It depends on the plugin. This is my basic structure for nearly every plugin:

my-plugin/
    inc/
        Any additional plugin-specific PHP files go here
    lib/
        Library classes, css, js, and other files that I use with many
        plugins go here
    css/
    js/
    images/
    lang/
        Translation files
    my-plugin.php
    readme.txt

This would be something that would go in the lib folder.

If it's particularly complex plugin, with a lot of admin area functionality, I'd add an admin folder to contain all of those PHP files. If the plugin does something like replace included theme files, there maybe a template or theme folder as well.

So, a directory structure might look like this:

my-plugin/
    inc/
    lib/
    admin/
    templates/
    css/
    js/
    images/
    lang/
    my-plugin.php
    readme.txt