Disabling plugins entirely means you loose many advantages.
There are distributions of wordpress that go further and rip out posts and links etc, but they'll always lag behind WordPress core and tend not to survive as long.
Here are some things that could be done
Putting this in your wp-config.php:
define( 'SHORTINIT', TRUE );
Or defining it somewhere before you load in wordpress, should reduce the loading process and pull it back to minimal core functions.
Secondary lightweight installs
Setup a second wordpress installation, with only the plugins and themes you want (if any). Then configure the wp-config.php to use the same wp-content directory and database values.
WordPress, the original BBpress and glotpress amongst others are built around the BackPress library. You could use this instead of WordPress to do your work though numerous APIs and features may be missing or in need of re-implementation/porting
I would also bear in mind that WordPress loads the entire options table into memory to cut down on queries, if you're saving any large values in there it'll impact on performance.
You could also try hooking into earlier functions in the WordPress load process and exiting the PHP process before WordPress is finished, but I can't advise on how safe that would be.
Selectively loading plugins
There is also this article on selectively loading plugins, but it does require hacks to the wordpress core files