Is it possible to choose not to use WP Bakery on a specific page even if it's used for the rest of the site? Would I be able to upload and use my own template (i.e template-pagename.php) + custom CSS for that one page while leaving the rest of the site intact?
Yes, it's possible. You can do it with Freesoul Deactivate Plugins.
I mean, use FDP to totally unload it on the frontend, so WPBakery will not load any asset, and will not add any database query on that page.
If you simply want to edit the page without WPBakery, you don't need FDP, just use Gutenberg or the classic editor, but WPBakery will load all on that page in any case if you don't disable it.