You're worrying too much about something that doesn't mean much. WordPress doesn't care how many PHP files you have; the PHP interpreter does. And the PHP interpreter really doesn't care about the number of files until it actually accesses those files and executes the code. And for all purposes, the interpreter doesn't access a file until told by the code.
Modern servers and PHP are so fast that you're not going to see an appreciable difference in the speed that the interpreter reads files. With 10,000 files, maybe there will be a slowdown. But that's more the server, the OS, the storage (HD or SSD), RAM, etc., dealing with the files themselves and the directory structure not the PHP code in each file.
Now, what is in those PHP files will make the difference. The speed of the PHP interpreter depends on the CPU of the server. Slow shared server = slow PHP execution. And if you're trying to calculate pi with PHP, you're find out that PHP will use lots of CPU and will be slow, depending on the server.
And if you're using PHP to call the database with complex queries, or calling external resources that depend on other servers, those are more performance hits having to do with PHP execution and not the number of files.
Yes, a plugin can be slow, even with only one PHP file, if you're doing extravagant things in that one file. The PHP code and what it does is the issue, not the raw number of PHP files. The files - and the code in them - just sit there, waiting to be executed; the execution is what takes the time and impacts the performance.
WordPress will load the main PHP of a plugin, and then PHP does what it is told to do in that one file; and if that main PHP file tells PHP to load 1,000 more files, and there are executable hooks, database calls, etc., in each file, then yes, the plugin could be slow.
Edit: And, as pointed out in comments, I'm not even taking PHP caching into account, and that can make a huge difference in performance of the OS, file system and the PHP interpreter.