Actions do things. Filters modify things. You do stuff in an action, whereas if you apply a filter, you do not expect any events or actions to occur, other than the modification of the value you're filtering. e.g. It's not okay to send an email or save a DB value in a filter, but it is okay in an action. There may be rare moments when you need to violate this convention but it is a useful and necessary one.
By replacing all add_action and do_action calls with their filter counterparts, you remove the distinction and force the knowledge of wether names like 'the_content' etc are actions or filters out of the world, and into your memory, encouraging mistakes.
So instead of making your code harder to read, realise that if your code is slow, it isn't because you didn't shave an instruction or two here and there, it's either your slow server, or your algorithmic structure of your code and the nature of the data being handled. You would get far more from refactoring your processes, upgrading hardware, or changing your paradigms.
To put it into perspective, when WordPress runs, PHP executes millions of instructions. Swapping actions for filters would barely make a dent into the top 1 thousandth of a percentile.
http://fabien.potencier.org/article/8/print-vs-echo-which-one-is-faster
I have tried on a fresh Wordpress installation. The script halts
before it ends with a "Bus Error" on my laptop, but the number of
opcodes was already at more than 2.3 millions. Enough said.
If you really want to try, look at how the guy used the VLD and test out your theory for yourself.