A while ago I wrote a question asking about storing easily queryable data associated with a given attachment.
The recommendation that I received at the time was that I should avoid using the WP attachment metadata because it cannot be quickly queried for. Instead, what was recommended (and what I opted to implement) was to store all of the values in an array within an option. This works, but the design of the thing makes me want to cry. It goes agains every fiber of my being which says, "that should be its own table!" See what I mean: http://pastebin.com/hBfrxaiH
So my question is, under what circumstances should a WP developer consider creating a custom table to support their plugin? I know that custom tables are generally frowned upon, but I feel like what I'm currently doing is an abomination and really would like to achieve my plugin's needs in a cleaner way.
My thinking with a custom table is that not only would it clean up this design disaster, but I would also have a higher degree of control over how and when values are loaded from the DB rather than just pulling EVERYTHING out before being able to do anything intelligent with the data. Additionally, unlike the current solution, a custom table would scale -- the current solution most definitely does NOT scale.