With raw SQL through
$wpdb, I believe that I could do something along the lines of this (untested):
SELECT post_id, meta_value FROM postmeta WHERE meta_key = '_wp_attachment_metadata' AND meta_value LIKE '%:"custom_key";a:%'
That said, if there is built-in functionality for this kind of query then I would rather not reinvent the wheel. All of my codex digging has just returned ways to get the attachment metadata for a given id or set of ids, which will not do what I need. I could also retain a list of IDs with this particular array key, but I'd rather not keep this data since it seems like extra clutter...