How to host different file formats/types for a media attachment without creating multiple attachments?

Question

For a better browser compatibility we often have to maintain multiple format of a media.

For example:

1. Image - JPG & WebP
2. Image animated - GIF & WebP
3. Image transparent - PNG & WebP
4. Video - MP4, WebM, OGG

Multiple format file of a same media belongs to a single media attachments logically. Having multiple media attachment for same media file doesn't make sense.

How to handle this situation? How can I have multiple format file for a media attachment?

PS: This is not a question for how to convert files to different format. Assuming we already have converted files or code which converts uploaded files into appropriate formats and saves into filesystem.

Additional Research & Possible Solutions

(I will write a solution after I am satisfied with the research findings and have a solid direction)

Observations

1. Currently WP Doesn't supports multiple uploads for a single attachment.
2. WordPress attachments is single format.
3. Instead of <img /> a <picture> tag with appropriate source tag can provide browsers with multiple image formats. A good WP way of multi format image solution could be created if we could modify output of wp_get_attachment_image() function but it doesn't have a filter. A trac ticket has been created.

Possible Solution steps

1. WordPress uses meta field (_wp_attachment_metadata) to save file location information. We could save format of each files into a separate metadata and later use WP actions and filters to load it.

2. WordPress has image sizes. File formats have to created for each images sizes.

3. Video attachments needs multiple formats as well but cover image for video tag is an important component. Extract image from video frame and save into metadata for video attachments. (?)