This is not a complete answer, just an design direction. I think the best approach is something like this:
In the admin edit post
Rip all the shortcodes from the saved post and render it inside a metabox, aside from the editor. Make shure they appears in the same order as the shortcodes occur in the tiny Editor.
In tinyMCE javascript API
Make a jQuery function, when user click on a shortcode, it swaps the HTML from the metabox into the editor. And vice versa. The order itselfs should be ok as association, but Im not shure about enclosing shortcodes. However there is many ways to design a nice ID-connection. The updates of shortcodes can be done on the fly with ajax.
Never save the rendered shortcode state
Before switch editors, save drafts, autodrafts and publish, make a API call to trigger the restore, so the rendered shortcode state never get saved...
This can be done, but you need to be familar with the tinyMCE API to understand where and when to access the contents of the editor, and hook into javascript actions before 'save' and more.
There can be several tinyMCE editor on the same edit post pageload.
The restore part can be investigated by looking at the [gallery]
shortcode beaviour. But the click on [MY_SHORTCODE]
has to be done by some jQuery tricks.
in the admin_footer script, access the content of where the cursor is active with:
var $editor_content = $(tinymce.activeEditor.getBody());
is a hint of how to start.