Searching this Stack, I can see two possible solutions for this question (not tested).
The answer is not fully developed, but can provide some insight.
Restricting access to files within a specific folder
How to Protect Uploads, if User is not Logged In?
Frank Bueltge's answer seems interesting but the code is quite complex. As he himself admits his english is nasty, but I can assure his coding skills are superb :)
And the answer provided by hakre could be operational:
- Protecting one or more folders inside
.htaccess as detailed there.
- Moving the desired files to this upload folders according to User or Post-ID, and for that the answer I provide to this question should do it. And, with the examples provided, maybe you can think of other kind of filtering.