Just because something is not in the media library it doesn't mean that it is not used in your site or by some other site. The problem that you may run into is deleting a file that it is actually used.
On the other hand, just because a file is "located" in the media library doesn't mean it is actually used.
Therefor a better question should be "How to identify media files which are not used" for which unfortunately I have no answer and would be off-topic on the wordpress development site.
I just don't delete files. It is a waste of space but unless you use services like S3 in which you pay for each byte of storage the risk of going into the trouble of locating the file in the backup just not worth the small saving you get from deleting the files.