Each of these have their own advantages and disadvantages. You might need to choose one over another, based on your needs. They are both well secured, otherwise they wouldn't exist in the core.
REST-API, The modern and well known API
The REST API was added to ( or better say, combined with ) the core more recently than Admin-AJAX. It is perfect to be used in mobile apps and API developments.
- Is simpler to write, develop and debug
- Does not need separate functions for logged-in and non-logged-in users
- The core already has some built-in handlers that speed up the development process
- The response can easily be used in applications or platforms that do not run on WordPress
- It does not produce any user-friendly response. The output is in JSON, which can't be used in some cases, such as SEO purposes. However, some might find this an advantage
Admin-AJAX, The ancient AJAX handler of WordPress
Admin AJAX existed in the core for as long as I remember, and is the way the core itself deals with the requests in the admin area.
- It directly outputs the content, which can be used anywhere, even by search engines. Some scripts such as Custombox only support this kind of response
- It has separate functions for logged-in and logged-out users. While you can do this with a conditional in the REST-API, some may find this useful
- Working with the response is easier, since all you need is to put it in a
div or wherever you need
- Since the output is plain HTML ( By default ) it shouldn't ( or maybe even can't ) be used in APIs and application development
It's very hard to say which one should be used, they are both useful handlers and if any of them wasn't secure, it certainly would not have existed in the core for so many years. So there is not a problem of security, but still if you are concerned about the security, you need an SSL certificate.
It's rather performance, type of request, and development platform that decides which one should be used.