There are several caching mechanisms in WordPress and their mechanics differ, depending on choice of object cache (native or not):
+-----------+-------------------------+---------------------+
| | Native | Object cache plugin |
+-----------+-------------------------+---------------------+
| Option | Persistent (database) | N/A |
| Transient | Persistent (database) | Persistent (varies) |
| Cache | Non-persistent (memory) | Persistent (varies) |
+-----------+-------------------------+---------------------+
In a nutshell what this means is that transient is always persistent (will survive between page loads unlike Cache natively), but it will make use of customized storage if provided (unlike Options).
This makes transients most versatile choice for caching.
However with flexibility comes undercurrent complexity and there are quite a few of nuances with them (such as limit on name length, different behavior with and without expiration, lack of garbage collection) that make them more complex than they seem.
Overall:
- use Options for saving things that must be persistent
- use Transients for caching anything else
- use Cache when you have very good grasp of all three and know that Cache fits use case better than others (which won't be often)