In general, should you try to visually adhere to default WordPress admin style (button shapes, colors, etc).
Are there any benefits of custom styling except for branding (I suppose that WP teams does some usability tests on their designs)?
Following the previous answers, in most cases I would stick to the WordPress default styles and patterns -- for design and also markup, functions, APIs and cod, not only the CSS --, specially if it's a plugin to the general public and to be hosted in the official repository.
That said, there will be times you will need to tweak, or event change completely, something. My advice is to build upon what WP does and change just what really needed or justified.
My company manages 3 WordPress multisite networks: A for our own clients, B for a specific niche managed sites/CMS/Digital Hub SAAS and C for other boarder niche managed sites SAAS.
And for plugins we release "in the wild", most of time we follow WP style/patterns.
It all depends. There are many cases where WordPress admin styling is not enough: complex controls like repeaters with one or more fields, complex nested settings and other things. If you have a lot of settings, default WP styling might be limiting.
On the other hand, there are plugins and themes that use styling that look very alien and over the top, with huge colourful buttons, replacements for default HTML controls and much more. They look very out of place, and can be confusing for the end user.
If you need to create simple controls and settings, it is best to stick to the WordPress default styling (same classes as WP uses), that will ensure that your plugin looks good with all WordPress versions. If your plugin has a lot of settings and you need a better way to organize things, you can develop your own styling that will remain close to the WordPress, but giving you more flexibility.
For all my plugins, I have my own standardized interface library that is based on the WordPress styling with few extra organizational elements, and so far, my users had no problem adjusting to my plugins UI because it looks a lot like WordPress.
This question reminds me the bad old days of the early java versions. java looked good if you focused just on its elements (well, good enough at least), but in the context of any OS it looked like an ugly bastard child.
No wordpress user is likely to avoid using your code because the UI do not follow conventions, but users hate to learn the new "snow flake" UI, especially when they have 20 plugin each with its own "snow flake" design.
UX design 101 is to always follow the conventions of the enviroment in which your code is running, and only if you do something drastically different, use different UX. Frankly even Apple and MS do not always follow this guide when they develop things for their own products on their own OS, but you are unlikely to be that big to not care.