any other way in Wordpress side to configure external user authentication but without developing anything?
No. The only authentication WordPress provides out of the box is cookie based authentication.
There are however plugin based solutions, specifically ones that rely on widely accepted standards that aren't unique to WordPress. These implement standardised authentication protocols such as:
- Oauth1, you can use this with the REST API to authenticate from Java
- Oauth2, like Oauth1 you could use this, I'd recommend this over OAuth1, assuming your site uses SSL/TLS/Https
- SAML, you could authenticate with a SAML provider, and use that instead of WP to handle your accounts, and install a WP plugin to make WordPress use the SAML provider too
- JWT, avoid if possible for security reasons, not just in WP, it's attractive but fundamentally flawed, I mention it for completeness
- Basic Auth, the simplest but least secure option
- Basic Auth with Application Passwords, marginally more secure than basic auth, but not as easy to build a nice UI around
- Rolling your own, you could implement a REST API endpoint or two to handle user login and registration, but this is a very bad idea. There's a reason people who figured this out wrote standards such as OAuth or SAML, think of it like rolling your own cryptography, unless you have a degree in the subject you'll build something insecure with bugs, not worth the hassle
Note that these protocols are not WordPress protocols, but general standards used across numerous pieces of software.
As an example, I have a CLI application that requires the user to be authenticated, using a WordPress site to store data/users/etc. In order to login the application uses OAuth1 to communicate with WordPress and exchange details.
For your Java application, you could communicate with WordPress via OAuth2 to authenticate your user, perhaps even show an inline dialog showing a login form to approve.
The REST API
You'll want to authenticate and communicate with the REST API, allowing easy JSON based communication with the WordPress site. This should simplify things, and provide data in a machine readable fashion that Java is happy to read and process. It'll also allow you to use common libraries to communicate with WordPress instead of building the API requests from scratch
Or the way by using Keyring or/and writing own filter is the right one?
This isn't a single problem, if you pick a standard then you can break this apart into a pure WP problem, and a pure Java problem. You'll get 100x as many results for implementing OAuth2 client in Java than you would for logging into WP with Java. The same for logging into WP with OAuth2.
However, Java questions are not in this stacks scope ( try stack overflow ), neither are plugin recommendations ( there's a software recommendations stack ).