Let's say you have a cleaner library. Is it okay to limit the free version that is hosted on the WordPress-repo to max clean 10.000 entries; after that you would have to pay for the pro version?
Trialware is not permitted (props @jacob): https://developer.wordpress.org/plugins/wordpress-org/detailed-plugin-guidelines/#5-trialware-is-not-permitted
With the only exception being
Attempting to upsell the user on ad-hoc products and features is acceptable, provided it falls within bounds of guideline 11 (hijacking the admin experience).
Sounds like the feature must be in a separate plugin itself to be within the guidelines for submitting a plugin to WordPress.org
Just make sure you're not that guy that has a plugin that's super annoying trying to upsell :)