Warning: Keep in mind that the only definitive answer you can get on a legal subject is from a lawyer trained in that area. If your defence in court is that you read a stack exchange answer that won't look good. Having said that I'll try to answer using information from the FSF FAQ.
Lots of people sell GPL licensed plugins, so you can make money this way. But the user is purchasing the ability to acquire the plugin, support, future updates, etc, not the plugins code itself.
- Your code must use a GPL compatible license, the best and easiest option is GPL.
- If your license forbids something the GPL allows it is not compatible. It must be as permissive as the GPL or more so. So no telling people they can only install on a single site, or limit to X users.
- Not all licenses are compatible, check beforehand, e.g. the BSD license is compatible, as is the MIT license. The Apache v1 license is not.
- See this article for more information on license compatibility: https://www.gnu.org/licenses/license-list.en.html
- You can charge for distribution, but you can't charge for the code itself.
- There's nothing preventing someone buying your plugin and redistributing the code once it's publicly distributed. GPL code is free by definition.
- You can charge for services such as updates and support, or other hosted services
- Lots of people have businesses selling or supporting GPL licensed plugins
How do plugins like Gravityforms protect their plugin from being redistributed to the wild ?
Since WordPress is GPL code, all code publicly distributed must also be licensed either as GPL, or GPL compatible.
The GPL states you cannot charge for code, but you can charge for distribution. So when you buy gravity forms, you're not paying for the plugin, you're paying for the downloading and acquisition of the plugin.
Once you have the plugin it is perfectly legal to burn it to a CD and mail it to 20,000 people free of charge. You won't get the support or updates ( unless you pay them again, or you have a support contract ), and it would be a pretty nasty thing to do ( they have a business! ), but it's perfectly legal.
No matter what you do, someone will figure out how to redistribute it for free, so stop worrying about it, those people were never going to pay anyway and won't get the support for updates.
The only method that appears to work is to do the functionality on your own server and use API keys, selling a service.
For the actual purchasing component:
Put a payment mechanism on your site, and use it to generate API keys. Make the user put these keys into their backend, or couple their sites URL with their purchase, and use that to enable the restricted features.
Paid for Plugin
Set up something like Woocommerce or Jigoshop and use the virtual downloadable product type. E-Junkie is nice if you want something simpler.