They say WordPress is GPL, and therefore all plugins and themes made with it are supposed to be GPL. Fine, but if I spent three months coding an extremely complex app theme with the intent on selling it repeatedly for profit, such as a medical office scheduling system theme, then how can I protect my investment, if even a moderate amount?
In addition to the other two suggestions, there is another possible approach: move all of your custom-app functionality out of the Theme, and into a hosted web service, to which the Theme connects via API key. That way, redistribution of the Theme itself does not impact your custom app-based business model, because the app would require Theme plus valid API key.
This approach may or may not work, depending on the nature of your custom app, but it is a successful model for some commercial Plugins, and is fully GPL compliant.
Legality aside, I generally look at it this way, write good code and offer good support and people will come to you. There are many premiume themes that are GPL and are doing great. Look at WooThemes,Headway,StudioPress (Genesis) to name just a few companies that write quality, fully GPL themes and make a living doing so.
In my opinion some of their sucess is credited to providing quility support and pricing their themes at an amount which they can afford to live but others can afford to pay for them.
I think this idea of "If I make my theme GPL someone is going to steal it and all my work is going to be gone" is just false. Sure, maybe someone will steal it, give it away. But if you offer support people will still be come to you and get it. Not to mention the fact that they know what they are getting. Free/stole premium themes (and some non-premium ones) often contain spyware/malware. I'd rather pay someone for something I know works then deal with a virus later.
One last example (and maybe my favorite) is Justin Tadlock's Theme Hybrid, He releases it for free as GPL and charges $25 a year for support. A fee I gladly pay because his support is amazing.
Bottom line, if you create a trusted environment and people will come.
Another solution would be a terr solution, $X for the product, $Y for support, $Z for additional add-ons
PS: personally I don't buy anything for WordPress that is NOT full-GPL.
If you want to apply some legal restrictions to your product and stay in line with GPL practices of WordPress your best option is split license:
Something that hasn't been mentioned in this thread are the topics Encryption and Obfuscation.
Encrypting your code with IonCube or Zend Encoder are but two popular methods for protection themes and or plugins that I've seen in use.
The problem with encryption is that with enough will and desire you can decrypt the files back into their original state. Sometimes the results will vary and depending upon how well the the type of encryption methodology is understood will often determine the success or failure in decrypting files.
There are unscrupulous individuals who have become quite skilled at the art of decrypting files from IonCube, Zend and others. For the average person, the hassle with often outweigh the worth.
The next methodology is obfuscation which I have rarely if ever seen used. In my opinion it can make it near impossible to decipher files that have been properly obfuscated which in turn also means you can not edit files with obfuscation in the traditional manner and need to keep copies of your master files for any modifications, updates, bug fixes which usually isn't a problem.
However a combination of both encryption and obfuscation would make it near impossible if not absolutely impossible steal your proprietary code. It wont stop people from using it, assuming it functions, but it will stop people from modifying it or copying functionality to create their own similar product.
Using an API Key as mentioned above is the other great method to help secure your products BUT there is a downside to this method and that is by storing some of your application logic out of the original theme or plugin means that the user needs to connect to your server to retrieve that logic in order for the theme or plugin to operate properly.
This sounds like a great thing and it is for the most part but consider what happens if your server were to go offline even for an hour or two. Would this render your theme or plugin unusable? No doubt it would. Then you'd need to consider what kind of impact that would have on the end user.
You could circumvent this, as best as possible, by having some failsafe server locations handle the distribution of your API logic such as using cloud based services from reliable companies such as Amazon and more in addition to directly accessing logic from your server.
Then you'd need to weigh up the cost in overhead and ultimately the worth to you. Is it really worth the time? I guess that's project specific and dependant but considerations one must make ultimately.
The bottom line is that most people who will pirate or steal your product, theme or plugin are most likely to have never bought your product, theme or plugin in the first place.
There are often thought to be three types of people in our environment,
Although pirating and stealing of themes and plugins in rife around the Internet, the amount of people who actually use your themes or plugins consistently enough to warrant any damage to your bottom-line is somewhat miniscule.
Its not to say that we shouldn't do everything within our power to minimize that loss, but often your efforts would be better spent in creating more products and or marketing existing products further, as well as diversifying the way in which you offer your product.
With the rate at which many products either update with new features or fix bugs, it often renders previously pirated products useless or not as fruitful had it been paid for.
As mentioned above, Encrypting and Obfuscating code, combined, are two methods well worth further investigation in addition to API style integration, to help secure your products, themes or plugins in the best possible manner.
If you are selling it then it doesn't need to be under GPL as you can't sell it on WordPress's Sites. You can just distribute it yourself under whatever license you like. The GPL restriction is just for the Wordpress.org repository's, and seen as you can't sell it under Wordpress.org, you can have whatever license you like.