- Go ahead and reduce polyphony on the non-commercial version. 6 voices is enough.
- Don't remove any features like OSCs or filters or mod-routings... that'd be lame and people would continue to steal.
- Limit the number of concurrent instances. Two is enough, but four would be better.
- Here's the really neat idea: enforce a cc-by like license. So that songs/albums that use the software must (legally) list the software being used. Free marketing for you!
- Of course, no technical support. (Support is where a whole lot of cost goes for software.)
Yes, you will get fewer sales of the commercial version. But I imagine the non-commercial version will sell profoundly well. I think in the end, you'll end up with more overall income.
My two cents.