I'm not sure what exactly you mean. What can you do in Windows that you can't in Linux?
unlike the majority of closed source windows apps
I think we should point out to NinjaRooster that the programs can be platform independent. Meaning the same program can be compiled to run on any type of device running the OS from a Linksys router, Mac OSX, Tivo, Sun, Intel PC etc..
unlike the majority of closed source windows apps
Yes, that's what I love about it. Even some free Windows apps are closed source. But in open source software, hell, you can even submit your changes back to the project!
I'll give it a try when I get the CD
What platform are you interested in running the compiler?
If you're using Windows, since you're starting, and don't need a pro version, Microsoft offers this great free compiler
http://msdn.microsoft.com/vstudio/express/visualcsharp/
Visual C# express, is free, powerful, and easy to use. I highly reccomend it. They also have others for Java, C++, VB, and web development as well, all free express editions. Make sure to get the SP1 patch, and if on Vista, you'll need a Vista patch.
I still recommend C++ it's the best you're gonna get without learning assembly and it's quite easy to pick up.
Sure pointers and the myriad of extensions can be a bit mind blowing, but it's very good just to get in to programming.
And don't put any effort into learning java, you'll get taught it at University so save yourself the hassle and just wait.