are you using the 2-pass mode? if you are, increase the bitrate, I use 900kps when using the h264 codec but you can go higher. The higher the bitrate the closer you get to the original.
Divx will usually get you the best quality, from my experience better than h264. But I prefer the h264 codec because it's free. From all the conversions I've done I never notice the problems you encountered, it might be a video player problem. I noticed the VLC player has the best quality when viewed on a computer monitor.
The best H.264 converter that I have used, FREE, is Handbrake(the DVD needs to be ripped to your hard drive first), or Fair Use Wizard.
Using HandBrake is a bit more intuitive.
The problems with the "ripples" might be due to interlacing.
Quote:Divx will usually get you the best quality, from my experience better than h264. But I prefer the h264 codec because it's free