I ran into the previous page from 2007 and wanting an answer myself I decided to research the issue using VidCoder 1.3.4.0 (x64). I'd love to add links to stuff, but I'm not allowed as I am a new user... =( Just use your favorite search engine for anything I list, and wikipedia to research any terminology I may use that you are un-familiar with.
Im using a PSP Go for playing video at the moment and have some profile settings that future forum searchers may want to see.
Id love to have other people give their input as well if they have additional advice for getting superior video quality while keeping the compression fairly high. (for a smaller file size) Remember that your files must be encoded as mp4s for use with a PSP, these settings should work for most PSPs.
Ive noticed that for playback on the actual PSP you dont need to go above the standard resolution of 480 x 272 and most video files will look good at 23.976 frames per second (fps), especially if you are aiming to make the file smaller.
I have my audio set to a throughput bitrate of 128 with 44.1khz in stereo, because I enjoy my audio. if you want to save a smidge of file space you can probably take this down to a bitrate of 64 and 22.05khz and maybe even mix sampling to mono instead of stereo. Id suggest sampling it to see what is to your liking.
The video bitrate can go pretty high on the PSP for video, up to 15000 Ive read, but this will yield a very large video file, and since the PSP doesnt really have much space to work with (PSP Go internal 16 Gig + 16 Gig Mem Card is still only 32 Gig) you may want to work with some more sane settings.
Ive noticed that for a great picture at 480x272 you usually dont need a video bitrate of more than 768, this still looks decent even when hooked up to an HD TV with the PSP video out, although the resolution is hard to miss the bit blocks are fairly well defined.
If however you are looking for a smaller file size a video bitrate of 512 works fairly well, looks great on the PSP itself and a bit buttery if output to an HD TV.
Now if you are really looking to save space, a video bitrate of 256 looks pretty decent on the PSP with blocking and smudging only apparent during parts with 1-3 tone colors occupying the majority of the screen, or during scenes of fast movement. This setting however looks absolutely horrible if the video is output to an HD TV.
Ive even encoded some simple videos to a bitrate of 128, but I only do this for videos that use sequential art instead of animation. (like Zero Punctuation Reviews)
I havent played with the higher resolutions since I just dont feel the need to occupy my storage with files so large, so feel free to add some commentary or guides for that.
Again for the highest settings I recommend:
480x272 resolution at 23.976 frames per second with a video bitrate of 768 and an audio bitrate of 128, 44.1 khz stereo.
For the lowest ok quality to view only on your PSP:
same resolution and framerate but with a video bitrate of 256 - 512 and audio bitrate no lower than 64, 22.05 khz, stereo or mono.
If you are using VidCoder you can alternatively use the constant quality settings for movies and large video files, with a setting of 25 - 26 the video files will be approximately the same size as a video encoded at a bitrate of 256. For a video file more comparable to a bitrate of 512 - 768 use a quality setting around 20, this will yield better overall quality of picture usually with a more manageable file size.
The other benefit of encoding with constant quality is VidCoder will only need to do one pass for encoding, (it won't do 2 pass with constant quality because it doesn't need to) The VidCoder team recommends using this setting to provide several benefits.
I hope this helps someone who searches for this in the future, everyone else feel free to add some commentary or your own guides.