|
|
|
Managing Files-A Loss-less Plean for the Future and Beyond!
|
|
|
jumbalia
Junior Member
|
11. February 2005 @ 15:00 |
Link to this message
|
|
OK, like most DV n00bs I am crawling my way forward, slowly forming a viable DV editing solution.
-I have a Sony DV Camera
-I have Firewire
-I have 200 Gigs of HDD space devoted to DV files
-I have a DVD burner
-I can make DVD movies
Today my problem is this:
I want LOSS-LESS storage of my DV files.
I see several possible solutions here (none I like)
1) Make 3 data DVD backups for each 12 Gig AVI file
2) Convert to Mpeg and make a data dvd backup
3) Keep on Tape (yuck)
4) Buy a ka-Jillion Gig Hard Drive and store raw avi files (double yuck)
Problems with each plan
1) I tried two ways to do this. I created a 12 gig avi file, but when I tried to burn it to DVD (as a data dvd) my burner program didn't span to muliple disks. (Do other programs work better?)
I then tried to MANUALLY CAPTURE the movie in three files. This was annoying and got old REALLY quick.
I can't find a program that will chop up an original into multiple files.
2) My second plan. I thought to myself "Jumbalia, why store avi files? It's not like you are going to make professional movies."
So...I was thinking that I'd convert it to "DVD Quality" and then I'd be OK because DVD's are my destination anyway....right?
But then I read comments like "..that Mpeg encoder sucks..."
So I am afraid that in 10 years I will be unhappy with the resulting conversion files.
3) I don't want to keep on tape for two reasons; Cost and deterioration.
4) Yeah....sure. I can buy a 300 gig HDD for 150 bucks, but that only like 20 tapes. And what if the HDD crashes?? Nah....DVD is best.
This message has been edited since posting. Last time this message was edited on 11. February 2005 @ 15:02
|
|
Advertisement
|
|
|
|
|
Mark7
Newbie
|
12. February 2005 @ 12:47 |
Link to this message
|
|
You?re wasting a lot of valuable space by using poorly encoded avi files. If you want to stay with avi files (although I can?t imagine why) so they take up much, much less room, then use a DivX encoder. It'll shrink a standard avi file considerably - to about 15-25% of it's original size without a loss in quality. But DivX may not be around in the 10 years you're talking about, although you could easily maintain a DivX codec on your DVD's for later use.
You're better off, though, using a good mpeg2 codec to compress your video files for safe keeping..but use a higher quality mpeg2 codec. Pegasys-inc.com offers their "TMPGEnc 3XPress" software that has a very good mpeg2 encoder, and is easy to use. Incidently, I've noticed that this software has become a "must have" item for serious video fanatics within the past year or so. 3XP will produce excellent mpeg2 files that will be better than the avi files you're now making. But to make a playable DVD from mpeg2 files you'll need to have a DVD Authoring program. It makes the .BUP, .IFO and .VOB files required by DVD players.
The 12 gig file you mention is unnecessary. A two hour movie encoded to mpeg2 properly will easily fit on a 4.3 gig DVD. Much less (about 2.7 gigs at 4000 kbits/sec encoding rate) if you want to drop the encode rate down to 4000-5000 kbits/sec. DVD players are mostly limited to 8000 kbits/sec, but encode rate above 4000 have the same visual quality usually.
There's no point in keeping your files in an uncompressed format using up far too much space on your hard drive!! And not too worry, mpeg2 will be around for quite a while. A lot of digital video on cable is now shifting to mpeg2 for quality. Currently, I capture exceptional quality mpeg2 video signals directly off a set top satellite receiver right now, for instance.
Tape: NEVER, NEVER store video on tape for any length of time!!!! Get your recording to a hard drive then onto media as soon as you can!!!! I've been burned in the past by tapes (as has every video fanatic that I know!!)
Which Sony camcorder do you have? If it has a ccd size of less than 680,000 bits than Firewire will work for you. If it's above that and you want to keep higher quality video then you're going to have to go to a video capture card to get the file into your computer with very little loss....
P.S. I chuckled over your comment: "Buy a ka-Jillion Gig Hard Drive and store raw avi files (double yuck)"
First off you would never want to store raw avi files. But the chuckle came when I read your comments, and then looked over my work area. I have 4 computers in front of me, two have six 300 gig hard drives (I use HiPoint pci extender cards to add extra drives). The other two systems have 2 and 4, respectively, 300 gig hard drives along with a couple 250 gig units. Most importantly, three of these systems have ATI video capture cards to capture video at very high resolution.
I started out nearly 6 years ago with essentially the same question that you have: How can I save valuable (to me and family) priceless family video's. As you can see, my hobby has gotten a bit out of hand!! So watch out!
This message has been edited since posting. Last time this message was edited on 12. February 2005 @ 12:50
|
|
ZombyHero
Junior Member
|
12. February 2005 @ 21:33 |
Link to this message
|
Alright, there seem to be a few misconceptions floating around in these posts.
First: Firewire is an appropriate capture method for ANY video encoded in 4:1:1 25Mb/s video. This includes miniDV, Digital8, DVCAM and DVCPRO. CCD size has nothing to do with it (and CCDs are usually measured in physical size, because pixel resolution often has little bearing on overall quality).
Second: If you originally captured your video on DV tape, its already been compressed by a lossy codec, and there's nothing you can do about it. Very few cameras exist that capture video uncompressed, and they're all very, very expensive). Once captured to tape, your video has been compressed with the color sampling at 4:1:1, data compression of 5:1 and bit rate of 25 Mb/s. For more information on DV compression, check out: http://www.adamwilt.com/DV.html
If you do all your editing and rendering using a DV25 codec (Microsoft DV in Premiere and DV/DVCPRO NTSC in Final Cut), you should see no quality loss in your video. DV25 has a storage rate of about 13GB per hour vs. somewhere around 160GB/hour for RAW uncompressed video. Saving DV25 footage as uncompressed will not increase the quality one bit.
As far as storage is concerned, keep a few things in mind. Encoding in mpeg2 WILL further compress the video. Using a proper encoder you may be able to achieve pretty high quality, though. Its going to be a decision between video quality and how much you can fit on a disc. There is a visual difference between recording 1hr per disc and 2hrs per disc. Also, there's a lot of people experiencing issues with CD-R "rot" (myself included) after a few years. CD-Rs and DVD-Rs are not made the same way commercial CDs and DVDs are made (commercial discs have grooves written into them, CD-Rs use a transparent dye). So consumer DVD-R discs might not have the long term storage capability you seek.
Tapes are not necessarily a bad idea, if you buy high quality master DV tape, and store them in a proper environment, they may last quite a long time. Another option would be to transfer them to DVCAM or DVCPRO tapes, which use larger tape tracks, and higher quality materials to be less error prone.
Hard drives are also an excellent idea, but obviously expensive. One thing to keep in mind is that if a hard drive crashes or is corrupted, there's a good chance the data would be very difficult to recover. Often tape is easier to recover because only certain sections of it become corrupted.
This isn't an easy question to give a definite answer to, it all boils down to money and what quality you'll be happy with.
|
|
Mark7
Newbie
|
13. February 2005 @ 12:16 |
Link to this message
|
|
First: You don?t capture with Firewire!! Firewire is ONLY a data transmission method using the IEEE 1394 specifications.
Secondly: Pixel resolution has EVERYTHING to do with quality. While physical size is a poor judge of quality because the manufacturers continue to shrink the physical size and increase the pixel sensing ability to improve quality. Look at the quality between a 375,000 pixel ccd and a 1 meg pixel ccd.- there?s no comparison unless your quality acceptance level is pretty low.
Thirdly: The specific problem with transferring the compressed data on the video tape is not specifically with the Firewire data rate (it can keep up - but barely); it?s because most camcorder manufacturers use Microsoft?s DV25 codec!!!! Consequently, the software manufacturers use Microsoft spec?s to try to turn the file into a good quality video presentation. If you?re accepting the output of the currently available video massaging software as great, then you?ve got a lot to learn!! By playing back a video using the camcorder manufacturers processor and codec you are NEARLY nulling out the problems inherent with Microsoft?s codec. In other words, the camcorders own processor will undo the codecs ?mistakes? as it converts the video to composite output. This is an easy item to prove. Capture the composite output of a high res camcorder using a high resolution video card, then transfer the same file via Firewire to ANY of the leading video programs. Blow up the frames from each and you will be startled by the difference. I, and a considerable number of others, have run through this sequence every time there?s been a ?New, Improved? video editor on the market - with the same results.
Your concern?s about CD-R "rot" are well founded. However, the major CD producers claim they have gotten around this issue by better edge sealing techniques; but, only time will tell. The same companies have made statements that their DVD production will never have this problem because of the research they?ve done. Here again, we?ll just have to wait and see, although I?ve not read or heard about anyone running into this problem with DVD blanks - so far. But it is a concern that lurks in the background!
Tapes: Industrial archival tape will stand up to some severe handling - but even then, when I was still a corporate manager, tests indicated that large mag disc paks were more secure for long term data storage. But either required storage in controlled environments - something most users can?t do. All to frequently the camcorder tape is stuck in a storage container of some sort, then banged around over a few years time. Then, when you want to retrieve the data you have not-so-good quality. You could install a 9 track tape system for storage - but I doubt that many users will really want to go to this extreme.
One thing that we both agree on whole heartedly is your statement:
?This isn't an easy question to give a definite answer to, it all boils down to money and what quality you'll be happy with.?
.
This message has been edited since posting. Last time this message was edited on 13. February 2005 @ 12:18
|
|
Mark7
Newbie
|
14. February 2005 @ 03:56 |
Link to this message
|
|
ADDENDUM: Yesterday I recorded my wife's Choir group as they performed. I then did my normal procedure of capturing the video to a hard drive using the composite video output from the camcorder, then compressed it as an mpeg2 file using 3XPress, then authored it as a viewable DVD with DVDAuthor. The mpeg2 final file was 2.944 gigs in size.
Then, I recorded the same file from the camcorder tape to disk by using Firewire and Windows Movie Maker (with the resolution set to DVI-AVI at 25 mbps). The resultant file produced was 10.985 gigs in size. I then burned a DVD from this method.
Later, when the choir group came by our home to practice, I first played the Movie Maker version, asking most members what they thought of the DVD. Most comments were the polite "Very nice", "Looks good", etc., etc.
Then I put in the mpeg2 version for all to view, then asked for comments. "Wow, what did you do?", "The difference is amazing!!", "It looks like HDTV in comparision!", etc...
And there is even more of a difference if you blow the individual frames up. For instance: As I would zoom in on each memebers head and shoulders, the intricate design of the embroidery insignia vas clear and concise on the mpeg2 version - but on the DVI-AVI based version the same insignia was not clear at all - more of a fuzzy image that you could recognize only if you knew what it was.
AND - I could easily have reduced the mpeg2 encoding down to about 2 gig in size by dropping the encoding rate - and the output would still have been somewhat better than the DVI-AVI version.
Now, back to jumbalia's question... How can a video file be stored and retrieved later RELIABLY. My answer still stands - at this point in time, compress it to mpeg2 with a GOOD encodec. Mpeg2 will be around for quite some time as it is rapidly becoming THE standard that should last for the next few years at least. Regardless, there will always be archival codecs for these files for a long time to come anyway.
.
This message has been edited since posting. Last time this message was edited on 14. February 2005 @ 04:03
|
|
jumbalia
Junior Member
|
14. February 2005 @ 14:22 |
Link to this message
|
Mark7, you are awesome!
(ZombieHero, you need to have a few beers and chill out. I'm buying)
I will reiterate Mark7's advice to me and see if I have it right
BUT FIRST!
An anecdote from church. I asked around at my church and was directed to the guy who handles our video. Well his advice was VALID, but FLAWED.
His advice:
Step 1: Buy a $1800 video editing package
Step 2: Buy a $1000 optical disk drive.
Sure.....Maybe I should upgrade my $600 DV camera first.
Anyway, So this is what Mark7 is telling me:
-Transfer my DV (via FIREWIRE!!!!!) to my HDD
-Encode to Mpeg2 using a high quality Mpeg2 encoder such as the TMPGEnc program from Pegasys-inc
-Burn the MPeg to DVD for storage
-ERASE the DV tapes and ERASE the .AVI files
Cool?
(BTW I searched for "TMPGEnc 3XPress" in Google and guess what I got....THIS VERY thread! Go figure.)
|
Senior Member
|
16. February 2005 @ 03:08 |
Link to this message
|
Well, for what it's worth. This is what I do.
Tranfer the video to my PC, make a nice edited movie (usually ending up with using not more than between 10 and 15% of my origianl material).
Encode it two MPEG-2 and put it on DVD (and a second one for backup).
Play my edited movie back to tape through DV-in to preserve an "original quality" backup.
Delete the avi's from my PC.
Maybe I should transfer the tape to a new one once every five years or so...
Unless, my house burns down I think I am covered well...
http://www.digitalvideoclub.com
|
Staff Member
2 product reviews
|
16. February 2005 @ 12:34 |
Link to this message
|
*Removed because I re-read the 2 posts.
Quote:
Thirdly: The specific problem with transferring the compressed data on the video tape is not specifically with the Firewire data rate (it can keep up - but barely);
Really? Firewire has a transfer rate of 50MB/second and some quick math tells me that 13GB/3600 seconds is less than 4MB/second. There can be transfer issues but they're not caused by the firewire connection.
Quote:
it?s because most camcorder manufacturers use Microsoft?s DV25 codec!!!! Consequently, the software manufacturers use Microsoft spec?s to try to turn the file into a good quality video presentation. If you?re accepting the output of the currently available video massaging software as great, then you?ve got a lot to learn!! By playing back a video using the camcorder manufacturers processor and codec you are NEARLY nulling out the problems inherent with Microsoft?s codec. In other words, the camcorders own processor will undo the codecs ?mistakes? as it converts the video to composite output. This is an easy item to prove. Capture the composite output of a high res camcorder using a high resolution video card, then transfer the same file via Firewire to ANY of the leading video programs. Blow up the frames from each and you will be startled by the difference. I, and a considerable number of others, have run through this sequence every time there?s been a ?New, Improved? video editor on the market - with the same results.
The choice of codec is indeed important, but your explanation isn't quite accurate. First off you're talking about looking at it in an editing application which will use the MS DirectX filter for decoding if nothing else is installed to override it. Second the MS DirectX filter uses the Panasonic codec (which is the same as the Sony codec). I agree that this isn't a particularly good codec. In fact it sharpens the image. The only other 2 codecs that I'm familiar with are the ones from MainConcept and Canopus and they also have some issues. The MainConcept codec sharpens more than Panasonic/Sony/MS codec does and will therefore alter the picture even more. The Canopus codec doesn't sharpen at all, but has a chroma upsampling bug that will cause a smearing effect. This can be fixed by frameserving with AviSynth and using the FixBrokenChromaUpsampling filter. These links demonstrate the differences between the 3 codecs:
Panasonic/MS/Sony
http://mywebpage.netscape.com/wlopensource/TestSignal_Panasonic_RGB.png
MainConcept
http://mywebpage.netscape.com/wlopensource/TestSignal_MainConcept_RGB.png
Canopus
http://mywebpage.netscape.com/wlopensource/TestSignal_Canopus_YUV2.png
And here's a comparison between frames viewed with the Canopus codec with the chroma bug present and then fixed:
Bad Upsampling
http://mywebpage.netscape.com/wlopensource/JW_Canopus_YUV2.png
Upsampling Fixed with AviSynth
http://mywebpage.netscape.com/wlopensource/JW_Canopus_YUV2_fixed.png
In order to use the Canopus codec you also need to change the FourCC of the file. You can download both the codec (actually it's technically only a decoder unless you pay for it) and a program to rewrite the AVI files with the correct FourCC code from the Canopus website. For a faster conversion you can also find other programs that will change the FourCC without rewriting the file.
As far as storing the data losslessly, the transfer to your computer is completely lossless. It's not until you open it up and save it again that you have to worry about what the codec might be doing to it. If you want to cut it into smaller pieces and store it a better method would be to use a program like WinRAR or WinZip to create a segmented archive. You can use WinDV (freeware) to transfer to your PC (I don't like to say capture because that's not really what it is) via firewire without ever opening an editor.
Personally I transfer with WinDV, use AviSynth to open the files, correct the chroma upsampling and do any editing, and then encode with Cinemacraft Encoder Basic which costs $58. I find it to be better than any TMPGEnc encoder I've ever tried although the best for interlaced content is supposed to be Canopus Procoder. Unfortunately Procder costs $500 compared to well under $100 for either CCE Basic making it the second most expensive piece of MPEG encoding software on the market (CCE SP is the most expensive at $2000).
This message has been edited since posting. Last time this message was edited on 17. February 2005 @ 13:45
|
Senior Member
|
17. February 2005 @ 22:27 |
Link to this message
|
Reading this thread again, one question comes up in my mind:
My assumption has always been that when I edit (with Premiere) and export my work with the option recompress off, non-edited frames are directly copied from the original files. Or do they actually get decoded and compressed again?
I would think it is the first, but I maybe wrong.
If my assumption is correct, only edited frames (which, in my productions will less than 1% of the total; mainly transitions and occasionaly PIP) are the only places where a codec can have a influence on quality on the path: transfer to PC - edit - transfer back to camera. This would definitely support my case to at least also send a copy of the edit back to tape and preserve it as "original quality".
BTW: I also assume that editing of audio does not require recompression of the video, because if that would be the case, suddenly more than 75% of my franes would become edited frames...
|
|
dinghao
Newbie
|
1. March 2005 @ 18:07 |
Link to this message
|
I specifically joined this forum just so I could say thanks....to all the posters, and especially to Mark7. I recently got a Panasonic PV-GS120 MiniDV camcorder (well, first I got a Sony DVD camcorder, then took it right the hell back, but that's another story).
I'm totally new to DV, and my first attempts at capturing and moviemaking, with Dazzle's MovieStar and Movie Maker, were a little unsettling.
However, going on information I gleaned from this thread, I tried using WinDV to capture, then TMPGEnc to encode to mpeg2, and the results were MUCH better.
Even in full screen with WinDVD, the picture was quite smooth and acceptibly sharp.
With my first attempts I seemed to be having a lot of problems with dropping frames, but WinDV fixed that. But then, pulling the AVI files into Movie Maker (MovieStar) editing and outputting still seemed to result in quality problems in the final product.
But, at the very least, now I know I can get good results with
camera -> AVI capture with WinDV -> mpeg2 encoding.
Now, I guess the question is, is there a way I can edit and combine and split and add effects while generally maintaining this quality? Seems when I try to use MovieMaker or MovieStar, quality goes downhill pretty quickly. Do I need better software or something? Or am I using the stuff I have incorrectly?
Thanks a bundle,
ding
|
Senior Member
|
1. March 2005 @ 23:43 |
Link to this message
|
I suggest you better start a new thread for this...
Anyway, if you use DV-AVI in your editing program and export it as DV-AVI again, you should have no loss of quality whatsoever. Just make sure that you export it in DV-AVI again.
For a list of some video editing programs check at the bottom of this article: http://www.digitalvideoclub.com/basics/editing.php
|
|
dinghao
Newbie
|
2. March 2005 @ 06:55 |
Link to this message
|
|
Yes, you're right about the off-topicness of my post, sorry about that....I just posted to say thanks, then got carried away and asked off-topic questions. I realized that just after I posted. Mea culpa :-)
Anyway thanks for the additional info, and I'll research the site for more information regarding the questions I asked.
ding
|
Staff Member
2 product reviews
|
2. March 2005 @ 10:17 |
Link to this message
|
Quote:
Anyway, if you use DV-AVI in your editing program and export it as DV-AVI again, you should have no loss of quality whatsoever.
If you follow the links in my earlier post you should see why this isn't really true for most people. Most editing apps edit in RGB which means anything that's changed during decoding (like sharpening in the Panasonic or MainConcept codecs or chroma upsampling errors in the Canopus codec) will be incorporated into the final video when it's compressed again at the end. If you used AviSynth to open the file with the Canopus codec and correct the chroma upsampling and then frameserved to your editing app you might not get any loss (I haven't tested saving back to DV) but if the decoder makes any changes (and aside from the method I just mentioned they always will) you'll end up with destructive changes to the source every time you decode and re-encode.
This message has been edited since posting. Last time this message was edited on 2. March 2005 @ 10:17
|
Senior Member
|
2. March 2005 @ 10:31 |
Link to this message
|
Well Vurbal, technically you are right, but I disagree if you say "why this isn't really true for most people".
"Most people" (and specifically "most people" in this thread) don't edit all of their frames. They simply cut and paste and use some transitions and therefor probably end up with less than 1% of their frames recompressed. The other 99% will be "smart-rendered" as long as you don't go messing around with several DV-codecs installed on your system so they will remain unchanged.
Furthermore, I think that the poster of the question refers to another level of quality decrease as may be the result of recompressing DV-AVI to DV-AVI...
|
Staff Member
2 product reviews
|
2. March 2005 @ 11:45 |
Link to this message
|
|
I can't speak for all editing programs since I've only used Premier 6/6.5 and AviSynth but I can say for a fact that Premier doesn't just copy frames that aren't changed. It re-encodes every single frame from the decoded RGB. I can't imagine other programs are any different.
This message has been edited since posting. Last time this message was edited on 2. March 2005 @ 11:48
|
Senior Member
|
2. March 2005 @ 22:26 |
Link to this message
|
I again have to challenge that statement.
In the viseo export settings in Premiere there is a box called "recompress". As far as I know if you uncheck that, unchanged frames are copied. I did not perform any in-depth mathematical frame comparison, but that is what I see stated in several threads in the Adobe Premiere forum (I also would not know what else this box would be for).
I also found this official Adobe technical document:
http://www.adobe.com/support/techdocs/318277.html
Read the very last sentence of that article....
If you have hard evidence that we are being mislead by Adobe, I certainly would like to get that.
|
Staff Member
2 product reviews
|
3. March 2005 @ 04:31 |
Link to this message
|
|
Since you posted that I did some more research and I stand corrected with regard to Premiere. As long as you have Recompression Always unchecked in Premiere 6.5 it will only compress frames that have been altered. Additionally Pinnacle VideoStudio & MediaStudio apparently has an option to 'Perform SmartRender' that will do the same thing. So the real key here appears to be making sure those options are set correctly. I also have to assume that Avid's software has a similar option. Thanks for the correction on that.
This message has been edited since posting. Last time this message was edited on 3. March 2005 @ 04:33
|
Senior Member
|
3. March 2005 @ 05:05 |
Link to this message
|
|
OK, thanks for letting me know. I recognized your comments here as knowledgeable, so you seriously made me doubt ;-)
I guess the only thing that can be wondered why the Adobe guys hide this setting somewhere in a corner of a window (where many people will not look) and why the default setting for "always recompress" is ON. I don't see why anybody would NOT want to use smart rendering...
|
Staff Member
2 product reviews
|
3. March 2005 @ 05:39 |
Link to this message
|
|
I have to agree with you on that. If you're marketing to the pro-sumer market it would seem to be a pretty important setting. Oh and I also found out that if you use an MPEG-2 encoder plugin to send the output directly to a third party encoder like CCE or MainConcept Premiere apparently uses the original stream for untouched frames by default (not even sure if this setting can be changed).
|
|
billbrem
Newbie
|
5. April 2005 @ 12:22 |
Link to this message
|
Mark7 makes an extrodinary claim: That transferring video using the analog composite output of a camcorder is superior to a Firewire connection.
While not doubting his assertion, I'm not sure that I follow his explanation as to why this would be true.
Can anyone else confirm Mark7's experience with this?
Would the installation of a different codec on the PC make any difference?
Thanks
|
|
Advertisement
|
|
|
|
ZombyHero
Junior Member
|
5. April 2005 @ 12:59 |
Link to this message
|
Its tough to compare composite transfer with digital transfer, since one is a string of digits, and the other an analog signal. I do know that when it comes to analog that a composite signal is not the highest quality analog signal out there. Composite is a single signal containing the chroma and luminance info. A component signal, which divides chroma and luminance into three signal will deliver a cleaner video output. How it compares to the quality of a DV codec, I don't know. Either way, if its coming into a computer, its going to be digitized by a codec anyway.
There are quality differences in codecs, though. The Microsoft codec is decent, but not spectacular. I prefer to use Apple codecs or the Avid DV codec. I hear really good things about the MainConcept DV codec as well. Of course, not all of these are free.
mark7's claim is technically unsound. It seems to work for him, but it simply not the way people should be working with digital video.
|
|
