Uncompressed video to lossless compression FFV1/MKV
Background: Digital video produces vast amounts of data. Take PAL-SD material (only image, no audio), stored as YUV422 with 8-bits per component (=16-bits per pixel) and a resolution of 720x576 pixels, for example. http://download.das-werkstatt.com/pb/mthk/info/video/comparison_video_codecs... * Applying no compression at all (=uncompressed <http://download.das-werkstatt.com/pb/mthk/info/video/comparison_video_codecs_containers.html#codec_uncompressed>) results in: * 720 x 576 x 16 bits x 25 frames ≥ 1.159 GiB/Minute * * For YUV422, 10 bpc (10+5+5 = 20 bits per pixel), this would be: * 720 x 576 x 20 bits x 25 frames ≥ 1.448 GiB/Minute So you would fit no more than 4 minutes of video on a whole 4.7 GiB DVD. Since these values are already uncomfortable (even for 422-subsampled PAL), imagine HD and beyond. Therefore, you will hardly find uncompressed video in the wild, except for temporary editing use cases. ----------- The examples below show the differences in storage space required for uncompressed, lossless and lossy video codecs: https://www.scart.be/?q=en/content/short-guide-choosing-digital-format-video... * uncompressed (e.g., v210) 10-bit -> approx. 100GB per hour of video; * lossless compression (FFV1 and JPEG 2000) 10-bit -> approx. 45-50 GB per hour of video; * lossy compression; * MPEG 2/DV50/IMX50 (50 Mbps) -> approx. 25 GB per hour of video; * to compare with * DV (DV25) -> approx. 12 GB per hour of video; * MPEG 2 (DVD quality) -> approx. 3.6 GB per hour of video. ------------- That is, if we manage to capture 422 uncompressed video via either an internal capture card or via an external SSD recorder, there is a need for * mathematical lossless compression with typical intraframe FFV1/MKV for preservation (archive masters) * and/or possibly "visually lossless" compression to a lossy intraframe intermediate/mezzanine format I had no uncompressed test file available, but tried this guide to generate a simple video (only) file: Compressing and uncompressing a preservation master video file https://bits.ashleyblewer.com/blog/2018/11/08/compressing-and-uncompressing-... ffmpeg -f lavfi -i mandelbrot=size=640x480:rate=25 -c:v v210 -t 5 uncompressed.mov and next to turn it to a FFV1/MKV lossless compressed file ffmpeg -i uncompressed.mov -map 0 -dn -c:v ffv1 -level 3 -g 1 -slicecrc 1 -slices 16 -c:a copy ffv1.mkv and if required a test to uncompress the FFV1/MKV file to uncompressed 10bit file again: ffmpeg -i ffv1.mkv -c:v v210 uncompressed-again.mov du -s * 105004 uncompressed.mov 22844 ffv1.mkv 105004 uncompressed-again.mov In this simple case the lossless compressed ffv1.mkv size is schrinked to 22% of the uncompressed.mov file. In other practical tests I read the FFV1 file size was schrinked to 1/3 to 46% of the uncompressed video. (I tested for fun to transcode a "visual lossless" compressed ProResHQ.mov file to FFV1/MKV, but that extented the file size to 180% ) I loaded the simple generated ffv1.mkv in Cinelerra, but also read about a problem in a previous thread [Cin] Single-frame step fwd/bak in viewer delayed? https://www.mail-archive.com/[email protected]/msg01799.html Is this problem solved? And will the lossless compressed (preservation) format FFV1.MKV work as a editing format, or will there be a need beside to have another intermediate format MPEG 2/DV50/IMX50 (50 Mbps) for editing? ---------- Terje J. H
On Thursday, December 23, 2021, Terje J. Hanssen via Cin < [email protected]> wrote:
Background:
Digital video produces vast amounts of data. Take PAL-SD material (only image, no audio), stored as YUV422 with 8-bits per component (=16-bits per pixel) and a resolution of 720x576 pixels, for example. http://download.das-werkstatt.com/pb/mthk/info/video/compari son_video_codecs_containers.html#lossy_vs_lossless
* Applying no compression at all (=uncompressed <http://download.das-werkstatt.com/pb/mthk/info/video/ comparison_video_codecs_containers.html#codec_uncompressed>) results in: * 720 x 576 x 16 bits x 25 frames ≥ 1.159 GiB/Minute *
* For YUV422, 10 bpc (10+5+5 = 20 bits per pixel), this would be: * 720 x 576 x 20 bits x 25 frames ≥ 1.448 GiB/Minute
So you would fit no more than 4 minutes of video on a whole 4.7 GiB DVD. Since these values are already uncomfortable (even for 422-subsampled PAL), imagine HD and beyond. Therefore, you will hardly find uncompressed video in the wild, except for temporary editing use cases.
-----------
The examples below show the differences in storage space required for uncompressed, lossless and lossy video codecs: https://www.scart.be/?q=en/content/short-guide-choosing-digi tal-format-video-archiving-masters
* uncompressed (e.g., v210) 10-bit -> approx. 100GB per hour of video; * lossless compression (FFV1 and JPEG 2000) 10-bit -> approx. 45-50 GB per hour of video; * lossy compression; * MPEG 2/DV50/IMX50 (50 Mbps) -> approx. 25 GB per hour of video; * to compare with * DV (DV25) -> approx. 12 GB per hour of video; * MPEG 2 (DVD quality) -> approx. 3.6 GB per hour of video.
-------------
That is, if we manage to capture 422 uncompressed video via either an internal capture card or via an external SSD recorder, there is a need for
* mathematical lossless compression with typical intraframe FFV1/MKV for preservation (archive masters) * and/or possibly "visually lossless" compression to a lossy intraframe intermediate/mezzanine format
I had no uncompressed test file available, but tried this guide to generate a simple video (only) file: Compressing and uncompressing a preservation master video file https://bits.ashleyblewer.com/blog/2018/11/08/compressing-an d-uncompressing-a-preservation-master-video-file/
ffmpeg -f lavfi -i mandelbrot=size=640x480:rate=25 -c:v v210 -t 5 uncompressed.mov
and next to turn it to a FFV1/MKV lossless compressed file
ffmpeg -i uncompressed.mov -map 0 -dn -c:v ffv1 -level 3 -g 1 -slicecrc 1 -slices 16 -c:a copy ffv1.mkv
and if required a test to uncompress the FFV1/MKV file to uncompressed 10bit file again:
ffmpeg -i ffv1.mkv -c:v v210 uncompressed-again.mov
du -s * 105004 uncompressed.mov 22844 ffv1.mkv 105004 uncompressed-again.mov
i think there is special filter (framecrc?) in ffmpeg for checking exactness of those frames (no example off hand.. http://underpop.online.fr/f/ffmpeg/help/framecrc.htm.gz ? )
In this simple case the lossless compressed ffv1.mkv size is schrinked to 22% of the uncompressed.mov file. In other practical tests I read the FFV1 file size was schrinked to 1/3 to 46% of the uncompressed video.
(I tested for fun to transcode a "visual lossless" compressed ProResHQ.mov file to FFV1/MKV, but that extented the file size to 180% )
I loaded the simple generated ffv1.mkv in Cinelerra, but also read about a problem in a previous thread [Cin] Single-frame step fwd/bak in viewer delayed? https://www.mail-archive.com/[email protected]/msg01799.html
Is this problem solved? And will the lossless compressed (preservation) format FFV1.MKV work as a editing format, or will there be a need beside to have another intermediate format MPEG 2/DV50/IMX50 (50 Mbps) for editing?
I think Andrea run into slowness/desyncro with some mkvs (unsolved) ffv1 can be quite cpu intensive, so fludity during edition depends on host cpu. Also, apparently there is two-level of crc codes both in ffv1 and mkv container. May be one can disable one level of checking for faster de/compression (I will look into it)
----------
Terje J. H
-- Cin mailing list [email protected] https://lists.cinelerra-gg.org/mailman/listinfo/cin
Den 23.12.2021 17:01, skrev Andrew Randrianasulu:
On Thursday, December 23, 2021, Terje J. Hanssen via Cin <[email protected]> wrote:
Background:
Digital video produces vast amounts of data. Take PAL-SD material (only image, no audio), stored as YUV422 with 8-bits per component (=16-bits per pixel) and a resolution of 720x576 pixels, for example. http://download.das-werkstatt.com/pb/mthk/info/video/comparison_video_codecs... <http://download.das-werkstatt.com/pb/mthk/info/video/comparison_video_codecs_containers.html#lossy_vs_lossless>
* Applying no compression at all (=uncompressed <http://download.das-werkstatt.com/pb/mthk/info/video/comparison_video_codecs... <http://download.das-werkstatt.com/pb/mthk/info/video/comparison_video_codecs_containers.html#codec_uncompressed>>) results in: * 720 x 576 x 16 bits x 25 frames ≥ 1.159 GiB/Minute *
* For YUV422, 10 bpc (10+5+5 = 20 bits per pixel), this would be: * 720 x 576 x 20 bits x 25 frames ≥ 1.448 GiB/Minute
So you would fit no more than 4 minutes of video on a whole 4.7 GiB DVD. Since these values are already uncomfortable (even for 422-subsampled PAL), imagine HD and beyond. Therefore, you will hardly find uncompressed video in the wild, except for temporary editing use cases.
-----------
The examples below show the differences in storage space required for uncompressed, lossless and lossy video codecs: https://www.scart.be/?q=en/content/short-guide-choosing-digital-format-video... <https://www.scart.be/?q=en/content/short-guide-choosing-digital-format-video-archiving-masters>
* uncompressed (e.g., v210) 10-bit -> approx. 100GB per hour of video; * lossless compression (FFV1 and JPEG 2000) 10-bit -> approx. 45-50 GB per hour of video; * lossy compression; * MPEG 2/DV50/IMX50 (50 Mbps) -> approx. 25 GB per hour of video; * to compare with * DV (DV25) -> approx. 12 GB per hour of video; * MPEG 2 (DVD quality) -> approx. 3.6 GB per hour of video.
-------------
That is, if we manage to capture 422 uncompressed video via either an internal capture card or via an external SSD recorder, there is a need for
* mathematical lossless compression with typical intraframe FFV1/MKV for preservation (archive masters) * and/or possibly "visually lossless" compression to a lossy intraframe intermediate/mezzanine format
I had no uncompressed test file available, but tried this guide to generate a simple video (only) file: Compressing and uncompressing a preservation master video file https://bits.ashleyblewer.com/blog/2018/11/08/compressing-and-uncompressing-... <https://bits.ashleyblewer.com/blog/2018/11/08/compressing-and-uncompressing-a-preservation-master-video-file/>
ffmpeg -f lavfi -i mandelbrot=size=640x480:rate=25 -c:v v210 -t 5 uncompressed.mov
and next to turn it to a FFV1/MKV lossless compressed file
ffmpeg -i uncompressed.mov -map 0 -dn -c:v ffv1 -level 3 -g 1 -slicecrc 1 -slices 16 -c:a copy ffv1.mkv
and if required a test to uncompress the FFV1/MKV file to uncompressed 10bit file again:
ffmpeg -i ffv1.mkv -c:v v210 uncompressed-again.mov
du -s * 105004 uncompressed.mov 22844 ffv1.mkv 105004 uncompressed-again.mov
i think there is special filter (framecrc?) in ffmpeg for checking exactness of those frames (no example off hand.. http://underpop.online.fr/f/ffmpeg/help/framecrc.htm.gz ? )
In this simple case the lossless compressed ffv1.mkv size is schrinked to 22% of the uncompressed.mov file. In other practical tests I read the FFV1 file size was schrinked to 1/3 to 46% of the uncompressed video.
(I tested for fun to transcode a "visual lossless" compressed ProResHQ.mov file to FFV1/MKV, but that extented the file size to 180% )
I loaded the simple generated ffv1.mkv in Cinelerra, but also read about a problem in a previous thread [Cin] Single-frame step fwd/bak in viewer delayed? https://www.mail-archive.com/[email protected]/msg01799.html <https://www.mail-archive.com/[email protected]/msg01799.html>
Is this problem solved? And will the lossless compressed (preservation) format FFV1.MKV work as a editing format, or will there be a need beside to have another intermediate format MPEG 2/DV50/IMX50 (50 Mbps) for editing?
I think Andrea run into slowness/desyncro with some mkvs (unsolved)
ffv1 can be quite cpu intensive, so fludity during edition depends on host cpu. Also, apparently there is two-level of crc codes both in ffv1 and mkv container. May be one can disable one level of checking for faster de/compression (I will look into it)
Better multithreading (than ffv1 -slices at the expense of file size) would help. There is a current suggestion for ffv1 v.4: "Appendix A. Multithreaded Decoder Implementation Suggestions" https://datatracker.ietf.org/doc/html/draft-ietf-cellar-ffv1-v4#appendix-A Else, interesting to read an equivalent article "Open Source FFV1 video capture workflow for MacOS" https://digitensions.home.blog/2019/03/18/moving-video-tape-capture-to-an-op...
participants (2)
-
Andrew Randrianasulu -
Terje J. Hanssen