[Cin] Uncompressed video to lossless compression FFV1/MKV

[Terje J. Hanssen]

Den 23.12.2021 17:01, skrev Andrew Randrianasulu:
>
>
> On Thursday, December 23, 2021, Terje J. Hanssen via Cin 
> <cin at lists.cinelerra-gg.org> wrote:
>
>     Background:
>
>     Digital video produces vast amounts of data. Take PAL-SD material
>     (only image, no audio), stored as YUV422 with 8-bits per component
>     (=16-bits per pixel) and a resolution of 720x576 pixels, for example.
>     http://download.das-werkstatt.com/pb/mthk/info/video/comparison_video_codecs_containers.html#lossy_vs_lossless
>     <http://download.das-werkstatt.com/pb/mthk/info/video/comparison_video_codecs_containers.html#lossy_vs_lossless>
>
>      * Applying no compression at all (=uncompressed
>      
>      <http://download.das-werkstatt.com/pb/mthk/info/video/comparison_video_codecs_containers.html#codec_uncompressed
>     <http://download.das-werkstatt.com/pb/mthk/info/video/comparison_video_codecs_containers.html#codec_uncompressed>>)
>        results in:
>      * 720 x 576 x 16 bits x 25 frames ≥ 1.159 GiB/Minute
>      *
>
>      * For YUV422, 10 bpc (10+5+5 = 20 bits per pixel), this would be:
>      * 720 x 576 x 20 bits x 25 frames ≥ 1.448 GiB/Minute
>
>
>     So you would fit no more than 4 minutes of video on a whole 4.7
>     GiB DVD. Since these values are already uncomfortable (even for
>     422-subsampled PAL), imagine HD and beyond.
>     Therefore, you will hardly find uncompressed video in the wild,
>     except for temporary editing use cases.
>
>     -----------
>
>     The examples below show the differences in storage space required
>     for uncompressed, lossless and lossy video codecs:
>     https://www.scart.be/?q=en/content/short-guide-choosing-digital-format-video-archiving-masters
>     <https://www.scart.be/?q=en/content/short-guide-choosing-digital-format-video-archiving-masters>
>
>      * uncompressed (e.g., v210) 10-bit -> approx. 100GB per hour of
>     video;
>      * lossless compression (FFV1 and JPEG 2000) 10-bit -> approx.
>     45-50 GB
>        per hour of video;
>      * lossy compression;
>      * MPEG 2/DV50/IMX50 (50 Mbps) -> approx. 25 GB per hour of video;
>      * to compare with
>      * DV (DV25) -> approx. 12 GB per hour of video;
>      * MPEG 2 (DVD quality) -> approx. 3.6 GB per hour of video.
>
>     -------------
>
>     That is, if we manage to capture 422 uncompressed video via either
>     an internal capture card or via an external SSD recorder, there is
>     a need for
>
>      * mathematical lossless compression with typical intraframe FFV1/MKV
>        for preservation (archive masters)
>      * and/or possibly "visually lossless" compression to a lossy
>        intraframe intermediate/mezzanine format
>
>
>     I had no uncompressed test file available, but tried this guide to
>     generate a simple video (only) file:
>     Compressing and uncompressing a preservation master video file
>     https://bits.ashleyblewer.com/blog/2018/11/08/compressing-and-uncompressing-a-preservation-master-video-file/
>     <https://bits.ashleyblewer.com/blog/2018/11/08/compressing-and-uncompressing-a-preservation-master-video-file/>
>
>     ffmpeg -f lavfi -i mandelbrot=size=640x480:rate=25 -c:v v210 -t 5
>     uncompressed.mov
>
>     and next to turn it to a FFV1/MKV lossless compressed file
>
>     ffmpeg -i uncompressed.mov -map 0 -dn -c:v ffv1 -level 3 -g 1
>     -slicecrc 1 -slices 16 -c:a copy ffv1.mkv
>
>     and if required a test to uncompress the FFV1/MKV file to
>     uncompressed 10bit file again:
>
>     ffmpeg -i ffv1.mkv -c:v v210 uncompressed-again.mov
>
>     du -s *
>     105004    uncompressed.mov
>     22844    ffv1.mkv
>     105004    uncompressed-again.mov
>
>
> i think there is special filter (framecrc?) in ffmpeg for checking 
> exactness of those frames (no example off hand.. 
> http://underpop.online.fr/f/ffmpeg/help/framecrc.htm.gz ? )
>
>
>     In this simple case the lossless compressed ffv1.mkv size is
>     schrinked to 22% of the uncompressed.mov file.
>     In other practical tests I read the FFV1 file size was schrinked
>     to 1/3 to 46% of the uncompressed video.
>
>     (I tested for fun to transcode a "visual lossless" compressed
>     ProResHQ.mov file to FFV1/MKV, but that extented the file size to
>     180% )
>
>     I loaded the simple generated ffv1.mkv in Cinelerra, but also read
>     about a problem in a previous thread
>     [Cin] Single-frame step fwd/bak in viewer delayed?
>     https://www.mail-archive.com/cin@lists.cinelerra-gg.org/msg01799.html
>     <https://www.mail-archive.com/cin@lists.cinelerra-gg.org/msg01799.html>
>
>     Is this problem solved?
>     And will the lossless compressed (preservation) format FFV1.MKV
>     work as a editing format, or will there be a need beside to have
>     another intermediate format MPEG 2/DV50/IMX50 (50 Mbps) for editing?
>
>
> I think Andrea run into slowness/desyncro with some mkvs (unsolved)
>
> ffv1 can be quite cpu intensive, so fludity during edition depends on 
> host cpu. Also, apparently there is two-level of  crc codes both in 
> ffv1 and mkv container. May be one can disable one level of checking 
> for faster de/compression (I will look into it)
>
>

Better multithreading (than ffv1 -slices at the expense of file size) 
would help. There is a current suggestion for ffv1 v.4: "Appendix A.  
Multithreaded Decoder Implementation Suggestions"
https://datatracker.ietf.org/doc/html/draft-ietf-cellar-ffv1-v4#appendix-A

Else, interesting to read an equivalent article "Open Source FFV1 video 
capture workflow for MacOS"
https://digitensions.home.blog/2019/03/18/moving-video-tape-capture-to-an-open-source-workflow/


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.cinelerra-gg.org/pipermail/cin/attachments/20211225/c55d0ef1/attachment-0001.htm>