[Cin] wildest dreams: stream-copy and/or skipping avoidable pixel format conversions

[Simeon Völkel]

Hi Einar,
Hi Phyllis,

as your highly appreciated responses are faster getting in than I can
respond adequately in individual mails, I'll try to respond to them
collectively below.

Am 24.04.20 um 14:53 schrieb Einar Rünkaru:
> Hi.
> 
> On 4/23/20 11:54 PM, Simeon Völkel wrote:
>> Hi Andrew,
>>
>> this was way quicker than I expected to get a response :D Please see below
>> for my interleaved comments.
>>
>> Am 23.04.20 um 16:45 schrieb Andrew Randrianasulu:
>>> В сообщении от Thursday 23 April 2020 13:36:30 Simeon Vc3b6lkel написал(а):
>>>> Hi all,
>>>>
>>>> coming back to cinelerra after a whole decade, I was really surprised
>>>> finding now even hardware accelerated decoding and encoding implemented...
>>>> Even though the landscape of cinelerra-forks seems to become increasingly
>>>> confusing, THANK YOU, to all who contributed in the last years and of
>>>> course especially to Adam, Einar and William for their tireless
>>>> commitment!
> 
> Thanks.
> 
>>> I think this was done in old Cinelerra for some DV variants and mjpeg
>>> (in mov and avi)
>>>
>>> In old CinelerraCV it was removed in commits
>>>
>>> https://github.com/cinelerra-gg/cinelerra-cv/commit/8364fc6af3eb9b105ecf0853f79885090b12005f
>>>
>>> https://github.com/cinelerra-gg/cinelerra-cv/commit/0ff51f4c53e17ff33701e8cc1096de33a87313b9
>>>
>>>
>>> I remember this because I tested this (mis)feature, and found it working
>>> for
>>> mjpeg avi (so i was not convinced by this reasoning and just keep my
>>> copy at commit before this :E)
>>>
>>> https://git.cinelerra-gg.org/git/?p=goodguy/cinelerra.git;a=blob;f=cinelerra-5.1/guicast/bccmodels.h;h=28b58459fb74eabb72e3fbf74f371ea51786cd18;hb=HEAD
>>>
>>>
>>> enum BC_CModel {
>>>    27         BC_TRANSPARENCY = 0,
>>>    28         BC_COMPRESSED   = 1,
>>>
>>> [..]
>>>
>>> I think this 'colormodel' define was used for this.
>>
>>
>>
>> This is a very interesting finding, and something I was not aware at all.
>> So thanks for bringing this piece of history to my attention!
>>
>> I'm wondering, too, whether the noticeable differences are really rounding
>> problems or rather mismatch between "full swing" and "studio swing", i.e.
>> using all possible (e.g. 8-bit) values or only a limited sub-range. I seem
>> to remember from the 2000s that pausing and playing a DV video could (or
>> would in every case?) switch between "darker black" and "brighter black",
>> reminding of a "MPEG YUV" vs "JPEG YUV" range mismatch.
>>
>> Maybe Einar can give additional details concerning the circumstances of the
>> removal?
> 
> Everything is said in commit messages. As I rememeber, someone asked the
> reasons and I explained. Search mail archives.


Thanks for the hint and sorry for me forgetting about the patch discussions
on CinCV-ML.

The thread
https://lists.cinelerra-cv.org/pipermail/cinelerra/2017q3/006525.html has
been brought to my attention, where the pro and cons were indeed discussed
in detail.


> 
> As Cinelerra-gg contains this feature try to use it. If you do not succeed,
> it is quite useless feature.


While I fully agree that supporting this feature only for formats popular
two decades ago makes it an extreme niche product, I for myself wouldn't
call it outright useless. But I have to admit, that my personal bias might
be unusually strong in this regard as my video-editing self literally grew
up in parallel with DV. :D

Towards the end of the thread linked above, Andrew reported on several of
his experiments for verifying the speedup offered by direct copy of DV:

https://lists.cinelerra-cv.org/pipermail/cinelerra/2017q3/006546.html

https://lists.cinelerra-cv.org/pipermail/cinelerra/2017q3/006569.html

So at least back then it wasn't broken per se and the commands he posted
should help if anyone wants to verify it again right now.


> 
> There is a good explanation why the requested feature could not be
> implemented in
> https://lists.cinelerra-gg.org/pipermail/cin/2020-April/001880.html
> 
> I agree with it.


I hope that especially my initial "wildest dreams" mail wasn't misconceived
as a "plz implement this, thx!!1eleven!" mail! I simply wanted to bring up
this topic in the context of cinelerra, as I have found quite some (rather
"techy") people working with video regularly completely unaware of the
feasibility (and benefits) of stream copying (under certain circumstances)
and didn't stumble upon cinelerra's direct copy capability (for DV) right
away either.

I understand that the quoted cost-benefit calculation from Phyllis and/or
William makes stream copying sound like a poor investment of (developer)
time. With the current limitation on number of tracks and supported codecs
as a prerequisite, I even agree on that point.


But I would like to challenge the estimated figure of only 2% of
Cinelerra's userbase benefiting of stream-copying as soon as nowadays
popular codecs were supported.

Cinelerra would have to be able to identify (even in the case of multiple
tracks) all frames for which only a single track is relevant for the final
output AND containing a clip not affected by any effect in the wider sense
of the desired output format (with stream-copying-compatible settings of
course). These are quite some prerequisites. But given that several people
involved with (or at least familiar with) cinelerra have already thought
about this, I could imagine that finding an exhaustive list of conditions
is absolutely feasible.

Given this identification ability (and support of nowadays relevant
intra-frame-only codecs), I would phrase the benefiting cases not as
"trivial edits" anymore but rather e.g. "in between fades if no effect is
applied either". Of course there can be projects where the number of frames
not being color-graded, scaled or overlaid by lower thirds can be counted
on the fingers of one hand. But I would be surprised if these were the
overwhelming majority ;)



Regarding the nowadays most popular codecs, I have to disagree with
Johannes's statement in
https://lists.cinelerra-cv.org/pipermail/cinelerra/2017q3/006539.html
that "It can certainly not be extended to formats that have inter-frame
compression.", as ffmpeg's -c:v copy and the concat demuxer working
perfectly for me simply proves this statement to be technically wrong.

I do not want to say that it would be easy to achieve. And as detailed in
my first mail, I'm not even sure about the general applicability when
thinking about possible restrictions on the GOP-length and -structure. In
addition I'm afraid that Phyllis estimation for the amount of developer
hours required for "full" stream-copying support in cinelerra might be not
too far off.

However I'm wondering how involved a hybrid approach would be. I imagine
that giving the user feedback in the GUI on the current frame type (I, B,
P, etc. similar to ffmpeg's showinfo filter) will be very beneficial for a
first mock-up. Together with an "only-single-unmodified-clip-relevant"
identification ability mentioned above, I think that at least a ffmpeg
concat demuxer export or even a direct forwarding from the demuxer to muxer
within cinelerra as discussed by Andrew should be feasible to hack, at
least for not-too-involved cases...

Again, please don't take this as an immediate feature request, but rather a
venturous proposal seeking futher in-depth discussion.

Still, in case anyone willing to fearlessly fight with a compiler (no need
to be a proven software engineer ;)) is reading this: I would definitely
welcome any coding experiments in this direction. As I see quite some hard
nuts to crack, I guess that additional architectural thinking should be
spread on many shoulders before anyone currently involved with Cinelerra's
development should be asked to go down to her codebase.



> Sorry,
> Einar

No need to say sorry... I feel honored to be able to discuss this with
Cinelerra's current experts, even if things should remain dreams!


Best regards,
Simeon