[Cin] Colors [was: Crash on playback]

[Andrea paz]

@Andrew

> @Andrea, for some reason I can't download attaches from forum (because I
have no login there?), can you add CMS.txt as attachment to email (again)?

I'm opening a new email because color has nothing to do with the
Vdpau/Vaapi issues.
It was just an attempt to put together all the information on how
CinGG treats color. Lately I found emails from Hermann Vosseler that
are very indicative. However, the fact remains that each plugins of
CinGG treats the color in its own way, so we should find a way to
understand how all the native plugins act, one by one (if I'm not
mistaken, the plugins of ffmpeg clip everything to 8-bit and sRGB).
Only with this information can we reconstruct a chain of steps that
accounts for color transformations and, perhaps, keep them under
control.

Sorry, I've been pushing too much color lately, it's best to let it go.
Hermann already knew in 2012 that it was not possible to treat it in
CinCV and that's also why he went for Lumiera.
-------------- next part --------------

CinGG does not have support for ICC color profiles or a global color management to standardize and facilitate the management of the various files with which it works. But it has its own way of managing color spaces and conversions; let's see how.


CMS

A color management describes how translates the colors of images/videos from their current color space to the color space of the output devices like monitors. The basic problem is to be able to display the same colors in every device we use for editing and every device on which our work will be watched. Calibrating and keeping our hardware under control is feasible, when instead we come out on the internet or DVD, etc. will be impossible to maintain the same colors on which we have worked us and so we just have to hope that there are not too many and bad alterations. But if the basis that we have set us is consistent, the alterations could be acceptable because they do not result from the sum of more alterations at each step.

There are two types of color management: Display referred (DRC) and Scene referred (SRC).
- DRC is based on having a calibrated monitor. What it displays is considered correct and becomes the basis of our color grading. The hope is that the colors of the final render won't change too much when displayed in other hardware/contexts. Be careful though, there must be a color profile for each type of color space you choose for your monitor. If we work for the web we have to set the monitor in sRGB with its color profile, if we go out for HDTV we have to set the monitor in rec.709 with its color profile, for 4k in Rec 2020; for Cinema in DCP-P3; etc.
- SRC instead uses three steps: 1- The input color space: whatever it is, it can be converted manually or automatically to a color space of your choice. 2- The color space of the timeline: we can choose and set the color space on which to work. 3- The colour space of the output: we can choose the colour space of the output (on other monitors or of the final rendering). ACES and OpenColorIO have an SRC workflow. NB: the monitor must still be calibrated to avoid unwanted color shifts.
- There would also be a third type of CMS: the one through the LUTs. In practice, the SRC workflow is followed through the appropriate 3D LUTs, instead of relying on the internal (automatic) management of the program. The LUT combined with the camera, used to display it correctly in the timeline and the LUT for the final output. Using LUTs, however, always involves preparation, selection of the appropriate LUT and post-correction. Also, as they are fixed conversion tables, they can always result in clipping and banding.


Color Space

A color space consists of primaries (gamut), transfer function (gamma) and matrix coefficients (scaler). The gamma adjustment is encoded in the transfer function (and not in the matrix, as many people mistakenly think).

- Color primaries: the gamut of the color space associated with the media, sensor or device (monitor, for example).
- Transfer characteristic function: converts linear values to non-linear values (e.g. logarithmic). It is also called Gamma correction.
- Color matrix function: converts from one color model to another. RGB <--> YUV; RGB <--> YCbCr; etc.

[Question: are these concepts correct? We needed more dettails?]

The camera sensors are always RGB and linear. Generally, those values get converted to YUV, in the files that produce, because it is a more efficient format thanks to chroma subsampling, and produces smaller files (even if of lower quality, i.e. you lose part of the color data). The conversion is nonlinear and so it concerns the "transfer characteristic" or gamma. The encoder gets input YUV and compresses that. It stores the transfer function as metadata, if provided. This is the type of file we will use in CinGG. Once imported into the program, the decoder generates YUV, ideally close to the input values, and shows it on the timeline. In CinGG, it is converted to RGB and displayed in sRGB [???]. As you can see, in this step the gamma doesn't play a role (but the metadata is attached, available when needed).


Monitors

Not having CinGG a CMS, it becomes essential to have a monitor calibrated and set in sRGB that is just the output displayed on the timeline of the program. You have these cases:

timeline: sRGB --> monitor: sRGB  (we get a correct color reproduction)
timeline: sRGB --> monitor: rec709 (we get slightly dark colors)
timeline: sRGB:--> monitor: DCI-P3 (we get over-saturated colors)

timeline: rec709 --> monitor: rec709 (we get a correct color reproduction)
timeline: rec709 --> monitor: sRGB (we get slightly faded colors)
timeline: rec709 --> monitor: DCI-P3 (we get over-saturated colors)



Pipeline CMS

INPUT --> PROCESSING --> OUTPUT --> DISPLAY

Input: color space of the source file; better if combined with an ICC profile.
Processing: How CinGG transforms and uses the input file (it is a temporary transformation, for use and consumption of the internal engine and plugins)
Output: our setting of the project for the output. This signal is taken in charge by the operating system and the graphics card (and then transformed, if any) to be sent to the monitor)
Display: As the monitor equipped with its color space (and profiled with ICC or LUT) displays the signal that reaches him and that we see.


How CinGG works

CinGG, like many other similar programs, works internally at 32 bits in RGB (and therefore 4:4:4, but the output in timeline is sRGB, as mentioned above) and performs a first conversion from YUV of the original file to RGB with the scaler, not the decoder, using its internal settings.
We can do without converting any media into the preferred color space, because the quality of the internal engine is already maximum.
Specifically, using X11: "playback single step” and “plugins” cause the render to be in the session color model, while continuous playback with no plugins tries to use the file’s best color model for the display (for speed). See manual 17.4.
This can create a visible effect of a switch in color in the Compositor, usually shown as grayish versus over-bright. The cause of the issue is that X11 is RGB only and it is used to draw the "refresh" frame. So single step is always drawn in RGB. To make a YUV frame into RGB, a color model transfer function is used. The math equations are based on color_space and color_range. In this case, color_range is the cause of the "gray" offset. The YUV mpeg color range is 16..235 for Y, 16..240 for UV, and the color range used by YUV jpeg is 0..255 for YUV". (GG/Phyllis) [Question: If we set OpenGL do we have the same behavior?]
In addition to forcing the internal motor to use sRGB, Each plugin we add converts and uses the color information in its own way. Some limit the gamut and depth of color by clipping (i.e. Histogram); others convert and reconvert color spaces for their convenience; others introduce artifacts and posterization; etc. For example, the Chroma Key (HSV) plugin converts any signal to HSV for its operation.
If we want to better control and target this color management in CinGG, we can take advantage of its internal ffmpeg engine:
"There is a optional feature that can be used via .opts lines from the ffmpeg decoded files. This is via the video_filter=colormatrix=...ffmpeg plugin. There may be other good plugins (lut3d...) that can also accomplish a desired color transform. This .opts feature affects the file colorspace on a file by file basis, although in principle it should be possible to setup a histogram plugin or any of the F_lut* plugins to remap the colortable, either by table or interp.

For output, the yuv<-->rgb transformations are via the YUV class, and its tables are initialized using YUV::yuv_set_colors. This sets up the transfer tables for just one version (one of bt601, bt709, bt2020) [Question: transfer function or scaler? I think scaler.] and the color range for mpeg or jpeg [question: primaries?]. This is limited, but since the product is usually a render, and this transform is designed to match display parameters, it is often correct to select the output colorspace once. If the render needs a colorspace mapping, then the session can be nested and the session output remapped by a plugin.

This is all not very glitzy or highly automated, but it does provide a wide color mapping capability." [GG]
In practice, in CinGG, you can have an SRC, but you have to do it manually step by step with the filters of ffmpeg because we do not have available automatisms and metadata management. However, with a good monitor well calibrated, we can always use the DRC (which, we remember, until a few years ago was the std).

From Ichthyostega mail (2012)

when discussing colour models, we should consider that any conversion is
(slightly) lossy. Because you always have to interpolate an exact value
when mapping it into the other colour space. Obviously, when we use
floating point numbers to represent values, these losses become small
and close to negligible.

So, theoretically speaking, it would always be the best options to perform
all effects in the same colour space, and that colour space should be the
one of the original footage. BUT -- since the two relevant colour models
(RGB and YUV) work somewhat different, basically this requires coding
each effect twice. And on top of that, it requires a thorough understanding
of the colour models. Because this way, basically the programmer does the
conversion "in pure math", i.e. in his brain, when he adapts the algorithm
to a different colour space. Sometimes this isn't even feasible, or it is
very demanding (requiring a solid knowledge of math). Thus, in practice,
many plug-ins just have their "natural" home colour model, and either
the other operation modes are sort-of broken, or implemented in a
sub optimal way.

So, bottom line: its not desirable to have just one colour model.
- YUV is the natural model of most camcorder produced video footage
- RGB is the most easy-to-understand model for the programmers
- additionally, float is the most precise and desirable model,
  but lacks support in (consumer) hardware

On colour spce selection, Cinelerra will configure the frame buffer for your
resulting video to be able to hold data in that colour model. Then, for each
plug-in, it will pick the variant of the algorithm coded for that model. If
that variant of the algorithm happens to be buggy, you've lost. We know since
several years, that some models are coded erroneously for some plug-ins,
especially when combined with an alpha channel. Unfortunately fixing that
requires really intense and systematic work; often it's not even outright
clear, how the "correct" implementation should work; thus, additionally
it would require some research and learning of theory.
We, as a community, simply didn't manage to get that done.

[...] according to Monty, Cinelerra already performs more colour
conversions than it ought to, under the hood.

The OpenGL plugin engine attempts to aggregate effects; treating a
chain of effects as one combined effect.  I think the same could be
done with colour formats, avoiding a few noisy conversions.

My tests yesterday showed that even _one_ conversion is visibly lossy:
The gradient effect becomes "striped" in YUV8 mode.  The gradient is
generated, so it does not process input pixels.  Hence, there is only

However, conversion in FLOAT looks pretty smooth.  It can't be
technically lossless, but it seems perceptually lossless, which is
our main concern here.  Doesn't blur work with floats in the
inner loops anyway?  And it works on a temporary buffer, since it
needs two passes, to you don't have to stick to the same component
size as the input and output.

What I'm suggesting is: Don't do the YUV<->RGB conversions in 8bit.
Always do them in FLOAT.  Cinelerra has functions for that; they are
called in the YUV effect plugin.  You probably have to move the
conversions closer to the inner loop to do this.

From mailing-list CV (2008)

Cinelerra loads MPEGs and still frames as-is, without thinking about it's color
range. This has a multitude of consequences. For example, when I load an MPEG4
in Cinelerra and also render it to MPEG4, the output file has the correct color
scale, because RGB value 16-16-16 was black in the original, and will be black
when playing the rendered file. However, when you render that same project to
PNGs and then use an external encoder (like mencoder) to encode to MPEG4 (which
I prefer, because mencoder is a very good encoder), the resulting video is too
bright, because the external encoder will think that the color range of PNG is
0-255, and it would be right about that. Therefore, 16-16-16 will turn out as
grey.

Right now, one has to use the RGB-601 filter to do it manually, but this is
very cumbersome. For example, if I render to any mpeg format with Cinelerra, I
have to compress the range of any format which uses 0-255, like still PNGs.
But, should I choose to render to a PNG sequence, I have to do the exact
reverse: expand the color range of all the mpeg source material and leave the
still frame as-is.

From renomath.org:
Cinelerra uses a 4:4:4 colorspace for editing; however, current versions of Cinelerra do not properly convert interlaced 4:2:0 source to 4:4:4. The upsampling of interlaced chroma fields is done using a progressive algorithm, even when the interlaced option is set in the video format section. Similarly, subsampling from 4:4:4 to 4:2:0 is always done using the progessive algorithm.

It should be pointed out that the conversion Cinelerra uses to get from 4:2:0 to 4:4:4 followed by the conversion from 4:4:4 to 4:2:0 results in the original frame. Therefore the interlaced chroma error only effects projects that use video effects or start with a DV source and render it to mpeg2. In particular, the video quality of a project consisting of HDV source spliced together with transitions and rendered through a yuv4mpeg stream to x264 will not be affected.

Suggestion

- Ultimately it would be better to set the project as RGB(A)-FLOAT, allowing system performance, because it collects all available data and does not make rounding errors. If we can't afford it, starting from YUV type media it is better to set the project as YUV(A)8, so as not to have a darker rendering in the timeline. On the contrary, if we start from RGB signals, it is better to use RGB(A)8. (Mag) [Question: Why this behavior?]. If we don't display correctly on the timeline, we'll make adjustments from the wrong base (metamerism) and get false results.
- If we import multiple media types with different color spaces into the project, an automatic CMS is counterproductive and it is worthwhile to manually convert each file to the color space we have decided for the project.
- If we go out for internet/TV it is better to set the colour space of the monitor to sRGB, which is the most used one. Some high quality TVs support bt.709, so it would be good to ask about the features of your TV.
- Another tip is to specify, when rendering, the color parameters in the relevant wrench options window. These parameters are:
        colorspace= ... (color matrix)
        color_trc= … (transfer characteristic or gamma)
        color_primaries= … (gamut)

They can take the values, for example, bt601, bt709 and bt2020. However, you have to understand that these parameters are only added metadata and do not affect the render. Their importance comes out when we open the file in a video player. “The color space information must be used explicitly so that it can be included in the video. CinGG or FFmpeg does not write it by itself. Without this information the players (e.g. MPV) stick to the dimensions of the video and take the assumed color model from a table. With videos in the dimensions from 720 to 1080 this is like written bt709. For smaller dimensions, e.g. DVD, bt601 is assumed and for 4k and above it is bt2020. Normally this is not a problem, but if you want to export a FullHD without color loss to a smaller size like 576 for example, you have to inform the encoder as well as the decoder of the player. This also applies if the videos are to be loaded on video platforms, where they are then converted into videos of different sizes. It is a security measure to prevent false colors, such as the color profiles in digital photos and the copies made from them. Some players always use one or the other matrix, even if the matrix is specified. And your graphics card settings can effect this too.” [Olaf]
- If we work with good quality files, it is always good to do the final rendering by setting the "pixel format" to at least 4:2:2, so as not to compromise too much the color rendering in the video produced. We have to evaluate the larger file size and rendering time, but if there are no problems in this sense, the resulting increase in quality is significant.
- Try setting your video card (if you allow!) to the highest quality or color setting. This is because it often produces a signal [16 - 235] instead of the full signal [0 - 255].
- Set CinGG with the "YUV color space" to bt709 (HD) or bt2020 (UHD) and the "YUV color range" to JPEG; this tip, like the previous one, does not limit the color range to [16 - 235].
- To avoid interference from the video card and operating system, which could impose their color space, you can use an IO device, also called mini monitor, which collects the signal of the software used and brings it unchanged to the monitor. They are expensive though.
-
[More suggestion?]