[Cin] Upd: More findings about openEXR and libtiff

Sun Mar 15 23:47:46 CET 2020

While my b44 patch for openEXR works, it produces nearly same 32Mb
big images as uncompressed tiff.. (for rgba-float project. - correspond to documentation
I cited! "Channels of type UINT or FLOAT are not compressed."
so, no use  in cinelerra currently? (HALF = half-float, 16-bit fp per component, IIRC)

there are LOSSY subtypes of compression, DWAA/DWAB - but they are mostly for proxies
and final renders where "small compression is acceptable"
but they require some more parameters (compression ratio), so not for this night

And my hacking on TIFFTAG_PHOTOMETRIC

was *wrong* .... resulted image was  treated as monochrome/1bit in GIMP .. :/

so, this hunk can be dropped. sorry.

----------  Пересланное сообщение  ----------

Тема: More findings about openEXR and libtiff
Дата: Понедельник 16 марта 2020
Отправитель: Andrew Randrianasulu <randrianasulu at gmail.com>
Получатель:  "Cinelerra.GG" <cin at lists.cinelerra-gg.org>

So, I digged deeper.

First, gimp source says:

https://github.com/GNOME/gimp/blob/mainline/plug-ins/file-tiff/file-tiff-load.c

else if (extra == 0 &&
                     ((photomet == PHOTOMETRIC_RGB && spp > 3) ||
                      /* All other color space expect 1 channel (grayscale,
                       * palette, mask). */
                      (photomet != PHOTOMETRIC_RGB && spp > 1)))
              {
                /* ExtraSamples field not set, yet we have more channels than
                 * the PhotometricInterpretation field suggests.
                 * This should not happen as the spec clearly says "This field
                 * must be present if there are extra samples". So the files
                 * can be considered non-conformant.
                 * Let's ask what to do with the channel.
                 */
                extra_message = _("Non-conformant TIFF: extra channels without 'ExtraSamples' field.")

https://www.awaresystems.be/imaging/tiff/tifftags/photometricinterpretation.html

TIFF Tag PhotometricInterpretation
IFD
  Image
  Code
262 (hex 0x0106)
  Name
PhotometricInterpretation
  LibTiff name
TIFFTAG_PHOTOMETRIC
  Type
SHORT
  Count
1
  Default
None
  Description
The color space of the image data.
The specification considers these values baseline:
 0 = WhiteIsZero. For bilevel and grayscale images: 0 is imaged as white.
 1 = BlackIsZero. For bilevel and grayscale images: 0 is imaged as black.
------>>>>>-----
 2 = RGB. RGB value of (0,0,0) represents black, and (255,255,255) represents white, assuming 8-bit components. The components are stored in the indicated order: first Red, then Green, then Blue.
------<<<<<-----
 3 = Palette color. In this model, a color is described with a single component. The value of the component is used as an index into the red, green and blue curves in the ColorMap field to retrieve an RGB triplet that defines the color. When PhotometricInterpretation=3 is used, ColorMap must be present and SamplesPerPixel must be 1.
 4 = Transparency Mask. This means that the image is used to define an irregularly shaped region of another image in the same TIFF file. SamplesPerPixel and BitsPerSample must be 1. PackBits compression is recommended. The 1-bits define the interior of the region; the 0-bits define the exterior of the region. 
These values are considered an extension:

 5 = Seperated, usually CMYK.
 6 = YCbCr
 8 = CIE L*a*b* (see also specification supplements 1 and 2)
 9 = CIE L*a*b*, alternate encoding also known as ICC L*a*b* (see also specification supplements 1 and 2) 
The TIFF-F specification (RFC 2301) defines:
10 = CIE L*a*b*, alternate encoding also known as ITU L*a*b*, defined in ITU-T Rec. T.42, used in the TIFF-F and TIFF-FX standard (RFC 2301). The Decode tag, if present, holds information about this particular CIE L*a*b* encoding. 
The DNG specification adds these definitions:
32803 = CFA (Color Filter Array)
34892 = LinearRaw

LibTiff defines mostly the same values, and adds support for Pixar LogL and LogLuv encoding:
 PHOTOMETRIC_MINISWHITE = 0;
 PHOTOMETRIC_MINISBLACK = 1;
 PHOTOMETRIC_RGB = 2;
 PHOTOMETRIC_PALETTE = 3;
 PHOTOMETRIC_MASK = 4;
 PHOTOMETRIC_SEPARATED = 5;
 PHOTOMETRIC_YCBCR = 6;
 PHOTOMETRIC_CIELAB = 8;
 PHOTOMETRIC_ICCLAB = 9;
 PHOTOMETRIC_ITULAB = 10;
 PHOTOMETRIC_LOGL = 32844;
 PHOTOMETRIC_LOGLUV = 32845;

So, *I think* tiff exporter shouldn't use this tag on alpha/32-bit per channel images ....

	if(frame->get_color_model() == BC_RGB888) // PHOTOMETRIC RGB only valid for 8-bit RGB
	TIFFSetField(stream, TIFFTAG_PHOTOMETRIC, PHOTOMETRIC_RGB);

currently recompiling Cin with this change .... probably additional tag needed for alpha samples ...
so, not final

Next into openEXR!
https://en.wikipedia.org/wiki/OpenEXR

B44
This form of compression is lossy for half data and stores 32-bit data uncompressed. It maintains a fixed compression size of either 2.28:1 or 4.57:1 and is designed for realtime playback. B44 compresses uniformly regardless of image content. [10]

B44A
An extension to B44 where areas of flat color are further compressed, such as alpha channels.

Mmm.. realtime playback!

https://rainboxlab.org/downloads/documents/EXR_Data_Compression.pdf
says basically same thing about B44/B44A, with slightly more tech details.

Added, currently compiles (see draft patch, made in analogy with my tiff patch)

OpenEXR lib multithreading topic:

https://lists.nongnu.org/archive/html/openexr-devel/2009-11/msg00001.html
--quote---
When threading is enabled in the library, multiple scanline blocks or tiles
are processed concurrently.   A scanline block is 32 lines for PIZ compression.
Threading doesn't kick in unless each writePixels() call supplies multiple
blocks' worth of pixels.  For best threading performance, write the entire
image with a single writePixels() call.
See also Section 7.2 of http://www.openexr.com/ReadingAndWritingImageFiles.pdf.
----quote end---

from this file ...

-----****----
Multithreaded I/O
The IlmImf library supports multithreaded file input and output where the library creates its own worker
threads that are independent of the application program's threads. When an application thread calls
readPixels(), readTiles(), writePixels() or writeTiles() to read or write multiple scan lines or
tiles at once, the library's worker threads process the tiles or scanlines in parallel.
During     startup,    the   application   program     must     enable   multithreading   by   calling function
setGlobalThreadCount(). This tells the IlmImf library how many worker threads it should create. (As a
special case, setting the number of worker threads to zero reverts to single-threaded operation; reading and
writing image files happens entirely in the application thread that calls the IlmImf library.)

Multithreaded I/O, Multithreaded Application Program
Function setGlobalThreadCount() creates a global pool of worker threads inside the IlmImf library. If
an application program has multiple threads, and those threads read or write several OpenEXR files at the
same time, then the worker threads must be shared among the application threads. By default each file will
attempt to use the entire worker thread pool for itself. If two files are read or written simultaneously by two
application threads, then it is possible that all worker threads perform I/O on behalf of one of the files, while
I/O for the other file is stalled.

In order to avoid this situation, the constructors for input and output file objects take an optional
numThreads argument. This gives the application program more control over how many threads will be
kept busy reading or writing a particular file.
-----****------

sounds like something to try, using available within Cin variable about cpu count.

-------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OPEN_EXR_B44.diff
Type: text/x-diff
Size: 1771 bytes
Desc: not available
URL: <https://lists.cinelerra-gg.org/pipermail/cin/attachments/20200316/0916c082/attachment.bin>