March 2020 - Cin - Cinelerra-GG Mailing Lists

UPD: OpenEXR multithreading, new compression patch
by Andrew Randrianasulu 17 Mar '20

17 Mar '20

Just included file.h in fileexr, and used file->cpus for auto cpu count ---------- Пересланное сообщение ---------- Тема: OpenEXR (hardcoded) multithreading, new compression patch Дата: Вторник 17 марта 2020 Отправитель: Andrew Randrianasulu <randrianasulu(a)gmail.com> Получатель: "Cinelerra.GG" <cin(a)lists.cinelerra-gg.org> I still don't know how to set attribute in OpenEXR, its here in /usr/include/OpenEXR/ImfStandardAttributes.h // // dwaCompressionLevel -- sets the quality level for images compressed // with the DWAA or DWAB method. // IMF_STD_ATTRIBUTE_DEF (dwaCompressionLevel, DwaCompressionLevel, float) but c++/c still definitely not my language ... I killed few blank lines in fileexr.C, I hope this is ok Additions to asset.c | h not included ..They not really usable without setting this new attribute Also, I still have no idea how to downpropagate cpu count .. file.C knows it, but File::EXR not ... -------------------------------------------------------

1 0

OpenEXR (hardcoded) multithreading, new compression patch
by Andrew Randrianasulu 17 Mar '20

17 Mar '20

I still don't know how to set attribute in OpenEXR, its here in /usr/include/OpenEXR/ImfStandardAttributes.h // // dwaCompressionLevel -- sets the quality level for images compressed // with the DWAA or DWAB method. // IMF_STD_ATTRIBUTE_DEF (dwaCompressionLevel, DwaCompressionLevel, float) but c++/c still definitely not my language ... I killed few blank lines in fileexr.C, I hope this is ok Additions to asset.c | h not included ..They not really usable without setting this new attribute Also, I still have no idea how to downpropagate cpu count .. file.C knows it, but File::EXR not ...

1 0

Just found "magical" EXR multithreaded call!
by Andrew Randrianasulu 17 Mar '20

17 Mar '20

in fileexr.C int FileEXR::write_frame(VFrame *frame, VFrame *data, FrameWriterUnit *unit) { EXRUnit *exr_unit = (EXRUnit*)unit; VFrame *output_frame; data->set_compressed_size(0); Imf::setGlobalThreadCount (4); <==== replace with project's cpu number? int native_cmodel = asset->exr_use_alpha ? BC_RGBA_FLOAT : BC_RGB_FLOAT; int components = BC_CModels::components(native_cmodel); Now, if only I found how to set this dwa compression thing .... PS: Render::render_single: Session finished. ** rendered 322 frames in 155.525 secs, 2.070 fps for DWAB compression, CPU set to 2.6Ghz Project - 1920x1080 float-rgba, with fader on video track set to 59 (so it will not take fast path)

1 0

Yet another round of tiff patch
by Andrew Randrianasulu 17 Mar '20

17 Mar '20

this time I set EXTRASAMPLE_UNASSALPHA; because associated alpha image was too dark, IMO, compared to what I saw in compositor window

1 0

UPD: Writing 16-bit per component png?
by Andrew Randrianasulu 17 Mar '20

17 Mar '20

Oh, I was setting functions in wrong order: int FilePNG::write_frame(VFrame *frame, VFrame *data, FrameWriterUnit *unit) { PNGUnit *png_unit = (PNGUnit*)unit; png_structp png_ptr = 0; png_infop info_ptr = 0; VFrame *output_frame; int result = 1; int is_float = (frame->get_color_model() == BC_RGBA_FLOAT || frame->get_color_model() == BC_RGB_FLOAT); data->set_compressed_size(0); //printf("FilePNG::write_frame 1\n"); native_cmodel = asset->png_use_alpha ? BC_RGBA8888 : BC_RGB888; if (is_float) native_cmodel = asset->png_use_alpha ? BC_RGBA16161616 : BC_RGB161616; if(frame->get_color_model() != native_cmodel) { if(!png_unit->temp_frame) png_unit->temp_frame = new VFrame(asset->width, asset->height, native_cmodel, 0); png_unit->temp_frame->transfer_from(frame); output_frame = png_unit->temp_frame; } else output_frame = frame; //printf("FilePNG::write_frame 1\n"); png_ptr = png_create_write_struct(PNG_LIBPNG_VER_STRING, 0, 0, 0); if( png_ptr ) { if( !setjmp(png_jmpbuf(png_ptr)) ) { info_ptr = png_create_info_struct(png_ptr); png_set_write_fn(png_ptr, data, (png_rw_ptr)write_function, (png_flush_ptr)flush_function); png_set_compression_level(png_ptr, asset->png_compression); png_set_IHDR(png_ptr, info_ptr, asset->width, asset->height, is_float ? 16 : 8, asset->png_use_alpha ? PNG_COLOR_TYPE_RGB_ALPHA : PNG_COLOR_TYPE_RGB, PNG_INTERLACE_NONE, PNG_COMPRESSION_TYPE_DEFAULT, PNG_FILTER_TYPE_DEFAULT); png_write_info(png_ptr, info_ptr); if(is_float) png_set_swap(png_ptr); <====== png_write_image(png_ptr, output_frame->get_rows()); png_write_end(png_ptr, info_ptr); result = 0; } png_destroy_write_struct(&png_ptr, &info_ptr); } if( result ) { char error[256]; png_error(png_ptr, error); fprintf(stderr, "FilePNG::write_frame failed: %s\n", error); } //printf("FilePNG::write_frame 3 %d\n", data->get_compressed_size()); return result; } hint found via https://github.com/GNOME/gimp/blob/mainline/plug-ins/common/file-png.c png_set_swap must come AFTER png_write_info! ---------- Пересланное сообщение ---------- Тема: Writing 16-bit per component png? Дата: Понедельник 16 марта 2020 Отправитель: Andrew Randrianasulu <randrianasulu(a)gmail.com> Получатель: "Cinelerra.GG" <cin(a)lists.cinelerra-gg.org> Probably stupid misuse of libpng api on my side ... I tried to add support for writing 16 bit per component PNGs into Cin. int FilePNG::write_frame(VFrame *frame, VFrame *data, FrameWriterUnit *unit) { PNGUnit *png_unit = (PNGUnit*)unit; png_structp png_ptr = 0; png_infop info_ptr = 0; VFrame *output_frame; int result = 1; int is_float = (frame->get_color_model() == BC_RGBA_FLOAT || frame->get_color_model() == BC_RGB_FLOAT); ==== here I try to get idea if project's colormodel actually *-FLOAT or not ==== data->set_compressed_size(0); //printf("FilePNG::write_frame 1\n"); native_cmodel = asset->png_use_alpha ? BC_RGBA8888 : BC_RGB888; if (is_float) native_cmodel = asset->png_use_alpha ? BC_RGBA16161616 : BC_RGB161616; === here I try to set native_colormodel to wide-channel integer format, if input was FP ==== if(frame->get_color_model() != native_cmodel) { if(!png_unit->temp_frame) png_unit->temp_frame = new VFrame(asset->width, asset->height, native_cmodel, 0); png_unit->temp_frame->transfer_from(frame); output_frame = png_unit->temp_frame; } else output_frame = frame; //printf("FilePNG::write_frame 1\n"); png_ptr = png_create_write_struct(PNG_LIBPNG_VER_STRING, 0, 0, 0); if( png_ptr ) { if( !setjmp(png_jmpbuf(png_ptr)) ) { info_ptr = png_create_info_struct(png_ptr); png_set_write_fn(png_ptr, data, (png_rw_ptr)write_function, (png_flush_ptr)flush_function); png_set_compression_level(png_ptr, asset->png_compression); png_set_IHDR(png_ptr, info_ptr, asset->width, asset->height, is_float ? 16 : 8, asset->png_use_alpha ? PNG_COLOR_TYPE_RGB_ALPHA : PNG_COLOR_TYPE_RGB, PNG_INTERLACE_NONE, PNG_COMPRESSION_TYPE_DEFAULT, PNG_FILTER_TYPE_DEFAULT); ==== here I try to call libpng with different bit depth depending on is_float ====== png_write_info(png_ptr, info_ptr); png_write_image(png_ptr, output_frame->get_rows()); png_write_end(png_ptr, info_ptr); result = 0; } png_destroy_write_struct(&png_ptr, &info_ptr); } if( result ) { char error[256]; png_error(png_ptr, error); fprintf(stderr, "FilePNG::write_frame failed: %s\n", error); } //printf("FilePNG::write_frame 3 %d\n", data->get_compressed_size()); return result; } I also modified int FilePNG::get_best_colormodel(Asset *asset, int driver) { if(asset->png_use_alpha) //return BC_RGBA8888; return BC_RGBA16161616; else //return BC_RGB888; return BC_RGB161616; } but probably those changes alone not enough, I get wrong colors or completely transparent picture in resulted png ? -------------------------------------------------------

2 2

Writing 16-bit per component png?
by Andrew Randrianasulu 16 Mar '20

16 Mar '20

Probably stupid misuse of libpng api on my side ... I tried to add support for writing 16 bit per component PNGs into Cin. int FilePNG::write_frame(VFrame *frame, VFrame *data, FrameWriterUnit *unit) { PNGUnit *png_unit = (PNGUnit*)unit; png_structp png_ptr = 0; png_infop info_ptr = 0; VFrame *output_frame; int result = 1; int is_float = (frame->get_color_model() == BC_RGBA_FLOAT || frame->get_color_model() == BC_RGB_FLOAT); ==== here I try to get idea if project's colormodel actually *-FLOAT or not ==== data->set_compressed_size(0); //printf("FilePNG::write_frame 1\n"); native_cmodel = asset->png_use_alpha ? BC_RGBA8888 : BC_RGB888; if (is_float) native_cmodel = asset->png_use_alpha ? BC_RGBA16161616 : BC_RGB161616; === here I try to set native_colormodel to wide-channel integer format, if input was FP ==== if(frame->get_color_model() != native_cmodel) { if(!png_unit->temp_frame) png_unit->temp_frame = new VFrame(asset->width, asset->height, native_cmodel, 0); png_unit->temp_frame->transfer_from(frame); output_frame = png_unit->temp_frame; } else output_frame = frame; //printf("FilePNG::write_frame 1\n"); png_ptr = png_create_write_struct(PNG_LIBPNG_VER_STRING, 0, 0, 0); if( png_ptr ) { if( !setjmp(png_jmpbuf(png_ptr)) ) { info_ptr = png_create_info_struct(png_ptr); png_set_write_fn(png_ptr, data, (png_rw_ptr)write_function, (png_flush_ptr)flush_function); png_set_compression_level(png_ptr, asset->png_compression); png_set_IHDR(png_ptr, info_ptr, asset->width, asset->height, is_float ? 16 : 8, asset->png_use_alpha ? PNG_COLOR_TYPE_RGB_ALPHA : PNG_COLOR_TYPE_RGB, PNG_INTERLACE_NONE, PNG_COMPRESSION_TYPE_DEFAULT, PNG_FILTER_TYPE_DEFAULT); ==== here I try to call libpng with different bit depth depending on is_float ====== png_write_info(png_ptr, info_ptr); png_write_image(png_ptr, output_frame->get_rows()); png_write_end(png_ptr, info_ptr); result = 0; } png_destroy_write_struct(&png_ptr, &info_ptr); } if( result ) { char error[256]; png_error(png_ptr, error); fprintf(stderr, "FilePNG::write_frame failed: %s\n", error); } //printf("FilePNG::write_frame 3 %d\n", data->get_compressed_size()); return result; } I also modified int FilePNG::get_best_colormodel(Asset *asset, int driver) { if(asset->png_use_alpha) //return BC_RGBA8888; return BC_RGBA16161616; else //return BC_RGB888; return BC_RGB161616; } but probably those changes alone not enough, I get wrong colors or completely transparent picture in resulted png ?

1 0

Upd: Re: How to add correctly BC_TumbleTextBox ?
by Andrew Randrianasulu 16 Mar '20

16 Mar '20

Ah, just run make clean and make in cinelerra directory, now it starts and I can make few PNGs .... ---------- Пересланное сообщение ---------- Тема: Re: [Cin] How to add correctly BC_TumbleTextBox ? Дата: Понедельник 16 марта 2020 Отправитель: Andrew Randrianasulu <randrianasulu(a)gmail.com> Получатель: "Cinelerra.GG" <cin(a)lists.cinelerra-gg.org> В сообщении от Monday 16 March 2020 19:01:27 Phyllis Smith написал(а): > Turns out there are widgets and gadgets (old school parlance). Widgets > usually involve a window base class, and gadgets usually involve a set of > widgets or gadgets. A BC_TumbleTextBox is a gadget, which has a textbox > widget and a tumbler widget. The "create_objects()" call is mandatory for > gadgets. A gadget tries to create an organized set of widgets as a > graphical component. It may have a window base class, and may need an > add_subwindow... or (as with most of them) it may just be a group of > objects, usually with a common x,y,w,h base class definition. For a > BC_TumbleTextBox, just call create objects, and do not add_subwindow. The > exact initialization order determines this, and can be variable. For > instance, a popup menu, or a list box may need add_window, and also a > create_objects to load the widget content prior to invoking the draw. > > It is frequently attractive (from a programming perspective) to add a > pointer to the widgets and gadgets in the gui base class so that you can > easily access them from the debugger. add_subwindow adds the subwindow to > the gui subwindow list, and makes it part of the window hierarchy. > Anything created and added with add_subwindow should probably not be > deleted in the destructor. The destruction of the gui will delete it > normally. You can delete a subwindow (lock the gui), and it is ok to do > that if your intent to to re-factor the gui appearance after construction. > Gadgets usually need to be deleted in the gui destructor since the object > is not in the gui subwindow set. If you dont delete it, it wont cause an > error, but it will be a memory leak. > > I tweaked/attached the patch for your review. It is not tested, > gg Thanks, I manually added it to source tree, with minor mod: @@ -351,21 +351,30 @@ PNGUnit::~PNGUnit() PNGConfigVideo::PNGConfigVideo(BC_WindowBase *parent_window, Asset *asset) : BC_Window(_(PROGRAM_NAME ": Video Compression"), parent_window->get_abs_cursor_x(1), parent_window->get_abs_cursor_y(1), - xS(200), yS(100)) + xS(200), yS(120)) { this->parent_window = parent_window; this->asset = asset; + use_alpha = 0; + compression = 0; } replaced alpha with use_alpha, otherwise it was not compileable. But now just compiled cin fails to start from its bin directory! LANG=C ./cin Cinelerra Infinity - built: Mar 16 2020 01:05:34 git://git.cinelerra-gg.org/goodguy/cinelerra.git (c) 2006-2019 Heroine Virtual Ltd. by Adam Williams 2007-2020 mods for Cinelerra-GG by W.P.Morrow aka goodguy Cinelerra is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. There is absolutely no warranty for Cinelerra. cin: malloc.c:2395: sysmalloc: Assertion `(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed. Аварийный останов system-installed cin still works normally ... > > On Mon, Mar 16, 2020 at 5:29 AM Andrew Randrianasulu < > randrianasulu(a)gmail.com> wrote: > > > I tried to add png compression level into filepng.c > > > > But I did something stupid, and whole thing doesn't compile: > > > > root@slax:/dev/shm/tmp/cinelerra-goodguy-20200316/cinelerra-5.1/cinelerra# > > LANG=C make > > g++ `cat i686/c_flags` -DMSGQUAL=filepng -c filepng.C -o i686/filepng.o > > filepng.C: In member function 'virtual void > > PNGConfigVideo::create_objects()': > > filepng.C:370:49: error: no matching function for call to > > 'PNGConfigVideo::add_subwindow(PNGCompression*)' > > add_subwindow(new PNGCompression(this, x+50, y)); > > ^ > > In file included from > > /dev/shm/tmp/cinelerra-goodguy-20200316/cinelerra-5.1/cinelerra/../guicast/bcsubwindow.h:26:0, > > from > > /dev/shm/tmp/cinelerra-goodguy-20200316/cinelerra-5.1/cinelerra/../guicast/bcbar.h:26, > > from > > /dev/shm/tmp/cinelerra-goodguy-20200316/cinelerra-5.1/cinelerra/../guicast/guicast.h:27, > > from edit.h:30, > > from filepng.C:23: > > /dev/shm/tmp/cinelerra-goodguy-20200316/cinelerra-5.1/cinelerra/../guicast/bcwindowbase.h:276:17: > > note: candidate: BC_WindowBase* BC_WindowBase::add_subwindow(BC_WindowBase*) > > BC_WindowBase* add_subwindow(BC_WindowBase *subwindow); > > ^ > > /dev/shm/tmp/cinelerra-goodguy-20200316/cinelerra-5.1/cinelerra/../guicast/bcwindowbase.h:276:17: > > note: no known conversion for argument 1 from 'PNGCompression*' to > > 'BC_WindowBase*' > > At global scope: > > cc1plus: warning: unrecognized command line option > > '-Wno-stringop-truncation' > > cc1plus: warning: unrecognized command line option '-Wno-format-overflow' > > cc1plus: warning: unrecognized command line option '-Wno-format-truncation' > > cc1plus: warning: unrecognized command line option '-Wno-stringop-overflow' > > cc1plus: warning: unrecognized command line option > > '-Wno-unknown-warning-option' > > make: *** [i686/filepng.o] Error 1 > > > > patch attached .... > > -- > > Cin mailing list > > Cin(a)lists.cinelerra-gg.org > > https://lists.cinelerra-gg.org/mailman/listinfo/cin > > > -------------------------------------------------------

1 0

OpenEXR 2.4.1 - still with CMake
by Andrew Randrianasulu 16 Mar '20

16 Mar '20

Not sure if reposting those news actually good idea or not but I have their github open next to Cin bugtracker I remember OpenEXR 2.3.x was reverted due to cmake build system conversion [it was 2.4.0, wrong memory!] At 2.4.x they still with CMake, but at least some bugs were fixed: https://github.com/AcademySoftwareFoundation/openexr/blob/master/CHANGES.md… Version 2.4.1 (February 11, 2020) Patch release with minor bug fixes. Summary Various fixes for memory leaks and invalid memory accesses Various fixes for integer overflow with large images. Various cmake fixes for build/install of python modules. ImfMisc.h is no longer installed, since it's a private header. from changelog ... Fix overzealous removal of if statements breaking all builds except win32 Re-enable Boost_NO_BOOST_CMAKE by default, document, clean up status messages Take DESTDIR into account when creating library symlinks Version 2.4.0 (September 19, 2019) Summary Completely re-written CMake configuration files Improved support for building on Windows, via CMake Improved support for building on macOS, via CMake All code compiles without warnings on gcc, clang, msvc Cleanup of license and copyright notices floating-point exception handling is disabled by default New Slice::Make method to reliably compute base pointer for a slice. Miscellaneous bug fixes ah, 2.4.0 was exactly reverted in september, 2019 for CinelerraGG. So, I'm doing more experiments with image sequences. It seems PNG is limited to 8-bit per pixel + alpha (at least in current Cin) In theory PNG supports 16 bit per sample integer formats ... http://www.libpng.org/pub/png/spec/1.2/PNG-Contents.html JPEG is also 8-bit only But both of above compiles fast! And they exactly use a lot of CPU in parallel, writing their index file (*.jpgs) at last moment, just as I hypothetized earlier. TIFF can create 32-bit per pixel floating-point images, they can be viewed in Gimp 2.10.18 but they are HUGE, like 32 MB for 1920x1080 image! It seems libtiff can compress with few algos, but not sure how many other apps can support those? https://www.awaresystems.be/imaging/tiff/tifftags/compression.html COMPRESSION_PIXARFILM = 32908; COMPRESSION_PIXARLOG = 32909; COMPRESSION_DEFLATE = 32946; COMPRESSION_ADOBE_DEFLATE = 8; something to experiment with, too! OpenEXR compressions: compression: pxr24 is slowest, at least for me. 7-10 min for 300-344 fullHD rgba-float images! around 3Mb per file compression: piz up to 4, 4.5 Mb per file but faster - 4 min (4 min 16 sec on second try)) for 344 images at 2.6 Ghz! cpu util: around 140% compression: zip So far up to 4.1 Mb per file 7 min 48 sec for 344 fullHD files at 2.6Ghz, 4-way compression: zips easily up to 6Mb files! 8 min 01 sec! compression: rle nearly as big as TIFF, nearly 32 Mb per file! It seems my tmpfs will be overflooded with 344 of them, and .. yeah CinGG gives NO WARNING about 0-sized OpenEXR files, so speed is irrelevant. (2 min 38 sec, but only 84 files written, for 2,693,421 Kbytes! ) Possible BUG! Shouldn't user be warned about data loss? It seems CinGG tries to compress 2 OpenEXR files at a time, by looking at directory with images via midnight commander (I can see briefly two 0-sized files). This is on 4-cpu machine. Yet, CPU utilization just hovers around 120% ..may be I need restart after upping image cache value from 16 mb to 256? (16 was clearly not enough for fullHD rgba-float frame) Utilization was up to 150% with rle OpenEXR compression, but for me this one hardly compress. may be on synthetic/rendered iamges .... Note, those numbers were done with CinGG's version of OpenEXR libs, NOT 2.4.1! I only announced my find of this release at the very start.

2 2

How to add correctly BC_TumbleTextBox ?
by Andrew Randrianasulu 16 Mar '20

16 Mar '20

I tried to add png compression level into filepng.c But I did something stupid, and whole thing doesn't compile: root@slax:/dev/shm/tmp/cinelerra-goodguy-20200316/cinelerra-5.1/cinelerra# LANG=C make g++ `cat i686/c_flags` -DMSGQUAL=filepng -c filepng.C -o i686/filepng.o filepng.C: In member function 'virtual void PNGConfigVideo::create_objects()': filepng.C:370:49: error: no matching function for call to 'PNGConfigVideo::add_subwindow(PNGCompression*)' add_subwindow(new PNGCompression(this, x+50, y)); ^ In file included from /dev/shm/tmp/cinelerra-goodguy-20200316/cinelerra-5.1/cinelerra/../guicast/bcsubwindow.h:26:0, from /dev/shm/tmp/cinelerra-goodguy-20200316/cinelerra-5.1/cinelerra/../guicast/bcbar.h:26, from /dev/shm/tmp/cinelerra-goodguy-20200316/cinelerra-5.1/cinelerra/../guicast/guicast.h:27, from edit.h:30, from filepng.C:23: /dev/shm/tmp/cinelerra-goodguy-20200316/cinelerra-5.1/cinelerra/../guicast/bcwindowbase.h:276:17: note: candidate: BC_WindowBase* BC_WindowBase::add_subwindow(BC_WindowBase*) BC_WindowBase* add_subwindow(BC_WindowBase *subwindow); ^ /dev/shm/tmp/cinelerra-goodguy-20200316/cinelerra-5.1/cinelerra/../guicast/bcwindowbase.h:276:17: note: no known conversion for argument 1 from 'PNGCompression*' to 'BC_WindowBase*' At global scope: cc1plus: warning: unrecognized command line option '-Wno-stringop-truncation' cc1plus: warning: unrecognized command line option '-Wno-format-overflow' cc1plus: warning: unrecognized command line option '-Wno-format-truncation' cc1plus: warning: unrecognized command line option '-Wno-stringop-overflow' cc1plus: warning: unrecognized command line option '-Wno-unknown-warning-option' make: *** [i686/filepng.o] Error 1 patch attached ....

2 2

OpenEXR DWA compression blog post from 2014
by Andrew Randrianasulu 16 Mar '20

16 Mar '20

http://fnordware.blogspot.com/2014/08/dwa-compression-in-openexr-22.html DWA compression in OpenEXR 2.2 "The default Compression Level setting is 45.0, which creates a visually lossless image that is often less than half the size of its truly lossless Piz counterpart. " So, I included it in my OpenEXR patch .... Not sure if you can call ILM library to do pixel format conversion so B44 will finally work? Time to more research!

1 0