Packaging for Arch Linux
Hey there, I have just dropped the cinelerra-cv package from the official repositories on Arch Linux (to the AUR). Initially I planned on replacing it with cinelerra-gg, but I don't think this is very feasible at this point in time. I started to attempt a build, that is dynamically linked against the required system libraries [1], but it seems to be an equal amount of work compared to cinelerra-cv, which is not at all feasible. This is not because of the libraries themselves failing to work or missing symbols or anything like that, but because the build system actively prevents building a dynamically linked application! However, I already hack-fixed a great deal of things compared to the upstream provided attempt [2] (on a static build), which really doesn't need to depend on all the packages, that the PKGBUILD lists in depends=() (because of its static nature). That aside, a static build of this size in the official repositories of Arch Linux is not ever going to become a reality (nor in any other distribution), if nearly all of the components are available as a package. Therefore I would like you to consider to fix a great deal regarding the build system of cinelerra-gg to remedy this situation and move towards a (mostly) dynamically linked application instead. I believe this can be fairly straight forward with dynamic rebuilds against different versions of the required libraries in a continuous integration pipeline! The tools are there. In case static builds is what you will be going for no matter what, disregard the rest of my e-mail. That being said, I had to hack in nearly all of the --libs (which could just easily be gotten by calls to pkg-config, instead of hardcoding some of them or just not making them available at all) and also some of the --cflags for lv2 integration. I'm left with a (working!) application (probably with a bunch of broken edges), that - unfortunately - doesn't feature full RELRO [3] (because LDFLAGS are not propagated throughout all the components), PIE [4] (not implemented), has executable stack [5] (which I haven't encountered before with any of the other software I package, but is apparently pretty bad) and *a lot* of "unused shared library" references (due to the bizarre "libs" files being used to store all the library names instead of dynamically assembling them, when needed). The naming of the project, its subdirectories and (un-)declared/undocumented thirdparties (which of them are real thirdparties, which are your own/modified?) are confusing. The project does not (yet) have a form of versioning (at least not reflected in actual git tags), unless you're up for using calver (I recommend semver to actually be able to discern different levels of stability, feature availability and to work towards a roadmap). Considering providing (unsupported) builds for Arch Linux: Please use the AUR for providing PKGBUILDs and name the package according to your upstream (cinelerra-gg), unless you have the intent to rename the project. Make sure to use devtools [8] (e.g. `extra-x86_64-build -c` in the directory of the PKGBUILD) to build your package. It will give you *a lot* of additional information, as the package build is done in a pristine containered build, using systemd-nspawn. In case cin [7] is done by you: Please, never push binary data to the repositories! Closing, please don't get me wrong: It's your project and all, but this is a situation, that will drive away any potential packager of the software as it's just too complicated to plainly build (no matter the eventual runtime problems one might get)! I know this project has come a long way, but I believe it would benefit from a general cleanup and restructuring, which will render it much more accessible! Best, David P.S.: If you have any questions regarding the Arch Build System (ABS), or packaging on Arch Linux, or my PKGBUILD (which was still a WIP, when I stopped), shoot away! [1] http://pkgbuild.com/~dvzrv/aur/cinelerra-gg/PKGBUILD [2] https://git.cinelerra-gg.org/git/?p=goodguy/cinelerra.git;a=blob;f=cinelerra... [3] https://mudongliang.github.io/2016/07/11/relro-a-not-so-well-known-memory-co... [4] https://en.wikipedia.org/wiki/Position-independent_code [5] https://en.wikipedia.org/wiki/Executable_space_protection [6] https://aur.archlinux.org [7] https://aur.archlinux.org/cgit/aur.git/tree/?h=cin [8] https://www.archlinux.org/packages/extra/any/devtools/ -- https://sleepmap.de
Hi, Thank you for your email regarding your effort. As it turns out, it is not all that hard to make a system deliverable build using the following procedure: rm -rf /tmp/arch mkdir /tmp/arch cd /tmp/arch cp -a /tmp/cin_5.1.20181229-src.tgz . # you can get a tarball from last month's build here: # https://www.cinelerra-gg.org/download/pkgs/src/cin_5.1.20181129-src.tgz cp /tmp/PKGBUILD . # attached to email chown -R goodguy:goodguy . # your user name here sudo -u goodguy makepkg 2>&1 | tee pkg-arch.log You will have to update the md5sum in PKGBUILD to match the tarball you use. The cin_5.1.20181129-src.tgz tarball has: 3a964b60b7f2e216474841f1d64587c4 Be SURE to change the date in the PKGBUILD to match the tarball version. line 4: pkgrel=20181129 This produces a build with no thirdparty libraries, and is dynamically linked to system shared libraries, not to any build local binaries. It does add a variety of shared libraries as plugins, and a few executables needed by the program for utilities. I did test this, and it worked at least once. If you have any problems, or just have questions, or just are very friendly... feel free to chat us up. Can't tell the future, but I suspect that you will be able to do this quite easily. gg PS: there are a few spurious warnings, due to differences in the fffmpeg build config between the system ffmpeg and the thirdparty build... I will look into this as time permits. TO TEST: pacman -U ./cin-5.1-20181129-arch-x86_64.pkg.tar.xz (or whatever) then type: cin On Sat, Dec 29, 2018 at 3:07 PM David Runge <[email protected]> wrote:
Hey there,
I have just dropped the cinelerra-cv package from the official repositories on Arch Linux (to the AUR). Initially I planned on replacing it with cinelerra-gg, but I don't think this is very feasible at this point in time.
I started to attempt a build, that is dynamically linked against the required system libraries [1], but it seems to be an equal amount of work compared to cinelerra-cv, which is not at all feasible. This is not because of the libraries themselves failing to work or missing symbols or anything like that, but because the build system actively prevents building a dynamically linked application! However, I already hack-fixed a great deal of things compared to the upstream provided attempt [2] (on a static build), which really doesn't need to depend on all the packages, that the PKGBUILD lists in depends=() (because of its static nature).
That aside, a static build of this size in the official repositories of Arch Linux is not ever going to become a reality (nor in any other distribution), if nearly all of the components are available as a package. Therefore I would like you to consider to fix a great deal regarding the build system of cinelerra-gg to remedy this situation and move towards a (mostly) dynamically linked application instead. I believe this can be fairly straight forward with dynamic rebuilds against different versions of the required libraries in a continuous integration pipeline! The tools are there.
In case static builds is what you will be going for no matter what, disregard the rest of my e-mail.
That being said, I had to hack in nearly all of the --libs (which could just easily be gotten by calls to pkg-config, instead of hardcoding some of them or just not making them available at all) and also some of the --cflags for lv2 integration. I'm left with a (working!) application (probably with a bunch of broken edges), that - unfortunately - doesn't feature full RELRO [3] (because LDFLAGS are not propagated throughout all the components), PIE [4] (not implemented), has executable stack [5] (which I haven't encountered before with any of the other software I package, but is apparently pretty bad) and *a lot* of "unused shared library" references (due to the bizarre "libs" files being used to store all the library names instead of dynamically assembling them, when needed).
The naming of the project, its subdirectories and (un-)declared/undocumented thirdparties (which of them are real thirdparties, which are your own/modified?) are confusing. The project does not (yet) have a form of versioning (at least not reflected in actual git tags), unless you're up for using calver (I recommend semver to actually be able to discern different levels of stability, feature availability and to work towards a roadmap).
Considering providing (unsupported) builds for Arch Linux: Please use the AUR for providing PKGBUILDs and name the package according to your upstream (cinelerra-gg), unless you have the intent to rename the project. Make sure to use devtools [8] (e.g. `extra-x86_64-build -c` in the directory of the PKGBUILD) to build your package. It will give you *a lot* of additional information, as the package build is done in a pristine containered build, using systemd-nspawn. In case cin [7] is done by you: Please, never push binary data to the repositories!
Closing, please don't get me wrong: It's your project and all, but this is a situation, that will drive away any potential packager of the software as it's just too complicated to plainly build (no matter the eventual runtime problems one might get)! I know this project has come a long way, but I believe it would benefit from a general cleanup and restructuring, which will render it much more accessible!
Best, David
P.S.: If you have any questions regarding the Arch Build System (ABS), or packaging on Arch Linux, or my PKGBUILD (which was still a WIP, when I stopped), shoot away!
[1] http://pkgbuild.com/~dvzrv/aur/cinelerra-gg/PKGBUILD [2] https://git.cinelerra-gg.org/git/?p=goodguy/cinelerra.git;a=blob;f=cinelerra... [3] https://mudongliang.github.io/2016/07/11/relro-a-not-so-well-known-memory-co... [4] https://en.wikipedia.org/wiki/Position-independent_code [5] https://en.wikipedia.org/wiki/Executable_space_protection [6] https://aur.archlinux.org [7] https://aur.archlinux.org/cgit/aur.git/tree/?h=cin [8] https://www.archlinux.org/packages/extra/any/devtools/
-- https://sleepmap.de -- Cin mailing list [email protected] https://lists.cinelerra-gg.org/mailman/listinfo/cin
Hi! On 2018-12-29 20:14:59 (-0700), Good Guy wrote:
As it turns out, it is not all that hard to make a system deliverable build using the following procedure: In fact, 'not that hard' would be: git clone https://aur.archlinux.org/cinelerrag-gg.git cd cinelerra-gg makepkg -LCcf
rm -rf /tmp/arch mkdir /tmp/arch cd /tmp/arch cp -a /tmp/cin_5.1.20181229-src.tgz . # you can get a tarball from last month's build here: # https://www.cinelerra-gg.org/download/pkgs/src/cin_5.1.20181129-src.tgz With build you mean source tarball, right?
cp /tmp/PKGBUILD . Where does the PKGBUILD come from? Where is it maintained? After all, it is not the same as the one in the source tarball (see my note about confusing sources and structures). Why is the --disable-static-build option to configure not documented, the various --disable-option possibilities are though?
# attached to email chown -R goodguy:goodguy . # your user name here sudo -u goodguy makepkg 2>&1 | tee pkg-arch.log You will want to use devtools for that to achieve reproducibility. Also, there's no reason to use sudo here. Additionally, makepkg has a builtin log feature ( -L ), so there is no need for piping to tee manually.
You will have to update the md5sum in PKGBUILD to match the tarball you use. The cin_5.1.20181129-src.tgz tarball has: 3a964b60b7f2e216474841f1d64587c4 Be SURE to change the date in the PKGBUILD to match the tarball version. line 4: pkgrel=20181129 That is not what pkgrel [1] is for (hence my quote on versioning)! What you want is a consistent versioning, reflected in pkgver [2].
This produces a build with no thirdparty libraries, and is dynamically linked to system shared libraries, not to any build local binaries. It does add a variety of shared libraries as plugins, and a few executables needed by the program for utilities. For me this ends right at configure in:
checking whether X_HAVE_UTF8_STRING is declared... no configure: error: Cinelerra requires utf8 support in X Windows. which is strange, as I have not run into this before!
I did test this, and it worked at least once.
If you have any problems, or just have questions, or just are very friendly... feel free to chat us up. Can't tell the future, but I suspect that you will be able to do this quite easily. There are `--without-firewire --without-dv --without-dvb --without-video4linux2`, but the package dependencies are still pulled in in depends (also, that array should be named lower-case, not all upper case letters).
To get clarification about the dependencies, as this really doesn't jump at me: It seems to be okay to disable libzmpeg and commercial, but what is the latter actually pulling in/providing? I assume, that people would want something like lv2 integration. What is the rationale behind it being disabled? Another thing I found out is, that for ladspa integration, there's always the local version used, even if I provide `--disable-ladspa`, it is set to "forced".
PS: there are a few spurious warnings, due to differences in the fffmpeg build config between the system ffmpeg and the thirdparty build... I will look into this as time permits. TO TEST: pacman -U ./cin-5.1-20181129-arch-x86_64.pkg.tar.xz (or whatever) then type: cin What one really should to do is to run namcap [1] on that package, instead of installing it! This would also help in decimating the huge amount of unneeded dependencies, this package pulls in (some of which should at best be optdepends, but many are already provided by pulling in one of the others)!
Best, David [1] https://wiki.archlinux.org/index.php/PKGBUILD#pkgrel [2] https://wiki.archlinux.org/index.php/PKGBUILD#pkgver [3] https://wiki.archlinux.org/index.php/Namcap -- https://sleepmap.de
David: your feedback is very much appreciated and we will do what we can to resolve the issues you brought up as best as possible, but if it is still too burdensome for you, you have to consider whether or not you feel it is worth your time. We do monthly builds for Arch, Centos, Debian 8 & 9, Fedora 26-29, Leap/Suse 15, 42 & 10bit, Mint 18 & 19, Slackware 32 and 64 bit, Ubuntu 14, 16, 17, 18 & 32bit v14, and FreeBSD. The build procedures we use have to work for all of these as automatically as possible using XEN. Even then we still have to do some manual work. GG's expertise is in programming languages and writing build scripts was only something he had to do to get cinelerra-gg software usable. It would be really great to have a person whose expertise is in writing build scripts help out and still leave our XEN working.
As it turns out, it is not all that hard to make a system deliverable
build using the following procedure: In fact, 'not that hard' would be: git clone https://aur.archlinux.org/cinelerrag-gg.git cd cinelerra-gg makepkg -LCcf
This build design would be the one that would fit "your" needs. Every month, I build: ub14/deb ub18/deb ub17/deb ub16/deb leap14/rpm leap15/rpm ubuntu/deb mint18/deb \ fc26/rpm fc27/rpm fc28/rpm debian9i/deb debian8/deb debian9/deb arch/arch mint17/deb \ freebsd slk32 slk64 As you can see, there are several build designs that need to be accommodated. Your suggestion does bare the fact that it is not as easy as it can be for you to do a bulid. I can add the PKGBUILD I provided (or similar, if you have recommended changes) as a file in the top level directory, so that it can operate as you wish. My build procedures are in the "blds" directory. In fact, the PKGBUILD you have is just a simple composition of blds/arch.bld and the monthly PKGBUILD which I normally use for each monthly build. The dependency list is not customized to the vastly reduced build target design needed when the thirdparty libraries are not built, where the build must rely only on system libraries. If you have all of the demands needed to change the build design limit, you should remove the --without features that can be removed. I am not very familiar with the current state of the arch library series, and I think that you are vastly more qualified to make this kind of judgement. If you can create a recommended PKGBUILD, I will be happy to add it to the top level directory as you suggest. Why is the --disable-static-build option to configure not documented,
the various --disable-option possibilities are though?
Disable static build defaults to "auto", which is no for system builds. It is listed in ./configure --help. It did not need to be specified, but I put it in as a redundant specification just to indicate that non-static builds are available. In fact, each monthly build includes system packages that include thirdparty static, and are dynamically linked to most normally available system libraries. The wide configuration of ffmpeg prevents just blasting in a "one size fits all" type build config. I am only one good-guy after all. That is not what pkgrel [1] is for (hence my quote on versioning)!
What you want is a consistent versioning, reflected in pkgver [2].
These are monthly build ids, not any trick canonical naming system. As before, if you can identify what you would like, I will be happy to add it as the top level build design for arch. You will want to use devtools for that to achieve reproducibility.
Also, there's no reason to use sudo here. Additionally, makepkg has a builtin log feature ( -L ), so there is no need for piping to tee manually.
The data I provided was to illustrate that it is "not difficult" to produce a build. I did not say it would be "perfect" for your needs. I am definitely not an expert on arch system build designs. Any polish you can add would be widely appreciated. checking whether X_HAVE_UTF8_STRING is declared... no
configure: error: Cinelerra requires utf8 support in X Windows.
Strange, it is in /usr/include/X11/Xlib.h at line 70 and the autoconf test in configure looks good and succeeds on my arch system. I have no idea why your getting this problem. I would check your system include /usr/include/X11/Xlib.h for this line, and check the configure output of autogen.sh to see if the test looks like: ac_fn_c_check_decl "$LINENO" "X_HAVE_UTF8_STRING" "ac_cv_have_decl_X_HAVE_UTF8_STRING" "#include <X11/Xlib.h> " if test "x$ac_cv_have_decl_X_HAVE_UTF8_STRING" = xyes; then : else no_utf=yes fi Works for me. There are `--without-firewire --without-dv --without-dvb
--without-video4linux2`, but the package dependencies are still pulled in in depends (also, that array should be named lower-case, not all upper case letters).
It is only recently that ffmpeg libraries have been widely published on mainstream distros. The blds/*.bld scripts were created in haste, from a template build procedure that relied on a reduced recipe. The --without feature set is more of a reflex action. I did not spend a lot of time on any particular build design to determine the boundary of supported features. It is definitely the case that using the thirdparty build when possible is more stable than most system library sets. Heck, libxft is everywhere and is very dilapidated. I would recommend that you use the thirdparty build if possbile for two reasons... 1) the version is known, current and is more tested with this product. 2) in the event of a bug, the debug info can be easily accessed since the source is always consistent to the binary. This is not easy to produce, but... after using this for years I can say unequivocally that it is better. Also, what ARRAY? To get clarification about the dependencies, as this really doesn't jump
at me: It seems to be okay to disable libzmpeg and commercial, but what is the latter actually pulling in/providing? I assume, that people would want something like lv2 integration. What is the rationale behind it being disabled?
A couple of years ago, I was so dismayed over rude TV commercials, that I began to implement an "ad-blocker" for video. I mostly got it to work, but it requires too much effort to maintain a database of signature clips of ads to block. I left it because it provides a means to access reduced mpeg images in real time for ad suppression. It still needs work, but the framework is valuable, and so it persists waiting for someone to make a billion bucks by implementing an ad blocker for commercial TV. The reason lv2 is disabled is that I ported lv2 at about the same time I created the *.bld scripts. At the time, the lv2 support was pretty shaky. I opted to leave it out, but now it may be ok to add it. It has not been tested at all in this way. It may or may not be possible. My experience with lv2 was quite variable. some work well, some don't. Another thing I found out is, that for ladspa integration, there's
always the local version used, even if I provide `--disable-ladspa`, it
is set to "forced".
When thirdparty is disabled, it is not built. I will fix the message in the ./configure output to not indicate that it is in the build.
TO TEST: pacman -U ./cin-5.1-20181129-arch-x86_64.pkg.tar.xz (or
whatever) then type: cin
Thanks for the suggestion. I really appreciate you took the time to look hard at this. Hope to hear from you soon. gg
participants (2)
-
David Runge -
Good Guy