I have a CPU 8 cores / 16 threads, but the final rendering uses only one core. I tried with mp4/h264-5; webm/VP9; mkv/h264-5 e Prores. A 45 min video takes 3h 30m. I tried adding the parameters "threads=15", "wpp" which should be active by default and "frame-threads=3"; but the result is the same. On average I do 15 fps. Using h264-5 with nvenc takes 1h20m; but the CPU still uses only one core. On average I make 115 fps. Using an external engine (handbrake, which uses ffmpeg) you can use both the Cpu and the GPU; the rendering will take 10m! (but with lower quality). Since I'm compiling CinGG (note: all threads are used for compiling), maybe I need to enable some flags or something like that? Are there any customizations to do? Or should I set something in the operating system? Another question: How do you configure the render farm to use only the local CPU cores?
Andrea, If I remember correctly, rendering is not threaded (I will verify with GG later when he gets back from a bike ride). When gg does a 2 hour movie rendering, he almost always uses a local render farm. This is described in 6.4.5 BUT I just noticed that an image showing this that was in Features5.pdf was eliminated (probably by me at some time). I will send a copy of the png to you since it shows a 16 core setup for a "root" user (the 400 range ports work only for root and I have not personally tried the 1025 range for ordinary user). In 6.4.1, just follow the basic steps 1-7 and skip step 2 since there are no clients. I have not tried the Shell Script RenderMux in a long time, but Olaf fixed this for us awhile back and it was working perfectly then. On Fri, Jun 26, 2020 at 9:27 AM Andrea paz <[email protected]> wrote:
I have a CPU 8 cores / 16 threads, but the final rendering uses only one core. I tried with mp4/h264-5; webm/VP9; mkv/h264-5 e Prores. A 45 min video takes 3h 30m. I tried adding the parameters "threads=15", "wpp" which should be active by default and "frame-threads=3"; but the result is the same. On average I do 15 fps. Using h264-5 with nvenc takes 1h20m; but the CPU still uses only one core. On average I make 115 fps. Using an external engine (handbrake, which uses ffmpeg) you can use both the Cpu and the GPU; the rendering will take 10m! (but with lower quality).
Since I'm compiling CinGG (note: all threads are used for compiling), maybe I need to enable some flags or something like that? Are there any customizations to do? Or should I set something in the operating system?
Another question: How do you configure the render farm to use only the local CPU cores? -- Cin mailing list [email protected] https://lists.cinelerra-gg.org/mailman/listinfo/cin
OK, I am WRONG -- *rendering is multi-threaded for h264* and many others so something else is wrong. I will have gg analyze what the problem could be. Sorry for being wrong. On Fri, Jun 26, 2020 at 9:27 AM Andrea paz <[email protected]> wrote:
I have a CPU 8 cores / 16 threads, but the final rendering uses only one core. I tried with mp4/h264-5; webm/VP9; mkv/h264-5 e Prores. A 45 min video takes 3h 30m. I tried adding the parameters "threads=15", "wpp" which should be active by default and "frame-threads=3"; but the result is the same. On average I do 15 fps. Using h264-5 with nvenc takes 1h20m; but the CPU still uses only one core. On average I make 115 fps. Using an external engine (handbrake, which uses ffmpeg) you can use both the Cpu and the GPU; the rendering will take 10m! (but with lower quality).
Since I'm compiling CinGG (note: all threads are used for compiling), maybe I need to enable some flags or something like that? Are there any customizations to do? Or should I set something in the operating system?
Another question: How do you configure the render farm to use only the local CPU cores? -- Cin mailing list [email protected] https://lists.cinelerra-gg.org/mailman/listinfo/cin
Hi Andrea,
Another question: How do you configure the render farm to use only the local CPU cores? You can run each cinelerra process in separate terminal emulator with -D key and different ports higher than 1024.
Best regards, Andrey пт, 26 июн. 2020 г., 18:27 Andrea paz <[email protected]>:
I have a CPU 8 cores / 16 threads, but the final rendering uses only one core. I tried with mp4/h264-5; webm/VP9; mkv/h264-5 e Prores. A 45 min video takes 3h 30m. I tried adding the parameters "threads=15", "wpp" which should be active by default and "frame-threads=3"; but the result is the same. On average I do 15 fps. Using h264-5 with nvenc takes 1h20m; but the CPU still uses only one core. On average I make 115 fps. Using an external engine (handbrake, which uses ffmpeg) you can use both the Cpu and the GPU; the rendering will take 10m! (but with lower quality).
Since I'm compiling CinGG (note: all threads are used for compiling), maybe I need to enable some flags or something like that? Are there any customizations to do? Or should I set something in the operating system?
Another question: How do you configure the render farm to use only the local CPU cores? -- Cin mailing list [email protected] https://lists.cinelerra-gg.org/mailman/listinfo/cin
Excuse my ignorance. I don't understand points 3) and 5) of the manual. In fact, I really don't understand how a render farm works. If I decide to use 14 jobs in the renderfarm, starting from node 1025, I have to open 14 terminals and in each of them write: {path}./cin -d (or -D?) 1025 then 1026 ... 1038 And then? - "cin" gives me "unknown command" error, "./cin" works instead. - RenderMux does nothing; I certainly don't know how to use it.
Andrea, If I decide to use 14 jobs in the renderfarm, starting from node 1025,
I have to open 14 terminals and in each of them write: {path}./cin -d (or -D?) 1025 then 1026 ... 1038
No, you just need to use 1 terminal window and keyin: {path}./cin -d 1025 {path}./cin -d 1026 {path}./cin -d 1027 {path}./cin -d 1028 and so on (I could not get uppercase D to work) Here is an example of what I did and saw: # ./cin -d 410 Cinelerra Infinity - built: Jun 25 2020 19:59:25 git://git.cinelerra-gg.org/goodguy/cinelerra.git (c) 2006-2019 Heroine Virtual Ltd. by Adam Williams 2007-2020 mods for Cinelerra-GG by W.P.Morrow aka goodguy Cinelerra is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. There is absolutely no warranty for Cinelerra. # RenderFarmClient::main_loop: client started Then when you keyin the next line, you will see these jobs sitting around waiting for something to do: # ps -ef | grep cin root 3621 3342 99 15:45 pts/1 02:39:35 ./cin root 4658 1272 0 16:05 pts/2 00:00:00 ./cin -d 401 root 4662 1272 0 16:05 pts/2 00:00:00 ./cin -d 402 root 4666 1272 0 16:05 pts/2 00:00:00 ./cin -d 403 root 4670 1272 0 16:05 pts/2 00:00:00 ./cin -d 404 root 4674 1272 0 16:05 pts/2 00:00:00 ./cin -d 405 root 4678 1272 0 16:05 pts/2 00:00:00 ./cin -d 406 root 4686 1272 0 16:05 pts/2 00:00:00 ./cin -d 407 root 4690 1272 0 16:06 pts/2 00:00:00 ./cin -d 408 root 4694 1272 0 16:06 pts/2 00:00:00 ./cin -d 409 root 4698 1272 0 16:06 pts/2 00:00:00 ./cin -d 411 root 4704 1272 0 16:06 pts/2 00:00:00 ./cin -d 412 root 4708 1272 0 16:06 pts/2 00:00:00 ./cin -d 413 root 4713 1272 0 16:06 pts/2 00:00:00 ./cin -d 414 root 4717 1272 0 16:06 pts/2 00:00:00 ./cin -d 415 root 4721 1272 0 16:06 pts/2 00:00:00 ./cin -d 416 root 4733 1272 4 16:07 pts/2 00:00:00 ./cin -d 410 THEN you start the render and if you watch with the command line "top", you will see the 15 jobs all running and if you look in the directory where you rendered the file to, you will see 15 files changing size. When each of the cin -d jobs finish, you will see in the terminal window where you started them, the following: RenderFarmClientThread::run: Session finished. RenderFarmClientThread::run: Session finished. RenderFarmClientThread::run: Session finished. RenderFarmClientThread::run: Session finished. and so on - "cin" gives me "unknown command" error, "./cin" works instead.
I will fix the manual to correct "cin" to be "./cin" instead.
- RenderMux does nothing; I certainly don't know how to use it.
RenderMux will not do anything until the render farm rendering is finished. It is just used to put the 15 files together. GG puts the files together all of the time and sees no weirdness between the files. You have to use RenderMux in the same session that you rendered right afterwards because it keeps track of the file name you chose (or you can run it manually later from {path}/doc/RenderMux.sh ) However, on my laptop with 16 cores, BigBuckBunny 1920x1080 rendered using all 11-14 cpus automatically and took almost 10 minutes. When I used the Render Farm method instead, it took 9 minutes so IT IS MORE IMPORTANT TO FIGURE OUT WHY YOUR COMPUTER IS NOT USING MULTIPLE THREADS TO RENDER.
-- Cin mailing list [email protected] https://lists.cinelerra-gg.org/mailman/listinfo/cin
Andrea, I have a CPU 8 cores / 16 threads, but the final rendering uses only
one core.
The only way I could get my 16 core laptop to NOT use multiple cores was to set the Operating System cpu count to a lower number or to change in {path}/bin/ffmpeg, the file encode.opts from "threads=auto" to "threads=1". I used ffmpeg filetype of mp4 in my renders. Even when I set "Project SMP cpus" to 1, it still used multiple (this is in Settings, Preferences, Performance tab). Also, be sure to uncheck "use Render farm" so that the next time when you do not want to use the render farm, it does not automatically start up and surprise you.
Since I'm compiling CinGG (note: all threads are used for compiling), maybe I need to enable some flags or something like that? Are there any customizations to do? Or should I set something in the operating system?
Step 5 in the Render Farm usage just means that in the Render Menu, where you typed in the name for "Select file to render to", the 15 files will be named accordingly. For example, if you keyed in "/tmp/mystuff.mp4", you will have files in /tmp named mystuff.mp4001, mystuff.mp4002, ..., mystuff.mp4016. The RenderMux procedure uses ffmpeg to concatenate them together into 1 file called mystuff.mp4.
Render farm is OK, now. Thanks. I did some more rendering tests: CPU 8c/16t "use HW device" --> none Container: mp4 codec: h264 Default (none or threads=0) --> 86.5 fps [note: does not vary by changing SMP (obviously); I tried SMP=4; 8 and 14 with the exact same result] threads=8 --> 64.2 fps threads=15 --> 72.8 fps (I read somewhere that ffmpeg handles up to 15 threads; I don't know if this is true) threads=16 --> 73.2 fps [Note: in all these tests, from Top you can see that the rendering starts from 1000 - 1200%, but then after about ten seconds, it goes to 400 - 600% until the end of the rendering.] Render farm OK: all the cores at 100% --> 170 fps When I compile CinGG, I use this "configure": env CUDA_PATH=/opt/cuda ./configure --without-oss --with-cuda --with-single-user --with-booby maybe I need to add/take out some flags? (e.g. --prefix=/usr)
Andrea, GG says maybe it is something to do with permissions on the proc filesystem. Does it use only 1 cpu when you run a small test as "root" instead of an ordinary user? On Sat, Jun 27, 2020 at 2:00 AM Andrea paz <[email protected]> wrote:
Render farm is OK, now. Thanks.
I did some more rendering tests: CPU 8c/16t "use HW device" --> none Container: mp4 codec: h264
Default (none or threads=0) --> 86.5 fps
[note: does not vary by changing SMP (obviously); I tried SMP=4; 8 and 14 with the exact same result]
threads=8 --> 64.2 fps
threads=15 --> 72.8 fps (I read somewhere that ffmpeg handles up to 15 threads; I don't know if this is true)
threads=16 --> 73.2 fps
[Note: in all these tests, from Top you can see that the rendering starts from 1000 - 1200%, but then after about ten seconds, it goes to 400 - 600% until the end of the rendering.]
Render farm OK: all the cores at 100% --> 170 fps
When I compile CinGG, I use this "configure":
env CUDA_PATH=/opt/cuda ./configure --without-oss --with-cuda --with-single-user --with-booby
maybe I need to add/take out some flags? (e.g. --prefix=/usr) -- Cin mailing list [email protected] https://lists.cinelerra-gg.org/mailman/listinfo/cin
Much better! The render starts at 1500% CPU and after a while stabilizes at 700 - 800% with an average of 118 fps. The best result obtained, apart from the Render Farm (170 fps). How do I give permission to the "proc" even as non-root?
Andrea, GG looked at an old dump that you had sent and he says the relevant /proc/cpuinfo looks correct there. How do I give permission to the "proc" even as non-root?
Check permissions on this file via:
ls -l /cat/procinfo They should be 'r r r' so everyone can read it. Meanwhile we will test (later today though) on Arch here as an ordinary user to see if we can duplicate.
Permits seem to be correct: dr-xr-xr-x 403 root root 0 27 giu 08.47 proc But don't waste your time with this problem of mine. Using Render Farm is a great alternative.
Andrea:
Permits seem to be correct:
dr-xr-xr-x 403 root root 0 27 giu 08.47 proc
Because GG had to update Arch today to prepare for June 30 builds, he decided to test anyway on that Arch. Permissions were the exact same as yours on the /proc/cpuinfo file.
But don't waste your time with this problem of mine. Using Render Farm is a great alternative.
Since he had it booted anyway, he logged in as an ordinary user, did a Render, and it used 1100% of a 16 core cpu. So we have no idea what the problem is but you are happy with your workaround. Do you think that the Render Farm usage in the manual needs improvements? I forgot to mention that the referenced section 6.4.1 is enhanced with more details in 6.4.3 so maybe that should be referred to instead of improvements?
For completeness I add the result of further tests: with mitigations=off/auto ==> same results with command line System ffmpeg: 1100- 1200% 260 fps with ffmpeg embedded into CinGG: 400 - 800% 102 fps
participants (3)
-
Andrea paz -
Phyllis Smith -
Андрей Спицын