As you can see from the responses already, there are a lot of factors that can affect render times! But I do not see this in a 720x576 video that I rendered with file format ffmpeg/mp4. It was a 4 minute video and took only 32 seconds to render. You can see in the attached file with another window running "top" that shows %CPU = 1149 (so while rendering it is using 11 1/2 cpus out of 16). You can see on the timeline that the video is about 4 minutes long and you can see in the lower left hand bottom corner, that Rendering took...0.32 seconds.

In Settings->Preferences, Performance tab I have Cache set to 4096 and SMP cpu count set to 16. I have not looked at your screenshot in detail yet, but will try to do so tomorrow. In the Render menu, using the Video wrench, I chose h264.mp4. I will also try on a 1080p video.

I am getting what I consider to be very slow render times and wanted to know if something is set up wrong.

As a test, I imported a 1080p video from my camera, looped it several times and rendered it with no effects or compositing. For a 4 minute video it took about 12 minutes to render as an .mp4.

CPU usage never went above 23% and memory usage never went above 19%.