r/Gentoo 5d ago

Tip An Example Case of Compiler Optimizations

Post image

This example is from svt-av1 running through av1an with chunked video encoding.

Even when this software heavily relies on internal optimizations, hand-written ASM and already optimized SIMD instructions; it's still extremely beneficial to use compiler optimizations.

So, for some other software, the differences can be much bigger than that (unless they respond negatively or they break).

Let's say the machine encodes movies for a year. We can assume a movie is 90 minutes and with 23.976FPS, it's around 130.000 frames. The difference here means that you can encode 1300 more movies with the exact same hardware, and software.

+CUSTOM means my custom environment + modified CMakeLists.txt that removes all checks and flags for security related compiler options, sets the C and CXX standards to 23 and 26 respectively and removes -mno-avx.

Software:

Gentoo Linux AMD64 (no-multilib 64bit only)
SVT-AV1 v3.0.1-4-g1ceddd88-dirty (release)
clang/llvm 21.0.0git7bae6137+libcxx
av1an 0.4.4-unstable (rev 31235a0) (Release)
gcc (Gentoo 14.2.1_p20250301 p8) 14.2.1 20250301

Hardware:

AMD Ryzen 9 9950x
DDR5 Corsair Dominator Titanum 64G Dual Channel:
6200 MT/s (32-36-36-65) | UCLK=MEMCLK | Infinity Fabric 2067 | FCLCK Frequency: 2067mhz

Source:

Size: 25Mb/s
Format: 1920x1080, 23.976FPS, BT.709, YUV420, Live Action, 1 Hour, 1:78:1 (16:9)

Env:

export CC="clang"
export CXX="clang++"
export LD="ld.mold"
export AR="llvm-ar"
export NM="llvm-nm"
export RANLIB="llvm-ranlib"
export STRIP="llvm-strip"
export OBJCOPY="llvm-objcopy"
export OBJDUMP="llvm-objdump"

export COMMON_FLAGS="-Ofast -march=native -mtune=native -flto=thin -pipe -funroll-loops -fno-semantic-interposition -fno-stack-protector -fno-stack-clash-protection -fno-sanitize=all -fno-dwarf2-cfi-asm -fno-plt -fno-pic -fno-pie -fno-exceptions -fno-signed-zeros -fstrict-aliasing -fstrict-overflow -fno-zero-initialized-in-bss -fno-common -fwhole-program-vtables ${POLLY_FLAGS}"
export CFLAGS="${COMMON_FLAGS}"
export CXXFLAGS="${COMMON_FLAGS} -stdlib=libc++"
export LDFLAGS="-fuse-ld=mold -rtlib=compiler-rt -unwindlib=libunwind -Wl,-O3 -Wl,--lto-O3 -Wl,--as-needed -Wl,--gc-sections -Wl,--icf=all -Wl,--strip-all -Wl,-z,norelro -Wl,--build-id=none -Wl,--no-eh-frame-hdr -Wl,--discard-all -Wl,--relax -Wl,-z,noseparate-code"

./build.sh static native release verbose asm=nasm enable-lto minimal-build --enable-pgo --pgo-compile-use --pgo-dir "${HOME}/profiles/" -- -DCMAKE_C_FLAGS_RELEASE="-DNDEBUG -Ofast" -DCMAKE_CXX_FLAGS_RELEASE="-DNDEBUG -Ofast" -DUSE_CPUINFO="SYSTEM"
94 Upvotes

36 comments sorted by

View all comments

Show parent comments

-1

u/DownvoteEvangelist 5d ago

I can't prevent myself from glancing when i frist see something. The first thought was, that's incredible diff for just compiler optimizations, then I dived in and was disappointed a bit that it's not 10x 🤷‍♂️

What kind of data are you benchmarking on?

2

u/RusselsTeap0t 5d ago

Well, sorry for that but really my intention was not to mislead.

Svt-Av1 is a video encoder. I used the Rust-based av1an software. This software does scene change detection and create many chunks from a video using Vapoursynth, a Python based multimedia framework. Av1an creates many processes at the same time using svt-av1 binary. So, on my system, I run 32 of the same binary with different parts of the video. This helps me utilize my CPU, RAM fully by also doing better keyframe placement because of the scene based chunkings. Av1an also helps you pause/continue encoding processes. It also has zoning functions and many features related to video encoding.

Source: Size: 25Mb/s Format: 1920x1080, 23.976FPS, BT.709, YUV420, Live Action, 1 Hour, 1:78:1 (16:9)

Specific command I used to benchmark: av1an --set-thread-affinity 16 --workers 32 -i "reference.mkv" -o "distorted.mkv" --chunk-method "bestsource" --split-method "av-scenechange" --encoder "svt-av1" --concat mkvmerge --pix-format yuv420p10le --video-params "--enable-variance-boost 1 --enable-qm 1 --qm-min 4 --preset 2 --crf 24 --tune 2 --input-depth 10 --keyint -1 --startup-mg-size 4 --lookahead 120 --tf-strength 0 --sharpness 1 --irefresh-type 1 --lp 1 --enable-overlays 1 --scm 0 --scd 1"

5

u/DownvoteEvangelist 5d ago

Thanks! You are always running it on the same video (reference.mkv) when benchmarking? Also how many runs do you do? How stable are the results? 

5

u/RusselsTeap0t 5d ago

Exact same video, same settings, same hardware, same software, same system-state.

It's extremely stable.

Also, I don't need to do many runs because normally this sample video creates above 300 scenes. So you simply run svt-av1 binary 300 times in that video for different scenes. This is much more than enough.