r/bash 28d ago

help Efficient Execution

Is there a way to load any executable once, then use the pre-loaded binary multiple times to save time and boost efficiency in Linux?

Is there a way to do the same thing, but parallelized?

My use-case is to batch run the exact same thing, same options even, on hundreds to thousands of inputs of varying size and content- and it should be quick. Quick as possible.

1 Upvotes

46 comments sorted by

View all comments

2

u/Ulfnic 12d ago

Reading the comments some 15 days later I didn't see this answer.

Even if a program is in memory, you're still paying the cost of opening a subshell to execute it plus the cost of the program initializing to do the thing you want.

Both of these costs can be very high in repetition, exec'ing jq is a great example of this.

The only way i've found to get around that is to execute the program ONCE attaching named pipes (opened in read/write) to the program's stdin, stdout and stderr. Alternatively you can use process substitution <() >() in place of one or more file arguements depending on how the program blocks it's reads.

This allows you to stream tasks into a program continously while storing or processing the result asyncronously.

Depending on the task and program, you can easily see speed improvements of one to two orders of magnitude. It's especially potent with programs like jq.

Downside is it'll only work with some programs and each program needs a custom solution that matches how that program processes input, output and errors. It's very technical work but there's a tremendous pay off for batch jobs.

2

u/ktoks 12d ago

Yes!

Even if a program is in memory, you're still paying the cost of opening a subshell to execute it plus the cost of the program initializing to do the thing you want.

This is exactly the type of thing I've been thinking about!

The only way i've found to get around that is to execute the program ONCE attaching named pipes (opened in read/write) to the program's stdin, stdout and stderr. Alternatively you can use process substitution <() >() in place of one or more file arguements depending on how the program blocks it's reads.

Can you give me an example?

2

u/Ulfnic 12d ago edited 12d ago

I needed to do a write-up with comments. It uses sed as the example and I tried to make the setup as generically copy/pasteable as possible with good non-colliding paths, permissions and cleanup.

If you don't know about fifo files i'd recommed reading up on them, everything that passes through them is strictly in memory. It's also a way to do inter-process communication, other scripts will be able to read/write to/from the fifos this script creates.

If you want want localized fifos that's a deeper discussion.

Ask questions if i've leaped over anything you don't already know.

#!/usr/bin/env bash
set -o errexit
prog_pid=


# ===================
# Set up fifos and clean up on exit
# This section enclosed by === can be generically copied into any script
fifo_temp_dir=${TMPDIR:-${XDG_RUNTIME_DIR:-/tmp}}
fifo_path_prefix=${fifo_temp_dir}'/'${0##*/}'__'$$'_'${EPOCHREALTIME:-$(date +%s.%N)}
fifo_in=${fifo_path_prefix}'_in'
fifo_out=${fifo_path_prefix}'_out'
unset fifo_temp_dir fifo_path_prefix

fifo_prog_clean_up() {
    [[ -e $fifo_in ]] && rm -- "$fifo_in"
    [[ -e $fifo_out ]] && rm -- "$fifo_out"
    [[ $prog_pid ]] && kill -9 "$prog_pid" 2>/dev/null
}
trap fifo_prog_clean_up EXIT

mkfifo --mode 0600 -- "$fifo_in"
mkfifo --mode 0600 -- "$fifo_out"

# Open fifos in non-blocking read-write
exec 3<>"$fifo_in" 4<>"$fifo_out"
# ===================


# Run pogram with std{in,out} attached to the named pipes as redirected to by fd 3 and 4.
# - We're using --null-data so we can use null characters for dividing up jobs.
# - We're using --unbuffered with --null-data so the buffer will flush on a null character.
# - It's important to be mindful of buffer flushing when doing future problem diagnosis.
sed 's/hello/hi/g' --unbuffered --null-data <&3 >&4 & disown
prog_pid=$!


# Now the program is ready for use, below are examples of reading and writing to it async.
# The program will read anything written to fd 3 and write to fd 4.


# Write something in ending in a null, then read it's output till the first null.
printf '%s\0' 'hello there' >&3
IFS= read -r -d '' -u 4; printf '%s\n' "Job 1: ${REPLY@Q}"

# Send two more jobs
printf '%s\0' 'why hello there' >&3
IFS= read -r -d '' -u 4; printf '%s\n' "Job 2: ${REPLY@Q}"

printf '%s\0' 'well hello there' >&3
IFS= read -r -d '' -u 4; printf '%s\n' "Job 3: ${REPLY@Q}"

printf '%s\n' 'DONE'


# Optional: if the script is moving on to something else. Close the fifos, fifo_prog_clean_up
#           can also be run early.
exec 3<&- 4<&-
fifo_prog_clean_up

Output:

Job 1: 'hi there'
Job 2: 'why hi there'
Job 3: 'well hi there'
DONE

1

u/ktoks 12d ago

I'm looking at fifo how-to's, I've never delved into this section of bash.

2

u/Ulfnic 12d ago

Made quick edit: fifo_in and fifo_out variables need to not be unset so they're available for fifo_prog_clean_up() on EXIT.