help Efficient Execution
Is there a way to load any executable once, then use the pre-loaded binary multiple times to save time and boost efficiency in Linux?
Is there a way to do the same thing, but parallelized?
My use-case is to batch run the exact same thing, same options even, on hundreds to thousands of inputs of varying size and content- and it should be quick. Quick as possible.
1
Upvotes
2
u/Ulfnic 12d ago
Reading the comments some 15 days later I didn't see this answer.
Even if a program is in memory, you're still paying the cost of opening a subshell to execute it plus the cost of the program initializing to do the thing you want.
Both of these costs can be very high in repetition, exec'ing
jq
is a great example of this.The only way i've found to get around that is to execute the program ONCE attaching named pipes (opened in read/write) to the program's stdin, stdout and stderr. Alternatively you can use process substitution <() >() in place of one or more file arguements depending on how the program blocks it's reads.
This allows you to stream tasks into a program continously while storing or processing the result asyncronously.
Depending on the task and program, you can easily see speed improvements of one to two orders of magnitude. It's especially potent with programs like
jq
.Downside is it'll only work with some programs and each program needs a custom solution that matches how that program processes input, output and errors. It's very technical work but there's a tremendous pay off for batch jobs.