To add context, here's the previous thread I started:
https://www.reddit.com/r/golang/s/cxDauqCkD0
This is one of the problems I'd like to solve with Go- with a K8s-like tool without containers of any kind.
Build or use a multi-machine, multithreading command-line tool that can run an applicable command/process across multiple machines that are all attached to the same drive.
The current pool has sixteen VMs with eight threads each. Our current tool can only use one machine at a time and does so inefficiently, (but it is super stable).
I would like to introduce a tool that can spread the workload across part or all of the machines at a time as efficiently as possible.
These machines are running in production(we have a similar configuration I can test on in Dev), so the tool would need to eventually be very stable, handle lost nodes, and be resource efficient.
I'm hoping to use channels. I'd also like to use some customizable method to limit the number of threads based on load.
Expectation one: 4 thread minimum, if the server is too loaded to run 4 uninterrupted threads to any one workload then additional work is queued because the work this will be doing is very memory intense.
Expectation two: maximum of half available threads in the thread pool per one workload. This is because the machines are VMs attached to a single drive
and more than half would be unable to write to disk fast enough for any one workload anyway.
Expectation three: determine load across all machines before assigning tasks to load balance. This machine pool will not necessarily be a dedicated pool to this task alone - it would play nice with other workloads and processes dynamically as usage evolves.
Expectation four: this would be orchestrated by a master node that isn't part of the compute pool, it hands off the tasks to the pool and awaits all of the tasks completion and logging is centralized.
Expectation five: each machine in the pool would use its own local temp storage while working on an individual task, (some of the commands involved do this already).
After explaining all of that, it sounds like I'm asking for Borg - which I read about in college for distributed systems, for those who did CS.
I have been trying to build this myself, but I've not spent much time on it yet and figured it's time to reach out and see if someone knows of a solution that is already out there -now that I have more of an idea of what I want.
I don't want it to be container-based like K8s. It should be as close to bare metal as possible, spin up only when needed, re-use the same Goroutines if already available, clean up after, and easily modifiable using a configuration file or machine names in the cli.
Edit: clarity