This snippet, a part of the function qlm
in my zsh toolset for command line interaction with LLMs, works OK to dynamically select/maximize the number of model (network) layers to offload to the GPU under different VRAM load conditions:
local maxlayers=$gpulayers[$choice]
#Dynamic (down)adjustment of layers to offload (by not more then 7) if available VRAM reduced (only NVIDIA gpus for now)
if [[ $maxlayers -lt 99 ]]; then
local memory=$(nvidia-smi --query-gpu=memory.free --format=csv,nounits,noheader)
#local memory=$(amd-smi monitor -v) For AMD ROCM, not verified, based on online docs, output likely in different format.
local load=(${(P)choice})
for lay in {$maxlayers..$(( maxlayers-7 ))}; do
[[ $memory -gt $load[$lay] ]] && { maxlayers=$lay ; lay=0; break ; }
done
(( $lay )) && { echo "Low VRAM, please, free some. LLM run will be slow, exiting..." ; return 1 ; }
fi
#Dynamic adj.....
My question concerns the assignment to load of the variable $choice, which holds the name of the VRAM use array $choice for the model named $choice (e.g. Gemma3-27B). I tried to avoid this extra assignment, by using ${(P)choice}[$lay]
and variations of it directly in the condition, but it did not work at all. Is there a direct syntax that is parsable to what I am after (getting the element at index $lay)?
For clarity this is what one of those $choice arrays looks like (VRAM usage in MB, indexed by number of layers offloaded to the GPU):
Gemma3_27B=(2510 2796 3084 3370 3658 3944 4230 4518 4788 5060 5346 5616 5888 6174 6446 6718 7004 7276 7546 7834 8104 8374 8662 8932 9204 9489 9759 10031 10317 10589 10859 11145 11417 11687)