estagiário

A Rust tool that binary-searches for the optimal GPU layer count (-ngl) when running llama-server with a GGUF model, then launches the server with that configuration.

How it works

Tests different -ngl values (0..255) using binary search.
For each candidate:
- Spawns llama-server with that -ngl.
- Checks if it outputs the expected response.
Narrows down the highest valid -ngl.
Spawns the final llama-server with best_fit - 1 to stay safe.

Requirements

Rust (edition 2024)
llama-server in $PATH
GGUF model(s) under ~/guff_models

Build and run

Build:
- cargo build --release
Run:
- ./target/release/estagiario

CLI arguments

Run with --help for full list:

-c, --ctx
- Context size (in 1024-byte units). Default: 32.
-t, --threads
- Threads to pass to llama-server. Default: 32.
--port
- Port for llama-server. Default: 11434.
--host
- Host to bind. Default: 0.0.0.0.
-b, --batch
- CPU offload batch size. Default: 512.
-u, --ubatch
- GPU offload batch size. Default: 256.
-v, --verbose
- Print llama-server logs instead of discarding them.

Model selection

The tool scans ~/guff_models for .gguf files.
Lists them as:
- a) model1.gguf
- b) model2.gguf
- ...
Prompts for a letter to select.

Behavior notes

During binary search:
- Each test run’s stdout/stderr is discarded.
- Results are printed to stderr as:
- [ok] or [fail]
- Final server:
  - Runs until interrupted (Ctrl+C).
  - Use --verbose to see its logs.