intern

estagiário

A Rust tool that binary-searches for the optimal GPU layer count (-ngl) when running llama-server with a GGUF model, then launches the server with that configuration.

How it works

• Tests different -ngl values (0..255) using binary search.
• For each candidate:
  • Spawns llama-server with that -ngl.
  • Checks if it outputs the expected response.
• Narrows down the highest valid -ngl.
• Spawns the final llama-server with best_fit - 1 to stay safe.

Requirements

• Rust (edition 2024)
• llama-server in $PATH
• GGUF model(s) under ~/guff_models

Build and run

• Build:
  • cargo build --release
• Run:
  • ./target/release/estagiario

CLI arguments

Run with --help for full list:

• -c, --ctx

  • Context size (in 1024-byte units). Default: 32.

• -t, --threads

  • Threads to pass to llama-server. Default: 32.

• --port

  • Port for llama-server. Default: 11434.

• --host

  • Host to bind. Default: 0.0.0.0.

• -b, --batch

  • CPU offload batch size. Default: 512.

• -u, --ubatch

  • GPU offload batch size. Default: 256.

• -v, --verbose

  • Print llama-server logs instead of discarding them.

Model selection

• The tool scans ~/guff_models for .gguf files.
• Lists them as:
  • a) model1.gguf
  • b) model2.gguf
  • ...
• Prompts for a letter to select.

Behavior notes

• During binary search:
  • Each test run’s stdout/stderr is discarded.
  • Results are printed to stderr as:
  • [ok] or [fail]
  • Final server:
    • Runs until interrupted (Ctrl+C).
    • Use --verbose to see its logs.

  Credits

  • This project was built using vibecoding with Qwen 3.6 quantized (served locally), in conjunction with opencode.