|
|
@@ -1,28 +1,35 @@
|
|
|
-# estagiário
|
|
|
+# Intern
|
|
|
|
|
|
A Rust tool that binary-searches for the optimal GPU layer count (-ngl) when running llama-server with a GGUF model, then launches the server with that configuration.
|
|
|
|
|
|
## How it works
|
|
|
|
|
|
-- Tests different -ngl values (0..255) using binary search.
|
|
|
-- For each candidate:
|
|
|
- - Spawns llama-server with that -ngl.
|
|
|
- - Checks if it outputs the expected response.
|
|
|
-- Narrows down the highest valid -ngl.
|
|
|
-- Spawns the final llama-server with best_fit - 1 to stay safe.
|
|
|
+```
|
|
|
+Tests different -ngl values (0..255) using binary search.
|
|
|
+For each candidate:
|
|
|
+ Spawns llama-server with that -ngl.
|
|
|
+ Checks if it outputs the expected response.
|
|
|
+Narrows down the highest valid -ngl.
|
|
|
+Spawns the final llama-server with best_fit - 1 to stay safe.
|
|
|
+```
|
|
|
|
|
|
## Requirements
|
|
|
|
|
|
-- Rust (edition 2024)
|
|
|
-- llama-server in $PATH
|
|
|
-- GGUF model(s) under ~/guff_models
|
|
|
+```
|
|
|
+Rust (edition 2024)
|
|
|
+llama-server in $PATH
|
|
|
+GGUF model(s) under ~/guff_models
|
|
|
+```
|
|
|
|
|
|
## Build and run
|
|
|
|
|
|
-- Build:
|
|
|
- - cargo build --release
|
|
|
-- Run:
|
|
|
- - ./target/release/estagiario
|
|
|
+```
|
|
|
+Build:
|
|
|
+ cargo build --release
|
|
|
+
|
|
|
+Run:
|
|
|
+ ./target/release/intern
|
|
|
+```
|
|
|
|
|
|
## CLI arguments
|
|
|
|
|
|
@@ -53,23 +60,30 @@ Run with `__help` for full list:
|
|
|
|
|
|
## Model selection
|
|
|
|
|
|
-- The tool scans ~/guff_models for .gguf files.
|
|
|
-- Lists them as:
|
|
|
- - a) model1.gguf
|
|
|
- - b) model2.gguf
|
|
|
- - ...
|
|
|
-- Prompts for a letter to select.
|
|
|
+```
|
|
|
+The tool scans ~/guff_models for .gguf files.
|
|
|
+Lists them as:
|
|
|
+ a) model1.gguf
|
|
|
+ b) model2.gguf
|
|
|
+ ...
|
|
|
+Prompts for a letter to select.
|
|
|
+```
|
|
|
|
|
|
## Behavior notes
|
|
|
|
|
|
-- During binary search:
|
|
|
- - Each test run’s stdout/stderr is discarded.
|
|
|
- - Results are printed to stderr as:
|
|
|
- - <ngl> [ok] or <ngl> [fail]
|
|
|
-- Final server:
|
|
|
- - Runs until interrupted (Ctrl+C).
|
|
|
- - Use --verbose to see its logs.
|
|
|
+```
|
|
|
+During binary search:
|
|
|
+ Each test run's stdout/stderr is discarded.
|
|
|
+ Results are printed to stderr as:
|
|
|
+ <ngl> [ok] or <ngl> [fail]
|
|
|
+
|
|
|
+Final server:
|
|
|
+ Runs until interrupted (Ctrl+C).
|
|
|
+ Use --verbose to see its logs.
|
|
|
+```
|
|
|
|
|
|
## Credits
|
|
|
|
|
|
-- This project was built using vibecoding with Qwen 3.6 quantized (served locally), in conjunction with opencode.
|
|
|
+```
|
|
|
+This project was built using vibecoding with Qwen 3.6 quantized (served locally), in conjunction with opencode.
|
|
|
+```
|