|
@@ -4,22 +4,11 @@ A Rust tool that binary-searches for the optimal GPU layer count (-ngl) when run
|
|
|
|
|
|
|
|
## How it works
|
|
## How it works
|
|
|
|
|
|
|
|
-```
|
|
|
|
|
-Tests different -ngl values (0..255) using binary search.
|
|
|
|
|
-For each candidate:
|
|
|
|
|
- Spawns llama-server with that -ngl.
|
|
|
|
|
- Checks if it outputs the expected response.
|
|
|
|
|
-Narrows down the highest valid -ngl.
|
|
|
|
|
-Spawns the final llama-server with best_fit - 1 to stay safe.
|
|
|
|
|
-```
|
|
|
|
|
|
|
+Tests different -ngl values (0..255) using binary search. For each candidate: Spawns llama-server with that -ngl. Checks if it outputs the expected response. Narrows down the highest valid -ngl. Spawns the final llama-server with best_fit - 1 to stay safe.
|
|
|
|
|
|
|
|
## Requirements
|
|
## Requirements
|
|
|
|
|
|
|
|
-```
|
|
|
|
|
-Rust (edition 2024)
|
|
|
|
|
-llama-server in $PATH
|
|
|
|
|
-GGUF model(s) under ~/guff_models
|
|
|
|
|
-```
|
|
|
|
|
|
|
+Rust (edition 2024), llama-server in $PATH, GGUF model(s) under ~/guff_models
|
|
|
|
|
|
|
|
## Build and run
|
|
## Build and run
|
|
|
|
|
|
|
@@ -52,30 +41,12 @@ Options:
|
|
|
|
|
|
|
|
## Model selection
|
|
## Model selection
|
|
|
|
|
|
|
|
-```
|
|
|
|
|
-The tool scans ~/guff_models for .gguf files.
|
|
|
|
|
-Lists them as:
|
|
|
|
|
- a) model1.gguf
|
|
|
|
|
- b) model2.gguf
|
|
|
|
|
- ...
|
|
|
|
|
-Prompts for a letter to select.
|
|
|
|
|
-```
|
|
|
|
|
|
|
+The tool scans ~/guff_models for .gguf files, lists them as a) model1.gguf, b) model2.gguf, etc., and prompts for a letter to select.
|
|
|
|
|
|
|
|
## Behavior notes
|
|
## Behavior notes
|
|
|
|
|
|
|
|
-```
|
|
|
|
|
-During binary search:
|
|
|
|
|
- Each test run's stdout/stderr is discarded.
|
|
|
|
|
- Results are printed to stderr as:
|
|
|
|
|
- <ngl> [ok] or <ngl> [fail]
|
|
|
|
|
-
|
|
|
|
|
-Final server:
|
|
|
|
|
- Runs until interrupted (Ctrl+C).
|
|
|
|
|
- Use --verbose to see its logs.
|
|
|
|
|
-```
|
|
|
|
|
|
|
+During binary search, each test run's stdout/stderr is discarded. Results are printed to stderr as `<ngl> [ok]` or `<ngl> [fail]`. The final server runs until interrupted (Ctrl+C). Use `--verbose` to see its logs.
|
|
|
|
|
|
|
|
## Credits
|
|
## Credits
|
|
|
|
|
|
|
|
-```
|
|
|
|
|
This project was built using vibecoding with Qwen 3.6 quantized (served locally), in conjunction with opencode.
|
|
This project was built using vibecoding with Qwen 3.6 quantized (served locally), in conjunction with opencode.
|
|
|
-```
|
|
|