VULKAN · LIVE
Per-matmul offload
Stage one: the projections of the chosen layers run on the device. Attention, norms and the KV cache stay on the CPU. Always available.
STAGE 1Every graphics card,one backend.
One Vulkan compute backend so the same engine runs faster on NVIDIA, AMD and Intel graphics, on Windows and Linux first. The model file and the CPU path stay untouched.
What it does.
One system.
Whole-layer spans
Stage two: a run of layers executes entirely on the device, with attention over a device-resident f16 KV cache. One submit per forward pass.
STAGE 2Online re-quantisation
Opt-in: the device gets a smaller copy of the weights so more layers fit in VRAM. The GGUF and the CPU path are not changed.
OPT-INAutomatic placement
The engine decides how many layers fit the card's memory budget and keeps the rest on the CPU.
AUTOHow it works.
Step 1
Detect
The Vulkan loader finds a usable device at runtime.
Step 2
Place
Layers are assigned to the GPU until the VRAM budget is used.
Step 3
Upload
Weights for those layers are copied, optionally re-quantised.
Step 4
Run
Each forward pass crosses the bus once each way.
Spec sheet.
01 / 05API
API.
Vulkan compute, GLSL shaders
- API
- Vulkan compute, GLSL shaders
- Vendors
- NVIDIA, AMD and Intel
- Platforms
- Windows and Linux first; Android and macOS later
- KV cache
- f16, device-resident in span mode
- Status
- In progress; the CPU path remains the default fallback
In short.
GPU offload.
Per-matmul offload of the heavy projections
Whole-layer spans on the device with an f16 KV cache
Opt-in re-quantisation so more layers fit in VRAM
FROM IDEA TO RELEASE
Builders, not managers.
Shubham Mallick
Founder & maintainer
The founder’s storyYou, possibly
Open-source contributors
Fork the repoor just email support@imutechz.in