Running LLMs locally on an SBC or mini PC is a good way to keep your data private, but they typically require a conventional operating system, which uses a good portion of system memory. NightRun is an experimental open-source project that addresses this issue by booting a machine directly into a local LLM from a USB drive or microSD card, without loading a conventional OS.
By removing the operating system, NightRun makes more RAM and memory bandwidth available for AI inference. The runtime is written in Rust, and during boot it loads a quantized model (1.3 GB to 2.4 GB) directly into RAM while verifying its CRC-32 checksums. After the model is loaded, storage is “sealed,” meaning any later attempt to read from the disk will trigger a hard fault.
Hardware support – Supports 64-bit x86_64 UEFI PCs (Secure Boot disabled) and Raspberry Pi 5 via USB or microSD card
Supported models – Supports Llama 3.2 1B (4GB RAM), Llama 3.2 3B and Granite 4.1 3B (6GB RAM), and Qwen3 4B Instruct 2507 (8GB RAM) models converted to the .nrm format
Optimized inference – Uses AVX2/FMA/F16C kernels on x86 and NEON kernels on Raspberry Pi
Zero-Copy design – Runs quantized Q8_0, Q4_K, and Q6_K models directly from RAM without dequantization
Built-in UI – Includes a framebuffer chat interface with live stats, prompt editing, and conversation history
NightRun generating a response with the Granite 4.1 3B model in its built-in framebuffer chat interface. Completed response from the Granite 4.1 3B model in NightRun’s framebuffer chat interface, with live inference statistics displayed in the status bar.