AXIOMINFERENCE
Research
Small models.
Real adaptation.
Project Kestrel
In active design — not yet implementedA small, dense, adaptively-learning architecture.
Two owned models: Kestrel‑Nano (~150M), trained entirely on a single GTX 1080 for $0, and Kestrel‑Mini (~500M), cloud-pretrained for under $25. Both are built to run on nearly any computer.
A kestrel is a small falcon that hunts by hovering — reading the wind and adjusting continuously. That's the design thesis: a small model that stays light but adapts in place, instead of a frozen giant.
Read the master plan
Why this project exists
Independence from the cloud. People in low-connectivity regions — and anyone who values privacy and ownership — deserve capable local models. Nano runs on roughly 100 MB quantized; Mini on roughly 300 MB. An 8 GB GPU, a CPU-only laptop, in principle a phone.
Small doesn't mean shallow. Recent evidence from looped models, hybrid attention, and sparse memory research shows small models close the gap through architecture shape, data quality, and adaptive compute — not just scale.
The "frozen weights" problem. Today's models largely stop learning the moment training ends. Kestrel's memory subsystem — recurrent session state, sparse memory layers, and nightly consolidation — is a direct engineering response to that.
Pascal-safe, by rule. Every component must run in pure PyTorch on a GTX 1080 locally, and in bf16 compiled form on a cloud pod for Mini — the same architecture, legal in both worlds.
The five pillars — v0.1
Hybrid sequence mixer
Three gated linear-attention blocks to one full-attention block — O(1) inference state per token, long context on 8 GB, a persistent "mind-state."
Looped core
The middle block group re-runs one to four times — deeper effective reasoning without more parameters, and compute that adapts to the problem.
Product-key memory layers
Large sparse key-value banks give knowledge capacity without extra FLOPs — the "plastic tissue" that makes continual learning possible.
Token efficiency
A superword BPE tokenizer plus a multi-token-prediction auxiliary head — roughly 20–30% more text per FLOP and per context window.
Layered memory & consolidation
Context flows into recurrent state, into memory slots, into nightly sparse consolidation — a model that learns from use without catastrophic forgetting.
Ground rules
Every deviation from a vanilla transformer has to beat the matched-compute baseline or it gets cut — evidence or ablation, no exceptions. The science is proven locally at 20M–60M parameters for $0 first; scaling to Nano and Mini follows a costed, resume-from-checkpoint ladder, capped at one week of local GPU time plus $25 of cloud spend. Nothing here is announced as finished until it's actually trained and evaluated — the current deliverables are the design documents themselves, published in full on GitHub.
View the repository