AXIOMINFERENCE
Research
Small models.
Real adaptation.
Project Kestrel
Kestrel-Nano v0.1 trained — Phase D nextA trained small-model program, not only a design.
Kestrel‑Nano v0.1 is a 157M-parameter from-scratch model trained on 1.311B tokens. The implementation, trainer, tokenizer, data pipeline, CPU smoke tests, Phase C dataset, and generalization evaluation suite are complete.
The evaluation is intentionally honest: v0.1 generates coherent text, but it remains undertrained and learned more form than content. Phase D will over-train on the completed 4.86B-token curriculum and compare directly against the v0.1 control.
Read the master planLast commit recently
Why this project exists
Independence from the cloud. People in low-connectivity regions — and anyone who values privacy and ownership — deserve capable local models. Nano targets roughly 100 MB quantized; Mini roughly 300 MB. The research path is designed for modest local hardware as well as a larger training run.
Small doesn't mean shallow. Recent evidence from looped models, hybrid attention, and sparse memory research shows small models close the gap through architecture shape, data quality, and adaptive compute — not just scale.
The "frozen weights" problem. Today's models largely stop learning the moment training ends. Kestrel's memory subsystem — recurrent session state, sparse memory layers, and nightly consolidation — is a direct engineering response to that.
Hardware-grounded, by rule. The current lab GPU is an RTX 3060 12 GB. The implementation remains pure PyTorch and matmul-heavy, with local and cloud configurations measured separately instead of assuming a paper design will run.
The five pillars — v0.1
Hybrid sequence mixer
Three gated linear-attention blocks to one full-attention block — O(1) inference state per token, long context on 8 GB, a persistent "mind-state."
Looped core
The middle block group re-runs one to four times — deeper effective reasoning without more parameters, and compute that adapts to the problem.
Product-key memory layers
Large sparse key-value banks give knowledge capacity without extra FLOPs — the "plastic tissue" that makes continual learning possible.
Token efficiency
A superword BPE tokenizer plus a multi-token-prediction auxiliary head — roughly 20–30% more text per FLOP and per context window.
Layered memory & consolidation
Context flows into recurrent state, into memory slots, into nightly sparse consolidation — a model that learns from use without catastrophic forgetting.
Ground rules
Every deviation from a vanilla transformer has to beat the matched-compute baseline or it gets cut — evidence or ablation, no exceptions. Kestrel-Nano v0.1, its limitations, and the Phase C preparation work are documented openly. The next claim will only be made after Phase D is trained and compared against the v0.1 control.
View the repositoryAgent alignment: SFT & Direct Preference Optimization
MalxLabs-Fable5
SFT + DPO post-training · benchmarked against base modelFable-Coder-7B-DPO — agent trajectory & reasoning alignment.
Where Kestrel trains small models from scratch and MalxLabs-V4 targets legacy Pascal GPUs, Fable-Coder focuses on autonomous agent tool execution and logical programming correctness. Starting from Qwen2.5-Coder-7B-Instruct, it applies multi-turn Supervised Fine-Tuning (SFT) with LoRA on agent trajectories (MoreThought/Fable-5.1-Max-Reasoning-Filtered-5000x), followed by Direct Preference Optimization (DPO) to penalize cyclic tool loops and reward clean execution syntax.
Fused into native BF16, patched for GGUF compilation, and quantized to an optimized 4.36 GB Q4_K_M binary running in ~5.1 GB VRAM. In a controlled 33-prompt comparative benchmark across 7 domains against its base model, Fable-Coder scored 80.0% (92/115) with an 85.0% vs. 65.0% (+20.0%) lead in complex reasoning, resolving Bayesian probability problems where the base model suffered decimal hallucination.
Last commit recently
A different angle: fine-tune, don't retrain
MalxLabs-V4
SFT fine-tune shipped & benchmarked against its base modelReasoning & logic optimization on legacy hardware.
Where Kestrel trains a small architecture from scratch, MalxLabs-V4 takes the opposite approach: start from an existing model — DeepSeek-R1-Distill-Qwen-1.5B — and supervised-fine-tune it (on OpenMathInstruct-2 and CodeFeedback-Filtered, via a rented RTX 4500) for math and code reasoning. The result is quantized to GGUF Q4_K_M and run entirely on a 2014-era GTX 1080 (8 GB, Pascal) — no cloud inference, no modern GPU required.
It ships with a real, measured head-to-head against the untuned base model across grade-school math, harder logic problems, and execution-verified code generation — same hardware, same settings — plus a separate inference benchmark on a non-GPU laptop (Intel i5-1035G7, integrated graphics only).
View the repositoryLast commit recently