local-ai

Everything in the inference and training path is written from the ground up: the tokenizer, the tensor and memory system, the training engine, the Metal kernels, quantization, the inference runtime, and the checkpoint format. No PyTorch, no MLX, no llama.cpp, no existing training or inference framework anywhere in the stack.

The thesis

capability = f( training compute, test-time compute, memory, tools )

I am seven to eight orders of magnitude short of frontier training compute, and no amount of cleverness fixes that. But on the other three terms a laptop is richer than a cloud provider: an idle machine can afford to think for sixty seconds on a single query, memory is free and local, and the host has a compiler and a filesystem sitting right there. The whole system is designed around that asymmetry rather than pretending the compute gap can be closed.

Why train rather than run open weights

Running someone else's weights was considered and rejected. Weights are inseparable from the tokenizer and architecture they were trained with, and those are precisely the two things this project exists to control — adopting them would foreclose the experiment on day one.

Someone else's text, on the other hand, costs nothing: text carries no architecture. So the corpus is downloaded and the model is mine. Target corpus is roughly 20B tokens (~85 GB), with the largest planned run at 6B tokens.

Milestone 1: the gates

The project is structured so that measurements, not opinions, decide whether a design survives. Four gating measurements had to pass before any of the architecture was allowed to proceed. All four did.

Matmul throughput — the gate was >55% of peak.

| Kernel | 4096³ | | --- | --- | | naive f32 | 0.9 TFLOP/s | | tiled f32 | 3.8 TFLOP/s | | simdgroup f16 | 8.9 TFLOP/s |

Memory-layer gather — the question was whether random access is even viable on unified memory. Measured against streaming bandwidth: random 77–97%, clustered about the same, and a zipf distribution — the pattern a trained model actually produces — exceeded 100%, because the hot slots simply stay in cache.

Backward scatter cost 1.5–2.2× the forward gather, with 2.2× the worst case across the sweep.

Put together, a full memory layer forward and backward runs in about 7 ms per 65,536 tokens, against roughly 556 ms for the dense SwiGLU it replaces — about a hundredfold saving end to end.

Rules that fell out of the measurements

• Keep memory-layer vdim >= 64. Below 128 bytes per slot the gather goes latency-bound and loses about 30%.
• Use native float atomics for the backward scatter.

Status

Milestone 1 complete, all four gates passed. Training engine work is ongoing.