HACT: Co-Designing LLM Architecture and Training with the CPU Cache Hierarchy for Native CPU Inference

Companion code for the HACT preprint — treating the CPU cache hierarchy as a training-time constraint for faster native CPU LLM inference.

Abstract

HACT treats the CPU cache hierarchy not as a fixed deployment target but as a constraint that shapes model architecture and training. It introduces three techniques: CCT (Cache-Capacity-Constrained Training) prunes FFN channels with hard-concrete L0 gates against an explicit cache budget; HPA-Attention (Hardware-Prefetch-Aligned Attention) reorders key/value access into contiguous, prefetcher-friendly block scans while staying numerically equivalent to standard attention; and BPA (Branch-Predictable Activation) regularizes activation firing patterns toward lower branch entropy. Motivating all three, we measure a 2.7× cache cliff in decode throughput at the 16 MB L2 boundary on an Apple M3 Pro — a non-linear step, not the smooth speedup that size-budgeted pruning assumes.

Key finding — the cache cliff

Decode throughput does not decline smoothly as a layer's working set grows. It steps non-linearly the moment the working set no longer fits in a cache level. In a single-thread fp32 matvec sweep on Apple M3 Pro, throughput drops from 279.77 GB/s to 103.34 GB/s (2.7×) as the weight working set crosses the ~16 MB L2 boundary (12.6 MB → 18.9 MB).

This cliff is the core motivation for HACT: sizing a model so each layer's working set lands below a cache boundary yields a step-change in speed, which uniform, budget-based pruning cannot capture.

Repository structure

File	What it does
cliff_detect.py	Cache-cliff sweep: fp32 matvec across sizes that cross L1/L2/LLC, plus a real Pythia-160M FFN-width decode sweep.
cct_l0_experiment.py	CCT with hard-concrete L0 gates on FFN channels + a cache-budget hinge; reports perplexity and genuinely prunable fraction.
cct_pythia.py	Scaled CCT: fine-tunes pretrained Pythia-160M, physically prunes dead channels, reports perplexity and decode tokens/sec.
hpa_experiment.py	HPA-Attention: proves numerical equivalence to standard attention, then microbenchmarks the key-access memory pattern.
bpa_experiment.py	BPA: trains baseline vs. branch-entropy-regularized model, measures firing-pattern entropy vs. perplexity cost.
plot_cliff.py	Regenerates cliff.png from cliff_results.json.

Quick start

Reproduce the cache cliff on your own machine:

pip install torch transformers
python cliff_detect.py --pythia

This runs the matvec sweep (no download) plus the Pythia-160M FFN-width decode sweep (downloads a 160M model) and writes cliff_results.json. Omit --pythia for the matvec sweep only. To redraw the figure: pip install matplotlib && python plot_cliff.py.

Results and reproducibility

Raw sweep data is committed for inspection and reuse:

• cliff_results.json — the matvec and Pythia FFN-width measurements behind the cache-cliff figure (Apple M3 Pro, single P-core, fp32).

The remaining experiment scripts (cct_l0_experiment.py, cct_pythia.py, hpa_experiment.py, bpa_experiment.py) write their own JSON results when run. Training runs use CUDA/MPS; cache and decode measurements are intended for Apple/Arm64 CPUs.

How to cite

@misc{sattineni2026hact,
  title={HACT: Co-Designing LLM Architecture and Training with the CPU Cache Hierarchy for Native CPU Inference},
  author={Sattineni, Jayanth},
  year={2026},
  doi={10.5281/zenodo.21251333},
  url={https://doi.org/10.5281/zenodo.21251333}
}

A machine-readable CITATION.cff is included, so GitHub shows a "Cite this repository" button.

Keywords

cache hierarchy, CPU inference, memory bandwidth, Arm64, structured pruning, cache cliff, hardware-aware training, LLM inference

License

MIT