Quantization Is a Half-Measure: On-Device LLMs Hit the Memory Bus Limit
Sh0ny
23 июля 20262 min read
Updated 25 июля 2026In short
Extreme quantization helps fit the model into RAM, but it doesn’t solve the main problem—bus bandwidth. The CaSA project proposes performing inference directly in DRAM, bypassing the SoC bus. An analysis of the hardware reality that software hype tends to ignore.
Everyone is talking about on-device LLMs: privacy, cost savings on cloud inference, and a new hardware upgrade cycle. Apple and chip manufacturers are actively moving in this direction. Quantization down to 1-bit and ternary weights—as seen in PrismML Bonsai—allows a 27B model to be compressed to a size that fits into a smartphone’s memory. But fitting into RAM and running in RAM are two different things.
A problem the software world overlooks: while the weights reside in memory, for each token they must be routed via the SoC bus to the NPU’s computing units. Gigabytes of data are shuttled back and forth at every step of inference. The result is NPU thermal throttling and rapid battery drain. This is the very “memory wall” that hardware engineers are well aware of, but which the ML community often overlooks.
The CaSA project proposes a radical approach: instead of moving data to the computation units, it performs computations directly within DRAM using charge-sharing. LLM ternary inference is performed in commercial off-the-shelf memory, completely bypassing the bus. The project’s author explicitly states that he is tired of academic Processing-In-Memory simulations that ignore the physics of “bare metal.”
The logic is clear: quantization is the first software step; it reduces the volume of data. But the physical bottleneck isn’t the size of the weights—it’s the cost of moving them. CaSA attempts to eliminate the movement itself. If the architecture reaches a working prototype on real devices, it will change not only on-device AI but also the entire balance between cloud-based and local inference.
The project is still in its early stages—it has zero comments and one upvote on Hacker News. But the question it raises is more important than the current implementation: the industry is optimizing software but has run up against a fundamental limitation of von Neumann architecture. As long as computation and memory are physically separated by a bus, even 1-bit models will pay for each token with power consumption and heat generation.