Attention as a finite-state machine: 0.00% DSP and sub-milliwatts—but only in simulation

Standard attention in LLMs involves heavy matrix multiplications that dictate the entire stack: from VRAM to GPU TDP. Yusuke Esaki’s preprint proposes rethinking attention as a discrete finite-state machine, where computations are distributed across stationary in-memory cells, and each cell locally performs deterministic state transitions. It sounds like a theoretical exercise, but the author has taken it all the way to synthesis on a real FPGA fabric—and that’s a rare combination.

The mathematical section includes complete proofs of quantization error bounds, the construction of a discrete Softmax with bounded logits, and a proof of equivalence to a deterministic finite automaton with spatial factorization. Belonging to DSPACE(O(n)) implies a linear space with respect to sequence length—potentially better than the quadratic dependence of classical attention. For practitioners, this is the key theoretical hook: if the formalism is correct, attention can be implemented architecturally not as a matrix, but as a hardware transition graph.

Two experiments on an FPGA provide concrete results. On a Zynq-7000 (xc7z020), an array of 1,024 cells (d=128) demonstrated a dynamic core power consumption below 1.0 mW and a normalized energy of 3.81×10⁻⁵ pJ/operation. On UltraScale+ (xcvu9p), a 256-cell subarray—a single attention head—was integrated into a full SoC with AXI4 and PCIe Gen3 x1. The timing is locked at 62.5 MHz, occupying 0.67% of the logic slices and exactly 0.00% of the DSP blocks. Functional simulation on 1,000+ random vectors yielded bit-for-bit agreement with the fixed-point software benchmark.

And here is a critical caveat that the author himself honestly notes: all power consumption figures are estimates from the simulation toolchain; not a single physical measurement was taken on the board. The power consumption of external system memory is strictly excluded. This means that the sub-milliwatt result describes only the synthesized core logic, not the actual end-to-end inference cost. In a real pipeline, memory often dominates the power balance, and 0.00% DSP usage does not guarantee that the system as a whole will be equally energy-efficient.

Nevertheless, the approach is interesting not as a finished product, but as proof of concept. If attention truly boils down to discrete transitions in a linear space, this opens up a path for custom accelerators that require neither tensor cores nor massive matrix engines. A Japanese patent application (No. 2026-177318) has been filed, so the author clearly plans to pursue this line of research.

The main question that the preprint does not yet answer is whether bit precision and the linear space are preserved when scaling beyond a single head and d=128. The 256-cell subarray is a demonstration fragment, not a replacement for a full-fledged attention layer in a production model. For now, it is an elegant theoretical framework with a promising but as yet unconfirmed hardware implementation.

Source: Hacker News - Newest: ""AI" "LLM""