On-Device AI on the Mac M2: How to Tag Thousands of Transactions on the Cheap

The millfolio project uses a two-tier architecture: the front-end model generates a small program based on the schema, while the local model reads the user’s actual files directly on the device. The main problem is that the local model does not scale with the volume of data. It’s impossible to run thousands of transactions accumulated over a couple of years through AI for every query: even a fraction of a second per record turns into minutes of inference, and the next query starts the whole process over again.

Design Principle

A model’s prediction is something that is calculated once and saved, not run at the moment of a query. All tags are assigned during the indexing phase and stored alongside the records.

Three Types of Tags

Tags are defined in a plain text file named categories.txt—one rule per line; this file is the single source of truth. The types are ordered by cost:

• String tags — the cheapest and most common case. Case-insensitive substring matching, with possible exceptions. Effectively free and covers most of the actual transaction history, since merchant strings are repeated.
• Reference tags — constructed from other tags using @tag. The rule applies if the specified tag is already present in the string. Cycles are safe: tags are only added, so the computation converges.
• AI tags — a fuzzy tail that isn’t captured by any keyword list. The rule is defined as a “yes/no” question, and the local model acts as the judge. This is the only type that requires inference.

At the time of the request, all three types look the same: the generated program filters by .tags, and the response for thousands of records boils down to comparing strings rather than calling the model.

Privacy

The front-end model is provided with tag names and notes on the scope of application, but not keywords. The actual merchant strings remain on the device.

Batch Processing and Priorities

AI tags must be computed at least once—this involves backfilling all existing records plus new arrivals. The work is managed by an on-device orchestrator: a disk-based queue that survives restarts and processes one task at a time. Indexing and backfilling AI tags do not compete with each other, but both take a back seat to interactive queries.

Within a task, classification is batch-processed and deduplicated:

• a set of descriptions is sent to the model in a single prompt; the response comes in the format 1: yes, 2: no, …—a single call classifies an entire batch;
• exact duplicates are merged before reaching the model: a monthly payment that repeats 24 times is classified once, and the verdict applies to all copies.

A small log tracks how much of each rule has already been applied, linked to the monotonically increasing insertion sequence. The finished tag never re-processes old records—new ones are processed incrementally. Between batches, the backfiller “sleeps,” and the length of the pause serves as a priority setting.

Source: Hacker News - Newest: ""AI" "LLM""