Voice Assistant on a Local PC: Gemini Plans, the Phone Takes the Lead

Desktop voice assistants still operate on the “send audio to the cloud, get a response” model. AnovaX does the opposite: the entire pipeline runs on the user’s machine, and the desktop becomes a surface for actions. The LLM (Gemini) doesn’t “chat” here; instead, it generates a JSON plan for tool invocations, which a multi-agent orchestrator converts into typed child agents running on a limited pool of threads.

The architecture consists of a single Python process that connects the wake word, the speech pipeline, the scheduler, the security layer (whitelist + denylist), and the adaptive recovery loop. Each tool is a separate agent class: AppAgent, TypingAgent, BrowserAgent, and six others. Each has its own timeout, retry policy, and locking mechanism for shared resources. MetaAgent allows the scheduler to delegate a sub-goal to itself, but no deeper than two levels of nesting—a strict limit against recursive proliferation.

The most interesting part is recovery and latency hiding. When a key step fails, a compact ReAct prompt kicks in to try to fix the situation. At the same time, read-only tools are launched speculatively to mask Gemini’s latency. This isn’t magic, but an engineering trick: while the LLM is thinking, the system performs useful work that doesn’t require its response.

Control is handled from a phone via local Wi-Fi. A Flask server streams every event in the agents’ lifecycle to the smartphone in real time and streams the laptop screen via MJPEG so the user can see the commands being executed. The LLM never interacts directly with the keyboard—it only generates a plan, which is then executed by typed agents.

The main value of the project isn’t competition with Siri or Alexa, but rather the proof that: a human-readable assistant consisting of several thousand lines of code is capable of opening apps, entering text, launching searches, coordinating parallel actions, recovering from errors, and being controlled via a phone from another room. For practitioners, this serves as a practical reference for how to build local agents without relying on the cloud.

Source: cs.AI updates on arXiv.org