ProcAgent: An LLM-powered assistant for assembling furniture directly on the Jetson Orin

Cloud-based multimodal assistants are ill-suited for everyday tasks: they require constant video streaming, rely on network connectivity, and pose privacy risks. ProcAgent solves this problem by running entirely on a single NVIDIA Jetson AGX Orin device.

The architecture is based on the “propose and verify” principle. The system maintains continuous, low-cost perception and a symbolic task graph, and only invokes expensive visual-linguistic analysis in cases of ambiguity or when the user is likely to deviate from the steps. The LLM-based agent provides reactive responses and proactive prompts with human confirmation.

On the device, the system responds to text queries in approximately 2 seconds and to queries with visual context in 8 seconds. In a study involving 10 participants performing assembly tasks, ProcAgent received high ratings for clarity, practicality, and privacy.

The main trade-off is clear: running locally on edge hardware means balancing speed against the depth of visual analysis. Calling heavy models only on demand conserves resources but creates the risk of missing an error at moments when continuous, lightweight monitoring would not detect it. For everyday scenarios like assembling furniture, this is acceptable, but applying this approach to more critical tasks will require finer-tuning of triggers for resource-intensive checks.

Source: cs.AI updates on arXiv.org