When it comes to auditing AI skills, a single code comparison is no longer enough
Sh0ny
7 августа 20261 min read
In short
The reusability of an LLM agent’s skill may lie not in the code, but in the instructions and the structure of the workflows. SKILLTRACE offers the ability to check three traces of origin at once and provide the auditor with an explainable basis for a decision.
Plagiarism in LLM-agent ecosystems cannot be reliably detected by comparing source code alone. The same skill can be reused by retaining only the instructions, implementation fragments, or the sequence of actions itself—and a conventional detector won’t catch this.
SKILLTRACE breaks down a skill into three independent traces: Expression for the author’s text, Implementation for the code, and Operational for the operational structure. The latter is represented as a Skill Operational Graph: it describes the skill’s activation, the steps of the procedure, and the resource flows between them.
There’s an important practical aspect here. The LLM is used only once—when extracting the operational trace during skill creation. During the audit phase, the system compares the already saved representations in a deterministic manner and indicates which specific trace confirms potential reuse. This is more useful than a simple binary “similar/not similar” flag: the auditor receives guidance for manual analysis.
On SKILLTRACE-BENCH, with 820 modified reuse examples, 100 source skills, and 751 negative controls, the method achieved an AUROC of 0.938 and an F1 score of 0.898. In a separate review of 36,446 skills, the system generated queues for additional auditing, which complemented the results of repository-level checks.
However, this is not a definitive automated verdict on plagiarism. The result depends on how successfully the three types of traces are extracted, and details regarding errors, processing costs, and the system’s behavior on other datasets are not disclosed in the description. Therefore, SKILLTRACE appears to be more of a tool for sorting suspicious cases and explaining alerts than a replacement for expert review.
If you were building a skills marketplace for agents, what would be the strongest evidence of reuse for you: matching code, matching instruction text, or an identical sequence of actions? Source: cs.AI updates on arXiv.org