ITSM data becomes useful only after a layer of interpretation

The problem with ITSM is not a shortage of data but that tickets are poorly suited to making decisions. Researchers built a pipeline that turns raw exports into a structure intelligible to managers, sales and customer success — and it was the interpretation layer, rather than the neural network, that proved the more important part.

First an LLM brings heterogeneous records to a single schema. Clustering algorithms then merge similar enquiries: HDBSCAN looks for small sub-themes while hierarchical clustering gathers them into larger Main-topics. What comes out is not a stream of tickets but a map of problems at two levels of detail.

That is a useful separation. A manager does not need to read hundreds of enquiries to see a large theme. The team that will act needs Sub-topics with enough specificity. Such a result is closer to a working instrument for discussing priorities than to yet another dashboard of charts.

The evaluation looked encouraging: five specialists from Sales Engineering and customer success assessed six artefacts. All four metrics — intelligibility, applicability, trust and likelihood of use — averaged above 4 points out of 5. Trust proved the most consistent signal.

But that is where the line runs between a promising approach and a proven product. The study involved only five assessors and six artefacts; high scores show the result seems useful and intelligible but do not prove it already improves sales, support or management decisions. Besides, the quality of the whole chain still depends on the original tickets: if the records are incomplete or chaotic, the LLM can only tidily structure poor material.

The main conclusion for companies is simple: the place to start is not the question of which model to connect to ITSM but the question of what form of decision a human should receive. The model here is one component of a pipeline, and the value appears at the junction of data normalisation, topic grouping and human review.

Would you trust a management decision to such a system at an average trust score above 4 out of 5, when only five specialists have tested it so far? Source: cs.AI updates on arXiv.org