Why the automation of incident reports using LLMs worries engineers

Lorin Hochstein, a systems reliability engineer and author of the blog Surfing Complexity, responded to Reginald Braithwaite’s sarcastic post about a future where incident reports would be written entirely by LLMs. Although Braithwaite was clearly being ironic, Hochstein is convinced that such a future is indeed approaching—and it worries him.

First, a disclaimer: Collecting data for an incident report is a labor-intensive routine, and AI can indeed help here by reducing the “toil.” The problem lies elsewhere—in the difference between using an LLM as an assistant to prepare the material and completely handing over the actual writing of the text to it.

Writing as a Way of Thinking

The author refers to a famous quote by cartoonist Dick Gindon: “Writing is nature’s way of showing you how sloppily you think.” Until a person tries to explain what happened in their own words, they won’t notice the gaps in their own understanding.

Leslie Lamport’s quote supports this same idea: “If you think without writing it down, you only think you’re thinking.” It is precisely this stage—self-verification through formulation—that LLMs completely eliminate from the process.

The Danger of Plausible Texts

When a model generates a report, the reader receives text that sounds convincing but may contain fabricated connections between systems or omit real, critical interactions. Since no one has done the hard work of synthesizing the data, errors will go unnoticed—after all, the less effort put into writing, the less effort will be put into verification.

Hochstein believes that the risks are higher for incident reports than when using AI to write code or in AI SRE tasks:

• When working with code, there is testing to determine whether the solution works;
• in operations tasks, AI either helps resolve the incident or it doesn’t—nature itself provides the evaluation;
• a report, however, lacks such an explicit criterion for correctness—it may look convincing but be factually incorrect.

Simulacra Instead of Analysis

In the author’s view, since writing reports is time-consuming and tedious, the temptation to offload this work onto AI will be enormous. But the model won’t interview the people involved in the incident—the resulting texts will turn into simulacra: formally correct, but lacking any real understanding of how the system works.

Hochstein fears that this will also reduce teams’ ability to learn from their own mistakes—and on top of that, such reports will likely be aggregated by AI later on, thus completing the cycle of superficial information processing.

Source: Hacker News - Newest: ""AI" "LLM""