Why the consensus of AI judges can be misleading

In today’s roundup: a study on the dependence of LLM judge errors, ideas for more reliable prompt engineering, and practical case studies of launching AI tools.

🔥 Hot:

🔹 Study shows that agreement among LLM judges does not guarantee the correct answer — Models are often trained in similar ways and may independently arrive at the same error, so consensus cannot be considered sufficient evidence of quality.

➡️ News:

🔹 Greece’s prime minister says governments are not prepared for the consequences of AI — In an interview, he noted that many regulators are already trying to solve yesterday’s problems while the technology is changing rapidly.

➡️ Useful materials:

🔹 Study shows that discussion between groups improves collective AI evaluations — The average opinion of several small deliberating groups may outperform the classic “wisdom of the crowd” effect. 🔹 The psychological model of “what we know and what we don’t know” helps improve prompt engineering — The approach suggests separating what is known, unknown, and unknowingly overlooked before tackling a complex task.

➡️ Discussions and case studies:

🔹 Developer examines five technical problems of an AI aggregator for Telegram channels — A solo project collects posts, removes duplicates and noise, and then turns the result into a short audio digest. 🔹 User runs Open Terminal in an isolated VM to give the agent safer access to the system — This gives the agent more freedom to work with the CLI, while potential damage is limited to the virtual machine.

📝 If you’d like to add other news and materials to the list, write in the comments.