The OSS AI stack is easy to test, but there's been no word from the developers

A discussion has popped up on Hacker News that sounds simple but hits a nerve: Who’s actually running open-source AI infrastructure in production, and what challenges are they facing?

The question covers the entire stack: inference, orchestration, observability, vector search, data pipelines, evaluation, and model management. The author honestly admits the key point—most OSS projects are easy to test, but operating them in production is a completely different challenge.

And here’s what’s telling: as of this writing, there are zero comments. Not because the topic isn’t interesting, but because those who are actually doing this usually don’t have time to write detailed responses on HN. And those who want to often lack production experience—they’ve only done proof-of-concept work.

This reflects a real gap in the industry. The hype surrounding open-source models and frameworks is enormous, but there’s a chasm between demos and production deployment: cost management, GPU availability, security, and manageability. Each of these points is a separate engineering challenge for which there is no ready-made playbook.

My conclusion: the market for OSS AI tools is growing faster than the practice of deploying them. If you’re choosing a stack for production—don’t trust the README or the stars on GitHub. Look for people who are already running it in production, and ask about failure modes, not features.

Source: Hacker News - Newest: ""AI" "LLM""