Operating AI in Production: Reliability, Risk, and Observability

Deploying AI is one challenge; operating it reliably in production is another. As organizations move from experimentation to production systems, engineering leaders are navigating new risks — quality regressions, model drift, unpredictable agent behavior, and rising operational costs — that don’t fit traditional software playbooks.

This roundtable brings leaders together to discuss how teams are building trust in AI systems once they are live: what to monitor, how to define reliability targets for probabilistic systems, where to invest in evaluations, synthetic testing, guardrails, and human oversight, and how feedback loops from production can continuously improve performance.

Together we’ll explore emerging practices for:

- Defining reliability signals and SLOs for AI systems

- Observability and tracing across LLM and agentic workflows

- Synthetic testing and production feedback loops to catch regressions early

- Balancing guardrails, autonomy, and human review

- Organizational and investment choices required to operate trustworthy AI at scale

Participants are encouraged to bring a production lesson learned (or near miss) and the metric they wish they had started tracking sooner.

You’ll leave with:

- A practical checklist of signals worth instrumenting early

- Frameworks for reliability and risk management in production AI

- Peer-tested patterns for testing, monitoring, and resilient system design

- New perspectives on the leadership and operating models emerging around AI systems