Scaling the Frontier of Agentic Workflows: Michael Levan on the Operational Realities of AI Agents

As the enterprise landscape shifts from simple generative AI implementations toward more complex, autonomous systems, the conversation around infrastructure has taken a turn toward the "agentic." In a recent and deeply technical discussion, industry commentators Ned and Kyler sat down with cloud and DevOps expert Michael Levan to unpack the burgeoning complexities of running AI agents securely and at scale. The conversation, which touched upon everything from specialized AI gateways to the intricacies of sandboxing, highlighted a pivotal moment in the evolution of artificial intelligence: the transition from experimental prototypes to production-ready operational workloads.

The concept of an AI agent—a system capable of not just generating text, but taking actions, making decisions, and interacting with other software tools—represents a significant leap in capability. However, as Levan pointed out during the session, with increased capability comes a unique set of architectural and security challenges that traditional DevOps and platform engineering practices are only beginning to address.

The Shift Toward Agentic Workflows

For much of the past two years, the corporate world has focused on Large Language Models (LLMs) as advanced chatbots. However, the industry is rapidly moving toward "agentic workflows." Unlike a standard chatbot that provides a static response to a prompt, an agent is designed to achieve a goal by breaking it down into smaller tasks, selecting the appropriate tools, and iterating until the objective is met.

This shift introduces a new layer of non-deterministic behavior into the software stack. During the discussion, Ned, Kyler, and Levan explored how this unpredictability necessitates a fundamental rethink of the underlying infrastructure. When an agent is empowered to execute code or query a database, the traditional boundaries of security and performance monitoring become blurred. The group emphasized that "day two" operations—the ongoing management, scaling, and securing of these systems—are where many organizations currently find themselves struggling.

The Role of Specialized AI Gateways

One of the most critical components identified by Levan in the quest for scalable AI is the implementation of specialized AI gateways. In a traditional microservices architecture, an API gateway manages traffic, authentication, and rate limiting. In the world of agentic workloads, the AI gateway serves a much more nuanced purpose.

An AI gateway acts as a sophisticated proxy between the application and the various LLMs or agentic frameworks being utilized. It provides a centralized point of control for governance, allowing organizations to manage API keys, monitor token usage, and enforce security policies across multiple providers. As Levan noted, without such a gateway, companies risk a "Wild West" scenario where individual developers or agents might inadvertently rack up massive costs or leak sensitive data through unmonitored API calls.

Furthermore, these gateways are becoming essential for "prompt governance." By intercepting requests, a gateway can check for prompt injections or ensure that PII (Personally Identifiable Information) is scrubbed before it ever reaches an external model provider. This layer of abstraction not only enhances security but also provides the flexibility to swap underlying models without re-architecting the entire application.

Semantic Routing and Intelligent Traffic Management

The discussion moved deeper into the mechanics of how these agents interact with infrastructure, specifically focusing on the concept of semantic routing. In traditional networking, routing is based on fixed paths, headers, or IP addresses. Semantic routing, however, uses the intent or the "meaning" of a request to determine its destination.

Levan explained that in an agentic system, a request might be routed differently depending on its complexity or the specific domain expertise required. For example, a simple query might be routed to a smaller, more cost-effective model, while a complex mathematical problem might be sent to a high-reasoning model or a specialized tool. Semantic routing allows the infrastructure to be "intent-aware," optimizing for both performance and cost.

This level of intelligent traffic management is crucial for running AI at scale. It prevents the over-provisioning of expensive resources and ensures that the most capable models are reserved for the tasks that actually require their advanced reasoning capabilities. For platform engineers, this represents a new frontier where networking logic is driven by natural language processing and embeddings rather than static rules.

The Necessity of Sandboxing and Execution Security

Perhaps the most pressing security concern discussed by Ned, Kyler, and Levan was the issue of execution. If an AI agent is tasked with analyzing a financial spreadsheet, it may decide to write and execute a Python script to perform the calculation. If that execution happens on a local server or within a sensitive environment, the risks are immense.

Sandboxing emerged as a non-negotiable best practice for agentic workloads. Levan highlighted the importance of creating isolated, ephemeral environments where agents can execute code without posing a threat to the broader system. These "digital playrooms" must be strictly governed, with limited access to the network and file systems.

The challenge, as the participants noted, is balancing security with performance. A sandbox that takes too long to spin up can cripple the responsiveness of an AI agent. Therefore, the infrastructure must support high-performance, low-latency containerization or micro-VM technologies that can provide isolation at the speed of the agent’s thought process.

Robust Observability in a Non-Deterministic World

Traditional observability focuses on metrics like CPU usage, memory, and request latency. While these remain important, the group agreed that AI agents require a new category of observability. Because agents are non-deterministic, understanding why an agent made a particular decision is just as important as knowing that it completed its task.

Levan and the hosts discussed the need for "traceability" in agentic loops. This involves capturing the full chain of thought, the tools used, and the intermediate outputs generated by the agent. Without this level of granular detail, debugging an agent that has "gone off the rails" becomes nearly impossible.

Observability also plays a key role in cost management. Since agents can enter loops where they repeatedly call an LLM to refine a task, organizations need real-time visibility into token consumption and "recursive depth." The discussion underscored that robust observability practices are the only way to ensure that autonomous agents remain both effective and economically viable.

AdSpot Sponsor: Gartner

The complexities discussed by Levan, Ned, and Kyler are exactly the types of challenges that the upcoming Gartner IT Infrastructure Operations and Cloud Strategies Conference aims to solve. Scheduled to take place from December 8th through the 10th in Las Vegas, the conference is a premier gathering for heads of infrastructure, cloud operations professionals, platform engineers, and security leaders.

As AI agents move from the periphery to the core of enterprise strategy, infrastructure leaders are being tasked with building the foundations that support these workloads. The Gartner conference offers a unique opportunity for these professionals to learn from industry experts, share best practices, and explore the latest technologies in cloud operations and infrastructure management. For those looking to stay ahead of the curve in a rapidly changing landscape, this event provides the strategic insights necessary to navigate the future of IT.

Interested parties can learn more and register at gartner.com/io. By using the code DEVOPS, attendees can receive a $450 discount off the standard registration rates, making it an accessible opportunity for teams looking to bolster their operational expertise.

Looking Ahead: The Operationalization of AI

As the conversation between Ned, Kyler, and Michael Levan concluded, the takeaway was clear: the era of "AI as a toy" is over. For organizations to truly leverage the power of agentic workflows, they must treat AI as a first-class citizen in their infrastructure stack. This means moving beyond simple API calls and investing in the gateways, routing mechanisms, security sandboxes, and observability tools that define modern, resilient systems.

The insights shared by Michael Levan serve as a roadmap for platform engineers and DevOps professionals who are now finding themselves at the intersection of traditional infrastructure and cutting-edge artificial intelligence. As these technologies continue to mature, the focus will increasingly shift from the models themselves to the robust systems that keep them running safely and efficiently.

Episode Links:

For those who wish to dive deeper into the technical nuances of this discussion, the full episode provides a wealth of additional context and real-world examples.

Watch this episode on YouTube

To stay updated on Michael Levan’s latest insights into DevOps, cloud architecture, and the evolving world of AI operations, you can follow him on social media.

Michael Levan on X