Scaling the Frontier of Agentic Workflows: Michael Levan on the Operational Realities of AI Agents

As the enterprise landscape shifts from simple generative AI implementations toward more complex, autonomous systems, the conversation around infrastructure has taken a turn toward the "agentic." In a recent and deeply technical discussion, industry commentators Ned and Kyler sat down with cloud and DevOps expert Michael Levan to unpack the burgeoning complexities of running AI agents securely and at scale. The conversation, which touched upon everything from specialized AI gateways to the intricacies of sandboxing, highlighted a pivotal moment in the evolution of artificial intelligence: the transition from experimental prototypes to production-ready operational workloads.

The concept of an AI agent—a system capable of not just generating text, but taking actions, making decisions, and interacting with other software tools—represents a significant leap in capability. However, as Levan pointed out during the session, with increased capability comes a unique set of architectural and security challenges that traditional DevOps and platform engineering practices are only beginning to address.

The Shift Toward Agentic Workflows

For much of the past two years, the corporate world has focused on Large Language Models (LLMs) as advanced chatbots. However, the industry is rapidly moving toward "agentic workflows." Unlike a standard chatbot that provides a static response to a prompt, an agent is designed to achieve a goal by breaking it down into smaller tasks, selecting the appropriate tools, and iterating until the objective is met.

This shift introduces a new layer of non-deterministic behavior into the software stack. During the discussion, Ned, Kyler, and Levan explored how this unpredictability necessitates a fundamental rethink of the underlying infrastructure. When an agent is empowered to execute code or query a database, the traditional boundaries of security and performance monitoring become blurred. The group emphasized that "day two" operations—the ongoing management, scaling, and securing of these systems—are where many organizations currently find themselves struggling.

The Role of Specialized AI Gateways

One of the most critical components identified by Levan in the quest for scalable AI is the implementation of specialized AI gateways. In a traditional microservices architecture, an API gateway manages traffic, authentication, and rate limiting. In the world of agentic workloads, the AI gateway serves a much more nuanced purpose.

An AI gateway acts as a sophisticated proxy between the application and the various LLMs or agentic frameworks being utilized. It provides a centralized point of control for governance, allowing organizations to manage API keys, monitor token usage, and enforce security policies across multiple providers. As Levan noted, without such a gateway, companies risk a "Wild West" scenario where individual developers or agents might inadvertently rack up massive costs or leak sensitive data through unmonitored API calls.

Furthermore, these gateways are becoming essential for "prompt governance." By intercepting requests, a gateway can check for prompt injections or ensure that PII (Personally Identifiable Information) is scrubbed before it ever reaches an external model provider. This layer of abstraction not only enhances security but also provides the flexibility to swap underlying models without re-architecting the entire application.

Semantic Routing and Intelligent Traffic Management

The discussion moved deeper into the mechanics of how these agents interact with infrastructure, specifically focusing on the concept of semantic routing. In traditional networking, routing is based on fixed paths, headers, or IP addresses. Semantic routing, however, uses the intent or the "meaning" of a request to determine its destination.

Levan explained that in an agentic system, a request might be routed differently depending on its complexity or the specific domain expertise required. For example, a simple query might be routed to a smaller, more cost-effective model, while a complex mathematical problem might be sent to a high-reasoning model or a specialized tool. Semantic routing allows the infrastructure to be "intent-aware," optimizing for both performance and cost.