News

Gopi Duddi is Chief Technology Officer at Couchbase, where he leads engineering for the company’s operational data platform for Ai.

getty​Most enterprise AI planning sessions today center on foundation models, orchestration frameworks and GPUs. The data architecture underneath and whether it can handle agentic AI in production barely comes up. Teams get the model right, get the framework right and watch the whole thing buckle because the data layer was never designed for the access patterns AI agents generate.

Making the right choice between scale-up and scale-out data systems determines what AI can accomplish in production. Most organizations are making that choice by default rather than by design because they’re reusing whatever database infrastructure was already in place before AI entered the picture. That’s a mistake.

What AI Agents Actually Do To A Data LayerTraditional applications generate predictable query patterns. Users log in, retrieve a record, update a field and log out.

Agentic AI workloads shatter that predictability because each agent action triggers a cascade of data operations spanning context retrieval, memory writes, vector similarity searches and state synchronization, often across thousands of concurrent sessions. A single autonomous agent might generate 10 or 20 times the data-layer traffic of a traditional application interaction and, unlike a human, never pause to read a screen or think about what to click next.

I come from a family of doctors and have watched what a full clinic day looks like. They see patient after patient, each one demanding full attention, with no real gap to reset before the next one walks in. AI workloads impose something similar on a data layer. There’s no idle time between requests and no natural pause for the system to catch its breath. A database built for human-paced traffic gets a lunch break; a database serving AI agents doesn’t.

Data infrastructure that performed admirably under conventional workloads hits walls that no amount of vertical scaling can move when AI is introduced. Adding more memory or faster processors to a single node helps for a while, but vertical scaling stops helping well before the needs are met.

Most enterprises today stitch together separate vector stores, caching layers, document databases and session stores. The operational complexity of keeping all of those synchronized under bursty agent loads is where production deployments quietly fall apart.

Three Failure Modes That Scale-Up Architectures Can’t EscapeEngineering teams that try to run production AI agents on top of scale-up database architectures hit the same three failure modes every time:

1. Single-Node Contention Under Concurrent Agent QueriesA distributed, scale-out architecture spreads those concurrent operations across many nodes, so adding agents means adding capacity rather than going off a cliff.

2. Replication Lag Making Real-Time Retrieval StaleAgents acting on current data for fraud detection, inventory management or customer interactions produce unreliable results when the data they read is even slightly behind. Systems built for distributed replication from the start treat cross-node consistency as core engineering, not a bolted-on feature.

3. Hidden Cost Of Sharding Strategies These strategies were never designed for mixed workloads. A multi-model distributed architecture that natively supports documents, key-value lookups, full-text search and vector operations within the same system avoids this trade-off because the data doesn’t need to be split across specialized stores in the first place. The cost of getting this wrong isn’t immediately visible; it shows up 18 months later as an unmaintainable web of sync jobs between five specialized stores.

Why The Architecture Decision Comes Before The Model DecisionMost enterprises have a sequencing problem when planning their AI infrastructure. The typical approach treats the data layer as the last thing to figure out, by which point the team has already committed to patterns that will break under AI load.

Models and orchestration frameworks are commoditizing rapidly. Data architecture, by contrast, is the one decision that’s genuinely difficult to change after the fact because it touches everything—from how agents retrieve context to how sessions persist across restarts to how the system behaves when a node fails at 2 a.m.

A model can be swapped out in a quarter. A data architecture takes years to rearchitect and a genuinely frightening amount of resources to fix if you get it wrong.

CIOs and senior IT leaders should start AI readiness assessments with a hard look at the data layer, not just end them with one. Can the current infrastructure scale horizontally without downtime when agent workloads spike? Can it serve vector searches and transactional queries from the same system without forcing data duplication? Most critically, can the team afford to discover the answer to these questions after the architecture is already load-bearing?

If the answer to any of these is no, the ceiling on AI ambitions is already set, regardless of how sophisticated the models are.

A Fair Word About Trade-OffsDistributed architectures aren’t free of their own operational costs. The CAP theorem guarantees trade-offs between consistency, availability and partition tolerance. Running a distributed cluster well requires genuine expertise in partitioning, rebalancing and failure recovery.

The claim that capacity scales linearly with nodes only holds when sharding and data placement were designed correctly from the start, which is exactly the skill set that most teams building their first production agent systems are still developing. None of this changes the core argument. Agentic workloads, with their concurrency and mixed access patterns, hit the limits of scale-up architectures faster than almost any other production use case. However, choosing scale-out is the beginning of an engineering discipline, not a substitute for one.

The enterprises that move from AI pilot to AI production fastest and create sustainable competitive differentiation in the AI era will be the ones that treated their data architecture as a deliberate, strategic investment rather than an afterthought assembled from point solutions on the shelf. None of this means every workload needs scale-out from day one, but agentic workloads, by their concurrency and access-pattern profile, are the case where the trade-off tips fastest.​

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?