How to Structure an AI Center of Excellence

An AI center of excellence structure falls into three basic patterns: centralized (one team owns AI capability and delivery), embedded (AI talent sits inside business units), or hybrid (a central team sets standards while embedded practitioners execute locally). Most organizations beyond 500 employees converge on a hybrid model within two years, because pure centralization creates a delivery bottleneck and pure embedding fragments standards and governance. The right starting point depends on AI maturity, the number of business units competing for AI resources, and how tightly regulated the organization's data environment is.

This article breaks down what each model actually looks like in practice, when to use it, and how to migrate as the organization matures.

What Is an AI Center of Excellence?

An AI center of excellence (AI CoE) is the organizational unit responsible for setting AI strategy, standards, and shared infrastructure, and for building or supervising the capability that turns AI initiatives into production systems. It is not a research lab and not a single data science team bolted onto IT. Its mandate spans four functions: governance (policy, risk, model approval), enablement (tooling, platforms, reusable components), delivery (building or co-building AI use cases), and capability building (training, talent development, change management).

Where a CoE sits organizationally, who staffs it, and how much authority it holds over business-unit AI spending are structural decisions, not defaults. Get the structure wrong and the CoE becomes either an ivory tower that ships nothing or a shadow IT problem with no coherent standards.

The Three Core Models

Centralized: One Team Owns AI

In a centralized structure, a single AI CoE reporting into the CIO, CTO, or a Chief AI Officer owns strategy, tooling, governance, and most delivery. Business units submit use case requests; the CoE prioritizes, builds, and hands back a working system.

This model suits organizations early in their AI maturity, where AI talent is scarce and concentrating it prevents duplicated effort and inconsistent risk practices. A single insurer building its first claims-automation model benefits from one team that has done the work once and can reuse the pattern for underwriting, next.

The failure mode is predictable: as demand grows, the central team becomes a queue. Business units wait months for a CoE that is already overcommitted, then start building AI capability on their own outside the CoE's visibility, which defeats the governance purpose the structure was meant to serve.

Embedded: AI Talent Lives in the Business

In an embedded structure, AI practitioners, data scientists, ML engineers, prompt engineers, sit inside individual business units or functions, reporting to that unit's leadership. A small central group, if it exists at all, provides light-touch standards and shared tooling but has no authority over headcount or priorities.

This model suits organizations where business units have genuinely different data environments and use cases, and where domain context matters more than shared infrastructure. A retailer's merchandising AI and its fraud-detection AI have almost nothing in common technically; embedding both close to the business they serve can produce faster, more relevant outcomes than routing both through one central queue.

The tradeoff is fragmentation. Without a coordinating layer, business units re-solve the same problems (model evaluation, vendor selection, prompt governance) independently, inconsistently, and often insecurely. Model risk management becomes close to impossible to audit centrally, which is a material problem the moment regulators or the board ask how AI risk is being managed enterprise-wide.

Hybrid: Central Standards, Distributed Delivery

The hybrid model, sometimes called "hub and spoke," keeps a central CoE responsible for strategy, governance, platform, and shared services, while embedding AI practitioners ("spokes") inside business units to deliver against local priorities. The hub sets the guardrails; the spokes build inside them.

This is the model most large organizations converge toward, because it solves the two failure modes above simultaneously: the hub prevents fragmented governance, the spokes prevent bottlenecked delivery. It costs more to run than either pure model, since it requires both central and distributed headcount, and it requires clear escalation paths so hub and spoke don't simply relitigate the same prioritization fights the centralized model had.

Which Model Fits Your Organization?

Four factors should drive the choice, not preference for one model as a philosophy.

• AI maturity. Organizations with fewer than five production AI systems rarely have enough proven pattern to distribute; start centralized, plan to hybridize.
• Number of business units with genuinely distinct data and use cases. Fewer than three, centralize. More than five, hybrid or embedded becomes necessary to avoid the hub drowning in unrelated domain complexity.
• Regulatory exposure. Financial services, healthcare, and public sector organizations need a strong central governance function regardless of delivery model; embedded-only is rarely defensible under audit.
• Existing talent distribution. If ML and AI talent already sits inside business units and is productive there, a forced centralization will trigger attrition before it delivers coordination benefits.

A useful diagnostic: if the organization cannot currently name every production AI system, who owns its risk sign-off, and what data it touches, governance is the immediate gap, not delivery speed. That argues for strengthening the hub before expanding the spokes.

How Should the CoE Report Into the Organization?

Reporting line matters as much as internal structure. A CoE that reports into IT tends to optimize for technical delivery and under-invests in change management and business adoption. A CoE that reports into a single business function tends to over-serve that function and lose credibility as an enterprise resource.

The structures that hold up best report the CoE into a role with enterprise-wide mandate and board visibility, typically a Chief AI Officer or equivalent, with a dotted line into both technology and the businesses it serves. This is less about title and more about ensuring the CoE has standing to say no to a business unit's request when it conflicts with risk policy, and standing to demand engineering discipline from a business unit's embedded team.

How Does the Structure Change as AI Maturity Grows?

An AI center of excellence structure is not a one-time decision. Organizations typically pass through three phases, and the structure that fits phase one actively slows down phase three.

Phase one: prove value. The organization has few or no production AI systems. A small, centralized team, often five to fifteen people, builds the first two or three use cases end to end. The goal is proof, not scale: pick high-value, contained problems, ship them, and use the results to build the internal case for investment.

Phase two: build the platform. Once the CoE has shipped several systems, the pattern of repeated work becomes visible: data access requests, model evaluation, prompt or fine-tuning pipelines, deployment infrastructure. This phase is about extracting that repeated work into shared platform and tooling, so the third phase can scale delivery without scaling headcount at the same rate.

Phase three: distribute delivery. With platform and governance in place, the organization embeds practitioners into business units who consume the shared platform rather than rebuilding it. The central CoE shifts from doing the delivery work itself to setting standards, running the approval gates, and supporting the embedded teams. This is the hybrid model, arrived at deliberately rather than by drift.

Skipping phase two is the most common structural error. Organizations under pressure to show enterprise-wide AI adoption jump straight from a small central team to fully embedded delivery, without ever building the shared platform. The result is exactly the fragmentation the embedded model risks: five business units, five different approaches to model evaluation, and no consistent answer to what data any given model was trained or fine-tuned on.

Who Should Staff the CoE?

Structure determines where people sit; staffing determines whether the structure actually functions. A CoE, at any phase, needs four capability types represented, even if one person covers more than one:

• Technical delivery. ML engineers, data engineers, and increasingly LLM/agent engineers who can actually build and ship a working system, not just prototype one.
• Governance and risk. Someone accountable for model risk management, data governance, and regulatory mapping, distinct from the technical delivery lead, so technical debt and risk decisions aren't made by the same person under delivery pressure.
• Business translation. A role, often the most underinvested in, that translates business problems into AI-solvable use cases and translates delivered systems back into business outcomes. Without this, the CoE ships technically correct systems nobody in the business asked for or adopts.
• Change management. AI systems that change how people work fail on adoption more often than they fail on technical performance. A CoE with no change management capacity underdelivers on every system it ships, regardless of model quality.

A common mistake is staffing the CoE entirely with technical delivery roles and treating governance and change management as someone else's job, usually compliance or HR, brought in only when something goes wrong. Both need a seat inside the CoE from day one, not as an external check applied after the fact.

Common Structural Mistakes

• Building the CoE as a delivery team with no governance mandate. It ships fast at first, then accumulates ungoverned risk that surfaces at the worst possible time, typically during a regulatory review or after a model failure.
• Giving the CoE governance authority with no delivery credibility. Business units route around a CoE that only says no and never demonstrates it can build. Policy without a track record of shipped, working systems gets ignored.
• Skipping the migration plan. Organizations pick centralized or embedded and never revisit the choice as AI maturity grows, leaving the structure years out of date with the actual scale of AI activity.
• Underestimating the platform layer. A hybrid model without genuinely shared MLOps, data access, and evaluation tooling is not hybrid, it is two disconnected embedded teams that happen to share a reporting chart.
• No single accountable owner. A CoE with a steering committee but no named executive accountable for outcomes tends to produce consensus-driven mediocrity rather than a functioning operating model.

Key Takeaways

• Centralized structures suit early AI maturity and concentrate scarce talent, but become delivery bottlenecks as demand scales.
• Embedded structures move fast on domain-specific use cases but fragment governance and make enterprise risk oversight difficult to sustain.
• Hybrid, hub-and-spoke structures are where most organizations land: a central function owns strategy, governance, and platform; embedded teams own delivery inside business units.
• The right choice depends on AI maturity, the number of distinct business-unit use cases, regulatory exposure, and where AI talent already sits.
• Reporting line matters as much as internal structure: the CoE needs enterprise-wide standing to both enforce governance and earn delivery credibility.

Structuring and leading this operating model, across strategy, governance, and organizational design, is the core of AICA's Certified Chief AI Officer (CCAIO) executive certification.