Skip to content

From AI pilot to production: what changes when you scale

A working pilot is not production readiness. See what owners, monitoring, support, governance and metrics must exist before an AI pilot scales.

6 min readBy AI Guru AcademyFacts last checked

By AI Guru Academy

In short

A pilot proves that an idea can work. Production proves that the organisation can run it, every day, with someone accountable when it fails. The AWS generative AI maturity model separates these into two levels, Experiment and Launch, and the difference is governance, monitoring and support rather than a better model.

Why is a successful pilot not the same as being ready for production?

A pilot answers "can this work?", while production must answer "who runs it, watches it and fixes it?". The AWS generative AI maturity model describes Experiment as validating potential through structured pilot projects and proofs of concept while building core capabilities. It describes Launch as systematically deploying production-ready solutions with governance, monitoring and support mechanisms.

Pilots are usually small, watched closely by their builders and forgiven when they stumble. Production has none of those protections. Real users arrive, nobody is watching every answer, and the people who built the pilot move on to other work.

What are the four levels of the generative AI maturity model?

The model has four levels: Envision, Experiment, Launch and Scale. Do not confuse them with the AWS Cloud Adoption Framework (CAF) phases, which are Envision, Align, Launch and Scale. They are different frameworks with different second steps.

What are the four levels of the generative AI maturity model?
LevelWhat the organisation is doingTypical activities named by AWS
1. EnvisionExploring concepts and use casesAwareness training, evaluating cultural readiness
2. ExperimentValidating value through pilotsProofs of concept, selecting models and tooling, measuring benefits realisation
3. LaunchRunning production-ready solutionsMonitoring and performance tracking, risk and governance management, operation and support structure
4. ScaleEnterprise-wide capabilityReusable components, standard patterns, formal operating model and RACI matrix

The overview table on the AWS page labels level 4 "Scaling" while the level name is Scale. AWS also states that no level is inherently better, and that an organisation can sit at different levels for different aspects or use cases. Treat the model as a conversation tool, not a score.

What must exist before a pilot goes live?

Six things should exist before launch: an owner, monitoring, support, rollback, governance and success metrics. The maturity model's Launch level names several of these directly, including monitoring and performance tracking, risk and governance management, and an operation and support structure.

Use this checklist as a gate:

  • Named owner. One accountable person for the business result, and one for the technical service.
  • Monitoring. Agreed measures for quality, errors and user feedback, reviewed on a schedule.
  • Support. A route for users to report problems and a team that answers them.
  • Rollback. A tested way to return to the previous version or to switch the feature off.
  • Governance. Written risk and responsible AI policies that the system has been checked against.
  • Success metrics. Business measures, not only technical ones, tracked after launch.

If any line is blank, the project is still a pilot, however many users it has.

How do you monitor an AI system once it is live?

Monitor outputs, inputs and the business result, and keep records you can inspect later. Amazon Bedrock model invocation logging can record request data, response data and metadata for supported calls. Supported destinations are Amazon S3 and Amazon CloudWatch Logs, and only destinations in the same account and Region are supported. Logging is off by default, so someone must decide to turn it on and own the data it creates.

Amazon Bedrock Guardrails can apply content filters and denied topics to model inputs and outputs. That is a control on behaviour, which is separate from measuring whether the system is still doing its job.

For machine learning models, drift means live data moving away from the data a model learned from. Amazon SageMaker Model Monitor can compare production data with a training baseline. AWS documents it as no longer open to new customers, so treat it as relevant mainly to existing users.

Who is responsible, and how do roles change at scale?

At Launch, responsibility becomes explicit, and at Scale it becomes a formal operating model. The maturity model names establishing operational and support teams at Launch, and formalising the operating model and a RACI matrix (responsible, accountable, consulted, informed) at Scale.

The AWS CAF groups its capabilities into six perspectives: Business, People, Governance, Platform, Security and Operations. They are a useful way to check that a launch plan has not missed a whole area. A technically strong team often covers Platform and Security well and forgets People and Governance.

Change management matters as much as the technology: training users, explaining what the tool will and will not do, and agreeing how staff escalate problems. An accurate system that nobody trusts or uses delivers no value.

Worked example (illustration only). A fictional insurer pilots an assistant that summarises claim notes. Staff like it. Before launch, the team finds that nobody owns the summaries, there is no switch to turn it off, and success was never defined beyond "people liked it". They name an owner in claims, add a one-click fallback to the old process, log outputs for review, and set a target for handling time. Only then do they call it a launch.

Key takeaways

  • A pilot shows value is possible. Launch requires owners, monitoring, support and governance.
  • The generative AI maturity model levels are Envision, Experiment, Launch and Scale. CAF phases are Envision, Align, Launch and Scale.
  • Rollback and a way to switch a feature off should be tested before users depend on it.
  • Logging and guardrails are controls you must choose to set up and own, not defaults.
  • Success metrics should be business measures, tracked continuously after launch.
  • Scale means reusable patterns and a formal operating model, not just more users.

FAQ

How long should an AI pilot run before moving to production?

There is no AWS-stated duration. Move when the Launch conditions are in place and the pilot has shown measurable business value against agreed metrics.

Is Launch always better than Experiment?

No. AWS states that no level is inherently superior or inferior, and the right level depends on the organisation's goals and readiness. A team can reasonably stay in Experiment on a use case that is not yet worth the cost of production controls.

What is the difference between the maturity model and the CAF?

The maturity model describes how far an organisation has progressed in using generative AI, with four levels. The CAF organises capabilities into six perspectives and uses its own phases. They answer different questions and can be used together.

Who should own an AI system in production?

One accountable business owner and one accountable technical owner, with support and operations roles agreed in advance. AWS describes formalising a RACI matrix as a Scale-level activity, but naming owners earlier avoids confusion.

Sources

Last checked 2026-11-16.

AI Guru Academy is an independent training provider and is not affiliated with or endorsed by Amazon Web Services. AWS service names are trademarks of Amazon.com, Inc. or its affiliates.

Next step: take the free AIB-C01 diagnostic at /diagnostic, then see the course page at /courses/aib-c01-exam-prep.

From AI pilot to production: what changes when you scale | AI Guru Academy