Chapter 3

Chapter 3

From Experimentation to Engineering Standards

Creating Consistency Without Freezing Innovation

This question matters because software engineering depends on consistency far more than individual experimentation does. A single developer can work successfully with a highly personalized AI workflow. An engineering organization maintaining hundreds of applications over many years cannot allow every important engineering decision to become a personal preference. Code changes hands. Engineers move between teams. Systems integrate. Security expectations cross application boundaries. Architecture depends on shared assumptions. Maintenance becomes easier when developers can enter an unfamiliar repository and recognize how the organization expects important work to be performed.

AI does not eliminate the need for that consistency. In some ways, it makes consistency more important because AI dramatically increases the amount of work that can be produced and repeated.

That requires us to rethink what an engineering standard is actually for. A standard should not exist merely because leadership prefers uniformity. A standard should protect an engineering outcome that matters.

Standards Should Protect Outcomes, Not Preserve Habits

AI makes this problem more dangerous because the pace of improvement is much faster. A useful AI practice can become outdated quickly, and a model or workflow that performs poorly today may become substantially better after a new release. If an organization defines its AI engineering standards primarily around particular products, prompts, or procedures, it risks creating rules that become obsolete almost as quickly as they are published.

A better approach is to begin with the protected outcome introduced in Chapter 2. Before making an AI practice a standard, ask what engineering result the organization is trying to preserve. The standard may define an approved method for achieving that result, but the reason for the standard should remain visible. This makes it possible for engineers to challenge the method without challenging the need for the outcome.

Consider code review. An organization might establish a rule that AI-generated production code must receive human review before it is merged. The important question is why. If the real protected outcome is that consequential software changes must be independently validated before entering the production codebase, then human review is the current mechanism for protecting that outcome. As AI validation capabilities improve, the organization can evaluate whether another mechanism eventually provides equivalent or stronger protection. The protected outcome remains stable even when the method changes.

The same principle can apply to architecture.

The standard tells engineers what currently works and what is expected. The protected outcome explains why it matters. The evidence record explains why the current method was selected. The challenge process gives engineers a legitimate way to demonstrate that something better has become available. Together, those elements prevent standards from becoming either arbitrary restrictions or permanent technical artifacts.

Not Everything That Works Should Become a Standard

Some AI practices may remain personal productivity choices indefinitely. If one developer prefers to use an approved AI assistant to help draft internal documentation while another prefers to write it manually, there may be little organizational benefit in requiring either approach because the outcome matters more than the method used to achieve it. Other practices may benefit from guidance without requiring enforcement. Perhaps the organization has discovered a particularly effective way to provide architectural context to AI when working within a certain application family. Sharing that technique can save other developers time and help spread what the organization has learned, but there may still be no reason to require every engineer to use it in exactly the same way.

Still other practices should become expected because inconsistency creates downstream problems or unacceptable risk. If AI-generated code repeatedly introduces architectural deviations unless a particular context package is supplied, requiring that context may be justified. If certain categories of AI-generated changes require independent review because of their potential consequence, that review requirement may need to become a standard. Likewise, if an agent must never modify a production environment without explicit approval, the organization should not leave that decision to individual preference. The challenge is distinguishing among these situations so that formal standards are applied where consistency protects an important engineering outcome, while flexibility is preserved where individual choice and continued experimentation still provide value.

• Experiment: We are still learning whether the practice works and where it belongs.
• Recommended: Evidence indicates that the practice is useful, and engineers are encouraged to use it where appropriate.
• Required: Evidence and organizational consequence justify making the practice an engineering expectation.
• Automated or enforced: The requirement is sufficiently important and technically definable that the engineering system should enforce it rather than relying solely on people to remember it.

This progression is not automatic. A successful experiment does not have to become recommended, and a recommended practice does not have to become required. Some practices should remain recommendations because engineers need flexibility. Others may move quickly toward enforcement because the protected outcome is too important to depend on individual judgment. The organization should use the least restrictive level that reliably protects the outcome. That principle helps preserve experimentation without sacrificing engineering discipline.

Guidance and Standards Are Different

AI will create an enormous number of potential practices, including recommendations for prompting, context construction, model selection, code generation, test generation, review, documentation, agents, tool access, and dozens of other activities. Turning all of those practices into requirements would quickly create a system too cumbersome to maintain, which is why guidance provides an important place for useful organizational knowledge that has not earned—or simply does not need—the force of a standard. For example, several teams might discover that providing an AI assistant with a short architectural summary before asking it to modify a particular application produces substantially better results. The organization can publish guidance showing developers how to provide that context and explaining the improvement that has been observed, allowing other engineers working in the application to benefit immediately without turning the technique into a mandatory process.

If later evidence shows that AI-generated changes made without this architectural context repeatedly violate important system boundaries, however, the practice has become more than helpful guidance. Providing approved architectural context may now need to become a required part of AI-assisted changes for that application because the organization has evidence that the practice protects an important engineering outcome. The method has moved from being useful to being necessary for protecting the result, and that is the point at which standardization becomes appropriate.

A Standard Needs a Scope

AI makes scope particularly important because performance can vary dramatically based on context. The same model may perform exceptionally well in a modern application with strong automated testing and clear architectural patterns while performing poorly in a legacy system with undocumented business rules and limited test coverage. A workflow proven in the first environment should not automatically become the standard for the second. Every AI engineering standard should therefore define where it applies.

Scope may include:

• Application types.
• Repositories or technology stacks.
• Categories of engineering work.
• Risk or consequence levels.
• Required test coverage or validation capability.
• Data classifications.
• Development environments.
• Agent authority levels.
• Teams or business areas.
• Conditions that must exist before the standard applies.

This allows the organization to say something much more useful than, “Developers should use AI to generate unit tests.” It can instead say that for a defined class of routine service-layer changes in applications meeting specified testing conditions, AI-assisted test generation is recommended because evaluation showed a meaningful reduction in engineering effort with acceptable review and quality results. That statement can evolve as evidence expands.

If the practice later proves successful in additional environments, the scope can grow. If failures reveal a limitation, the scope can narrow. The standard becomes a living engineering decision rather than a universal declaration about what AI can do.

A Standard Also Needs an Owner

For every meaningful AI engineering standard, identify:

• Owner: Who is responsible for maintaining the standard?
• Protected outcome: What engineering result is the standard intended to preserve?
• Evidence basis: Why was the current approach selected?
• Scope: Where does the standard apply?
• Review trigger: What would cause the organization to reconsider it?
• Last review: When was the standard last examined?
• Related tooling: Is any part of the standard automated or enforced?
• Challenge path: How can an engineer propose a better alternative?

Review triggers are particularly useful because they are often better than arbitrary calendar reviews. An organization can certainly review standards periodically, but significant technological change should also trigger reconsideration. A major model improvement, new development platform capability, repeated exception requests, evidence of unnecessary cost, or a successful alternative experiment may all justify reviewing a standard before its scheduled date.

This keeps the standards system responsive to evidence rather than time alone.

Do Not Standardize a Vendor When You Mean to Standardize an Outcome

Separating these concepts makes future change much easier because the organization does not have to redefine its engineering expectations every time the technology changes. When a better model appears, the organization can test it against the protected outcome rather than reopening the entire AI engineering strategy. If the new model satisfies the established requirements more effectively, the implementation can change while the engineering expectation remains intact. This approach also prevents vendor capabilities from defining organizational policy by accident. A product may suddenly introduce an agent feature capable of performing additional actions, but that does not mean the organization should automatically expand its standards or permissions to allow those actions simply because the capability now exists. Product capability can change much faster than organizational readiness, which is why the engineering operating model should determine what authority is appropriate, while the tool should be selected and configured to implement that decision.

Standards Should Become Machine-Readable Where Practical

Some expectations can be implemented through conventional tooling. Formatting, dependency restrictions, security rules, test requirements, static analysis, repository policies, and deployment controls often do not require AI interpretation at all. If a rule can be enforced reliably with deterministic software, that may be preferable to repeatedly explaining the rule to a model and hoping it follows it. Other expectations require context because architectural principles, application-specific constraints, domain terminology, approved patterns, or engineering decision records cannot always be reduced to a simple automated rule. In those cases, the relevant knowledge may need to be made available to the AI at the appropriate point in the workflow. Later in the book, particularly in Chapter 6, we will examine how this engineering knowledge can be organized so AI can actually use it.

For now, the important principle is that a standard that matters to AI-assisted work should eventually exist somewhere the AI-assisted workflow can encounter it. This does not mean converting every policy document into a prompt or attempting to feed the entire engineering handbook into every AI interaction. It means examining each important standard and asking how the engineering system actually protects it, whether through deterministic controls, accessible engineering context, human review, or some appropriate combination of the three.

• Static analysis.
• Repository rules.
• Automated tests.
• CI/CD validation.
• Policy-as-code.
• Approved context supplied to AI.
• Human review.
• Explicit approval.
• A combination of these controls.

The stronger the consequence of violating the standard, the less desirable it is to depend solely on the AI remembering an instruction. This gives us another useful principle: Do not ask AI to remember a rule that the engineering system can reliably enforce. AI is most valuable where reasoning is required. Deterministic controls remain extremely valuable where the outcome can be defined precisely.

The Standard Must Be Challengeable

That distinction is essential because the engineer is not simply asking for permission to ignore the standard. The engineer is proposing an alternative and accepting responsibility for producing evidence that the alternative is better. A useful challenge should identify the current standard, the proposed alternative, what the alternative is expected to improve, and the protected outcome that must still be satisfied. The proposed approach can then be evaluated using representative engineering work and the same general evidence discipline developed in Chapter 2. If the alternative performs better while continuing to protect the required outcome, the standard should be capable of changing; if it does not, the existing standard remains. Either result is useful because the decision is based on evidence rather than preference, seniority, or enthusiasm for a new technology.

This is how an organization avoids the false choice between standards and innovation, because the two can reinforce rather than oppose each other. A good standard captures the best proven practice the organization currently knows, while a good challenge process provides a structured way to replace or improve that standard when someone can demonstrate that the organization has learned something better.

The Standard Improvement / Challenge Process

The process should begin with the protected outcome, not the engineer’s preferred technology. Suppose the current standard requires a particular approach to AI-assisted code review. An engineer believes a newer model and workflow can produce better results with less human effort. The challenge is not simply, “I want to use this new model.” The challenge is, “I believe this alternative can reduce review effort while preserving or improving the outcomes this standard exists to protect.” That framing forces both the challenger and the organization to focus on engineering value rather than novelty.
The engineer should then be allowed to test the alternative in a controlled environment using representative work. The evidence requirements should be proportional to the consequence of the standard. Challenging a recommended prompting technique should require far less evidence than challenging a security control or an approval requirement for an agent with production access. The organization is not trying to create one universal approval process. It is trying to create a repeatable way to compare an alternative against the reason the existing standard exists.

When the evaluation is complete, there are four useful outcomes. The organization may keep the current standard because the alternative did not demonstrate enough improvement. It may update guidance without changing the required standard because the alternative is useful in some situations. It may revise or replace the standard because the evidence shows a better approach. Or it may return the alternative to experimentation because the results are promising but not mature enough to support a broader change.

Those decisions should be explicit:

• Keep current standard — Existing evidence still supports the current approach.
• Update guidance only — The alternative is useful under some conditions but does not justify changing the requirement.
• Replace or revise the standard — Evidence supports changing the current engineering expectation.
• Return the alternative to experimentation — More evidence or refinement is required before a decision can be made.

The decision should include a short explanation of the evidence behind it. This prevents the challenge process from becoming a popularity contest and gives future engineers something useful to examine if the question arises again.

There is one final field that belongs in the challenge record: communication and workflow updates required. This phrase can sound more complicated than it is. It simply asks: If we change this standard, what else needs to change so people and systems begin following the new approach? If a standard changes, the organization may need to update developer guidance, repository instructions, CI/CD checks, agent context, onboarding material, templates, internal documentation, or automated controls. If nothing else needs to change, the answer can simply be “none.” The purpose is to prevent a decision from being made on paper while the engineering environment continues enforcing or teaching the old approach.

For example, suppose the organization changes a standard requiring human review of a particular low-risk AI-generated change because evidence shows that automated validation now protects the same outcome reliably. Changing the standards document is not enough. The pull-request workflow may still require the old approval. Team documentation may still instruct developers to request it. An agent may still be configured to stop and wait for approval. The communication and workflow updates required field identifies those dependent changes so the new decision actually becomes operational.

This gives the organization a complete improvement loop:

Standard → Challenge → Evidence → Decision → Implementation → New Standard

That loop is important because the objective is not to create standards that never change. The objective is to create standards that change for good reasons.

Exceptions Are Evidence Too

Useful questions include:

• Are the same exceptions appearing repeatedly?
• Are they concentrated in a particular application type or team?
• Does the pattern reveal that the standard’s scope is too broad?
• Has technology changed since the standard was established?
• Are engineers consistently finding a less expensive or more effective alternative?
• Is a supposedly temporary exception quietly becoming normal practice?
• Does the exception expose a readiness problem elsewhere in the engineering system?

This turns exception management into part of the AI Engineering Learning Loop established in Chapter 1.

Standardize the Successful Conditions, Not Just the Successful Technique

Imagine that one team demonstrates excellent results using an agent to perform routine dependency updates. The agent examines the dependency, modifies the appropriate files, executes tests, resolves straightforward issues, and prepares a pull request. Human review is minimal, and the process saves meaningful engineering time. Leadership might reasonably want to expand the practice.

But suppose the successful team also has excellent automated tests, standardized build processes, well-defined repository structure, clear dependency policies, and strong CI validation. Another team has limited test coverage, several manual deployment steps, and an application containing undocumented integration behavior. Deploying the same agent to both teams does not create the same engineering system. The agent is only one part of the successful result.

When a practice moves toward standardization, record the conditions required for success alongside the practice itself.

• Minimum automated validation.
• Required architectural or business context.
• Repository structure or conventions.
• Defined human review.
• Appropriate permissions.
• Observable rollback or recovery.
• Clear task boundaries.
• Approved tools and integrations.
• Escalation conditions.
• Required engineering expertise.

This becomes increasingly important as we move toward agents because an agent should not inherit authority simply because another agent performing similar work was successful elsewhere; the surrounding engineering environment determines how safely and reliably that authority can be exercised. Standardization therefore requires a more precise conclusion than simply saying “this worked.” We need to understand what worked, where it worked, why it worked, and what conditions must remain true for us to reasonably expect it to work again.

Standards Should Reduce Repeated Human Decisions

This becomes particularly valuable with AI because AI can increase the frequency at which these decisions occur. A human developer might encounter a particular architectural choice occasionally, while an agent performing many tasks may encounter the same decision repeatedly. If that decision can be captured as a standard, context rule, automated check, or deterministic control, the organization stops paying for the same reasoning every time. This gives us another way to identify standardization opportunities: during the Human Correction analysis from Chapter 2, look for corrections that occur repeatedly across developers and AI interactions. Repetition is a signal that the engineering system may be missing a reusable decision, which leads to an important question: Are our engineers repeatedly correcting AI for something the organization already knows?

If the answer is yes, the solution may not be better prompting. The better solution may be to capture that knowledge once and make it part of the engineering system so it can be applied consistently wherever it is needed. Some of that knowledge belongs in standards, some belongs in architecture documentation or business context, some should become automated validation, and some may be better implemented as conventional software rather than AI instruction. Determining where that knowledge belongs is an important part of the transition, and we will examine it more deeply in Chapter 6.

Keep the Standards Portfolio Small Enough to Matter

An individual practice should not move upward in this hierarchy simply because it has existed for a long time; it should move because evidence and consequence justify greater consistency. Practices should also be capable of moving downward as conditions change. If a required standard becomes unnecessary because automated tooling now protects the outcome, the human procedure may be removed. If supporting evidence weakens, a recommended practice may return to experimentation, and if a better technique replaces an older one, obsolete guidance should be retired rather than left beside the new guidance indefinitely. A healthy standards portfolio, therefore, is not one that continually grows, but one that continually improves as the organization learns what deserves to be required, what should remain recommended, and what is still being explored.

The Chapter 3 Working Checklist

Do not review every AI experiment for standardization. Begin with practices where evidence or organizational consequence justifies the effort. For each candidate, confirm:

• The engineering activity has been clearly defined.
• Representative work has been evaluated.
• The expected benefit has been demonstrated sufficiently for the decision being considered.
• Protected outcomes have remained acceptable.
• Known limitations have been documented.
• Conditions required for success are understood.
• There is an organizational reason for greater consistency.

If the final item is missing, the practice may be valuable without needing to become a standard.

2. Identify the Protected Outcome

State clearly why the organization is considering a standard. Record:

• What engineering outcome must remain protected?
• What failure or inconsistency is the standard intended to prevent?
• Why does consistency matter in this case?
• What evidence shows that the proposed practice protects the outcome?

Avoid defining the protected outcome as compliance with the method itself. “Developers must use Tool X” is not a protected outcome. Explain what Tool X currently helps the organization accomplish.


3. Determine the Appropriate Level

Choose the least restrictive level that reliably protects the outcome.

• Experimental — Continue learning; no broader recommendation yet.
• Recommended — Evidence supports the practice, but engineering discretion remains appropriate.
• Required — Consistency is necessary to protect an important engineering outcome.
• Automated/enforced — The requirement is sufficiently important and technically definable that the engineering system should enforce it where practical.

Record why the selected level is appropriate.

4. Define the Scope

Specify where the guidance or standard applies. Include relevant boundaries such as:

• Engineering activity.
• Application or repository types.
• Technology stack.
• Risk or consequence level.
• Required testing or validation capability.
• Environment.
• Data classification.
• Agent authority.
• Team or organizational area.
• Conditions that must exist before the standard applies.

Also record where the standard does not apply if that distinction could prevent misuse.

5. Record the Evidence Basis

Link the standard to the reasoning that created it. Capture:

• AI Engineering Value Record.
• Representative work evaluated.
• Material productivity findings.
• Quality findings.
• Human correction findings.
• Known limitations.
• Conditions required for success.
• Evidence that remains incomplete.

This allows future engineers to understand why the standard exists rather than inheriting a rule without its reasoning.

6. Determine How the Standard Will Be Protected

Decide how the engineering system will make the standard effective. Possible mechanisms include:

• Developer guidance.
• AI context or instructions.
• Code templates.
• Static analysis.
• Automated tests.
• Repository controls.
• CI/CD validation.
• Policy-as-code.
• Human review.
• Required approval.
• Access or permission controls.

Prefer deterministic enforcement when a requirement can be defined reliably. Use AI reasoning where judgment is genuinely required.

7. Assign Ownership

For every required standard and important recommendation, record:

• Standard owner.
• Date established.
• Last review date.
• Evidence or technology changes that should trigger reconsideration.
• Related systems, documentation, or tooling.
• Where exceptions are recorded.
• How engineers submit a challenge.

Ownership should make the standard maintainable, not create a gatekeeper who prevents change.

8. Establish the Standard Improvement / Challenge Log

When an engineer believes a current recommended or required practice can be improved, capture:

Current standard or practice

• What is currently expected?
• Where does it apply?
• What protected outcome does it preserve?

Proposed alternative

• What is the alternative?
• What is expected to improve?
• Why does the engineer believe it may be better?

Evidence plan

• Test the alternative using representative engineering work.
• Compare it with the current practice using the same protected outcome.
• Record lifecycle effort, quality, correction, risk, and downstream consequences that could materially affect the decision.

Decision

• Keep current standard.
• Update guidance only.
• Replace or revise standard.
• Return alternative to experimentation.

Reason / evidence for decision

Record the evidence that led to the decision rather than simply the decision itself.

Communication / workflow updates required
If the standard changes, identify what must also change so the new decision becomes operational. This may include:

• Developer documentation.
• AI instructions or context.
• Repository guidance.
• CI/CD checks.
• Automated policies.
• Templates.
• Agent configuration.
• Training or onboarding material.
• Review or approval workflows.
• No additional changes required.

9. Review Exceptions for Patterns

Do not allow exceptions to disappear after approval. Periodically ask:

• Are similar exceptions recurring?
• Is the standard’s scope incorrect?
• Has technology changed?
• Is a better practice emerging?
• Does the exception reveal an engineering-readiness problem?
• Should the exception become a formal challenge?
• Has a temporary exception quietly become normal practice?

Repeated exceptions are evidence that deserves examination.

What You Should Have When Chapter 3 Is Complete

You should now have:

• A clear distinction between experimental, recommended, required, and enforced practices.
• Protected outcomes associated with important AI engineering standards.
• Defined scope and conditions for each standard.
• Evidence explaining why the current practice was selected.
• Named ownership for maintaining important standards.
• A method for enforcing standards through the engineering system where practical.
• A Standard Improvement / Challenge Log.
  A process for learning from recurring exceptions.

• A mechanism for replacing existing standards when evidence supports something better.

The most important result is cultural as much as procedural. Engineers should understand that a standard is not an announcement that experimentation is over; it represents the organization’s current best evidence about how an important engineering outcome should be protected. They should also know that if they discover something better, there is a legitimate path for demonstrating its value and improving the standard. That gives the organization consistency without making consistency permanent, allowing proven practices to become part of the engineering system while preserving the ability to challenge and improve them as the technology and the organization’s experience evolve.

The first three chapters have now established an important foundation. We know what engineers are doing with AI, we have a method for determining whether those practices create real engineering value, and we have a way to turn proven practices into shared engineering expectations while continuing to improve them. The next question changes the nature of the transition because, so far, a human engineer has remained clearly responsible for the work. AI may explain, recommend, generate, analyze, or assist, but the human remains at the center of the workflow and ultimately decides what happens next.

Increasingly, that will not always be the case. An AI system may be able to receive an objective, determine a sequence of steps, call tools, inspect files, modify software, execute tests, react to the results, and continue working without waiting for a developer to direct each individual action. At that point, we are no longer deciding only how developers should use AI; we are deciding what AI should be allowed to do on its own. That is the subject of Chapter 4.

• Introduction → AI engineering framework
• Chapter 1 → AI in software development
• Chapter 2 → AI developer productivity
• Chapter 3 → AI engineering standards
• Chapter 4 → AI agents in software development
• Chapter 5 → AI technical debt
• Chapter 6 → AI engineering context
• Chapter 7 → AI cost governance
• Chapter 8 → AI engineering operating model

If you have questions, please let us know.

Take a few minutes to complete the assessment and gain a clear, practical view of your organization’s AI readiness—and what to do next.