Chapter 2

Chapter 2

Is AI Actually Saving Engineering Time?

Measuring the Work That Disappears—and the Work That Simply Moves

Those examples matter. They are one reason the AI Engineering Usage Baseline developed in Chapter 1 asks developers where they perceive meaningful benefit. Engineers closest to the work are often the first people to recognize where AI is creating leverage, and an organization would be foolish to disregard those observations simply because they have not yet been converted into formal measurements. The problem begins when the organization assumes that time saved during generation is equivalent to engineering time saved throughout the lifecycle.

It may be. Sometimes AI really does remove work. A developer may use an assistant to understand an unfamiliar API in ten minutes instead of spending an hour moving between documentation, source code, and search results. AI may generate routine tests that require only minor review. A repetitive conversion task may be completed dramatically faster with no measurable increase in downstream effort. When this happens consistently, the organization has found real leverage.

In other cases, however, the work does not disappear. It moves.

If ten minutes of generation replaces an hour of manual coding but creates forty minutes of additional review and twenty minutes of correction, the engineering result is very different from the initial productivity claim. If the generated work also increases maintenance effort during the next several changes, the difference becomes larger still. Conversely, if ten minutes of AI-assisted work replaces an hour of manual work and the resulting implementation moves through review, testing, deployment, and future maintenance with no additional burden, the organization has discovered a highly valuable practice. The purpose of measurement is to tell those situations apart.

Measure Engineering Outcomes, Not AI Activity

A developer who submits twice as many prompts is not necessarily twice as productive. A team that generates more code is not necessarily delivering more useful software. An agent that performs one hundred actions is not necessarily more valuable than one that performs ten. In fact, as we will see when we examine AI economics, additional activity can sometimes indicate inefficiency rather than success. The measurement needs to remain connected to the engineering outcome.

If the organization introduced AI to reduce the effort required to create unit tests, measure the lifecycle effort associated with creating and maintaining those tests. If AI is being used to understand legacy code, measure whether developers reach reliable understanding faster and whether that understanding produces correct changes. If an agent is being considered for routine dependency updates, measure the amount of human engineering effort required to move a successful update through validation and completion.

The question is not, “How much AI did we use?” The question is, “What engineering work became better because we used it?”

Establish the Baseline Before the Improvement

Suppose a team believes AI substantially improves the creation of unit tests for existing services. Before declaring the practice successful, understand the current workflow. How much engineering effort does test creation typically require? How much review is necessary? How often are tests rewritten because they do not represent the intended behavior? How useful are those tests when the code changes later? How much maintenance do they require?

The baseline does not have to become a scientific research project. Software engineering contains too much variation for many tasks to be measured with laboratory precision. A complex feature is not directly comparable to a simple one, and two defects that appear similar may require very different investigation. Attempting to eliminate every variable can consume more effort than the measurement is worth. What we need is decision-quality evidence, not perfect evidence.

That may come from representative samples, historical work, repeated tasks, paired comparisons, engineering estimates supported by actual examples, or a combination of quantitative and qualitative evidence. The measurement method should be rigorous enough to challenge assumptions without becoming so burdensome that the organization spends more time measuring AI than benefiting from it.

• What engineering outcome is expected to improve.
• How the work is performed today.
• A reasonable estimate or sample of current engineering effort.
• Existing quality, review, rework, and maintenance expectations.
• What evidence will be collected during AI-assisted work.
• What result would justify continuing or expanding the practice.
• What result would cause the organization to modify or abandon it.

Defining the decision before collecting the evidence reduces the temptation to reinterpret disappointing results after the experiment has begun.

Follow the Work Beyond the Developer

Consider code review. AI can allow a developer to produce a substantial implementation very quickly. The developer may legitimately report that the task took far less time than expected. The reviewer, however, still needs to understand the change well enough to accept responsibility for approving it. If the implementation is larger than it would otherwise have been, uses unfamiliar patterns, or includes decisions the author cannot explain clearly because AI generated them, review effort may increase.

This creates an important measurement principle: The organization should measure the work at the point where it disappears, but also look for where it may have reappeared. For many AI-assisted engineering activities, the lifecycle view should include some combination of:

• Initial implementation effort.
• Developer review and correction.
• Peer review.
• Testing and defect correction.
• Security or architecture review where applicable.
• Deployment-related rework.
• Maintenance during subsequent changes.
• Defects attributable to the implementation.
• Documentation or knowledge-transfer effort.
• Human intervention required when an AI workflow fails.

Not every experiment needs every measurement. A low-risk documentation task does not require the same evaluation as an agent modifying production-facing software. The measurement should be proportional to the consequence and the decision being made. What matters is that leadership does not allow a local improvement to automatically become an organizational productivity claim.

The Review Paradox

If leadership measures only development throughput, the organization may celebrate increased output while reviewers become overloaded. Review quality can then decline, changes can wait longer in queues, or engineers may begin approving work they have not examined deeply enough simply because the volume has increased.

The answer is not to slow AI down merely to preserve the old balance. The answer is to recognize that the engineering system has changed and the bottleneck may have moved.

This is one reason successful AI adoption may require changes beyond the AI tool itself. Better automated tests, clearer architectural standards, smaller change sets, improved static analysis, stronger validation, and more structured context can reduce the amount of human reasoning required for routine verification. AI may even assist portions of review, provided the organization does not make the mistake of allowing the same assumptions that generated the work to become the only mechanism validating it. A useful AI productivity measurement therefore asks not only whether the author became faster, but whether the flow of acceptable engineering work improved.

Rework Is Where Many Productivity Claims Become Less Clear

If an engineer normally spends ninety minutes implementing a task and thirty minutes correcting it, while AI reduces implementation to twenty minutes but increases correction to forty-five minutes, the organization still gained meaningful time. The AI-assisted process does not have to be perfect to be valuable, nor does every additional correction mean the workflow was unsuccessful. What matters is the total result and whether the additional effort introduced by AI is smaller than the effort or risk it removed from the overall engineering process.

Likewise, additional review is not automatically evidence that AI reduced productivity. If AI enables substantially more valuable work while requiring modestly more review, the trade may be excellent. The objective is not to find AI workflows with zero additional cost, correction, or human involvement, because that would set an unrealistic standard that we do not apply to human development either. The objective is to identify workflows in which the value of the engineering improvement exceeds the consequences introduced by the new way of working and continues to do so as the practice is repeated.

This is why simplistic productivity claims can be misleading in both directions. Enthusiasts may count every minute saved during generation while ignoring the downstream effort required to review, correct, integrate, and maintain the result, while skeptics may point to every correction as evidence that AI provides no meaningful value. Neither approach gives leadership the information it needs to make a sound engineering decision. The useful question is whether, compared with the previous way of accomplishing the same engineering outcome, the AI-assisted workflow improved the work that actually mattered across the full engineering lifecycle.

Quality Cannot Be Separated From Productivity

For example, AI may generate routine tests considerably faster than developers do manually while occasionally missing business-specific edge cases. If developers can identify and add those cases efficiently, the workflow may still be extremely valuable. If the generated tests instead create a false sense of confidence and important behavior routinely goes untested, the same apparent productivity gain has a very different meaning. Quality evidence might therefore include defect rates, review findings, test effectiveness, security findings, architectural deviations, maintainability concerns, or the amount of human correction required before acceptance, with the appropriate indicators depending on the engineering activity being evaluated.

The point is not to surround every AI experiment with an enormous quality-management program or create so much measurement overhead that experimentation becomes difficult. It is to establish enough visibility to understand whether faster work is still producing an acceptable engineering result and to make sure the organization does not define productivity in a way that allows quality problems, rework, or future maintenance costs to disappear from the calculation.

Maintenance Is the Measurement We Usually Learn Too Late

This is where short evaluation windows can exaggerate AI productivity because they capture the immediate benefit without necessarily capturing the longer-term engineering consequence. An organization does not need to wait a year before allowing a successful AI practice to expand, but it should recognize when a claim has only been validated for the initial development stage. A useful evidence record might state that the practice has demonstrated significant implementation savings with no increase in review or testing effort while noting that the long-term maintenance impact remains unknown. That is a much stronger management statement than pretending the unknown does not exist or allowing an early productivity result to imply that every stage of the engineering lifecycle has already been validated.

Over time, the organization can revisit AI-assisted work and determine whether maintenance experience changes the original conclusion, creating a growing evidence base without requiring every possible question to be answered before the organization can move forward. The important principle is to measure what can be measured now, clearly identify what cannot yet be known, and avoid confusing evidence that has been established with assumptions that still need to be tested.

Do Not Measure Developers by Their AI Metrics

The same problem occurs when leadership creates an expectation that every team must demonstrate a certain percentage of AI adoption. Teams will find ways to satisfy the metric, but whether satisfying that metric improves software engineering is an entirely different question. AI usage is a means, not an outcome, and a highly experienced engineer who recognizes that a particular task is faster, safer, or more predictable to complete conventionally should not be penalized for exercising good judgment. Likewise, a team that replaces an unreliable AI workflow with a deterministic script may have substantially improved its engineering system even though its measured AI usage decreased. The measurement system should therefore reward better engineering decisions and outcomes rather than greater AI consumption.

This distinction becomes increasingly important as organizations move toward agents, because greater AI activity does not necessarily indicate a more mature AI engineering system. An agent that requires fewer attempts, less context, fewer model calls, and less human correction to produce the same successful result may represent a considerable improvement even though traditional adoption metrics show less activity. The objective throughout the transition remains the same: use AI where it earns its place by producing a better engineering outcome. The next step is creating a practical way to gather the evidence that proves when it has.

Define the Outcome Before Measuring the Tool

Suppose a team is evaluating AI-assisted defect investigation. The objective should not be “increase use of AI for debugging.” The objective might be to reduce the engineering effort required to identify the likely cause of a defect without increasing incorrect diagnoses or unnecessary code changes. That definition immediately gives us a more useful measurement. We can compare representative defects, observe the amount of engineering effort required to reach a correct diagnosis, record how often AI suggestions send the engineer in the wrong direction, and determine whether the overall investigation improves.

The same principle applies to code generation. If AI is being used to generate routine application components, the desired outcome might be to reduce implementation effort while preserving architectural consistency, review quality, test effectiveness, and maintainability. For documentation, the objective might be to reduce the effort required to keep technical documentation synchronized with engineering changes while maintaining accuracy. For test generation, the objective might be to reduce the effort required to create useful tests without reducing the team’s ability to identify meaningful failure conditions.

Writing the outcome this way forces the organization to acknowledge both sides of the value equation. We want the benefit, but we also identify the engineering qualities we are unwilling to sacrifice to obtain it.
We will refer to these as protected outcomes.

A protected outcome is something that must remain acceptable while the organization pursues an AI productivity improvement. If AI makes code generation faster but causes maintainability to fall below an acceptable level, the productivity gain has not satisfied the protected outcome. If AI dramatically reduces test-writing effort but produces tests that merely confirm the implementation rather than validate intended behavior, the practice may not be creating the value we think it is.

Protected outcomes will also become important in Chapter 3 when we decide whether a successful AI practice is mature enough to become a shared engineering standard. A standard should not simply prescribe a technique. It should protect an outcome.

• Engineering outcome: What are we trying to improve?
• Expected AI benefit: What work do we believe AI can remove or improve?
• Protected outcomes: What must remain acceptable while we pursue that benefit?
• Evidence: What would tell us whether the result improved?
• Decision threshold: What would make the practice worth continuing or expanding?

This small amount of discipline dramatically improves the quality of an AI experiment because the team knows what it is trying to learn before the result is known.

Use Representative Work

Representative work does not mean selecting the hardest possible cases to make AI prove itself. Nor does it mean selecting only simple tasks that make the technology look successful. The objective is to capture the range of work the practice would encounter if adopted.

For example, if AI is being evaluated for generating unit tests, do not test it only against small methods with obvious inputs and outputs. Include examples containing business rules, dependencies, error conditions, and the types of edge cases the team normally encounters. If AI is being evaluated for legacy-code comprehension, include code with some of the characteristics that make the actual application difficult to understand. If an agent is eventually expected to modify code across multiple files, a demonstration that changes one isolated method tells us very little about readiness for the intended use.

Representative work also helps reveal where the boundary of a successful practice lies. The organization may discover that AI performs extremely well for one class of tasks but becomes unreliable when a particular type of business context is required. That does not invalidate the practice. It defines it.

A mature conclusion might therefore be: “This approach consistently reduces engineering effort for routine service-layer test creation when expected behavior is visible in the code and existing requirements, but it should not be relied upon to identify undocumented business edge cases.” That is much more useful than either “AI is great at writing tests” or “AI misses edge cases.” We now know where the practice belongs.

Measure the Successful Outcome, Not Just the Attempt

This becomes particularly important with agents because agents can continue working. A failed attempt may trigger another model call, additional context retrieval, more tool usage, another test run, and another attempt. The developer may not be actively involved during those steps, but the workflow is still consuming resources. Eventually, the agent may succeed, escalate to a human, or stop. For measurement purposes, the meaningful unit is not necessarily the prompt, model call, or agent attempt. It is the successful engineering outcome.

That outcome might be a correctly diagnosed defect, an accepted pull request, a completed dependency update, a useful set of tests, an accurate migration, or another clearly defined engineering result. Once the outcome is defined, the organization can ask how much human effort, AI consumption, review, correction, and downstream work were required to achieve it.

This idea will become central to the economics discussion in Chapter 7. For now, it gives us a better way to think about productivity: the organization is not trying to make AI interactions efficient. It is trying to make engineering outcomes efficient.

Look for Displaced Work

Displaced work does not automatically make the practice unsuccessful. The organization may intentionally move work from an expensive or constrained part of the engineering system to another place where it can be performed more efficiently. What matters is that the movement is visible. A simple Lifecycle Effort Map can help. For the activity being evaluated, identify the stages that materially contribute to the result and compare the before-and-after effort.

Typical stages might include:

• Understanding or preparation.
• Initial implementation.
• Developer validation and correction.
• Peer review.
• Testing.
• Security or architecture review.
• Deployment correction.
• Early production support.
• Subsequent maintenance.

Not every stage will apply to every use case, and the organization should resist the temptation to measure stages that cannot reasonably affect the decision. The purpose is to find material changes, not create a time-tracking exercise.

If AI reduces implementation by two hours and increases review by ten minutes, the conclusion is probably straightforward. If it reduces implementation by forty minutes but adds thirty minutes of review and fifteen minutes of correction, the result deserves closer examination. If it reduces initial effort substantially but repeatedly creates maintenance problems later, the practice may need stronger standards or better context rather than abandonment. The Lifecycle Effort Map makes those tradeoffs visible.

Account for Human Correction

If developers repeatedly correct formatting or naming conventions, the problem may be solved through better standards or automated tooling. If they repeatedly correct architectural decisions, the AI may lack architectural context. If they correct business logic, important domain knowledge may be missing. If they continually remove unnecessary complexity, the model or prompting approach may be inappropriate for the task. If engineers have difficulty identifying what needs correction at all, the workflow may be operating beyond the team’s ability to validate it safely.

A useful correction record can therefore capture:

• How much meaningful correction was required.
• What category of correction occurred.
• Whether the same correction appears repeatedly.
• Whether the cause can be addressed through standards, context, tooling, or workflow design.
• Whether the correction changes the conclusion about the practice’s value.

Repeated corrections are especially important because they represent an opportunity to improve the system. If every developer has to tell AI the same thing manually, the organization may have discovered information that should be supplied automatically. If every reviewer catches the same architectural deviation, the organization may need to make the architectural expectation machine-accessible or enforce it through conventional tooling. In this way, measurement does more than judge AI. It helps improve the environment in which AI operates.

Distinguish Efficiency From Capacity

Engineering organizations contain backlogs of work that previously could not justify the available time. Technical debt remains unresolved because feature work takes priority. Documentation is incomplete. Tests could be stronger. Modernization is deferred. Developers spend limited time improving internal tooling. Architecture needs attention. Security work competes with delivery schedules. Knowledge transfer is postponed until someone announces a departure. AI-created capacity can be used to address some of this work.

This matters because the value of AI may appear not only as greater feature throughput but also as an improvement in the engineering system itself. If AI reduces routine implementation effort and engineers use part of that capacity to strengthen tests, simplify architecture, reduce technical debt, or improve documentation, the business may receive significant value even though the number of features delivered does not increase proportionally.

Leadership should therefore define what it intends to do with meaningful capacity gains. Otherwise, saved time tends to disappear into the general workload and becomes difficult to distinguish from ordinary fluctuations in engineering activity. The question should be: If AI removes meaningful engineering effort, where do we want that capacity to go?

Possible answers may include greater delivery throughput, modernization, technical-debt reduction, improved quality, stronger testing, security work, documentation, innovation, or simply absorbing growth without increasing engineering headcount at the same rate. The correct answer depends on the organization’s priorities. The important point is that productivity becomes much more meaningful when saved capacity has a destination.

Avoid the False Precision Trap

Those studies can be useful, but they do not tell you what AI is doing inside your engineering organization. Your applications and architecture are different, your engineers have different levels of experience, and your testing environment, business rules, development processes, code quality, documentation, and risk tolerance create conditions that may be very different from those represented in an outside study. A productivity improvement measured in a controlled task or another organization should therefore be treated as evidence that an opportunity may exist, not as a forecast of the results your organization should expect to achieve. Internal evidence does not need to be perfect to be valuable, but it does need to be honest about what it can support.

It is entirely reasonable, for example, to conclude that for a particular class of routine work, your sample shows that AI substantially reduces implementation effort with no material increase in review or correction, providing enough evidence to expand the practice to two additional teams while continuing to monitor its maintenance impact. That conclusion is actionable even without claiming a company-wide productivity percentage. It is equally reasonable to conclude that developers are reporting significant initial time savings, but your evaluation found that much of those savings currently returns during review and correction, suggesting that the workflow should be improved before it is expanded. That is not evidence of a failed AI initiative; it is exactly the kind of information a managed AI engineering process is supposed to discover.

The purpose of measurement, therefore, is not to produce an impressive number that can be repeated in a presentation. It is to give leadership enough reliable evidence to determine what should happen next—whether a practice should be expanded, modified, investigated further, or stopped. The value of the measurement is ultimately determined by whether it improves the quality of the next decision.

Build an Evidence Record That Can Follow the Practice

For each AI Leverage Candidate evaluated, create a concise AI Engineering Value Record containing the problem, baseline, AI-assisted approach, protected outcomes, evidence, findings, limitations, and decision. This record becomes part of the organizational knowledge surrounding the practice.

A useful value record should answer:

• What engineering activity was evaluated?
• What problem or source of effort were we trying to improve?
• What was the previous workflow?
• What AI-assisted workflow was tested?
• What representative work was included?
• What lifecycle effort changed?
• What human correction was required?
• What happened to review, quality, testing, and maintainability?
• What protected outcomes had to remain acceptable?
• What did we learn about where the approach works?
• Where did it fail or become less useful?
• What conditions appear necessary for success?
• What decision did we make?
• What evidence remains incomplete and should be revisited later?

The value record does not need to become a long report. In most cases, one or two well-structured pages supported by the relevant engineering evidence will be more useful than a presentation filled with adoption metrics. The purpose is to preserve the reasoning behind the decision.

That reasoning becomes especially important when the practice begins spreading. A team receiving a recommended AI workflow should understand not only what the organization recommends, but why, where the recommendation has been proven, and what limitations remain. This is the bridge between Chapter 2 and Chapter 3.

We began Chapter 1 by discovering what engineers were already doing. We then selected promising practices rather than trying to manage every experiment equally. In this chapter, we are testing those practices against actual engineering outcomes. When the evidence becomes strong enough, the organization faces the next question: When does something that works stop being an experiment and become part of the way we engineer software?

The Chapter 2 Working Checklist

The checklist also needs to remain connected to the engineering work. This is not an AI adoption scorecard. We are not trying to reward the team with the most prompts, the greatest number of generated lines of code, or the highest percentage of AI usage. We are examining whether a particular AI-assisted practice changes the amount of engineering effort required to produce an acceptable result and whether any of that effort or risk reappears elsewhere in the lifecycle.

Begin by describing the work without describing the AI. This keeps the evaluation centered on the engineering problem rather than the technology being tested. For each leverage candidate, record:

• Engineering activity: What work is being performed?
• Current workflow: How is the work completed today?
• Current source of effort or friction: What makes the work expensive, slow, repetitive, or difficult?
• Desired improvement: What do we want to become better?
• Reason for evaluating AI: What has already been observed that suggests AI may create meaningful leverage?

A useful description might be: Developers working in an established application spend substantial time creating and updating unit tests for routine service-layer changes. Several engineers are already using AI to create initial test implementations and report significant reductions in writing time. We want to determine whether the practice reduces total test-development effort while preserving test quality and maintainability.

That statement gives the evaluation a purpose. By contrast, “We are evaluating AI-generated unit tests” tells us what technology is being used but very little about the engineering problem we are trying to solve.

2. Define the Expected AI Benefit

Next, state what you believe AI will improve before examining the results. This is important because vague expectations make almost any outcome look successful afterward. Record:

• What engineering effort should AI reduce?
• What part of the workflow should become faster or easier?
• Is AI expected to improve quality as well as speed?
• Is the benefit primarily time, capacity, consistency, quality, knowledge access, or another engineering outcome?
• Where should the improvement become visible?

Avoid broad statements such as “increase developer productivity,” and instead define the expected improvement closely enough that the evidence can challenge it. For example, the expected benefit might be to reduce the engineering effort required to produce the initial unit-test implementation while maintaining comparable review effort, test effectiveness, and future maintainability. Defining the benefit this way gives the organization a clear understanding of what it is trying to improve while also identifying what it is unwilling to sacrifice merely to achieve faster generation.

3. Identify the Protected Outcomes

Protected outcomes are the engineering qualities that must remain acceptable while the organization pursues the AI benefit. This prevents a local productivity improvement from being declared successful while important costs are transferred elsewhere. Depending on the activity, protected outcomes may include:

• Correctness.
• Security.
• Reliability.
• Test effectiveness.
• Architectural consistency.
• Maintainability.
• Reviewability.
• Regulatory or contractual requirements.
• Business-rule accuracy.
• Operational stability.

Do not automatically select every possible outcome; instead, identify the ones that could materially change the decision and state them explicitly. For example, a protected outcome might be that AI-assisted test generation must reduce creation effort without producing materially weaker tests or increasing correction and maintenance enough to eliminate the savings. Defining the protected outcome this way is important because the organization is not merely asking whether AI can perform the task; it is determining whether AI can perform the task while preserving the engineering outcomes that matter.

4. Establish the Current Baseline

The baseline should describe the existing workflow well enough that the organization can recognize meaningful change. Use representative samples where exact historical measurements do not exist rather than inventing precision. Capture what is reasonably available:

• Typical engineering effort.
• Review effort.
• Correction or rework.
• Testing effort.
• Common defects or failure patterns.
• Maintenance concerns.
• Waiting or handoff time where material.
• Existing tool or infrastructure cost where relevant.
• Qualitative engineering observations that could affect the decision.

If exact time data is unavailable, use representative examples or reasonable ranges, since a baseline stating that a task typically requires between two and four hours under comparable conditions can still support a useful decision, while an unsupported claim that the task takes exactly 2.7 hours creates a level of precision the evidence cannot justify. The purpose of the baseline is comparison, not accounting.

5. Select Representative Work

Choose work that reflects what the practice would actually encounter if adopted. Avoid evaluating AI only against examples selected because they are unusually easy or unusually difficult. Document:

• Types of work included.
• Complexity represented.
• Applications or repositories involved.
• Engineers participating.
• Important conditions represented.
• Known conditions excluded from the evaluation.

If the evaluation excludes an important category of work, record the limitation. The final conclusion can then reflect what was actually tested. A result such as “effective for routine service-layer changes in applications with strong automated testing” is far more useful than a broad claim that “AI works well for implementation.”

6. Map the Lifecycle Effort

Use the discovery process to identify weaknesses in the engineering environment that may limit future AI leverage.Now compare the work before and after AI at the stages that matter to the outcome. This is where displaced effort becomes visible.

For the relevant stages, record whether effort:

• Decreased.
• Remained approximately the same.
• Increased.
• Could not yet be determined.

Potential lifecycle stages include:

• Understanding and preparation.
• Initial implementation.
• Developer review.
• Correction and rework.
• Peer review.
• Testing.
• Architecture or security review.
• Deployment correction.
• Early production support.
• Subsequent maintenance.

Where possible, add a reasonable estimate of the magnitude of the change. The organization does not need stopwatch precision, but “implementation decreased substantially while review increased slightly” is much more informative than “developers were faster.”

7. Record Human Correction

Do not treat correction as a single number. Record what engineers repeatedly have to fix because recurring correction tells us how the workflow itself might be improved. For meaningful corrections, capture:

• What AI produced incorrectly or inadequately.
• How frequently the problem occurred.
• How much engineering effort correction required.
• Whether the problem was obvious or difficult to detect.
• Whether the cause appears to be missing standards, missing context, model capability, workflow design, or another issue.
• Whether the problem can reasonably be prevented in future runs.

This information will become particularly useful in later chapters. A recurring architectural correction may eventually lead to a stronger engineering standard. Repeated business-rule corrections may identify missing reusable context. Excessive unnecessary code may change model or workflow selection. Repeated failures that are expensive to detect may limit how much authority the AI should receive. Measurement should therefore improve the system, not merely grade it.

8. Evaluate Quality and Downstream Consequences

Determine whether the protected outcomes remained acceptable. Review the evidence that matters for the activity:

• Were defects materially different?
• Did reviewers identify more or fewer significant issues?
• Did testing reveal different failure patterns?
• Did the implementation follow established architecture and engineering practices?
• Were security or reliability concerns introduced?
• Was the resulting work understandable to engineers who did not generate it?
• Did subsequent changes expose maintainability problems?
• Did AI introduce unnecessary complexity or dependencies?
• Did the workflow create new operational work elsewhere?

Where evidence is not yet available, mark it as unknown rather than assuming there is no consequence. This is particularly important for maintenance. A practice may have strong evidence for immediate implementation savings while still having limited evidence about long-term maintainability. The correct response is not necessarily to stop. It is to preserve that uncertainty in the decision.

9. Calculate the Successful Engineering Outcome

Where the workflow includes retries, repeated AI interactions, agent activity, or substantial correction, evaluate the effort required to reach an accepted result, not merely the cost of the first attempt. Capture:

• AI attempts or retries where material.
• Human intervention.
• Review and correction.
• Tool or model consumption where significant.
• Whether escalation to a person or stronger model was required.
• Percentage or number of representative tasks that reached an acceptable result.
• Approximate total effort per successful outcome.

This prevents an inexpensive but unreliable workflow from appearing more efficient than a slightly more expensive workflow that consistently succeeds.

10. Determine Where the Saved Capacity Goes

If the evidence shows meaningful engineering savings, decide how the organization wants to use that capacity. Potential uses include:

• Increased feature delivery.
• Technical-debt reduction.
• Modernization.
• Improved automated testing.
• Security work.
• Documentation and knowledge capture.
• Architecture improvement.
• Internal tooling.
• Innovation or experimentation.
• Absorbing organizational growth without proportional headcount growth.

This does not require assigning every saved hour to another activity. It requires leadership to recognize that productivity becomes strategically meaningful when the organization knows what it intends to do with the capacity created.

11. Make the Decision

The evidence should result in an explicit decision rather than simply another presentation. Choose one:

• Continue experimenting — promising, but evidence is not yet sufficient.
• Improve and retest — value appears possible, but the workflow has identifiable weaknesses.
• Recommend the practice — evidence supports broader voluntary use under defined conditions.
• Candidate for standardization — evidence is strong enough to consider moving the practice into the standards process developed in Chapter 3.
• Do not expand — evidence does not currently justify broader use.
• Stop — the practice creates insufficient value or unacceptable consequences.

Record the reason for the decision and the evidence supporting it. This final step matters because an experiment without a decision can remain an experiment indefinitely. The organization continues paying for it, developers continue using it, and nobody knows whether it has actually earned a place in the engineering system.

The AI Engineering Value Record

A completed AI Engineering Value Record should contain:

• Practice / engineering activity
• Engineering problem
• Current baseline
• AI-assisted approach
• Expected benefit
• Protected outcomes
• Representative work evaluated
• Lifecycle effort findings
• Human correction findings
• Quality and downstream findings
• Cost or consumption findings where material
• Known limitations
• Conditions required for success
• Evidence still incomplete
• Decision
• Reason for decision
• Date for reassessment if appropriate

The value of this record becomes clearer over time. Six months later, when someone asks why the organization recommends a particular AI practice, the answer should not be, “We tried it and everyone liked it.” The organization should be able to show what was evaluated, what improved, what remained protected, what limitations were discovered, and why the decision was made. That is the difference between enthusiasm becoming policy and evidence becoming engineering practice.

What You Should Have When Chapter 2 Is Complete

You should now have:

• A defined engineering outcome for each evaluated
• AI practice.
• A baseline against which improvement can be judged.
• Protected outcomes that cannot simply be traded away for generation speed.
• Representative engineering work used for evaluation.
• A Lifecycle Effort Map showing where work decreased, remained, or moved.
• Human Correction findings that reveal recurring weaknesses in the AI workflow.
• Quality and downstream evidence.
• An AI Engineering Value Record for each evaluated practice.
• A decision to continue experimenting, improve and retest, recommend, consider standardization, not expand, or stop.

Perhaps more importantly, leadership should now be able to answer a much better question than “Are our developers faster with AI?” It should be able to say, “Here are the engineering activities where AI is creating measurable value, here is the evidence supporting that conclusion, here are the conditions under which that value appears, and here is what we still do not know.” That represents a meaningful change in the maturity of the AI conversation because leadership is no longer relying primarily on vendor demonstrations, industry productivity claims, or individual developer enthusiasm. The organization has begun creating its own evidence and can use that evidence to make better decisions about where AI should be expanded, where a workflow still needs improvement, and where the results remain uncertain.

Evidence, however, introduces the next challenge. Suppose an AI-assisted practice works, developers like it, representative engineering work shows that it removes meaningful effort, review and correction remain acceptable, protected outcomes are satisfied, and several teams could benefit from using the same approach. It might seem reasonable for leadership to simply tell everyone to begin using it, but a successful practice is not automatically a practice that should become an organization-wide standard. What works for one team may depend on conditions that do not exist elsewhere, and a useful technique may be mature enough to recommend without being mature enough to require. Introducing a standard too early can restrict valuable experimentation around a rapidly changing technology, while introducing it too late can allow unnecessary inconsistency, duplicated learning, and avoidable problems to spread across the engineering organization.

The next stage of the transition, therefore, is not simply about standardization. It is about learning what should become standard, what should remain flexible, what engineering outcome the standard is intended to protect, and how engineers can challenge or improve that standard when they discover something better. Once an organization can make those distinctions deliberately, successful AI experimentation can begin evolving into repeatable engineering practice, which is where we turn next.

• Introduction → AI engineering framework
• Chapter 1 → AI in software development
• Chapter 2 → AI developer productivity
• Chapter 3 → AI engineering standards
• Chapter 4 → AI agents in software development
• Chapter 5 → AI technical debt
• Chapter 6 → AI engineering context
• Chapter 7 → AI cost governance
• Chapter 8 → AI engineering operating model

If you have questions, please let us know.

Take a few minutes to complete the assessment and gain a clear, practical view of your organization’s AI readiness—and what to do next.