Chapter 4

Chapter 4

Assistants, Agents, and Where Each Belongs

Giving AI More Authority Without Giving Away Engineering Control

This creates enormous potential because the organization is no longer using AI only to accelerate individual engineering actions; it can begin allowing AI to perform portions of an engineering workflow.

The question is not whether agents are good or bad, nor whether an organization should be aggressive or conservative about adopting them. The useful question is much more specific: For this engineering activity, what decisions and actions can AI perform reliably enough to receive authority, and where should human responsibility remain? Answering that requires us to separate several AI operating models that are often discussed as though they were the same thing.

Three Operating Models for AI-Assisted Engineering

For the purposes of this book, we will distinguish AI engineering work according to who directs the work, who determines the steps, and who has authority to act. There are three useful operating models.

In the first model, the human engineer remains in control of the workflow and uses AI to assist with individual tasks. The engineer decides what to ask, evaluates the response, determines the next action, and remains responsible for moving the work forward.

This is the model most developers first encounter with AI coding assistants. The engineer may ask AI to explain a method, suggest an implementation, generate a test, identify a possible defect, refactor a section of code, or draft documentation. AI may produce substantial work, but the human remains responsible for deciding what happens with that work.

The important characteristic is not how much content AI generates. AI could generate three lines of code or three hundred. If the engineer is still directing the workflow and deciding whether and how each meaningful result is used, the operating model remains human-directed assistance.

This model is particularly useful when the task requires substantial judgment, the environment contains important undocumented context, the work is unfamiliar, or the consequences of an incorrect decision make close human involvement appropriate. It is also often the correct starting point for new AI uses because the engineer can observe AI behavior while retaining direct control.

Human-directed assistance should not be viewed as an immature state that every organization must eventually eliminate. There will always be engineering work where human judgment remains valuable enough that assistance is the appropriate model. The objective of AI transformation is not to move everything toward autonomy. It is to place work at the right level of autonomy.

The second model changes who determines the sequence of work. AI may receive a broader objective, determine the steps required to accomplish it, use tools, inspect information, produce changes, and perform validation. The human no longer directs every individual action, but the AI cannot complete a consequential transition without human approval.

An agent might receive a request to update a dependency across an application. It identifies the relevant files, makes the changes, executes tests, resolves straightforward failures, prepares a summary, and creates a pull request. The engineer does not tell the agent which file to open or which command to execute. The agent determines those steps.

But the engineer still controls the consequential boundary. The agent may prepare the pull request but not merge it. It may prepare an infrastructure change but not apply it. It may diagnose a production issue but not modify production. It may recommend a database migration but not execute it. This operating model is extremely important because it allows organizations to capture much of the value of agentic work without immediately transferring final authority.

Human approval, however, should not be treated as a magical safety mechanism. Approval provides protection only when the person approving the work has enough information, time, expertise, and visibility to make a meaningful decision. If an agent produces enormous changes that reviewers cannot reasonably understand, requiring a human to click Approve does not necessarily create strong governance. It may create the appearance of human control without the substance.
The quality of the approval boundary therefore matters as much as the existence of the approval.

A well-designed agent should make approval easier by presenting what it changed, why it changed it, what evidence it used, what tests or validation it performed, what remains uncertain, and where human judgment is specifically required. The objective is not to bury the engineer under the agent’s activity log. It is to provide the evidence necessary to make the decision.

In the third model, AI is allowed to complete a defined engineering activity without requiring human approval for each execution. The agent receives authority to act inside an established boundary because the organization has enough evidence and sufficient controls to trust the workflow for that class of work. The word bounded is essential because autonomy should not mean that an agent has general permission to do whatever it believes is necessary. It means the organization has defined the objective, the permitted actions, the systems the agent may access, the validation required, the conditions under which it must stop, and the circumstances that require escalation.

A bounded autonomous agent might be permitted to update a known class of low-risk dependencies, execute an established test suite, create and merge a change when all required validation passes, and stop when it encounters a failure outside a defined category. Another might maintain generated documentation based on approved source material, while an infrastructure agent might correct a narrow class of known configuration drift while escalating anything outside the established pattern. In each case, the engineering organization is not trusting AI in the abstract; it is trusting a particular workflow inside a particular boundary under particular conditions. This distinction prevents one of the most dangerous assumptions in agent adoption: that evidence showing an agent can perform one task reliably establishes that the agent should receive broader authority. Authority should attach to the proven activity, not to the AI system itself.

The Same AI Can Operate in All Three Models

An AI coding system might operate as an assistant when a developer asks it to suggest a change. It might operate in the second model when given a feature request and allowed to modify the repository, run tests, and prepare a pull request for human review. The same system might operate autonomously for a narrow category of routine maintenance changes after the organization has accumulated enough evidence to permit those changes without individual approval. This is why product labels are insufficient for governance. Asking, “Are we using agents?” tells leadership very little.

• What objective is the AI receiving?
• What decisions can it make independently?
• What tools can it call?
• What systems can it access?
• What can it read?
• What can it modify?
• What validation must succeed?
• Where is human approval required?
• When must the AI stop?
• When must it escalate?
• What happens if it makes the wrong decision?

Those questions define the operating model.

Authority Should Be Divided, Not Granted as a Single Permission

An agent may reasonably need permission to read source code without receiving permission to modify it. It may modify a development branch without being allowed to merge into the main branch. It may execute a test suite without being allowed to change the test configuration. It may inspect production logs without being allowed to modify production systems. It may create an infrastructure plan without being allowed to apply that plan. Authority should therefore be decomposed into specific capabilities.

For an agentic workflow, consider separately:

• Read authority: What information can the agent inspect?
• Write authority: What can it modify?
• Execution authority: What commands, builds, tests, or tools can it run?
• Transaction authority: Can it create, submit, merge, deploy, publish, or otherwise complete a consequential action?
• Credential authority: What identities, secrets, or service permissions can it use?
• Delegation authority: Can it invoke other agents, models, or systems?
• Persistence authority: Can it continue operating after the initial task or schedule future work?
• Recovery authority: Can it reverse or repair its own actions?

Separating these permissions allows the organization to give the agent enough authority to create value without granting capabilities that the task does not require, which is simply the principle of least privilege applied to AI engineering. Agents, however, add another dimension because permissions alone do not define authority; an agent can make decisions about when and why to use the permissions it has. An engineer may give a conventional automation script permission to modify a file, but the script follows a deterministic sequence, while an agent may reason that modifying the file is necessary based on the objective it has been given. This makes task boundaries and decision boundaries just as important as access controls, because responsible agent authority must define not only what the agent is technically capable of doing, but also the circumstances under which it is permitted to decide that an action should be taken.

The AI Autonomy Map

Without an explicit map, authority often expands through convenience. An agent begins by suggesting changes. Developers become comfortable with the suggestions, so it is allowed to modify files. Because developers are already accepting most changes, it is allowed to create pull requests. Eventually someone asks why humans are still approving routine changes, and the agent is allowed to merge them. Each step may appear reasonable in isolation, but the organization may never stop to ask whether the evidence and engineering controls grew at the same rate as the authority. The AI Autonomy Map forces that question.

For each agentic engineering activity, record:

• Engineering activity: What outcome is the agent responsible for producing?
• Current operating model: Human-directed assistance, AI-directed with human approval, or bounded autonomous execution.
• Permitted actions: What can the AI actually do?
• Systems and tools: What can it access?
• Validation: How does the engineering system determine whether the result is acceptable?
• Human boundary: Where is human review, approval, or judgment required?
• Stop conditions: When must the agent stop rather than continue trying?
• Escalation conditions: When must responsibility return to a human?
• Failure consequence: What could happen if the agent is wrong?
• Recovery: Can the action be reversed, and how?
• Evidence level: What has the organization actually demonstrated about the workflow?
• Next autonomy level: What additional authority is being considered?
• Evidence required for expansion: What must become true before that authority is granted?

The final field is especially important because it turns autonomy into an engineering progression rather than a technology aspiration. The question is no longer, “When can we make this agent fully autonomous?” but instead, “What evidence would justify the next increase in authority?” Sometimes the answer will be straightforward and the evidence will support allowing the agent to take another action independently, while in other situations the organization may determine that there is no meaningful reason to increase the agent’s authority at all. That is a perfectly valid outcome, because the objective is not maximum autonomy; it is the appropriate level of autonomy for the engineering work being performed.

Separate Experimentation From Established Practice Autonomy Should Solve a Problem

Autonomy is valuable when it removes a meaningful constraint from the engineering system, which means the organization should ask, “What engineering cost or bottleneck are we removing by increasing AI authority?” If there is no good answer, additional autonomy may not be necessary. This principle protects organizations from pursuing autonomous engineering simply because the technology makes an impressive demonstration or because greater autonomy appears to represent a more advanced stage of AI adoption. The objective remains engineering value, not autonomy for its own sake.

Reversibility Changes the Autonomy Decision

An activity with occasional errors may still be a strong candidate for autonomy if failures are easy to detect, inexpensive to reverse, and contained within a safe environment. Another activity may require human approval even when the AI performs extremely well because the rare failure would be difficult to detect or costly to recover from. A useful autonomy decision therefore considers three questions together: How likely are we to detect a bad result? How serious is the consequence if we do not? How easily can we recover? Taken together, those questions often tell leadership far more about the appropriate level of autonomy than a model accuracy percentage alone.

Validation Is What Makes Autonomy Possible

This creates an important relationship: the ceiling on useful AI autonomy is often determined less by the intelligence of the model than by the organization’s ability to validate its work. A more capable model can certainly improve performance, but intelligence cannot compensate indefinitely for an engineering environment that cannot clearly define or detect success. This is one reason AI transformation frequently exposes the value of engineering investments that were important long before AI arrived. Better testing, clearer architecture, stronger automation, improved observability, and reliable deployment controls may suddenly provide a second return because they allow the organization to delegate more work safely. Before asking whether an agent is capable of doing more, leadership should therefore ask whether the engineering system is capable of knowing when the agent did it correctly, because that ability becomes one of the most important factors in determining where human approval can eventually be reduced, moved, or where it should remain.

Human Approval Should Be a Designed Control

This becomes more important as agents become faster. An agent may be able to complete several hours of implementation work in minutes, but the human ability to understand consequential engineering decisions has not accelerated at the same rate. If an organization deploys enough agents, it can create a new bottleneck in which engineers spend increasing amounts of time reviewing machine-generated work. Eventually, one of two things happens: the expected productivity gains begin disappearing into review, or review becomes increasingly superficial because the volume exceeds the amount of attention humans can reasonably provide.

The objective should therefore not be to place a human approval step everywhere. It should be to place meaningful human judgment at the points where it changes the risk or quality of the outcome.

For each approval boundary, ask what decision the human is actually making. Is the engineer verifying technical correctness? Confirming a business decision? Evaluating an architectural tradeoff? Accepting a security consequence? Authorizing access to a sensitive environment? Deciding whether an ambiguous requirement has been interpreted correctly? If the organization cannot explain why human judgment is required at that point, the approval may be unnecessary. Conversely, if the decision depends on context, accountability, or judgment the AI does not reliably possess, removing the approval merely to increase automation would be premature.

A useful approval should also give the reviewer the evidence necessary to make the decision. Depending on the activity, the agent may need to present the objective it received, the files or systems it changed, the significant decisions it made, the validation it performed, any assumptions it made, failures or retries it encountered, and anything it remains uncertain about. This does not mean the reviewer needs a transcript of every model interaction. In fact, excessive agent logs can make meaningful review harder. The goal is to provide decision-relevant evidence, not an exhaustive history of machine activity.

As the workflow matures, the organization should also examine what reviewers actually do. If humans repeatedly approve a particular class of agent work without finding meaningful problems, and independent validation reliably protects the required outcomes, the approval step may have become a candidate for removal. If reviewers frequently catch important issues, the opposite conclusion is appropriate: the human boundary is still providing significant value, and the organization should understand what information or capability the agent lacks before considering greater autonomy.
Approval should therefore be treated as something that can move based on evidence rather than as a permanent feature of the workflow.

Design the Escalation Before the Agent Needs It

An agent might be required to escalate when it encounters a business rule it cannot resolve from approved sources, when required tests continue failing after a defined number or category of corrective attempts, when the proposed change expands beyond the expected scope, when it discovers a security-sensitive condition, when required information conflicts, when it needs access beyond its assigned permissions, or when the confidence required for a consequential decision cannot be established. These conditions are not all technical failures because an agent may be functioning exactly as designed and still encounter a decision it should not make. An agent that encounters two conflicting interpretations of a business requirement does not necessarily need more reasoning time; it may need a product owner. An agent that discovers a potential architectural change does not necessarily need a more powerful model; it may need an architect. Likewise, an agent that requires a new production permission to complete its objective should not necessarily attempt to find another technical path because it may simply have reached the boundary the organization deliberately established.

Escalation, therefore, should not be viewed as agent failure because correct escalation is successful agent behavior. An engineering organization should evaluate agents partly on whether they recognize the limits of their authority, available information, and ability to continue safely. An agent that completes ninety tasks and correctly escalates ten ambiguous ones may be considerably more valuable than an agent that attempts all one hundred and confidently makes several consequential mistakes. The goal is not to create an agent that always finds a way to finish; it is to create an agent that can distinguish between situations it has been authorized and equipped to resolve and those in which the correct engineering action is to return the decision to a human.

Retry Limits Are Not Enough

An agent should be able to recognize when the problem is no longer the same problem it was authorized to solve. A dependency update that begins requiring changes to unrelated application behavior has crossed a scope boundary. A defect correction that begins modifying architecture has crossed another. An infrastructure adjustment that requires broader permissions than originally expected should stop before obtaining or attempting to work around those permissions.

Useful stop conditions can therefore include:

• The work expands beyond the authorized scope.
• Required validation fails in a way the agent is not authorized to resolve.
• The agent encounters conflicting authoritative information.
• A change would require an architectural or business decision.
• Additional system access or credentials are required.
• The agent reaches a defined resource or cost boundary.
• The potential consequence of the next action exceeds the authority granted.
• Recovery from the proposed action would no longer be straightforward.
• The agent cannot produce the evidence required for the next approval or autonomous action.

The resource and cost boundary will become particularly important in Chapter 7. Once agents can continue working independently, the organization must decide not only what they are technically allowed to do but how much resource they are allowed to consume trying to do it. An agent should not spend fifty dollars—or five hundred dollars—solving a problem that an engineer could resolve in ten minutes simply because nobody told it when to stop. Technical authority and economic authority eventually need to meet.

Agents Need Observability, Not Just Logs

This creates an audit trail that serves several purposes. It allows engineers to investigate an incorrect result, helps the organization determine whether an agent is repeatedly struggling with the same class of problem, and provides evidence for deciding whether authority should expand. It can also reveal unnecessary tool calls or expensive loops and help distinguish a model problem from an environment problem. If an agent repeatedly fails because it cannot find an authoritative business rule, for example, changing models may not solve anything because the observability data may reveal that the real problem is missing engineering context. The useful question, therefore, is not simply, “Can we see what the agent did?” but rather, “Can we understand enough about what happened to improve the workflow and make the next authority decision?” This is the same evidence-oriented philosophy we used in Chapters 2 and 3, now applied to autonomous behavior.

Avoid the Single-Agent Trap

A narrowly defined agent performing dependency updates can be given permissions, validation, stop conditions, and escalation rules specifically suited to dependency updates. A general engineering agent capable of performing almost any development task requires much broader authority and encounters a far wider range of situations, forcing the organization to determine whether the controls appropriate for one activity remain appropriate for every other activity the agent might undertake. This does not mean organizations must build a separate agent for every tiny task; it means authority becomes easier to manage when the purpose of the agent is bounded. The same principle already exists in conventional system design, where services, roles, permissions, and automation are generally safer and easier to reason about when responsibilities are clear, and AI does not eliminate that principle.

A bounded agent can also use other specialized agents or services where appropriate, provided the delegation itself is controlled. For example, an engineering agent might invoke a security-analysis capability without receiving permission to alter security policy, while a documentation agent might consume information produced by a code-analysis workflow without receiving write access to the repository. The architecture can separate responsibilities rather than accumulating every capability inside one autonomous identity, making it easier to understand what each component is allowed to do and what controls should apply to its actions.

This separation becomes particularly useful when models differ in capability and cost because the agent responsible for orchestrating a workflow does not necessarily need to use the same model for every step. Some activities may require strong reasoning, while others can be handled by smaller models or conventional software. We will return to this in Chapter 7 because agent architecture and AI economics are closely connected, but for now the important point is that more capable does not automatically mean better designed.

Autonomy Can Move in Both Directions

Suppose an agent has been autonomously completing a routine class of updates successfully for six months. A major application change then alters the architecture, automated test coverage falls, or a new integration introduces business behavior the agent does not understand. The conditions that originally justified autonomy may no longer exist.
The correct response may be to move the workflow temporarily back to AI-directed work with human approval.

This should not be viewed as failure. It is evidence that the organization is actually managing autonomy rather than treating it as a status level that, once achieved, can never be surrendered.

The same can happen when a model changes, a vendor modifies behavior, a new regulatory requirement appears, or production experience reveals a failure mode that was not visible during evaluation. Authority should remain attached to current evidence.

A mature operating model therefore allows autonomy to:

• Increase when evidence and controls improve.
• Remain unchanged when additional authority provides little value.
• Narrow when the scope of proven success becomes clearer.
• Decrease when conditions or evidence deteriorate.
• Stop entirely when the workflow no longer produces sufficient value.

This makes the AI Autonomy Map a management instrument rather than an adoption scoreboard.

The Chapter 4 Working Checklist

Begin with the outcome rather than the agent. Record:

• Engineering activity to be performed.
• Desired outcome.
• Frequency of the activity.
• Current human workflow.
• Source of engineering effort or delay.
• Why agentic execution may improve the workflow.

Do not begin with, “We want an agent that can…” Begin with the engineering problem the organization is trying to improve.

2. Select the Current Operating Model

Classify the activity according to how authority currently operates:

• Human-directed AI assistance — The engineer directs the work and decides each meaningful next step.
• AI-directed work with human approval — AI determines and performs multiple steps but a human approves consequential completion.
• Bounded autonomous execution — AI may complete the defined activity without individual human approval inside established boundaries.

Record why the current model is appropriate.

3. Decompose the Agent’s Authority

Do not grant authority as a single permission. Define it by capability. Record what the agent may:

• Read.
• Modify.
• Execute.
• Submit or create.
• Merge.
• Deploy.
• Publish.
• Access through credentials.
• Invoke through other agents or tools.
• Schedule or continue independently.
• Reverse or repair.

For every capability, ask whether it is actually required to accomplish the defined engineering activity.

4. Define the System Boundary

Identify where the agent can operate. Document:

• Repositories.
• Development environments.
• Build systems.
• Test systems.
• Issue or work-management systems.
• Documentation systems.
• Databases.
• Cloud or infrastructure systems.
• Production systems.
• External services.
• Other agents or models.

Explicitly identify systems that remain outside the agent’s authority where that distinction is important.

5. Define Independent Validation

For every consequential output, identify how the engineering system determines whether the result is acceptable. Potential validation may include:

• Automated tests.
• Static analysis.
• Security scanning.
• Architecture checks.
• Build validation.
• Schema validation.
• Policy-as-code.
• Comparison against authoritative data.
• Human review.
• Human approval.
• Production monitoring.
• Rollback verification.

Ask one particularly important question: If a human does not inspect every action, how will we know when the agent is wrong? If the organization cannot answer that question adequately, the proposed autonomy level may be too high.

6. Design the Human Approval Boundary

Where human approval remains required, record:

• What decision the human is making.
• Why human judgment is necessary.
• What evidence the agent must present.
• Who is qualified to approve.
• What happens if approval is denied.
• Whether approval itself has become a material bottleneck.

Periodically examine approval results. If reviewers routinely catch significant problems, the boundary remains valuable. If they rarely intervene and independent validation is strong, the boundary may become a candidate for reevaluation.

7. Define Stop Conditions

Specify conditions under which the agent must stop rather than continue trying. Examples include:

• Scope expands beyond the assigned activity.
• Required validation repeatedly fails.
• Conflicting authoritative information is encountered.
• An architectural decision is required.
• A business decision is required.
• Additional permissions are needed.
• The next action would exceed the permitted consequence level.
• Recovery would become difficult.
• Resource or cost boundaries are reached.
• Required evidence cannot be produced.

The agent should not be rewarded simply for finishing. Correctly stopping can be the best engineering outcome.

8. Define Escalation

For each stop condition, determine where responsibility goes next. Record:

• Who receives the escalation.
• What information the agent must provide.
• What work has already been completed.
• What failed or remains uncertain.
• What decision is required from the human.
• Whether the agent may resume after the decision.

Design escalation so the human does not have to reconstruct the entire task before helping.

9. Assess Failure Consequence and Reversibility

For the proposed authority, ask:

• What is a plausible incorrect action?
• How quickly would it be detected?
• What systems or users could be affected?
• Can the action be reversed?
• How difficult is recovery?
• Could incorrect work propagate before detection?
• Is the potential consequence consistent with the authority being proposed?

Activities with low consequence, strong detection, and easy recovery are generally better candidates for greater autonomy than activities where rare failures are difficult to detect or reverse.

10. Establish Agent Observability

Ensure the organization can reconstruct meaningful agent behavior. Capture as appropriate:

• Objective received.
• Systems accessed.
• Tools invoked.
• Material actions performed.
• Changes produced.
• Validation performed.
• Failures encountered.
• Retries or alternative approaches.
• Escalations.
• Human approvals.
• Final outcome.
• Resource consumption where material.

Capture what is necessary to understand and improve the workflow rather than indiscriminately preserving every interaction.

11. Define the Next Authority Decision

Do not ask whether the agent can become fully autonomous. Identify the next meaningful increase in authority, if one would create value.

• Proposed additional authority.
• Engineering benefit expected from granting it.
• Evidence required.
• Validation that must exist.
• Failure conditions that must be understood.
• Required recovery capability.
• Human approval that would be removed or changed.
• Conditions that would cause authority to be reduced again.

If greater authority does not remove a meaningful engineering cost or constraint, leave the operating model where it is.

The AI Autonomy Map

For each activity, the map should contain:

• Engineering activity
• Current operating model
• AI objective
• Permitted actions
• Systems/tools available
• Validation
• Human approval boundary
• Stop conditions
• Escalation path
• Failure consequence
• Reversibility
• Current evidence
• Proposed next authority level
• Evidence required before expansion

This map should be maintained as the organization learns. It is not a one-time governance document.

What You Should Have When Chapter 4 Is Complete

You should now have:

• A common definition of the three AI engineering operating models.
• A clear distinction between AI capability and organizational readiness.
• Granular authority rather than all-or-nothing agent permissions.
• Independent validation associated with consequential AI actions.
• Designed human approval boundaries.
• Explicit stop and escalation conditions.
• Failure and reversibility analysis.
• Agent observability requirements.
• An AI Autonomy Map for significant agentic workflows.
• Evidence requirements for increasing—or decreasing—AI authority.

The most important change is that autonomy is no longer the goal; appropriate authority is the goal. An organization may ultimately have some engineering activities performed autonomously, others performed by AI with human approval, and still others where an engineer remains directly in control. That mixture is not evidence of an incomplete AI transformation. It is what we should expect from an engineering organization that deliberately applies different levels of authority to different kinds of work based on the evidence, risk, validation, and consequences involved.

Even a carefully bounded AI workflow, however, can create another problem. AI can produce code quickly, and agents can produce and modify even more of it. If the organization does not control the quality, consistency, and maintainability of what enters its systems, the same technology that accelerates development can also accelerate the accumulation of engineering problems. The next question, therefore, is not simply how much more AI can do, but whether the software being created today will still be understandable, maintainable, and affordable to change years from now.

• Introduction → AI engineering framework
• Chapter 1 → AI in software development
• Chapter 2 → AI developer productivity
• Chapter 3 → AI engineering standards
• Chapter 4 → AI agents in software development
• Chapter 5 → AI technical debt
• Chapter 6 → AI engineering context
• Chapter 7 → AI cost governance
• Chapter 8 → AI engineering operating model

If you have questions, please let us know.

Take a few minutes to complete the assessment and gain a clear, practical view of your organization’s AI readiness—and what to do next.