luismori.

Modern Agent Engineering

AI Engineering AI Agents Developer Productivity Product

How AI Helps Engineers Evolve and Scale Modern Apps

AI can help teams move faster, but the real unlock is designing agents, skills, evals, traces, and self-correction loops around your app.

15 min read

TL;DR

AI helps engineers scale modern apps when it is treated as part of the engineering system, not as a magic code generator. The most useful pattern is: give AI agents clear goals, reusable skills, safe tools, strong observability, evaluation datasets, and human review loops. Then use real failures from production to improve prompts, code, tools, policies, and tests over time.

The practical shift is this: instead of asking “Can AI write this feature?”, ask “Can our team design a system where AI can propose, test, observe, correct, and safely escalate work?”

What You Will Learn Here

  • How AI changes engineering work beyond autocomplete.
  • What agents are, and when they are worth building.
  • What “skills” mean in an AI engineering system.
  • How engineers and PMs can use AI to evolve modern apps without losing quality.
  • How to implement a self-correction loop with traces, evals, review, and regression tests.
  • Where AI still needs strong product judgment, engineering judgment, and human oversight.

The Real AI Shift Is Not Just Speed

AI coding tools are already mainstream. Google’s 2025 DORA research reported that AI adoption among software development professionals reached 90%, with 65% heavily relying on AI for software development work. More than 80% of respondents said AI improved their productivity, and 59% reported a positive effect on code quality.

That sounds like a pure acceleration story, but the more useful interpretation is subtler.

AI amplifies the system around it.

If a team has clear product goals, good architecture, clean documentation, reliable tests, and fast review cycles, AI can help that team move faster. If the team has vague requirements, missing ownership, fragile infrastructure, and no feedback loops, AI usually makes those weaknesses louder.

This is why modern AI engineering is less about “prompt harder” and more about improving the environment where AI works:

  • Clear specs and acceptance criteria.
  • Small tasks with testable success conditions.
  • Tools that expose real app state.
  • Evals that catch regressions.
  • Traces that show what the agent actually did.
  • Human review at important checkpoints.

For engineers, that means the job evolves from writing every line manually to designing a high-trust delivery system. For PMs, it means product definition becomes more important, not less. When implementation gets cheaper, choosing the right problem becomes more valuable.

Where AI Helps Engineers Scale Modern Apps

AI is useful across the whole app lifecycle, but different use cases need different levels of control.

1. Product Discovery and Planning

AI can help summarize customer interviews, extract user pain points, compare competitors, create first-pass specs, and generate edge-case checklists. This is especially useful when PMs and engineers are trying to turn messy context into a clear build plan.

Good AI-assisted planning does not replace product judgment. It gives the team more raw material to inspect.

Example prompts that tend to work well:

Read this feature brief and produce:
1. The user problem.
2. The riskiest assumptions.
3. Missing requirements.
4. Testable acceptance criteria.
5. Questions for engineering and design.

2. Architecture and Technical Design

AI can compare architecture options, generate sequence diagrams, identify failure modes, and pressure-test decisions. It is useful as a second reviewer when you ask it to argue against your design.

The key is to ground the model in real constraints: existing code, traffic patterns, latency targets, team skills, compliance requirements, and operational limits.

3. Implementation

This is the familiar use case: generate code, refactor code, write tests, scaffold APIs, migrate modules, and explain unfamiliar parts of a repo.

But the best teams avoid the “big vague prompt, giant diff” trap. They use AI in small loops:

Goal -> Small plan -> Edit -> Run tests -> Review diff -> Improve -> Merge

Small loops keep humans in control and make defects easier to find.

4. QA, Security, and Reliability

AI can generate test cases, inspect logs, explain failing CI, create threat models, and review pull requests for specific classes of bugs. It is especially useful when connected to tools: test runners, linters, static analysis, observability platforms, issue trackers, and deployment logs.

This is where agents become interesting. A QA agent that can open the app, reproduce a bug, inspect network calls, propose a fix, and rerun tests is doing more than chat. It is executing a workflow.

5. Operations and Continuous Improvement

Modern apps are never “done”. AI can help classify incidents, cluster support tickets, find repeated failure modes, update runbooks, and convert production failures into regression tests.

This is the bridge into self-correction systems: every production issue becomes structured learning for the app and its agents.

What Is an Agent?

A chatbot responds. An agent works.

More concretely, an agent is a system that uses a model to pursue a goal, choose tools, observe results, and continue until it reaches a stopping condition.

User goal
   |
   v
+---------+
| Agent   |
+---------+
   |
   +--> plan next step
   +--> call tool
   +--> observe result
   +--> decide whether to continue
   |
   v
Final answer, code change, ticket, report, or escalation

An app agent usually has five parts:

  • Model: The reasoning engine.
  • Instructions: The rules, persona, constraints, and workflow.
  • Tools: APIs, databases, browser actions, test runners, search, file access, and internal services.
  • Memory or context: User state, prior decisions, docs, traces, and relevant files.
  • Guardrails: Permissions, validation, policy checks, rate limits, and human approval.

OpenAI’s agent guidance frames the core pieces as model, tools, and instructions. Anthropic’s agent guidance adds an important architectural distinction: sometimes you want a workflow with fixed steps, and sometimes you want a more autonomous agent that can decide its own path.

That distinction matters.

If the task is predictable, use a workflow. If the task is ambiguous, tool-heavy, and requires judgment across changing context, consider an agent.

Workflows vs Agents

Not every AI feature should be an autonomous agent.

Use a workflow when the process is known:

Input -> classify -> retrieve data -> generate draft -> validate -> send to review

Use an agent when the process needs dynamic decisions:

Goal -> inspect context -> choose tool -> observe -> choose next action -> stop or escalate

For example:

  • A password reset assistant is usually a workflow.
  • A support investigation assistant that checks orders, policies, logs, and prior tickets may need an agent.
  • A code migration task across many files may need an agent.
  • A marketing copy generator probably does not need an agent unless it also researches, tests, publishes, and monitors results.

The engineering instinct should be conservative: start with the simplest reliable workflow, then introduce more autonomy only where the workflow becomes too rigid.

What Are Skills?

Skills are reusable task knowledge for AI systems.

A skill can be a short instruction, a checklist, a script, a tool wrapper, a policy, a prompt template, or a full workflow. The point is to turn repeated human know-how into something the agent can reliably reuse.

Think of skills as “onboarding documents for agents”.

Examples:

  • “How to run this repo locally.”
  • “How to write an API endpoint in our style.”
  • “How to debug failed payments.”
  • “How to create a safe database migration.”
  • “How to review AI-generated code.”
  • “How to convert a production trace into an eval.”

Skills help because they reduce ambiguity. Instead of asking the model to rediscover your team’s conventions every time, you give it stable operating knowledge.

+-------------------+
| Team Knowledge    |
+-------------------+
          |
          v
+-------------------+       +------------------+
| Skill Documents   | ----> | Agent Execution  |
+-------------------+       +------------------+
          |                         |
          v                         v
  prompts, scripts,          better plans,
  checklists, tools          fewer mistakes

A useful skill is:

  • Small enough to apply to a clear task.
  • Specific to your app, team, or workflow.
  • Written with steps and success conditions.
  • Connected to tools or commands when possible.
  • Updated when production reality teaches you something new.

A Simple Agent Pattern for Modern Apps

Here is a minimal TypeScript-style sketch. The important idea is not the exact SDK. It is the loop: decide, act, observe, validate, and stop.

type ToolResult = {
  ok: boolean;
  data?: unknown;
  error?: string;
};

type AppTool = {
  name: string;
  description: string;
  run: (input: unknown) => Promise<ToolResult>;
};

const tools: Record<string, AppTool> = {
  searchDocs: {
    name: "searchDocs",
    description: "Find relevant internal documentation.",
    run: async (input) => searchDocs(String(input)),
  },
  runTests: {
    name: "runTests",
    description: "Run the focused test suite for the current change.",
    run: async () => runCommand("npm test -- --runInBand"),
  },
};

async function runEngineeringAgent(goal: string) {
  const trace = [];
  let state = { goal, done: false, attempts: 0 };

  while (!state.done && state.attempts < 6) {
    const decision = await modelDecideNextStep({
      goal: state.goal,
      availableTools: Object.values(tools).map(({ name, description }) => ({
        name,
        description,
      })),
      trace,
    });

    const tool = tools[decision.toolName];
    if (!tool) {
      return escalateToHuman("Unknown tool requested", trace);
    }

    const result = await tool.run(decision.input);
    trace.push({ decision, result, at: new Date().toISOString() });

    if (!result.ok) {
      state.attempts += 1;
      continue;
    }

    const validation = await validateProgress(goal, trace);
    state.done = validation.done;
    state.attempts += 1;
  }

  if (!state.done) {
    return escalateToHuman("Agent reached max attempts", trace);
  }

  return {
    status: "ready_for_review",
    trace,
  };
}

Notice the boring but important parts:

  • The agent has a max number of attempts.
  • Every step is traced.
  • Unknown tools are blocked.
  • Progress is validated.
  • The final status is “ready for review”, not “magically shipped”.

That is the posture that keeps agentic systems useful.

The Self-Correction Loop

Self-correction does not mean the app silently rewrites itself in production. That is a dangerous fantasy.

In a mature engineering system, self-correction means the product creates evidence, the team converts that evidence into tests and improvements, and agents help perform the work under guardrails.

Production use
     |
     v
Traces, logs, feedback, failures
     |
     v
Human and automated review
     |
     v
Failure patterns
     |
     v
New evals and regression tests
     |
     v
Prompt, tool, policy, or code changes
     |
     v
CI gate and human review
     |
     v
Deploy
     |
     v
Observe again

This is how an AI-enabled app gets better without pretending the model is always right.

Step 1: Capture Traces

For normal software, logs tell you what happened. For agents, traces tell you how the agent got there.

A trace should capture:

  • User input or task goal.
  • Retrieved context.
  • Tool calls and tool results.
  • Intermediate decisions.
  • Final output.
  • Latency, cost, and errors.
  • Human feedback or review labels.

LangChain’s 2026 guidance puts this clearly: code tells you what an agent is allowed to do; traces tell you what it actually did in a specific run.

Step 2: Add Evaluations

Evals are tests for AI behavior. They can be deterministic checks, human labels, LLM-as-judge rubrics, golden datasets, or full scenario tests.

Use evals for questions like:

  • Did the answer follow policy?
  • Did the agent call the right tool?
  • Did it avoid unsafe actions?
  • Did it solve the user’s real problem?
  • Did it preserve required data?
  • Did it produce code that passes tests?

OpenAI’s evals guidance describes evaluations as the process of validating and testing LLM application outputs. The practical engineering version is simple: if you cannot measure whether a change made the agent better, you are guessing.

Step 3: Turn Failures Into Datasets

Every repeated failure should become a reusable test case.

Bad production run
     |
     v
Reviewer labels root cause
     |
     v
Create test input + expected behavior
     |
     v
Add to offline eval suite
     |
     v
Run before future releases

This is where PMs and domain experts are extremely valuable. They can label whether the agent solved the actual user need, not just whether the response looked good.

Step 4: Choose the Right Fix

Not every AI failure is a prompt problem.

Use this map:

Failure type                 Likely fix
--------------------------   --------------------------------
Missing product context      Improve retrieval or documentation
Wrong tool selected          Improve tool descriptions or routing
Unsafe action attempted      Add permissions or human approval
Bad final answer             Add evals, examples, or clearer rubric
Repeated edge-case miss      Add a skill or workflow branch
Code regression              Add tests and tighten review gates
High cost or latency         Route simpler tasks to cheaper models

The fix might be a prompt update, but it might also be a better tool, a smaller workflow, a stricter permission boundary, or a product decision.

Step 5: Keep Humans in the Loop Where It Matters

Human review is not a failure of automation. It is part of the system design.

Require approval for:

  • Writing to production systems.
  • Sending external messages.
  • Handling money, identity, privacy, or security.
  • Changing access controls.
  • Publishing user-visible content.
  • Deploying code.

Over time, you can reduce review for low-risk, high-confidence actions. But earn that autonomy with evidence.

A Practical Self-Correction Backlog

If you are designing a new app today, add this backlog from day one:

  • Trace storage: Keep structured records of agent runs.
  • Feedback capture: Let users and reviewers label bad outputs.
  • Eval dataset: Save real examples with expected behavior.
  • Regression gate: Run evals in CI before prompt, model, or workflow changes ship.
  • Tool permissions: Separate read tools from write tools.
  • Escalation rules: Define when the agent must stop and ask a human.
  • Skill registry: Store reusable workflow instructions near the code.
  • Release notes for agents: Track prompt, model, tool, and policy changes like code changes.

This makes the app easier to improve because the system remembers what it learned.

How Engineers and PMs Should Work Together

AI-assisted engineering works best when PMs and engineers share ownership of quality.

PMs can help define:

  • User goals.
  • Acceptable outcomes.
  • Edge cases.
  • Business rules.
  • Escalation moments.
  • What “good” means in an eval rubric.

Engineers can help define:

  • System boundaries.
  • Tool contracts.
  • Observability.
  • Test strategy.
  • Security and permissions.
  • CI and deployment gates.

Together, they create a better operating system for AI.

PM judgment                Engineering judgment
-----------                --------------------
What matters?              How should it work?
What is risky?             How can it fail?
What is good enough?       How do we verify it?
When should a human step in? What should be automated?

Where Teams Get AI Wrong

The common mistakes are predictable:

  • Treating AI output as finished work.
  • Asking for huge changes without intermediate checks.
  • Measuring only speed, not quality or rework.
  • Building multi-agent systems before a single-agent workflow is reliable.
  • Giving agents too many overlapping tools.
  • Skipping traces because “we have logs”.
  • Shipping prompt changes without regression evals.
  • Ignoring PM and domain expert feedback.

The METR 2025 study is a useful caution here. In one randomized controlled trial with experienced open-source developers, AI tools slowed participants down on realistic tasks, even though participants believed AI had helped them move faster. That does not mean AI is useless. It means productivity is context-dependent, and self-reported speed is not enough.

A Good First Implementation Plan

If I were adding AI to a modern app from scratch, I would start here:

  1. Pick one painful workflow with clear business value.
  2. Write the human workflow as steps.
  3. Add an AI assistant for the narrowest useful part.
  4. Give it read-only tools first.
  5. Capture traces from every run.
  6. Review the first 50 to 100 real runs manually.
  7. Convert common failures into evals.
  8. Add write actions only behind approval.
  9. Measure speed, quality, user satisfaction, rework, and incident rate.
  10. Expand autonomy only after the eval suite and trace data support it.

The goal is not to create a futuristic demo. The goal is to create a learning system.

Gaps and Sections to Add Later

This article intentionally stays framework-neutral. A stronger follow-up could go deeper into implementation choices:

  • A concrete OpenAI Agents SDK or Vercel AI SDK implementation.
  • A LangSmith, LangWatch, or custom tracing setup.
  • A sample eval dataset for support, code review, or QA agents.
  • A security model for read/write tool permissions.
  • A PM-friendly rubric for evaluating AI product quality.
  • A real CI pipeline that blocks agent regressions before deploy.

The biggest open gap is measurement. Teams still need better ways to connect AI usage to durable product outcomes: less rework, fewer incidents, faster learning, better retention, and higher customer satisfaction. Lines of code and prompt counts are not enough.

Final Thought

AI helps engineers scale modern apps when we design the system around it.

Agents need tools. Tools need permissions. Permissions need guardrails. Guardrails need traces. Traces need evals. Evals need human judgment. Human judgment needs product context.

That is the modern loop.

The teams that master it will not just ship faster. They will learn faster.

Sources