Once a company begins adopting AI Agent, managers quickly receive a stream of easy-to-count metrics: how many people logged in, how many tasks were run, how many tokens were used, and how many documents were generated. These numbers show whether employees are using the tool, but they cannot answer the more important question: Has the business problem the company set out to solve actually improved?
We believe AI Agent usage metrics are useful for managing costs and understanding usage patterns, but they should not be treated as adoption outcomes. Companies should first define verifiable business outcomes, then review whether spending, execution records, and human approvals comply with requirements. If this order is reversed, teams may focus on increasing usage while failing to improve the work itself.
Why doesn't higher token usage mean better results?
On October 27, Amazon released updated role guidelines for software development engineers. The guidelines state that engineer performance should continue to be measured by actual work outcomes, rather than directly by AI tool adoption rates, token usage, or lines of code. These figures may be related to productivity, but they do not directly prove that the product has improved, the business problem has been solved, or the quality of the deliverable meets requirements.
Amazon made this change in response to a specific incident. In May, Amazon employees created KiroRank, an internal leaderboard that ranked employees by their token usage in the Kiro AI development tool. To improve their rankings, some engineers had AI Agent run unnecessary tasks, which eventually prompted Amazon to shut down the leaderboard. Once a company turns usage into a performance target, employees have a natural incentive to consume more tokens or produce more code. The result may be higher costs and greater system complexity.
The same problem can arise in enterprise workflows. Having AI Agent draft more customer service responses does not mean customer complaints have been resolved. Generating more procurement comparison documents does not mean the buyer has found an option that meets the required specifications and delivery schedule. A manager's first question should therefore be: "Which work outcome were we originally trying to improve?" Usage should only be reviewed after that, so easily counted activity metrics are not mistaken for actual results.
Define one verifiable business outcome for each project
A company can begin with a clearly scoped process that managers already know how to evaluate, then define one primary outcome for the AI Agent project. For sales quotations, the company could check whether all required information is included and whether the quotation is delivered within the promised timeframe. For procurement comparisons, it could verify that the specifications, prices, and delivery schedules are supported by traceable sources. For customer service, it could check whether the issue has been closed and whether cases requiring human intervention were properly handed off. Business managers must define these conditions first, after which IT and the project team can convert them into acceptance methods that produce auditable records.
In addition to the primary outcome, the team should record supporting data to understand why results improved or where problems occurred. Recommended data includes task start and completion times, reasons for human rejection, the number of reruns, token usage or cost, and the identity of the final approver. This information should explain the outcome, not become a set of performance targets where higher is always better. For example, if delivery times decrease but human rejections increase, managers should investigate whether AI Agent is merely producing noncompliant content faster.
The evaluation also needs a baseline for comparison. Before the project begins, record delivery results and common reasons for rejection over a defined period using the current process. After launch, compare results using the same definitions to avoid applying different standards before and after implementation. If reliable historical data is unavailable, have the old and new processes handle comparable work, then ask the same group of managers to review the results under the same rules. The project owner should now select one process and clearly document the outcome name, acceptance criteria, data sources, and designated approver.
Review outcomes, costs, and accountability together
During each monthly review of an AI Agent project, the business manager, IT, and project owner should jointly confirm three points: whether the business outcome has been achieved, whether costs remain within the approved limit, and whether the execution process can be traced and verified. If the outcome has not improved, the approach should be adjusted or discontinued, even when usage is high. If the outcome is achieved but costs continue to rise, the team should check for unnecessary reruns or determine whether expensive methods are being used for simple tasks.
Situations that require human judgment must also be explicitly built into the workflow. For example, a quotation workflow should pause and wait for approval from a designated manager if the amount exceeds the department's limit, the terms are unclear, or the quotation is about to be sent to the customer. If the existing AI Agent cannot improve a particular type of work, employees should also be able to record the reason and return the task to a manual process. A low adoption rate should not automatically be treated as poor performance. Managers must first determine whether the tool is unsuitable, the available data is insufficient, or employees have not yet learned how to use it.
EgentWrX allows companies to set tiered cost budgets by tenant, business unit, member, and AI Agent. Once the budget at any level is exhausted, the system can block further usage. It also maintains exportable audit logs and can verify the integrity of their contents. Workflows can include human approval gates, allowing AI Agent to complete the initial work and then pause until a manager authorizes the next step. These mechanisms do not decide what success means for the company, but they help managers review outcomes, spending, and accountability together. As a next step, the company can schedule a monthly review meeting and clearly define the conditions for continuing use, adjusting the approach, or discontinuing use.
FAQ
Can AI Agent adoption success be measured solely by time saved?
Measuring time alone is not recommended. Even when processing becomes faster, the company still needs to verify whether deliverables meet specifications, whether human rejections have increased, and whether the work was actually completed. Otherwise, the team may simply be producing content faster that still needs to be redone.
Does token usage have no management value at all?
Token usage does have management value. It is useful for tracking costs, identifying unusual increases in consumption, and comparing the cost of different approaches. However, it should not be used directly as a performance target. Managers should review token usage alongside business outcomes, rerun counts, and approval records.
Can adoption success still be evaluated without reliable historical data?
Yes. The company can establish a short-term comparison baseline. Have the current and new processes handle comparable work, apply the same acceptance rules, and ask people in the same roles to review the results. The important point is to use consistent definitions before and after implementation rather than changing the measurement criteria only after launch.
What should the team do when AI Agent is unsuitable for a particular task?
The team should document why AI Agent is unsuitable and allow the workflow to return to manual processing. Managers can then determine whether the problem lies with the data, the tool's capabilities, or the nature of the work itself. Employees who provide a reasonable explanation should not be considered poor performers simply because their adoption rate is low.
References
- Amazon Revises Software Engineer Role Guidelines, Emphasizing Outcomes over Token Usage in the AI Era — iThome's summary of Amazon's updated engineer role guidelines and principles for measuring AI-related performance.
