Last week, a salesperson told an AI Agent: “For quotes to this type of customer, deduct the annual rebate first, and ask a manager to confirm any lead time longer than six weeks.” When the AI Agent was asked to process another quote this week, however, it did not apply the same rules. This does not necessarily mean the model forgot them. The critical information may have been buried in a long conversation history, leaving the AI Agent unable to find it when it was actually needed.
When enterprises adopt AI Agents, they often assume that once an AI Agent has “seen” something, it will follow the same guidance next time. We believe conversations are best suited to the task at hand, personal preferences can be stored in controlled memory, and information that must be verified should be placed in a knowledge base. Company rules that must always be followed, meanwhile, should be organized as skills or task instructions. Information should not be classified by length alone. First determine how it will be used and who would be affected if it were wrong.
Conversation History Preserves Context but Should Not Store Enterprise Rules
OpenAI’s LifeVal benchmark includes 169 tasks spanning caregiving, education, shopping, personal finance, government services, and other domains. The research team had AI use a browser and operate a computer to complete tasks such as finding a preschool, organizing flood-prevention volunteers, and searching for housing. They then assessed whether the required actions had been completed and whether the results met the user’s needs.
The findings were clear: where background information is stored affects whether AI can find it when needed. Evaluation criteria that required information from conversation history achieved an average success rate of 33.33%. When the research team moved background information that had originally been provided directly into conversation history, the corresponding success rate fell from 55.8% to 41.5%. However, LifeVal was primarily tested in simulated environments, and the full task dataset, evaluation code, and model used for the personalization analysis were not made public. Enterprises therefore cannot apply these figures directly to estimate the performance of their own production systems.
Conversation history remains useful for recording how far a customer complaint has progressed, which options procurement is comparing, or what temporary conditions were added during a quotation process. This information is closely tied to the current task and may become less important once the work is complete. Discount thresholds, approval requirements, quality-control criteria, and standard output formats are different. Enterprises cannot rely on an AI Agent to search through old conversations for them.
Start with one frequently performed process and identify each company rule within it. Then ask: “If a different employee performs this task next month, must the AI Agent still follow this rule?” If the answer is yes, move the rule out of the conversation and organize it as a skill or task instruction, with a clearly assigned owner and review process.
Content That Requires Documentary Evidence Belongs in a Searchable Knowledge Base
Policies, operating procedures, product specifications, and supplier information all have authoritative sources. Users also need to know which document supports an answer. This type of content belongs in a knowledge base rather than being scattered across different conversations. In procurement comparisons, for example, the AI Agent can receive an instruction such as “prioritize lead time in this comparison” within the current conversation. Payment terms, acceptance requirements, and approved supplier lists, however, should be retrieved from official sources.
The research team also found that some evaluation criteria labeled as requiring conversation history—and achieving a 100% success rate—could in fact be answered using other data or the conversation title. In other words, an AI Agent completing a task correctly does not prove that it consulted the designated source. If enterprise acceptance testing checks only the final answer, it may overlook a significant risk: the AI Agent may have arrived at the correct answer by chance and could continue citing the wrong source after the underlying data changes.
Acceptance testing can be divided into two layers. The first checks whether the task was completed—for example, whether a quote was generated. The second verifies which policy and version the answer cited, and whether the AI Agent stops answering when it cannot find sufficient evidence. Test cases should deliberately include questions that the knowledge base cannot answer, confirming that the AI Agent will not invent content simply to complete the task. At this stage, begin with one currently valid policy document and one obsolete version. Test whether the AI Agent cites the correct version and clearly explains when supporting evidence is unavailable.
Memory Is Only Appropriate for Personal Context—and Users Must Be Able to View and Edit It
Memory is suitable for personal information that can be reused across tasks, such as a manager’s preferred report format, the product line assigned to a salesperson, or a user’s preference for reviewing a summary before the details. In LifeVal, evaluation criteria requiring the use of ChatGPT memory achieved an average success rate of 47.8%. However, the study did not disclose which model was used for the personalization analysis. These results cannot be used directly to estimate success rates in enterprise environments, but they are sufficient to highlight an important requirement: users must be able to view and correct stored memories.
Enterprises should first define which types of information may be written to memory. General preferences can be stored directly. Work-related context that may affect departmental operations should be reviewed first. Sensitive information or content that could alter an approval outcome should require authorization from designated personnel. Users must also be able to view, modify, and delete incorrect memories, while administrators should assign someone to conduct periodic reviews. “Prefers Traditional Chinese” can be treated as a personal preference, but “this supplier is exempt from acceptance inspection” affects a formal process and must not be written directly to memory merely because it was mentioned in a previous conversation.
EgentWrX provides mechanisms for knowledge bases, memory, and skills. Documents can be organized as wikis, vectors, or tables, and the system can refuse to answer when it cannot find sufficient evidence. Users can view, modify, or delete memories, while administrators can configure policies that allow direct storage, require review, or require authorization. Standard practices can be organized as skills. If an employee-created skill is to be rolled out across the company, it must first pass an administrative review. We recommend starting with one frequently performed process and labeling each piece of information as current-conversation context, personal memory, formal knowledge, or a reusable skill. The process owner should then validate the classification and confirm how exceptions will be handled.
FAQ
Is it most reliable to paste the full context into every conversation?
Providing the context required for the current task directly is easier to verify than expecting an AI Agent to search through old conversations, but there is no need to paste all information every time. Include only the conditions relevant to the current task, organize standing rules as skills, and place content that must be cited in the knowledge base.
What information should not be written to memory?
Sensitive data, unapproved judgments, and information that directly changes pricing, eligibility, or approval outcomes should not be written to memory automatically. Enterprises should establish storage policies that require review or authorization and allow users to view, modify, and delete memories.
What is the difference between a knowledge base and a skill?
A knowledge base answers “What is the supporting source?” A skill defines “What should be done every time?” Product specifications, policy documents, and lists belong in the knowledge base. Quotation procedures, approval thresholds, output formats, and exception-handling rules should be organized as skills or task instructions.
How can we verify that an AI Agent actually consulted the designated source?
In addition to reviewing the answer, acceptance testing should verify the cited source and include scenarios in which other files contain similar answers or the knowledge base contains no answer at all. If the AI Agent still produces an answer when no supporting information can be found, the source of that answer must be investigated. A correct final result alone is not sufficient grounds for approval.
References
- OpenAI Reveals Personalization Bottlenecks for AI Agents, as Models Still Struggle to Use Past Conversations — iThome’s overview of the findings and limitations of OpenAI’s LifeVal research.
