AI Agent Operations and Maintenance: Monitoring, Updates, Support
・ Employee Store Operations
Summary
Even after delivery, an AI agent's behavior can change when an AI model is retired or a connected service changes. This article explains what tends to happen after delivery, what to monitor, how to run updates, and how to set the support scope and monthly terms, based on public documents and each company's official documentation.
Deciding AI agent operations and maintenance before delivery makes later communication easier. If no one has decided who responds when something happens, even a small defect takes time to sort out with the client. This article explains, in order, what to decide, from the view of the party that builds and operates AI agents.
The content of this article was checked on October 2, 2026 against public documents from Japan's Ministry of Internal Affairs and Communications (MIC) and Ministry of Economy, Trade and Industry (METI), and against each company's official documentation. Company policies may change. Before using it, check the latest content through the sources at the end.
What tends to happen after delivery: model updates, API changes, data changes
Even if it worked correctly at delivery, outside changes can change the results. There are three main causes.
AI model retirement
AI model providers retire older models over time. OpenAI says it gives at least 6 months' notice for generally available models and at least 3 months' notice for specialized derivative models. It says models with preview in their name may be retired with shorter notice, around 2 weeks. Anthropic says it gives at least 60 days' notice for its publicly released models. Anthropic's documentation explains that exporting usage from the Claude Console shows usage by API key and model. This helps you find where you use a model that has a retirement notice. Keep a list of which models each delivered AI uses.
Features can be retired too, not just models. OpenAI plans to retire custom GPTs (GPTs) on December 11, 2026. For Enterprise workspaces approved for an extension, the date is February 11, 2027.
Changes to connected services' APIs and terms
METI's “Contract Checklist for the Use and Development of AI” notes that terms of use may change periodically and asks you to check for changes as needed. For services built on general-purpose AI services, it says you may need to change your own service's terms to match changes in the underlying service's terms.
Changes in data
When the client's work or data changes, the same system can produce different results. The “AI Guidelines for Business (Version 1.2)” from MIC and METI ask AI providers to regularly verify, even after provision, that the AI is being used for appropriate purposes.
What happened
What to monitor and logs
To make verification possible, the AI Guidelines for Business list recording and storing logs, such as inputs and outputs during use, within a reasonable scope. They say the recording method, frequency and retention period should be set based on whether they are needed to investigate the cause of incidents and prevent recurrence.
| Item to monitor | What to look at | Example features |
|---|---|---|
| Execution success or failure | Number of failed executions, and where they stopped | n8n execution list (filter by failed, running, succeeded, waiting), error workflows |
| Response time and cost | Time and token volume per run | Dify logs (token usage and response time per message) |
| Output quality | User ratings, examples of errors | Ratings and comments in Dify logs, Langfuse evaluations |
| Alerts when thresholds are exceeded | Whether failures or costs exceeded set values | n8n error workflows, Langfuse alerts |
In n8n, you can set an error workflow for each workflow. When an execution fails, a separate workflow that starts with an Error Trigger runs and can send alerts by email or Slack. In Dify logs, you can see node-level execution records for each workflow run, and find which node failed or slowed down.
Langfuse is an open-source AI engineering platform. It records AI model calls, retrieval and API calls, and shows quality, cost and response time on dashboards. It also has an alert feature that sends notifications through Slack, Webhooks and other channels when a metric exceeds a set value. Its documentation includes steps for connecting to n8n and Dify.
Catching failures is not enough for AI agent monitoring. The AI Guidelines for Business also list regularly evaluating AI model inputs, outputs and the basis for decisions, and monitoring for bias. Set a frequency, such as once a month, pull output examples from logs, and look for patterns of bias or errors.
Logs may contain the client's company information or personal information. Dify's documentation also warns that logs include full conversation text and may contain confidential information. Agree with the client on which logs to keep, where, and for how long, and write it in the contract.
Update steps and how to notify
For updates, do not change what is running in production directly. Test first, then swap it in.
- 1Learn of the changeProvider notices, monitoring results
- 2Fix it in a test environment
- 3Compare on evaluation examples
- 4Tell the client what and why
- 5Apply to production
- 6Keep the option to roll back
The AI Guidelines for Business list providing information about the content of and reasons for updates when an AI system is updated. Send the client a short note on when, what and why you changed something. If the update changes output tendencies, say so too. The guidelines also list checking the latest risk trends, such as attack methods, after provision, and considering responses to vulnerabilities. Include security fixes in the scope of updates.
Tools also have features for rolling back. Dify Chatflow and Workflow record published versions and let you load an earlier version back into the draft. n8n also lets you view a workflow's change history and restore an earlier version. How to build evaluation examples is covered in How to evaluate AI agents.
How to set and write the support scope
Listing what is in scope and out of scope side by side makes it easier for the client to judge.
Examples in scope
- Investigating and fixing defects in the delivered system
- Switching models when an AI model is retired
- Responding to API changes in connected services
- Answering questions about usage
Examples out of scope
- Adding new features
- Expanding the target tasks
- Defects caused by settings the client changed
- AI model and service usage fees
Besides the scope, also write the following: how and when you accept inquiries, the target response time, the contact for emergencies, and how out-of-scope work is priced. Also decide how far errors in AI output count as defects. Using results measured on the evaluation examples set in the requirements document as the standard helps avoid disagreements. How to set this in the contract is covered in How to run AI agent development projects.
Terms for offering maintenance as a monthly plan
If you offer operations and maintenance as a monthly plan, write what the monthly fee includes.
- Included work: monitoring, answering inquiries, fixes within the agreed scope
- Excluded costs: AI model usage fees and server costs, and whose account pays them
- Work limit: a guide to the number of cases or hours handled per month
- Reporting: what, how often and how you report
- Cancellation: how far in advance to give notice, and how data and workflows are handled after cancellation
If it runs on your API key, your costs rise as usage grows. If it runs on the client's API key, tell them from the start that those costs fall on them.
Communicating in the Employee Store transaction room
On Employee Store, buyers and sellers exchange messages on a page for each transaction (the transaction room). Checking deliverables, confirming receipt, ratings and reports also happen in the transaction room. Sending defect reports and update notices as transaction room messages keeps them together as a record of that transaction.
Pricing can be a one-time purchase, a monthly plan, or an upfront fee plus a monthly plan. Payments are processed by Stripe. The buyer cancels a monthly plan from contract management in the transaction room, which stops the next charge. There are no prorated refunds. If you list on a monthly plan, write the support scope included in the monthly fee on the listing page. For how to write it, see How to write an AI agent description. For the transaction flow, see How to use Employee Store.
FAQ
- Should switching models after an AI model retirement be included in the maintenance fee?
- There is no fixed rule. Before delivery, decide whether switching due to retirement is within the support scope, and write it in the contract or on the listing page. OpenAI says it gives 6 months' notice or more for generally available models, and Anthropic gives 60 days' notice or more for publicly released models.
- How long should logs be kept?
- There is no uniform rule. The AI Guidelines for Business say the recording method, frequency and retention period should be set based on whether they are needed to investigate the cause of incidents and prevent recurrence. If logs include personal information, agree on the period with the client and write it in the contract.
- Is a monthly plan or pay-per-request better for operations and maintenance?
- If monitoring and inquiries come up every month, a monthly plan fits. If there are only occasional fixes, pay-per-request fits. If you choose a monthly plan, write down the included work and the limit.
Sources
- Ministry of Internal Affairs and Communications, “AI Guidelines for Business” page (Japanese)
- Ministry of Internal Affairs and Communications and Ministry of Economy, Trade and Industry, “AI Guidelines for Business (Version 1.2)” main text (March 31, 2026) (Japanese)
- Ministry of Economy, Trade and Industry, “Contract Checklist for the Use and Development of AI” (February 2025) (Japanese)
- OpenAI, “Deprecations” (checked October 2, 2026)
- Anthropic, “Model deprecations” (checked October 2, 2026)
- OpenAI Help Center, “Custom GPT retirement and migration FAQ” (Japanese)
- n8n Docs, “Handle errors gracefully”
- n8n Docs, “View all executions”
- n8n Docs, “View change history”
- Dify Docs, “Logs” (Japanese)
- Dify Docs, “Version Control” (Japanese)
- Langfuse Docs, “Overview”
- Employee Store, “User Guide” (Japanese)
