A production planner understands that just because a production order needs to move forward, it’s not necessarily as simple as just having an available machine to do so. There will be material in place, correct tooling and potentially a changeover needed from the last batch. Quality checks will need to be done. A seemingly efficient decision made on the schedule board could cause delays or quality issues further down the line. An AI agent has been tasked with reducing downtime and improving output. It notices a production line is not working and checks the schedule. It takes some action to change this. Whilst this may be technically correct, within the AI systems capabilities and aligns with what it’s been given to do, it may not be the correct decision for the factory.
This is the issue manufacturers have to face. As we move from recommendation based AI solutions to operation based AI solutions, they don’t understand whether the action they’re carrying out is appropriate for the plant. They can follow an instruction correctly. The issue becomes how much autonomy should be given to AI systems, and how to limit them.
A factory decision is rarely an isolated decision
Traditionally a manufacturing system will consist of many parts, each performing separate roles. There will be an Enterprise Resource Planning (ERP) system to handle ordering, stock and business resources. A Manufacturing Execution System (MES) to monitor production. Machines and industrial control systems. Then there will be supervisors, planners, maintenance engineers and operators looking at the data, resolving issues and making decisions.
But with AI agents, these steps can be seen in a fluid process. If an issue is detected by an AI agent it can look up maintenance history, check if spares are available, create a work order and talk to the production planners. This can help greatly to increase how quickly an issue can be resolved by multiple people and systems. But this doesn’t mean that the outcome will be better.
By carrying out some maintenance to improve the availability of a piece of machinery they may cause a delay to a critical customer order. By changing the production sequence they may reduce downtime but increase stock waste and changeover losses. By making a decision on stock they may cause the production line to be vulnerable to an unexpected shortage.
This isn’t necessarily down to incorrect information or the AI not reasoning correctly, it’s down to the fact that manufacturing decisions often involve conflicting goals. The National Institute of Standards and Technology (NIST) roadmap on AI and machine learning in smart manufacturing indicates that integrating complex industrial systems and trusted and reliable AI are key barriers. This means that the intelligence of industry needs to be measured based on how well it can advise, not just how good the advice is.
The AI agent shouldn’t be allowed to optimise an activity at the expense of the overall factory performance.
Human approval is not enough if the consequences remain invisible
Human in the loop is well recognised as an important consideration. However, if not done correctly, it can become ineffective. Imagine that an AI makes a recommendation that several batches need to be rescheduled, resulting in better machine utilization. However, there is no information provided about how this will affect availability of materials, setup times, effects on downstream processes and customer delivery commitments.
Whilst the production manager has the final say, they will be making a largely administrative decision as they wouldn’t understand the effects of this. To effectively have human oversight, the individual needs to know why an action is being recommended, what limitations have been considered, what assumptions have been made and what could go wrong.
This also means that manufacturers should consider how they measure AI performance. A scheduling agent should not only be measured on how much quicker it makes the planning process, but how stable the schedule is, quality results, availability of materials and how often decisions need to be made.
If a maintenance agent is providing accurate predictions but consistently recommending actions that are not appropriate from a commercial or operational standpoint, it can not be seen as fully effective. The aim isn’t to eliminate human involvement, but ensure that decisions are left up to a human where competing priorities, uncertainty or operational consequences make that judgement necessary.
The real risk emerges when one agent’s action becomes another system’s instruction
The increased usage of Application Programming Interface (APIs) and Model Context Protocol (MCP) can help AI agents easily interact with enterprise and integrated systems. However, there are increased risks due to the ability to pass actions between systems without considering how they impact the constraints of plant operation.
If an agent identifies a possible failure and requests maintenance activity then this could cause a request for spare parts, change production schedules and provide new delivery times. Each step may be acceptable, but the overall effect may disrupt operations in a way that none of the individual systems were expecting.
Manufacturers therefore need to consider the full series of actions. There needs to be defined limits to what the agent can do (systems it can interact with, actions it can request and when the action needs to cease and be passed back to a person).
Also, for any safety critical equipment/processes, the current validated systems and controls should be the authoritative source. Just because a language model can understand instructions does not mean it should replace deterministic machine control.
This is important as implementing AI within industry is not just another piece of enterprise software. The actions the AI will perform will ultimately affect processes, assets and people.
Accountability begins with the design of the decision
If an issue occurs due to an action being performed by an AI you will benefit from having an audit log to see what system has taken the action. However you still need the manufacturer to be able to understand why the action was allowed to take place. Why was there certain operational data available? Was the constraint considered? And who gave the agent permission to act at that level?
An example of this could be if an agent changes a batch because it has incorrect inventory data. Was the issue that the agent made an incorrect interpretation? The data was incorrect? Or there wasn’t a validation process? Or the agent had too much permission given to it?
Depending on the issue you will need to make sure the correct actions are taken to prevent this happening again. This helps share responsibility. The technology provider needs to prove that the system enforces permissions, handles uncertainty and captures decisions. The manufacturer needs to decide what processes can be passed to the agent, what level of risk is acceptable and ensure they have the correct authority and data to intervene.
Human approvals should have clear responsibilities but shouldn’t automatically be blamed for issues caused by the system design, inaccurate data or lack of controls. A useful framework for accountability should allow you to understand why the agent took the action and why the system allowed it to take place.
India’s manufacturing opportunity lies in disciplined autonomy
As India looks to enhance its manufacturing capabilities, it needs to consider increasing productivity, quality, responsiveness, and complexity of processes. AI agents can help to reduce the lag time between problem detection and coordination of actions.
However, manufacturers should not aim to increase autonomous workflows. Instead manufacturers should look at whether the workflow has improved the output of the process using AI. This should ensure that there is no undue risk or cost incurred elsewhere in the process.
Start by clearly defining where problems exist in the process, having good data feeds, decisions that can safely be made autonomously. Finally testing the AI under scenarios where assumptions about the process may not hold true. A factory should understand what will happen if the inventory data is delayed, the machine changes state unexpectedly, there are conflicting instructions or the agent cannot confirm key conditions. These are not unusual questions to ask but will determine whether the system is ready to be used.
The most valuable AI systems will not be those that can make the most decisions autonomously but instead those that know their boundaries, understand the dependencies and know when to hand back control to people. The future of intelligent factories will not be determined by how many decisions the machines can make but how accurately they can determine which decisions they can and should make.
