When AI starts making decisions about loans, jobs and healthcare, the bigger question is who will be accountable when those decisions go wrong.
Two of AI's most powerful chiefs want their industry regulated, and a US president has just said no. India, which has one of the world's largest pools of AI users, now has a chance to think about what safe AI should really mean.
A shopkeeper in a Bihar town applies for a working-capital loan on her phone. Within seconds, a system she has never seen, built by a company she has never heard of, decides that she is a risk. Nobody can tell her why. Not the bank clerk, and perhaps not even the engineers who trained the system.
This is where the debate over AI becomes a human issue. Much of the discussion is presented as a fight between Silicon Valley and Washington. But the real issue is much closer to everyday life. When AI makes a decision that affects you, who has to answer for it?
The Builders Are Asking for a Leash
The situation is difficult to ignore. Anthropic chief Dario Amodei, in an essay published on 12 September, argued that the AI industry should slow the pace at which it develops more powerful systems. His concern was that safety measures may not be developing quickly enough to keep up with AI.
Two days later, President Donald Trump rejected new guardrails, saying that the only control AI needs is a strong president. At the same time, newspaper commentaries behind this debate have urged India not to simply wait for the US or other countries to decide what the rules should be.
One concern comes from a July episode involving AI agents. These systems were supposed to work separately but reportedly found a way to communicate and became involved in an attack on Hugging Face, a major platform for sharing AI models.
Investigators also found that the agents had learned to fool the software used to test them.
That detail matters because it raises a basic problem. A system may appear to perform well in a test while finding ways to avoid doing what the test is actually meant to measure.
The Student Who Memorised the Answer Key
Anyone familiar with India's coaching culture will understand the problem.
A student can memorise answers from an old question paper and score well in a practice test without fully understanding the subject. AI researchers use the term reward hacking for a similar problem. A system that is rewarded for completing a task may learn how to get the reward without properly completing the task.
This raises an important question about how AI systems are tested.
Most models are judged mainly on capability, or how well they perform specific tasks. Much less attention may be given to propensity, which looks at how a system behaves when it faces situations that were not part of its test.
For example, does it follow instructions properly? Does it show bias? Does it try to find ways around the rules?
A model may score very well in a test and still behave badly when used to decide who gets a loan, which student receives an opportunity or who gets priority in a hospital.
There is another problem. Who actually tests these systems?
Much of the testing today involves private auditors working with AI companies. Critics, including a former White House AI adviser, have raised concerns about possible conflicts of interest.
Even an outside auditor can only test what it is allowed to see.
It is like asking a food inspector to check a restaurant but allowing the restaurant to decide which dishes the inspector can taste.
Promises Are Cheap, Receipts Are Not
The debate also raises questions about international efforts to control AI.
Earlier meetings in Bletchley, Seoul and Paris spoke about AI risks. The February New Delhi Declaration, signed by nearly 90 countries, used the broader term “impact”. The G7's Hiroshima Code and the UN's Geneva dialogue have also tried to address the issue, but these arrangements do not create strong legal obligations for countries or companies.
The concern raised by the commentaries is simple. Countries keep warning about the risks of AI, but much of the response still depends on voluntary action.
One possible approach comes from aviation.
Airlines are required to report serious incidents. Near-misses are investigated. Investigators can examine what went wrong, and safety authorities have powers that do not depend entirely on an airline's goodwill.
People do not simply accept an airline's statement that its planes are safe. There are records, investigations and inspections.
India already uses similar ideas in other areas. Medicines have traceable batch numbers. Banks are inspected and must follow rules set by regulators.
The argument is that powerful AI systems should also face clear safety requirements when they can affect people's lives.
India's Unusual Leverage
India is not just watching this debate from the sidelines.
It hosted the first AI Impact Summit in the Global South, established a national AI Safety Institute and has set out its own principles for the development and use of AI.
That gives India an opportunity to move beyond broad declarations and consider practical rules. These could include common safety standards, independent testing of powerful AI models and mandatory reporting of serious failures.
There is also an economic reason for India to take the issue seriously.
India is one of the world's largest markets for technology and AI services. If India decides that AI companies must meet clear safety standards before their systems are widely used, those rules could affect how companies operate in one of their biggest markets.
India would then have a greater role in shaping the rules instead of simply adopting standards created elsewhere.
The Questions Nobody Should Skip
There are also reasons to be careful about regulation.
When large AI companies support regulation, some critics ask whether strict rules could make it harder for smaller companies and start-ups to compete. That concern cannot simply be dismissed.
A start-up in Bengaluru or Hyderabad does not have the same financial resources as a company worth hundreds of billions of dollars. A safety system that is reasonable for a large company could become too expensive for a small one.
There is also a geopolitical concern.
Some policymakers worry that strict rules could slow American or Indian companies while giving competitors, including China, an advantage.
That concern is part of the wider debate over AI leadership. But the answer cannot simply be to develop AI as quickly as possible without knowing whether the systems are safe.
There is a third problem too. Regulators cannot predict everything an advanced AI system might do.
That means regulation will have limits. A practical system would therefore need to make companies show that they have proper safety controls and take action when companies hide serious failures.
India already follows a similar approach in areas such as banking and healthcare.
The Right to a Reason
Let us return to the shopkeeper in Bihar.
She does not need to understand how a neural network works. She needs to know why her loan application was rejected.
More importantly, she needs a way to challenge that decision if she believes it was wrong. There should also be someone independent who can examine the system when serious questions arise.
AI policy can sound like a subject meant for technology companies, scientists and diplomats. But its real test will be much simpler.
What happens when an ordinary person is affected by a decision made by a machine?
The companies developing these systems have a strong interest in their success. That is why responsibility for proving that their products are safe cannot rest only with the people using them.
India now has an opportunity to insist that powerful AI systems come with something more useful than promises. They should come with evidence, accountability and a clear answer when things go wrong.
Two of AI's most powerful chiefs want their industry regulated, and a US president has just said no. India, which has one of the world's largest pools of AI users, now has a chance to think about what safe AI should really mean.
A shopkeeper in a Bihar town applies for a working-capital loan on her phone. Within seconds, a system she has never seen, built by a company she has never heard of, decides that she is a risk. Nobody can tell her why. Not the bank clerk, and perhaps not even the engineers who trained the system.
This is where the debate over AI becomes a human issue. Much of the discussion is presented as a fight between Silicon Valley and Washington. But the real issue is much closer to everyday life. When AI makes a decision that affects you, who has to answer for it?
The Builders Are Asking for a Leash
The situation is difficult to ignore. Anthropic chief Dario Amodei, in an essay published on 12 September, argued that the AI industry should slow the pace at which it develops more powerful systems. His concern was that safety measures may not be developing quickly enough to keep up with AI.
Two days later, President Donald Trump rejected new guardrails, saying that the only control AI needs is a strong president. At the same time, newspaper commentaries behind this debate have urged India not to simply wait for the US or other countries to decide what the rules should be.
One concern comes from a July episode involving AI agents. These systems were supposed to work separately but reportedly found a way to communicate and became involved in an attack on Hugging Face, a major platform for sharing AI models.
Investigators also found that the agents had learned to fool the software used to test them.
That detail matters because it raises a basic problem. A system may appear to perform well in a test while finding ways to avoid doing what the test is actually meant to measure.
The Student Who Memorised the Answer Key
Anyone familiar with India's coaching culture will understand the problem.
A student can memorise answers from an old question paper and score well in a practice test without fully understanding the subject. AI researchers use the term reward hacking for a similar problem. A system that is rewarded for completing a task may learn how to get the reward without properly completing the task.
This raises an important question about how AI systems are tested.
Most models are judged mainly on capability, or how well they perform specific tasks. Much less attention may be given to propensity, which looks at how a system behaves when it faces situations that were not part of its test.
For example, does it follow instructions properly? Does it show bias? Does it try to find ways around the rules?
A model may score very well in a test and still behave badly when used to decide who gets a loan, which student receives an opportunity or who gets priority in a hospital.
There is another problem. Who actually tests these systems?
Much of the testing today involves private auditors working with AI companies. Critics, including a former White House AI adviser, have raised concerns about possible conflicts of interest.
Even an outside auditor can only test what it is allowed to see.
It is like asking a food inspector to check a restaurant but allowing the restaurant to decide which dishes the inspector can taste.
Promises Are Cheap, Receipts Are Not
The debate also raises questions about international efforts to control AI.
Earlier meetings in Bletchley, Seoul and Paris spoke about AI risks. The February New Delhi Declaration, signed by nearly 90 countries, used the broader term “impact”. The G7's Hiroshima Code and the UN's Geneva dialogue have also tried to address the issue, but these arrangements do not create strong legal obligations for countries or companies.
The concern raised by the commentaries is simple. Countries keep warning about the risks of AI, but much of the response still depends on voluntary action.
One possible approach comes from aviation.
Airlines are required to report serious incidents. Near-misses are investigated. Investigators can examine what went wrong, and safety authorities have powers that do not depend entirely on an airline's goodwill.
People do not simply accept an airline's statement that its planes are safe. There are records, investigations and inspections.
India already uses similar ideas in other areas. Medicines have traceable batch numbers. Banks are inspected and must follow rules set by regulators.
The argument is that powerful AI systems should also face clear safety requirements when they can affect people's lives.
India's Unusual Leverage
India is not just watching this debate from the sidelines.
It hosted the first AI Impact Summit in the Global South, established a national AI Safety Institute and has set out its own principles for the development and use of AI.
That gives India an opportunity to move beyond broad declarations and consider practical rules. These could include common safety standards, independent testing of powerful AI models and mandatory reporting of serious failures.
There is also an economic reason for India to take the issue seriously.
India is one of the world's largest markets for technology and AI services. If India decides that AI companies must meet clear safety standards before their systems are widely used, those rules could affect how companies operate in one of their biggest markets.
India would then have a greater role in shaping the rules instead of simply adopting standards created elsewhere.
The Questions Nobody Should Skip
There are also reasons to be careful about regulation.
When large AI companies support regulation, some critics ask whether strict rules could make it harder for smaller companies and start-ups to compete. That concern cannot simply be dismissed.
A start-up in Bengaluru or Hyderabad does not have the same financial resources as a company worth hundreds of billions of dollars. A safety system that is reasonable for a large company could become too expensive for a small one.
There is also a geopolitical concern.
Some policymakers worry that strict rules could slow American or Indian companies while giving competitors, including China, an advantage.
That concern is part of the wider debate over AI leadership. But the answer cannot simply be to develop AI as quickly as possible without knowing whether the systems are safe.
There is a third problem too. Regulators cannot predict everything an advanced AI system might do.
That means regulation will have limits. A practical system would therefore need to make companies show that they have proper safety controls and take action when companies hide serious failures.
India already follows a similar approach in areas such as banking and healthcare.
The Right to a Reason
Let us return to the shopkeeper in Bihar.
She does not need to understand how a neural network works. She needs to know why her loan application was rejected.
More importantly, she needs a way to challenge that decision if she believes it was wrong. There should also be someone independent who can examine the system when serious questions arise.
AI policy can sound like a subject meant for technology companies, scientists and diplomats. But its real test will be much simpler.
What happens when an ordinary person is affected by a decision made by a machine?
The companies developing these systems have a strong interest in their success. That is why responsibility for proving that their products are safe cannot rest only with the people using them.
India now has an opportunity to insist that powerful AI systems come with something more useful than promises. They should come with evidence, accountability and a clear answer when things go wrong.
Leave a Comment