AI Self-Governance Is Necessary — But It Is Not Enough

Late Sept 2026 witnessed some major movement in AI leaders honestly and openly acknowledging that AI risks (Burke & Boak, 2026). For years, we discussed Artificial Intelligence primarily through the lens of capability: What can AI do? How quickly can it learn? How much productivity can it unlock? Regardless of the motive behind these discussions, I am happy to see the conversation steer towards the more fundamental question: What happens when the capability itself begins to outpace our ability to govern it? 

In recent weeks, leaders and researchers associated with Anthropic, OpenAI, Microsoft, Meta and others have raised concerns about AI systems increasingly participating in the development of future AI systems (Burke & Boak, 2026; Fiegerman, 2026). A recent paper warns that automating AI research and development could, under certain conditions, produce an “intelligence explosion,” (Milmo, 2026) compressing years of technological progress into months or less. All these authors call for greater visibility, independent oversight and mechanisms to constrain development if necessary. Naturally, all roads lead to risks and compliance that pretty much every technical frontier had always been. The issue is not technology but we are forgetting the lessons each industrial revolution taught us with every technology introduced.

Interestingly, this is not an entirely new conversation (Rajagopalan, 2025). In 2020, I argued (Rajagopalan, 2020) that developing AI solutions required more than technical capability. We needed to ask whether our knowledge extended to the situation, who would be accountable for abuse or misuse, what value the solution actually created, and whether it was secure, stable, scalable and sustainable. In 2021, I extended that thinking by arguing that we should redefine “value” beyond business outcomes (Rajagopalan, 2021) to include the impact on people and the planet. I proposed four considerations for AI solutions: fairness, alternatives, risk management and efficacy. The underlying principle was simple: those who create and deploy AI cannot outsource responsibility for its consequences. 

Currently, people are just raising the alarms for self-governance! Interesting, encouraging and important, I would say!  AI organizations should absolutely govern themselves. Developers should identify risks before deployment, test beyond the “happy path,” consider misuse and abuse scenarios (Rajagopalan, 2015), provide meaningful alternatives to automated recommendations, document assumptions, establish accountability and continuously monitor systems after deployment. In fact, I actually proposed the risk driven development (Rajagopalan, 2023).  Recent proposals for independent evaluators are particularly relevant because they introduce some separation between the people building systems and those assessing their risks. Honestly, a much-needed relief that the conversation is at least acknowledged by the leaders finally as I believe this was leadership myopia focusing on capability hype getting caught in acceleration trap.

But self-governance should not become the end state of AI governance. It is one layer of governance and not the entire governance architecture. Let me offer an useful everyday analogy: Most people can manage themselves well enough to stop at a traffic light. We still have traffic laws. The existence of laws does not imply that every driver is irresponsible; rather, laws establish a common minimum standard when individual judgment fails, when incentives conflict, when someone does not know the rules, when governance is missed, or when controls are deliberately circumvented. 

My 2023 discussion (Rajagopalan, 2023) of risk management made a similar distinction between standards, organizational practices and regulations. Standards and frameworks can establish repeatable practices, while regulation provides an external obligation and accountability mechanism. AI needs this same layered approach: responsible people, responsible organizations, independent assessment, industry standards and enforceable regulation working together. In fact, I further expand on why "explainability"needs to shift towards an accountable AI growth mechanism (Rajagopalan, 2026b) and why we need to move from human-in-the-loop to human-in-the-middle mindset (Rajagopalan, 2026a)

The real opportunity, therefore, is not to choose between self-governance and government regulation. It is to recognize that we need both and that each solves a different problem. Self-governance can move faster, understand the technology more deeply and embed responsibility into everyday engineering decisions. Think about GDPR regulations that put a new spin on privacy controls! While we can argue whether it is required or not, it enforced processes and standards within the organization (as part of the organizational process assets) to protect the end consumer. External governance provides the backstop when self-governance is absent, inadequate, conflicted, bypassed or compromised. The recent warnings from AI leaders should therefore not be interpreted simply as a reason to give the industry more control over its own future. 

They should be a reminder that the people building increasingly powerful systems have a responsibility to govern themselves—and that society has an equally important responsibility to establish the guardrails that remain in place when self-governance does not. The question is no longer merely whether AI can govern itself. The question is whether we are building a governance system resilient enough to protect people when it cannot.

References

Burke, G. & Boak, J. (2026, Sep 28). Anthropic and OpenAI sound the alarm on AI safety — and seek to shape how it's controlled. Retrieved from https://www.pbs.org/newshour/world/anthropic-and-openai-sound-the-alarm-on-ai-safety-and-seek-to-shape-how-its-controlled

Fiegerman, S. (2026, Sep 28). Anthropic, OpenAI Executives urge oversight of Self-Improving AI. Retrieved from https://bloomberg.com/news/articles/2026-09-28/anthropic-openai-executives-urge-oversight-of-self-improving-ai

Milmo, D. (2026, Sep 28). AI godfathers warn of runaway ‘intelligence explosion’. Retrieved from https://www.theguardian.com/technology/2026/sep/28/ai-godfathers-warn-of-runaway-intelligence-explosion

Veiga, A. (2026, Sep 14). New warnings about the risks of AI to humanity to revive a long-running debate. Retrieved from https://thebusinessjournal.com/anthropic-ceo-ai-development-safety-risks/

Rajagopalan, S. (2015, Sep). Abuser Stories: What should the software do? Retrieved from https://agilesriram.blogspot.com/2015/09/abuser-storieswhat-shouldn-software-do.html

Rajagopalan, S. (2020, Sep). Artificial Intelligence Solutions: Four Considerations extended from Digital Bioethics, Retrieved from https://agilesriram.blogspot.com/2020/09/artificial-intelligence-solutions-four.html

Rajagopalan, S. (2021, Jan). Redefining Value in Artificial Intelligence Solutions. Retrieved from https://agilesriram.blogspot.com/2021/01/redefining-value-in-artificial.html

Rajagopalan, S. (2023, August). Risk Management: Birds' Eye View of some Standards and Regulations, https://agilesriram.blogspot.com/2023/08/risk-management-birds-eye-view-of-some.html

Rajagopalan, S. (2025). Leadership Unleashed: Game Changing Insights. Denver, CO: Outskirts Press.

Rajagopalan, S. (2026a, Aug). LEAD Framework: Differentiating HIL from HIM. Retrieved from https://agilesriram.blogspot.com/2026/08/lead-framework-differentiating-hil-from.html

Rajagopalan, S. (2026b, Jun). Explainability is the measure of Accountability: Why Responsible AI must evolve into Accountable AI. Retrieved from https://agilesriram.blogspot.com/2026/06/explainability-is-measure-of.html