Enforcing an AI Moratorium: Lessons from the ICC?

[Oscar Pearce is a Research Officer at the Australian National University’s College of Law]

The recent proliferation of artificial intelligence in the form of Large Language Models (‘LLMs’) has amplified conversations about the social and political risks of such technology. While the threats of existing AI capabilities are daunting, experts believe that in the not-too-distant future, advanced AI (or Artificial General Intelligence, ‘AGI’) could pose an existential risk.

This post will provide a brief background on the existential risks of AI, before discussing the emerging proposals for a multinational treaty aimed at mitigating those risks. The conversation around enforcement mechanisms for such a treaty is vital, but only just emerging. 

I will argue that as the enforcement conversation gathers momentum, scholars and practitioners alike should pay careful attention to the model of the International Criminal Court (‘ICC’). I do not mean to suggest that the ICC represents best practice, or that its model can be neatly applied to AGI governance. It is, however, the leading example of a multilateral enforcement mechanism targeting non-state legal persons. As such, ignoring the strengths and weaknesses of the ICC would be to the detriment of efforts to mitigate existential AGI risk.

Background on Artificial General Intelligence

Existing AI technology poses severe social and political risks. Alarming developments include deepfakes, social scoring and, more generally, algorithmic bias. In the Australian judicial context, the Victorian Law Reform Commission has recently provided a helpful categorisation of these risks. Privacy and data security are the primary risks when assembling the underlying data sets. Once training and testing begins, AI systems can develop harmful biases. For example, if a model is trained on biased data (eg, higher crime rates among a certain social group due to discriminatory policing), then that model is vulnerable to reproducing or amplifying those biases. Upon becoming operational, models may be inaccurate; a prominent topic following the rise of LLMs with their tendency to ‘hallucinate’. Moreover, and in any event, these AI systems are black boxes that reach conclusions through opaque or outright inexplicable methods

Even if all these inherent shortcomings are avoided, there remains the risk that AI acts as a force multiplier, worsening existing threats. In the field of public international law, it is this last category of risk that has received the most attention, in the context of autonomous weapons and AI targeting (see, eg, here, here and here).

An altogether different category of risk, however, is the existential risk posed by AGI. While current technologies can only outperform humans in certain discrete metrics, it is reasonably foreseeable that an AI system could greatly surpass human capabilities. The consequences of creating such a superintelligence, or “godlike AI”, are unpredictable. What would motivate the rogue system? How, if at all, could it be constrained? These and many other questions are still to be answered.

Difficult moral, epistemological, and technical questions aside, the opacity and scale of AGI risk is sufficient to warrant serious concern. In his book ‘The Precipice’, Oxford philosopher Toby Ord estimates a 1 in 10 chance that ‘unaligned’ (ie, ill-motivated) AI spells the end for humanity in the next 100 years. Indeed, a majority of surveyed experts estimate the probability of AI causing human extinction to be at least 10%. Even UN Secretary General António Guterres and The Elders have issued warnings over AGI’s potentially existential threat. 

States are responding. UNESCO and the UN General Assembly have adopted relevant recommendations, the EU passed its AI Act, and the Council of Europe created a Framework Convention on Artificial Intelligence. Of greater relevance to AGI systems in particular, 28 states and the EU signed the Bletchley Declaration and US President Biden has signed an executive order. 

Most of these developments set out general principles and basic recommendations, including, inter alia, the implementation of risk assessment frameworks. The EU AI Act warrants particular attention, however. While it is concerned with the immediate risks of AI (deepfakes, social scoring, bias, etc) rather than the existential threat posed by AGI, it is the closest analogue to the proposals for a multilateral AGI treaty discussed below. It imposes direct obligations on developers, backed by a penalty regime. This includes a prohibition on certain AI systems punishable by a 35 million euro fine. It is too early to assess the effectiveness of that regime, and it must be conceded that EU regional enforcement is often an exception to the ordinary limitations of multilateral treaty enforcement. Nevertheless, the EU AI Act is a helpful case study of enforceable international obligations on AI developers.

Emerging Proposals for Multilateral AGI Governance

Proposals for mitigating AGI risks through multilateral governance fall into two general categories. First, many have called for a standalone global AGI body to be established. The tasks of such a body vary between proposals, ranging from scientific consensus-building to global political alignment to policy coordination/standard-setting. 

A second, and rather intuitive, response to the risk posed by AGI is to impose a moratorium on the AI arms race. The emergence of AGI could plausibly be dependent on AI systems reaching a particularly massive scale. Developers could be prevented from reaching that dangerous scale by, for example, introducing global compute limits. ‘Compute’ refers to the raw computational inputs (processing power, memory, storage, etc) used to develop AI systems. Alternatively, the scale or nature of data centres, or even individual AI training runs, could plausibly be constrained. These are not the only models for an AI moratorium, but they are indicative of the measures that a multilateral AGI treaty could impose. 

While it is far from guaranteed that international support can be mustered for multilateral AI regulations, this post will consider a world in which this major political hurdle is cleared. 

As ever, the verification of compliance with such moratoriums would not be straightforward. Initial proposals have reasoned by analogy to the verification of nuclear disarmament and other arms-related obligations. Despite the stark difference between AI system development and weapons manufacturing, there are reasons to believe that effective verification methods are achievable.

Beyond mere verification, the complexities of enforcing a hypothetical AGI treaty are novel. A weakness in the analogy to weapons prohibitions is that AI development is likely to be conducted in the private sector. A multilateral compute cap, for example, would be seeking to regulate corporate entities and individual researchers, rather than public entities alone. Consequently, AI researchers should not limit themselves to the International Atomic Energy Agency, the Organisation for the Prohibition of Chemical Weapons, and so on in searching for enforcement inspiration. Rather, due consideration should be given to the leading public international law enforcement mechanism for obligations on individuals: the International Criminal Court. 

There are, of course, shortcomings with drawing inspiration from the ICC. The ICC prosecutes natural persons, while corporations are likely to be the relevant subject of AGI governance. The ICC’s remit covers genocide, crimes against humanity, war crimes and crimes of aggression. These may reasonably be seen as far more heinous than a developer pushing the boundaries of permissible AI development. Indeed, even for the purpose it is intended to serve, the ICC attracts frequent criticism (see, eg, here, here, here, here). Moreover, it is improbable that bringing dangerous AI development under the jurisdiction of an international criminal court is politically feasible. 

Due to the scale of existential AGI risk, however, some form of non-compliance penalty would be an appropriate addition to a hypothetical AGI treaty, akin to the EU AI Act’s penalty regime. 

For instance, a corporation engaging in unlawful AI development in one jurisdiction ought not to reap the rewards of that development in another jurisdiction (let us call this the ‘Lax AI Regulation Loophole’). While individual jurisdictions may have capacity to review AI development operations on their own soil, it is unreasonable for them to be tasked with reviewing the practices of a given corporation across all the jurisdictions in which they operate. Thus, without some form of international oversight of AI developers, there is serious vulnerability to the Lax AI Regulation Loophole.

If the oversight of international AI developers is to be pursued, the ICC is relevant for several reasons. As already noted, it demonstrates a model for enforcing international obligations on non-state legal persons. As such, the principles that it has developed are highly relevant to AGI enforcement. The ICC’s fundamental principle of complementarity, for example, could be readily embedded in AGI regulation enforcement, thereby avoiding the need to construct a jurisdiction regime from scratch.

Likewise, AI scholars ought to review the sophisticated (and relatively demanding) cooperation arrangements outlined in Part 9 of the Rome Statute. Contraventions of an AI moratorium are likely to involve complex cross-border elements, due to the possibility of data centres, individual developers and responsible corporations being physically dispersed entities. As such, even the most demanding enforcement mechanism could be rendered toothless if it cannot also compel cooperation from third parties. 

Finally, the political history of the ICC could provide important lessons. AI safety discourse has been relatively effective at avoiding partisanship and polarisation on the domestic level. If a multilateral AGI treaty is to be developed, scholars and practitioners in the field must grapple with the distinct international political pitfalls surrounding notions of sovereignty, regional bias, and legitimacy.

The ICC model is not flawless, nor perfectly applicable to the AI governance context. That said, the ICC’s principles, practices and political history should absolutely be given due consideration in the emerging conversation on AI treaty enforcement.