Meaningful Human Control over Autonomous Weapon Systems: An (International) Criminal Law Account

[Marta Bo is a Researcher at the Graduate Institute (LAWS and War Crimes Project) and at the T.M.C. Asser Institute.]

Meaningful Human Control is at the core of regulatory and ethical debates on autonomous weapon systems. In international discussions and writings, the problem of meaningful human control has been addressed from different angles: from philosophical, ethical and legal (here and here), to operational, cognitive and technical (here and here), and recently from an engineering, socio-technical and governance perspective.

Despite the increasing use of parallel concepts such as ‘human element’ or ‘human oversight’, a recent, very comprehensive, SIPRI-ICRC report, which combines ethical, legal and operational considerations, adopts human control as a conceptual framework to shape limits to autonomy in weapons. In addition, a general obligation to maintain meaningful human control and positive obligations to ensure meaningful human control are the first and third pillars of the banning treaty advocated by HRW and the Harvard Law School International Human Rights Clinic.

This post provides an account of meaningful human control based on criminal law and on conditions to attribute principal criminal responsibility (rather than command responsibility). More specifically, it suggests that criminal law could provide elements not only to ground a theory of meaningful human control but also for the concrete operationalisation of this norm. A treaty obligation enshrining a duty to ensure meaningful human control should be adopted. This obligation should hinge on whether the individual had the capacity to fulfil the duty to take precautions in attacks and thus should provide for the attribution of criminal responsibility by omission (thus partly addressing responsibility gaps) limited to situations when it is fair to do so.

Why using criminal responsibility as conceptual framework for Meaningful Human control?

While concerns surrounding responsibility (and in particular the so-called responsibility gap) have been raised by civil society and academia since the inception of the debates on autonomous weapons, only recently have international regulatory discussions opened up and moved from a compliance-centric focus to encompassing issues of responsibility. Guiding Principles b) and d) adopted by the GGE on LAWS both refer to accountability.

The use of autonomous weapon systems might result in the commission of violations of the IHL principles of distinction and proportionality potentially amounting to war crimes. Criminal responsibility for unlawful attacks against protected persons (e.g. civilians) or objects is one kind of responsibility that could arise in this scenario.

In light of possible responsibility gaps and the cardinal notion of ‘control’ in criminal law, meaningful human control must be shaped so that individual criminal responsibility for unlawful attacks amounting to war crimes can be attributed. Amoroso and Tamburrini rightly speak of meaningful human control as ‘responsibility attractor’. As I have previously argued ‘the degree of human involvement and the corresponding level of autonomy of weapon systems are also crucial for the determination of criminal responsibility [and must] be able to ensure that the ‘human’ will be held criminally responsible.’

‘Control’ figures prominently in criminal law. It is a crucial notion within the international criminal law doctrine of command responsibility (Mettraux, 143 and 156 ff.),as well of some forms of complicity (Ashworth and Horder, p. 427)including joint perpetration before the ICC. Besides these forms of criminal participation or accessory liability, ‘control’ also plays a central role in the attribution of principal responsibility. Arguably, the applicability of command responsibility to an autonomous weapon is not self-evident (since it has so far been postulated on the existence of responsibility of human subordinates for crimes that the commander failed to prevent and punish). Thus, principal responsibility must be the starting point. obvious

Criminal law blames individuals for what they intended or foresaw (mens rea) and for what lay within their control (actus reus and causation). Key concepts such as intent, foreseeability, voluntary act, and causality entail some control conditions. Among those, importantly, criminal responsibility is causal responsibility: it requires human causal control over events. A theory of ‘meaningful human control’ must ensure that conditions for the ascription of principal criminal responsibility– and in particular foreseeability and influence over the act and the causal chain  – could be met if war crimes are committed.

It is beyond the scope of this post to develop a theory of human control ensuring human dominance over the act and human causal contribution. When human actions are intermediated by autonomous processes, foreseeability is irremediably affected, thus making the determination of mens rea and causality difficult.

Instead, I will focus on human omissions: that is, humans’ failures to stop autonomous systems.

Criminal responsibility for what and by what? Failures to stop and responsibility by omission for conduct of hostilities war crimes

Whilst intentionally committing a war crime and causing the death of civilians with an autonomous weapon system falls squarely within war crimes provisions, this will not be the most likely scenario when these systems are employed. Most probably, there will be some forms of (reckless or negligent) failures to stop these systems.

According to a recent SIPRI-ICRC report, ‘human supervision has been and is likely to remain a necessary form of control measure to ensure safety, reliability and efficiency of military operation (p. 21) and  seems ‘to be required in all but the most static and predictable targeting and conflict environments’’ (p. 10). Supervising the functioning of an autonomous system entails ‘authorizing, vetoing or overriding the targeting functions, aborting a task or deactivating the system’ (p. 28).

Thus, at the operational phase, the failure to stop an autonomous targeting cycle will be one of the most common situations arising when an autonomous weapons are deployed causing the violation of the principle of distinction or proportionality. Responsibility will arise for having failed to stop an attack against protected objects or individuals, such as civilians. This is a type of responsibility by omission, which refers to responsibility for not doing something (so-called commission by omission).

War crimes are defined in terms of affirmative action. If we take grave breaches in Article 85 AP I, it is a war crime to wilfully launch attacks against civilians causing death or serious injury and to wilfully launch attacks causing disproportionate death or serious injury are war crimes. According to Article 8 of the ICC Statute, it is a war crime to intentionally direct attacks against civilians or to intentionally launch disproportionate attacks.

In the context of principal responsibility, can these war crimes be committed by omission as well as by action? In other words, is ‘commission by omission’ of conduct of hostilities war crimes punishable? Can the lack of intervention/failure to stop an autonomous targeting process resulting in an attack in violation of the IHL principles of distinction or proportionality qualify as a war crime? This depends on the legal framework.

In most national systems, it is accepted that failing to prevent harm is equated to causing it, if a positive legal duty to act exists. The equivalence of act and omission is founded on the existence of such legal duty to act.

AP I accepts this equivalence and at Article 86 requires states and belligerents to ‘take measures necessary to suppress all other breaches of the Conventions or of this Protocol which result from a failure to act when under a duty to do so’’.

When it comes to the duty to prevent unlawful attacks, we must turn to the IHL duty to take precautions in Articles 57 and 58 AP I and in customary law that requires, among other things, that those responsible for planning, deciding upon or executing attacks do everything feasible to verify the objective of an attack and suspend attacks if it becomes apparent that they would violate the requirements of IHL – either because the target is recognised as a protected person or object or because the attack is disproportionate. Regardless of whether Articles 57 and 58 AP I and customary law impose these obligations directly on individuals, these duties are enacted in most national laws, military laws, manuals or rules of engagement, which make individuals the bearers of these obligations.

Legal duties to act create so-called ‘guarantor’ positions. The concept of guarantor in the context of omission responsibility in national criminal law refers to the ‘duty to safeguard any protected interest against potential dangers which are under [one’s] control (position of supervisor)’ (Duttwiler, pp. 35 and 36).  Supervisory positions ‘can be subdivided into duties to control potential dangers posed by third persons or by things’ (Duttwiler, p. 36). Importantly, autonomous weapons are to be equated to ‘things’. Thus, omission liability in the context of principal responsibility is in principle applicable, whilst the applicability of command responsibility to ‘things’ is arguably more uncertain. Once a legal duty to act exists, the capacity to perform the required act and to discharge the obligation is a requirement (in most national legal systems and in the ICTY case law) for responsibility by omission to be imposed.

Within the Rome Statute there is no general provision on commission by omission and the status of criminal responsibility by omission is more uncertain (save in instances of command responsibility). Dörmann contends that violations of the duty to take precautions do not give rise per se to criminal responsibility. Violations of this duty, and, importantly, failures to suspend unlawful attacks, amount to war crimes only to the extent that the violation evidences intent to unlawfully attack civilians or other protected persons or objects (p. 382). In the absence of intent, the impossibility of attributing responsibility for omission, i.e. for failures to suspend attacks launched by autonomous systems, would be very problematic and the criminal responsibility gap is apparent.

An obligation to ensure MHC as precondition for attributing criminal responsibility by omission and criminal responsibility as one of the drivers of the concept of MHC

Arguably, when commission by omission is accepted, an obligation to ensure meaningful human control is functional to a fair ascription of criminal responsibility. Human control is a precondition for the duty to take precautions insofar as it allows obligations stemming from this duty to be discharged in a comprehensive and informed manner.

According to Sassoli and Quintin (pp. 32 and 33): ‘What counts is not the authority under rules, regulations and instructions, but only who possesses the practical possibility to take precautions and in particular who has the necessary knowledge of risks’. A treaty obligation to ensure meaningful human control across the difference stages of the targeting process is key for the ‘practical possibility’ of discharging the duty to take precautions. However, such an obligation should also be implemented in national military laws and rules of engagement. The obligation to ensure meaningful human control over an autonomous system should be articulated in specific legal duties to act and be imposed upon individuals. All bearers of these duties should be identified taking into account the distributed nature of military decision-making processes, the specific features of the weapon system in use, as well as the stage of its use.

The purpose of the obligation would be twofold: to create duties to act and to place the individual bearers of these duties in the condition (i.e. capacity) to exercise them and suspend the launching of unlawful attacks. Legal obligations on individuals stemming from a general treaty obligation to ensure meaningful human control together with the IHL duty to take precautions create so-called ‘guarantor’ or supervisory positions.

Criminal responsibility of individuals in a position to exercise human control over the autonomous weapons system would arise for having neglected positive duties to act, as a result of which attacks in violation of the IHL principles of distinction or proportionality are not prevented and launched. In case these unlawful attacks occur, criminal responsibility would arise from failures to suspend attacks launched by autonomous weapons based on the violation of the duty to exercise control over the weapons system and the duty to exercise precautions. The framework of omission responsibility offers surrogates for the arduous assessment of causality and mens rea in cases of responsibility by commission (Fletcher, Rethinking Criminal Law, OUP 2000, p. 606).  In the context of human omissions, the assessment of mens rea and causality becomes less cumbersome than in cases of human actions intermediated by autonomous processes and thus allows a partial closing of the responsibility gap.

In sum, obligations to exercise meaningful human control across the different stages of the targeting process are necessary for the duty to take precautions in attacks to be discharged.

This allows for a fair attribution of criminal responsibility: only in cases where the individual/human was under a clear duty to act (e.g. control and suspend) and only when he/she was capable to do so (i.e. he/she had the ability to affect the autonomous process).

Conclusion:

In this post, I offered an account of the relationship between meaningful human control and criminal law whereby they can be mutually beneficial. A treaty obligation on meaningful human control must be adopted and articulated in terms of legal duties to act on individuals and must ensure that human operators are capable to exercise the duty to take precautions. Under these conditions, criminal responsibility for failing to stop unlawful attacks launched by autonomous weapons is fairly attributed.

The creation of supervisory positions and responsibility for omission are policy choices, which seem appropriate in light of the present and future relevance of supervised autonomy. This however highlights a wider potential responsibility gap when it comes to the punishment of conduct of hostilities war crimes committed by omission within the framework of the Rome Statute.