Autonomous weapons: Decisions to kill and destroy are a human responsibility

Convention on Certain Conventional Weapons
Statement of the ICRC, read at the Meeting of Experts on Lethal Autonomous Weapons Systems, held in Geneva from 11-16 April.

The International Committee of the Red Cross (ICRC) is pleased to contribute its views to this third CCW Meeting of Experts on "Lethal Autonomous Weapon Systems".

Discussions at the previous two CCW meetings of experts, as well as expert meetings convened by the ICRC and other organizations, have underscored the significant legal, ethical and societal questions raised by weapon systems that can select and attack targets without human intervention.

Although views on this complex subject continue to evolve, including those of the ICRC, discussions have indicated broad agreement that, for legal, ethical or military-operational reasons, human control over weapon systems and the use of force must be retained.

The ICRC has called on States to set limits on autonomy in weapon systems to ensure they are used in accordance with international humanitarian law (IHL) and within the bounds of what is acceptable under the principles of humanity and the dictates of public conscience.

As an additional contribution and further to the first ICRC meeting held in March 2014, the ICRC convened a second meeting of international experts ("Autonomous weapon systems: Implications of increasing autonomy in the critical functions of weapons") on 15-16 March 2016. Representatives of 20 States together with individual experts and representatives of the United Nations and civil society organizations participated in the meeting.

The aim of the meeting was to link the real-world development of autonomy in the critical functions of weapon systems more closely to consideration of the legal and ethical implications of, and potential policy approaches to, autonomous weapon systems. A summary report of the ICRC's expert meeting will be published shortly. In the meantime, as a contribution to this CCW meeting, we are circulating a short paper highlighting the ICRC's perspective on some of the key issues raised at our expert meeting. In particular, we would like to highlight the following points:

Definitions: The ICRC has proposed that "autonomous weapon systems" is an umbrella term encompassing any weapon system that has autonomy in the critical functions of selecting and attacking targets. We wish to stress that the purpose of this working definition is to promote better understanding of the issue and to help frame related discussions. The advantage of such a broad definition is that it enables consideration to be given to experience of existing weapons systems with autonomy in their critical functions and to lessons learned. This could facilitate the process of determining the boundaries of what is acceptable under IHL and the dictates of public conscience.
Existing weapons: Some weapon systems in use today can select and attack targets without human intervention. The ability to effectively control these weapons and the use of force seems to be closely linked to their predictability and reliability, as well as to strict operational constraints with respect to the task carried out, the targets attacked, the operational environment, the geographical space and time of operation, the scope to enable human oversight of the operation of the weapon system, and the human ability to deactivate it if need be.
Emerging technology: There is a likelihood that increasingly autonomous weapon systems could become less predictable, particularly in case of increased mobility, increased adaptability and/or increased interaction of multiple systems (as swarms). The loss of predictability regarding the outcomes of using an autonomous weapon may point to the loss of human control over that weapon's operation, with human decision-making over the use of force being replaced by machine processes. Therefore, significant questions may arise regarding compliance of such a weapon with IHL and its acceptability under the dictates of public conscience.
Legal reviews: The ICRC welcomes the recognition by States of the importance of reviewing new weapons to ensure their compatibility with international law, and stresses that efforts to strengthen national legal review processes are complementary and mutually reinforcing of discussions at the international level, in particular in the CCW meetings of experts. However, the legal review faces certain practical challenges regarding the assessment of whether an autonomous weapon system will perform as anticipated in the intended or expected circumstances of use. Two particular questions are raised: How is the predictability and reliability of the weapon assessed? What level of predictability and reliability is considered necessary?
The principles of humanity and the dictates of public conscience: Debates among States, experts, civil society and the general public have shown that there is a sense of deep discomfort with the idea of any weapon system that places the use of force beyond human control. Given the extreme gravity of the consequences and regardless of whether or not such systems are lawful, humanity may very well insist that decisions to kill or to destroy must continue to be taken by humans.
Human control: The notion of human control is the overarching issue in this debate. Whether for legal, ethical or military-operational reasons, there is broad agreement on the need for human control over weapons and the use of force. However, it remains unclear whether human control at the stages of the development and the deployment of an autonomous weapon system is sufficient to overcome minimal or no human control at the stage of the weapon system's operation – that is, when it independently selects and attacks targets. There is now a need to determine the kind and degree of human control over the operation of weapon systems that are deemed necessary to comply with legal obligations and to satisfy ethical and societal considerations.
The way forward: Since the need to maintain human control – whether "meaningful", "appropriate" or "effective" – over weapon systems and the use of force is consistent with legal obligations, military operational requirements and ethical considerations, the ICRC encourages States to use it as a framework for ongoing CCW discussions. Human control and consideration of human-machine interaction may provide a useful baseline from which common understandings can be developed among States, and through which limits on autonomy in weapon systems can be agreed. The ICRC urges CCW States Parties to agree at the Review Conference in December that future work focus on determining where these limits on autonomy in weapon systems should lie. Weapon systems that operate with autonomy in their critical functions already exist and new advances are constantly being made. There is therefore a need for States – and indeed for all stakeholders – to address this issue with a sense of responsibility and urgency, so as to ensure that technological developments do not outpace our legal and ethical deliberations.

We encourage delegations to read the paper we have circulated, which goes into greater detail on these points. We will be pleased to elaborate further on our views during the thematic sessions.

Thank you.

Introduction