AI‑Driven Cyber Defense Workshop Discusses Autonomous Agent Risks

Esa Origins 7 February 24 — 26, 2017 PROJECT An Origins Project Scientific Workshop Challenges of Artificial Intelligence: Envisioning and Addressing Adverse Outcomes ARIZONA STATE UNIVERSITY in the past with Chess and Go. Computer systems are initially inferior to their human counterparts but quickly come to dominate the space. The purpose of ACWs means they will be equipped with strategies for replication, persistence, and stealth, all attributes that will make it hard to defend against them were they to “go rogue.” Because of this concern, it is likely a good idea for designers to add built-in “kill switches”, lifetimes, or other safety limitations. Figuring out how to effectively limit the actions of an ACW while maintaining its usefulness is likely a very hard problem. Current practices of cyber defense (especially against advanced threats) continue to be heavily reliant on manual analysis, detection and risk mitigation. Unfortunately, human-driven analysis does not scale well with the increasing speed and data amounts traversing modern networks. There is a growing recognition that the future cyber defense should involve extensive use of autonomous agents that actively patrol the friendly network, and detect and react to hostile activities rapidly (faster than human reaction time), before the hostile malware can inflict major damage, or evade elimination, or destroy the friendly agent. This requires cyber defense agents with a significant degree of intelligence, autonomy, self-learning and adaptability. Autonomy, however, comes with difficult challenges of trust and control by humans. The scenario considers intelligent autonomous agents in both defensive and offensive cyber operations. Their autonomous reasoning and cyber actions for prevention, detection and active response to cyber threats will become critical enablers for both industry and military in protecting large networks. Cyber weapons (e.g., malware) rapidly grow in their sophistication, and in their ability to act autonomously and to adapt to specific conditions encountered in a system/network. Agent’s self-preservation tactics are important for the continuous protection of networks, and if defeat is inevitable the agent should self-destruct (i.e., corrupt itself and/or the system) to avoid being compromised or tampered with by the adversary. Also, the notion of adversary must be defined and distinguishable for the agent. The system design and purpose is well intentioned — meant to reduce the load of human security analysts and network operators, and speed up reaction times in cyber operations. The agent monitors the systems in order to detect any adversarial activity, takes action autonomously, and reports back to the central command unit regarding the incident and the action taken. Since the agents are designed to be persistent, autonomous and learn, there are several implicit problems that can arise: e False reactions due to limited or misinformation — The agent has only a limited amount of technical information that does not always correspond to what is happening in the human layer. This can create false positives when trying to determine the adversary or adversarial activity. Since IZ HOUSE_OVERSIGHT_014708