Academic discussion of counterfactual models for assessing intelligent agents

140 7 A Formal Model of Intelligent Agents We suggest to view the definitions of pragmatic and efficient pragmatic general intelligence in terms of a “possible worlds” semantics — i.e. to view them as asking, counterfactually, how an agent would perform, hypothetically, on a series of tests (the tests being goals, defined in relation to environments and reward signals). Real-world intelligent agents don’t normally operate in terms of explicit goals and rewards; these are abstractions that we use to think about intelligent agents. However, this is no objection to characterizing various sorts of intelligence in terms of counterfactuals like: how would system S operate if it were trying to achieve this or that goal, in this or that environment, in order to seek reward? We can characterize various sorts of intelligence in terms of how it can be inferred an agent would perform on certain tests, even though the agent’s real life does not consist of taking these tests. This conceptual approach may seem a bit artificial but we don’t currently see a better alternative, if one wishes to quantitatively gauge intelligence (which is, in a sense, an “artificial” thing to do in the first place). Given a real-world agent X and a mandate to assess its intelligence, the obvious alternative to looking at possible worlds in the manner of the above definitions, is just looking directly at the properties of the things X has achieved in the real world during its lifespan. But this isn’t an easy solution, because it doesn’t disambiguate which aspects of X’s achievements were due to its own actions versus due to the rest of the world that X was interacting with when it made its achievements. To distinguish the amount of achievement that X “caused” via its own actions requires a model of causality, which is a complex can of worms in itself; and, critically, the standard models of causality also involve counterfactuals (asking “what would have been achieved in this situation if the agent X hadn’t been there”, etc.) [MW07]. Regardless of the particulars, it seems impossible to avoid counterfactual realities in assessing intelligence. The approach we suggest — given a real-world agent X with a history of actions in a particular world, and a mandate to assess its intelligence — is to introduce an additional player, an inference agent 6, into the picture. The agent 7 modeled above is then viewed as 7x: the model of X that 6 constructs, in order to explore X’s inferred behaviors in various counterfactual environments. In the test situations embodied in the definitions of pragmatic and efficient pragmatic general intelligence, the environment gives 7x rewards, based on specifically configured goals. In X’s real life, the relation between goals, rewards and actions will generally be significantly subtler and perhaps quite different. We model the real world similarly to the “fantasy world” of the previous section, but with the omission of goals and rewards. We define a naturalistic context as one in which all goals and rewards are constant, i.e. g; = 99 and r; = 7p for all 7. This is just a mathematical convention for stating that there are no precisely-defined external goals and rewards for the agent. In a naturalistic context, we then have a situation where agents create actions based on the past history of actions and perceptions, and if there is any relevant notion of reward or goal, it is within the cognitive mechanism of some agent. A naturalistic agent X is then an agent 7 which is restricted to one particular naturalistic context, involving one particular environment pe (formally, we may achieve this within the framework of agents described above via dictating that X issues constant “null actions” ao in all environments except ;:). Next, we posit a metric space (2, d) of naturalistic agents defined on a naturalistic context involving environment yp, and a subspace A € &’, of inference agents, which are naturalistic agents that output predictions of other agents’ behaviors (a notion we will not fully formalize here). If agents are represented as program trees, then d may be taken as edit distance on tree space [Bil05]. Then, for each agent 6 € A, we may assess HOUSE_OVERSIGHT_013056