Technical exposition on formal goal models for AGI agents

7.2 A Simple Formal Agents Model (SRAM) 131 7.2.1 Goals We define goals as mathematical functions (to be specified below) associated with symbols drawn from the alphabet G; and we consider the environment as sending goal-symbols to the agent along with regular observation-symbols. (Note however that the presentation of a goal- symbol to an agent does not necessarily entail the explicit communication to the agent of the contents of the goal function. This must be provided by other, correlated observations.) We also introduce a conditional distribution y(g, 4) that gives the weight of a goal g in the context of a particular environment ju. In this extended framework, an interaction sequence looks like 410191714202g92P9... or else a1 Yy1a2yo... where g; are symbols corresponding to goals, and y is introduced as a single symbol to denote the combination of an observation, a reward and a goal. Each goal function maps each finite interaction sequence Jg,.2 = ays: with gs to g¢ corre- sponding to g, into a value rg(Ig,s,2) € [0,1] indicating the value or “raw reward” of achieving the goal during that interaction sequence. The total reward 7; obtained by the agent is the sum of the raw rewards obtained at time ¢ from all goals whose symbols occur in the agent’s history before t. This formalism of goal-seeking agents allows us to formalize the notion of intelligence as “achieving complex goals in complex environments” — a direction that is pursued in Section 7.3 below. Note that this is an external perspective of system goals, which is natural from the perspective of formally defining system intelligence in terms of system behavior, but is not necessarily very natural in terms of system design. From the point of view of AGI design, one is generally more concerned with the (implicit or explicit) representation of goals inside an AGI system, as in CogPrime’s Goal Atoms to be reviewed in Chapter 22 below. Further, it is important to also consider the case where an AGI system has no explicit goals, and the system’s environment has no immediately identifiable goals either. But in this case, we don’t see any clear way to define a system’s intelligence, except via approximating the system in terms of other theoretical systems which do have explicit goals. This approximation approach is developed in Section 7.3.5 below. The awkwardness of linking the general formalism of intelligence theory presented here, with the practical business of creating and designing AGI systems, may indicate a shortcoming on the part of contemporary intelligence theory or AGI designs. On the other hand, this sort of situation often occurs in other domains as well — e.g. the leap from quantum theory to the analysis of real-world systems like organic molecules involves a lot of awkwardness and large leaps a well. HOUSE_OVERSIGHT_013047