Journal of Theoretical Biology 399 (2016) 103-116

Journal of Theoretical Biology 399 (2016) 103-116 Contents lists available at ScienceDirect Journal of Theoretical Biology journal homopago::::vw.elsevier.com/locate/yjtbi Jam. 1 or Theoretical lhoOgy Evolution of worker policing Jason W. Olejarza, Benjamin Allen b•a'c, Carl VelleraA, Raghavendra Gadagkai Martin A. Nowaka•d-gs ▪ Program for Evolutionary Dynamic. Harvard University, Cambridge. MA 02738, USA b Department of Mathematics. Emmanuel Cortege, Boston, MA 02115. USA Center for Mathematical Sciences and Applications. Harvard University, Cambridge. MA 02738, USA ▪ Department of Organismic and Evolutionary Biology. Harvard University, Cambridge, MA 0213B, USA ' Centre for Ecological Sciences and Centre for Contemporary Studies. Indian Institute of Science, Bangalore 560 012. India Indian National Science Academy, New Delhi 170 002. India 2 Department of Mathematics, Harvard University, Cambridge. AM 02138, USA CrossMatk ARTICLE INFO Ankle history. Received 2 February 2015 Received in revised form 23 January 2016 Accepted 2 March 2016 Available online II March 2016 Keywords: Sociobiology Natural selection Evolutionary dynamics Models/simulations ABSTRACT Workers in insect societies are sometimes observed to kill male eggs of other workers, a phenomenon known as worker policing. We perform a mathematical analysis of the evolutionary dynamics of policing. We investigate the selective forces behind policing for both dominant and recessive mutations for dif- ferent numbers of matings of the queen. The traditional, relatedness-based argument suggests that policing evolves if the queen mates with more than two mates, but does not evolve if the queen mates with a single male. We derive precise conditions for the invasion and stability of policing alleles. We find that the relatedness-based argument is not robust with respect to small changes in colony efficiency caused by policing. We also calculate evolutionarily singular strategics and determine when they are evolutionarily stable. We use a population genetics approach that applies to dominant or recessive mutations of any effect size. e. 2016 Elsevier Ltd. All rights reserved. 1. Introduction In populations with haplodiploid genetics. unfertilized female workers are capable of laying male eggs. Thus, in a haplodiploid colony, male eggs can in principle originate from the queen or from the workers. Worker policing is a phenomenon where female workers kill the male eggs of unmated female workers (Ratnieks, 1988: Ratnieks and Visscher.1989: Ratnieks et al„ 2006: Gadagkar, 2001: Wenseleers and Ratnieks, 2006a). Worker policing is observed in many social insects, including ants, bees, and wasps. Yet the precise conditions for the evolution of worker policing are still unclear. Worker policing (Ratnieks, 1988: Rah-licks et al.. 2006: Gadagkar, 2001: Wenseleers and Ratnieks, 2006a) and worker sterility (Wilson, 1971: Hamilton, 1972: Olejarz et al.. 2015) are two distinct phe- nomena that are widespread in the eusocial Hymenoptera. In addi- tion to worker policing. a subset of workers in a colony may forego their own reproductive potential to aid in raising their siblings. Prior relatedness-based arguments have suggested that queen monogamy is important for the evolution of a non-reproductive worker caste 'Corresponding author. E-mail address: martin_nowaloPharvard.edu (MA Nowak). http:fidx.dokorg/10.1016/mtbi.2016.03.001 0022-51930 2016 Elsevier Ltd. All rights reserved. (Hughes et al., 2008: Cornwallis et al.. 2010: Queller and Strassmann, 1998: Foster et al.. 2006: Boomsma, 2007, 2009). In contrast• it is believed that polygamy—not monogamy—is important for the evo- lution of police workers. Several papers have studied the evolution of policing. Starr (1984) explores various topics in the reproductive biology and sociobiology of eusocial Hymenoptera. He defines promiscuity as 1/(E7_,J1), where n is the number of matings of each queen. and .h is the fractional contribution to daughters by the i-th male mate. He writes, regarding workers, that "They are on average less related to nephews than brothers whenever (promiscuity is greater than twol and should prefer that the queen lay all the male eggs. Workers would therefore be expected to interfere with each others reproduction.' Thus, Starr (1984) was the first to suggest that workers should raise their nephews (sons of other workers) if the queen mates once, but should only raise their brothers (sons of the queen) if the queen mates more than twice. Starr (1984) uses a relatedness-based argument, but he does not provide any calcu- lation of evolutionary dynamics in support of his argument: he uses neither population genetics nor inclusive fitness theory. In a book on honeybee ecology. Seeley (1985) also proposed, using a relatedness-based argument, that worker policing should occur in colonies with multiply mated queens, but that worker policing should be absent if queens are singly mated. EFTA01070997 104 OkJorz et al /Journal of theatrical Biology 399 (2016) 103-116 Woyciechowski and Lomnicki (1987) perform a calculation based on population genetics and conclude that workers should raise their nephews (sons of other workers) if the queen mates once, but should only raise their brothers (sons of the queen) if the queen mates more than twice—the case of double mating is neu- tral with respect to preference. From this result, they claim that. under multiple mating of the queen, natural selection should favor non-reproductive workers. Woyciechowski and Lomnicki (1987) consider both dominant and recessive alleles affecting worker behavior, but they do not consider colony efficiency effects. Ratnieks (1988) considers the invasion of a dominant allele for policing. Using population genetics, he arrives at essentially the same conclusion as Woyciechowski and Lomnicki (1987): In the absence of efficiency effects, policing evolves with triple mating but not with single mating. But Ratnieks also considers colony efficiency effects, focusing mainly on the case where policing improves colony efficiency. Since policing occurs alongside other maintenance tasks (such as cleaning of cells, removal of patho- gens. incubation of brood), and since eating worker-laid eggs might allow workers to recycle some of the energy lost from laying eggs, Ratnieks supposes that policing improves colony efficiency. He finds that worker policing with singly mated queens may evolve if policing improves colony reproductive efficiency. He also finds that worker policing with triply mated queens may not evolve if policing reduces colony reproductive efficiency, but he considered this case to be unlikely on empirical grounds. Ratnieks does not study recessive policing alleles. He also does not calculate evolutionary stability conditions. Both papers (Woyciechowski and Lomnicki. 1987: Ratnieks, 1988) offer calculations based on population genetics without mentioning or calculating inclusive fitness. These early studies (Starr, 1984: Seeley, 1985: Woyciechowski and Lomnicki, 1987: Ratnieks. 1988) were instrumental in establishing the field of worker policing. Testing theoretical predictions on the evolution of worker policing in the field or in the lab is difficult. Due to the complex- ities inherent in insect sociality, published empirical results are not always easy to interpret. While, so far, worker policing has been found in all species with multiple mating that have been studied, it has also been found in about 20% of species with singly mated queens (Hammond and Keller, 2004: Wenseleers and Ratnieks. 2006b: Bonckaert et al.. 2008). Herein lies the difficulty: When worker policing is found in multiply mated species and found to be absent in singly mated species, this is taken as evi- dence supporting the relatedness argument, and when worker policing is found in singly mated species, it is explained away as not being evidence against the theory, but as having evolved for other reasons (such as colony efficiency). See, for example, the following quotation by Bonckaert et al. (2008): "Nevertheless, our results are important in that they show that V. germanico forms no exception to the rule that worker reproduction should be effec- tively policed in a species where queens mate multiple times (Ratnieks. 1988). Indeed, any exception to this pattern would be a much bigger challenge to the theory than the occurrence of worker policing in species with single mating, which can be readily explained (Ratnieks. 1988: Foster and Ratnieks, 2001b)' This is precisely why a careful simultaneous consideration of relatedness, male parentage, and colony efficiency is important for understanding worker policing. We do not aim to provide an exhaustive catalog of all species in which worker policing has been studied. We merely cite some specific examples to add context Policing is rampant in colonies of the honeybee (Ratnieks and Visscher, 1989), the wasp Vespula vulgaris (Foster and Ratnieks. 2001c), and the wasp Vespula ger- manica (Bonckaert et al.. 2008), which are all multiply mated. (As mentioned above, worker policing has been found in all of the studied species to date that are multiply mated.) Worker removal of worker-laid eggs is much less prevalent in colonies of the bumblebee (Velthuis et al.. 2002), the stingless bee. (Peters et al.. 1999), and the wasp, Vespula rub (Wenseleers et al.. 2005). which are predominantly singly mated. (As mentioned above, worker policing has been found only in about 20% of the studied species to date that are singly mated.) There are some studies based on observational evidence that find policing in singly mated species: examples of species with single mating and worker policing are Vespa crabro (Foster et al., 2002), Camponotusfloridanus (Endler et al.. 2004). Aphaenogasrer smythiesi (Wenseleers and Ratnieks. 2006b), and Diacamma (Wenseleers and Ratnieks, 2006b). Interspecies comparisons are somewhat problematic, because even though phylogeny can be controlled for, there are many (known and unknown) ways in which species differ in addition to mating frequency that may also affect the absence or presence of worker policing. Furthermore. many empirical studies are based on genetic analyses of male parentage. (Though studies of some species are based on actual observational evidence: see, e.g., Wenseleers and Ratnieks. 2006b.) Regarding species for which the study of policing is based on genetic analyses, policing is often inferred if males are found to originate predominantly from the queen. But such an inference. in cases where it is made. pre- supposes that workers actively try to lay male eggs in the first place. It is therefore not clear how reliably genetic investigations can measure policing. The small number of attempts at measuring the prevalence of worker policing in intraspecific experiments have also returned conflicting results. Foster and Ratnieks (2000) report that facul- tative worker policing in the saxon wasp. Dotichovespula scuronica, is more common in colonies headed by multiply mated queens. But their sample size is only nine colonies. The phenomenon was reinvestigated by Bonckaert et al. (2011) who report no evidence of facultative worker policing depending on queen mating fre- quencies. and argue that the previous result may have been flawed or that there were interpopulational variations. Many empirical studies have emphasized that factors besides intracolony relatedness—including the effects of policing on a colony's rate of production of offspring—may play a role in explaining evolution of worker policing (Foster and Ratnieks 2001a,c: Hartmann et al., 2003: Hammond and Keller, 2004: Wenseleers and Ratnieks. 2006b: Helantera and Sundstrom, 2007: Khila and Abouheif, 2008: Zanette et al.. 2012). Yet reliable pub- lished data on the effect that policing has on colony reproductive efficiency are often hard to find. (For some exceptions, see Wenseleers et al., 2013 and references therein.) In this paper. we derive precise conditions for the evolutionary invasion and evolutionary stability of police alleles. We consider any number of matings, changes in the proportion of queen- derived males. changes in colony efficiency, and both dominant and recessive mutations that affect the intensity of policing. Our paper is based on an analysis of evolutionary dynamics and population genetics of haplodiploid species (Nowak et al., 2010: Olejarz et al.. 2015). It does not use inclusive fitness theory. Spe- cifically. we adapt the mathematical approach that was developed by Olejarz et al. (2015) for the evolution of non-reproductive workers. We derive evolutionary invasion and stability conditions for police alleles. Mathematical details are given in Appendix A. In Section 2, we present the basic model and state the general result for any number of matings for dominant policing alleles. In Sections 3-5. we specifically discuss single. double, and triple mating for dominant policing alleles. We take dominance of the policing allele to be the more realistic possibility because the policing phenotype is a gained function. Nonetheless. for com- pleteness. we give the general result for recessive policing alleles in Section 6. In Section 7. we discuss how the shape of the EFTA01070998 J.W. OleJeri et al. /Journal of theoretical Biology 399 (2016) 103-116 105 efficiency function determines whether or not policing is more likely to evolve for single or multiple matings. In Section 8. we analyze our results for the case where the phenotypic mutation induced by the mutant allele is weak (or. equivalently in our formalism, the case of weak penetrance). In this setting, the quantity of interest is the intensity of policing. We locate the evolutionarily singular strategies. These are the values of intensity of policing for which mutant workers with slightly different policing behavior are, to first order in the mutant phenotype, neither advantageous nor disadvantageous. We then determine if a singular strategy is an evolutionarily stable strategy (ESS). In Section 9. we discuss the relationship between policing and inclusive fitness theory, together with the limitations of the relatedness-based argument. Section 10 concludes. 2. The model We investigate worker policing in insect colonies with haplo- diploid genetics. Each queen mates n times. We derive conditions under which a mutation that effects worker policing can spread in a population. We make the simplifying assumption, as do Woyciechowski and Lomnicki (1987) and Ratnieks (1988), that the colony's sex ratio is not affected by the intensity of worker policing. First we consider the case of a dominant mutant allele. Because the policing allele confers a gain of function on its bearer, the assumption that it is dominant is reasonable. There are two types of males. A and a. There are three types of females. AA, Ac. and aa. If the mutant allele is dominant, then Aa and aa workers kill the male eggs of other workers, while AA workers do not. (Alter- natively. AA workers police with intensity 44 while Aa and aa workers police with intensity ZA0 = Z 2a = 44+w. We consider this case in Section 8.) For n matings, there are 3(n + 1) types of mated queens. We use the notation AA. m: Aa, m: and aa,m to denote the genome of the queen and the number. m. of her matings that were with mutant males, a. The parameter m can assume values 0,1...., n. A schematic of the possible mating events is shown in Fig. 1(a). There are three types of females, AA. Act, and ca. and there are n+1 possible combinations of males that each queen can mate with. (For example, a queen that mates three times (n=3) can mate with three type A males, two type A males and one type a male, one type A male and two type a males, or three type a males.) Fig. 1(b) shows the different colony types and the offspring of each type of colony when each queen is singly mated. Fig. 1 (c) shows the different colony types and the offspring of each type of colony when each queen mates n times. The invasion of the mutant allele only depends on a subset of colony types. The cal- culations of invasion conditions are presented in detail in Appendix A. 2.1. Fraction of male offspring produced by the queen Pr represents the fraction of males that are queen-derived if the fraction of police workers is z. (This quantity was already employed by Ratnieks. 1988.) The parameter z can vary between 0 and 1. For z=0. there are no police workers in the colony, and for z=1. all workers in the colony are policing. We expect that pr is an increasing function of z. Increasing the fraction of police workers increases the fraction of surviving male eggs that come from the queen (Fig. 2). a Virgin Queens n Males Fertilized Queens AA +(n —m) A +m a AA,m Aa +(n -m) A +m a Aa,m aa +(n —m) A +m a aa,m b Oman Gomm OM Ousts MO•Onellos MO Oen mina Man On VIManelms MO AA A A A•• M1/21 As A An MA AA•Pa 5A•• AM *aim An Ana ma As A•• M1 la a C IsehOINOI MOE • TOM Omen Omega Omen Oman I Clain- MINIONIfielt• la na AnmPa A ft...inns fan-r••ta•h•Ael.- Mon I 05-MM•Mann oil A • Ilp 16-nIsa•maa In-MA•ln•mla Flg. I (a) The possible mating events with haplodiploid genetics are shown. Each queen mates with n males. in denotes the number of times hat a queen mates with mutant type a males and can take values between 0 and n Thus, there are 3(n +1) types of colonies. (b) If each queen mates with only a single male, then there are six types of colonies. The female and male offspring (right hree columns) of each colony (leftmost column) are shown. For example. AA. I co onies arise from a type AA female mating with a single mutant type a male. M.1 queens produce female offspring of type Aa and male offspring of type A. 50% of the offspring of workers in M. I colonies are of type A, while the remaining 50% of the offspring of workers in M. I colonies are of type a. (c) The female and male offspring (right three columns) of each colony (leftmost column) when each queen mates it times are shown. 0 0 2E 0.75 rngi 0.50 Possible Forms for the Function p : wo 0.25 O LI- 0 0 0.25 0.50 0.75 Fraction of police workers, z Fig 2. The queen's production of male eggs, p., increases with the fraction of workers that are policing, 2. This is intuitive, since having a larger worker police force means that a greater amount of worker-laid eggs can be eaten of removed. Three possibilities for a monotonically increasing function pt are shown. 22. Colony efficiency as a function of policing re represents the rate at which a colony produces offspring (virgin queens and males) if the fraction of police workers is z. (This quantity was also employed by Ratnieks. 1988.) Without loss of generality, we can set ro = 1. For a given mutation that affects the intensity of policing, and for a given biological setting, the efficiency function re may take any one of a variety of forms (Fig. 3). Colony efficiency depends on interactions among police work- ers and other colony members. It also depends on the interactions of colonies and their environment. There are some obvious nega- tive effects that policing can have on colony efficiency. By the act EFTA01070999 106 OkJarz et al. 1 puma, of theoretical Biology 399 (2016)193-116 1.3 1.2 0 1.1 z 0.9 0.8 Possible Forms for the Function r 0 0.25 0.50 0.75 Fraction of police workers, z Hg. 3. The functional dependence of colony efficiency. rz. on the fraction of workers that are policing. z, may take any one of many possibilities. of killing eggs, police workers are directly diminishing the number of potential offspring. In the process of identifying and killing nephews, police workers may also be expending energy that could otherwise be spent on important colony maintenance tasks (Cole. 1986: Naeger et al.. 2013). Policing can also be costly if there are recognitional mistakes. Le, queen-laid eggs may accidentally be removed by workers. Recognitional errors could result in modifications to the sex ratio, which is an important extension of our model but is beyond the scope of this paper. We can also identify positive effects that policing may have on colony efficiency. It has been hypothesized that the eggs which are killed by police workers may be less viable than other male eggs (Velthuis et al.. 2002: Pirk et al., 1999: Cadagkat 2004: Nonacs. 2006), although this possibility has been disputed (Beekman and Oldroyd. 2005: Helantera et al., 2006: Zanette et al., 2012). If less- viable worker-laid eggs are competing with more-viable queen-laid male eggs, then policing may contribute positively to overall colony efficiency. Moreover, policing decreases the incentive for workers to expend their energy laying eggs in the first place (Foster and Ratnieks, 2001a: Wenseleers et al.. 2004a,b: Wenseleers and Ratnieks, 2006a), which could be another positive influence on colony efficiency. (However, the decrease in incentive for workers to reproduce due to policing would only arise on a short time scale if there is a facultative response to policing, which is unlikely.) As another speculative possibility: Could it be that worker egg- laying and subsequent policing acts as a form of redistribution within the colony? That is, suppose that it is better for colony efficiency to have many average-condition workers than to have some in poor condition and some in good condition. Suppose further, as seems realistic, that good-condition workers are more likely to lay eggs (which are high in nutritional content. of course) If the average police worker is of condition below the average egg- laying worker, then worker egg-laying and policing serves to redistribute condition among the workers, improving overall col- ony efficiency. The special case, where policing has no effect on colony effi- ciency and which has informed the conventional wisdom. is ungeneric, because policing certainly has energetic consequences for the colony that cannot be expected to balance out completely. An early theoretical investigation of colony efficiency effects regarding invasion of dominant mutations that effect worker policing was performed by Ratnieks (1988). Although monotonically increasing or monotonically decreas- ing functions rz are the simplest possibilities, these cases are not exhaustive. For example, a small or moderate amount of policing may be expected to improve colony efficiency. However, the pre- cise number of police workers that are needed to effectively police the entire worker population is unclear. It is possible that a frac- tion ze 1 of police workers can effectively police the entire population, and adding additional police workers beyond a certain point could result in wasted energy, inefficient use of colony resources, additional recognitional errors. etc. These effects may correspond to colony efficiency rz reaching a maximum value for some 0 < z < 1. As another possibility, suppose that police workers, when their number is rare, directly decrease colony efficiency by the act of killing male eggs. It is possible that for some z < 1. police workers are sufficiently abundant that their presence can be detected by other workers. Assuming the possibility of some type of facultative response, the potentially reproductive workers may behaviorally adapt by reducing their propensity to lay male eggs. instead directing their energy at raising the queen's offspring. In this scenario. colony efficiency rz may reach a minimum value for some 0 ez e 1. 2.3. Main results for dominant police alleles We derive the following main results for dominant police alleles. If the queen mates with n males, then the a allele for policing can invade an A resident population provided the fol- lowing -evolutionary invasion condition" holds: Ptin +Pia Cm) (rig) >2_ CI /2) _ _ Cin) (1) ro ro ro Pito ro When considering only one mutation. ro can be set as 1 without loss of generality. Why are the four parameters. rim. rla, pun. and pia. sufficient to quantify the condition for invasion of the mutant allele, a? Since we consider invasion of a. the frequency of the mutant allele is low. Therefore, almost all colonies are of type AA, 0. which means a wild-type queen. AA, has mated with n wild- type males. A, and 0 mutant males, a. In addition, the colonies Act 0 and AA, 1 are relevant These are all colony types that include exactly one mutant allele. Colony types that include more than one mutant allele (such as Aa, 1 or AA,2) are too rare to contribute to the invasion dynamics. For an Ac. 0 colony, half of all workers are policing, and therefore the parameters r112 and p1/2 occur in Eq. (1). For an AA. 1 colony, 1/n of all workers are policing, which explains the occurrence of run and pim in Eq. (1). Next. we ask the converse question: What happens if a popu- lation in which all workers are policing is perturbed by the introduction of a rare mutant allele that prevents workers from policing? If the a allele for worker policing is fully dominant. and if colony efficiency is affected by policing, then a resident policing population is stable against invasion by non-police workers if the following 'evolutionary stability condition" holds: +n)(2 +Pi ) + Pan - oita,o(n —2) (2) 2(2+n+npi) roan-mart What is the intuition behind the occurrence of the four para- meters, ri, ran _ wan). Pt. and Pan-102M? The condition applies to a population in which all workers are initially policing. Note that. because the allele. a, for policing is fully dominant in our treat- ment. non-policing behavior arises if at least two mutant A alleles for non-policing are present in the genome of the colony, which is the combination of the queen's genome and the sperm she has stored. To study the invasion of a non-policing mutant allele. we must consider all colony types that have 0.1. or 2 mutant A alleles: these are aa.n: aa,n — 1: Aa,n: n-2: Aa,n-1: and AA, n. The colonies aa.n: mai —1: Aa,n: aa, n —2: and AA, n do not contain non-police workers: the efficiency of those colonies is r1, and the fraction of male eggs that originate from the queen in those colonies is pi. Both of these parameters occur in Eq. (2). Colonies of type At n —1 produce a fraction of 1/(2n) non-police workers. EFTA01071000 J.W. Oleforz er al. /Journal of theoretical Biology 399 (2016) M3-116 107 Single mating. n=1 10 10 1 0 To 104 '1104 .6 10 4 4 1. 10 ! 1.1. 104 .7 Double mating, n=2 2 Time (104) 6 Fig. a Numerical simulations of the evolutionary dynamics of worker policing confirm the condition given by Eq. (1). The policing allele is dominant. For numerically probing invasion, we use the initial condition XA40 —1-10 s and Kai — 10 a. We set ra —1 without loss of generality. Other parameters are: (a) pi,2 —0.75, pi —0.9, and rig-1A1: (b)pla —0.6, p,—0.8. — 1.005, and r,-1.01. 0 Police allele can invade and is evolutionarily stable can invade, but is not stable cannot invade. but is stable cannot invade and is not stable 1 kequency of police allele S. There are four possibilities for the dynamical behavior in the proximity of two pure equilibria a n= 1 mating 1.2 Does Not Invade Stable a 8 0.9 0.8 08 0.9 1 1.1 12 Colony efficiency. r„2 P1/2 = 0.75. p1 =1 Stable Does Not Invade Unstable Invades Unstable Colony efficiency. r1 b n = 1 mating 1.2 1.1 1 0.9 0.8 08 Does Not Invade Stable pv2 = 0.99. Pi = I Invades Stable Does Not Invade Invades Unstable 0.9 1 1.1 Colony efficiency. r112 12 Mg. 6. If queens are singly mated (n-1). then a plot of ri versus rin clearly shows all four possibilities for the behavior around the two pure equilibria. For (a). we set pi,2 —0.75 and p, — I. For (b), we set pc.2 —0.99 and pi — I. which explains the occurrence of rota _ Iva," and pan _ i)!2,0 in Eq. (2). Numerical simulations of the evolutionary dynamics with a dominant police allele are shown in Fig. 4. Generally, four scenarios regarding the two pure equilibria are possible: Policing may not be able to invade and be unstable, policing may not be able to invade but be stable, policing may be able to invade but be unstable, or policing may be able to invade and be stable. The possibilities are shown in Fig. 5. In the cases where policing cannot invade but is stable, or where policing can invade but is unstable, Brouwer's fixed-point theorem guarantees the existence of at least one mixed equilibrium. In the case where policing can invade but is unstable, police and non-police workers will coexist indefinitely. We will now discuss the implications of our results for parti- cular numbers of matings. EFTA01071001 108 Okjarz er aL /journal of theoretical Biology 399 (2016)103-n6 a The police allele invades and is stable. p„.2=0.75. rue 1.0344. r1=1.0767 Frequency of policing allele, a 2' 2 LL 1 2 Time (101) 3 4 The police and non-police alleles are bistable. pw=0.75, p1=1. r„2=1.0244. r1.1.0667 1 0.8 0.8 0.4 0.2 1 10 15 Time (10) b to 0.8 co :9. • 0.6 o 1 0.4 8. 0.2 d 1 0.8 0.8 8 0.4 0.2 LL The non-police allele invades and is stable. p.„2=0.75, p1=1, r„e1.0344, r,=1.0567 82 84 86 88 Time (101) The police and non-police alleles coexist. pm l3.75. p1=1, re 1.0444. r1=1.0667 OB 8 2 4 Time (104) 10 Fig. 7. Numerical simulations of the evolutionary dynamics of worker policing that show the four behaviors in fig. 6(a). The policing allele is dominant. For each of the four panels. we use the initial conditions: la) Xm,0 and443 -10-1; (b)X„,2 — 1 —10"1 and X„0 (C)440 0.02 and X2A.I — 0.98 (lower cum), and XAA0 —0.01 and X,,,„, —0.99 (upper cum): (d)X.„,„ — 1-10-2 and X.,," — 10-2 (lower curve), and X„,,, —1-10 2 and Xmo —10 2 (upper curie) We set re —1 without loss of generality. 1.003 1.002 1.001 1 O O 0.999 0.998 O 1/2 Fraction of poice workers, Fig a Possible rz efficiency curves for n-1 mating which demonstrate different behaviors. For this plot. we set pi.,7 0.99 and p, — 1. Here, each curve has the functional form r, — I + art/IS. For example. we can have: (blue) policing invades but is unstable, a—0.003, p--0.O004: (green) policing invades and is stable, a—0.0026,1-0: (red) policing does not invade and is unstable, a — 0.0024. p—0: (black) policing does not invade but is stable, a—0.002. 0-0.0004. (For inter- pretation of the references to color in this figure caption, the reader is referred to die web version of this paper.) n = 1 mating 3. Single mating 1 For single mating. n=1. the invasion condition for a dominant police allele is > 2(1 -1:01)+(P +P tp) p 2(2 — rip) (Recall that r0 = 1.) (3) The stability condition for a dominant police allele is 6 —pia +3pt r 1 > (4) 6+2p1 Evolution of policing is highly sensitive to changes in colony effi- ciency. For example, let us consider p in = 0.99 and pi =1. This means that if half of all workers police then 99% of all males come from the queen. If all workers police then all males come from the queen. In this case, efficiency values such as r112 = 1.001 and rt = t0031 lead to the evolution of policing. In principle, arbitrarily small increases in colony efficiency can lead to the evolution of policing for single mating. A plot of rt versus r112 for singly mated queens (Fig. 6) illus- trates the rich behavior highlighted in Fig. 5. Numerical simula- tions of the evolutionary dynamics are shown in Fig. 7. Another intriguing feature is that increases in colony efficiency due to policing do not necessarily result in a higher frequency of police workers at equilibrium Fig. 8 illustrates this phenomenon. Four possibilities for the efficiency function rz are shown. Notice that the r, curve which results in coexistence of police workers and non- police workers (blue, top) is strictly greater than the rz curve which results in all workers policing (green, second from top). How can increased efficiency due to policing possibly result in policing being less abundant at equilibrium? If a mutation for non-policing behavior is introduced into a resident policing population, then the evolu- tionary success of the non-policing mutation depends on the success of M. 0 colonies relative to aa. 1, aa.0, Aa, 1, and AA, 1 colonies. Aa. 0 colonies have an efficiency parameter r 112. while the other four relevant colonies each have an efficiency parameter rt. Thus, if r112 is too large relative to rt, then the non-police allele can invade a resi- dent policing population, and there is coexistence. Also notice that the rz curve which results in bistability of police workers and non-police workers (black, bottom) is strictly less than EFTA01071002 J.W. OleJarz er al. /Journal of Theoretical Biology 399 (2016) 103-116 n = 2 matings 109 n=3 matings 1.003 1.003 POIleng InvedeS but IS Pa cog yaks INA is unstable °Datable Poems' 1.002 1.002 POrcna ades and is stable Porteng does not invade and is unstable invades and is stable - P011eng We, nOlInvaile BM unSbable PoIrceig does but PcI cog does not invade txt 4 Stable not invade is stable 1.001 ro 1.001 1 1 g. O 8 0.999 0 0 0.999 0.998 0.998 0 1/2 3/4 1 0 1)3 112 (213) 516 1 Fraction of police workers, z Fig. 9. Possible r, efficiency curves for n-2 matings which demonstrate different behaviors. Here, each curve has the functional form r, — az+rs.z2. For example, we can have: (blue) poking invades but is unstable. a-0.0005. p--0.0004: (green) policing invades and is stable. a-0.0001, p-0; (red) policing does not invade and is unstable, a — 0.0001, 0-0; (black) policing does not invade but is stable, a — -0.0005.p-0.0004. (For interpretation of the references to color in this figure caption, the reader is referred to the web version of this paper.) the r, curve which results in policing being dominated by non-policing (red, second from bottom). This phenomenon arises in a similar way: if ri ;2 is too small relative tor,, then the non-police allele cannot invade a resident policing population, and there is bistability. 4. Double mating For double mating. n=2. the invasion condition for a dominant police allele is given by ria >1 (5) Thus, policing can invade if there is an infinitesimal increase in colony efficiency when half of all workers police. Policing cannot invade if there is an infinitesimal decrease in colony efficiency when half of all workers police. The stability condition for policing is given by rl > F3/4 (6) Therefore, the policing allele is stable if the colony efficiency is greater for z=1 (when all workers police) than for z= 3/4 (when only three quarters of the workers police). Four possible efficiency curves r, and the corresponding behavior of the police allele are shown in Fig. 9. 5. Triple mating For triple mating n=3. the invasion condition for a dominant police allele is given by 4- 20 - PO115 rig> (7) 2+(Pia+Ptp), 113 The stability condition for policing is given by 10+Josio+Spi ri > F5/6 10+6pi (8) As a numerical example, let us consider pv3=0.98 and p112 =0.99. If z= 1/3 of workers police, then 98% of males come from the queen. If z = 1/2 of workers police, then 99% of males come from the queen. In this case, policing cannot invade if ri/3 = 0.9990 and ria =0.9979. In principle, arbitrarily small reduc- tions in colony efficiency can prevent evolution of policing for triple mating. Fraction of police workers, z Fig. 10. Possible rt efficiency curves for n-3 matings which demonstrate different behaviors. For this plot we set pl.., — 0.986, p,, a — 0.99, p, — 0.996, and pi —1. Here, each curve has the functional form r, -1 + a2 pz2. For example. we can have: (blue) policing invades but is unstable. a— —0.0006. p- -0.0006: (green) policing invades and is stable, a--0.0012, p-0; (red) policing does not invade and is unstable, a - 0.0015, 0-0; (black) policing does not invade but is stable, a- -0.0021, /1-0.0006. Note that the value r2,., affects the population dynamics but does not appear in the conditions for invasion and stability of the police allele, hence the parentheses on the horizontal axis. (For interpretation of the references to color in this figure caption, the reader is referred to the web version of this paper.) Just as for single mating we observe the intriguing feature that increases in colony efficiency due to policing do not necessarily result in a higher frequency of police workers at equilibrium. Fig. 10 illus- trates this phenomenon. Four possibilities for the efficiency function r, are shown. Notice that the r, curve which results in coexistence of police workers and non-police workers (blue, top) is strictly greater than the r, curve which results in all workers policing (green, second from top). Also notice that the r, curve which results in bistability of police workers and non-police workers (black, bottom) is strictly less than the r, curve which results in policing being dominated by non- policing (red, second from bottom). 6. Recessive police allele We have also derived the conditions for the emergence and evo- lutionary stability of worker policing if the police allele is fully reces- sive. In this case. M and Aa workers are phenotypically identical and do not police, while aa workers do police. (Alternatively, M and Aa workers police with intensity 44= Zilch while aa workers police with intensity Zoo =Zvi+ W = ZA0 -1-w. We consider this case in Section 8.) 6.1. Emergence of worker policing The invasion condition for a recessive police allele, a, is given by Two > 2(2+n+npo) ro (2 +n)(2 +Po) +Pin2A)(n — 2) Note that Eq. (9) for invasion of a recessive police allele has the same mathematical form as Eq. (2) for evolutionary stability of a dominant police allele. Starting from Eq. (2), making the substitution z -.1-z. and reversing the inequality, we recover Eq. (9). The intuition behind this correspondence is described in Appendix A. 62. Stability of worker policing A recessive police allele, a. is evolutionarily stable if [2( \ 11 (1 pi ri >cfa_iv.+pia Vrn -run/ I. Via./ ' kria) 2 Note that Eq. (10) for evolutionary stability of a recessive police allele has the same mathematical form as Eq. (1) for invasion of a (9) (10) EFTA01071003 110 J.W. OkJazz et al. &Journal of theoretical Biology 399 (2016) 103-116 Single mating, n=1 b le To To 2 rr Double mating, n=2 - ru4=1.002 - ru4=1.001 - r =0.999 - r =0.998 16 20 0 8 12 16 20 Time (105 F1/4 .11. Numerical simulations of the evolutionary dynamic of worker policing confirm the condition given by Eq. (9). The policing allele is recessive. For numerically probing invasion, we use the initial condition XAm, - 1 -10-2 and XAki - 10-e. We set I'D 1 without loss of generality. Other parameters are: (a)p0-0.6. en -0.8, and 6-1.06: (b) Po- 035. R14 —0.9. ri,2 - recra, and G -1.012. Under what conditions does worker policing invade? n a 1 Mathlg n = 2 footings as 3 matings is constant (equal to 1) NEUTRAL a decreases monotonically YES Increases monotonically NO YES YES a YES YES YES reaches a maximum for scaitt0<z< I NO NO NO YES NO YES YES YES reaches a minimum for some 0< z <1 NO YES NO NO NO ki tL YESA YES YES NO NO YES NO YES NO YES YES NO NO NO YES YES YES NO YES Fig. 12. Depending on the funttional form of colony efficiency. G. on the fraction of police workers. z. policing alleles may or may not invade for single, double, of triple mating. Various possibilities of r, are shown. The outcomes hold for both dominant and recessive police alleles. If G is constant, then policing does not invade for single mating, is neutral for double mating and invades for triple mating. If rz decreases monotonically, then policing does not invade of invades only for triple mating, If i, increases monotonically. then policing either invades only for double and triple mating or for single, double, and tnpk mating. If G reaches a maximum at an intermediate value 0 <z < 1, then policing does not invade of may invade for tnpk mating only. for double and triple mating or for single, double, and triple mating, If rz reaches a minimum at an intermediate value 0 <z < 1. then any pattern is possible. dominant police allele. Starting from Eq. (1). making the sub- stitution z-.1-z. and reversing the inequality, we recover Eq. , 10 '. Again, the intuition behind this correspondence is described in Appendix A. Numerical simulations of the evolutionary dynamics with a recessive police allele are shown in Fig. 11. 7. Shape of the efficiency function, rj, The shape of the efficiency function. rz. determines whether policing is more likely to evolve for single mating or multiple matings. Recall that r, is the colony efficiency (defined as the rate of generation of reproductives) if a fraction, z. of all workers per- form policing. The variable z can assume values between 0 and 1. If no workers police. z=0. then the colony efficiency is at baseline. which we set to one: therefore, we have ro =1. Policing can in principle increase or decrease colony efficiency (Fig. 12). We have the following results regarding the invasion and sta- bility of police workers. We discuss single (n=1). double (n=2). and triple (n=3) mating. All results apply to both dominant and recessive police alleles. They can be instantiated with arbitrarily small changes in colony efficiency. 7.1. Evotutionary invasion of policing (i) If r2 is strictly constant (which is ungeneric), then policing does not invade for single mating, is neutral for double mating, and does invade for triple mating. (ii) If rz is monotonically decreasing, then policing either invades not at all or only for triple mating. (iii) If rz is monotonically increasing, then policing either invades for single, double, and triple mating or only for double and triple mating. (iv) If rz reaches an intermediate maximum (which means colony efficiency is highest for an intermediate fraction of police EFTA01071004 Olefarz er al. /Journal of Theoretkaf Biology 399 (2016) 103-116 a 1.012 1.010 1.088 1.006 1.004 _g 1.002 8 0.998 0 Policing invades for n=1 but not for n=2 - • • • 1/2 3/4 Fraction of police workers.: Policing invades but is unstable (for n=1 and n=2) 1 b g. 0.999 1 0.998 6 0.997 d 0.996 Policing does not evade but is stable (for n=1 and n=2) 0 1/2 3/4 Fraction of police workers.: Policing does not invade but is stable (fix n=1) Policing invades but is unstable (for n=2) 1.002 • • 1.003 C • g .2 1.002 € 1.001 0 1.001 1 - • 0 1/2 3/4 Fraction of police workers.: 0 1/2 314 Fraction of police workers,: Fig. 13. Non-monotonic efficiency funcnons can lead to rich and counterintuitive behavior. We consider invasion and stability of a dominant police allele for single (n-1) and double (n-2) mating. The baseline colony efficiency without policing is ro — 1.Three other values must be specified: r,„2, 44. and r1. Moreover, we need to specify two values for how the presence of police workers affects the fraction of male offspring coming from the queen: we choose oil —0.99 and p, — I. A variety of behaviors can be realized by a very small variation in colony efficiency. (a) Policing invades for single mating but not for double mating. (b) Policing does not invade but is stable for single and double mating. (c) Policing invades but is unstable for single and double mating. (d) Policing does not invade but is stable for single mating, while policing invades but is unstable for double mating. workers), then policing can invade for n = 1,2,3 or n=2.3 or n=3 or not at all. (v) If rz reaches an intermediate minimum (which means colony efficiency is lowest for an intermediate fraction of police workers), then policing can invade with any pattern of mat- ings. For example, it is possible that policing invades only for single mating but neither for double nor triple mating. Or it invades for single and double mating but not for triple mating. 72. Evolutionary stability of policing (i) If rz is constant, then policing is unstable for single mating, is neutral for double mating, and is stable for triple mating. (ii) If rz is monotonically decreasing, then policing is unstable for single and double mating. For triple mating it can be stable or unstable. (iii) If rz is monotonically increasing, then policing either is always stable or is stable only for double and triple mating. (iv) If rz reaches an intermediate maximum, then policing can be stable for any pattern of matings. For example, policing can be stable for single mating but neither for double nor triple mating. (v) If rz reaches an intermediate minimum, then policing can be stable for n = 1.2.3 or n=2,3 or n=3 or not at all. 7.3. Examples for single and double mating Fig. 13 gives some interesting examples for how non- monotonic efficiency functions can influence the evolution of policing for single (n=1) and double (n=2) mating. In order to discuss the invasion and stability of a dominant police allele for single and double mating, we need to specify efficiency at three discrete values for the fraction of police workers present in a colony: ria. r314. and r 1. Note that ro =1 is the baseline. Moreover. we need to specify the fraction of male offspring coming from the queen at two values: po and pi. For all examples in Fig. 13, we assume pla =0.99 and pi = 1. We show four cases: (a) policing invades for single mating but not for double mating: (b) for both single and double mating. policing does not invade but is stable: (c) for both single and double mating, policing invades but is unstable (leading to coexistence of policing and non-policing alleles): (d) policing does not invade but is stable for single mat- ing: policing invades but is unstable for double mating. These cases demonstrate the rich behavior of the system, which goes beyond the simple view that multiple matings are always favor- able for the evolution of policing. 8. Gradual evolution of worker policing Our main calculation applies to mutations of any effect size. In this section, we calculate the limit of incremental mutation (small mutational effect size). Our calculations in this section are remi- niscent of adaptive dynamics (Nowak and Sigmund, 1990; Hof- bauer and Sigmund, 1990; Dieckmann and Law, 1996; Metz et al., 1996; Geritz et al., 1998), which is usually formulated for asexual and haploid models. The analysis in this section applies both to the case of small phenotypic effect and to the case of weak penetrance. Mathematically, we consider the evolutionary dynamics of poli- cing if the phenotypic mutations induced by the a allele are smalL If an allele affecting intensity of policing is dominant, then it is intui- tive to think of wild-type workers as policing with intensity 4.,, EFTA01071005 112 Okjorz et al /journal of theorerkol Biology 399 (2016) 10-116 while mutant workers police with intensity Im =Zoo =ZoAl-w. If an allele affecting intensity of policing is recessive, then it is intuitive to think of wild-type workers as policing with intensity ZM =140, while mutant workers police with intensity 1,0 = ZAA = ZA0+ w. In the limit of incremental mutation, the fraction, p. of queen- derived males and the colony efficiency. r, become functions of the average intensity of policing in the colony, which is Z+wz, where z is the fraction of mutant workers in the colony. We have -. p(Z+ wz)= P(z)+PV)wz+1 PV)w22+0(w3) r,-.R(Zl-wz)= Rail+ K(Z)wz +1 R.(Z)w2z2 +O(w3) (11) We have made the substitutions p,—.P(Zl-wz) and r1—.5t(Z+wz). and (11) gives the Taylor expansions of these quantities in terms of their first and second derivatives at intensity 1 (For conciseness, we will often omit the argument Z from the functions P and R and their derivatives.) Here. I wi 41. so that workers with the phenotype corresponding to the mutant allele only have an incremental effect on colony dynamics. Thus, the expansions (11) are accurate approximations. We assume that P > 0. The sign of w can be posi- tive or negative. If w is positive, then the mutant allele's effect is to increase the intensity of policing. If w is negative, then the mutant allele's effect is to decrease the intensity of policing. Note that this formalism could also be interpreted as describing the case of weak penetrance, in which only a small fraction of all workers that have the mutant genotype express the mutant phenotype. For considering the dynamics of a dominant police allele with weak phenotypic mutation, we introduce the quantity Qom= Pun +Pi a 01 Oa) [2 _ trial _ _Pim) ( .1 1. )1 (12) 2 ro 11/4 re / ro I f ro If Corm > 0. then increased intensity of policing is selected, and if Com <0. then increased intensity of policing is not selected. This is just a different way of writing (1). We substitute (11) into (12) and collect powers of w. To first order in w we get Cdom = Wr-2)KRA-2(2+n+nP)R1 4nR +O(w2) (13) For considering the dynamics of a recessive police allele with weak phenotypic mutation, we introduce the quantity Aram 2(2-i-nl-npo) Crec — ro (2+n)(2 +Po)+Pita,o(n —2) (14) If Crec > a then increased intensity of policing is selected, and if Crec < 0, then increased intensity of policing is not selected. This is just a different way of writing (9). We substitute (11) into (14) and collect powers of w To first order in w. we get Cre, — —2)PRA-2(2+n-EnP)K1 4nR(2+n+nP) +O0412) (15) Notice that (13) and (15) are, up to a multiplicative factor, the same to first order in w Using Eqs. (13) and (15), the condition for policing to increase from a given level Z is 1O2.) > -1n-2) R(Z) P(Z) 2(2+ n+nP(Z)) Policing decreases from a given level Z if the opposite inequality holds. We have explicitly written the Z dependencies in Eq. (16) to emphasize that the quantities P.P.& and K are all functions of the intensity of policing, Z. The left-hand side of Eq. (16) can be understood as a ratio of marginal effects. To be specific, the left-hand side gives the ratio of the marginal change in efficiency over the marginal increase in the proportion of queen-derived males, if policing were to increase by (16) a small amount. For selection to favor increased policing, this ratio of marginals must exceed a quantity depending on the current values of R and P. Notice that the sign of the right-hand side is determined by n-2. So we get different behavior for different numbers of matings: • For n=2 (double mating), policing increases from Z if and only if R'(Z)> 0. This means that evolution maximizes the value of it, regardless of the behavior of P. In other words, for double mating, evolution maximizes colony efficiency regardless of the effect on the number of queen-derived males. • For n=1 (single mating), the right-hand side of Eq. (16) is positive. So the condition for Z to increase is more stringent than in the n=2 case. Increases in policing may be disfavored even if they increase colony efficiency. • For n a 3 (triple mating or more than three matings), the right- hand side of Eq. (16) is negative. So the condition for Z to increase is less stringent than in the n=2 case. Any increase in policing that improves colony efficiency will be favored, and even increases in policing that reduce colony efficiency may be favored. Eqs. (13) and (15) also allow us to determine the location(s) of evolutionarily singular strategies (Geritz et al.. 1998). Intuitively, a singular strategy is a particular intensity of policing, denoted by r. at which rare workers with slightly different policing behavior are, to first order in w, neither favored nor disfavored by natural selection. The parameter measuring intensity of policing. Z. can take values between 0 (corresponding to no policing) and 1 (cor- responding to full policing). There are several possibilities: There may not exist a singular strategy for intermediate intensity of policing: in this case, there is either no policing (Z* =0) or full policing (r = 1). If there exists a singular strategy for 0 <Z• <1. then there are additional considerations: There may be convergent evolution toward intensity Z" or divergent evolution away from intensity Z. In a small neighborhood for which Za-a*, further analysis is needed to determine if the singular strategy corre- sponding tor is an ESS. To determine the location(s) of evolutionarily singular strate- gies. we set the quantity in square brackets that multiplies w in (13) and (15) to zero, yielding Kr) +(n-2) (Z*) —0 (17) per) 2(2+n A- nP(n) Eq. (17) gives the location(s) of singular strategies for both domi- nant and recessive mutations that affect policing. For a given singular strategy r. there is convergent evolution toward r if el [K(Z)+(n 2 R(Z) 1 <0 dZ P(Z) 2(2+ n+nP(Z)) .1 There is divergent evolution away from r if the opposite inequality holds. It is helpful to consider some examples. If the functions P(Z) and R(Z) are known for a given species, then the behavior of worker policing with gradual evolution can be studied. It is pos- sible that policing is at maximal intensity. r =1 (Fig. 14(a)). is nonexistent, r =0 (Fig. 14(b)). is bistable around a critical value of intensity. 0 e r < 1 (Fig. 14(c)). or exists at an intermediate value of intensity, 0 <Z. < I (Fig. 14(d)). Note that a singular strategy may or may not be an evolutio- narily stable strategy (ESS). (For example, it is possible that there is convergent evolution toward a particular singular strategy r which is not an ESS. In this case, once .zzo. evolutionary EFTA01071006 J.W. OleJarz et al. /puma/ of Theoregicaf Biology 399 (2016) 103-116 113 a 120 1.15 = 1.10 g L05 w I 1 O 7. 0.8 0.6 2 c 0.4 - 0.2 0 C S 120 1.15 1.10 ig 1.05 ILI 0.8 0.6 0.4 g 0.2 There is full policing. (I-1) 0.2 0.4 0.6 0.8 Intensity of policing. Z There is histabdity. (1-113) CIO 0.2 0.4 0.6 0.8 Intensity of policing, Z b S 1.20 re • 1.15 1.10 g Los 1 0.8 0.6 8 0.4 2 & 0.2 00 There is no policing. (1.0) 0.2 0.4 0.6 0.8 Intensity of policing, Z d There is intermediate policing. (1.0.7986...) S 1.20- a 1.15 - 11.10- g 1 IF; 0.8 °'. 0.6 I 0.4 2 0.2 ° O 02 OA 0.6 0.8 Intercity of policing. Z Fig. 14. Several simple examples of functions P(Z) and R(Z) are shown. For single mating, the corresponding dynamics of policing intensity with gradual evolution are also shown. We use the forms FM- 1 -IF +102 and R(Z)-l+CIZI-(1/2)C2.22. For each of the four panels, we set: (a) fr -0.5, C, -0.2, C2 0, corresponding to r -1: (b) P - OZ. CI -0.1, C2 -0, corresponding to 2" - (c) P -0.8, Ci -0.12, C2 -0, corresponding to bistability around Zs - 1/3: (d)f - - 0.2. C2 - 0.18, corre- sponding to an intermediate level of policing around r ft 0.7986.... branching may occur: Geritz et al., 1998) To determine if (17) is an ESS, we must look at second-order terms in (12) and (14). For a dominant police allele, we return to (12) with the sub- stitutions (11). We focus on a singular strategy given by (17). For a singular strategy, Cdom is zero to first order in w. To second order in w. we get Cam = (n2 — 4)P.R2 + 2(n2 + 4n —4),1212 w2 16n2R2 8nPR42 +2(n2 + n2P+4)R-R1 +Ow') 16n2R2 (18) We may alternatively write (18) by substituting for RP using (17): Qom w2[(2+n+nP)2((n2 —4)" R +2(n2 n2P 4)12. 16n2R(2+n+nP)2 (n —4)(n2+n2P+4n-4)PeR +000) (19) 16n2R(2+n+nP)2 For a recessive police allele, we return to (14) with the sub- stitutions (11). We focus on a singular strategy given by (17). For a singular strategy, Crec is zero to first order in w. To second order in w. we get crec [(n — 2)(2 +n + nP)P R — (n — 2y2P2R = 16n2R(2 +n + nP)2 2(2 + n + nP)2R. +O(W3) (20) 16n2R(2+n nP)2 Inspection of (18) and (20) allows us to determine if a singular strategy is an ESS. If the bracketed quantity multiplying w2 is negative, then mutations that change policing in either direction are disfavored. If the bracketed quantity multiplying w2 is positive, then mutations that change policing in either direction are favored. Thus, for a dominant allele that affects intensity of policing, the singular strategy (17) represents a local ESS if (n2 —4)1:Die + 2(n2 +4n —4)Pfrft+8nW2+2(n2+ n2P+4)R-R <0 (21) We may alternatively write (21) by substituting for R' using (17): (2 +n +nP)2[(n2 — 4)P.R + 2 (n2 +rt2P+4)r] — (n2 —4) (n2 A- n2P+ 4n — 4)112R <0 (22) Similarly, for a recessive allele that affects intensity of policing, the singular strategy (17) represents a local ESS if (n —2)(2+n+ nP)P. R — (n — 2)20 /2+2(2 + n + nP)2R. <0 (23) Here, P. P. R. R. and 12' are all functions of the intensity of policing, Z The local ESS conditions (22) and (23) are quite opaque and do not allow for simple analysis. Notice that, although the locations of evolutionarily singular strategies are the same for dominant and recessive mutations that influence policing, the conditions for a singular strategy to be a local ESS are different 9. Policing and inclusive fitness theory It has been claimed that policing is a test case of inclusive fit- ness theory (Abbot et al.. 2011). But the first two papers to theo- retically establish the phenomenon (Woyciechowski and Lomnicki, 1987: Ratnieks. 1988) use standard population genetics: they do not mention the term "inclusive fitness", and they do not calculate inclusive fitness. Therefore, the claims that theoretical investiga- tions of worker policing emerge from inclusive fitness theory or that empirical studies of policing test predictions of inclusive fit- ness theory are incorrect In light of known and mathematically proven limitations of inclusive fitness theory (Nowak et al., 2010: Allen et al., 2013). it is unlikely that inclusive fitness theory can be used to study general EFTA01071007 114 J.W. Odejorz er aL /puma( of Theorerkol Biology 399 (2026) la3-116 questions of worker policing. Inclusive fitness theory assumes that each individual contributes a separate, well-defined portion of fitness to itself and to every other individual. It has been shown repeatedly (Cavalli-Sforza and Feldman, 1978; Uyenoyama and Feldman, 1982; Matessi and Karlin, 1984; Nowak et al., 2010; van Veelen et al.. 2014). that this assumption does not hold for general evolutionary processes. Therefore, inclusive fitness is a limited concept that does not exist in most biological situations. Our work shows that the evolution of worker policing depends on the effectiveness of egg removal (p1) and the consequences of colony efficiency (rz). Each of these effects can be nonlinear (not the sum of contributions from separate individuals), with impor- tant consequences for the fate of a policing allele. Moreover, the invasion and stability conditions involve the product of p- and r- values, indicating a nontrivial interaction between these two effects which does not reduce to a simple sum of costs and ben- efits. We also found that there are separate conditions for invasion and stability, with neither implying the other. Inclusive fitness theory, which posits a single, linear condition for the success of a trait, is not equipped to deal with these considerations. Attempts to extend inclusive fitness theory to more general evolutionary processes (Queller. 1992; Frank, 1983; Gardner et al., 2011) rely on the incorrect interpretation of linear regression coefficients (Allen et al., 2013; see also Birch and Okasha, 2014). This misuse of statistical inference tools is unique to inclusive fitness theory, and differs from legitimate uses of linear regression in quantitative genetics and other areas of science. It was also recently discovered that even in situations where inclusive fitness does exist, it can give the wrong result as to the direction of nat- ural selection (Tarnita and Taylor, 2014). Relatedness-based arguments are often seen in conjunction with inclusive fitness, but there is a crucial difference. Consider the following statement: if the queen is singly mated, then workers share more genetic material with sons of other workers than with sons of the queen. This statement is not wrong and could be useful in formulating evolutionary hypotheses. Such hypotheses can then be checked using exact mathematical methods. The problem arises when one attempts to formulate the quantity of inclusive fitness by partitioning fitness into contribu- tions from different individuals and reassigning these contribu- tions from recipient to actor. A worker does not make separate contributions to fitnesses of others. and therefore does not have Inclusive fitness". Arguments such as "the worker maximizes her inclusive fitness by not policing' are meaningless. since they are based on maximizing a nonexistent quantity. Moreover, even when evolution leads individuals to maximize some quantity, that quantity is not necessarily inclusive fitness (Okasha and Martens, 2015; Lehmann et al., 2015). It is true that genes (alleles) can be favored by natural selection if they enhance the reproduction of copies of themselves in other individuals. But that argument works out on the level of genes and can be fully analyzed using population genetics. Inclusive fitness only arises when the individual is chosen as the level of analysis. which is a problematic choice for many cases of complex family or population structure (Akcay and Van Cleve. 2016). Bourke (2011) has proposed that inclusive fitness remains valid as a concept even when it is nonexistent as a quantity. But why is such an uninstantiable concept useful? The mathematical theory of evolution is clear and powerful Exact calculations of evolutionary dynamics (Antal et al.. 2009; Allen and Nowak, 2014; Fu et al.. 2014; Hauert and Doebeli, 2004; Szabo and Fath, 2007; Antal and Scheuring, 2006; Traulsen et al., 2008; van Veelen et al.. 2014; Simon et al.. 2013) demonstrate that inclusive fitness is not needed for understanding any phenomenon in evolutionary biology. This realization is good news for all whose primary goal is to understand evolution rather than to insist on a particular method of analysis. By releasing ourselves from the confines of a mathematically limited theory, we expand the possibilities of scientific discovery. 10. Discussion We have derived analytical conditions for the invasion and stability of policing in situations where queens mate once or several times and where colony efficiency can be affected by policing. In the special case where policing has no effect on colony efficiency, our results confirm the traditional view that policing does not evolve for single mating, is neutral for double mating, and does evolve for triple mating or more than three matings. If colony efficiency depends linearly or monotonically on the fraction of workers that are policing, then our results support the view that multiple mating is favorable to evolution of policing (Ratnieks. 1988). Our results also show that non-monotonic relations in colony dynamics and small changes in colony efficiency necessi- tate a more careful analysis. We find that policing can evolve in species with singly mated queens if it causes minute increases in colony efficiency. We find that policing does not evolve in species with multiply mated queens if it causes minute decreases in colony efficiency. For non- monotonic efficiency functions, it is possible that single mating allows evolution of policing, while multiple mating opposes evo- lution of policing. Our analysis is the first to give precise conditions for both the invasion and stability of policing for both dominant and recessive mutations that effect policing. We study the evolutionary invasion and evolutionary stability of policing both analytically and numerically. For any number of matings, there are four possible outcomes (see Fig. 5): (i) policing can invade and is stable; (ii) policing can invade but is unstable, leading to coexistence; (iii) policing cannot invade but is stable. leading to bistability; (iv) policing cannot invade and is unstable. We give precise conditions for all outcomes for both dominant and recessive police alleles. All outcomes can be achieved with arbitrarily small changes in colony efficiency. Our calculations are not based on any assumption about the strength of phenotypic mutation induced by an allele. The condi- tions (1). (2). (9), and (10) also describe the dynamics of mutations that have an arbitrarily small phenotypic effect on colony dynamics. This facilitates investigation of the evolution of complex social behaviors that result from gradual accumulation of many mutations (Kapheim et al.. 2016). We derive a simple relation, Eq. (17). for the location(s) of evolutionarily singular strategies. We also derive precise conditions for a singular strategy to be an ESS. These results are applicable for understanding both the case of weak phenotypic effect and the case of weak penetrance. Our analysis does not use inclusive fitness theory. Given the known limitations of inclusive fitness (Nowak et al., 2010; Allen et al.. 2013). it is unlikely that inclusive fitness theory could provide a general framework for analyzing the evolution of worker policing. In summary, the main conclusions of our paper are: (i) The prevalent relatedness-based argument that policing evolves under multiple mating but not under single mating is not robust with respect to arbitrarily small variations in colony efficiency; (ii) for non-monotonic efficiency functions, it is possible that policing evolves for single mating, but not for double or triple mating; (iii) careful measurements of colony efficiency and the fraction of queen-derived males are needed to understand how natural selection acts on policing; (iv) contrary to what has been claimed (Abbot et al.. 2011), the phenomenon of worker policing is no empirical confirmation of inclusive fitness theory; the first two mathematical papers on worker policing (Woyciechowski and Lomnicki, 1987; Ratnieks. 1988) do not use inclusive fitness theory. EFTA01071008 J.W. Olejorz et al. /puma/ of Theoretkof Biology 399 (2016) 103-116 115 The present paper, which also does not use inclusive fitness the- ory. is the first detailed analysis of policing for any number of matings and taking into account effects on colony efficiency. Acknowledgments We are grateful to the referees and editor for helpful comments that have significantly benefited this manuscript. This publication was made possible through the support of a grant from the John Templeton Foundation. The opinions expressed in this publication are those of the authors and do not necessarily reflect the views of the John Templeton Foundation. Appendix A. Supplementary data Supplementary data associated with this paper can be found in the online version at http://dx.doi.org/10.1016njtbi2016.03.001. References Abbot. It. et al_ 2011. Inclusive fitness theory and eusociality. Nature 471. (1-(9. Akcay. L. Van Cleve. J.. 2016. There is no fitness but fitness. and the lineage is its bearer. Philos. Trans. R. Soc. B 371.20150085. Allen. B.. Nowak. MA.. Wilson. E.O.. 2013. Limitations of inclusive fitness. Proc. Natl. Acad. Sci. USA. 110.20135-20139. Allen. B.. Nowak. MA. 2014. Games on graphs. EMS Surv. Math. Sc. 1 (1). 115-151. Antal. T.. Scheming. I.. 2006. Fixation of strategies for an evolutionary game in finite populations. BuIL Math. Biol. 68.1923-1944. Antal. T.. Ohtsula. H. Wakeley. J.. Taylor. P.U.. Nowak. M.A. 2009. Evolution of cooperation by phenotypic similanty. Proc. Natl. Mad. Sa. USA. 106 (21). 8597-8600. Beekman. M.. Oldroyd. B.P.. 2005. Honeybee workers use cues other than egg via- bility for policing. Biol. Lett. I. 129-132. Birch. J.. °kasha. S.. 2014. Kin selection and its critics. BioScrence 65.22-32. Bonckaert. %V_ Vuerincicc K.. Billen. J.. Hammond. R.L. Keller. L. Wenseleers. T.. 2008. Worker policing in the German wasp Wspulo gembonko. Behay. (col. 19. 272-278. Bonckaen. W.. van Zweden. J.S.. d'Ettorre. P.. Billen. J.. Wenseleers. T.. 2011. Colony stage and not facultative policing explains pattern of worker reproduction in the Saxon wasp. Mol. EcoL 20.3455-3468. Boomsma. J.J. 2007. Kin selection versus sexual selection: why the ends do not meet. Curr. Biol. 17. R673-R683. Boonisma. J.J.. 2009. Lifetime monogamy and the evolution of eusociality. Trans. IL Soc. B 164.3191-3207. Bourke. A.F.C.. 2011. The validity and value of inclusive fitness theory. Proc. FL Soc B 278.3313-3320. Cavalli-Sforza. LL. Feldman. M.W.. 1978. Darwinian selection and 'altruism". Theor. Popul. BioL 14.268-280. Cole. Bj.. 1986. The social behavior of Leptothorax olkodyce: (Hymenoptera. For- nuadae): time budgets and the evolution of worker reproduction. Behay. (col. SociobioL 18.165-173. Cornwallis. CK.. West. S.A.. Davis. K.E.. Gnffin. AS.. 2010. Pronuscuity and the evolutionary transition to complex societies. Nature 466.969-972. Dieckmann. U.. Law. R.. 1996. The dynamical theory of coevolution: a denvanon from stochastic ecological processes. J. Math. Biol. 34.579-612. Endler. A.. Liebig. J.. Schnutt. T.. Parker. J.E. Jones. C.R.. Schreier. P. Holldobler. B.. 2004. Surface hydrocarbons of queen eggs regulate worker reproduction in a social insect. Proc. Natl. Acad. Sci. USA 101.2945-2950. Foster. K.R. Ratnieks. F.LW.. 2000. Facultative worker policing in a wasp. Nature 407.692-693. Foster. K.R. Ratnieks. F.LW.. 2001a. Convergent evolution of worker policing by egg eating in the honeybee and common wasp. Proc. R. Soc. Lend. B 268.169-174. Foster. K.R. Ratnieks. F.LW.. 2001b. The effect of sex-allocation biasing on the evolution of worker policing in hymenopteran societies. Am. Nat. 158.615-624. Foster. K.R.. Ratnieks. F.LW.. 2001c. Paternity. reproduction and conflict in vespine wasps: a model system for testing kin selection predictions. Behay. (col. SociobioL 50.1-8. Foster. KR.. Gulliver. J.. Ratnieks. F.LW.. 2002. Worker policing in the European homer Vespo <wino. Insectes Soc. 49.41-44. Foster. K.R.. Wenseleers. T_ Ratnieks. F.LW.. 2006. Kin selection is the key to altruism. Trends (col. (vol. 21.57-60. Frank. SA_ 1983. Foundations of Social Evolution. Pnnceron University Press. Princeton. NJ. Fu. F.. Rocker. S.U.. Nowak M.A.. 2014. The nsk-retum trade-off between solitary and eusocial reproduction. Ecol. Lett. 18 (1). 74-84. Gadagkar. R. 2001. The Social Biology of Ropalidta Marginata: Toward Under- standing the Evolution of Eusociality. Harvard University Press. Cambridge. MA Gadagkar. It. 2004. Why do honey bee workers destroy each other's eggs/ J. Biosci. 29 (3). 213-217. Gardner. A_ West. SA. Wild. C.. 2011. The genetical theory of kin selection. J. (vol. Biol. 24.1020-1043. Gentz. SAH.. Kisdi. E.. Meszena. C.. Metz. JAJ.. 1998. Evolutionarily singular stra- tegies and the adaptive growth and branching of the evolutionary tree. Evol. (col. 12.35-57. Hamilton. W.D. 1972. Altruism and related phenomena, mainly in social insects. Annu. Rev. EcoL Syst. 31.193-232. Hammond. R.L. Keller. L. 2004. Conflict over male parentage in social insects. PLoS 8101.2 (e248). 1472-1482. Hartmann. A. Wantia. J.. Torres. JA_ Heinze. J.. 2003. Worker policing without genetic conflicts in a clonal ant. Proc. Natl. Acad. Sit. USA 100.12836-12840. Hauen. C.. Doebeli. M_ 2004. Spatial structure often inhibits the evolution of cooperation in the snowdrift game. Nature 428.643-646. Helantera. H.. Tofilsla. A.. Wenseleers. T. Ratnieks. ELW.. 2006. Worker policing in the common wasp Vespulo wagons is not aimed at improving colony hygiene. Insectes Soc. 53.399-401 Helantera. H.. Sandstrom. L 2007. Worker reproduction in formica ants. Am. Nat. 170. (14-E25. Hofbauer. J.. Sigmund. K.. 1990. Adaptive dynanucs and evolutionary stability. Appl. Math. Len. 3.75-79. Hughes. W.O.H.. Oldroyd. B.R. Beekman. M.. Ratnieks. F.LW.. 2008. Ancestral monogamy shows kin selection is key to the evolution of eusociality. Science 320. 1213-1216. Kapheim. K.M.. Pan. H.. U. C.. Salzberg. S.L.. Pula. D.. et al.. 2015. Cenonuc signatures of evolutionary transitions from solitary to group living. Science 348. 1139-1143. Khila. A.. Abouheif. E_ 2008. Reproductive constraint is a developmental mechan- ism that maintains social harmony in advanced ant societies. Proc. Natl. Acad. Sci. U.S.A. 105.17884-17889. Lehmann. L. Alger. I.. Weibull. J_ 2015. Does evolution lead to maximizing beha- vior/ Evolution 69. 1858-1873. Matessi. C.. Karlin. S.. 1984.0n the evolution of altruism by kin selection. Proc. Natl. Mad. Sa. USA 81.1754-1758. Metz. JAJ.. Gentz. SA.H.. Meszena. G.. Jacobs. EJA. van Heerwaarden. J.S.. 1996. Adaptive dynamics. a geometncal study of the consequences of nearly faithful reproduction. In: van Stnen. S.J.. Verduyn Lund SM. (Eds.). Stochastic and Spatial Structures of Dynamical Systems. KNAW Verhandelingen. Afd.. Amsterdam. pp. 183-231. Naeger. N.L. Peso. M.. Even. N.. Barron.A.B.. Robinson. G.E.. 2013. Altruistic behavior by egg-laying worker honeybees. Curt. BioL 23.1574-1578. Nonacs. P.. 2006. Nepotism and brood reliability in the suppression of worker reproduction in the eusocial Hymenoptera. Biol. Lett. 2.577-579. Nowak. MA.. Sigmund. K.. 1990. The evolution of stochastic strategies in the pris- oners dilemma. Acta AppL Math. 20.247-265. Nowak. MA. Tarrora. C.E. Wilson. E.0.. 2010. The evolution of eusociality. Nature 466. 1057-1061 °kasha. S.. Martens. J_ 2016. Hamilton's rule. inclusive fitness maximization, and the goal of individual behaviour in symmetric two-player games. J. (vol. Biol. 29.473-482. ▪ J.W.. Allen. B.. yeller. C.. Nowak. M.A.. 2015. The evolution of non- reproductive workers in insect colonies with haplodiploid generics. elzfe 4. e08918. Peters. J.M.. Queller. D.C. Imperarriz-Fonseca. V.L. Roubik. D.W.. Strassmann. J.E.. 1999. Mate number. kin selection and social conflicts in stingless bees and honeybees. Proc. It Soc. Lond. B 266.379-384. Pirk. C.W.W.. Neumann. P.. Hepburn. R. N10111z, R.F.A.. Tautz. J_ 1999. Egg viability and worker policing in honey bees. Proc. Natl. Mad. Sci. U.S.A. 101.8649-8651. Queller. U.C.. 1992. A general model for kin selection. Evolution 46.376-380. Queller. D.C.. Strassmann. J.E.. 1998. Kin selection and social insects. Bioscience 48. 165-175. Barracks. F.L.W_ 1988. Reproductive harmony via mutual policing by workers in eusocial hymenoptera. Am. Nat 132.217-236. Barracks. F.L.W.. Visscher. HK.. 1989. Worker policing in the honeybee. Nature 342. 796-797. Ratnieks. F.LW.. Foster. K.R. Wenseleers. T_ 2006. Conflict resolution in insect societies. Annu. Rev. EntomoL 51.581-608. Seeley. T.D.. 1985. Honeybee Ecology: A Study of Adaptation in Social Life. Princeton University MI5. Princeton. NJ. Simon. B.. Fletcher. JA. Doebeli. M_ 2013. Towards a general theory of group selection. Evolution 67.1561-1572. Starr, C.K., 1984. In: Smith, RI. (Ed.). Sperm Competition, Kinship, and Sociality in the Aculeate Hymenoptera. Academic Press, Orlando, Florida, pp. 427-464. Szabo. G.. Fath. C.. 2007. Evolutionary games on graphs. Phys. Rep. 446.97-216. Tarnita, C.E., Taylor, P.D. 2014. Measures of relative fitness d social behaviors in finite structured population models. Am. Nat. 184(4) in press. Traulsen. A.. Shoresh. N.. Nowak. MA.. 2008. Analytical results for individual and group selection of any intensity. Bull. Math. BioL 70.1410-1424. Uyenoyama. M.K.. Feldman. M.. 1982. Population genetic theory of kin selection. II. The multiplicative model. Am. Nat. 120.614-627. van Veelen. M.. Luo. S.. Simon. B.. 2014. A simple model of group selection that cannot be analyzed with inclusive fitness. J. Theor. Biol. 160.279-289. EFTA01071009 116 J.W. Olejarz er al. /Journal of Theoreckal Bioko. 399 (2016)103-126 Velthuis. H.H.W.. de Araujo Alves. D.. Imperatriz-Fonseca. V.L. Jose. M.. 2002. Worker bees and the fare of their eggs. Proc. Exp. Appl. Enromol. NEV Amsterdam 13.97-102. Wenseleers. T.. Helantera. H.. Hart. A.. Rarnieks. ELW.. 2004a. Worker reproduction and policing in insect societies: an ESS analysis. J. Evol. Biol. 17. 1035-1047. Wenseleers. T.. Hart. A.G.. Ratmeks. ELW.. 2004b. When resistance is useless: policing and the evolution of reproductive acquiescence in insect societies. Am. Nat. 164. E154-E167. Wenseleers. T.. Badcock. N.S. Erven. K.. Tofilski. A. Nascimenro. ES. Hart. A.C.. Burke. T.A. Archer. M.E. Ratnieks. F.LW.. 2005. A test of worker policing theory in an advanced eusocial wasp. Vespula tufa. Evolution 59. 1306-1314. Wenseleers. T.. Rain reks. F.LW.. 2006a. Enforced altruism in insect societies. Nature 444. 50. Wenseleers. T.. Ratmeks. F.LW.. 2006b. Comparative analysts of worker reproduc- tion and policing in eusocial hymenoptera supports relatedness theory. Am. Nat 168. E163-E179. Wenseleers. T.. Helantera. H.. Alves. DA. Duenez-Cuzman. E.. Panto. P.. 2013. Towards greater realism in inclusive fitness models: the case of worker reproduction in insect societies. Biol. Lett. 9. 20130334. WdSOA E.0.. 1971. The Insect Societies. Harvard University Press. Cambridge. MA Woyaechowski. M.. Lommcki. A.. 1987. Multiple manng of queens and the sterility of workers among eusocial hymenoptera. J. Theor. Biol. 128. 317-327. Zanette. LRS.. Miller. S.U.L. Farrel. C.M.A.. Almond. Ej.. Huggins. T.J_ Jordan. W.C.. Bourke. AEC.. 2012. Reproductive conflict in bumblebees and the evolution of worker policing. Evolution 66. 3765-3777. EFTA01071010