Negative Reinforcement - The good, the bad, and the ugly

Disclaimer: I am mostly a behavioural scientist. I know about animal behaviour from the outside. I know about measuring emotional states, assessing welfare, and looking for behavioural indicators of stress. Some of the topics in these blog posts on negative reinforcement are not my strongest areas. I can only offer my interpretation of the literature. I am certainly open to discussing alternative interpretations.

Negative reinforcement (R-, NR) is an operant conditioning quadrant. Quadrants basically predict how stimuli will affect the frequency, duration, and/or intensity of future behaviour depending on whether they are rewarding or punishing and whether they are added or taken away. In the case of negative reinforcement, future behaviour increases in frequency, duration and/or intensity when something is taken away. In other words, the animal will learn to perform a behaviour in order to gain relief from something they find aversive. Some well known trainers have created what's called a humane hierarchy of training methods as a guide to rank training methods on how humane an intervention they represent. Negative reinforcement is conspicuously far up on these humane hierarchies, prompting many trainers to stringently avoid it. This seems like a tenuous reason to condemn an entire learning quadrant. It makes a few very broad assumptions, such as even the mildest aversive experiences strong enough for an animal to want to avoid have no place in training behaviour. Is having someone brush past you, making you feel too close to them so that you step away really the same beast as having someone scream in your face until you move? I use extreme examples to show the breadth of the quadrant we're talking about. Some forms of negative reinforcement are extremely mild and some are jumping on the toes of downright punishment. And is avoiding an aversive experience always less humane than, say, reinforcing a different behaviour. Has anyone ever asked you to do something still like lie down when you'd rather be running away? But you can have a chocolate if you lie down. Thanks, but I think I'll run.  So what's the story? Can negative reinforcement be a humane way to train an animal? How do we assess that? Let's have a good look at the related scientific literature and see if we can reason out an answer. 

Susan Friedman's "humane hierarchy"

It would seem like the place to start is examining what it feels like for an animal to be negatively reinforced. Perhaps this is the most important question and yet the hardest to answer. I can present what we know of emotions and welfare in animals. I am hesitant to delve into neurotransmitters, neuroanatomy, and stress physiology because it is not my area of expertise. My understanding of the literature may be simplistic. Yet, there are claims being made that negative reinforcement should be avoided on that basis, so let's have a quick look at it. 

Neuroscience

These are chemicals that carry signals throughout the brain. The type of neurotransmitter is important, as is where it is going, but do not for a second think that this is remotely straight forward. The brain is crazy complicated, and very adaptable. Our understanding of it is not complete. Some neurotransmitters are associated with 'good' emotional experiences. One example is dopamine, which is heavily implicated in reward and the anticipation of good things happening. That is a good feeling! But, the lack of a certain type of dopamine receptor inhibits learning of both positive and negative associations with a physical place, and an active avoidance task. That suggests dopamine is implicated in unpleasant feelings as well... All right, so there's also serotonin involved in negative reinforcement. According to recent research, serotonin may have a role to play in how both rewarding (appetitive) and punishing (aversive) stimuli are processed. What does that mean? It means we don't really know exactly what serotonin does and we are probably going to need to study specific serotonin receptors to better understand it. So then there is noradrenaline, (or norepinephrine to use the American name), which is implicated in negative reinforcement learning and also features strongly in stress responses. Its role in stress responses is largely an arousing one, which, as it happens, appears to enhance memory and learning in discrimination tasks. It would be important in understanding what this all means to an animal to know where the neurotransmitters were going and what neural systems were involved. I would go into this except it would probably take me years to understand it enough to be able to condense it into a blog post. Suffice to say, nothing is straight forward in the brain. Even the amygdala, which everyone 'knows' is all about fear, flight and fight and freezing, also plays a critical role in positive emotional states. 

Stress

The concern about neurotransmitters and neuroanatomy involved in negative reinforcement may be in its association with stress and negative emotional states. So let's try there for some clearer answers. Is negative reinforcement stressful to animals? First, let's define what we mean by 'stressful', here. The body's stress response is very adaptive and can handle everything from minor stressors such as being hungry to major "I'm going to die" moments. And it covers positive experiences as well. A dog that is chasing a ball is very aroused and will be experiencing elevated stress hormones. The strength of a stress response is typically proportional to the intensity of the emotion associated with it. A strong stress response associated with a negative event basically means a lot of fear or anger, but a weak stress response means being a little perturbed. The strength of stress responses can be measured in a variety of ways, but most commonly through concentrations of stress-related hormones such as cortisol, either in blood, urine, or saliva. It is not quite an exact science. Check out this excellent blog post for a really nice summary of the issues. At any rate, lots of normal, everyday things raise cortisol concentrations, and cortisol (and glucocorticoids, and other stress-related hormones) are not "bad" per se; we need them! This system is fabulous at what it does, which is to keep us engaged and motivated when we need to be, and to keep us safe and help us recognise opportunities and threats and take appropriate action. We can't learn without stress, and we don't remember things that weren't very stressful all that well. But stress can be very unpleasant, and prolonged or frequent stress responses are dangerous and can cause serious disease and illness. Most people in modern society have experienced chronic stress in some form. It is not fun. To delve into this fascinating topic more, I cannot recommend Professor Robert Sapolsky enough. He has many videos available free on YouTube and his book "Why Zebras Don't Get Ulcers" is entertaining and very informative and still one of my favourites. 

Stress is also not always as simple as isolated events. There are unique stress responses for all kinds of stressors, which speaks to just how finely tuned stress responses are. There is a large body of literature on uncontrollable versus controllable stressors and how exposure to various kinds of stressors moulds future stress responses. All of these changes are 'good' in that they are adaptive and help animals best handle the cards they have been dealt. But some hands are terrible and the best you can do is try to minimise how much you lose out. Some hands offer the beginnings of a better hand down the track.  An example of where stress adaptation may be the beginnings of something better is in what is usually called stress inoculation. Animals that have learned to control stressors are more resilient in the face of future stressors. They try for longer to make things better for themselves when they are stressed, which means they are more likely to succeed and less likely to sink into despondency. They also show more curiosity, better emotional processing, and better cognitive control. All of these things are good for an individual animal. It will help an animal respond appropriately to stress and also handle mild challenges in their life better, such as social situations and impulse control.

So if stress is an important part of everyday life and plays critical roles not just in keeping us safe, but also motivating us, helping us remember and learn, and in some cases has a positive effect on future experiences and responses to stress, then how do we assess the role of stress in negative reinforcement and whether it is good stress, unpleasant stress, or beneficial stress?

At a basic level, looking at what animals find negatively reinforcing can tell us what they wish to avoid, and therefore, what they find unpleasant. Indeed, this has been proposed as an indicator of welfare by a leading animal welfare scientist. And we do find indications that training animals with negative reinforcement results in less approach behaviour towards people, and stronger emotional responses towards people, and these animals are also less engaged with their handlers than animals trained with positive reinforcement. And this brings us to an enormous body of research in avoidance learning, which is where things get interesting.

Active avoidance and stress inoculation

Avoidance is something we all do, keeping ourselves safe, comfortable, and healthy. Lots of research has been done on avoidance learning in animals, but I'm going to focus on active avoidance because it is most like what trainers do when they use negative reinforcement to train animals. Active avoidance is where an animal performs a behaviour on cue to avoid an aversive event. An example we might see often with animals is dashing under furniture to hide when a child comes into the room. The presence of the child cues the avoidance behaviour. Another variation is escape behaviour, where the animal learns to perform a behaviour to 'switch off' an aversive experience. So the animal dashes under the furniture when the child is too rough with it. Both are sensible strategies. The animal gains immediate refuge. But we wouldn't think this was necessarily an ideal sequence of events. Active avoidance is associated with a large elevation in cortisol concentration during the learning phase. It is distressing for animals when they are exposed to something unpleasant. Their distress is generally proportional to how strong the unpleasant experience is (brushing past vs yelling in face). And if their response (run and hide) is one that requires a fair bit of energy, their arousal will be elevated as soon as they see what they will need to run from. Heightened arousal means more intense feelings. Furthermore, if their response isn't always reliable in gaining them refuge, or they sometimes miss the cue, or the cue comes often and randomly, that is a layer of uncertainty that will make them very vigilant and very focused on potential threats to the point where they are likely to see them where they don't exist. All in all, not good.

But what if they are exposed to something unpleasant and have a reliable way to handle it? Is it just as distressing? A significant drop in cortisol concentrations occurs once an avoidance behaviour has stabilised. It does not drop to baseline, because it is arousing to perform an avoidance behaviour no matter how nonchalantly. But the drop in cortisol correlates with an increase in behaviours associated with a relaxed state, suggesting that fear has diminished quite a lot. Commonly, there is no outward appearance of fear or distress, and this nonchalance has been noted by several researchers. Animals are not frantically performing an avoidance behaviour to stave off the bad things. In fact, they tend to wait until the last possible moment to perform it and otherwise go about their usual business. Humans also report reduced fear and an increased sense of control when they use active avoidance during phobia treatment. Furthermore, studies suggest that animals that are taught active behaviours to successfully avoid aversive experiences learn to suppress their natural freezing responses, which in turn enables them to control the stressor through operant means. Freezing up helplessly is considerably worse from a stress perspective than actively coping because it is probably experienced more intensely. Active coping also leads to stress inoculation, which we have already discussed. 

So it seems like successful avoidance is not necessarily associated with significant fear or distress. How could that be? The animals still don't like the aversive they are trying to avoid, right? If they are trying to avoid something it must be unpleasant. That's what Dawkins' suggestion for negative reinforcers as indicators of poor welfare was all about. It turns out there may be some pretty cool and strange things going on...

Safety signals

Bear with me while I give a brief but necessary background, here. A safety signal tells an animal that they are safe in the immediate future. It is trained by giving the animal an aversive experience and then pairing the ABSENCE of that aversive experience with a particular signal. So the signal comes to mean "You are safe for now". Safety signals appear to be able to inhibit fear surrounding an uncontrollable aversive experience AND inhibit the anxiety expressed after that event. This is pretty amazing stuff when you think about it. Safety signals themselves are not entirely negative reinforcement because no response is necessary so no particular behaviour is being reinforced, although an aversive experience is required in order to train one. But, an operant behaviour can take on a similar role, and these are learned through negative reinforcement.

Safety behaviours

 New research shows that it is inherently rewarding to avoid an expected aversive event - in other words, avoidance itself can be a form of positive reinforcement. Wha...? How does THAT work?? It has been suggested that there comes a point where an animal is not so much avoiding the aversive stimulus but approaching safety. Let's call these safety behaviours. It's not necessarily an official name. This is a pretty poorly understood area of science and I am not aware of anyone that has tested whether these behaviours have the same properties as a safety signal. They can be escape behaviours or avoidance behaviours in that they may switch off something unpleasant or avoid it completely, and that's how they become safety behaviours. It seems like a slippery distinction that could be used to justify some quite terrible things that are done to animals using negative reinforcement in the name of training, so let's be very clear about what this means. Safety has to be real to be sought. This means several things for the development of safety behaviours: 

1) It needs to be very clear to the animal that safety has now been attained - ideally, a specific signal (safety signal, possibly a cue or marker can take on this role). 

2) That safety has to be real and meaningful to the animal, not simply declared by a trainer or handler. 

3) Learning a behaviour to attain safety is actually quite hard in many circumstances, because animals already have natural behaviours they will tend to use when they feel threatened. If they are being taught a behaviour that runs counter to their goals (i.e. get distance from the scary thing), they may never be reliable or never learn it at all. 

4) The effectiveness of safety signals are inversely proportional to the strength of the threatening stimulus - in other words, the ability of safety signals to inhibit fear is influenced by arousal. The more threatening something is, the more aroused an animal becomes, and the more aroused they are, the less effective a safety signal will be in inhibiting fear. In short, if someone routinely threatens to clobber you with a baseball bat, it won't make you feel very safe to know that they never will as long as you run to the other side of the room whenever they make the threat. But if someone routinely threatens to swat you with a newspaper, knowing they won't as long as you run to the other side of the room probably will make you feel pretty nonchalant about the threat. 

5) Animals that are regularly becoming afraid are most likely going to become pessimistic, even if they are practiced at avoiding aversive experiences and seeking safety. 

There is no line between "animals like safety" and "I should therefore create scenarios in training where I can reward them with safety." They like safety, but they like tangible rewards more, and if we want happy, optimistic animals, we should be very much focused on giving them as many opportunities to access tangible rewards as we can. We should also be very aware that their sense of safety comes first to the point where if they feel unsafe they will be primarily motivated to seek safety. Food, play, social contact... all of these things come secondary to seeking safety. We should therefore make their safety our first priority. Compromising their sense of safety ourselves is not clever and runs counter to our goals if we want happy animals first and foremost. Training safety signals and safety behaviours can and should be done opportunistically. If you are able to keep your dog safe and protected from aversive experiences at all times, you do not need safety signals. Although it may mean your animal is both more sensitive to aversive experiences and may experience them more intensely. I do not believe it is in any animal's best interests to attempt to protect them from all aversive experiences. They have evolved a truly wondrous system to handle them, just as we have. We just have to be careful that what they experience is well within their coping abilities and does not have a lasting impact on their mood or health. 

Animals look after their safety first and foremost.

Safety behaviours can also play a role in the treatment fears and phobias by increasing the acceptability of exposure. This has, to my knowledge, only been done in humans where safety behaviours can be quite problematic. However, in some circumstances they can offer a stepping stone to further treatment, making sufferers feel less fear during exposure and they tend to approach closer to the object of their fear. I have done this with a wild hare and found indications of similar results. The hare was taught to 'ask' for space by pulling away from me. If he pulled away, I respectfully did not follow or I backed up. He soon began allowing me to touch his flanks, head, and legs, which put him in a very vulnerable position, as if I had wanted to grab him (something that is probably always on a hare's mind), I was in an excellent position to do so. 

Emotional state

All this talk about the ambiguity in neuroscience, stress, and even approach and avoidance behaviours has left us with a bit of a quandary. If neuroscience is crazy complicated, and sometimes stress is good, and sometimes avoidance is positive reinforcement, and safety signals inhibit fear, and negative reinforcement can be very scary or barely register and everything in between, how can we tell if negative reinforcement is an ethical training approach? I believe the answer is in emotional states. Possibly because I have done a lot of work in detecting emotional states. But really, emotional states are at the center of all this. Whatever the animal is experiencing, it should be detectable in behavioural changes, although they may be subtle and take some careful observation. Whether stress or avoidance is good or bad will directly influence emotional state, either positively or negatively, and that will affect behaviour.

An animal will tend to develop a positive mood if they experience a lot of good things, and if they experience unpleasant things, they will tend to develop a negative mood. There are passing few ways to reliably measure emotional state in animals, which is why I was able to do a PhD on it. Lacking the ability to measure neural activity, cortisol concentrations, reward sensitivity, and cognitive bias, we are left with the terrible inadequacy that is behavioural indicators. The biggest problem with behavioural indicators is that they are hard to identify and open to interpretation. There are a few behaviours we know in dogs are associated with elevated cortisol concentrations in an environment where this is almost certainly due to emotional distress, such as increased urinating, physical activity, and increased displacement behaviours (lip licking and paw lifts in particular). In turn, there are very few indicators of positive emotional state. Play is one, and anticipatory behaviour surrounding rewards is another. It is pretty hard to identify positive anticipatory behaviour in dogs because the work hasn't been done, but it's probably fair to say if they are looking something like the picture below, we're on the right track. 

Anticipatory! Image Eric Danley

This is not as useful as we might hope. These are behaviours generally associated with major, chronic stress. It is quite unlikely that we would see this as a result of simply using negative reinforcement in training. Even if we ONLY used negative reinforcement in training. We need something more sensitive. While we can't formally measure cognitive bias and reward loss sensitivity, we can look for it in everyday behaviour. 

1. Exploration and approach behaviour - We would expect a reduction in exploration and approach behaviour if emotional state is tipping towards negative, and an increase where the emotional state is tipping towards positive. It fits in very nicely with what we know about optimism. The horse study cited earlier showed horses trained with negative reinforcement were less explorative and approached people less, which suggests the training has a negative effect on the horses' emotional state. This will hurt our training goals if we like to shape behaviours!

2. Interest in training - If an animal becomes less willing to participate in training, which may manifest in distractibility, nervousness, skittishness, lots of displacement behaviour (sniffing the ground, staring into the distance, scratching, anything to delay having to train), disinterest, and general unwillingness to approach either the trainer or the training environment, this is BAD. It suggests the animal does not enjoy training. We want to see them engaged and readily coming to you, prick eared and leaning forward. Unless they don't have visible ears. Then just leaning forward.

3. Willingness to offer behaviours - Animals in a negative emotional state are expected to be behaviourally suppressed to some degree. They will not really want to try new behaviours and may be reticent to offer those they know even when cued because it is risky to them. If we have an animal that is either offering behaviours on its own or can easily be coaxed into doing something new, they are most likely in a positive emotional state. 

4. Sensitivity to reward loss - Animals that are in a negative emotional state feel keenly when they think they have missed out on a reward they were expecting. In training this may manifest in relative slowness and reluctance particularly where reward rate has decreased or when attempting to move to a variable reinforcement schedule. 

So... what's the verdict?

Click here for an analysis of the literature and my take on the ethics of using negative reinforcement in training and behaviour modification (plus a reference list). 

Negative Reinforcement - Is it ethical?

In the previous post I did a brief review of the literature relevant to negative reinforcement. So are there ever cases where negative reinforcement should be viewed as more humane than its current placement on humane hierarchies?

In my opinion it's not a black and white issue. On the one hand, there is ample evidence that negative reinforcement is associated with elevated cortisol and a reduction in approach and explorative behaviour and therefore can be assumed to be more stressful and unpleasant than training with positive reinforcement. This certainly makes sense in the context of literature on emotions as well. So, it seems its place fairly late in the humane hierarchy is justified in most scenarios.

Image source

But wait... What about safety? Safety signals are a powerful inhibitor of fear, which is surely a good thing, particularly in behaviour modification where problem behaviour is motivated by fear. On the other hand, safety signals are only relevant where an animal anticipates danger. Shouldn't our goal be to stringently avoid our animals anticipating danger? On the face of it, I would say yes, we don't want our animals to anticipate danger. But what if they are already anticipating danger in spite of our efforts to protect them from this? So many problem behaviours are distance increasing behaviours - they are designed to buy an animal distance from something they find threatening. The animal is already anticipating danger. They are already in the exact situation we were hoping they wouldn't experience. How do we get them out? Generally the answer is to use counter-conditioning and desensitisation to change their emotional response to that threatening thing so that they no longer find it threatening. But what if you could tell them straight away "It's okay, you are safe" and have them believe you? Is that a worthwhile trick to have up your sleeve?

In my experience, absolutely. These safety signals generalise easily and can be attached to behaviours that are sensible for animals to do when they feel threatened. For example, one of my dogs falls into a formal heel when he feels threatened and the other walks between my feet. They do this because they firmly believe they will be safe if they do. Not only does it calm them and according to them, magically fix scary situations so they don't have to be scared, but it puts them right by me where I can best protect them and my proximity can make them feel more secure. It puts their attention on me so that they are less likely to react to changes in the threatening thing, for example, a dog starting to run instead of walking. It also means they will move with me, so I can calmly walk them right out of danger, and I have done this on occasion around loose dogs that are acting a bit volatile. Is it a replacement for counter-conditioning and desensitisation? Nope. It is an ace in the hole. It can help you out of a sticky situation, it can buffer your animal from an otherwise upsetting moment by inhibiting stress and anxiety during and following it, and there is almost certainly counter-conditioning occurring at the same time, as feeling relieved and confident around something scary instead of scared and anxious is going to be incorporated into associations made with that scary thing and should make it less scary. At least, that has been my experience and the literature supports this.

But don't you need to deliberately apply an aversive in order to train a safety signal? I thought you said that was stupid. Yes, I did. Because it is, and it's also unnecessary. I don't train safety behaviours like they do in studies. I just train them opportunistically. Sooner or later something will upset my animal and it's already too late to avoid it. I can, however, pair a sound with the moment when they realise they are now a comfortable distance from the scary thing, or to make things easier, the moment they retreat (at a run if you like). With my dogs, I can also ask for a behaviour at the moment when my dog has calmed down enough that they are able to perform it and reward that with a treat. If I always ask for the same behaviour, my dog comes to associate that behaviour with the end of the aversive experience, a sense of relief, and a period of immediate safety. The result is my dog starts performing this behaviour earlier and earlier until realising something scary might be going to happen cues the behaviour. They calm down, they focus on me, they successfully get themselves out of the situation without losing their marbles. Why wouldn't I just counter-condition? Because sometimes the environment isn't as controllable as we would like. I need to walk my dogs for their wellbeing. Sometimes our walks unexpectedly dump us way too close to something scary. There is no way around it. This is just a way to use these unfortunate scenarios for everyone's ultimate benefit.

What about escape behaviours and active avoidance? It might reduce fear, but surely teaching an animal to perform a behaviour in order to escape from an unpleasant experience is ethically questionable? Again, this is not black and white. In general, no, we do not want our animals to feel the need to escape. And again, it happens anyway, just as it happens to us. Sharing the road with a vehicle that looks unsafe makes me want to escape. I feel relief when I successfully distance myself from this naturally occurring situation that makes me intensely uncomfortable. I think it is inevitable that our animals will also find themselves in similar situations, particularly dogs who are out and about in the community with us, and dogs with anxiety problems, and dogs that are highly emotionally reactive. There is every possibility they will bump into a dog that frightens them, for example. Teaching them a controlled escape behaviour that will make them feel calm and in control and also avoid troublesome behavioural outbursts seems humane to me. I sure like it when someone tells me how I can escape from situations I dislike. They are much less stressful when you can quickly and confidently handle them. Have you ever set up an agreement with someone to have them call you away from a situation if you signal to them you want to leave? It gives you an almost guaranteed fast and effective out. Does it make you feel more confident going into that situation? Developing reliable escape and active avoidance behaviours gives animals the means to signal they want out. Sometimes it is argued that this means the animal will forever be asking for outs and handlers will need to be vigilant for the rest of the animal's life. I have not found this to be the case. I mentioned that safety behaviours do have a place in treatment of human fears and phobias, allowing people to feel safer and more in control so they can get closer. Exposure is important for overcoming fears. If the animal clings to their avoidance behaviours to the point where they won't venture any closer willingly, they may need a wee bit of encouragement to find they don't need the avoidance behaviours so much anymore. For animals, I believe that on the odd occasion this happens, it is likely to be an antecedent arrangement at the core of it. Change the setup slightly, or change the sequence of events or the behaviours cued slightly and they should pop right out of their rut and make huge bounds forward. 

This is Kivi's expression when heeling for treats, and when heeling away from dogs that scare the bejesus out of him. 

                   



What about stress inoculation? Improving resilience and giving animals the skills and confidence to work at solving problems seems like a positive thing in general. But is it worth exposing animals to stressors? That is a difficult question to answer, because what kind of stressor would be necessary? Puppies are typically exposed to stressors while they are still with their mother, such as being left alone for short periods, handling physical obstacles like uneven ground, and dealing with frustrations such as siblings that are in competition. On another level, there is frustration later in training where dogs may need to learn to persist in trying to solve a problem in order to access something they are motivated to have. And on another level still, there are more significant stressors like older dogs that do not appreciate puppy behaviour, being confined or restrained, experiencing car rides... As you can see, stress is part of everyday life. The positive effects of stress inoculation are likely to benefit any animal that is going to find themselves in novel environments or in novel contexts (which may include training a new behaviour, incidentally). This does not mean we should all rush out and expose our animals to some form of controllable stressor. Just make sure that when your animal does encounter mild stress, they are equipped to control it. If they are not, sometimes letting them find their own way to controlling it if it is not far beyond their comfort zone can have lasting benefits. See the series on risk aversion for more information, particularly this one.

In summary, there may be situations where negative reinforcement doesn't contravene our training goals if those goals are to have optimistic, confident animals that expect good things to happen to them. Those situations can be generally categorised as where the animal has already encountered something that has threatened their sense of safety and where a sense of control and safety would aid rehabilitation as long as arousal (which may be considered a surrogate for how scared the animal is in this context) is low to moderate and no higher.

CAUTION!!

As we have seen, negative reinforcement is not without risk. It behooves us to be careful with this. 

1. Do not use it to train approach behaviours - you don't want an animal approaching something in order to make it leave them alone. This is how we end up with things like dogs rushing and lunging in the first place. Some horse trainers do this kind of thing and it works, but personally, I am wary of it. I rely on whether my animals will approach to tell me if they like or are comfortable with something. Teaching an approach behaviour with negative reinforcement robs me of that information. 

2. Know when to bail - This will vary from animal to animal, but you should know before you do anything how far you will go. Generally speaking, I would abandon training if my animal is trying to escape, if they are frozen, and if their arousal has climbed to the point where they are darting glances around. They should be calm enough to respond to their name readily and be able to perform cued behaviours reliably. 

3. Keep track of indicators of emotional state. The point of using negative reinforcement in the contexts suggested here is to improve welfare and give your animal some flexibility in how they cope with stressors. You need to stop and rethink if it is not obviously doing that. 

4. Beware sticky avoidance behaviours hampering progress - Many psychiatric disorders in people are maintained to some degree by safety and avoidance behaviours. This seems like a minor concern in animals, who do not have such complex psyches, but it is well known that animals can continue with avoidance behaviours long after they are necessary. There is a problem here with prediction errors. In order for an animal to learn that they do not need their avoidance behaviour, they need to see that they didn't use their avoidance behaviour and nothing bad happened. If they are practiced avoiders, they may not have the opportunity to see this. As mentioned earlier, it's not that big a deal. Change the context just a little and they should change their behaviour. Sometimes allowing an animal to continue to avoid if they are comfortable with that is smart. We don't necessarily need them to approach everything and may not want them to.

References

Macoveanu, Julian. "Serotonergic modulation of reward and punishment: Evidence from pharmacological fMRI studies." Brain research (2014).

Smith, J. W., et al. "Dopamine D2L receptor knockout mice display deficits in positive and negative reinforcing properties of morphine and in avoidance learning." Neuroscience 113.4 (2002): 755-765.

Christianson, John P., et al. "Safety signals mitigate the consequences of uncontrollable stress via a circuit involving the sensory insular cortex and bed nucleus of the stria terminalis." Biological psychiatry 70.5 (2011): 458-464.

Christianson, John P., et al. "Inhibition of fear by learned safety signals: a mini-symposium review." The Journal of Neuroscience 32.41 (2012): 14118-14124.

Helmreich, Dana L., et al. "Active behavioral coping alters the behavioral but not the endocrine response to stress." Psychoneuroendocrinology 37.12 (2012): 1941-1948.

Moscarello, Justin M., and Joseph E. LeDoux. "Active avoidance learning requires prefrontal suppression of amygdala-mediated defensive reactions." The Journal of Neuroscience 33.9 (2013): 3815-3823.

Kim, Hackjin, Shinsuke Shimojo, and John P. O'Doherty. "Is avoiding an aversive outcome rewarding? Neural substrates of avoidance learning in the human brain." PLoS biology 4.8 (2006): e233.

Coover, Gary D., and Holger Ursin. "Plasma-corticosterone levels during active-avoidance learning in rats." Journal of comparative and physiological psychology 82.1 (1973): 170.

Innes, Lesley, and Sebastian McBride. "Negative versus positive reinforcement: an evaluation of training strategies for rehabilitated horses."Applied animal behaviour science 112.3 (2008): 357-368.

Lyons, David M., and Karen J. Parker. "Stress inoculation‐induced indications of resilience in monkeys." Journal of traumatic stress 20.4 (2007): 423-433.

Sankey, Carol, et al. "Reinforcement as a mediator of the perception of humans by horses (Equus caballus)." Animal cognition 13.5 (2010): 753-764.

Beerda, B., et al. "Behavioural and hormonal indicators of enduring environmental stress in dogs." ANIMAL WELFARE-POTTERS BAR- 9.1 (2000): 49-62.

Cain, C. K., J. S. Choi, and J. E. LeDoux. "Active avoidance and escape learning." Encyclopedia of Behavioral Neuroscience. New York: Elsevier(2010).

Levy, Hannah C., and Adam S. Radomsky. "Safety Behaviour Enhances the Acceptability of Exposure." Cognitive behaviour therapy 43.1 (2014): 83-92.

Hood, Heather K., et al. "Effects of safety behaviors on fear reduction during exposure." Behaviour research and therapy 48.12 (2010): 1161-1169.

Rachman, Stanley, et al. "Does escape behavior strengthen agoraphobic avoidance? A replication." Behavior Therapy 17.4 (1986): 366-384.

Dalley, Jeffrey W., et al. "Distinct changes in cortical acetylcholine and noradrenaline efflux during contingent and noncontingent performance of a visual attentional task." The Journal of Neuroscience 21.13 (2001): 4908-4914.

Burgdorf, Jeffrey, and Jaak Panksepp. "The neurobiology of positive emotions." Neuroscience & Biobehavioral Reviews 30.2 (2006): 173-187.

Dawkins, Marian Stamp. "The science of animal suffering." Ethology 114.10 (2008): 937-945.

Deldalle, Stephanie, and Florence Gaunet. "Effects of 2 training methods on stress-related behaviors of the dog (Canis familiaris) and on the dog–owner relationship."Journal of Veterinary Behavior: Clinical Applications and Research 9.2 (2014): 58-65.

Koolhaas, J. M., et al. "Coping styles in animals: current status in behavior and stress-physiology." Neuroscience & Biobehavioral Reviews 23.7 (1999): 925-935.

Burman, Oliver HP, et al. "Sensitivity to reward loss as an indicator of animal emotion and welfare." Biology letters 4.4 (2008): 330-333.

Risk Aversion 3 - Training for persistence, resilience, confidence and optimism

This is the third in a series of posts about risk aversion, or pessimism, in dogs. The first instalment looks at general tendencies of risk averse dogs, and the second looks at how risk averse dogs behave in day-to-day life, whether we can treat risk aversion, and if we should. In this instalment I will talk about how to make risk averse dogs less risk averse. This is pretty experimental, but is based on literature. I can at least say that I did it with my risk averse dog and had phenomenal results. 

If we have decided that it is in our dog's (and our) best interests to reduce their risk aversion, we would do well to break risk aversion down into smaller pieces and treat each one. However, we will have trouble picking and choosing which pieces we want to work on and which we don't because they are all kind of related. Working on one will probably benefit others to a lesser extent as well.

Persistence

As discussed in the first post, risk averse dogs lack persistence. Training persistence is simple, but not necessarily easy (to use a saying from Bob Bailey, godfather of modern animal training). Literature on persistence training reports on partial reinforcement procedures, in other words, instead of a reward after every time a dog performs a behaviour, you would reward after only some of the times. You might choose to do this randomly, or predictably. I suggest predictably, because it's easier! For example, count a set number of responses. Research shows that partial reinforcement leads to animals that persist longer in learning situations than animals that get continuous reinforcement (reward every time). However, starting with continuous reinforcement and then later moving to partial reinforcement is even better!

Another aspect of persistence is called "generalised industriousness". What this means is that you can teach a dog to put in more effort, to work harder, and for fewer rewards. It can be done with a variation on the persistence training outlined above. The dog needs to put in a little more effort each time before they get a reward. But don't just hold out for them to work harder. With a risk averse dog you will need to make sure whatever you ask them to do is easily achievable. Start very easy, then gradually make it harder either by counting a few seconds more before you reward, or ask for an extra behaviour or two before you reward, or make a well understood task slightly harder the next time. Our favourite industriousness game we call "up-up". We find an obstacle in the environment that is low and safe for our dogs to jump onto, point to it, and say "up-up". Dog jumps onto obstacle and gets a treat. Think of this like a video game. The dog just finished level 1. Level 2 might be a little higher, or the surface might be uneven, or smaller. Level 3 might have one very simple obstacle to jump on in order to get to the next obstacle where we are pointing. When we train a dog to be industrious, we are pairing the feeling of working hard whether that be physical or mental with rewards, improving self-control and contributing to teaching dogs that if they just try that little bit harder it will pay off. 

Persistent dogs tend to get more rewards. Image source.

More variety in training tasks also leads to increased effort. Get your dog learning a variety of skills. Pay them for trying, even if they are nowhere near the behaviour you want, or they make a mess of it. All you want them to do is give it a go, so make giving it a go rewarding. As they grow more confident, you may decide to hold off and wait for a 'better' try with more effort or conviction from your dog. Reward handsomely when they deliver, again, even if all they do is put in a little extra effort. 

Resilience

I am not going to detail how to train this here because it is a little bit controversial and probably deserves its own dedicated discussion. Think of it this way: if you have a dog that starts to fall apart if faced with minor problems they don't know how to solve, what do we need to teach them? Why are they easily distressed and what would make them better able to cope with stressful situations? By my reckoning, we need to teach them how to problem solve on their own without needing much help from us. There is a body of literature on "mastery" and "resilience" or "inoculation", which refers to an individual's ability to learn to control something stressful and the positive effects of this in later stressful situations. See here for a nice review. Resilience may be trained using very careful exposure to low level stress that the dog can resolve on their own. This is something to be cautious about as a little too much stress or the wrong kind of stress is likely to backfire and make things worse. More about this in a later post.

Confidence

This one really depends on the ways in which a dog is lacking confidence. My risk averse dog was quite clumsy and found things like balancing or moving his back feet with precision very challenging. I felt that this was probably holding him back, because when you fall or feel unbalanced a lot, it seems risky to try things with your body that you haven't done before. I did a lot of balance and body awareness training with him. I used logs at our local dog park to train him to balance better, and to move his back feet independently of his front feet. This is known as rear end awareness and is popular training for dogs in dog sports because it helps them be more agile and move with more precision and efficiency. Here is a video of Erik demonstrating 'log games' for rear end awareness and balance. 

Learning what he could do physically and making him feel more balanced and agile had a huge positive effect on his confidence. He went from standing in front of something the height of his chin and staring helplessly at it while we spent an age trying to coax him to jump over it or onto it, to going out of his way to find things to climb on. This opened up a whole new world for him where he could seek rewards. See him in the video below giving me heart palpitations negotiating rocks at some height on a rock platform at the beach. 

Body awareness and balance exercises are a good place to start for general confidence building, but it is worthwhile trying to identify where a dog lacks confidence the most and applying similar principles. Start small, keep it easy. For example, little Erik has a lot of confidence in general, but lacks confidence around water. We let him take it at his own pace but encourage him to challenge himself. He is much more confident about creek crossings, now, but still doesn't want to swim. That's okay. 

Optimism

Optimism is basically the expectation that good things are going to happen. So when you are optimistic, you tend to feel pretty good, because at any moment probably something terrific will happen to you. This is associated with mental wellbeing, better outcomes in serious illness, and better physical health in humans. Learning to be optimistic is relatively simple for an otherwise healthy dog. If a dog has a lot of good things happen to them, they will expect more good things to happen to them. Training a dog to be optimistic doesn't just mean you throw a lot of rewards at them, though. It's important to note that in some situations, non-contingent reinforcement (reinforcement a dog can't control) can also interfere with learning in much the same way as learned helplessness. This is usually called "learned irrelevance". So for best results, offer opportunities for reinforcement in a large variety of situations. Make it easy to earn rewards often. Set them up sometimes to find their own way to success rather than have you show them or tell them what to do. This can take some skill. You want to make the path to success obvious enough that they won't have to try very hard, but just hard enough that they will have to think their way through it. The video of Kivi in the rocks is not a bad example. I move, so it is obvious to him he should follow me, but he has to find his own way through the rocks. The goal is getting access to rewards, whatever your dog loves. The game is for them to figure out how to get it. Again, think of video game levels. Getting them to use their nose and search for things is great. 

Final Word

These are all fairly general tips light on details. Treating risk aversion should be considered a broad thing with many components, because risk aversion itself is quite broad with a variety of components. It is quite easy to make things too hard for a risk averse dog. If you have a dog like this, above all remember to be patient and do one or two steps at a time and then just leave it for another day. Otherwise you risk making things worse by overdoing it. If they don't get success easily they will likely become stressed and give up. Let them find their own way as much as possible and set their own pace. You are there to encourage and guide (and provide lots of rewards). If you have a seriously risk averse dog that may have other fear related problems as well, get professional help from a behaviourist. Treating risk aversion doesn't treat specific fears or behaviour problems. 

Further reading

Nation, Jack R.; Cooney, John B.; Gartrell, Karen E., 1979. Durability and generalizability of persistence training. Journal of Abnormal Psychology, Vol 88(2), 121-136

Eisenberger, Robert; Masterson, Fred A.; McDermitt, Maureen 1982. Effects of task variety on generalized effort. Journal of Educational Psychology, Vol 74(4),

Nation, J. R., & Boyajian, L. G. (1980). Continuous before partial reinforcement: Effect on persistence training and resistance to extinction in humans. The American Journal of Psychology, 697-710.

Martin E. P. Seligman, Jane E. Gillham, 2000. The Science of Optimism and Hope: Research Essays in Honor of Martin E.P. Seligman. Templeton Foundation Press.

Job, R. F. S. 1988. Interference and facilitation produced by noncontingent reinforcement in the appetitive situation. Animal Learning & Behavior, 16(4), 451-460.

Building a reward system, or "My dog won't work for food"

Getting an animal to work for food is sometimes a little bit tricky. It is absolutely worthwhile, though, even if you have your animal working for another kind of reward. Food as a reward comes with several advantages. For example, it can be delivered and consumed very quickly so that the flow of the training is not disrupted and more repetitions can be fit into a short training session. It can also be delivered slowly (e.g. in lots of little bits one after another), stretching out the moment of reinforcement to make it seem like the animal has just hit the jackpot. It's a very flexible kind of reward. It can increase and decrease arousal, it can be delivered close to you or farther away, it's easy to carry, and it's one of the most reliable ways to improve emotional state, so invaluable for behaviour modification.

If your dog (or other animal) currently won't work for food, don't write the whole thing off. All animals will work for food. They have to eat, after all. Work through the following steps and pretty soon your animal will be working for their regular food whenever you give them the opportunity.

Step 1: Desensitisation

Animals won't eat unless they feel safe. If an animal is not safe it is just like when we are very anxious. Our stomach roils and we may even feel nauseous. We do not want to eat. They feel much the same way. Therefore, the first step in building a food-based reward system is to make sure your animal is comfortable with the environment. This means they are comfortable with their surroundings, the other animals and people in it, and the trainer. With dogs we tend to skip this step because many of them are very opportunistic about food and always seem ready to eat, but I can't emphasise enough how important it is to keep in the back of your mind. If you have a dog that is glancing around a lot, staring at other things in the environment, or seems very distractible or unwilling to move much, chances are the dog is not comfortable in the environment. For other animals you may see freezing and staring and a lack of response to you or any food you offer. The fix for this is desensitisation. Give them time to just take in their surroundings and get used to it. If they are in a whole new place, like they have just been rehomed, give them a few weeks. If they are in novel or unfamiliar surroundings but have a good relationship with you, give them a few days or sessions in those surroundings. If they are still not comfortable you can help them along with some counter-conditioning and active desensitisation. Stay tuned for an article detailing this.

Step 2: Establishing Food as a Reward

Some animals have not really had the opportunity to realise that food is something they actually can have control over. They are used to having it dished out to them and left to eat it in their own time. They don't know there might be other ways to get food, or that it might be much better than what they are used to. Other animals may naturally eat food that doesn't exactly run away from them, like grass. Basically they can eat whenever they feel like it and they won't have the motivation of a more opportunistic animal like a dog to take advantage of when it is available.

My preferred way of handling this is to find a food the animal particularly likes and make sure they only ever get it from my hand. For dogs, fresh or cooked meat is usually a winner. For very fussy dogs, try cooked heart. I boil lamb hearts in a saucepan until they are cooked through and then cut them up small. For herbivores, sometimes fresh fruit can hit the spot, or a favoured vegetables. I've had success with berries and carrot tops (the leafy bit from dutch carrots is a rabbit favourite). Grain-eating birds can be trickier. Watch what grains they choose first to work out their favourites. Try dark, oily seeds like a canary tonic mix, or egg and biscuit. For parrots, you may find fruit and nuts or a commercially available treat work. Make sure you check with experts if your animal's treats may cause health or digestive problems.

Step 3: Practice

Once we have some tasty rewards to choose from and our animal is interested in us and our food, then it just comes down to practising working for food. Many companion animals just don't realise they can earn food any time. If they can only earn food at meal times, or in training sessions, then how will they know it's worthwhile listening to you at other times? Make it really easy for them to earn food. They need to learn that just interacting with you pays. Making eye contact, following you, staying close, following gestures, and of course, lots of sits and downs. I love a default sit or down. Keep treats handy and play a game of trying to surprise your animal with treats or opportunities to earn treats when they are not expecting it. Note: This may work better for dogs than other animals.

Transitioning to using treats other than the amazing ones you found in Step 2 is quite easy. Wait until your animal is looking for a treat. This will usually happen a few days into Step 3. They will do something that has recently been rewarded and their ears will come forward and they will look at you and may dance around a little in anticipation. This is when you can try popping them a different treat. If they won't take it, go back to the good one for another day or half a day, then try again. If they do take it, good! Start offering it only 20% of the time at first, then after a day or so, move to 50% of the time, then 80%. Once you get to about 50% of the time, introduce a third treat in much the same way. Phase out the best treats at the same time. It is good to save these ones for when you really need them.

Next step - the big wide world. These dogs are in the habit of looking to humans for rewards, even somewhere as exciting and full of rewards as the beach.

Step 4: Taking it on the Road

It's easy to trip up, here. I know it's a drag, but if you have treats on you all the time you will see things to reward and you will have the means to do so. Why be stingy? The more opportunities your animal has to earn rewards the more attentive they will be to you. Once you get into the big wide world (if you take your animal there), they may completely forget about treats and earning food. It's okay, just be patient. The work you put in here will pay off in a big way. Go back to Step 1 and work through the protocol again somewhere that is not the most exciting place on the planet. You need to have your animal relaxed enough that they can stop running around and be content to look around instead, or even sit or lie down. Alternatively, teach them a cue that means "I have your favourite treats right here, right now." Use the amazing treats you discovered in Step 2 and pair them with a word or sound. I use "Hey!", but other people use a kissy sound, or clicked fingers, or "Oi!". Start at home, and every time you say the word, wait for your animal to look at you and then feed them the amazing treat. Do this a lot. Until when you say the world they whirl towards you. This can sometimes help you get your foot in the door once you are out in the world again by getting their attention.

Remember not to ask much of your animal at first. Be a good bet. All they need do is look at you when you say their name and you can reward that. If you ask for a sit, expect a one second sit and reward it if you get it. You can work up to more attention and longer sits later. To begin with this is all about teaching your animal that you frequently have good things for them, so it's worth their while to keep track of you and listen to you. If you can get their attention you're 90% there.