Some prominent brain researchers have argued that “emotions” are something that can only be studied in humans, and not in animals. To those of us who are pet owners, this position seems absurd. Isn’t it obvious that our dogs and cats, including my cat, have emotions? Maybe, but intuition is not enough. We must seek evidence, because animals are not little people in furry costumes and we can be fooled.

We typically attribute emotions to an animal species we can identify with. If a squirrel in Central Park freezes or runs away from me, it must be afraid—because I would feel afraid if I encountered an animal 12 times taller than me. Yet without access to the animal’s inner life, how can we be sure that it’s not simply exhibiting an automatic reflex? If a fruit fly freezes or jumps away from us, is it also “afraid?” If it’s just a reflex, why wouldn’t that also be true of the squirrel?

The temptation to project our own feelings onto other species is strong, especially other mammals. Monkeys frolicking with each other must be enjoying themselves. An elephant’s eyes leak fluid when a relative dies; we infer it is sad. Our dogs roll on their backs with their paws in the air; we conclude they are happy to see us. Whales singing in the ocean’s depths sound lonely and lions roaring after a kill must feel “triumphant.”

But we are even willing to attribute emotions to animals that are nothing like us. A captive octopus floridly changing color as children tap on its tank invites us to believe it is expressing irritation. But it may simply be trying reflexively to match its skin coloration to the flashing reflections of its human visitors. Moreover, if we insist that the octopus has emotions, then why not the same for its molluscan cousins? When a sea scallop encounters a predatory starfish, it rapidly snaps its shells open and shut as it somersaults to safety; is that panic? We often refer to bees swarming from their hive to attack an intruder as “angry.” If so, are fighting fruit flies (yes, even male fruit flies fight over females) also “angry?” Or are all of these diverse creatures just performing automatic survival behaviors, hard-wired into their brains by eons of evolution?This is more than just an academic issue. Answers about animals could provide much-needed aid to research into human mental health. Because of our lack of understanding of how the brain controls emotions, there has hardly been a fundamentally new drug for treating mental illness in the last 50 years. Indeed, most pharmaceutical and biotech companies have given up the search after costly failures.

Current treatments for serious psychiatric illnesses like depression, schizophrenia or bipolar disorder remain inadequate —and those that work often have damaging side effects, likely because most such drugs just flood the brain with chemicals like serotonin or dopamine. That’s like changing the oil in your car by opening up the hood and pouring a can of lubricant all over the engine, in the hope that some of it will dribble into the right place. Maybe so, but a lot of it will seep into places where it does more harm than good.

Human research into mental health and emotion typically relies on brain scans. But such studies alone can only identify correlations, not cause and effect. For that we need to enter and perturb the brain, its neurons and circuits. For ethical reasons, this cannot be done in human subjects; we need well-controlled neuroscience studies of emotions in laboratory animals. That means we need to determine whether a given animal’s behavior expresses an emotion or is just an adaptive reflex.

My Caltech colleague

Ralph Adolphs

and I have argued that to study emotions in animals, we should go beyond “feelings,” since animals can’t communicate those to us. Conscious feelings in humans are just the exposed tip of the brain’s emotional iceberg; there is a huge unconscious part below the surface that we share with many other creatures. The part below the surface involves internal brain states, or characteristic patterns of electrical and chemical activity. These brain states, the building blocks of emotion, are manifested by behaviors that have telltale signs that distinguish them from reflexes.

One such building block is “scalability.” Emotional behaviors often escalate in their intensity, from threats to attack or from sniffling to sobbing. In contrast, reflexes tend to be all-or-nothing. Another feature is “persistence.” Emotional behaviors tend to linger after their inciting stimulus disappears, whereas reflexes terminate quickly. And unlike reflexes, internal emotional states show “generalization.” A bad day at the office for humans will influence how you respond to a screaming child at home, and animals have their own equivalents.

Recent research has revealed evidence of these emotional states in “fight-or-flight” responses in both mice and fruit flies. For example, mice exposed briefly to a natural predator exhibit lingering avoidance of open spaces lasting minutes, indicating persistence. Moreover, their responses escalate from avoidance to freezing to running and jumping, as the predator approaches. Male mice exposed to a predator delay the resumption of interrupted mating or feeding until some time has passed, indicating generalization. These indicators collectively suggest that the response to the predator is not simply a reflex, but more likely a manifestation of an internal brain state of defensive arousal or threat alert.

Even fruit flies can exhibit emotional states in their escape behavior when exposed repeatedly to a shadow passing overhead (which mimics an approaching aerial predator). If the flies are enclosed in a transparent arena so they can’t fly away, their response escalates with each successive pass of the shadow, from interruption of feeding, to running around the arena perimeter and hopping like popcorn. These responses persist for minutes after the shadows have terminated, as the flies gradually “calm down” and return to their food. These tiny insects behave remarkably like birds that scatter into the trees when you approach their feeder, and which gradually return to their food only after a while, when the danger has passed.

Once we have identified behaviors that exhibit emotion in a given species, we can use powerful new neuroscience methods to understand how they are generated. In one such method, called optogenetics, specific populations of neurons are genetically modified to activate proteins that convert light into electricity. The neurons can then be activated or inhibited at the flick of a switch that delivers light pulses through tiny optic fibers inserted into the brain. Using such methods, my laboratory and others have discovered small groups of neurons in a brain region called the hypothalamus that control the strength and length of fear or aggression states in mice. Optogenetics cannot yet be performed in humans, for technical reasons and because the long-term safety of the genetic modifications required is not yet known.

If we understood better the neurons, circuitry and chemistry that control internal emotion states, we might eventually be able to develop new drugs or brain-stimulation therapies directed exclusively to those neurons. Such a treatment would, in effect, be like pouring the oil just into that part of the engine where it belongs. We need to study emotion in animals to get there.

—Dr. Anderson, a biology professor at California Institute of Technology, is the author of “The Nature of the Beast: How Emotions Guide Us” (Basic Books), from which this essay is adapted.