Posted
In College of Human Ecology, Psychology

Feelings, although central to human existence, are our most mysterious and subjective sense. Feelings, we are told, are personal and internal, while reality is physical and external. New research published in Nature Communications combines machine learning, perception, neuroscience, and generative AI to reveal that the brain may treat feelings as real as any aspect of the physical facts of the external world. 

It has been long understood that feelings and emotions emerge from a complex subjective mixture of thoughts, memories and body sensations. A new study uncovers how humans might experience feelings from visual data, perceived directly from the visual world, much like how the brain computes features such as color, shape, or size. 

The authors are Saeedeh Sadeghi Ph.D. '23; Zijin Gu Ph.D. '23; Eve De Rosa, the Mibs Martin Follett Professor in Human Ecology; Amy Kuceyeski, adjunct professor in computational biology; and Adam Anderson, professor of psychology.

Using machine learning, the researchers first trained a machine, which had never experienced pain or pleasure and that was blind to the meaning of images, to find objective patterns in visual data that reflect feelings of subjective valence—seeing things as positive or negative. Despite relying only on data from combinations of colors, shapes, and complex visual patterns, the machine could predict people’s valence experience to photographs. Subsequently, the machine was tested on predicting people’s feelings in response to abstract art, which contained no recognizable objects or meaning. The results were even stronger. Visual data appear to contain objective cues to how people may feel about the world. 

To examine whether these uncovered patterns afford a direct and objective way we experience feelings, researchers asked people to report their feelings in response to neutral photographs that contained patterns that machines indicated were positive or negative. Under conditions to limit recognition (presentation of a fraction of a second and upside down), people’s feelings of what they saw aligned closely with what machines predicted, especially when making snap judgments. People agreed with the machine; the visual world contains clues that could serve as direct sources and causes of feelings.

To more deeply examine whether brains can derive feelings from the direct perception of these cues from the external world, the research next assessed brain activity. Pictures containing patterns that were most strongly interpreted by machines as predicting positive and negative feelings, almost exclusively engaged neural activity in the brain’s visual regions; not brain regions traditionally associated with subjective value. The brain treated these feeling cues as external facts. 

In collaboration with Cornell engineers, including Kuceyeski, who innovated NeuroGen, a tool that synthesizes images based on brain activity patterns, they were able to visualize what these neural visual patterns ‘see.’ From visual activity patterns in the brain, NeuroGen used generative AI to create new images that human observers also experienced as positive or negative. This means that brain activity in the visual system contains information that can not only interpret valence but also can generate or ‘imagine’ pictures of what should cause these feelings. 

Providing hints at the origins of these visual cues, the generated images from positive valence neural activity patterns tended to be natural and living organisms, while negative patterns tended to be human-made objects. This supports the concept of biophilia—a potentially innate comfort for patterns derived from nature. The visual brain may be evolutionarily and experientially tuned to objective features as emotionally significant, directly informing us something is likely good or bad. 

This research may provide new understanding how feelings, by design, are experienced as objective facts of our reality. “Due to their survival value, our feelings are deeply intertwined with the fabric of perception itself," said Anderson. "As the brain sees it, feelings can be as objective as, and indeed a part of, the physical world.”