The dangers of the return of physiognomy through facial recognition technology

With the Cambridge Analytica + Facebook scandal, more people have begun to worry and question the security of their information and privacy. At the same time, however, more and more cameras are being installed in public spaces, whether for security reasons or for facial data collection—which led, for example, to the lawsuit against ViaQuatro, the company responsible for the yellow line of the São Paulo subway, which recently installed advertising screens with cameras on public transport doors.

On the other hand, facial recognition and the use of biometrics have been implemented in different market sectors—from Gol's selfie check-in to automated kiosks that perform facial recognition based on electronic passport registration. Furthermore, commercial establishments, such as this Japanese coffee shop, are selling coffee to be paid for with customer data. What all this proves is that data, whether generated from our online habits or constituting our very being (biometrics, genetic code, etc.), has proven to be the true new oil: both for its market value and for its versatility in generating wealth and products.

However, one of the problems raised by Jesse Emspak's article for the website Undark is how the combination of facial recognition offered by artificial intelligence systems may be promoting the resurgence of a pseudoscience known as physiognomy. As if the fake news and the problem of post-truth were not enough, researchers are pointing to the dangers of reviving this pseudoscience that relates an individual's facial features to possible moral and character aspects.

“Modern computers are much better at scanning minute details in human physiology, as the proponents of this recent research point out, and therefore the inferences they make are more reliable. Critics, however, argue that this is a fallacy. There is little evidence of a connection between a person's physical characteristics and anything like behavioral prediction, they claim. And, in any case, machines only learn what we teach them, and humans—held hostage by biases and prejudices, from the most obvious to the most subtle ignorance—are terrible teachers.”

Emspak cites as an example the research conducted by engineering professor Xialon Wu, from McMaster University, together with Xi Zhang from Shanghai Jiao Tong University. The study was published as a paper on ArXiv.org, in which the researchers point to the possibility of a computer predicting who could be convicted of a crime based simply on scanning photographs. Faced with the criticism received, the researchers took the study down, and Wu continued investigating another machine learning methodology that could be used to infer personality traits in women whose photographs were associated with different words such as attractive, captivating, sweet, or even imposing, according to a group of young Chinese men.

As Emspak points out, the abstract of this research argues that the algorithm only learned, aggregated, and then “regurgitated” human perceptions, which means that the program would be subject to making errors. Additionally, the authors also say that their empirical evidence points to “the possibility of training machine learning algorithms using facial image examples to predict personality traits and behavioral propensity.”

Something similar was also raised earlier this month, when Michal Kosinski and Yilun Wang, both from Stanford University, released a preview of their study suggesting that it would be possible to detect whether a person is homosexual or not based on photos used on dating sites. Although the study was approved by the respected publication Journal of Personality and Social Psychology, due to public pressure, the reviewers decided to check the work once more through a review focused on the ethical aspect of the research.

All these hypotheses raised by these studies, however, have a very old origin, dating back to the time of Mesopotamia. However, it was in the mid-19th century that the Italian criminologist Cesare Lombroso revived this pseudoscience to detect crimes. Lombroso believed he could detect criminals based on anomalies in facial structure; that is, having an asymmetry in the face or an oblique curvature of the forehead could indicate that these people had some type of character deficiency. Although his ideas were abandoned after World War II, some scientists continued trying to establish a connection between the appearance of the body and the mind.

“Many of these studies deal with human perception—whether we react differently because of what we see, even if that perception is uncertain. Others look for morphological differences between humans, seeking indicators of traits such as aggression, introversion, or homosexuality. Several studies, for example, have indicated that the ratio of the height and width of the face—a metric that can be mitigated by varying testosterone levels—can give clues about a person's tendency toward aggression.

But with the growth of databases and the advancement in computer processing power, some scientists and psychologists have come to believe that ‘neural networks’—systems connected in a network and used in machine learning—can analyze physiological and psychological traits in a less biased way than before. And if before scientists could only study a few factors by using photos of students, for example, and measuring the traits of a few dozen people, the internet has opened up infinite possibilities for samples. Facial recognition software can take measurements of thousands, even millions of people using photos taken from the internet, enabling them to access more data and, hopefully, of better quality.”

Although, at first glance, it may seem like something incredible and even positive to some people, some of the criticisms made of this endeavor are that it is just bad science hidden under mathematical arguments. Emspak argues that, instead of people making these unfounded measurements, the only difference is that now it would be algorithms, which does not eliminate the ethical, privacy, and potential bias problems that can be intentional or not. In any case, these biases end up generating false positives, that is, fallacious conclusions such as those that lead one to believe that, for example, Black people have a greater propensity to commit crimes due to the proportion of the ethnicity's presence in the prison population.

What Emspak presents next is that there are indeed some scientists who argue that there is a connection between some physical characteristics and a person's mental functioning—especially if one takes into account the action of the hormone testosterone and how it acts in the formation of a person's face, for example. Heading the facial research laboratory at the University of Glasgow, Lisa DeBruine describes herself as an evolutionary psychologist, but she does not hide the fact that there are several problems in her field of study. In the case of the conclusions made regarding the height-to-width ratio of a face, DeBruine says that the studies are all confusing and messy, especially due to the inconsistencies in how humans and machines measure data such as the distance from the chin to the eyebrow or between the cheekbones.

“DeBruine was one of several critics of Kosinski's study, which sought to train a computer to identify who was gay and who was straight based only on scanning the faces of people on dating sites, where users clearly indicate what their sexual orientations and preferences are. Kosinski argued that, from two new photos, one of a heterosexual and another of a gay person, the computer was able to guess with a high proportion of accuracy which one was gay. He also tested the algorithm on Facebook photos that the computer had never accessed, and it seemed to get it right again—at least in the case of gay people. The algorithm seemed less effective at identifying heterosexuals. (…)

In his article, Kosinski said that the computer was zeroing in on certain physical aspects of the face that are consistent with the prenatal hormone theory, which suggests that people are homosexual because they were exposed to different levels of androgens in the womb. ‘The faces of gay men and lesbians had gender-atypical aspects,’ the scientist wrote, ‘and predicted by the prenatal hormone theory.’ Kosinski argues that this theory is ‘widely accepted’ as a model for the origin of homosexuality.”

However, as Emspak continues, critics also pointed to several other problems in Kosinski's formulation, including different stereotypical conclusions and an inability to accept that the prenatal hormone theory is not as absolute as it seems. In other words, it is possible that computers are picking up on certain uncertain aspects that, nevertheless, were pointed out as being capable of defining a “degree of homosexuality.”

For Greggor Mattson, a sociology professor at Oberlin College, the idea that testosterone exposure in the womb causes people to “become” homosexual depends heavily on supporting the stereotype that gay men are necessarily effeminate and that lesbians are masculinized women—something reinforced by Kosinski's work. For Mattson, the problem is not in the programming and computational logic of the research, but rather in the way these data are being interpreted.

“In any case, even if there really are physiological differences between the faces of criminals and non-criminals in the photographs analyzed by the computer, this simply reveals more about stereotypes within the Chinese criminal system than the people in the photos. Critics also noted the lack of transparency in the origin of these photos pointed out as being of convicted criminals. (Wu says that, in the study, some of the photos are from the Chinese police department, with whom he signed a non-disclosure agreement).”

Wu’s subsequent studies on reactions to the faces of attractive women have a similar problem, Bjork-James suggests. It might be a good study on stereotypes of Chinese men, he says, or even on the effect women are trying to provoke in Chinese men, but it says nothing regarding the behavior or personality of those women.

Beyond China, a startup in Israel called Faception is also bringing back physiognomy to identify pedophiles and terrorists by using facial recognition technology combined with scanning for certain physical traits. Affectiva, a Boston-based company, also offers a facial recognition system that claims to be able to detect emotions in response to advertisements, thereby supporting advertising and marketing professionals.

For Jonathan Frankle, a doctoral student at MIT and former employee of the Center on Privacy & Technology at Georgetown University, it is often unknown how these algorithms actually work. “Unlike standard code, a neural network involves nodes that are interconnected, with processes happening in parallel. A neural network isn’t simply executing a series of instructions in sequence; instead, the nodes are talking to each other, giving feedback to one another. There really is no way to track how they make their decisions—no one can look at a single line of code or a subroutine. It is effectively a black box,” writes Emspak.

In other words, it becomes very difficult to say how these algorithms are reaching the conclusion that a person’s face indicates whether they are homosexual or not, as predicted by Kosinski’s study. According to Emspak, when the researcher tried to create a generic face that represented the typical face of a gay man, it wore glasses. Beyond the fact that there is no correlation between homosexuality and vision problems, Kosinski justified that wearing glasses was an element that had more to do with a style choice among gay men. In other words, the researcher’s argument, in itself, already undermines the hypothesis that facial features are determinants for indicating a person’s sexual orientation.

“Not knowing what an artificial intelligence is doing is particularly concerning when a startup announces that it can detect threats, as Faception claims. Citing the need to keep its operations secret, the company’s CEO, Shai Gilboa, does not say exactly how the algorithm was designed, or what kind of data they are using to train it. What he emphasizes is that Faception’s systems are based on solid research and that they have no biases. ‘It is very important from an ethical point of view,’ he says. ‘We have proven repeatedly that there are no measurements of color or race.’”

Affectiva, on the other hand, does not claim it can detect anything about innate personality traits from facial structure, only a person’s emotional states. Furthermore, CEO Rana el Kaliouby comments that her company made a point of training its artificial intelligences with people from many countries, using those populations as benchmarks (so, for example, a version of the software running in China or Brazil would be trained using images of people from that country).

Finally, just as difficult as the technical aspect of this technology is the ethical scope surrounding such development. Just as Kosinski used photos for his studies without authorization, the researcher also did not work in partnership with any organization representing LGBTQ+ rights. Emspak comments on how GLAAD, for example, which is one of the most prominent LGBTQ rights organizations in the United States, published a statement alongside the Human Rights Campaign repudiating the methods of Kosinski’s study.

In the case of Wu’s research, the photos of convicted criminals were provided by the Chinese police department, although they did not specify which ones. Thus, both studies end up raising the issue of data privacy as these companies and governments appear to be leaking such information without authorization and, worse, for the creation of discriminatory algorithms.

“But it is really a thin series of culturally constructed conclusions that are building the algorithmic foundation of facial recognition that are the most concerning part, according to critics. ‘Social psychology has a serious cis, straight, English-fluent male problem (like many other fields), which definitely leads to these types of studies,’ DeBruine said in an email. ‘I can’t imagine that [Kosinski’s] study would have gone as far as it did if there had been more diversity among the people with influence at certain points in that journey (fellow researchers, ethics committees, editors, reviewers, etc.).’”

On the other hand, Frankle is glad that these studies are happening publicly. “I would rather this be done in the open by academic researchers than by private companies that will sell it to some police agency,” he says.

Fortunately, most of the experts interviewed by Emspak stated that these problems are not impossible to solve: it would just take more diversity in the field of computer science, as well as greater dialogue between that area and other disciplines. For Mark Frank, a professor of communication at the University at Buffalo in New York, a little more humility and skepticism is also necessary. “Computer scientists’ understanding of behavioral issues is cute, that’s the word I’ll use,” he said in an interview. This is because, as Frank notes, there is much more complexity in human behavior than what is being measured by these researchers. For the communication professor, any connections between physical and behavioral aspects are much more subtle than these studies seem to point out.

For evolutionary biologist Carl Bergstrom of the University of Washington, this type of research will still take a long time to be perfected. “I really think there is this very old notion in human thought that our external appearance reflects our mind and our morality,” he states. For Bergstrom, therefore, it is not surprising to see this type of conclusion return from time to time in an attempt to reaffirm itself and find some foundation based on the prevailing logic of each era—in our case, it is in science and technology that this reasoning seeks a new affirmation.

‍