The specific beautification attributes I used comes from Woctezuma’s code, who in turn used code from a user’s project called “seeprettyface.” The website for seeprettyface notes that for creating the labeled data required to determine a generated picture’s attributes (e.g., beauty, gender presentation, smile, etc.), they used a combination of Microsoft’s image recognition API and Baidu Cloud. Baidu Cloud’s Chinese language API documentation describes features for measuring and enhancing the image subjects’ beauty. I could not find any documentation in Microsoft’s API documentation on beauty measurement or manipulation, which suggests to me that the measurements of beauty seeprettyface used to create the labeled data necessary to make the “beautification” filters I used comes from Baidu.

I cannot find a specific reference that identifies how Baidu measures beauty. This may be a failure of my searching, a language barrier, or because the company wishes to protect trade secrets. In lieu of specific information on Baidu’s methods, I can speak to some of the methods and factors that impact other popular beauty filters or digital measurements.

It is important to understand that in order to measure or create a filter for facial beauty, Baidu’s algorithm very likely first needed to find or create some form of labeled data on facial beauty, as is common practice in most machine-learning tasks. They may have created their own dataset from scratch and had people rate each picture, used an existing dataset of faces and had people rate those pictures, or they may have used a dataset created specifically for facial beauty perception that already contains peoples’ ratings for the subjects’ attractiveness. It’s possible that Baidu used a different method, unlikely given the state of the art.

Many sources, including experts speaking on the AI beauty contest in the Guardian article linked in the main text, describe lack of representation of minorities in underlying datasets as a key issue in algorithmically measuring or creating beauty. This is a real issue. Though the dataset FFHQ described previously uses images posted on Flickr, other datasets are more narrow in the subjects represented. For example, the CelebFaces Attributes (CelebA) Dataset created by researchers at The Chinese University of Hong Kong is popular for a variety of data science tasks related to human faces. One paper associated with the creation and use of the dataset was cited by over 4,500 people at the time of writing according to Google Scholar. The popular machine learning library TensorFlow includes CelebA in its data catalog.

As implied by its name, CelebFaces Attributes contains pictures of over 200,000 celebrities’ faces. Though the creators don’t appear to disclose how they chose which celebrities to include, the images appear to be of predominantly Western celebrities. This is supported by sources like the creators of the Diversity in Faces (DiF) dataset, who determined that about 86% of subjects pictured in CelebA had lighter skin tones, and about 78% of subjects were under the age of 46–which doesn’t exactly sound out of line with the demographics of Western celebrities. The authors behind DiF found similar levels of unbalanced age and skin tone representation in other popular face datasets.

We can also consider as an example ​​the SCUT-FBP5500 dataset–a dataset created specifically for algorithmically measuring facial beauty–contains only pictures of Asian and Caucasian subjects. Each photo in SCUT-FBP5500 was given a score between 1-5 for beauty from a team of volunteers aged between 18-27, which invites us to consider the bias introduced by those determining what constitutes beauty in datasets labeled for those purposes.

A final dataset worth considering in this discussion is the HotOrNot dataset. In this dataset, the authors use a subset of photos and their ratings from the website HotOrNot.com. HotOrNot was a website that invited users to rate the hotness of user-submitted photos from 1-10. Mashable recounts its rise to fame and eventual downfall in this piece honoring the site’s 20th birthday. Anecdotally, as a former dumbass kid with the open mindedness of an early 2000s Friends episode, HotOrNot was mostly popular with dumbass kids with the open mindedness of early 2000s Friends episodes. Though not as popular as the other datasets described, the HotOrNot dataset has been used in research on measurement or adjustment of facial “beauty,” such as in this paper on a method for facial beauty prediction.

In reality, the researchers behind all of these datasets bear little of the blame for the lack of representation in their data or methods. It’s a normal practice for researchers to scope their experiments just so that they can test a specific thing. One intended use of CelebA, for example, was to develop and test a method for identifying a person’s face in a variety of different circumstances. So, it makes sense that the authors opted to use subjects that it is easy to find pictures of. One output discussed in the creation of SCUT-FBP5500 is to provide data that can specifically be used to discuss the difference in perception of beauty (specifically the distribution of scoring) in Asian subjects versus Caucasian subjects. Though not explicitly stated by its creators, it is likely the HotOrNot dataset was conceived as a way of gaining a large number of images that had already been rated by a large number of people (the authors only used images with more than 100 ratings) as these kind of tasks can be time consuming and expensive, particularly for that amount of input.

Though there’s certainly more many researchers can do to address lack of representation and other bias in their datasets, much of the folly that comes from attempting to algorithmically measure or enhance beauty–beyond the question of why?–stems from the misuse of data produced for specific tests under specific circumstances. Or, in some cases, the issue may stem from the lazy creation of new datasets for general-purpose commercial products without due consideration of the biases introduced by minimal diversity in the subjects pictures, or in preferences introduced by a largely homogenous group evaluating the images. We can see the results of these issues in the biases found in everything from AI beauty contests, to commercial product APIs, to satirical essays on corporate culture.