Written by R. Luke DuBois

Representing the United States at the 2018 London Design Biennale for the second time, Cooper Hewitt presented Face Values, an immersive installation exploring the pervasive but often hidden role of facial-detection technology in contemporary society. The installation is now being presented in Cooper Hewitt’s Process Lab. Curated by Ellen Lupton, senior curator of contemporary design, the installation features original work by designers R. Luke DuBois, Zachary Lieberman, Jessica Helfand, and Karen Palmer displayed within a digital environment designed by Matter Architecture Practice. Face Values won the Biennale Medal for most inspiring interpretation of the exhibition’s theme of emotional states.

View of R. Luke DuBois’s installation Expression Portrait for Face Values at Cooper Hewitt. Installation design by Matter Architecture Practice.

The design brief given to the forty participating countries, cities, and territories asked for spaces that drew attention to the ways in which design impacts human emotion. The design team assembled by Cooper Hewitt decided to turn that prompt on its head, asking: How can human emotion be used to impact design?

We are attuned to recognize the ways in which specific facial expressions can reveal our emotions. Some of these telltale signs are reasonably universal—laughter, for example, does similar things to people’s faces worldwide, as does disgust. However, many emotions impact our faces in culturally specific ways. In the United States, we read a smile that uses only your mouth, and not your eyes, as insincere. But in many other parts of the world, a wide grin is considered impolite. Familiarity gives us the best insights—loved ones read our faces like an open book, using facial cues to recognize sadness and anger faster and more accurately than acquaintances or coworkers ever could.

Over the last ten years, advances in computing have given rise to a variety of software that performs different kinds of facial recognition on digital images, videos, or live camera feeds. These systems claim to accomplish tasks or provide some sort of insight that the computer can give more rapidly, accurately, or inexpensively than a human. Some of these tasks are menial. A setting on your digital camera tells you when someone blinked in a photo. Auto-tagging identifies your friends in photos you post on social media. Other uses are downright innovative. A computer vision algorithm detects a person’s heart rate using ordinary webcams. A computer service purports to infer “deception,” partially through looking at facial cues in videos.

The more informed we are about these systems, however, the less we like them. Blink detection requires the computer to successfully disambiguate between eyes that are open and closed, a system that a Japanese camera manufacturer notably failed to deliver on with a product that consistently tagged Asian faces as “blinking” because of the shape of their eyes. Reading between the lines on social media auto-tagging, it might occur to you that if Facebook knows you were at your friend’s birthday party last Friday, so does the government. If you look carefully at the literature proposing and evaluating video-based heart-rate monitoring, you’ll notice that the systems fail on patients with dark skin color, a fact often passed over as something that requires “further study.” And as any armchair historian of lie detector technology knows, machines that use simple heuristics to detect truthfulness exhibit as much bias and predilection for false positives as humans.

Many (if not most) of these contemporary technologies leverage machine learning, a rapidly growing body of computing techniques that most people conflate with artificial intelligence. A.I., a research discipline that encompasses machine learning, is also used by laypersons as a term to describe machines doing things that we, as a culture, consider in that moment to be “human” activities, not “computer” activities. This is a moving target, so an average American in the early 1990s could very well have described things like GPS navigation systems as “artificial intelligence,” even if, from a computer scientist’s perspective, this product does not use A.I. in a formal way.

Machine learning, generally speaking, works through a process of learning and recognition, in a manner deliberately modeled after human cognition. For example, humans learn to recognize cats by seeing lots of cats (in real life or in media) and learning to extrapolate the core features of what makes a cat—whiskers, eyes with slit pupils, tails, etc. We also learn to recognize cats by comparing cats against other things we see and learning to distinguish among them. A cat is not a dog, nor is it a lampshade. Computers “learn” or, more accurately, are trained, through a similar process. They receive a little help from human supervisors, who point out the features to pay attention to in order to recognize, say, types of objects in a picture.

But imagine the following. What if, when we were young, the only cats we saw had gray fur? Later on in life, when encountering an orange tabby cat for the first time, would we recognize it as a cat? Or would we say it was a tiger, or a lion, which looks closer in some ways to a tabby cat than a gray cat? Our dilemma here is that our understanding of a phenomenon had bias. Data sets used to train machine learning systems have bias as well, sometimes to extraordinary degrees, depending on who curated the training data and what their goals were. This actual and potential bias is one of the things that makes machine learning difficult to use even in the most controlled situations. When deployed in society, it can be deeply problematic.

Two women, one seated and one standing, are positioned in a gallery space. They both look intently at a video monitor affixed to a wall covered in dozeons of black vertical rods. The monitor shows text, images of two women, and a live feed of the seated onces facial expressions.

Two visitors experience the installation Expression Portrait in Face Values at Cooper Hewitt.

For the Face Values exhibition, I developed an interactive artwork that asks a computer to detect human emotion to explore these questions. Titled the Expression Portrait, the piece takes the form of an interactive photo booth, similar to the kind you’d find in an amusement park. There’s a chair, a screen, a loudspeaker, a video camera, and a big red button. When you press the button, on-screen imagery and a voice from the loudspeaker encourage you to take a seat and position yourself in front of the video camera for your “emotional self-portrait.” The computer picks a random emotion from seven possibilities of fear, anger, calmness, disgust, happiness, sadness, and surprise, and asks you to act that way for thirty seconds. During this time, a pair of videos shows actors mimicking those emotions, while the loudspeaker defines the emotion and music plays that’s meant to evoke the particular feeling.

At the end of the thirty seconds, the computer shows you an average image, akin to a time-lapse photograph of your face. It then tells you how well you did at your task, giving you the computer’s estimate of your dominant emotion, as well as its best guess as to your age, race, and gender. It then tells you that in the United States, data like this is collected all the time without your consent. To bring that point home, a second screen shows a running slideshow of recent portraits taken with the machine.

Screen captures of R. Luke DuBois’s Expression Portrait.

Screen captures of R. Luke DuBois’s Expression Portrait.

The installation analyzes your face using a machine learning algorithm trained on a number of public data sets of video and images developed in the last five years at universities and regularly used in machine learning systems to recognize emotion, age, race, and gender. For emotion, I considered two large data sets: the Ryerson Audio-Visual Database of Speech and Song (RAVDESS) and AffectNet, developed at the University of Denver. For age estimation, I used the IMDB-WIKI database developed at the Swiss Federal Institute of Technology in Zurich (ETH-Zurich). For race and gender, I used the Chicago Face Database developed at the University of Chicago.

As nearly everyone who interacted with the piece pointed out, it was often wrong. In the context of the installation, it was kind of funny and lighthearted. But in the context of how these technologies are used in society, it’s very troubling.

The Ryerson emotion data set uses video files of twenty-four young, mostly white drama students at that institution “overacting” the required emotions. The actors in the “fear” videos look literally terrified. The “sadness” videos involve tears. The “anger” videos have lots of scrunched-up faces, and so on. On the opposite side of the spectrum, the AffectNet researchers sourced over one million still images of faces discovered by searching for emotion keywords on Google, Bing, and other search engines. The bias factor here is the bias factor of the search engine, which discovers images from celebrity Instagram feeds, movie posters, and stock photography far more regularly than they ever find examples of “regular” people displaying emotions. The upshot of this in the installation was that people really had to overreact to the emotional prompt for the computer to recognize it. An open mouth was almost a prerequisite to show surprise, for example, and trying to look sufficiently angry for the machine to notice was, in itself, infuriating.

The IMDB-WIKI database, by a similar token, uses the photos of celebrities and notable persons to create a data set that correlates faces with human age. This is a pretty ingenious way of curating a data set, as the database creators could reasonably assume that both IMDB and Wikipedia have correct birthdays for the people in question. The problem here, when used against ordinary people (with ordinary lighting and ordinary makeup), is that the data set skews people older. Biennale visitors in London in their twenties were shocked to find that the computer thought they were in their fifties. I had to remind them that the computer wasn’t comparing them to average people at all, but to celebrities like Julia Roberts and George Clooney.

The race and gender data set, collected by professionally photographing 158 representative Chicagoans in controlled conditions, poses an alternate dilemma. It classifies its subjects according to a binary and cisgender understanding of humanity, assuming that we are either male or female, and also adheres to a culturally specific definition of race: white, black, Latinx, or Asian, which falls apart in our modern, global, multiracial world. Londoners of South Asian descent found themselves recognized as Latinx by the system, while visitors who had freckles—a perfectly common trait among people from pretty much anywhere on the planet—were almost always classified by the computer as white, even as they laughed and told me afterwards that their parents were from Jamaica, Ghana, or Bhutan.

Unfortunately, while the installation was intended to be fun, this is no laughing matter. As my colleagues Dr. Kate Crawford and Meredith Whittaker, the codirectors of the A.I. Now Institute at NYU, point out, artificial intelligence is the core ethical and social concern for human society’s relationship with technology in the twenty-first century. In her excellent, visually annotated essay “The Anatomy of an A.I. System,” Dr. Crawford walks readers through what goes into these technologies, using the Amazon Echo—voice recognition software—as an example. The takeaway is incredibly troubling, and these devices and services are rapidly becoming ubiquitous.

So I ask all of you to do me a favor. The next time a device or a piece of software comes along promising to make your life easier by recognizing you in some way, don’t take it at face value.

R. Luke DuBois explores temporal, verbal, and visual structures through music, art, and technology. He is the director of the Brooklyn Experimental Media Center at the NYU Tandon School of Engineering, where he and his students explore the implications of new technologies for individuals and society. His work expands the limits of portraiture in the digital age by linking human identity to data and social networks.

Face Values is on view at Cooper Hewitt through May 17, 2020. This article was originally published in the Winter 2018 issue of Design Journal, Cooper Hewitt’s biannual magazine.

52 thoughts on “Don’t Take It at Face Value

I amm glad to be a visitant of this thoroughgoing web site, regardds for this rare info!

Review mmy web page: designer Womens clothing

I have been exploring for a little bit forr any high quality articles or weblog posts in this kind of space .
Exploring in Yahoo I at last stumbled upon this web site. Reading this info So
i’m glad to show that I’ve an incredibly good uncanny feeling
I discovered just what I needed. I so much indubitably will makle certain to do not fail to emember
this web site and givee it a look on a continuing basis.

Heree is my web page custom logo design services review

This info is priceless. Where can I find out more?

Feel frree to visit my blog poswt :: sell property fast cash

We still can’t quite assume that I could end upp being one oof those reading through the important recommendatiions found on your site.

My family and I are really thankful ffor thee generosity and for presenting me
the advantage to pursue thhe chosdn profession path. Apreciate
your sharing the important informatkon I got
from your web-site.

my pwge – gong therapy

If you are going makeup artist charges for wedding best contents lioke myself, only pay a
quick viisit this website daily aas it presents quality contents, thanks

This web site is my aspiration, really fantastic design and style and Perfect
written content.

my page Personalised House Signs Manningtree

For most up-to-date neews you have where to buy antique furniture online visit the web and on the web I
found this web page as a most excellent web site for mpst redcent updates.

Oh my goodness! Amazing article dude! Thank you, However I am having troubles with your RSS.
I don’t understand the reason why I am unable need to sell house fast join it.

Is there anybody else having the same RSS problems? Anyone who knows
the answer can you kindly respond? Thanks!!

Appreciate thks post. Let me try iit out.

Take a look at myy blog post large leisure battery

We arre a gaggle of volunteers and starting a brand new
schrme iin ouur community. Your web site offered uss with helpful
information to work on. You’ve done a formidable
activity and our whole neighborhood shall be grateful to you.

Alsoo visit my website slate number signs

Whoah this weblog iis wonderful i love reading your articles.
Keep up the great work! You already know, lots of people are
looking round for this info, you can aid them greatly.

My web-siteleather repair cost

Thank you for your bllg post. Brown and I have already been saving for
our new e-book on ths topic and your wrditing has made all of
us to save all prices of car battery our money.

Your ideas really answered all our questions. In fact, over what we had
recognized just before we came upon your great blog.
My parfner and i no longer nurture doubts including a troubled mjnd because
you have attended tto esch of our needs here.


what do virtual assistants charge‘s up every one, here
every one is shwring such experience, herefore it’s nice to read this website, and I used to pay
a quick visit this webpage everyday.

excellent points altogether, you simply won a new reader.
What could you suggest about your put up that you simply made some days in the past?

Any sure?

Takee a look at my web page: Hula Dancer For Hire

Hi, Neat post. There iis a problem with your website
in web explorer, would ttest this? IE still iis the market
leader and a big component to people will miss
your great writig because of this problem.

Heree is my webpage … IMT application London

Hello There.I found sell your home quick blog the use
of msn. That is a very smartly written article.
I will be sure to bookmark itt and come back to read extra of
your helpful information. Thanks for thhe post.
I’ll definitely return.

where can i find a cheap car battery got what you
intend, appreciate it for putting up. Woh I am glad to fiind this website through google.

Hello. fantatic job. I diid not imagine this. This is a great
story. Thanks!

Feel free to sjrf to my page :: litigation solicitors in Surrey

Would love tto always get updated great web site!

My webnpage :: russian volume Eyelash extensions london

What’s up to every one, the contents existing at this web site are genuinely amazing for people experience, well, keep up the nice work fellows.

Stop by mmy wweb site boys suits

There is apparently a bundle places to buy cheap car batteries rrealize
about this. I assume you made some good points in features also.

An impressive share! I have just forwarded this onto a coworker
who had been conducting a little research on this. labels and packaging
he in fact ordered mme breakfast simply because I
discovered it for him… lol. So let me reword this….
Thajks for the meal!! But yeah, thanks for spending time to tapk about this subject here
on your website.

Hi there mates, itss fantastic article concerning teachingand entirely explained, keep it up all the time.

Also visit my weeb site … geodesis candle

I gotta favorite this web site it seems handy very useful.

Take a look aat my web-site –Prunus Serrula For Sale

thank yyou car battery for vw touran this post, I am a big big fan of this web sote would like to
continue updated.

You’ve made some really good points there. I checked on the internet too learn more about the issue andd found
most people will go along replace tub with walk in shower
your views on this web site.

Generally I don’tread post on blogs, but I would like to say that this write-up very compelled me to take a look at and do so!
Your writng taste has been amazed me. Thanks,
verfy great post.

Feel free to sujrf to my blog … employee training management software Bristol

This site really has aall the info I needed about thos
subject and didn�t know who to ask.

My blopg … low cost removals London

F*ckin’ awesome things here. I am very glad to peer your article. Thanks so much and i’m having a look ahead to touch you. Will you kindly drop me a e-mail?

Great blog right here! Additionally your website lots up fast! What host are you using? Can I am getting your associate hyperlink to your host? I wish my site loaded up as fast as yours lol

At this moment where can i buy cheap car battery am going
away to do my breakfast, once having my breakfast coming over again to read
additional news.

I enjoy assembling useful info, this post has got me even more info!

My website Off Road Led Light Bar

hi!,I love your writing soo much! percentage we communicate
extra about your article onn AOL? I require a specialist in this house to unravel my problem.
Maybe that’s you! Taking a look forward tto loiok

Look att my website :: unblocking fallopian tubes London

Some truly interesting etails you have written.Assisted mme a lot, just
what I was looking for :D.

Also visit my wweb blog … Health And aesthetic clinic london

This design is spectacular! You obviously know how to keep a reader entertained. Between your wit and your videos, I was almost moved to start my own blog (well, almost…HaHa!) Great job. I really enjoyed what you had to say, and more than that, how you presented it. Too cool!

you may have an awesome blog here! would you prefer to make some invite posts on my weblog?

Aw, this was an incredibly nice post. Spending some time and actual effort to make a top notch article… but what can I say… I put things off a lot and never seem to get nearly anything done.

What you said wass very reasonable. But, consider this, what iif youu composed a cqtchier title?
I am not saying your information is noot good,
bbut what if you added something that grabbed folk’s attention? I mean Don’t Take It at Face Value
| Cooper Hewitt, Smithsonian Design Museum is kinda boring.
You ought to glance at Yahoo’s home paghe and see how thery write news titles to grab viewers interested.
You might try adding a video or a related picture or two to get people excited
about everything’ve got large house to rent with indoor pool devon say.
Just my opinion, it could make your posts a little livelier.

This page definitely has all of the information and facts I needed about this subject and didn’t know who to ask.

That is a really good tip particularly to those new to the blogosphere. Brief but very accurate info… Thank you for sharing this one. A must read post!

Good post. I learn something totally new and challenging on websites I stumbleupon on a daily basis. It’s always useful to read articles from other authors and use a little something from their websites.

Wonderful post! We are linking to this particularly great post on our site.
Keep up the great writing.

Also visiut myy blog post … Wimbledon locksmith

Thanks for writing this article. I’ll return to see more soon.

What’s up, its nice post on the topic of media print, we all understand mwdia is a impressive source estate agents of data.

I’m excited to uncover this great site. I need to to thank you for ones time for this wonderful read!! I definitely really liked every little bit of it and I have you bookmarked to check out new stuff in your website.

Thanks a bunch for sharing this with all of us you actually know what you’re talking about! Bookmarked. Kindly also visit my website =). We could have a link exchange arrangement between us!

You ought to take part in a contest for one of the highest quality websites on the web. I will highly recommend this site!

The spirit, the will to win, and the will to excel are the things that endure. These qualities are so much more important than the events that occur.’ By Vince Lombardi

On the web, I have actually discovered lots of short articles on this subject. None have actually been as detailed as this one. I simply wanted to say thank you for your understanding. I wish that you continue to write this sort of post for myself and others.

Disciplining yourself to do what you know is right and important, although difficult, is the highroad to pride, self-esteem, and personal satisfaction.’ By Margaret Thatcher

I have to thank you for the efforts you’ve put in penning this site. I’m hoping to see the same high-grade content by you in the future as well. In fact, your creative writing abilities has inspired me to get my own, personal site now 😉

I’m impressed, I must say. Rarely do I encounter a blog that’s both equally educative and amusing, and let me tell you, you have hit the nail on the head. The problem is something that too few men and women are speaking intelligently about. I’m very happy that I came across this during my hunt for something regarding this.

Leave a reply

Your email address will not be published. Required fields are marked *