Facial Recognition: a role for Open Source software and Transparent data

The accuracy of facial recognition has improved dramatically over the past five years. Advances in both hardware and software have led this technology to the cusp of becoming an everyday part of daily life.

It has the potential to bring many practical benefits. However, it is surrounded by accusations of inaccuracy, bias against ethnic groups and has the potential for flagrant abuse by both corporations and governments.

This post provides a basic introduction to the technology, the controversies which surround the use of facial recognition. It then explores what role open source software and transparent data sets can play in helping people understand the technology.

What is Facial Recognition technology?

Type the phrase “facial recognition” into your favourite search engine and you are liable to get back many millions of hits. The results should leave you in no doubt that this is one of the hottest and most controversial technologies around at the present time.

Facial Recognition has its origins as an obscure research project back in the 1960s. But today, market analysts are predicting the global market for facial recognition technology and services to grow to be worth billions of dollars over the next decade. For example, Markets & Markets are predicting that the global facial recognition market size will grow from USD 3.2 billion in 2019 to USD 7.0 billion by 2024, at a CAGR of 16.6% between 2019 to 2024.

Another metric of growth is to track the amount of venture capital that is being pumped into seed funding for startups in the Facial Recognition field. A quick search shows that prospects of growth for this sector has attracted many millions of dollars of investment over the last few years.

This rapid progress owes its growth to major advances in both imaging hardware and sophisticated AI-based identification software. Now we have reached a point where Facial Recognition applications are becoming increasingly ubiquitous. And it’s ubiquity will only grow as innovators and marketers continue to dream up novel uses for this remarkable technology.

How does it work?

In simple terms, the basic technology works by capturing the image of a human face using one or more cameras. This image is then analysed by software to pick out key facial features such as the spacing of the eyes, shape of the nose, the contour of the lips. Once these have been analysed, they can then be compared an online library of facial images in the search for a match.
However, analysis of the raw image is complicated by a range of factors, such as:

Distance of the subject from the camera
Lighting conditions
The viewing angle between the camera and the subject
Head or facial coverings such as glasses, hats, beards, make-up
Facial expressions such as laughing and frowning

Researchers have developed some clever adaptations to overcome these problems. Although this has led to improvements in technical accuracy, it has not prevented accusations of bias. In simple terms, some implementations of facial recognition technology have exhibited a poor track record for identifying faces of people in specific gender and ethnic groups.

In the extreme, this has led to sensational headlines such as the Newsweek article from 2017 which posed the question “Is the iPhone X racist?” .This was in response to the revelation that following the launch of the iPhone X in China, Apple’s AI based facial recognition app was unable to accurately tell Chinese people apart.

How accurate is it?

Facial Recognition technology has become mired in controversy for a wide variety of reasons. Civil rights groups warn of the danger for potential abuse of this technology by governments, private companies and law enforcement agencies.

On a more subtle level, we should perhaps not be surprised by allegations of inaccuracy and bias. After all, the algorithms used in these types of applications are conceived by human designers. So perhaps inevitably, the outcomes reflect the design trade-offs and unconscious biases of the algorithm designer.

Fortunately, we can attempt to address the question of bias in a rational manner. At regular intervals since 2000, NIST has conducted evaluations of the facial recognition products which are available on the market. The most recent of these was in 2019.

A summary of NIST’s experience with these products was given to a Homeland Security committee session in February 2020. NIST concluded that:

There had been massive gains in both search speed and accuracy of the Facial Recognition algorithms in recent years. This was attributed to a new generation of algorithms which employed ‘deep learning’ techniques.
‘False positive’ outcomes are much more common that the incidence of ‘false negatives’: this defect was common to many of the algorithms which were tested.
‘False negatives’ outcomes tend to be more associated with specific algorithms
The impact of ‘false positive’ or ‘false negative’ outcomes was dependant on the type of application domain. For example, a false positive identification could have serious implications for an individual who has been wrongly accused of a crime. By contrast, a false negative could result in an unauthorised person being wrongly granted access to a resource.

In terms of the specific accusations of bias, NIST concluded:

‘False positives’ are higher in women than in men and are higher in the elderly and the young compared to middle-aged adults
‘False positives’ were found to be more common for faces of Asian, African American and Native American subjects

The suspicion voiced by NIST is that the wide variation in performance between the different algorithms might be due to the quality and diversity of the data sets which are used to train the algorithm.

Clearly, this presents us with a problem. In many cases both the algorithms and the data training sets are proprietary and not open to public inspection.

This naturally leads us to wonder if open source software and public data sets could help to reduce some of the problems associated with Facial Recognition technology.

A role for open source solutions

It’s no exaggeration to say that for many investors and companies in the tech sector, facial recognition technology represents a new ‘gold rush’. There is a large and growing list of companies which have filed patents for face recognition technologies: Facebook, LG, Apple, Amazon, Intel, Canon, IBM, Samsung, Sony, Google, Microsoft are just a few of the more familiar names.

Given that so many proprietary facial recognition techniques have been developed, is there any room for open source solutions?

It goes without saying that for the developers of open source facial recognition software, the existence of so many patents in this field makes it a minefield for the unwary. And yet, despite these constraints, there is significant activity in this space. The AwesomeOpenSource website currently lists more than 150 such projects.

In this section, we will look at a sample of the common open source implementations which are available for use. Included in the list are some emerging uses of facial recognition, such as Emotion Detection and Face Tracking for Augmented Reality.

OpenFace: this was released in 2015 under an Apache 2.0 license. It’s an implementation in Python and Torch of the FaceNet face recognition framework that was developed by Google. The original concept was described in the 2015 paper FaceNet: A Unified Embedding for Face Recognition and Clustering

DeepFace was released in 2014, as part of a Facebook research programme. It’s written in Python and is published using an MIT license.

OpenEBTS is an open source initiative which is fostered by ImageWare Systems. It is published under an Apache v2.0 license. It’s remit is rather broader than just facial recognition technology.

Flandmark is an open source C library ( with an interface to MATLAB ). The primary role is for analysis of static facial images The software is published using a GPL v3.0 license.

Open BR is a Facial Recognition framework for facial image analysis, written mainly in C++ . It is published using an Apache v2.0 license.

Open CV provides a foundational Facial Recognition tookit for Computer Vision and Image Processing. It is leveraged by other projects, such as Open BR. The software is published using a 3-clause BSD license.

OpenFace addresses the emerging field of ‘facial emotion recognition’. This is where an image of a subject is analysed and an inference is drawn about the emotional state of the subject. There are a number of commercial products out there. However, this particular software is firmly anchored in the world of academic research. It’s published with a ‘ACADEMIC OR NON-PROFIT ORGANIZATION NONCOMMERCIAL RESEARCH USE ONLY’ license.

EmoPy: this is a Python toolkit for ‘facial emotion recognition. It was developed by ThoughtWorks Arts and is published using an AGLP v3.0 license.

Nvidia AR SDK: enables real-time 3D tracking of a person’s face using a standard web camera. This SDK enables Augmented Reality applications. It is published using an MIT license.

The above list is by no means comprehensive. It intended to provide the reader with the broad overview of the type of open source solutions which are available, together with the range of software licenses which are employed by developers.

Open source data sets

As the NIST study has suggested, the quality of a training data set can influence the accuracy of your facial recognition software.
There are a few points which you need to bear in mind when selecting a data set:

What type of facial recognition problem you are trying to solve? For example, you might need a set of images which use different lighting conditions or faces which display a range of emotions.
Are there any licensing constraints on the uses of the image set? For example, is the set only available for research purposes? And don’t forget that individual images may be subject to copyright restrictions.
How diverse is the image set? For example, you may wish to have a broad set of images which reflect age, gender and ethnicity.

One final and really important factor is to ask “Where do the images come from?”. Some data sets may have ‘scraped’ images from the web. If the images are purely for research purposes, this approach can perhaps be justified.

However, there is a a darker side to this practice. For example, Clearview AI is a company which has recently found itself mired in scandal and under investigation from privacy regulators. It is rumoured to have a built a database of about three billion images, which it scraped from publicly accessible websites such as Google and Facebook. While this case refers to a commercial service provider, the basic principle is clear. If you have any doubt as to the origin of the images, it’s best to avoid that particular data set.

Now let’s take a look at a sample of the data sets which are out there and the type of licenses which are in use.

Flickr-Faces consists of 210,00 images, each file with its own metadata which describes the licensing constraints. It comes with a Creative Commons license and is only available for non-commercial purposes.

Labelled faces in the Wild consists of 13,233 images of 5,749 people, which have been harvested from the Internet. This dataset was assembled by the University of Massachusettsfor research purposes. Licensing information is difficult to find.

Celebrity Faces (CelebA) contains 202,599 images of celebrities. It is licensed only for non-commercial research and educational purposes.

Tufts Face Database consists of more than 100,000 images of various types. For example, it includes photographic portraits, thermal images and 3D images. It is licensed only for non-commercial research and educational purposes.

UMD Faces is an extremely large dataset which consists of both still images and video clips.

The above list should give you a flavour of what is out there. For a more complete list ( with licensing information ), take a look at the datasetlist.com website.

Benefits of open source

Having surveyed the current range of open source solutions and public data sets, it’s possible to identify a number of benefits:

Education: the ability to examine source code can help both students and technologists learn more about Facial Recognition
Transparency: the availability of public data sets invites people to question the provenance of data sets and issues of possible bias
Ethical benchmarking: the concern for ethics amongst the open source community stands in sharp contrast to some of the more questionable practices found in the commercial world
Productivity: using open source as a starting point for a project can save both time and coding effort. But be careful to choose a license that is appropriate to your intended use.

Conclusions

Facial recognition technology is rapidly becoming a ubiquitous part of everyday life. It has become almost trivial to code a basic application. Indeed. Microsoft Azure’s ‘Face’ service boasts: “No machine learning expertise is required.”

Consequently, all practitioners in the Information Technology now need to have at least a basic understanding of this technology. This is important from the perspective of understanding the benefits of Facial Recognition technology, as well as calling out abuses, wherever they may be found.

Open Source Specialist Group

A Specialist Group of the British Computer Society