Unlocking Facial Recognition Systems with Master-Key Images

We have invested a considerable amount of time into this project. Once we have completed all the necessary steps, I will provide a comprehensive update on this page, including project details, core concepts, algorithms, and outcomes. I will also publicize the code on my GitHub. For the time being, here I will share both early and recent experiment results. You can also find the final set of the experiments and results on arXiv:2309.05879

Problem Definition

Face verification is the process of determining if two facial images belong to the same individual.

Face Recognition systems have not only surpassed human performance in Face verification, but have also advanced to the extent that these technologies are now employed in various public safety applications, such as border control procedures, as well as commercial applications like unlocking smartphones.

The objective of this project was to create an image or a collection of images that could serve as master keys to unlock a Facial Recognition system. The effectiveness of a master key image is determined by its ability to unlock multiple locks. In this context, a lock refers to a face recognition system, such as the one found on smartphones.

Idea One

In the early attempts, we could generate such images that can fool a soft-max based neural network used for Face Verification task with a high success rate.

Idea Two

By implementing an other idea, we could generate such images that can fool a soft-max based neural network used for Face Verification task with even a higher success rate.

Idea Two - Incorporating a Different Loss Function

By changing the loss function of our model, we generated other nonsensical but instersting images:

Idea Three

Still, we have not yet been able to generate a convincing human face. This poses a problem as it would not pass the initial step in a Facial Recognition System, thus rendering the attack ineffective. In such systems, like a smartphone’s facial recognition feature that utilizes the phone’s camera, the first stage involves extracting the face from the captured image. Unfortunately, the previously generated images are unable to pass this face detection process due to their lack of recognizable human facial features. As a result, our next attempt focused on generating images that not only function as master keys but are also detected as faces by the face detector. In the following section, we present an endeavor to address this issue with a new approach:

However, based on preliminary results, it appears that our attempts with this idea have not produced promising outcomes in terms of both the face detection step and their effectiveness as master keys.

Idea Four

Next, we tried an other idea, that inputs a human face into the system, and generates an image that has the master key propery we want. In the following, I used my face image as the input, and successfully generated an image that can open more than 50% of the locks, effectively acting as a master key:

Idea Five

After conducting tests on softmax loss-based neural networks, we applied our methods to triplet loss-based neural networks. Unfortunately, the initial testing of these methods did not produce satisfactory results for triplet loss-based neural networks. This led us to explore another idea to address this issue. It took considerable effort and time, but eventually, we successfully cracked a triplet loss-based neural network for face verification with a success rate of 58%. This represents a significant improvement compared to the previous work, which only achieved a success rate of 43%. I invite you to take a look at the set of ten images below, where nine of them are modified images of me that can act as master keys. Can you spot the original image among these master-key images?

The Full Study on arXiv

As I highlighted earlier, I’ve presented only the preliminary experiment results here. For the complete methodology and final experiments and results, please refer to arXiv: