27643 Deep Face Recognition in Historical Portraits

Probleemstelling:

While deep learning is now already for some time very successful in face recognition for identity authentication and automatic retrieval of digital photos of the same person [1], this thesis presents a very refreshing challenging task: face recognition and portrait identification in paintings! This topic becomes increasingly more important also in the scope of a wider initiative 'AI & Arts' .

An often encountered problem in art historical praxis is the identification of portraits, certainly when no historical data are available to prove the identification. Comparison with other known portraits of a person is often dubious because of stylistic differences, different age, etc. As an example, we mention the documented portrait of the famous humanist Thomas More by Hans Holbein the Younger (New York, Frick Collection)(Pl. 1). It is tempting to see in a Laughing Democritos by Quinten Massys (Pl. 2) a ‘hidden’ portrait of Thomas More, but simply based on visual comparison, it is impossible to prove this hypothesis. The circumstantial historical context is complelling. Erasmus dedicated his ‘Praise of Folly’ to his friend Thomas More and addressed his as ‘Democritos’.

The leads to the research question of this thesis: Is it possible to solve such intriguing art historical problems using advanced face recognition techniques?

This research fits in the emerging domain  AI & Arts, where the research group GAIM is very active (e.g., in AI&Arts webinars at the Alan Turing Institute, international AI&Art webinars, and VAIA's AI in Cultural Heritage).


 

 

 Today, deep learning models like DeepFace [2] are surpassing humans in face recognition tasks. Fig. 1 shows a typical face recognition system comprising three modules: face detector (detects faces in images or videos),  face alignment (normalization to a canonical view) and a deep learning architecture for face recognition. 

Despite the outstanding results of deep learning in traditional face recognition tasks, the application to historical portraits is not sufficiently explored yet and this problem is considered to be very challenging [3.,4]. Obvious reasons are differences in style and painterly techniques as well as the imagination of the artist that affects in different ways likeness of the portrait to the person represented in it. Furthermore, common deep learning-based models in computer vision based on convolutional neural networks (CNNs) rely on the availability of a significant amount of training data. Although in some cases we do have access to multiple paintings of the same person made by the same artist (like examples in Fig. 1) or by different artists (like examples in  Fig. 2), these examples are never abundant and may not be sufficient  to reliably train CNN models. Hence there is need to make further research on how deep  face recognition models can be applied to historical portaits.


                                                                  Figure 1: Deep face recognition with face detector and alignment. [3]

 

   
                          Figure 2: Mary Queen of Scotts by Clouet: works of the same sitter by the same artist. [3] 


                                                             Figure 3:  Paintings of Newton by different artists. [3]


Doelstelling:

The goal of this thesis is to adapt existing and/or develop a novel deep learning model that should learn to identify portraits of the same person in different paintings.  State-of-the-art face recognition methods are typically based on modified CNN architectures, ResNets and most recent ones often apply attention mechanisms too. A recent overview can be found in [5], which also gives pointers to standard benchmarks and datasets for traditional face recognition problems. For the particular problem of this thesis, approapriate dataset of historical portraits will be available to the student. It will be of interest to explore the potential of generative models for this task [6]. This work is exploratory and very innovative - since there is very little known so far about the potential success of deep learning for face recognition in paintings, all the findings resulting from the work can be very impactful.

At the beginning of the semester, the student will be provided with the available literature, initial code, and data.

 

 

References:

  1. Taigman, Yaniv, et al. DeepFace: Closing the gap to human-level performance in face verification. Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR2014) (2014)
  2. M. Wang, W. Deng, Deep Face Recognition: A Survey. Neurocomputing, 429, 215-244 (2021)
  3. R. Srinivasan, R. Conrad, and K. R. Amit. Computerized face recognition in renaissance portrait art: A quantitative measure for identifying uncertain subjects in ancient portraits. IEEE Signal Processing Magazine 32.4. 85-94. (2015)
  4. R. Srinivasan et al. Quantitative modeling of artist styles in Renaissance face portraiture. Proceedings of the 2nd International Workshop on Historical Document Imaging and Processing (2013)
  5. M.D.T. Fuad et al: Recent Advances in Deep Learning Techniques for Face Recognition. IEEE Access 2021 https://arxiv.org/pdf/2103.10492.pdf
  6. Z. Ding et al: Generative One-Shot Face Recognition. Available from arXiv (2019) https://arxiv.org/pdf/1910.04860.pdf