27796 Transfer learning using local image descriptors for single-cell electron microscopy

Probleemstelling:

Electron microscopy (EM) imaging is emerging as an important imaging tool in biology and medicine, allowing for the detection of structures with nanometer resolution. Labeling different structures (e.g. mitochondria, endoplasmic reticulum) in EM datasets is a difficult and time-consuming task that can be done only by experts whose time is very valuable [1].

It is of great importance to develop methods that enable efficient and reliable search of specific biological structures of interest (like mitochondria) in large EM datasets. 

Local image descriptors (like SIFT [2], or our in-house autoencoder-based descriptor [3]) have shown a great potential for this application. They are enabling us to search for similar patches in an image/dataset. If, for example, a doctor notices an unusual type of tissue, he could easily look up other examples of such issue structure.

Furthermore, although there are various EM datasets available, many of them are (partially) unlabeled. It is thus of interest to explore transfer learning where the labels from one dataset would be leveraged in order to (semi-)automatically label new datasets. This could be achieved using local image descriptors, by using the self-similar search to match the unlabelled patches to the most similar labelled patches. It would also be interesting to explore whether labelling can be performed completely automatically or with a human in the loop.

Figure: Extracted features of a U-Net on two examples of EM images

The student will research different deep learning architectures (based on autoencoders and variational autoencoders [4]) that suit best for this type of descriptor for transfer learning.

This Master thesis is in collaboration with the research institute VIB in connection to the 'AI Flanders' research program, Research Challenge: AI-driven Data Science [5].


Doelstelling:

The goal of this thesis is to explore the use of local image descriptors to perform transfer learning by matching unlabelled electron microscopy image patches to the most similar labelled image patches.

The concrete tasks of this work are:

At the beginning of the next semester, the student will be provided with the available literature, initial code, and data.


References

[1] J. Roels, F. Vernaillen, A. Kremer, A. Gonçalves, J. Aelterman, H. Luong, B. Goossens, W. Philips, S. Lippens, and Y. Saeys. “An Interactive ImageJ Plugin for Semi-Automated Image Denoising in Electron Microscopy.” Nature Communications, 2020. https://biblio.ugent.be/publication/8651412
[2] D. G. Lowe, "Object recognition from local scale-invariant features," Proceedings of the Seventh IEEE International Conference on Computer Vision, 1999. https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=790410
[3] N. Zizakic and A. Pizurica, "Efficient Local Image Descriptors Learned With Autoencoders," in IEEE Access, 2022 https://biblio.ugent.be/publication/8732341
[4] https://en.wikipedia.org/wiki/Autoencoder
[5] https://airesearchflanders.be/ai-driven/