Introduction to Generative AI for Microscopy 

When discussing generative AI today, most initially think of large language models like ChatGPT capable of producing human-like text. While traditional AI models are designed to recognize patterns and make predictions, generative AI advances further: it creates new, authentic-appearing data nearly indistinguishable from real data, enabling the creation of novel content—whether text, images, music, or videos. Particularly in image processing, generative AI has achieved remarkable advancements. 

Notable applications include: 

• Style Transfer: Transferring an artwork’s style to another image, making a photograph resemble a Van Gogh painting 

• Image Restoration: Reconstructing and enhancing old or damaged images through AI 

• Image Generation: Models like DALL-E or Stable Diffusion creating realistic images from text descriptions 

A particularly compelling application is Super-Resolution, utilized in microscopy. This technique artificially upscales low-resolution images via AI to reveal finer details while simulating higher resolution. 

Inspired by style transfer and super-resolution work, we’ve developed a “Microscopy Modality Transfer” in collaboration with Saarland University and the University of Michigan, Ann Arbor, USA: utilizing a generative AI model to transform low-resolution light microscopy (LM) images into high-resolution scanning electron microscopy (SEM) images, enabling microstructure quantification at scales typically inaccessible via LM. 

Motivation: Overcoming Light Microscopy Limitations 

LM remains the standard tool for microstructural analysis, enabling quick and simple examination of large areas, particularly in daily quality control. However, certain microstructural components’ submicron structures cannot be resolved due to the visible light wavelength limitations inherent to LM. SEM has established itself in research and industry for analyzing these structures. Compared to LM, SEM’s high resolution and various contrast mechanisms offer superior microstructural analysis capabilities, though SEM imaging is more time-consuming and costly. 

We selected dual-phase steels as our use case material. These provide excellent property combinations through tailored microstructures. The mechanical properties strongly depend on the type, phase fraction, distribution, and submicron features of the secondary phase carbides (e.g., bainite lath width or pearlite interlamellar spacing). The latter cannot be resolved via LM, typically necessitating SEM deployment. 

Data Foundation for AI Models in Microscopy 

As with all AI models, underlying data is key to success. In this case, correlative microscopy forms the data foundation—we imaged the same sample location via both LM and SEM, subsequently registering these images congruently. This process yields fully correlated, pixel-accurately superimposed LM and SEM images, enabling us to train the AI model to transform LM images into high-resolution SEM images. 

Various generative AI models were employed: 

• Traditional encoder-decoder networks (AdaIN variant) 

• Generative Adversarial Networks (Pix2PixHD variant) 

• Modern Diffusion Models (Guided Diffusion (GD) and Super-Resolution via Repeated Refinement (SR3) variants) 

Results of AI-Assisted Microscopy 

The subsequent figure displays synthetic SEM images generated with various generative models. Expert assessments and analysis of multiple image metrics demonstrate:

Diffusion Models deliver the most realistic images. 

Comparison of LM and generated synthetic SEM images with different AI models 

Addressing the value proposition for potential later application, we investigated the accuracy with which we could determine microstructural parameters such as size and shape of precipitated carbides, bainite lath width, or pearlite interlamellar spacing from these LM-generated synthetic SEM images. Microstructural parameters determined from the Diffusion Models’ synthetic images exhibit relative errors between only 1-12%. 

How can we explain generative AI’s capability to master this complex task?

First, despite low resolution, LM images likely contain hidden information barely perceptible to the human eye but detectable by AI. Additionally, AI can learn the relationship between LM and SEM from correlated training data and, once the LM resolution limit is reached, estimate how these structures would appear in SEM—similar to a human expert with years of experience. 

Implications and Outlook for AI-Assisted Microscopy 

• We establish Diffusion Models as the state-of-the-art for Microscopy Modality Transfer, demonstrating AI-enhanced microscopy’s potential for improving light microscopy with structural reconstruction capabilities in the submicron range. 
• We’ve taken the first step toward “Surrogate Microscopy”: through AI-enhanced light microscopy, we can sufficiently approximate SEM analysis, significantly accelerating microstructural analysis. 
• For the examined dual-phase steels, we can quickly and easily determine microstructural parameters such as size and shape of carbides, bainite lath widths, or pearlite interlamellar spacing via LM—parameters that would otherwise require more time-consuming SEM analysis. 
• Further studies are necessary to determine these approaches’ limitations and define a secure parameter space for generative AI application. 

Overview of the methodology and results of AI-assisted microscopy

Additional research information is available in our open-access publication:
https://www.sciencedirect.com/science/article/pii/S1044580324009811