Vienna, Austria

ESTRO 2023

Session Item

May 14
10:30 - 11:30
Strauss 1
CT reconstruction and synthetic CTs
Bertrand Pouymayou, Switzerland;
Carsten Brink, Denmark
Proffered Papers
11:20 - 11:30
Bridging the gap between radiology and histology through AI-driven registration and reconstruction
Amaury Leroy, France


Bridging the gap between radiology and histology through AI-driven registration and reconstruction

Amaury Leroy1,2,3, Alexandre Cafaro1,2,3, Vincent Lepetit4, Nikos Paragios1, Eric Deutsch2, Vincent Grégoire3

1Therapanacea, Artificial Intelligence, Paris, France; 2Gustave Roussy, Paris-Saclay University, Inserm 1030, Molecular Radiotherapy and Therapeutic Innovation, Villejuif, France; 3Centre Léon Bérard, Radiation Oncology, Lyon, France; 4Ecole des Ponts, Université Gustave Eiffel, CNRS, Laboratoire d'Informatique Gaspard-Monge, Marne-la-Vallée, France

Show Affiliations
Purpose or Objective

Co-registration between in vivo radiological imaging and ex vivo histopathologic Whole Slide Image (WSI) enables pixel-wise mapping of ROIs. It brings biological insights for radiation oncologists towards new guidelines and homogenization of clinical practice across centers. In addition, AI-based segmentation methods often fail to learn ROIs on radiology due to interobserver contour variability and can benefit from these cross-modality ground truth labels. However, in addition to the differences in resolution scales, color intensities, and nature of data (2D vs. 3D), the task is often manual and challenging because the tissue undergoes severe deformation and shrinkage during the histological process. We propose a cutting-edge deep-learning framework for the automatic registration of 2D histopathology with 3D radiology. The pipeline is generalizable to any radiological modality and allows the reconstruction of synthetic 3D histology.

Material and Methods

We collected a cohort from 77 patients (joint collaboration Gustave Roussy Institute - Centre Léon Bérard, France) on whom were acquired both pre-operative H&N 3D CT scans and 4 to 11 digitalized 2D WSIs after total laryngectomy. Our novelty is two-fold: first, to solve the multimodal issue, we developed a transfer model based on cycleGAN to bring both images to the same modality. Second, for the 2D-3D problem, we built a slice-to-volume unsupervised registration pipeline. Both blocks are trained in parallel for mutual benefit (Figure 1). Moreover, correct mapping is difficult to achieve without proper initialization because of the tissue artifacts mentioned above. To overcome it, we guided the registration by rigidly aligning both thyroid and cricoid cartilages which are supposedly not distorted during tissue preparation procedures.


Figure 2 highlights some visual samples. The deformed WSI (c) is closely aligned to the fixed CT scan (b), for rigid regions like cartilage as well as for soft tissue at the edge of slide inclusion. Quantitatively for all patients, we report a mean Dice Score of 0.91/1 between cartilage masks and a mean Normalized Cross Correlation of 0.89/1 across all tissue. For the modality transfer generation, we can reconstruct synthetic images with a Structure Similarity index of 0.82/1, enabling a 3D reconstruction of the histology specimen by filling empty slices with generated histology from CT.


Co-registration between 3D radiology and 2D histology is a challenging problem, especially for H&N for which the larynx on WSI is often halved. We solve it through an automatic deep learning framework, which is by construction also able of generating synthetic images from one modality to the other one. It paves the way for a better understanding of the tissue microenvironment by overlaying histology-based features on the radiological acquisition, while the reconstruction task enables the generation of noninvasive virtual biopsy. Such tools should yield more precise and personalized RT treatment.