medXGAN - Visual Explanations for Medical Classifiers Through a Generative Latent Space

Amil Dravid, Florian Schiffers, Boqing Gong, Aggelos K. Katsaggelos

CVPR Workshops 2022

my alt text
Figure 1. Overview. We propose a GAN framework medX-GAN that takes in two latent vectors (z1, z2) to encode anatomical structure and classifier-specific pathology, respectively. After training with a fixed classifier providing feedback, the generator can be used to explain the classifier’s decision. Given a ground truth positive image, the latent code can be found via an optimization scheme. The positive image then can be turned into a negative realization by relying on the medXGAN’s latent sampling scheme. The classifier-specific features can be visualized via a pixel-wise difference between the negative and positive images or by integrating the gradients (IG) by traversing the latent space (LIG).

Abstract: Despite the surge of deep learning in the past decade, some users are skeptical to deploy these models in practice due to their black-box nature. Specifically, in the medical space where there are severe potential repercussions, we need to develop methods to gain confidence in the models’ decisions. To this end, we propose a novel medical imaging generative adversarial framework, medXGAN (medical eXplanation GAN), to visually explain what a medical classifier focuses on in its binary predictions. By encoding domain knowledge of medical images, we are able to disentangle anatomical structure and pathology, leading to fine-grained visualization through latent interpolation. Furthermore, we optimize the latent space such that interpolation explains how the features contribute to the classifier’s output. Our method outperforms baselines such as Gradient-Weighted Class Activation Mapping (Grad-CAM) and Integrated Gradients in localization and explanatory ability. Additionally, a combination of the medXGAN with Integrated Gradients can yield explanations more robust to noise. The project page with code is available at: https://avdravid.github.io/medXGAN page.

Paper Display
Paper Thumbnail
medXGAN - Visual Explanations for Medical Classifiers Through a Generative Latent Space
Amil Dravid, Florian Schiffers, Boqing Gong, Aggelos K. Katsaggelos
CVPR Workshops 2024
Paper
my alt text
Fig. 2: The two figures visualize the composition of the loss term used for the training process of the cycleGAN architecture. IF and IC are the input images for the color fundus image generator GC and the angiographic image generator GF, respectively. Similarly, DC and DF denote the respective discriminator networks. Cycle consistency is enforced so that the backwards translation resembles the input image for both ways, see LCycle_Angio and LCycle_Color. The adversarial loss, i.e. the capacity of the network to distinguish between real and fake images, is modeled by LDC and LDF.
my alt text
Fig. 3: Each row shows from left to right the real and generated angiographic image, the authentic color image, and the reconstructed color image to show cycle consistency. The first three rows are from dataset [10], the remaining is taken from our own data.

Introduction:

We delve into synthesizing angiographic images from conventional color fundus images using deep neural networks, aiming to provide safer diagnostic alternatives due to the potential risks associated with angiographic imaging.

Method:

CycleGAN Approach:

Utilizing CycleGAN, we translate between conventional and angiographic fundus images. This model comprises generators and discriminators, trained to create images that are nearly indistinguishable from real ones. The images were preprocessed and augmented to enhance the dataset size, and the network underwent training with a learning rate that decreased over epochs.

Results:

The synthesized images closely resembled the real angiographic images, enhancing certain structures like vessels. However, there were variations in brightness and contrast, and some small details were not accurately synthesized.

Discussion & Future Work:

This study demonstrates the potential of using synthetic angiographic images for developing robust algorithms, but the practical utility for medical practitioners needs further exploration. Future research will focus on refining image resolution and exploring advanced data-augmentation methods.

This browser does not support PDFs. Please download the PDF to view it: Download PDF.