Parameter Efficient Fine Tuning for
Multi-scanner PET to PET Reconstruction

Accept @ MICCAI 2024
Yumin Kim*, Gayoon Choi*, Seong Jae Hwang
Yonsei University
Paper arXiv Code
Visualizing EAGLE and Baselines

We introduce PETITE,
Parameter Efficient Fine Tuning for MultI-scanner PET to PET REconstruction.

Abstract

Reducing scan time in Positron Emission Tomography (PET) imaging while maintaining high-quality images is crucial for minimizing patient discomfort and radiation exposure. Due to the limited size of datasets and distribution discrepancy across scanners in medical imaging, fine-tuning in a parameter-efficient and effective manner is on the rise. Motivated by the potential of Parameter-Efficient Fine-Tuning (PEFT), we aim to address these issues by effectively leveraging PEFT to improve limited data and GPU resource issues in multi-scanner setups. In this paper, we introduce PETITE, Parameter-Efficient Fine-Tuning for MultI-scanner PET to PET REconstruction which represents the optimal PEFT combination when independently applying encoder-decoder components to each model architecture. To the best of our knowledge, this study is the first to systematically explore the efficacy of diverse PEFT techniques in medical imaging reconstruction tasks via prevalent encoder-decoder models. This investigation, in particular, brings intriguing insights into PETITE as we show further improvements by treating encoder and decoder separately and mixing different PEFT methods, namely, Mix-PEFT. Using multi-scanner PET datasets comprised of five different scanners, we extensively test the cross-scanner PET scan time reduction performances (i.e., a model pre-trained on one scanner is fine-tuned on a different scanner) of 21 feasible Mix-PEFT combinations to derive optimal PETITE. We show that training with less than 1% parameters using PETITE performs on par with full fine-tuning (i.e., 100% parameter).

Method


Pipeline

Main figure

The overview of PETITE: Scheme for single source-target settings in PET scan time reduction with PEFT.
The optimal PEFT combination independently applying encoder-decoder components to each model architecture.
To the best of our knowledge, this extensive study represents the first application of the PEFT methodology within the field of medical imaging.
Contribution

  • We leverage the PEFT methodology in a medical reconstruction task to reduce the scan time of PET images on scanners with different dimensions, voxel spacing, and institutions. To the best of our knowledge, this extensive study represents the first application of the PEFT methodology within the field of medical imaging.
  • Upon experimenting with possible Mix-PEFT, we found that using less than 1% of parameters can achieve performance comparable to Full-FT, carefully considering encoder and decoder architecture.
  • We provide novel insights into the optimal PEFT settings tailored for the reconstruction model.


Encoder-Decoder structure of each ViT-based model

Main figure

The pipeline of the encoder-decoder structure of each ViT-based model. (a) 3D CVT-GAN features a generator with a ViT-based encoder and decoder. Only the first three layers of the encoder and the first two layers of the decoder are trained. (b) UNETR consists of a ViT-based encoder and a CNN-based decoder.


Modified structures of PEFT methods

Main figure

Illustrations of the modified structures of PEFT Additive methods.

Visualization of Error Maps


3D CVT-GAN


UNETR


Scan time reduction examples using 3D CVT-GAN and UNETR.
First row: PET scans. Second row: error maps comparing the reconstructed PET scans to the ground-truth (GT).

Quantitative Main Results


cocostuff table

Quantitative comparison with 3D CVT-GAN [25] and UNETR [8]. The table compares the performance of various PEFT methods and PETITE (Ours) using PSNR, SSIM, and NRMSE evaluation metrics.
Best : Bold; Second best : Underline.

Ablation Results


cocostuff table

Optimal Mix-PEFT (Ours) outperforms each PEFT method with or without BitFit.
Best : Bold; Second best : Underline.

Details of Experiments


Scanner Specifications

cocostuff table


Computation costs of PEFT methods

cocostuff table

21 Feasible Experimental Mix-PEFT with BitFit


3D CVT-GAN

cocostuff table

UNETR

cocostuff table

BibTeX

@article{petite2024,
      title={Parameter Efficient Fine Tuning for Multi-scanner PET to PET Reconstruction},
      author={Kim, Yumin and Choi, Gayoon and Hwang, Seong Jae},
      journal={arXiv preprint arXiv:2407.07517},
      year={2024}
    }