input images mismatch problems

[davidiomminucsai]

Hi , thank you for sharing your work.

I tried your code for lesions tracking on full body PET images. I provided the two PET scans and the corresponding masks. Below is my command line:

"LesionLocator_track -bl first/pet.nii.gz -fu second/pet.nii.gz -p first/TumorLesions.nii.gz second/TumorLesions.nii.gz -t prev_mask -o out/PSMA_744C_SKIN_prevMask -m ../LesionLocatorCheckpoint"

This is the error message: Predicting Lesions with indices: range(1, 3)
50%|███████████████████████████████████████████████████████████████████████████████████▌ | 1/2 [01:47<01:47, 107.17s/it]
Traceback (most recent call last):
File "/home/david/miniconda3/bin/LesionLocator_track", line 8, in
sys.exit(segment_and_track())
~~~~~~~~~~~~~~~~~^^
File "/mnt/shared-access/network-fs/david/LesionLocator/lesionlocator/inference/lesionlocator_segment_and_track.py", line 287, in segment_and_track
predictor.track(baseline_image=args.bl, follow_up_images=args.fu, output_folder=args.o, prompt=prompt)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/shared-access/network-fs/david/LesionLocator/lesionlocator/inference/lesionlocator_segment_and_track.py", line 195, in track
filename = self.track_single_lesion(lesion_index, bl, fu, bl_path, fu_path, output_folder, prompts, properties, preprocessor)
File "/mnt/shared-access/network-fs/david/LesionLocator/lesionlocator/inference/lesionlocator_segment_and_track.py", line 99, in track_single_lesion
bl, prompt, _ = preprocessor.run_case([bl_path],
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
[prompts[lesion_index - 1]],
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
self.plans_manager,
^^^^^^^^^^^^^^^^^^^
self.configuration_manager,
^^^^^^^^^^^^^^^^^^^^^^^^^^^
self.dataset_json)
^^^^^^^^^^^^^^^^^^
File "/mnt/shared-access/network-fs/david/LesionLocator/lesionlocator/preprocessing/preprocessors/default_preprocessor.py", line 142, in run_case
data, seg, data_properties = self.run_case_npy(data, seg, data_properties, plans_manager, configuration_manager,
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dataset_json)
^^^^^^^^^^^^^
File "/mnt/shared-access/network-fs/david/LesionLocator/lesionlocator/preprocessing/preprocessors/default_preprocessor.py", line 47, in run_case_npy
assert data.shape[1:] == seg.shape[1:], "Shape mismatch between image and segmentation. Please fix your dataset and make use of the --verify_dataset_integrity flag to ensure everything is correct"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Shape mismatch between image and segmentation. Please fix your dataset and make use of the --verify_dataset_integrity flag to ensure everything is correct

First, I do not understand why the code start segmenting again, because the masks are already provided. Secondly, the scan has the same resolution of the mask nifti file (same resolution and size). Should I provide the centroid coordinates to bypass this issue?

Thirdly, do you think that your code will work on PET images, where we have bone lesions and highly irregular lesion shapes? On your paper this case was not fully covered, but I wanted to test it anyway.

Thanks in advance

[mrokuss]

Hey @davidiomminucsai

Thanks for using LesionLocator and reaching out!

So from what it looks to me in your provided command

LesionLocator_track -bl first/pet.nii.gz -fu second/pet.nii.gz -p first/TumorLesions.nii.gz second/TumorLesions.nii.gz -t prev_mask -o out/PSMA_744C_SKIN_prevMask -m ../LesionLocatorCheckpoint

it seems you are providing the lesion masks of the baseline AND follow-up scan to LesionLocator (-p first/TumorLesions.nii.gz second/TumorLesions.nii.gz). The model is actually meant for tracking lesions only from the baseline and should output the segmentation masks of the follow-up scan. Thus, could you try again with:

LesionLocator_track -bl first/pet.nii.gz -fu second/pet.nii.gz -p first/TumorLesions.nii.gz -t prev_mask -o out/PSMA_744C_SKIN_prevMask -m ../LesionLocatorCheckpoint

Regarding your other question, the model was not trained on PET data (only CT), since we did not have such data available, so performance may vary. If your PET is from a PET/CT scan, you might get better results using the CT image as input. Bone lesions and highly irregular shapes are indeed challenging, so the results could be less robust, but it’s definitely interesting to test.

Curious to hear how it works out on your PET data, feel free to let me know what you find!