Volumetric cross-modality data registration

[NichD]

Hello! I've been using VALIS for a month now and found it a very useful and robust tool for registering WSI histology stacks. Thank you!!

This task is part of a larger workflow I'm exploring that will culminate in cross-modal registration of a volumetric histology dataset (registered histology stack mentioned above, effectively a volumetric dataset) with a live-tissue fluorescent dataset of the same sample prior to histological processing. This fluorescent dataset may be single plane, or volumetric.

I know from reading the docs that VALIS was designed with the intent of registering fluorescent and histology data. I wondered if this capability has been developed / explored with respect to volumetric data. Is VALIS limited to 2D registration problems, or is registration of volumetric data possible? If yes, can you provide any guidance on how that process is set up? If not, have you encountered in your travels any other image registration tools that might be well suited to this task?

Thanks in advance!!

Cheers,

Nicholas Dana

[cdgatenbee]

Hi @NichD, this sounds like a very cool project! It is true that valis was originally intended for 2D registration, but I recently tested valis to see how it performs with 3d histology reconstruction. The dataset came from Kartasalo et al 2018, and involved aligning 260 serially sliced images. The authors provided landmarks to asses alignment quality, and for the prostate sample the error was 11um, which is an improvement over the reference result presented in the paper (15um). Here's a figure I was putting together for the website that shows the before and after:

So, the registration remains at its core a series of pairwise alignments, but it seems to work well for 3d registration. I did make a few small adjustments to the code to improve the quality of 3d registration, which will be in the next update (1.0.0rc13). However, in the meantime you should be able to get similar results if you set imgs_ordered=True, compose_non_rigid=False when you initialize the Valis object. Note that this does assume that the files are named according to their position in the z-stack, e.g. 001.tif, 002.tiff...100.tiff. I hope to have the update posted soon (along with a Docker image to simplify installation), but maybe you could try this in the meantime?

Best,
-Chandler

[NichD]

Hi @cdgatenbee - Thank you for your thoughts and sharing the related work you've been doing. Looking at your numbers, a ~25% improvement in landmark registration error is pretty impressive, as is that figure! I'm interested in what software you used to render the registered H&E volume in the right panel, it looks great!

As I read your response, several comments and clarifications come to mind. I've got a few questions, but I wanted to further explain the project. After testing VALIS with a dozen datasets, I've been able to generate a volumetric 'stack' of H&E images using similar parameters to what you suggested (see below). This is the first step of this project. The final registration goal is to take the H&E volume and co-register that with fluorescent image data collected on the intact tissue, prior to histological processing. The fluorescent images can be volumetric themselves (stack of Z-planes) , due to the fluorescent system being able to excite and capture in 3D. As such, this becomes a cross-modality volumetric registration problem between two 3D datasets. As you know, this task is further complicated by the distortion that can occur during fixation, making it a pretty challenging task. From reading the paper and docs, I speculate that VALIS may not be able to handle such a task 'out of the box', given the feature extractors look to be targeted to 2D data, but I wanted to confirm that. Is there any capability within VALIS to handle 3D registration, knowing that rotation, translation and warping would need to occur in the Z-dimension?

Additionally, I did have one question about the parameters you suggested. You indicated I should set compose_non_rigid=False to generate a H&E volume. I understand preserving the image order, but I was unclear why we would avoid the non-rigid registration step?

As always, thanks for your work and comments, and kudos on the awesome result you shared above!

Best,
Nich

[cdgatenbee]

Hi @NichD, thanks for the clarification, I think I have a better idea of your project's goal. It does seem that valis might not be suited to the 2nd part of this task, as registering the H&E stack to the fluorescent stack using valis would probably involve aligning corresponding Z-planes between the 2 modalities. It seems this could work if the fluorescent images were collected the same/similar regular intervals as the H&E slices though (e.g. every 4um). If that is the case, let me know and maybe I could put something together. If not, I'm afraid I don't have much experience with 3D registration, and haven't really encountered this challenge before :(

Regarding the parameters I suggested, I should clarify that setting compose_non_rigid=False will still perform non-rigid registration, it just doesn't combine the deformation fields. Combining them can be useful when there are a few slices, but when there are larger numbers of them the images can become quite distorted as one approaches the ends of the image stack. So, setting compose_non_rigid=False should result in a less distorted volumetric image.

To create that figure, I used PyVista to create the 3D image. The code was something like this, where draw_img is a list containing each image (as a numpy array), while slice_cent_r and slice_cent_c are the row and column where the image is "cut" to show the interior:

import pyvista as pv
from skimage import io
from tqdm import tqdm

p = pv.Plotter()
p.store_image = True
p.enable_depth_peeling()
for i, pln in tqdm(enumerate(draw_img)):

    pv_im = pv.UniformGrid(dimensions=(pln.shape[1], pln.shape[0], 1), origin=(0, 0, i))

    slice_img = pln.copy()

    slice_img[:slice_cent_r, slice_cent_c:, 3] = 0
    slice_img[slice_cent_r, slice_cent_c] = [0, 0, 0, 255]

    if i == 0 or i == len(draw_img)-1:
        slice_img[slice_cent_r, slice_cent_c:] = [0, 0, 0, 255]
        slice_img[:slice_cent_r, slice_cent_c] = [0, 0, 0, 255]


    p.add_mesh(pv_im, scalars=slice_img.reshape((-1, slice_img.shape[2])), show_scalar_bar=False, rgb=True, opacity=1, use_transparency=False)


p.show()
io.imsave("3d.png", p.image)

Hopefully some of that is useful, and if you have any more questions, please don't hesitate to ask :)

Best,
-Chandler

[NichD]

@cdgatenbee thank you for that more detailed explanation. To respond to some of your comments, the fluorescent images were captured on the same tissue, but prior to fixation, making it a pretty challenging non-rigid registration task with considerable deformation expected.

Regarding the setting compose_non_rigid=False - I'm not sure I understand the distinction you're making regarding combining the deformation fields. I certainly understand the potential pitfall of accumulating deformation fields introducing extreme distortions as you register the farthest slides (relative to the reference slide at the center of the stack) . However, I assumed that, without composing the deformations in a 'greedy' fashion, the registered stack/volume would look disjoint. Is this not an issue, or such a minor one that it can be ignored? FWIW I haven't observed any severe distortions on the registration results obtained thus far (save on a few that failed due to bugs or non-VALIS issues).

In terms of volumetric cross-modal registration, have you explored any tools (open or closed source) that you've had success with? No worries if you haven't explored that.

Thank you for pointing me toward PyVista! This looks like a great tool and I'm looking forward to exploring it. Much appreciated!!

Incidentally, if you're developing a Docker image and build process for VALIS and you would like an additional tester, I'm happy to discuss this. I think, for the deploy that I'm using, a stable image would be preferred to having multiple environments on the shared computation resource I currently use. I'm happy to explore images built in Windows or Linux. Please let me know if I can be of assistance.

Thank you again for your thoughts and suggestions. Much appreciated!!

[cdgatenbee]

Hi @NichD,
I think whether or not composition is needed depends on the non-rigid method being used. In the case of DeepFlow (the default in valis), it handles large displacements quite well, so the volume doesn't appear disjoint. If the algorithm can only account for smaller displacements, then I think it could be more of an issue. Having said that, if I could find a better way to remove the unwanted distortions I would probably still use composition.

Unfortunately, I don't have any experience with volumetric cross-modal registration, and so haven't explored any relevant tools :( It does sound like a very interesting task though, I just don't have any data to experiment with.

That would be great if you could test the Docker image! I don't have easy access to a Windows machine, so it would be extremely helpful if you are able to test it on a PC. I'd share the Dockerfile now, but I switched to using poetry to install python packages so I could have a lockfile, and so the Dockerfile wouldn't work given the set of files currently on github. I should be pushing that update with the lockfile soon though, and will also put the Docker image on DockerHub.

Best,
-Chandler

[NichD]

Hi @cdgatenbee - That additional detail regarding DeepFlow and displacements was helpful, thank you. I haven't noticed any disjoints in the transformed images using the default settings, but I'll pay attention to it.

I'd be happy to test the Docker image and build in a Windows (and Linux) environment. I had originally attempted to use VALIS on my local Windows laptop, but ran into issues with the Maven - Scipy dependency chain. I think this is a great solution to try. Please let me know when those updates are ready and I'll check them out.

Cheers,
-Nich

[cdgatenbee]

Hi @NichD,
I've just pushed the valis docker image to docker hub if you would like to try it out :) You can find it here: https://hub.docker.com/r/cdgatenbee/valis-wsi

Best.
-Chandler