Hi,
This is a great work and I'd like to try it out on my custom data. However, not able to perform training/inference/visualization of results on custom data. Could anyone please guide me towards using a simple API to train/infer and visualize inferred results?

Thanks in advance,
Amar

New PR fixes numerical stability issues that necessitated developing pytorchse3

Try the latest version of pytorch3d

facebookresearch/pytorch3d#1609

See eigenvivek/DiffDRR#183

• Remove diffpose.calibration.RigidTransform and all related conversion functions, use diffdrr.pose.RigidTransform

Dear Vivek,

Thank you for the help with my previous questions regarding train.py. I have one more question about it.
Q. In a scenario where multi-processing is not possible due to hardware limitations and training needs to be done with only 1 GPU, how can the training be performed?

This is how I proceeded:

parameterizations = [
"se3_log_map",
"so3_log_map",
"axis_angle",
"euler_angles",
"quaternion",
"rotation_6d",
"rotation_10d",
"quaternion_adjugate",
]

Should I pass these parameters to the main() function with a for loop and train them sequentially? The essence of the question is how to overcome the situation when hardware specifications do not allow multi-processing.

And, What is mean each parameterization item.

Thank you in advance for your response.

• Can you compile the DRR module before running pose optimization?
• Does it make a difference?

Dear Vivek,
I'm currently experimenting with your work and have some possibly basic questions. I hope you wouldn't mind enlightening me.

1. If I collect an X-ray image intraoperatively, how do I obtain its corresponding intrinsic and extrinsic parameters? Can this information be obtained from the manufacturer or the device itself?
2. How is the pose corresponding to this projection captured, in the world coordinate system or the camera coordinate system?
3. If the CT scan covers the entire spinal region, but only a portion of the lumbar vertebrae is captured in the intraoperative X-ray, how would one go about constructing DRR models and training in such a scenario?
   My questions might be a bit numerous, I hope you won't find me annoying, and provide valuable guidance.

I am writing code to plot the registration process. My registration work has been completed. I also want to have the effect in GIF, but I am not familiar with pyvista. Can you share how this is achieved? In addition, marker points can be removed during the process.

Dear @eigenvivek,

I found some compaticility issues. DiffPose cannot run if the version of DiffDRR is 0.3.10 or later.

I noticed that you have refactored RigidTransofrm in the rencent version update of DiffDRR, and removed most of the functions in utils. However, some functions such as convert need to be imported in DiffPose.

To solve this problem, I rolled back the version of DiffPose to 0.3.9. Although no errors were found during importing, when running some jupyter files in '/notebooks/', I found another problem: "'RigidTransform' object has no attribute 'dtype'".

I wonder if there are any solutions to these problems? I would be grateful if you could reply soon.

Best wishes,
Yongze

1. Please, what version of Python does the environment require to run for Diffpose?
2. Currently, I've built a Conda environment with Python 3.10, but when I try to directly 'pip install diffpose', it throws an error due to missing PyTorch3D dependency.
3. Is it correct for me to reconfigure the environment by installing 'environment.yml' from Diffpose, like 'pip install -e .'?

Dear vivek:
I hope this message finds you well. I am reaching out to discuss an issue I encountered while working with the register.py file in the diffpose module.

In the register.py file, there's a section where the convert function from diffdrr.pose is called with certain keyword arguments. However, upon inspecting the definition of the convert function in the source file, I noticed that it does not accept one of the provided keyword arguments, namely 'input_parameterization'.

Here's the relevant code snippet from diffpose.registration.py:

rotation, translation = convert(
    pose,
    input_parameterization="se3_exp_map",
    output_parameterization=parameterization,
    output_convention=convention,
)

And here's the definition of the convert function from diffdrr.pose:

def convert(*args, parameterization, convention=None) -> RigidTransform:
    # Function implementation...

As you can see, the convert function does not expect the 'input_parameterization' keyword argument. This inconsistency results in a TypeError during execution.

Here's the error message I encountered:

TypeError: convert() got an unexpected keyword argument 'input_parameterization'

I'm reaching out to bring this to your attention and seek guidance on how to resolve this issue. If you have any insights or suggestions on how to correct this problem, I would greatly appreciate your assistance.

Thank you for your attention to this matter. I look forward to hearing from you soon.

Best regards,
cyxyz

Hello,

when I try to use img and pred_img to draw projected landmarks in register.py, I find that the landmarks and img are not in the same scale. The specific code and the observed result are as follows:

pred_img, mask = registration()
loss = self.criterion(pred_img, img)
losses.append(loss.item())
times.append(0)
est_pose = registration.get_current_pose()
true_fiducials_2d, pred_fiducials_2d = self.specimen.get_2d_fiducials(idx, est_pose)

Create a new figure and subplots

fig, axs = plt.subplots(1, 2, figsize=(10, 5)) # Adjust figsize as needed

Display the original image

axs[0].imshow(img.squeeze().cpu().numpy(), cmap='gray')
axs[0].axis('off') # Hide axes
axs[0].set_title('Original Image')
axs[0].scatter(
pred_fiducials_2d[0, ..., 0].detach().numpy(),
pred_fiducials_2d[0, ..., 1].detach().numpy(),
marker="x",
c="tab:orange",
)

print("pred_img", pred_img)

Display the registered image

axs[1].imshow(pred_img.detach().cpu().numpy()[0,0], cmap='gray')
axs[1].axis('off') # Hide axes
axs[1].set_title('Registered Image')
axs[1].scatter(
true_fiducials_2d[0, ..., 0].detach().numpy(),
true_fiducials_2d[0, ..., 1].detach().numpy(),
label="True Fiducials",
)
axs[1].scatter(
pred_fiducials_2d[0, ..., 0].detach().numpy(),
pred_fiducials_2d[0, ..., 1].detach().numpy(),
marker="x",
c="tab:orange",
label="Predicted Fiducials",
)
for idx in range(true_fiducials_2d.shape[1]):
axs[1].plot(
[true_fiducials_2d[..., idx, 0].item(), true_fiducials_2d[..., idx, 0].item()],
[true_fiducials_2d[..., idx, 1].item(), true_fiducials_2d[..., idx, 1].item()],
"w--",
)

I tried another approach, and here's the code. It resulted in an error.

true_xray , _ = self.specimen[idx]
pred_xray = self.drr(est_pose)
true_fiducials_2d, pred_fiducials_2d = self.specimen.get_2d_fiducials(idx, est_pose)
       
error:
File "/home/data/cyx/autodl-tmp/DiffPose-main/experiments/deepfluoro/register.py", line 224, in run
pred_xray = self.drr(est_pose)
^^^^^^^^^^^^^^^^^^
File "/home/data/cyx/miniconda3/envs/preop/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/data/cyx/miniconda3/envs/preop/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: DRR.forward() missing 2 required positional arguments: 'translation' and 'parameterization'

Can you help me identify the reason?

I have found better initial points in traditional rigid registration before, but I want to migrate to DiffDRR and find that it is not ideal（I used the astra library to generate DRR (https://github.com/astra-toolbox/astra-toolbox), which I set based on the actual projection conditions of the X-ray image）. So if I don't use my previous better initial points, will it affect the subsequent registration results? This question is bugging me right now, if you could answer it I would really appreciate it！

eigenvivek/DiffDRR#179

Hi, thank you for your paper and code, very nice work!

I was wondering if you could illuminate me as to where in your code you access (or use) the ground truth 2D landmarks for the projections when performing the registration test.

I am asking this to test your code with my own annotated CT and generated DRR with manual landmark annotations. I notice I can run the code and have some results without ever specifying where those ground truth 2D landmarks should be collected from, which is odd.

Thank you in advance for this.

AutoExecutor.map_array (main, id_numbers) code is not running.
I tried to output random characters as a printf function inside the main function to see if it is entering the function, but no characters are output and the entire code ends normally.

May I know how to enter the main sentence and learning my data in train.py?
Thank you.

main function enter before

main function inner

whole code result console

I want to get the pose regression model.
So, I'll construct to the dataset.
What dataset did you use to make your pose regression model?
DeepFluoro Dataset?
What are the 1 million composite images you used to create the pose regression model?

I would appreciate it if you could answer the part that can be answered.

Thank you.

https://arxiv.org/abs/2308.10902

I have a question about getting the volume

I understood the operation the swapaxes of the volume, but I don’t understand why [::-1] is needed.

Dear Vivek,

I hope this email finds you well. I am encountering a rather peculiar issue while training the DiffPose model. Even after approximately 800 epochs, I'm still facing NaN (not a number) problems. Here's a snippet of the error message:
Aaaaaaand we've crashed...
tensor([0.9912, 0.9938, 0.9717, 0.9773], device='cuda:0',
grad_fn=)
tensor([ nan, 3.1945, 3.8267, 5.6667], device='cuda:0',
grad_fn=)
tensor([ 0.0000, 8.2518, 9.6319, 10.1889], device='cuda:0',
grad_fn=)
tensor([2.0651, 3.1944, 3.8266, 5.6665], device='cuda:0',
grad_fn=)
tensor([ 2.0651, 8.8485, 10.3642, 11.6586], device='cuda:0',
grad_fn=)
tensor([[[ 4.3618e-02, 9.9902e-01, -6.9611e-03, 0.0000e+00],
[-1.1242e-01, -2.0154e-03, -9.9366e-01, 0.0000e+00],
[-9.9270e-01, 4.4124e-02, 1.1222e-01, 0.0000e+00],
[ 2.8978e+02, 2.8499e+02, 1.9082e+02, 1.0000e+00]],

    [[ 4.5034e-02,  9.8938e-01, -1.3820e-01,  0.0000e+00],
     [-1.1919e-01, -1.3203e-01, -9.8405e-01,  0.0000e+00],
     [-9.9185e-01,  6.0788e-02,  1.1198e-01,  0.0000e+00],
     [ 3.4505e+02,  3.3442e+02,  9.7708e+01,  1.0000e+00]],

    [[ 2.2675e-01,  9.6538e-01, -1.2891e-01,  0.0000e+00],
     [ 8.8859e-02, -1.5231e-01, -9.8433e-01,  0.0000e+00],
     [-9.6989e-01,  2.1174e-01, -1.2032e-01,  0.0000e+00],
     [ 3.1068e+02,  2.0512e+02,  1.4144e+02,  1.0000e+00]],

    [[-1.3417e-01,  9.8313e-01,  1.2428e-01,  0.0000e+00],
     [-2.6461e-02,  1.2182e-01, -9.9220e-01,  0.0000e+00],
     [-9.9061e-01, -1.3641e-01,  9.6706e-03,  0.0000e+00],
     [ 1.6994e+02,  2.5727e+02,  1.1012e+02,  1.0000e+00]]],
   device='cuda:0')

tensor([[[ 4.3480e-02, 9.9903e-01, -6.9389e-03, 0.0000e+00],
[-1.1256e-01, -2.0024e-03, -9.9364e-01, 0.0000e+00],
[-9.9269e-01, 4.3985e-02, 1.1237e-01, 0.0000e+00],
[ 2.8970e+02, 2.8327e+02, 1.9196e+02, 1.0000e+00]],

    [[ 5.0833e-02,  9.8690e-01, -1.5309e-01,  0.0000e+00],
     [-1.1977e-01, -1.4616e-01, -9.8198e-01,  0.0000e+00],
     [-9.9150e-01,  6.8254e-02,  1.1077e-01,  0.0000e+00],
     [ 3.4530e+02,  3.3386e+02,  9.4573e+01,  1.0000e+00]],

    [[ 2.3226e-01,  9.6155e-01, -1.4656e-01,  0.0000e+00],
     [ 8.4274e-02, -1.7000e-01, -9.8183e-01,  0.0000e+00],
     [-9.6900e-01,  2.1569e-01, -1.2052e-01,  0.0000e+00],
     [ 3.1034e+02,  2.0452e+02,  1.3768e+02,  1.0000e+00]],

    [[-1.4363e-01,  9.8366e-01,  1.0859e-01,  0.0000e+00],
     [-1.6376e-02,  1.0735e-01, -9.9409e-01,  0.0000e+00],
     [-9.8950e-01, -1.4456e-01,  6.8980e-04,  0.0000e+00],
     [ 1.6963e+02,  2.5368e+02,  1.0575e+02,  1.0000e+00]]],
   device='cuda:0', grad_fn=<CloneBackward0>)

I have followed the instructions provided in the following link to modify the so3.py file: link.

Additionally, when attempting to utilize the content from this branch after installing the environment.yml file, I encountered an error during the execution of pip install diffpose. Here's a snippet of the error message:

Using cached diffpose-0.0.1-py3-none-any.whl.metadata (7.8 kB)
...
ERROR: Could not find a version that satisfies the requirement pytorch3d (from diffpose) (from versions: none)
ERROR: No matching distribution found for pytorch3d

It seems that the DiffPose package still requires the PyTorch3D code even after the modification in the so3.py file. Could you please assist me in resolving this issue?

Thank you for your attention to this matter.
Best regards,
cyxyz

Dear Vivek,

I hope this email finds you well. I am writing to address an issue I encountered while working with the register.py file in both the main and refactor-se3 branches of our project.

Specifically, I noticed that when I attempt to input the parameterization "euler_angles," an error occurs, as follows:

Traceback (most recent call last):
  File "/home/data/cyx/autodl-tmp/DiffPose-main/experiments/deepfluoro/register.py", line 381, in <module>
    main(id_number, parameterization=parameterization)
  File "/home/data/cyx/autodl-tmp/DiffPose-main/experiments/deepfluoro/register.py", line 348, in main
    df = registration.run(idx)
         ^^^^^^^^^^^^^^^^^^^^^
  File "/home/data/cyx/autodl-tmp/DiffPose-main/experiments/deepfluoro/register.py", line 189, in run
    registration = self.initialize_registration(img)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/data/cyx/autodl-tmp/DiffPose-main/experiments/deepfluoro/register.py", line 124, in initialize_registration
    return SparseRegistration(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/data/cyx/miniconda3/envs/preop/lib/python3.11/site-packages/diffpose/registration.py", line 99, in __init__
    rotation, translation = convert(
                            ^^^^^^^^
  File "/home/data/cyx/miniconda3/envs/preop/lib/python3.11/site-packages/diffpose/calibration.py", line 119, in convert
    transform.get_rotation(output_parameterization, output_convention),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/data/cyx/miniconda3/envs/preop/lib/python3.11/site-packages/diffpose/calibration.py", line 52, in get_rotation
    R = convert_so3(R, "matrix", parameterization, None, convention)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/data/cyx/miniconda3/envs/preop/lib/python3.11/site-packages/diffdrr/utils.py", line 91, in convert
    return _convert_from_rotation_matrix(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/data/cyx/miniconda3/envs/preop/lib/python3.11/site-packages/diffdrr/utils.py", line 126, in _convert_from_rotation_matrix
    rotation = matrix_to_euler_angles(matrix, convention)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/data/cyx/miniconda3/envs/preop/lib/python3.11/site-packages/diffdrr/utils.py", line 419, in matrix_to_euler_angles
    if len(convention) != 3:
       ^^^^^^^^^^^^^^^
TypeError: object of type 'NoneType' has no len()

This issue seems to persist regardless of the branch I am working in(main and refactor-se3). Upon further investigation, I realized that the code snippet provided below is likely the cause of the error:

id_numbers = [1, 2, 3, 4, 5, 6]
parameterizations = [
    "se3_log_map",
    "so3_log_map",
    "axis_angle",
    "euler_angles",
    "quaternion",
    "rotation_6d",
    "rotation_10d",
    "quaternion_adjugate",
]
id_numbers = [i for i, _ in product(id_numbers, parameterizations)]
parameterizations = [p for _, p in product(id_numbers, parameterizations)]

Moreover, I am considering using alternative parameterizations in the train.py file. Would it be necessary to modify the function signature in train.py, particularly the parameterization argument, to accommodate these changes? For instance, should I change parameterization="se3_log_map" to parameterization="euler_angles" ?

def main(
    id_number,
    height=256,
    restart=None,
    model_name="resnet18",
    parameterization="se3_log_map",
    convention=None,
    lr=1e-1,
    batch_size=4,
    n_epochs=400,
    n_batches_per_epoch=100,
):

Could you kindly provide guidance on how to resolve the aforementioned issue? I would greatly appreciate your assistance in this matter.

Thank you for your attention to this matter.

Best regards,