Allow loading of images that aren't in MNI space about nsviewer

[tyarkoni]

The heavy lifting is in NSViewer. It's actually not that heavy, but it will require generalizing some of the hardcoded coordinate transformations (in the models.Transform class) to handle arbitrary dimensions. This is on the agenda, but probably not very near-term, unless someone else wants to take it on.

[njvack]

I wonder if one could punt to nipy to generate a set of ijk -> xyz mappings...

[tyarkoni]

Probably, but I'd like to keep this all client-side, since there's no guarantee all end users will have access to nipy. Unless you just mean we should adapt code to do this from nipy, in which case I support this. Although having said that, now that I went and actually looked at the code, there's probably less to this than I thought. Essentially there are just four transformation methods (viewerToAtlas(), atlasToImage(), etc.) that are currently hardcoded for MNI. All that really needs to happen is that these methods should take an extra dimensions argument (passed from the currently active image) and then the values in the transformation matrix need to be computed dynamically.

[tyarkoni]

Actually, part of what should happen is that the Image class needs to start storing other information in the Nifti header--e.g., the qform and sform matrices. This is easy enough to do when reading from Nifti using xtk, since all the header info is available. To do it from JSON will require updating the Neurosynth code that generates the JSON file. Although I'm thinking of entirely deprecating the option to load from JSON, just because it involves that extra conversion step (even though it's blazingly fast compared to reading in the binary file).

I think it's probably worth waiting to see what functionality is included in the standalone I/O library that @haehn and the xtk folks are developing (https://github.com/xtk/XIO). I'm guessing it won't include conversions from image space to arbitrary spaces, but it might make sense to build this into that package rather than this one.

[mwaskom]

I don't think nsviewer needs to be able to take data from image coordinates
into an arbitrary space, just into the MNI coordinate system. Nipy has
plotting functions that do this that could serve as inpiration, e.g.:
https://github.com/nipy/nipy/blob/master/nipy/labs/viz_tools/activation_maps.py#L43.
The main idea is that the function takes map, a 3D array of values and
affine, a 4x4 matrix that takes you from ijk to some real-world xyz.
Within the Python ecosystem this is pretty straightforward because the
nibabel image object gives you both the data and the corresponding
transformation matrix (which I guess is not always stored as such in the
image header itself).

Copying this division of labor seems right. The image reader should be able
to tell you about the values in the image and how to put them in the MNI
coordinate system, and the viewer app should take those values and that
matrix (or an object that has both attributes) and then draw them properly.

But this might just be obvious noise -- because these issues are so
annoying, I've never thought about them very hard :)

[njvack]

Huh! I wonder why reading the binary data is slower than the JSON data; it should be like an order of magnitude faster.

[tyarkoni]

@mwaskom Well if you assume that the image is going to pass in the right transformation matrix, then pretty much everything is already set up that way in NSViewer. The various viewerToAtlas() methods are just convenience wrappers around a generic transformCoordinates() method that takes coords and transform matrix as input and returns the result. Currently there's no reason to ever transform an entire image, because painting 2D slices to the HTML5 canvas is fast, and memory is what's at a premium. So in practice the only time these transformation methods get called right now is when, e.g., the user clicks on the screen and we need to index into the right slices in the image.

I don't see any reason to force MNI coordinate system on the user; right now there's no place other than those wrapper transformation methods that hardcode the coordinate system at all. It's no more work to implement a completely dynamic viewer that adjusts the world coordinates to what's specified in the image than to convert to MNI. In other words, there's really only one degree of freedom the image has to tell the viewer about (the xyz <--> ijk mapping), and everything else can be done dynamically. If we go this route, I would much prefer to leave the coordinate system unconstrained so that users can display non-standard images if they like (of course, they'll then need to provide their own anatomical underlay in the same space).

@njvack Yeah, I don't know why that is. I suspect it's mostly overhead from xtk that will hopefully disappear in the refactored I/O code. Although one major difference is that once a volume is initialized (which is fast), the JSON reader only needs to index into values where there's a non-zero value (because of the way the information is represented), whereas the binary reader needs to read every chunk. But given the TypedArray implementation I would still expect the binary reader to be much faster.

[tyarkoni]

Thanks for jogging my memory, @mwaskom... though as you can obviously tell, this project is basically dead. What are you using these days for web-based visualization? Papaya.js?

[mwaskom]

I had moved over to Papaya for the project I'd been using nsviewer for originally, but I haven't found much use for web-based viewers lately and mostly stick with older technology (Freeview and FSLView).

[haehn]

people still use slicedrop.com and xtk pretty often :) unfortunately i have 0 time to maintain at the moment