scilus / ismrm_2015_tractography_challenge_scoring

Scripts used to recreate the results of the ISMRM 2015 Tractography Challenge

ISMRM 2015 Tractography Challenge Scoring system

This system contains the scripts and tools that can be used to recreate the results of the ISMRM 2015 Tractography Challenge and to evaluate new datasets.

The release used to produce results for the website and paper is archived on Zenodo .

Configuration and installation

Make sure the "pip" version is recent enough. You can update it with

pip install -U pip

Then, install the needed dependencies using the requirements file. The team recommends using a virtual environment (with the virtual env wrapper), but it is not mandatory. Once setup, run

pip install -r requirements.txt

You can then clone or download the scoring system. Once cloned or downloaded, go inside the ismrm_2015_tractography_challenge_scoring directory, and run

python setup.py build_ext --inplace
python setup.py install

Once those steps are all done, the system is configured.

Fetching the Ground Truth Dataset

To be able to run the scoring system, a directory containing the ground truth dataset is needed. It can be downloaded from the Tractometer website.

Scoring a tractogram

Once the ground truth dataset is unarchived (for example, to the scoring_data directory, one needs to create a directory where all results will be saved. Let's call it results for now.

Then, an example call to the scoring system will be

./scripts/score_tractogram.py YOUR_TRACTOGRAM_FILE scoring_data/ results/

where YOUR_TRACTOGRAM_FILE is replaced with the path of the tractogram file that will be scored.

Additional flags use to control the saving behavior of the script are available. Call score_tractogram.py -h to get the list of such flags.

ismrm_2015_tractography_challenge_scoring's Issues

Orientation assignment

ismrm_2015_tractography_challenge_scoring/scripts/score_tractogram.py

Line 154 in db3c423

parser.error('--orientation is needed for your tractogram format')

@jchoude Hello, should there be an additional line:

tract_attribute['orientation'] = args.orientation

Immediately after line 154

Testing the scoring system using the ground-truth TRK data

Hi,

I am trying to understand how to use the scoring system, and to keep things simple I am trying to run the system to evaluate the ground-truth .trk files.

The steps I follow are:

1. export the path, i.e.:
   export PYTHONPATH=~/Desktop/ismrm_2015_tractography_challenge_scoring
2. Run the scoring script on the CST_Left.trk ground-truth file downloaded from http://www.tractometer.org/ismrm_2015_challenge/data
   ./scripts/score_tractogram.py ~/Downloads/ISMRM_2015_Tracto_challenge_ground_truth_bundles_TRK_v2/CST_left.trk scoring_data/ results/

However, I get the following error:

WARNING:root:The number of streamlines specified in header ([0]) does not match the actual number of streamlines contained in this file (7217). The latter will be used.
WARNING:root:The number of streamlines specified in header ([0]) does not match the actual number of streamlines contained in this file (7217). The latter will be used.
WARNING:root:The number of streamlines specified in header ([0]) does not match the actual number of streamlines contained in this file (7217). The latter will be used.
Need "vox_to_ras" field to get affine with which to convert points; this is present for headers >= version 2
Traceback (most recent call last):
  File "./scripts/score_tractogram.py", line 175, in <module>
    main()
  File "./scripts/score_tractogram.py", line 168, in main
    segments_dir, base_name, args.verbose)
  File "/home/aritche/Desktop/ismrm_2015_tractography_challenge_scoring/challenge_scoring/metrics/scoring.py", line 154, in score_submission
    full_strl = [s for s in streamlines_gen]
  File "/home/aritche/Desktop/ismrm_2015_tractography_challenge_scoring/challenge_scoring/io/streamlines.py", line 94, in _get_tracts_over_grid
    str(er))
ValueError: 
------ ERROR ------

TrackVis header is malformed or incomplete.
Please make sure all fields are correctly set.

If I instead use the .tck files downloaded from the same website, I don't have any issues and the expected results are generated (f1 for everything is 0, except for CST_left which is 0.8294, valid bundles = 1, etc.).

What do I change in the .trk header files to fix this problem?