brain-score / brainio_collection

collection of stimuli and data

Brain-Score is a platform to evaluate computational models of brain function on their match to brain measurements in primate vision. The intent of Brain-Score is to adopt many (ideally all) the experimental benchmarks in the field for the purpose of model testing, falsification, and comparison. To that end, Brain-Score operationalizes experimental data into quantitative benchmarks that any model candidate following the BrainModel interface can be scored on.

Note that you can only access a limited set of public benchmarks when running locally. To score a model on all benchmarks, submit it via the brain-score.org website.

See the documentation for more details, e.g. for submitting a model or benchmark to Brain-Score. For a step-by-step walkthrough on submitting models to the Brain-Score website, see these web tutorials.

See these code examples on scoring models, retrieving data, using and defining benchmarks and metrics. These previous examples might be helpful, but their usage has been deprecated after the 2.0 update.

Brain-Score is made by and for the community. To contribute, please send in a pull request.

Local installation

You will need Python = 3.7 and pip >= 18.1.

pip install git+https://github.com/brain-score/vision

Test if the installation is successful by scoring a model on a public benchmark:

from brainscore_vision.benchmarks import public_benchmark_pool

benchmark = public_benchmark_pool['dicarlo.MajajHong2015public.IT-pls']
model = my_model()
score = benchmark(model)

# >  <xarray.Score ()>
# >  array(0.07637264)
# >  Attributes:
# >  Attributes:
# >      error:                 <xarray.Score ()>\narray(0.00548197)
# >      raw:                   <xarray.Score ()>\narray(0.22545106)\nAttributes:\...
# >      ceiling:               <xarray.DataArray ()>\narray(0.81579938)\nAttribut...
# >      model_identifier:      my-model
# >      benchmark_identifier:  dicarlo.MajajHong2015public.IT-pls

Some steps may take minutes because data has to be downloaded during first-time use.

Environment Variables

Variable	Description
RESULTCACHING_HOME	directory to cache results (benchmark ceilings) in, ~/.result_caching by default (see https://github.com/brain-score/result_caching)

License

MIT license

Troubleshooting

`ValueError: did not find HDF5 headers` during netcdf4 installation

pip seems to fail properly setting up the HDF5_DIR required by netcdf4. Use conda: `conda install netcdf4`

repeated runs of a benchmark / model do not change the outcome even though code was changed

results (scores, activations) are cached on disk using https://github.com/mschrimpf/result_caching. Delete the corresponding file or directory to clear the cache.

CI environment

Add CI related build commands to test_setup.sh. The script is executed in CI environment for unittests.

References

If you use Brain-Score in your work, please cite "Brain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like?" (technical) and "Integrative Benchmarking to Advance Neurally Mechanistic Models of Human Intelligence" (perspective) as well as the respective benchmark sources.

@article{SchrimpfKubilius2018BrainScore,
  title={Brain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like?},
  author={Martin Schrimpf and Jonas Kubilius and Ha Hong and Najib J. Majaj and Rishi Rajalingham and Elias B. Issa and Kohitij Kar and Pouya Bashivan and Jonathan Prescott-Roy and Franziska Geiger and Kailyn Schmidt and Daniel L. K. Yamins and James J. DiCarlo},
  journal={bioRxiv preprint},
  year={2018},
  url={https://www.biorxiv.org/content/10.1101/407007v2}
}

@article{Schrimpf2020integrative,
  title={Integrative Benchmarking to Advance Neurally Mechanistic Models of Human Intelligence},
  author={Schrimpf, Martin and Kubilius, Jonas and Lee, Michael J and Murty, N Apurva Ratan and Ajemian, Robert and DiCarlo, James J},
  journal={Neuron},
  year={2020},
  url={https://www.cell.com/neuron/fulltext/S0896-6273(20)30605-X}
}

brainio_collection's Issues

old s3 url in test_lookup

tests/test_assemblies.py:test_lookup:79 fails with

E       AssertionError: assert 'https://mkgu...l_features.nc' == 'https://brain...l_features.nc'
E         - https://mkgu-dicarlolab-hvm.s3.amazonaws.com/hvm_neuronal_features.nc
E         ?         ^^^^        -------
E         + https://brainio-dicarlo.s3.amazonaws.com/hvm_neuronal_features.nc
E         ?         ^^^^^^^

attributemodel (name, type) has unique index on name, but type might be different across stimulus sets

If a new stimulus set for instance has a batch that is now a string instead of an integer, the database won't allow it due to a unique index on attributemodel.name.
Could fix by setting unique index on joint (name, type) instead of just name

403 on tolias.Cadena2017

brainio_collection/tests/test_assemblies.py

Line 36 in d4601e4

assert brainio_collection.get_assembly("tolias.Cadena2017") is not None

fails with a 403. Did the travis account loose access rights for tolias.Cadena2017?

dicarlo.Majaj2015.temporal-10ms missing image_id field

#19 commented out the unit test for dicarlo.Majaj2015.temporal-10ms because the image_id field is not present.
This assembly needs to be fixed (by renaming id --> image_id) and the unit test added again.

get rid of knownfile

blocks being able to make this repository public

what is StimulusSetLoader.cls used for?

brainio_collection/brainio_collection/fetch.py

Line 139 in 7805a30

self.cls = cls

self.cls is not used for anything, is this dead code?

write_netcdf: reset_index on DataAssembly

netcdf cannot serialize indexes to disk.
We thus reset the index before writing it out (https://github.com/brain-score/brainio_collection/blob/master/brainio_collection/packaging.py#L171).
However, if you use the DataAssembly (or a sub-assembly-class of it), it will always re-index, hence leading to the serialization issue.

The temporary solution is to re-cast the assembly to a DataArray to avoid re-indexing (assembly = xarray.DataArray(assembly)).

We should also allow users to package with DataAssembly without running into this error.
The simplest solution is probably to re-cast to DataArray right before resetting the index in write_netcdf.

no separation of conceptual image from image file

even though it makes sense to only have a unique image file for unique images (i.e. if the exact same image occurs in two StimulusSets, only store one image), this does not apply to the conceptual image.
For instance, if StimulusSet A and B have the same image, they might put different metadata on it. E.g., A might set label='beach' and label='umbrella'.

@jjpr-mit suggests this might be fixed by a foreign key from attributemodel to stimulussetmodel and selecting for that key when constructing StimulusSet DataFrames

list_stimulus_sets() lists each StimulusSet twice

S3 access

currently, travis cannot access private assemblies.
need a new user @cshay.
then, we can follow https://stackoverflow.com/questions/37267916/how-to-run-aws-configure-in-a-travis-deploy-script and https://docs.travis-ci.com/user/encryption-keys/

create unit tests for dicarlo.Rajalingham2018

just testing for existence of the two stimulus sets and two assemblies is fine imo.

upload to S3 only on merge?

currently, we give the user access to S3 so that they can upload assembly/stimulus_set directly themselves.
This potentially generates files that are later unused and unnecessarily exposes data storage.

Would it be worth to have a solution where the user side only mocks the upload and then the real upload happens upon merge?
The downside of this is that the user needs to put the raw files used for packaging somewhere we can access them so we might be back to the same problem..

(copied from brain-score/brainio_contrib#37)