Frequently Asked Questions

What is the purpose of Brain-Score?

The Brain-Score platform aims to yield strong computational models of the brain and mind. We enable researchers to quickly get a sense of the alignment of their model(s) to currently dozens of neural and behavioral measurements, and provide these models to experimentalists to prototype new experiments and make sense of biological data.

What is a benchmark?

A benchmark is composed of data, a metric, and an experimental protocol to test models. The data is typically primate (human or non-human) neural and/or behavioral data. When a model is run on a benchmark, the metric outputs a score of how well the model predictions match the data.

What is a metric?

A metric scores how similar two sets of data are. Typically these two sets are model and primate measurements, but metrics are agnostic of the data source and can also be used to compare two primate measurements (e.g. for ceiling estimates).

What is a model?

Specifically, "model" here refers to any computational model that implements the BrainModel (for Brain-Score Vision) or ArtificialSubject (for Brain-Score Language) API, which defines the interface that allows a conformant model to run on Brain-Score benchmarks. Note that Brain-Score is agnostic to the exact model family, e.g. this could be an artificial neural network or a hand-crafted predictor. As long as the model makes predictions that adhere to the API, it can be tested.

What is a score?

The scores that Brain-Score computes are a normalized value ranging from 0.0 to 1.0 . The score reflects how closely model predictions align with biological data, e.g. neural activity or behavior, as evaluated by each benchmark's metric. 0 means the model does not match the measurements at all and 1 means the model matches the measurements at ceiling level such that no more data can be explained due to noise in the signal (e.g. if the model obtains a score of 0.8 and the data ceiling is also 0.8 , the score output by this method should be 1 ).

What am I able to submit?

You can submit new plugins to Brain-Score. A plugin can be a dataset, a metric, a benchmark, or a model. You can submit plugins directly on the website as a zip file or via a pull request on Github (zip file submissions will automatically be converted into a PR). The plugin directory has to contain at least two files: an __init__.py which adds your data/metric/benchmark/model to the corresponding registry, and a test.py file in which you can define tests to ensure your plugin works as expected. If your plugin requires third-party libraries, you can include a setup.py or arequirements.txt file. You can optionally add more files, e.g. to separate the detailed implementations.

How do I submit a model?

We have created two tutorials outlining how to submit a new Brain-Score plugin- a (Quickstart tutorial) and three (Deep Dive tutorials). The Quickstart tutorial outlines how to set up your local environment quickly and how to run the pipeline locally. The Deep Dive tutorials contain direction on how to submit plugins and even provide a template plugin file.

What happens to a model after I submit it?

A pull request is opened on the Brain-Score GitHub of whichever plugin type you submitted, either (vision or language). If this pull request passes all of the Jenkins tests, the PR will be merged and the scoring of your plugin will begin. Once scored, the Brain-Score of your plugin will appear on the leaderboard as long as you didn't ask for your model's score to be private when submitting it.

Am I able to keep my submissions private?

While we do not currently support private code submissions -- at the moment, all submitted code is merged into our public GitHub repository -- this functionality is under development and anticipated to be available soon. An automated option for including private files with your submission is also planned; in the interim, please contact us with any files that need to remain private and we will upload them directly to our AWS S3. This is a good way to enable model testing on your data without releasing it if you wish to keep it private: Your data will not be publicly accessible but by submitting models, the community can still get a sense of model alignment.

I submitted a model but don't see it in my profile. Why not?

There are a variety of reasons that your model may not be showing up in your profile. The submission and scoring process can vary depending on server availability, so if you haven't received an email indicating an error, it's possible that tests or scoring are still running. As every successful submission should result in a pull request in the relevant GitHub repository (vision or language), you can check to see if there is either an open or closed PR with your model name in the title, which can provide further information about the status of your submission. If you need assistance, you can either create a GitHub issue or reach out via Slack explaining your problem and someone in our developer community will assist you.