CCN Schedule

Tuesday, August 6, 2024

2:00pm Competition Findings (Martin Schrimpf)

2:15pm Behavioral Benchmark Winners

1. Do Neural Networks Care How Parts are Arranged? (Nick Baker, James Elder)

2. Brain-Score on Dotted Line Drawings and Textured Silhouettes (Marin Dujmović, Gaurav Malhotra, Jeff Bowers)

2:30pm Neural Benchmark Winners

1. Strong amodal completion of occluded objects across early and high-level visual cortices (David Coggan)

2. The Animate Space of Inanimate Perception (Andrea Ivan Costantino)

3:00pm Pros and Cons of Model Benchmarking (Kohitij Kar)

3:10pm Panel Discussion (Jenelle Feather, Dan Yamins, Kohitij Kar, Ev Fedorenko, Jeff Bowers, Eero Simoncelli; moderated by Jim DiCarlo)

Common critiques of the Brain-Score platform

These and others are common critiques in our community of the Brain-Score platform. We hear you!

We take pride in developing and making openly available an integrative benchmarking platform that allows the community to operationalize and actively engage on this and other feedback. In 2022, we organized the first Brain-Score competition for model submissions in which we evaluated the models on our existing benchmarks. In 2024 — we have turned the table! This year, in the spirit of an adversarial collaboration — we invite experimentalists and the community at large to turn their legitimate concerns into concrete benchmarks that will challenge and hopefully expose the explanatory gaps between our current models of primate vision and the biological brain.

Our ability to recognize objects in the world relies on visual processing along the ventral visual stream, a set of hierarchically-organized cortical areas. Recently, machine-learning researchers and neuroscientists have produced computational models that have achieved moderate success at explaining primate object recognition behavior and the neural representations that support it. This second edition of the Brain-Score Competition aims to find benchmarks on which the predictions of our current top models break down.

In 2022, the first Brain-Score Competition led to new and improved models of primate vision that predicted existing benchmarks reasonably well. In this year's Brain-Score Competition 2024, we will close the loop on testing model predictions by rewarding those benchmarks that show where models are the least aligned to the primate visual ventral stream. The Competition is open to the scientific community and we provide an infrastructure to evaluate a variety of models on new behavioral and neural benchmarks in a standardised and unified manner. In addition, we will incentivize benchmark submissions by providing visibility to participants and a $10,000 prize pool to the winning benchmarks.

Submissions are open until June 30 July 8, 2024. For regular updates related to the competition, please follow Brain-Score on Twitter and join our community. Good Luck!

Overview

Participants should submit their benchmarks through the Brain-Score platform. Brain-Score currently accepts any benchmark engaging on testing model predictions of core object recognition behavior and neural responses (spike rates) along the ventral visual stream areas. We encourage all submissions of behavioral and neural benchmarks. To facilitate benchmark submission, we provide helper code and tutorials. You can also use existing datasets or metrics and re-combine them in novel ways. Note that by using the Brain-Score platform, no model knowledge is required -- you can treat them like another primate subject.

Benchmarks have to use models in their current "operating regime". This means that there are three behavioral tasks (imagenet-labeling, probabilities, odd-one-out), and four brain regions to record from (V1, V2, V4, IT). Until April, we will accept suggestions of other behavioral tasks provided you work with us to make sure the models can engage on them. The stimuli should be static images (e.g. no videos this round). In this round, benchmarks should not test temporal dynamics (but you can still specify a single "time_bin" from the models to record from.)

FAQ