Scores on benchmarks
Model rank shown below is with respect to all public models.| .075 |
average_vision
rank 431
81 benchmarks |
|
| .015 |
neural_vision
rank 466
38 benchmarks |
|
| .059 |
V1
rank 452
24 benchmarks |
|
| .178 |
Coggan2024_fMRI.V1-rdm
v1
rank 30
|
|
|
||
| .135 |
behavior_vision
rank 224
43 benchmarks |
|
| .098 |
Geirhos2021-error_consistency
[reference]
rank 223
17 benchmarks |
|
| .179 |
Geirhos2021colour-error_consistency
v1
[reference]
rank 190
|
|
|
||
| .079 |
Geirhos2021contrast-error_consistency
v1
[reference]
rank 230
|
|
|
||
| .160 |
Geirhos2021cueconflict-error_consistency
v1
[reference]
rank 159
|
|
|
||
| .084 |
Geirhos2021edge-error_consistency
v1
[reference]
rank 132
|
|
|
||
| .077 |
Geirhos2021eidolonI-error_consistency
v1
[reference]
rank 276
|
|
|
||
| .173 |
Geirhos2021eidolonII-error_consistency
v1
[reference]
rank 224
|
|
|
||
| .123 |
Geirhos2021eidolonIII-error_consistency
v1
[reference]
rank 241
|
|
|
||
| .218 |
Geirhos2021falsecolour-error_consistency
v1
[reference]
rank 170
|
|
|
||
| .048 |
Geirhos2021highpass-error_consistency
v1
[reference]
rank 169
|
|
|
||
| .090 |
Geirhos2021lowpass-error_consistency
v1
[reference]
rank 196
|
|
|
||
| .051 |
Geirhos2021phasescrambling-error_consistency
v1
[reference]
rank 219
|
|
|
||
| .013 |
Geirhos2021powerequalisation-error_consistency
v1
[reference]
rank 280
|
|
|
||
| .063 |
Geirhos2021rotation-error_consistency
v1
[reference]
rank 207
|
|
|
||
| .052 |
Geirhos2021silhouette-error_consistency
v1
[reference]
rank 290
|
|
|
||
| .044 |
Geirhos2021sketch-error_consistency
v1
[reference]
rank 223
|
|
|
||
| .159 |
Geirhos2021stylized-error_consistency
v1
[reference]
rank 185
|
|
|
||
| .061 |
Geirhos2021uniformnoise-error_consistency
v1
[reference]
rank 185
|
|
|
||
| .091 |
Baker2022
rank 154
3 benchmarks |
|
| .274 |
Baker2022fragmented-accuracy_delta
v1
[reference]
rank 116
|
|
|
||
| .000 |
Baker2022frankenstein-accuracy_delta
v1
[reference]
rank 146
|
|
|
||
| .000 |
Baker2022inverted-accuracy_delta
v1
[reference]
rank 55
|
|
|
||
| .146 |
BMD2024
rank 127
4 benchmarks |
|
| .166 |
BMD2024.dotted_1Behavioral-accuracy_distance
v1
rank 84
|
|
|
||
| .126 |
BMD2024.dotted_2Behavioral-accuracy_distance
v1
rank 109
|
|
|
||
| .134 |
BMD2024.texture_1Behavioral-accuracy_distance
v1
rank 128
|
|
|
||
| .157 |
BMD2024.texture_2Behavioral-accuracy_distance
v1
rank 118
|
|
|
||
| .233 |
Hebart2023-match
v1
rank 142
|
|
|
||
| .417 |
Maniquet2024
rank 167
2 benchmarks |
|
| .164 |
Maniquet2024-confusion_similarity
v1
[reference]
rank 196
|
|
|
||
| .671 |
Maniquet2024-tasks_consistency
v1
[reference]
rank 70
|
|
|
||
| .095 |
Coggan2024_behavior-ConditionWiseAccuracySimilarity
v1
rank 174
|
|
|
||
| .104 |
engineering_vision
rank 275
25 benchmarks |
|
| .520 |
Geirhos2021-top1
[reference]
rank 156
17 benchmarks |
|
| .948 |
Geirhos2021colour-top1
v1
[reference]
rank 153
|
|
|
||
| .855 |
Geirhos2021contrast-top1
v1
[reference]
rank 95
|
|
|
||
| .158 |
Geirhos2021cueconflict-top1
v1
[reference]
rank 239
|
|
|
||
| .194 |
Geirhos2021edge-top1
v1
[reference]
rank 206
|
|
|
||
| .497 |
Geirhos2021eidolonI-top1
v1
[reference]
rank 140
|
|
|
||
| .511 |
Geirhos2021eidolonII-top1
v1
[reference]
rank 143
|
|
|
||
| .498 |
Geirhos2021eidolonIII-top1
v1
[reference]
rank 161
|
|
|
||
| .929 |
Geirhos2021falsecolour-top1
v1
[reference]
rank 134
|
|
|
||
| .500 |
Geirhos2021highpass-top1
v1
[reference]
rank 66
|
|
|
||
| .419 |
Geirhos2021lowpass-top1
v1
[reference]
rank 134
|
|
|
||
| .539 |
Geirhos2021phasescrambling-top1
v1
[reference]
rank 187
|
|
|
||
| .598 |
Geirhos2021powerequalisation-top1
v1
[reference]
rank 183
|
|
|
||
| .674 |
Geirhos2021rotation-top1
v1
[reference]
rank 128
|
|
|
||
| .319 |
Geirhos2021silhouette-top1
v1
[reference]
rank 236
|
|
|
||
| .487 |
Geirhos2021sketch-top1
v1
[reference]
rank 213
|
|
|
||
| .329 |
Geirhos2021stylized-top1
v1
[reference]
rank 205
|
|
|
||
| .381 |
Geirhos2021uniformnoise-top1
v1
[reference]
rank 153
|
|
|
||
How to use
from brainscore_vision import load_model
model = load_model("resnet50_eMMCR_Vanilla")
model.start_task(...)
model.start_recording(...)
model.look_at(...)
Benchmarks bibtex
@inproceedings{santurkar2019computer,
title={Computer Vision with a Single (Robust) Classifier},
author={Shibani Santurkar and Dimitris Tsipras and Brandon Tran and Andrew Ilyas and Logan Engstrom and Aleksander Madry},
booktitle={ArXiv preprint arXiv:1906.09453},
year={2019}
}
@article{geirhos2021partial,
title={Partial success in closing the gap between human and machine vision},
author={Geirhos, Robert and Narayanappa, Kantharaju and Mitzkus, Benjamin and Thieringer, Tizian and Bethge, Matthias and Wichmann, Felix A and Brendel, Wieland},
journal={Advances in Neural Information Processing Systems},
volume={34},
year={2021},
url={https://openreview.net/forum?id=QkljT4mrfs}
}
@article{BAKER2022104913,
title = {Deep learning models fail to capture the configural nature of human shape perception},
journal = {iScience},
volume = {25},
number = {9},
pages = {104913},
year = {2022},
issn = {2589-0042},
doi = {https://doi.org/10.1016/j.isci.2022.104913},
url = {https://www.sciencedirect.com/science/article/pii/S2589004222011853},
author = {Nicholas Baker and James H. Elder},
keywords = {Biological sciences, Neuroscience, Sensory neuroscience},
abstract = {Summary
A hallmark of human object perception is sensitivity to the holistic configuration of the local shape features of an object. Deep convolutional neural networks (DCNNs) are currently the dominant models for object recognition processing in the visual cortex, but do they capture this configural sensitivity? To answer this question, we employed a dataset of animal silhouettes and created a variant of this dataset that disrupts the configuration of each object while preserving local features. While human performance was impacted by this manipulation, DCNN performance was not, indicating insensitivity to object configuration. Modifications to training and architecture to make networks more brain-like did not lead to configural processing, and none of the networks were able to accurately predict trial-by-trial human object judgements. We speculate that to match human configural sensitivity, networks must be trained to solve a broader range of object tasks beyond category recognition.}
}
@article {Maniquet2024.04.02.587669,
author = {Maniquet, Tim and de Beeck, Hans Op and Costantino, Andrea Ivan},
title = {Recurrent issues with deep neural network models of visual recognition},
elocation-id = {2024.04.02.587669},
year = {2024},
doi = {10.1101/2024.04.02.587669},
publisher = {Cold Spring Harbor Laboratory},
URL = {https://www.biorxiv.org/content/early/2024/04/10/2024.04.02.587669},
eprint = {https://www.biorxiv.org/content/early/2024/04/10/2024.04.02.587669.full.pdf},
journal = {bioRxiv}
}