Sample stimuli

How to use

from brainscore_language import load_benchmark
benchmark = load_benchmark("Futrell2018-pearsonr")
score = benchmark(my_model)

Benchmark API

Code examples

Model scores

Score Legend Min Alignment Max Alignment

Rank	Model	Score
1	openai-gpt	.387
2	gpt2	.367
3	distilgpt2	.361
4	gpt-neo-125m	.358
5	gpt2-medium	.350
6	gpt2-large	.331
7	gpt-neo-1.3B	.324
8	gpt2-xl	.318
9	gpt-neo-2.7B	.308
10	lm1b	.286
11	glove-840b
12	randomembedding-100
13	randomembedding-1600
14	earley-parser-minivocab	nan

Benchmark bibtex

@proceedings{futrell2018natural,
  title={The Natural Stories Corpus},
  author={Futrell, Richard and Gibson, Edward and Tily, Harry J. and Blank, Idan and Vishnevetsky, Anastasia and
          Piantadosi, Steven T. and Fedorenko, Evelina},
  conference={International Conference on Language Resources and Evaluation (LREC)},
  url={http://www.lrec-conf.org/proceedings/lrec2018/pdf/337.pdf},
  year={2018}
}

Ceiling

0.86.

Note that scores are relative to this ceiling.

Data: Futrell2018

Metric: pearsonr