Leaderboard Explore Compare Profile Community Blog FAQ Competition 2024 Report Issue

Scores on benchmarks

from scoring API

Model rank shown below is with respect to all public models.

.319	average_language rank 18 5 benchmarks	.319 0 ceiling best median

.280	neural_language rank 24 4 benchmarks	.280 0 ceiling best median

.523	Pereira2018-ridge rank 23 2 benchmarks	.523 0 ceiling best median

.504	Pereira2018.243sentences-ridge v1 rank 24	.504 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.542	Pereira2018.384sentences-ridge v1 rank 20	.542 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.000	Blank2014-ridge v1 rank 21	.000 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.316	Fedorenko2016-ridge v3 rank 26	.316 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.358	behavior_language rank 7 1 benchmark	.358 0 ceiling best median

.358	Futrell2018-pearsonr v1 [reference] rank 7	.358 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.736	engineering_language rank 12 30 benchmarks	.736 0 ceiling best median

.736	SyntaxGym [reference] rank 12 30 benchmarks	.736 0 ceiling best median

.964	syntaxgym-center_embed v1 [reference] rank 7 1 benchmark	.964 0 ceiling best median

.857	syntaxgym-center_embed_mod v1 [reference] rank 18	.857 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

1.0	syntaxgym-cleft v1 [reference] rank 1 1 benchmark	1.0 0 ceiling best median

.925	syntaxgym-cleft_modifier v1 [reference] rank 12	.925 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.000	syntaxgym-fgd_hierarchy v1 [reference] rank 1	.000 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

1.0	syntaxgym-fgd_object v1 [reference] rank 1	1.0 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.917	syntaxgym-fgd_pp v1 [reference] rank 3	.917 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.625	syntaxgym-fgd_subject v1 [reference] rank 1	.625 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.786	syntaxgym-mvrr v1 [reference] rank 5 1 benchmark	.786 0 ceiling best median

.786	syntaxgym-mvrr_mod v1 [reference] rank 7	.786 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.947	syntaxgym-npi_orc_any v1 [reference] rank 17	.947 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

1.0	syntaxgym-npi_orc_ever v1 [reference] rank 1	1.0 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.737	syntaxgym-npi_src_any v1 [reference] rank 17	.737 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.868	syntaxgym-npi_src_ever v1 [reference] rank 17	.868 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.833	syntaxgym-npz_ambig v1 [reference] rank 9 1 benchmark	.833 0 ceiling best median

.917	syntaxgym-npz_ambig_mod v1 [reference] rank 8	.917 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

1.0	syntaxgym-npz_obj_mod v1 [reference] rank 1	1.0 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.421	syntaxgym-number_orc v1 [reference] rank 17	.421 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.579	syntaxgym-number_prep v1 [reference] rank 17	.579 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.737	syntaxgym-number_src v1 [reference] rank 11	.737 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.105	syntaxgym-reflexive_orc_fem v1 [reference] rank 16	.105 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.526	syntaxgym-reflexive_orc_masc v1 [reference] rank 15	.526 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.421	syntaxgym-reflexive_prep_fem v1 [reference] rank 11	.421 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.579	syntaxgym-reflexive_prep_masc v1 [reference] rank 15	.579 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.263	syntaxgym-reflexive_src_fem v1 [reference] rank 11	.263 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.632	syntaxgym-reflexive_src_masc v1 [reference] rank 9	.632 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.957	syntaxgym-subordination v1 [reference] rank 7 3 benchmarks	.957 0 ceiling best median

.826	syntaxgym-subordination_orc-orc v1 [reference] rank 19	.826 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.957	syntaxgym-subordination_pp-pp v1 [reference] rank 13	.957 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.913	syntaxgym-subordination_src-src v1 [reference] rank 15	.913 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

How to use

from brainscore_language import load_model
model = load_model("gpt-neo-125m")
model.start_task(...)
model.start_recording(...)
model.look_at(...)

Model API

Code examples

Metadata has not been added yet.

Brain Encoding Response Generator (BERG)

Through the BERG you can easily generate neural responses to text sentences of your choice using any Brain-Score language model.

For more information on how to use BERG, see the documentation and tutorial.

Benchmarks BibTeX

@proceedings{futrell2018natural,
  title={The Natural Stories Corpus},
  author={Futrell, Richard and Gibson, Edward and Tily, Harry J. and Blank, Idan and Vishnevetsky, Anastasia and
          Piantadosi, Steven T. and Fedorenko, Evelina},
  conference={International Conference on Language Resources and Evaluation (LREC)},
  url={http://www.lrec-conf.org/proceedings/lrec2018/pdf/337.pdf},
  year={2018}
}
        @inproceedings{gauthier-etal-2020-syntaxgym,
    title = "{S}yntax{G}ym: An Online Platform for Targeted Evaluation of Language Models",
    author = "Gauthier, Jon and Hu, Jennifer and Wilcox, Ethan and Qian, Peng and Levy, Roger",
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations",
    month = jul,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.acl-demos.10",
    pages = "70--76",
    abstract = "Targeted syntactic evaluations have yielded insights into the generalizations learned by neural network language models. However, this line of research requires an uncommon confluence of skills: both the theoretical knowledge needed to design controlled psycholinguistic experiments, and the technical proficiency needed to train and deploy large-scale language models. We present SyntaxGym, an online platform designed to make targeted evaluations accessible to both experts in NLP and linguistics, reproducible across computing environments, and standardized following the norms of psycholinguistic experimental design. This paper releases two tools of independent value for the computational linguistics community: 1. A website, syntaxgym.org, which centralizes the process of targeted syntactic evaluation and provides easy tools for analysis and visualization; 2. Two command-line tools, {`}syntaxgym{`} and {`}lm-zoo{`}, which allow any user to reproduce targeted syntactic evaluations and general language model inference on their own machine.",
}

metadata.yaml · Schema v2.0

This annotated template shows the structure of a model’s metadata.yaml. Each section groups related information; comments explain the expected types and allowed values.

Required: schema_version, domain, and models keyed by model identifier. Other fields may be omitted or set to null when undocumented. Use [] for empty lists.

The values below are illustrative, not this model’s metadata. Metadata changes are proposed in repository pull requests and published after merge.

# metadata.yaml — proposed schema v2.0
# Example values below are illustrative, not metadata for a real model.
# Required: schema_version, domain, and models keyed by exact model identifier.
# Other fields may be omitted or set to null when undocumented.
# Use [] for an empty list. Do not use "unknown" as a numeric/boolean value.
# Keep identifiers exact, including checkpoint suffixes.

schema_version: "2.0"  # String; quote the version.

domain: vision  # Must match the repository's registered domain.
models:
  example-model:
    sources:
      example-source:
        kind: other  # paper | huggingface | other | unreviewed
        url: https://example.org/model-card
        citation: Illustrative source, not evidence for a real model.
    model:
      display_name: Example model
      version: null  # String, e.g. "1.0".
      architecture:
        family: convolutional_neural_network  # String describing the architecture family.
        description: A convolutional model trained for image classification.
      parameter_count: 25000000  # Nonnegative integer; count, not "25M".
      parameter_count_exact: false  # Boolean; false means the count is approximate.
      trainable_layers: "All layers"  # Text; can describe partial fine-tuning.
      recurrent: false  # Boolean.
      input_resolution:
        channels: 3  # Nonnegative integer.
        height: 224  # Nonnegative integer, pixels.
        width: 224  # Nonnegative integer, pixels.
      visual_degrees: null  # Nonnegative number; vision models only.
      visual_degrees_description: null  # Text explaining the field of view.
      supervision:
        type: supervised  # String, e.g. supervised, self_supervised, weakly_supervised.
        description: Trained using class labels.

    training:
      process: null  # Text describing the training procedure.
      objective: null  # Text describing what the model learns.
      loss: null  # Text, e.g. "Cross-entropy".
      learning_rate: null  # Text; supports schedules, e.g. "0.1 with cosine decay".
      batch_size: null  # Text; supports different values across training stages.
      preprocessing: null  # Text describing normalization and augmentation.

    data:
      summary: null  # Text summarizing the data used.
      training_datasets:
        - identifier: example-training-set  # Optional dataset identifier.
          name: Example training set  # Required for each dataset entry.
          role: training  # training | pretraining | fine_tuning
          description: null  # Text describing this dataset's use.

    eval:
      test_datasets: []  # Entries: {identifier: optional, name: required, description: optional}.
      validation_datasets: []  # Same entry structure; role is implied by the list.

    io:
      modality: image  # String, e.g. image or text.
      interface: null  # Text describing the model's interface.
      input_format: null  # Text describing accepted inputs and their shape.
      output_format: null  # Text describing returned values and their shape.
      tokenizer: null  # Text; language models, where applicable.

    provenance:
      weights_provider: null  # Organization or repository providing the weights.
      checkpoint: null  # Exact checkpoint identifier.
      source_url: null  # Link to the original code or model card.
      curation_confidence: null  # Text summarizing confidence in the curated record.

    legal:
      license: null  # License identifier or full terms; distinguish code and weights.

    people:
      creators: []  # List of names, e.g. ["Example Researcher"].
      organizations: []  # List of organization names.

    use:
      applications: []  # List of intended applications (strings).
      users: []  # List of intended user groups (strings).
      limitations: []  # List of known limitations (strings).
      biases: []  # List of known biases (strings).

    lineage:
      base_models:
        - identifier: example-base  # Optional; exact identifier in the same domain.
          name: Example base model  # Required for each base-model entry.
          relationship: fine_tuned_from  # variant_of | fine_tuned_from | derived_from

    # Optional verification records. Omit this section if no verification is recorded.
    # Paths refer to fields in this YAML document, not database columns.
    assertions:
      - path: /model/architecture
        status: probable  # verified | probable | uncertain | undocumented
        sources: [example-source]  # References the structured sources above.