Scores on benchmarks

from scoring API
Model rank shown below is with respect to all public models.
.304 average_language rank 21
5 benchmarks
.304
0
ceiling
best
median
.607 neural_language rank 2
4 benchmarks
.607
0
ceiling
best
median
.995 Pereira2018-ridge rank 1
2 benchmarks
.995
0
ceiling
best
median
1.0 Pereira2018.243sentences-ridge v1 rank 1
1.0
0
ceiling
best
median
sample 0 sample 1 sample 2 sample 3 sample 4 sample 5 sample 6 sample 7 sample 8 sample 9
.989 Pereira2018.384sentences-ridge v1 rank 1
.989
0
ceiling
best
median
sample 0 sample 1 sample 2 sample 3 sample 4 sample 5 sample 6 sample 7 sample 8 sample 9
.209 Blank2014-ridge v1 rank 2
.209
0
ceiling
best
median
sample 0 sample 1 sample 2 sample 3 sample 4 sample 5 sample 6 sample 7 sample 8 sample 9
.618 Fedorenko2016-ridge v3 rank 9
.618
0
ceiling
best
median
sample 0 sample 1 sample 2 sample 3 sample 4 sample 5 sample 6 sample 7 sample 8 sample 9

How to use

from brainscore_language import load_model
model = load_model("llada-8b-base")
model.start_task(...)
model.start_recording(...)
model.look_at(...)

Brain Encoding Response Generator (BERG)

Through the BERG you can easily generate neural responses to text sentences of your choice using any Brain-Score language model.

For more information on how to use BERG, see the documentation and tutorial.

Benchmarks BibTeX

@proceedings{futrell2018natural,
  title={The Natural Stories Corpus},
  author={Futrell, Richard and Gibson, Edward and Tily, Harry J. and Blank, Idan and Vishnevetsky, Anastasia and
          Piantadosi, Steven T. and Fedorenko, Evelina},
  conference={International Conference on Language Resources and Evaluation (LREC)},
  url={http://www.lrec-conf.org/proceedings/lrec2018/pdf/337.pdf},
  year={2018}
}
        @inproceedings{gauthier-etal-2020-syntaxgym,
    title = "{S}yntax{G}ym: An Online Platform for Targeted Evaluation of Language Models",
    author = "Gauthier, Jon and Hu, Jennifer and Wilcox, Ethan and Qian, Peng and Levy, Roger",
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations",
    month = jul,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.acl-demos.10",
    pages = "70--76",
    abstract = "Targeted syntactic evaluations have yielded insights into the generalizations learned by neural network language models. However, this line of research requires an uncommon confluence of skills: both the theoretical knowledge needed to design controlled psycholinguistic experiments, and the technical proficiency needed to train and deploy large-scale language models. We present SyntaxGym, an online platform designed to make targeted evaluations accessible to both experts in NLP and linguistics, reproducible across computing environments, and standardized following the norms of psycholinguistic experimental design. This paper releases two tools of independent value for the computational linguistics community: 1. A website, syntaxgym.org, which centralizes the process of targeted syntactic evaluation and provides easy tools for analysis and visualization; 2. Two command-line tools, {`}syntaxgym{`} and {`}lm-zoo{`}, which allow any user to reproduce targeted syntactic evaluations and general language model inference on their own machine.",
}
        

metadata.yaml · Schema v2.0

This annotated template shows the structure of a model’s metadata.yaml. Each section groups related information; comments explain the expected types and allowed values.

Required: schema_version, domain, and models keyed by model identifier. Other fields may be omitted or set to null when undocumented. Use [] for empty lists.

The values below are illustrative, not this model’s metadata. Metadata changes are proposed in repository pull requests and published after merge.

# metadata.yaml — proposed schema v2.0
# Example values below are illustrative, not metadata for a real model.
# Required: schema_version, domain, and models keyed by exact model identifier.
# Other fields may be omitted or set to null when undocumented.
# Use [] for an empty list. Do not use "unknown" as a numeric/boolean value.
# Keep identifiers exact, including checkpoint suffixes.

schema_version: "2.0"  # String; quote the version.

domain: vision  # Must match the repository's registered domain.
models:
  example-model:
    sources:
      example-source:
        kind: other  # paper | huggingface | other | unreviewed
        url: https://example.org/model-card
        citation: Illustrative source, not evidence for a real model.
    model:
      display_name: Example model
      version: null  # String, e.g. "1.0".
      architecture:
        family: convolutional_neural_network  # String describing the architecture family.
        description: A convolutional model trained for image classification.
      parameter_count: 25000000  # Nonnegative integer; count, not "25M".
      parameter_count_exact: false  # Boolean; false means the count is approximate.
      trainable_layers: "All layers"  # Text; can describe partial fine-tuning.
      recurrent: false  # Boolean.
      input_resolution:
        channels: 3  # Nonnegative integer.
        height: 224  # Nonnegative integer, pixels.
        width: 224  # Nonnegative integer, pixels.
      visual_degrees: null  # Nonnegative number; vision models only.
      visual_degrees_description: null  # Text explaining the field of view.
      supervision:
        type: supervised  # String, e.g. supervised, self_supervised, weakly_supervised.
        description: Trained using class labels.

    training:
      process: null  # Text describing the training procedure.
      objective: null  # Text describing what the model learns.
      loss: null  # Text, e.g. "Cross-entropy".
      learning_rate: null  # Text; supports schedules, e.g. "0.1 with cosine decay".
      batch_size: null  # Text; supports different values across training stages.
      preprocessing: null  # Text describing normalization and augmentation.

    data:
      summary: null  # Text summarizing the data used.
      training_datasets:
        - identifier: example-training-set  # Optional dataset identifier.
          name: Example training set  # Required for each dataset entry.
          role: training  # training | pretraining | fine_tuning
          description: null  # Text describing this dataset's use.

    eval:
      test_datasets: []  # Entries: {identifier: optional, name: required, description: optional}.
      validation_datasets: []  # Same entry structure; role is implied by the list.

    io:
      modality: image  # String, e.g. image or text.
      interface: null  # Text describing the model's interface.
      input_format: null  # Text describing accepted inputs and their shape.
      output_format: null  # Text describing returned values and their shape.
      tokenizer: null  # Text; language models, where applicable.

    provenance:
      weights_provider: null  # Organization or repository providing the weights.
      checkpoint: null  # Exact checkpoint identifier.
      source_url: null  # Link to the original code or model card.
      curation_confidence: null  # Text summarizing confidence in the curated record.

    legal:
      license: null  # License identifier or full terms; distinguish code and weights.

    people:
      creators: []  # List of names, e.g. ["Example Researcher"].
      organizations: []  # List of organization names.

    use:
      applications: []  # List of intended applications (strings).
      users: []  # List of intended user groups (strings).
      limitations: []  # List of known limitations (strings).
      biases: []  # List of known biases (strings).

    lineage:
      base_models:
        - identifier: example-base  # Optional; exact identifier in the same domain.
          name: Example base model  # Required for each base-model entry.
          relationship: fine_tuned_from  # variant_of | fine_tuned_from | derived_from

    # Optional verification records. Omit this section if no verification is recorded.
    # Paths refer to fields in this YAML document, not database columns.
    assertions:
      - path: /model/architecture
        status: probable  # verified | probable | uncertain | undocumented
        sources: [example-source]  # References the structured sources above.

Layer commitment

Not documented

No layer commitments found for this model. Older submissions might not have stored this information but will be updated when evaluated on new benchmarks.

Visual angle

Not documented

Drives stimulus scaling. A null here means benchmarks run on the default.