Leaderboard Explore Compare Profile Community Blog FAQ Competition 2024 Report Issue

gpt2

Contributed by Brain-Score Team No reference provided Find on GitHub source

View details

≈124M parameters

Self-supervised MIT

Scores on benchmarks

from scoring API

Model rank shown below is with respect to all public models.

.397	average_language rank 10 5 benchmarks	.397 0 ceiling best median

.427	neural_language rank 17 4 benchmarks	.427 0 ceiling best median

.746	Pereira2018-ridge rank 10 2 benchmarks	.746 0 ceiling best median

.835	Pereira2018.243sentences-ridge v1 rank 9	.835 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.656	Pereira2018.384sentences-ridge v1 rank 10	.656 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.121	Blank2014-ridge v1 rank 10	.121 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.413	Fedorenko2016-ridge v3 rank 21	.413 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.367	behavior_language rank 4 1 benchmark	.367 0 ceiling best median

.367	Futrell2018-pearsonr v1 [reference] rank 4	.367 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.762	engineering_language rank 10 30 benchmarks	.762 0 ceiling best median

.762	SyntaxGym [reference] rank 10 30 benchmarks	.762 0 ceiling best median

.929	syntaxgym-center_embed v1 [reference] rank 15 1 benchmark	.929 0 ceiling best median

.857	syntaxgym-center_embed_mod v1 [reference] rank 18	.857 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

1.0	syntaxgym-cleft v1 [reference] rank 1 1 benchmark	1.0 0 ceiling best median

.925	syntaxgym-cleft_modifier v1 [reference] rank 12	.925 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.000	syntaxgym-fgd_hierarchy v1 [reference] rank 1	.000 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.958	syntaxgym-fgd_object v1 [reference] rank 3	.958 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.875	syntaxgym-fgd_pp v1 [reference] rank 4	.875 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.500	syntaxgym-fgd_subject v1 [reference] rank 5	.500 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.786	syntaxgym-mvrr v1 [reference] rank 5 1 benchmark	.786 0 ceiling best median

.750	syntaxgym-mvrr_mod v1 [reference] rank 11	.750 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.974	syntaxgym-npi_orc_any v1 [reference] rank 9	.974 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

1.0	syntaxgym-npi_orc_ever v1 [reference] rank 1	1.0 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.579	syntaxgym-npi_src_any v1 [reference] rank 18	.579 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.921	syntaxgym-npi_src_ever v1 [reference] rank 16	.921 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.667	syntaxgym-npz_ambig v1 [reference] rank 11 1 benchmark	.667 0 ceiling best median

.875	syntaxgym-npz_ambig_mod v1 [reference] rank 11	.875 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.958	syntaxgym-npz_obj_mod v1 [reference] rank 9	.958 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.632	syntaxgym-number_orc v1 [reference] rank 12	.632 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.789	syntaxgym-number_prep v1 [reference] rank 8	.789 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.789	syntaxgym-number_src v1 [reference] rank 4	.789 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.474	syntaxgym-reflexive_orc_fem v1 [reference] rank 3	.474 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.842	syntaxgym-reflexive_orc_masc v1 [reference] rank 9	.842 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.211	syntaxgym-reflexive_prep_fem v1 [reference] rank 17	.211 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.789	syntaxgym-reflexive_prep_masc v1 [reference] rank 6	.789 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.158	syntaxgym-reflexive_src_fem v1 [reference] rank 16	.158 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

.632	syntaxgym-reflexive_src_masc v1 [reference] rank 9	.632 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

1.0	syntaxgym-subordination v1 [reference] rank 1 3 benchmarks	1.0 0 ceiling best median

1.0	syntaxgym-subordination_orc-orc v1 [reference] rank 1	1.0 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

1.0	syntaxgym-subordination_pp-pp v1 [reference] rank 1	1.0 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

1.0	syntaxgym-subordination_src-src v1 [reference] rank 1	1.0 0 ceiling best median
	Sample stimuli cannot be displayed publicly for this benchmark.

How to use

from brainscore_language import load_model
model = load_model("gpt2")
model.start_task(...)
model.start_recording(...)
model.look_at(...)

Model API

Code examples

Curated metadata

At-a-glance

Architecture Probable: show confidence source Probable — Confidence not documented: 12 decoder blocks; width 768; 12 attention heads
Parameters Probable: show confidence source Probable — https://huggingface.co/openai-community/gpt2/blob/607a30d783dfa663caf39e06633721c8d4cfcd7e/README.md; https://github.com/openai/gpt-2/blob/9b63575ef42771a015060c964af2c3da4cf7c8ab/model_card.md: ≈124M
Input resolution: Not documented
Recurrent Probable: show confidence source Probable — https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf: No
Supervision Probable: show confidence source Probable — Confidence not documented: Causal language modeling on English text
Weights provider Probable: show confidence source Probable — https://github.com/openai/gpt-2/blob/9b63575ef42771a015060c964af2c3da4cf7c8ab/model_card.md: OpenAI
Trainable layers: Not documented
Checkpoint Probable: show confidence source Probable — https://github.com/brain-score/language/blob/f1128a65e8aeb302e388b7f9195deac9a10fd391/brainscore_language/models/gpt/__init__.py; https://github.com/brain-score/language/blob/f1128a65e8aeb302e388b7f9195deac9a10fd391/brainscore_language/model_helpers/huggingface.py: gpt2

Training and data

WebText pretraining; no task fine-tuning in the registered loader

Pretrain WebText

About 8 million documents, 40 GB; Wikipedia excluded; links through December 2017

Objective: Probable: show confidence source Probable — https://huggingface.co/openai-community/gpt2/blob/607a30d783dfa663caf39e06633721c8d4cfcd7e/README.md: Next-token prediction
Loss:: Not documented
Learning rate:: Not documented
Batch size: Probable: show confidence source Probable — https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf: 512 sequences (paper-wide setting)
Preprocessing: Probable: show confidence source Probable — https://huggingface.co/openai-community/gpt2/blob/607a30d783dfa663caf39e06633721c8d4cfcd7e/README.md; https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf: Byte-level BPE; 1,024-token sequences

Creators and licensing

Organization: Probable: show confidence source Probable — https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf; https://github.com/openai/gpt-2/blob/9b63575ef42771a015060c964af2c3da4cf7c8ab/model_card.md: OpenAI
Creators: Probable: show confidence source Probable — https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf: Alec Radford; Jeff Wu; Rewon Child; David Luan; Dario Amodei; Ilya Sutskever
License: Probable: show confidence source Probable — https://huggingface.co/openai-community/gpt2/blob/607a30d783dfa663caf39e06633721c8d4cfcd7e/README.md: MIT

Dashed Not documented fields are awaiting curation.

Intended use and limits

Recommended applications

Probable: show confidence source
Probable — https://github.com/openai/gpt-2/blob/9b63575ef42771a015060c964af2c3da4cf7c8ab/model_card.md; https://huggingface.co/openai-community/gpt2/blob/607a30d783dfa663caf39e06633721c8d4cfcd7e/README.md

Language-model research
Text generation

Target users

Probable: show confidence source
Probable — https://github.com/openai/gpt-2/blob/9b63575ef42771a015060c964af2c3da4cf7c8ab/model_card.md

AI researchers and practitioners

Known limitations

Probable: show confidence source
Probable — https://github.com/openai/gpt-2/blob/9b63575ef42771a015060c964af2c3da4cf7c8ab/model_card.md

Generated text can be factually incorrect
Assess bias before human-facing deployment

Biases

Probable: show confidence source
Probable — https://github.com/openai/gpt-2/blob/9b63575ef42771a015060c964af2c3da4cf7c8ab/model_card.md

Reflects biases in internet training text

Brain Encoding Response Generator (BERG)

Through the BERG you can easily generate neural responses to text sentences of your choice using any Brain-Score language model.

For more information on how to use BERG, see the documentation and tutorial.

Benchmarks BibTeX

@proceedings{futrell2018natural,
  title={The Natural Stories Corpus},
  author={Futrell, Richard and Gibson, Edward and Tily, Harry J. and Blank, Idan and Vishnevetsky, Anastasia and
          Piantadosi, Steven T. and Fedorenko, Evelina},
  conference={International Conference on Language Resources and Evaluation (LREC)},
  url={http://www.lrec-conf.org/proceedings/lrec2018/pdf/337.pdf},
  year={2018}
}
        @inproceedings{gauthier-etal-2020-syntaxgym,
    title = "{S}yntax{G}ym: An Online Platform for Targeted Evaluation of Language Models",
    author = "Gauthier, Jon and Hu, Jennifer and Wilcox, Ethan and Qian, Peng and Levy, Roger",
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations",
    month = jul,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.acl-demos.10",
    pages = "70--76",
    abstract = "Targeted syntactic evaluations have yielded insights into the generalizations learned by neural network language models. However, this line of research requires an uncommon confluence of skills: both the theoretical knowledge needed to design controlled psycholinguistic experiments, and the technical proficiency needed to train and deploy large-scale language models. We present SyntaxGym, an online platform designed to make targeted evaluations accessible to both experts in NLP and linguistics, reproducible across computing environments, and standardized following the norms of psycholinguistic experimental design. This paper releases two tools of independent value for the computational linguistics community: 1. A website, syntaxgym.org, which centralizes the process of targeted syntactic evaluation and provides easy tools for analysis and visualization; 2. Two command-line tools, {`}syntaxgym{`} and {`}lm-zoo{`}, which allow any user to reproduce targeted syntactic evaluations and general language model inference on their own machine.",
}

Evaluation and I/O

Test datasets: Not documented
Validation datasets: Not documented
Input format Probable: show confidence source Probable — https://github.com/brain-score/language/blob/f1128a65e8aeb302e388b7f9195deac9a10fd391/brainscore_language/models/gpt/__init__.py; https://github.com/brain-score/language/blob/f1128a65e8aeb302e388b7f9195deac9a10fd391/brainscore_language/model_helpers/huggingface.py; https://huggingface.co/openai-community/gpt2/blob/607a30d783dfa663caf39e06633721c8d4cfcd7e/README.md; https://huggingface.co/openai-community/gpt2/blob/607a30d783dfa663caf39e06633721c8d4cfcd7e/config.json: English text; 1,024-token model context
Output format Probable: show confidence source Probable — https://github.com/brain-score/language/blob/f1128a65e8aeb302e388b7f9195deac9a10fd391/brainscore_language/model_helpers/huggingface.py; https://github.com/brain-score/language/blob/f1128a65e8aeb302e388b7f9195deac9a10fd391/brainscore_language/models/gpt/__init__.py; https://github.com/brain-score/language/blob/f1128a65e8aeb302e388b7f9195deac9a10fd391/brainscore_language/model_helpers/huggingface.py: Behavioral predictions or requested layer activations
Tokenizer Probable: show confidence source Probable — https://huggingface.co/openai-community/gpt2/blob/607a30d783dfa663caf39e06633721c8d4cfcd7e/README.md; https://huggingface.co/openai-community/gpt2/blob/607a30d783dfa663caf39e06633721c8d4cfcd7e/config.json: Byte-level BPE; 50,257-token vocabulary
Model version Probable: show confidence source Probable — https://huggingface.co/openai-community/gpt2/blob/607a30d783dfa663caf39e06633721c8d4cfcd7e/README.md; https://github.com/brain-score/language/blob/f1128a65e8aeb302e388b7f9195deac9a10fd391/brainscore_language/model_helpers/huggingface.py: GPT-2 small; evaluated Hub revision unpinned

metadata.yaml · Schema v2.0

This annotated template shows the structure of a model’s metadata.yaml. Each section groups related information; comments explain the expected types and allowed values.

Required: schema_version, domain, and models keyed by model identifier. Other fields may be omitted or set to null when undocumented. Use [] for empty lists.

The values below are illustrative, not this model’s metadata. Metadata changes are proposed in repository pull requests and published after merge.

# metadata.yaml — proposed schema v2.0
# Example values below are illustrative, not metadata for a real model.
# Required: schema_version, domain, and models keyed by exact model identifier.
# Other fields may be omitted or set to null when undocumented.
# Use [] for an empty list. Do not use "unknown" as a numeric/boolean value.
# Keep identifiers exact, including checkpoint suffixes.

schema_version: "2.0"  # String; quote the version.

domain: vision  # Must match the repository's registered domain.
models:
  example-model:
    sources:
      example-source:
        kind: other  # paper | huggingface | other | unreviewed
        url: https://example.org/model-card
        citation: Illustrative source, not evidence for a real model.
    model:
      display_name: Example model
      version: null  # String, e.g. "1.0".
      architecture:
        family: convolutional_neural_network  # String describing the architecture family.
        description: A convolutional model trained for image classification.
      parameter_count: 25000000  # Nonnegative integer; count, not "25M".
      parameter_count_exact: false  # Boolean; false means the count is approximate.
      trainable_layers: "All layers"  # Text; can describe partial fine-tuning.
      recurrent: false  # Boolean.
      input_resolution:
        channels: 3  # Nonnegative integer.
        height: 224  # Nonnegative integer, pixels.
        width: 224  # Nonnegative integer, pixels.
      visual_degrees: null  # Nonnegative number; vision models only.
      visual_degrees_description: null  # Text explaining the field of view.
      supervision:
        type: supervised  # String, e.g. supervised, self_supervised, weakly_supervised.
        description: Trained using class labels.

    training:
      process: null  # Text describing the training procedure.
      objective: null  # Text describing what the model learns.
      loss: null  # Text, e.g. "Cross-entropy".
      learning_rate: null  # Text; supports schedules, e.g. "0.1 with cosine decay".
      batch_size: null  # Text; supports different values across training stages.
      preprocessing: null  # Text describing normalization and augmentation.

    data:
      summary: null  # Text summarizing the data used.
      training_datasets:
        - identifier: example-training-set  # Optional dataset identifier.
          name: Example training set  # Required for each dataset entry.
          role: training  # training | pretraining | fine_tuning
          description: null  # Text describing this dataset's use.

    eval:
      test_datasets: []  # Entries: {identifier: optional, name: required, description: optional}.
      validation_datasets: []  # Same entry structure; role is implied by the list.

    io:
      modality: image  # String, e.g. image or text.
      interface: null  # Text describing the model's interface.
      input_format: null  # Text describing accepted inputs and their shape.
      output_format: null  # Text describing returned values and their shape.
      tokenizer: null  # Text; language models, where applicable.

    provenance:
      weights_provider: null  # Organization or repository providing the weights.
      checkpoint: null  # Exact checkpoint identifier.
      source_url: null  # Link to the original code or model card.
      curation_confidence: null  # Text summarizing confidence in the curated record.

    legal:
      license: null  # License identifier or full terms; distinguish code and weights.

    people:
      creators: []  # List of names, e.g. ["Example Researcher"].
      organizations: []  # List of organization names.

    use:
      applications: []  # List of intended applications (strings).
      users: []  # List of intended user groups (strings).
      limitations: []  # List of known limitations (strings).
      biases: []  # List of known biases (strings).

    lineage:
      base_models:
        - identifier: example-base  # Optional; exact identifier in the same domain.
          name: Example base model  # Required for each base-model entry.
          relationship: fine_tuned_from  # variant_of | fine_tuned_from | derived_from

    # Optional verification records. Omit this section if no verification is recorded.
    # Paths refer to fields in this YAML document, not database columns.
    assertions:
      - path: /model/architecture
        status: probable  # verified | probable | uncertain | undocumented
        sources: [example-source]  # References the structured sources above.