Compare model versions

Diff

Before

ESMC is a state-of-the-art protein language model that has learned the rules of protein biology from training on billions of protein sequences. ESMC provides representations of proteins enabling novel AI applications from therapeutic protein engineering to unlocking basic insights into protein biology across life.

The ESMC 6B model has 6 billion parameters, with 80 layers and 2.37e23 training flops. We additionally release overtrained 300M and 600M parameter variants of ESMC for local inference and finetuning.

The [ESMFold2](https://huggingface.co/biohub/ESMFold2) structure prediction models are trained on top of a frozen ESMC 6B language model. ESMFold2 is a state-of-the-art model for protein structure prediction and design that defines a new frontier for speed and accuracy.

The [ESMC sparse autoencoder](https://huggingface.co/biohub/ESMC-6B-sae-layer60-k64-codebook16384), `ESMC-6B-sae-layer60-k64-codebook16384`, is built on the ESMC 6B model and provides human-interpretable, agent-generated feature descriptions. See the [ESMC SAE overview card](https://huggingface.co/biohub/ESMC-SAE-Overview) for the full set of ESMC SAE variants.

To run this model with the Biohub Platform API, visit the [Biohub Platform](https://biohub.ai/).

Read more about ESMC in our paper [here](https://biohub.ai/papers/esm_protein.pdf).

After

ESMC is a state-of-the-art protein language model that has learned the rules of protein biology from training on billions of protein sequences. ESMC provides representations of proteins enabling novel AI applications from therapeutic protein engineering to unlocking basic insights into protein biology across life.

The ESMC 6B model has 6 billion parameters, with 80 layers and 2.37e23 training flops. We additionally release overtrained 300M and 600M parameter variants of ESMC for local inference and finetuning.

The [ESMFold2](https://huggingface.co/biohub/ESMFold2) structure prediction models are trained on top of a frozen ESMC 6B language model. ESMFold2 is a state-of-the-art model for protein structure prediction and design that defines a new frontier for speed and accuracy.

The [ESMC sparse autoencoder](https://huggingface.co/biohub/ESMC-6B-sae-layer60-k64-codebook16384), `ESMC-6B-sae-layer60-k64-codebook16384`, is built on the ESMC 6B model and provides human-interpretable, agent-generated feature descriptions. See the [ESMC SAE overview card](https://huggingface.co/biohub/ESMC-SAE-Overview) for the full set of ESMC SAE variants.

To run this model with the Biohub Platform API, visit the [Biohub Platform](https://biohub.ai/).

Read more about ESMC in our paper [here](https://www.biorxiv.org/content/10.64898/2026.06.03.729735).

Diff

Before

Install `esm` from GitHub (a PyPI release is coming soon):

```
pip install esm@git+https://github.com/Biohub/esm.git@main
```

```py
import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

GFP = "MSKGEELFTGVVPILVELDGDVNGHKFSVSGEGEGDATYGKLTLKFICTTGKLPVPWPTLVTTFSYGVQCFSRYPDHMKQHDFFKSAMPEGYVQERTIFFKDDGNYKTRAEVKFEGDTLVNRIELKGIDFKEDGNILGHKLEYNYNSHNVYIMADKQKNGIKVNFKIRHNIEDGSVQLADHYQQNTPIGDGPVLLPDNHYLSTQSALSKDPNEKRDHMVLLEFVTAAGITHGMDELYK"

# optionally use "biohub/ESMC-600M" or "biohub/ESMC-300M"
model = AutoModelForMaskedLM.from_pretrained("biohub/ESMC-6B", device_map="auto").eval()
tokenizer = AutoTokenizer.from_pretrained("biohub/ESMC-6B")

inputs = tokenizer(GFP, return_tensors="pt", padding=True)
inputs = {k: v.to(model.device) for k, v in inputs.items()}

with torch.inference_mode():
    output = model(**inputs)

print(f"logits shape: {tuple(output.logits.shape)}")
```

By default, the model returns only the final layer representations. To return hidden states from **all transformer layers**, set:

```py
output = model(**inputs, output_hidden_states=True)
```

For detailed usage, refer to the [Usage section below](#usage).

After

Install `esm` from PyPI:

```
pip install esm
```

```py
import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

GFP = "MSKGEELFTGVVPILVELDGDVNGHKFSVSGEGEGDATYGKLTLKFICTTGKLPVPWPTLVTTFSYGVQCFSRYPDHMKQHDFFKSAMPEGYVQERTIFFKDDGNYKTRAEVKFEGDTLVNRIELKGIDFKEDGNILGHKLEYNYNSHNVYIMADKQKNGIKVNFKIRHNIEDGSVQLADHYQQNTPIGDGPVLLPDNHYLSTQSALSKDPNEKRDHMVLLEFVTAAGITHGMDELYK"

# optionally use "biohub/ESMC-600M" or "biohub/ESMC-300M"
model = AutoModelForMaskedLM.from_pretrained("biohub/ESMC-6B", device_map="auto").eval()
tokenizer = AutoTokenizer.from_pretrained("biohub/ESMC-6B")

inputs = tokenizer(GFP, return_tensors="pt", padding=True)
inputs = {k: v.to(model.device) for k, v in inputs.items()}

with torch.inference_mode():
    output = model(**inputs)

print(f"logits shape: {tuple(output.logits.shape)}")
```

By default, the model returns only the final layer representations. To return hidden states from **all transformer layers**, set:

```py
output = model(**inputs, output_hidden_states=True)
```

For detailed usage, refer to the [Usage section below](#usage).

Diff

Before

```
@misc{candido2026language,
  title  = {Language Modeling Materializes a World Model of Protein Biology},
  author = {Candido, Salvatore and Hayes, Thomas and Derry, Alexander and Rao, Roshan
            and Lin, Zeming and Verkuil, Robert and Wu, Bryan and Lee, Jin Sub
            and Bruguera, Elise S. and Keval, Jehan A. and Kopylov, Mykhailo
            and Pak, John E. and Wu, Wesley and Thomas, Neil and Mataraso, Samson
            and Hsu, Alvin and Trotman-Grant, Ashton C. and Fatras, Kilian
            and dos Santos Costa, Allan and Badkundri, Rohil and Ak{\i}n, Halil
            and Oktay, Deniz and Deaton, Jonathan and Montabana, Elizabeth
            and Sitwala, Hrishita and Yu, Yue and Wiggert, Marius
            and Carlin, Dylan Alexander and Goering, Anthony W. and Blazejewski, Tomasz
            and Sandora, McCullen and Hla, Michael and Jia, Tina Z.
            and Kloker, Leon H. and Sofroniew, Nicholas J. and Uehara, Masatoshi
            and Pannu, Jassi and Bachas, Sharrol and Liu, Daniel S.
            and Sercu, Tom and Rives, Alexander},
  year   = {2026},
  url    = {https://biohub.ai/papers/esm_protein.pdf},
  note   = {Preprint}
}
```

After

```
@misc{candido2026language,
  title  = {Language Modeling Materializes a World Model of Protein Biology},
  author = {Candido, Salvatore and Hayes, Thomas and Derry, Alexander and Rao, Roshan
            and Lin, Zeming and Verkuil, Robert and Wu, Bryan and Lee, Jin Sub
            and Bruguera, Elise S. and Keval, Jehan A. and Kopylov, Mykhailo
            and Pak, John E. and Wu, Wesley and Thomas, Neil and Mataraso, Samson
            and Hsu, Alvin and Trotman-Grant, Ashton C. and Fatras, Kilian
            and dos Santos Costa, Allan and Badkundri, Rohil and Ak{\i}n, Halil
            and Oktay, Deniz and Deaton, Jonathan and Montabana, Elizabeth
            and Sitwala, Hrishita and Yu, Yue and Wiggert, Marius
            and Carlin, Dylan Alexander and Goering, Anthony W. and Blazejewski, Tomasz
            and Sandora, McCullen and Hla, Michael and Jia, Tina Z.
            and Kloker, Leon H. and Sofroniew, Nicholas J. and Uehara, Masatoshi
            and Pannu, Jassi and Bachas, Sharrol and Liu, Daniel S.
            and Sercu, Tom and Rives, Alexander},
  year   = {2026},
  url    = {https://www.biorxiv.org/content/10.64898/2026.06.03.729735},
  note   = {Preprint}
}
```

Diff

Before

Performance metrics are detailed in our [ESMC & ESMFold2 paper](https://biohub.ai/papers/esm_protein.pdf).

After

Performance metrics are detailed in our [ESMC & ESMFold2 paper](https://www.biorxiv.org/content/10.64898/2026.06.03.729735).

Diff

Before

To get interpretable features from ESMC 6B hidden states and per-layer MLP outputs, you can choose from our pretrained SAEs. We provide the follow three:

* [ESMC SAEs for hidden states (all layers)](https://huggingface.co/collections/biohub/esmc-saes-for-hidden-states-all-layers)
* [ESMC SAEs for one layer (different sparsity/codebook size)](https://huggingface.co/collections/biohub/esmc-saes-for-one-layer-different-sparsity-codebook-size)
* [ESMC SAEs for MLP outputs (all layers)](https://huggingface.co/collections/biohub/esmc-saes-for-mlp-outputs-all-layers)

```py
import torch
from transformers import AutoModel, AutoTokenizer

GFP = "MSKGEELFTGVVPILVELDGDVNGHKFSVSGEGEGDATYGKLTLKFICTTGKLPVPWPTLVTTFSYGVQCFSRYPDHMKQHDFFKSAMPEGYVQERTIFFKDDGNYKTRAEVKFEGDTLVNRIELKGIDFKEDGNILGHKLEYNYNSHNVYIMADKQKNGIKVNFKIRHNIEDGSVQLADHYQQNTPIGDGPVLLPDNHYLSTQSALSKDPNEKRDHMVLLEFVTAAGITHGMDELYK"

model = AutoModel.from_pretrained("biohub/ESMC-6B", device_map="auto").eval()
tokenizer = AutoTokenizer.from_pretrained("biohub/ESMC-6B")

sae_models = []
sae = AutoModel.from_pretrained(
    "biohub/ESMC-6B-sae-sweep-layer60-k64-codebook16384", device_map="auto"
)
sae_models.append(sae)

model.add_sae_models(sae_models)

inputs = tokenizer(GFP, return_tensors="pt", padding=True)
inputs = {k: v.to(model.device) for k, v in inputs.items()}

with torch.inference_mode():
    output = model(**inputs)

print(f"num SAE outputs: {len(output.sae_outputs)}")
for i, sae_out in enumerate(output.sae_outputs):
    print(f"  SAE[{i}]: {type(sae_out).__name__}")
```

After

To get interpretable features from ESMC 6B hidden states and per-layer MLP outputs, you can choose from our pretrained SAEs. We provide the follow three:

* [ESMC SAEs for hidden states (all layers)](https://huggingface.co/collections/biohub/esmc-saes-for-hidden-states-all-layers)
* [ESMC SAEs for one layer (different sparsity/codebook size)](https://huggingface.co/collections/biohub/esmc-saes-for-one-layer-different-sparsity-codebook-size)
* [ESMC SAEs for MLP outputs (all layers)](https://huggingface.co/collections/biohub/esmc-saes-for-mlp-outputs-all-layers)

```py
import torch
from esm.models.esmc import EsmcModel, EsmcSaeModel, EsmcTokenizer

GFP = "MSKGEELFTGVVPILVELDGDVNGHKFSVSGEGEGDATYGKLTLKFICTTGKLPVPWPTLVTTFSYGVQCFSRYPDHMKQHDFFKSAMPEGYVQERTIFFKDDGNYKTRAEVKFEGDTLVNRIELKGIDFKEDGNILGHKLEYNYNSHNVYIMADKQKNGIKVNFKIRHNIEDGSVQLADHYQQNTPIGDGPVLLPDNHYLSTQSALSKDPNEKRDHMVLLEFVTAAGITHGMDELYK"

model = EsmcModel.from_pretrained("biohub/ESMC-6B").eval()
tokenizer = EsmcTokenizer.from_pretrained("biohub/ESMC-6B")

sae = EsmcSaeModel.from_pretrained(
    "biohub/ESMC-6B-sae-layer60-k64-codebook16384", device=model.device
)
model.add_sae_models([sae.layers["60"]])

inputs = tokenizer(GFP, return_tensors="pt", padding=True)
inputs = {k: v.to(model.device) for k, v in inputs.items()}

with torch.inference_mode():
    output = model(**inputs)

print(f"SAE features: {tuple(output.sae_outputs['layer60'].shape)}")
```

Diff

Before

```py
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForMaskedLM

model = AutoModelForMaskedLM.from_pretrained("biohub/ESMC-6B", device_map="auto")

lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    lora_dropout=0.01,
    target_modules=["layernorm_qkv.1", "out_proj", "ffn.1", "ffn.3"],
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
```

After

```py
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForMaskedLM

model = AutoModelForMaskedLM.from_pretrained("biohub/ESMC-6B", device_map="auto")

lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    lora_dropout=0.01,
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
    ],
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
```

Diff

Before

To extract attention maps, pass `output_attentions=True`. Note: this is incompatible with `attn_implementation="flash_attention_2"`.

```py
output = model(**inputs, output_attentions=True)
# output.attentions: tuple of (batch, n_heads, seq_len, seq_len) tensors, one per layer
```

`output_attentions=True` triggers a manual, unoptimized attention path to extract the attention maps, which will reduce inference speed.

After

To extract attention maps, pass `output_attentions=True`. Note: this is incompatible with `attn_implementation="flash_attention_2"`.

```py
model = AutoModelForMaskedLM.from_pretrained(
    "biohub/ESMC-6B", device_map="auto", attn_implementation="eager"
).eval()
output = model(**inputs, output_attentions=True)
# output.attentions: tuple of (batch, n_heads, seq_len, seq_len) tensors, one per layer
```

`output_attentions=True` triggers a manual, unoptimized attention path to extract the attention maps, which will reduce inference speed.