Journal of Chemical Information and Modeling 2024

SES-Adapter

A simple, efficient, and scalable structure-aware adapter for protein language models — serialize structure, attend across modalities, and fine-tune faster with stronger downstream accuracy.

Yang Tan, Mingchen Li, Bingxin Zhou, Bozitao Zhong, Lirong Zheng, Pan Tan, Ziyi Zhou, Huiqun Yu, Guisheng Fan, Liang Hong

Paper Code
SES-Adapter overview: structure serialization with cross-modal attention on PLMs
SES-Adapter injects serialized structural tokens into a PLM via cross-modal attention without redesigning the backbone.

Motivation

Sequence-only protein language models (PLMs) already transfer well, but many property labels — localization, function, solubility, annotation — still benefit from 3D context. Full structure-aware pretraining is expensive; heavy fusion architectures can be hard to scale across heterogeneous PLM families.

SES-Adapter treats structure as a lightweight adapter: keep the frozen (or lightly updated) PLM, serialize geometric information into a parallel token stream, and let cross-modal attention mix structure with sequence embeddings during fine-tuning.

Method in brief

Given a PDB / AlphaFold structure aligned to the sequence, SES-Adapter:

  1. Serializes local structural context into discrete structure tokens (compatible with multiple structure tokenizers, including an ESM3 structure-tokenizer path in the repo).
  2. Runs the PLM to obtain residue embeddings.
  3. Applies cross-modal attention between structure tokens and PLM states so geometry can steer representations for the downstream head.

The design is intentionally simple (adapter-style, not a new foundation model), efficient (faster wall-clock fine-tuning), and scalable across PLM backbones and dataset formats (JSON or Hugging Face CSV configs).

Benchmark scope

Evaluation covers 9 PLMs (including ESM2, ProtBert, ProtT5, and Ankh families) × 9 datasets × 4 task types:

  • Localization
  • Function
  • Solubility
  • Annotation

Versus vanilla PLM fine-tuning, SES-Adapter improves downstream performance by up to +11% (average +3%), accelerates training by up to +1034% (average +362%), and converges roughly 2× faster.

Using the code

The tyang816/SES-Adapter repository ships dataset configs, training scripts, and metric helpers (accuracy, F1, MCC, AUROC, Spearman, and multi-label f1_max). Point a JSON/CSV config at your train/valid/test splits (or a Hugging Face dataset id), choose a PLM backbone, and run train.py — see the scripts/ folder for examples.

Hardware note from the README: a 24 GB GPU (e.g. RTX 3090) is a practical baseline; memory demand mainly tracks the chosen PLM size.

Citation

If you use SES-Adapter, please cite:

@article{tan2024ses-adapter,
  title={Simple, Efficient, and Scalable Structure-Aware Adapter Boosts Protein Language Models},
  author={Tan, Yang and Li, Mingchen and Zhou, Bingxin and Zhong, Bozitao and Zheng, Lirong and Tan, Pan and Zhou, Ziyi and Yu, Huiqun and Fan, Guisheng and Hong, Liang},
  journal={Journal of Chemical Information and Modeling},
  volume={64},
  number={16},
  pages={6338-6349},
  year={2024},
  publisher={American Chemical Society},
  doi={10.1021/acs.jcim.4c00689}
}