NeurIPS-2021 Language models enable zero-shot prediction of the effects of mutations on protein function
Paper: Language models enable zero-shot prediction of the effects of mutations on protein function
ESM-1v: Zero-Shot Prediction of Protein Mutations
Abstract
Evolutionary information is encoded in protein sequences, and unsupervised models can learn the impact of variants from sequence data alone. Existing approaches fit models to sequences from individual families, which requires training a new model for each prediction task and is highly limiting. This work proposes a zero-shot model instead.
Introduction
The functional effects of sequence variants can be measured with deep mutational scanning experiments.
The authors introduce the unsupervised model ESM-1v.
Zero-Shot Transfer
Zero-shot learning requires that the model acquire during pretraining all information needed for downstream tasks. Here, the mechanism that enables zero-shot prediction is pretraining protein language models on large databases rich in evolutionary signal. When the database is sufficiently large and diverse, the model can learn sequence patterns spanning the entire tree of life and, in pretraining, capture patterns characteristic of the protein families to which it will be applied—so no additional training is needed at transfer time.
Method
Amino-acid conservation at a site gauges how strongly a variant may matter: if, relative to the wild type, other amino acids have very low probability at that position, the sequence is conserved there and substitution is unlikely, suggesting that the residue is important for structure and function.
To score how a variant affects protein function, the authors compare, at each masked position, the probability of the mutant amino acid versus the wild-type amino acid and take their log-odds (the log ratio of the two probabilities).
ESM-1v uses the same architecture as ESM-1b.
-
Training: masked language modeling
-
Training data: UR90, with 98 million diverse protein sequences—substantially greater in scale and diversity than UR50 used for ESM-1b and MSA Transformer—yielding stronger transfer performance
Result
<HR align=left color=#987cb9 SIZE=1>