Cell Systems-2024 Convolutions are competitive with transformers for protein sequence pretraining

Paper: Convolutions are competitive with transformers for protein sequence pretraining

Code: https://github.com/microsoft/protein-sequence-models/tree/main

CARP: convolutional masked networks for protein language model pre-training

Abstract

CNNs can match transformers with competitive performance and strong results on many downstream tasks, including structure prediction, zero-shot mutation effect prediction, and out-of-domain fitness prediction.

Introduction and background

A major limitation of transformers is that compute and memory scale quadratically with input sequence length, which leads to strict length caps during training—for example, ESM’s maximum length is 1022. UniRef50 contains 42M sequences, of which 1.1M (2.6%) exceed 1022, including many proteins of broad interest such as the SARS-CoV-2 spike glycoprotein and Streptococcus pyogenes CRISPR-associated endonuclease Cas9.

CARP (Convolutional Autoencoding Representations of Proteins) was trained on the March 2020 release of UniRef50.

Convolutional protein sequence masked language models

15% of tokens are masked: 80% are replaced with a mask token, 10% are replaced with a randomly chosen amino acid, and 10% are left unchanged.

Downstream Tasks

Protein Structure

Zero-shot Mutation Effect Prediction

Out-of-Domain Fitness Prediction

In-Domain Property Prediction