EMNLP-2021 CodeT5：Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation

2021-09-02 · EMNLP· Software · code-understanding, transformer

Paper: CodeT5：Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation

Code: https://github.com/salesforce/CodeT5

CodeT5: full Transformer, identifiers, and bimodal learning

Abstract

Most existing pre-training methods use encoder-only or decoder-only architectures. This work proposes CodeT5, a unified encoder–decoder Transformer that supports both code understanding and generation, together with an identifier-aware pre-training objective.

Introduction

Pre-trained models have succeeded widely on language tasks, yet most are either encoder-only (e.g., BERT) or decoder-only (e.g., GPT), which is suboptimal when both understanding and generation are required. For instance, CodeBERT needs an extra decoder for code summarization, and that decoder cannot leverage the pre-training stage.

Conventional NLP-style pipelines also treat source code as a token sequence much like natural language, largely overlooking the rich structured semantics of code.

CodeT5 uses denoising seq2seq pre-training, exploits developer-assigned identifiers, learns stronger natural-language–programming-language (NL–PL) alignment from code and comments, and is pre-trained on CodeSearchNet.

CodeT5

Pre-training Tasks

Identifier-aware Denoising Pre-training

The source sequence is first corrupted with noise functions; the decoder then reconstructs the original text—this is masked span prediction (MSP).

To incorporate more code-specific structure (identifier node types in the AST), two auxiliary tasks complement denoising pre-training: Identifier Tagging (IT) and Masked Identifier Prediction (MIP).

Identifier Tagging (IT): indicates whether a token is an identifier.
Masked Identifier Prediction (MIP): unlike masking a contiguous span, MIP masks all identifiers in a code fragment.