ICSE-2022 Cross-Domain Deep Code Search with Few-Shot Meta Learning

2022-07-05 · ICSE· Software · code-search, meta-learning

Paper: Cross-Domain Deep Code Search with Few-Shot Meta Learning

Code: https://github.com/fewshotcdcs/CDCS

CDCS: meta-learning with random task splits for code search and extensive experiments

Abstract

Pre-trained models such as CodeBERT have advanced code search considerably in recent years, but fine-tuning them to align semantic distributions typically requires large corpora, which limits practicality in domain-specific settings. The authors propose CDCS: pre-train on common corpora such as Python or Java, transfer to other languages such as SQL, and use a MAML-based few-shot meta-learning algorithm to obtain a strong parameter initialization.

Introduction

Data for domain-specific languages is scarce, so practitioners often train a large multilingual model first and then fine-tune on task-specific datasets. The challenge is how to transfer prior knowledge: languages differ in features and representations, which can yield conflicting shared representations—especially for domain-specific code search, where target-language examples are sparse in the training data.

CDCS combines CodeBERT with a meta-learning procedure to find an initialization that adapts well to new tasks with small training sets.

Approach

Pre-training

Similar to CodeBERT, MLM is used for pre-training; RTD is omitted because it did not help.

Each input is a <NL, PL> pair: [CLS], w…, [SEP], c…, [EOS], with natural language first and program language second.

Meta Learning

Training data are split into $k$ batches ${D_1,…,D_k}$; each $D_i$ is treated as a local task $T_i$, further divided into support and validation sets for inner updates, followed by an outer (global) update every $M$ steps.

Fine-Tuning

For code-search fine-tuning, retrieval is cast as binary classification: 1 for relevant and 0 for irrelevant. [CLS] serves as the pooled sequence representation; a fully connected head predicts whether a given <NL, PL> pair is relevant.