BigCode 背後的大規模近乎重複資料刪除技術

This technical blog post from Hugging Face takes an in-depth look at the challenges the BigCode project (the collaborative initiative behind StarCoder) faced when processing large-scale code datasets, and the solutions they developed. When training code LLMs, datasets are frequently filled with large numbers of duplicate or highly similar code files — for example, forked repositories or copy-pasted code snippets. Training directly on such data causes the model to overfit and memorize specific code, and can also raise privacy and licensing concerns.