About
The GitHub Java corpus is a set of Java projects collected from GitHub.
Download
You can download the set of git URLs along with the commit SHAs of the project commit we used in code analysis and the train-test split that we've used RepositoryState.tar.gz [tar.gz 720kB]
A snapshot of all .java files in the corpus (no revisions included) can be downloaded from here [java_projects .tar.gz 1.8GB]
Corpus Statistics
| Train | Test | |
|---|---|---|
| Number of Projects | 10,968 | 3,817 |
| LOC | 264,225,189 | 88,087,507 |
| Tokens | 1,116,195,158 | 385,419,678 |
Use
The GitHub Java Corpus is a collection of Java code at a large scale, while it has been filtered to have an above-average quality (see next section). It can be used to study coding practice in Java at a large scale. It contains all project code, tracked by git with all the associated files. The containing open-source projects are from multiple domains. The code is not ours but is open-source. Please respect the license of each project.
If you would like to cite the corpus please use:
@inproceedings{githubCorpus2013,
author={Allamanis, Miltiadis and Sutton, Charles},
title={{Mining Source Code Repositories at Massive Scale using Language Modeling}},
booktitle={The 10th Working Conference on Mining Software Repositories},
year={2013},
pages={207--216},
organization={IEEE}
}
How the corpus was collected
We processed GitHub's event stream, provided by the GitHub Archive and filtered individual projects that were forked at least once. For each of these projects we collected the URL, the project's name and its language. Filtering by forks aims to create a quality corpus: Since we filter away the majority of projects that have a below-average "reputation", we obtain code of better quality. This criterion is not clear-cut and we could have chosen another method (e.g. GitHub's star system), but we found this preprocessing sufficient.
We then downloaded (clone in Git's terms)
the full repositories for all remaining Java projects
and removed any duplicates: We found about 1,000 projects that shared
common commit SHAs indicating that they were most likely forks of the
same project (but not declared on GitHub)
and we manually picked those projects that seemed to be the
original source. Project duplicates need to be removed to
avoid a "leak" of data from the test
to the train projects and also to create
a more representative corpus of repositories.
We split the repository 75%-25% based on the lines of code assigning projects into either the training or test set. The split can be found in the RepositoryState.tar.gz
Top projects in the corpus
We have also ordered the projects in the corpus according to popularity, defined as the number of forks plus the number of watchers, where each is first seprately normalized into a z-score. The ordered list can be downloaded from here [top_projects.tar.gz 5.3MB]
More Information
For more information on the corpus, please contact us:
- Miltiadis Allamanis: m [dot] allamanis (at) ed {dot} ac.uk
- Charles Sutton: csutton {at} inf [dot] ed (dot) ac.uk