Can We Predict a Breakthrough?

A 2023 Nature paper argued that groundbreaking papers are getting rarer. The assignment: predict a paper's disruption score (the CD index) using only information available on publication day, with at least two ML approaches, one of them a neural network.

I cut a 149,078-paper SQLite citation database down to 44,553 clean training papers and built five features known at publication: authors, references, mean reference age, publication year and number of subject areas. Messy reference years were recovered by sharing known years across every appearance of the same DOI. I trained gradient-boosted trees (XGBoost) and a two-layer Keras network. I also clustered research fields with TF-IDF and K-means, but left it out of the final models because it could leak information from later papers.

XGBoost reached R² 0.96 on the held-out test set and the neural network an MAE of 0.021. Looking back, the 0.96 flatters the model: papers with no recorded references score CD = 1 by definition, so a simple baseline, a separate model for papers with references, and a time-based split would make the result more honest.