npm malware detection

A detector that scored 0.999 PR-AUC on a public benchmark flagged 16.6% of 5,000 random fresh npm uploads, about 4,900 false alarms a day at npm's publish rate. I found out why, and fixed it.A malware detector that scored almost perfectly on a public test wrongly flagged 16.6% of 5,000 real, newly published code packages: about 4,900 false alarms a day at the rate packages are published. I found out why, and fixed it.

Code Paper Article

01/08 · 0:00The problem

0.999 out of 1 on a research benchmark. On real npm uploads, it flagged one package in six.

The project video: the benchmark score, the shortcut behind the false alarms, and the two fixes.

0.999PR-AUC of the LightGBM model on a random split of the ASE 2026 benchmark (Guo et al.). ASE 2026 benchmark

PR-AUC on the benchmarkscore on the standard test, out of 1

16.6%5,000 random fresh npm uploads, at the threshold tuned for 1% false alarms on the benchmark's validation set. README

of 5,000 real uploads flaggedof 5,000 real packages wrongly flagged

81% at 0.7%Retrained with fresh uploads and the threshold set on uploads. Mean of five training runs, on a second, later sample of 5,000 uploads. results/experiments

new malware caught, false alarms, after the fix

The problem

Benchmarks for npm malware detectors report near-perfect scores. But a detector is used on live uploads, where almost every package is benign. So I tested one the way it would actually be used: on malware published after it was trained, on attack campaigns it had never seen, and on a random sample of real registry traffic.Tests for these detectors report near-perfect scores. But in real use, almost every new package is harmless. So I tested one the way it would really be used: on malware that appeared after it was built, on attack campaigns it had never seen, and on a random sample of real new packages.

What broke

With its threshold tuned for 1% false alarms on benchmark validation, a LightGBM model flagged 16.6% of fresh uploads, far above the target.Set to raise false alarms on only 1% of the test's packages, the model flagged 16.6% of real new packages, far above the target.

Bar chart: false-alarm rates on fresh npm uploads at a threshold tuned for 1%; LightGBM with all features reaches 16.6% — False alarms on real uploads, at thresholds tuned for 1%.

Why

The model had learned a shortcut. 99.8% of the benchmark's benign packages ship a README, against 21% of its malware and 85% of fresh uploads, so it learned "no README" as a sign of malware and flagged 79% of real packages without one. Random forest and logistic regression learn the same shortcut.The model had learned a shortcut. In the test data, 99.8% of harmless packages include a README (a description file), against 21% of the malware and 85% of real new packages. So it treated a missing README as a sign of malware, and flagged 79% of real packages without one. Two other kinds of model learned the same shortcut.

To check it was the cause and not just a correlate, I marked the 773 uploads without a README as having one and rescored them: false alarms fell from 16.6% to 5.5%.To check this was the cause and not a coincidence, I marked the 773 packages without a README as having one and scored them again: false alarms fell from 16.6% to 5.5%.The 773 uploads without a README, marked as having one and scored again by the same model. README

Bar chart: the full LightGBM model flags 79.2% of uploads without a README, against about 5% for the other groups — The README shortcut: 773 of 5,000 fresh uploads have no README.

The fix

I retrained with about 4,000 random fresh uploads as extra benign examples and set the threshold on uploads. Tested on a second, later sample of 5,000 uploads, the model catches 81% of 2025–26 malware at 0.7% false alarms, against 72% when only the threshold is raised. These are means of five training runs.I retrained it with about 4,000 real new packages as extra examples of harmless code, and set its alarm level on real packages. On a second, later sample of 5,000 packages, it catches 81% of 2025–26 malware with 7 false alarms per 1,000 packages, against 72% when only the alarm level is changed. These are averages of five training runs.

Limits

Hijacked releases of real libraries stay hard: the retrained model catches 36% of them, and a behaviour-only model 53%. The behaviour-only model, without package metadata, is the fallback: 2.8% false alarms, catching 75% of new malware.Malware slipped into real, trusted libraries is still hard to catch: the retrained model catches 36% of it, and a model that only looks at what the code does catches 53%. That second model is the fallback: 28 false alarms per 1,000 packages, catching 75% of new malware.

Every step that touches live malware runs in a locked-down Docker sandbox.Every step that touches real malware runs in a sealed-off container.

Stack

PythonLightGBM

Docker sandboxMinHash LSH

← All work Read the code on GitHub