Lodestar PAC
← Research
Paper 02 · Machine learning

Training, testing, and improving the taxonomy model.

How Lodestar predicts what an asteroid is made of, how we keep it honest, and what we did to make it materially better.
scroll ↓
01 — The training set

Learning from the few we have measured.

Of 41,884 near-Earth asteroids, only 116 carry both a measured class and an albedo, the pairing the model learns from. And they are lopsided: 83 stony, 29 carbonaceous, just 4 metallic. Find the four bright sparks. That scarcity decides what the model can and cannot learn.

02 — The model

Reflectivity does most of the work.

A class-balanced random forest reads albedo and orbital dynamics. The single strongest signal is albedo: dark carbonaceous rock reflects a few percent of sunlight, bright stony rock far more. Lay the asteroids out by brightness and the classes mostly fall into place on their own.

03 — Testing without cheating

Never scored by a model that has seen it.

Every accuracy number is leave-one-out: pull one asteroid out, train on the other 115, predict the one held out, and repeat 116 times. Nothing is ever graded by a model that learned from it. Under that test, the forest reaches 78% with infrared added, well past the 72% baseline.

04 — Honest confidence

It knows when not to answer.

The model attaches a calibrated confidence to every call, and the wrong answers cluster at low confidence. Keep only the confident ones and the mistakes (the red) fall away first, lifting accuracy from 78% toward 89%. Honest coverage beats confident error.

05 — Data-limited, not model-limited

The curve is still climbing.

Feed the model more training examples and accuracy keeps rising, right out to the largest set we have. A model that had run out of room would flatten. Ours is still going up at the edge, which means the bottleneck is labelled data, not the algorithm.

06 — The infrared upgrade shipped

We added infrared. It pulled the classes apart.

We folded in NEOWISE near-infrared albedo and SDSS colour. With a second dimension the model can see, carbonaceous and stony separate further and accuracy rose 72% → 78%. The four metallic asteroids still sit stuck in the middle. That is not a feature problem anymore. It is four examples. The next paper goes after the labels.

albedo only + infrared + colour
07 — The roadmap

What we did, and what is left.

Add infrared + colour · shipped, +6 pts

NEOWISE near-infrared albedo and SDSS colour, cross-matched and retrained, lifted leave-one-out accuracy 72% → 78% and improved carbonaceous recall. Done.

Grow the metallic labels · next

Metallic recall is still 0% because there are only 4 examples. The fix is more measured spectra (SMASS, Bus-DeMeo, MITHNEOS, modern taxonomies), not more features.

Ship calibration + abstention

Calibration already halves the error and abstention lifts accuracy to 89%. Wire both into the live model and expose an "indeterminate" state below a confidence threshold.

Conformal prediction sets

Return an honest set ("C or S, 90% covered") instead of one forced label, once probabilities are calibrated.

Honest by construction, and getting better on the evidence.

Every figure carries its provenance and its uncertainty, no object is ever graded by a model that trained on it, and where the model is weak we show it. The infrared upgrade is the first proof that the roadmap works: name the lever, pull it, measure the gain.