Weak supervision for knee MRI: 58 gold-labelled studies, 4,407 reports
An entry to the 2026 RSNA Knee Abnormality Detection competition on Kaggle, now closed: radiology reports taught an EfficientNetV2 and DINOv2 ensemble to read knee MRI from the images alone. It reached 0.941 on the public leaderboard by itself.
- Reportsevery training study has a free-text radiology report
- Labelsonly 58 of 4,407 studies carry the 12 gold labels, so labels are read from the reports
- Image modela 2.5D EfficientNetV2 and DINOv2 ensemble, 12 outputs; evolved from a timm and BiGRU baseline
- Image-only inferenceno report at test time
- Result0.941 public on its own; 1,849th of 6,425 with a public-notebook blend
The underlying scans are patient data and cannot be shown, so this page shows aggregate results and model outputs only. The competition has closed; the standing is reported below with its provenance.
The hypothesis
The 2026 RSNA Knee Abnormality Detection competition was a research code competition on Kaggle: twelve clinically important knee abnormalities, predicted per MRI study, scored by the ROC AUC averaged over the twelve. It closed on 22 October 2026 with 6,425 teams.
My entry rested on a hypothesis about labels. Only 58 of the 4,407 training studies carry the 12 gold labels, far too few to train an image model on directly. But every training study comes with a free-text radiology report. If the reports can be turned into labels, the image model can learn from thousands of studies instead of dozens, and still run on the images alone when it is scored. The 58 gold studies are then worth more as a test than as training data.
The result showed the hypothesis was half right. The report is a usable teacher: on report-derived labels my ensemble reached 0.941 on the public leaderboard, ahead of every public single model of its day. It is also a biased teacher. On the 58 gold studies the report labels agree with the image-verified truth on 97% of ACL cells but only 69% of synovitis and 79% of effusion and contusion cells. The test set is graded by readers of the images, so a model trained on reports inherits what the reports leave out, and averaging more such models removes variance but never that bias. The winning entry built its labels by having language models read the images as well as the reports, and finished a full point ahead on the same data with a smaller model.
What I did
- Preprocessed about 500 GB of raw DICOM in shards on Kaggle, writing windowed, rescaled uint8 arrays at 384 pixels, and synced the result to a GPU cluster.
- Laid the pipeline out as four stages: report, labels, image model, image-only inference.
- Built a report-to-label extractor over reports in nine languages and validated it on the 58 gold studies: macro AUC 0.887 against image-verified truth, with the per-label agreement shown below.
- Built a first baseline over 2.5D slice stacks: a timm slice encoder feeding a BiGRU, then attention pooling, series aggregation and 12 logits, trained with DistributedDataParallel on two Quadro GV100 GPUs.
- Replaced it with a 2.5D multiple-instance ensemble of EfficientNetV2-S and DINOv2-S: five patient-grouped folds, best and EMA checkpoints, 28 to 38 members merged by percentile rank per label with weights fitted on out-of-fold predictions; trained on 3× A100 80GB over more than 300 GPU-hours.
- Audited the pipeline for silent defects and fixed what the audit found: no augmentation in the dataset, ViT members training in fp16 by default, and a flip test-time augmentation that cost 0.004 because the model never saw mirrored knees in training.
- Measured every lever against a frozen validation and the gold holdout with paired bootstraps, and dropped the ones that did not hold: a DINO rebuild, label surgery, per-head attention, a budget rebalance.
- Shipped 37 scored submissions through a fail-closed inference notebook that runs the ensemble on two T4 GPUs inside Kaggle’s nine-hour limit, with a budget ladder that drops members when the clock runs short.
- In the final week, when a public notebook reached 0.958, forked it, pinned its dependencies, and blended my ensemble into it as a separate stage on both GPUs. That blend is where the final standing comes from; my own models’ scores are reported separately.
Decisions that mattered
- The report is the teacher
- With 58 gold-labelled studies the supervised route is too thin to trust. Every study has a report, so the report supplies the training signal and the image model is the student. At inference the report is gone: the model that is scored sees the MRI and nothing else.
- Preprocess once, in shards
- About 500 GB of raw DICOM is windowed and rescaled to uint8 arrays at 384 pixels, shard by shard on Kaggle, and the result is synced to the GPU cluster. The training loop reads the prepared arrays.
- Start with a baseline, then replace it
- The first model was a timm slice encoder with a BiGRU reading the slices in order. It proved the data path and the per-label logging. The approach then evolved into an ensemble of two image encoders, EfficientNetV2 and DINOv2, still over 2.5D slice stacks.
- Split by patient, before anything else
- GroupKFold at patient level, the same discipline as in my MSc work on prostate MRI. A split that lets one patient sit on both sides makes every later number unreliable, so it was settled first.
- Look at every label, study by study
- There are 12 labels and very few gold examples of each. A single averaged score would hide a label the model has not learned, so each label is tracked on its own and the readout shows whole studies, misses included.
- Measure the teacher, not only the student
- In September I scored the report labels themselves against the 58 gold studies, cell by cell. That audit is the most important number on this page: it showed where the training signal was wrong before any model could, and it predicted the private leaderboard better than the public leaderboard did. What I did not do was act on it by making better labels; the winning entry did.
- Treat a public notebook as a floor, not a plan
- When the public frontier reached 0.958 in the final week, I forked it rather than ignore it, kept every blend fail-closed so a broken stage could never ship a broken file, and wrote down before the private board what a copy could and could not do. The private board agreed: every entry built on that notebook, 1,633 of them, lost 0.016 as one block.
- 2.5D stacksneighbouring slices grouped from the 384 px uint8 shards
- Slice encodera timm backbone encodes each stack
- BiGRUreads the encoded slices in order, in both directions
- Attention poolingweighs the slices within a series
- Series aggregationcombines the series of one study
- 12 logitsone per gold label, trained with binary cross-entropy
What the model sees in a knee
A 2.5D EfficientNetV2 and DINOv2 ensemble, read against the 46 image-verified gold studies held out of training (macro AUC 0.904). For each study the ensemble gives a confidence for each of the twelve abnormalities; at or above 0.50 it calls the abnormality present. Three studies are set out below against the radiologist’s ground truth.
Case A. Advanced degenerative knee
11 of 12 labels correct at the 0.50 threshold
Case B. Mixed presentation, a hard read
9 of 12 labels correct at the 0.50 threshold
Case C. Near-normal, isolated meniscus tear
12 of 12 labels correct at the 0.50 threshold
- Across all 58 gold studies
- The ensemble calls 83.5% of label cells correctly at 0.50. It slips most on synovitis and effusion.
- Compute
- 3× A100 80GB, more than 300 GPU-hours.
- Standing
- The competition is still running. This is held-out validation on my side: no leaderboard position, no medal.
Results
Final standing: 1,849th of 6,425 teams on the private leaderboard (preliminary), private score 0.942, public 0.958. The standing came from a blend built on the strongest public notebook; my own ensemble reached 0.941 on the public leaderboard on its own, and 0.943 blended with the public notebook of its day.
| Date | Submission | Public (30%) | Private (70%) |
|---|---|---|---|
| 24 Aug | Prior-only baseline, pipeline check | 0.500 | |
| 25 Aug | Fold-0 EfficientNetV2, half-trained | 0.818 | |
| 27 Aug | Fold-0 EfficientNetV2 through the fixed inference pipeline | 0.861 | |
| 28 Aug | Public floor notebook, forked byte-exact as a calibration | 0.936 | |
| 28 to 30 Aug | My 4-fold arm blended 50/50 with the public floor | 0.938 to 0.940 | |
| 1 Sept | My ensemble alone, 28 members | 0.941 | |
| 1 to 7 Sept | My 28- to 38-member arm blended with the public floor | 0.943 | |
| 23 Sept | The same blend at 0.60 public, 0.40 mine | 0.939 | 0.921 |
| 20 Oct | Copy of the public 0.958 notebook | 0.958 | 0.942 |
| 21 Oct | 0.958 notebook with my arm blended in at 0.20 on all labels | 0.957 | 0.941 |
| 22 Oct | 0.958 notebook with my arm on three labels at 0.20 (selected) | 0.958 | 0.942 |
| 22 Oct | 0.958 notebook with my arm on five labels at 0.30 (selected) | 0.958 | 0.942 |
Private scores are shown where they were read from the submissions page after the close. On the private 70% the whole field moved down: first place from 0.970 to 0.960, the gold line (rank 23) from 0.965 to 0.954, the 1,633 copies of the public notebook from 0.958 to 0.942, and my own blend from 0.939 to 0.921.
Per-label readout on the gold studies
The ensemble, read label by label against the image-verified gold studies held out of training (46 of the 58; the other 12 sat inside training folds). Macro AUC 0.904.
| Abnormality | Ensemble AUC on the gold holdout |
|---|---|
| ACL | 0.948 |
| MCL | 0.828 |
| Medial meniscus | 0.966 |
| Lateral meniscus | 0.817 |
| Medial OA | 0.982 |
| Lateral OA | 0.845 |
| Patellofemoral OA | 0.879 |
| Effusion | 0.949 |
| Synovitis | 0.792 |
| Baker’s cyst | 0.997 |
| Contusion | 0.929 |
| Fracture | 0.912 |
| Macro average | 0.904 |
How far the reports agree with the images. The report-derived training labels, scored cell by cell against the 58 gold studies: 86.5% of the 696 cells agree. ACL agrees on 96.6%, the menisci on 86 to 90%, contusion and effusion on 79%, synovitis on 69%. Fluid and inflammation are visible on MRI but under-dictated in reports, which is where the training signal was weakest.
Where the pixels out-know the reports. On the same gold studies, the ensemble’s own predictions beat the report labels at predicting image truth on effusion (+0.113 AUC), synovitis (+0.098), contusion (+0.080), fracture (+0.080), lateral OA (+0.066) and Baker’s cyst (+0.059); the reports win on MCL (+0.168 for the reports), patellofemoral OA (+0.091), lateral meniscus (+0.065) and ACL (+0.012). A model trained on the reports had learned to see things its own labels missed.
Evidence
The competition has closed. The standing is stated with its provenance, and my own models' scores are given separately from the blend that produced it.
| Part | Final state, October 2026 | Context |
|---|---|---|
| Data engineering | done | about 500 GB of raw DICOM, preprocessed in shards on Kaggle to uint8 at 384 px, synced to a GPU cluster |
| Report-to-label extractor | done; macro AUC 0.887 on the gold studies | reports in nine languages; 86.5% of label cells agree with image truth |
| Baseline model | built, superseded | timm slice encoder, BiGRU, attention pooling, series aggregation, 12 logits; two Quadro GV100 GPUs |
| Current model | trained | 2.5D EfficientNetV2 and DINOv2 ensemble, 28 to 38 members; 3× A100 80GB, more than 300 GPU-hours |
| Held-out validation | macro AUC 0.904; 83.5% of label cells correct at 0.50 | gold studies held out of training; weakest on synovitis, lateral meniscus and MCL |
| Public leaderboard, own ensemble | 0.941 alone, 0.943 blended with the public floor | 1 September 2026; ahead of every public single model of its day |
| Final standing | 1,849th of 6,425; private 0.942, public 0.958 | a blend on the public 0.958 notebook; preliminary pending Kaggle’s verification |
| Medal | none | the bronze line on the private board sat above 0.942 |
Limitations and status
Fifty-eight studies is a small test set, and the 46 held out of training are smaller still: per-label AUCs on that many cases carry uncertainties of several hundredths, and the three cases in the readout are illustrations, not a sample. Label-cell accuracy at a fixed threshold is a blunt measure. Most labels are absent in most knees, so a model earns a good share of its cells by saying no, and 83.5% should be read with that in mind.
The final standing has a provenance that should be stated plainly. The private score of 0.942 belongs to a blend built on a public notebook that 1,633 other teams also submitted; my ensemble added to it changed nothing on the private board. My own models’ scores are 0.941 on the public leaderboard alone and 0.921 private in a blend with the public notebook of September. The private leaderboard is preliminary until Kaggle verifies the top entries.
Two of the five final-day submissions timed out on Kaggle’s nine-hour rerun and one threw an exception: the rerun hosts vary several-fold in data-mount speed, and my weights copy had no per-file timeout. None of the three would have scored differently from the two that finished.
The training repository is private. The scans are patient data and are not shown; this page shows aggregate results and model outputs only.
What I learned
In a weak-label competition, the labels are the model.
The question I set out with, how far a label read from a report can be believed, turned out to be the whole competition. My extractor’s labels scored 0.887 on the gold studies; the winning team’s, made by language models reading the report and the images together, scored 0.936 to 0.943. On the same data, with a smaller network, that difference was worth a full point on the private board, and nothing I did on the model side moved it by more than 0.002.
Measure the teacher first. The September audit of my labels against image truth was the most predictive number of the campaign: it said synovitis, effusion and contusion would be weak, and they were. The right response to that audit was better labels, not better models. I had the diagnosis in week three and spent the next six weeks on ensembling.
A leaderboard printed to three decimals cannot see a lever worth less than 0.005. Five changes that were positive on validation were null on the board. The only honest instrument was the gold holdout.
A copied public notebook is a floor. It was tuned to the 30% public slice and fell 0.016 on the private 70%, while the leaders fell 0.010. Everything built on it fell with it.
Compute follows validation, not the other way round. More than 300 A100-hours bought 0.002 on the board. The winner’s single model trained in hours, on labels that cost model-reading time instead of GPU time.
The earlier lessons stand: the data engineering has to come first, the patient-level split has to be settled before anything is measured, and a model is easier to judge one study at a time than through one averaged number.
What the winning entry did
Team Kangaroo (tascj and liushuzhi) finished first with 0.958 on the private board and published a write-up and their code. They treated it as a labelling competition: one non-interactive language-model session per training study, with tools to view the slices, reading the report and the images together and writing an evidence-cited probability for each of the twelve findings; a reading guide refined on the 58 gold studies; and labels judged by log loss and confident errors as well as AUC. They added 3D region boxes per finding on 751 studies and per-slice lesion boxes on 1,800, and trained a single-slice ConvNeXt-T per series that pools its features inside the predicted region. One such model scored 0.954 on the private board; an ensemble of five scored 0.958. Their labels and training notebooks are public.
Competition and profile
The competition page on Kaggle (opens in a new tab) describes the task and the data. My Kaggle profile (opens in a new tab) is public. The winning team’s 1st-place write-up (opens in a new tab) is public too. The training repository is private.
Inspect
- Competition on Kaggle (opens in a new tab)
- Kaggle profile (opens in a new tab)
- 1st-place write-up (opens in a new tab)
External links open in a new tab.