COHE
Menu

When does a sample difficulty proxy actually help?

COHE: Auditing Non-Transitivity in Sample Difficulty Proxies for Vision Models

NeurIPS 2026 · Evaluations & Datasets

Dayena Jeong1 and Sunglok Choi2

Seoul National University of Science and Technology

Affiliations

1 Department of Defense Artificial Intelligence Applications

2 Department of Computer Science and Engineering

TL;DR: DINOv2 isolation predicts when samples are first learned, yet training on its hardest-ranked samples reduces accuracy in the tested settings.

Conference record

The study

COHE compares reconstruction error, denoising error, token likelihood and DINOv2 isolation with supervised difficulty targets. Experiments measure rank agreement and held-out prediction, then test subset selection, full-data sample ordering and retrieval against matched baselines.

Read the full abstract in the paper

Dependence, prediction and transfer

  1. G1 — Dependence

    Do the rankings agree?

    Compare the proxy with the target on matched samples using Pearson r and Spearman ρ.

  2. G2 — Prediction

    Can it predict unseen samples?

    Fit on the training split and measure R² on held-out target values.

  3. G3 — Transfer

    Does using the score help?

    Compare the declared policy with a matched baseline, including uncertainty in the downstream metric.

The paper calls this non-transitivity: ranking agreement, held-out prediction and downstream benefit do not automatically imply one another.

COHE checks a proxy, target and protocol for dependence, held-out prediction and operational transfer, then reports one of the six outcomes defined below.
Audit flow with the paper’s six reporting outcomes. View at full size

Six reporting outcomes

The paper groups audit reports into six tiers for the stated proxy, target and protocol.

Validated surrogate
All three gates pass for the stated target and protocol.
Predictive validity only
G1 and G2 pass for the target, without a G3 pass.
Target-specific relation
Relational evidence is confined to an auxiliary target, such as first-learning or GradNorm.
Operational-only effect
G3 passes without both G1 and G2 passing for the same setting.
Weak evidence
Support is incomplete or inconclusive, including agreement at G1 alone.
No reliable transfer
G2 is measured weak or G3 has no reliable benefit, without matched positive G1 and G2 evidence.

An unmeasured gate is not a failure. A failed G3 preserves positive G1/G2 evidence; auxiliary-target relations retain their own label.

Definitions and assignment order

Experimental results

Selection, ordering and retrieval test different uses of a difficulty score.

In the DINOv2 and DDPM difference plots, negative values favor the baseline. Whiskers are paired Student-t 95% confidence intervals.

DINOv2 hard-sample selection

CIFAR-100, DINOv2 isolation

Hard-only selection underperforms Random in every tested condition.

For first-learning, rank correlation is ρ = 0.4363and held-out prediction is R² = 0.2126. This auxiliary-target relation measures when samples are learned; the selection experiment below measures test accuracy.

ResNet-18, 4 conditions with 5 paired seeds each. Seed IDs are reused across conditions. Balanced selection uses balanced Random.

Seed records
Mean difference with 95% paired CITop-1, percentage points
10% subset−14.87 pp
[−15.62, −14.12]
30% subset−17.25 pp
[−17.99, −16.52]
50% subset−11.12 pp
[−11.54, −10.70]
Balanced 10%−11.40 pp
[−12.91, −9.89]
Mean accuracy and matched baselines
DINOv2, Top-1 accuracy (%)
SelectionHardRandom
10% subset10.7725.64
30% subset26.4743.72
50% subset40.5751.68
Balanced 10%14.1125.51

DDPM sample ordering

CIFAR-100, DDPM difficulty scores

Easy-to-Hard and Shuffled-Proxy Order both fall below global Random ordering.

ResNet-18, 8 paired seeds. Both blocked schedules use the same full data and optimizer step count.

Seed records
Mean difference with 95% paired CITop-1, percentage points
Easy-to-Hard−0.77 pp
[−1.21, −0.33]
Shuffled-Proxy Order−0.50 pp
[−0.86, −0.14]
Easy − Shuffled−0.27 pp95% CI [−0.82, 0.28]; paired p = 0.2896

ImageNet-1K subset selection

ImageNet-1K, VAE reconstruction scores

Class-balanced VAE-hard selection underperforms balanced Random.

ResNet-18 from scratch, 5 matched seed pairs. Both arms use the same best-validation-Top-1 checkpoint rule.

Seed records
Best-checkpoint Top-1Accuracy (%)
Proxy-Hard53.646%
Random61.412%
03570%
Mean accuracy across 5 seeds. The interval below compares the paired difference.
Hard − Random−7.765 pp95% CI [−8.170, −7.360]
Endpoint and evaluation details

VAE: stabilityai/sd-vae-ft-mse normalized-pixel reconstruction MSE. The largest scores are selected within each training class. Validation data is used for checkpoint selection and evaluation.

Final-epoch sensitivity: −7.854 pp, 95% CI [−8.442, −7.266].

Held-out predictive validity

Held-out R-squared for single and multi-proxy predictors. CE and margin are close to zero; first-learning has a larger value.
Prediction from generative proxy scores. Error bars show 95% confidence intervals over held-out splits.

Run COHE on your CSVs

From the artifact repository root, provide your scalar scores, a fixed prediction split and completed policy results. These interface commands produce statistics; they do not decide whether a gate passes or assign a reporting tier. Match scores by unique sample_id; the scripts use the IDs shared by the input files. Define both scores so larger values mean harder samples, and record the target, learner and policy being audited.

CSV files for the commands below
FileColumnsContents
my_proxy.csvsample_id,proxy_scoreOne scalar proxy score per sample.
my_target.csvsample_id,target_scoreOne scalar value of the declared target per sample.
my_splits.csvsample_id,splitFixed train or test assignment; at least two training samples and a nonconstant held-out target.
my_transfer.csvmethod,budget,seed,accuracyCompleted runs, with a Random baseline at each budget and accuracy in the same units for every method.
# G1: Pearson r and Spearman rho
python scripts/run_stage1_dependence.py \
  --proxy my_proxy.csv --target my_target.csv \
  --out outputs/my_g1.csv

# G2: held-out R² from a one-dimensional linear fit
python scripts/run_stage2_prediction.py \
  --proxy my_proxy.csv --target my_target.csv \
  --splits my_splits.csv --out outputs/my_g2.csv

# G3 summary only, no pass/fail: mean differences versus Random
python scripts/run_stage3_transfer_summary.py \
  --transfer my_transfer.csv --out outputs/my_g3_summary.csv

G3 summary only, no pass/fail. The script averages supplied runs by method and budget without pairing seeds or estimating confidence intervals. Its output cannot establish a G3 pass. G2 likewise reports a single linear fit; split variability and the declared prediction criteria require separate evaluation.

For the paper’s accuracy comparisons, pair policy and baseline runs by seed and compute a 95% confidence interval for the paired differences. The lower bound must be above zero, together with the declared endpoint and protocol checks, before reporting a G3 pass. A positive mean difference alone is insufficient.

Assign the reporting outcome

After evaluating uncertainty and the predeclared criteria, record each gate as passed, failed, inconclusive or unmeasured for the same proxy, target and protocol. Then apply the paper’s assignment order manually:

  1. When all three gates pass, report Validated surrogate.
  2. When G3 passes without both G1 and G2 passing, report Operational-only effect.
  3. For remaining relational evidence, use Target-specific relation if it is confined to an auxiliary target; otherwise, positive G1 and G2 support Predictive validity only.
  4. Without matched positive G1/G2 evidence, measured weak G2 or an evaluated G3 without reliable benefit supports No reliable transfer. Use Weak evidence for remaining incomplete or inconclusive cases.

Keep unmeasured gates distinct from failures, and report the audited setting and uncertainty with the assigned outcome.

Runnable example CSVsReporting guide

Reproducibility

The v1.0-neurips2026 release includes code, seven CIFAR-100 proxy arrays, aggregate and seed-level records, and 234 author-trained checkpoints.

Reproduce the paper

From the artifact repository root, verify all released DINOv2 conditions, DDPM ordering pairs and ImageNet-1K endpoints against completed experiment records:

python scripts/reproduce.py --experiment paper_results
Result verification script
Resources available for each experiment
ExperimentCodeAggregatesSeedsCheckpointsNotes
CIFAR-100, ResNet-18TrainingAvailableAvailable120Dataset and score inputs required.
CIFAR-100, ResNet-50TrainingAvailableAvailableNot retainedSecond-learner results remain available.
DINOv2 selectionSelectionAvailableAvailableNot retainedScalar ranks and all reported conditions included.
DDPM orderingOrderingAvailableAvailable18 of 24Random and Easy-to-Hard seeds 0–2 are missing.
ImageNet-10Scoring / fine-tuningAvailableAvailableNot retainedImagenette and score inputs required.
ImageNet-1K VAE selectionScoring / trainingAvailableAvailable20ImageNet, VAE weights and training scores required.

CPU commands check released records. Regenerating scores and training models require upstream datasets, weights and experiment inputs; some historical source settings remain incomplete.

Reproduction guideCheckpoint inventory

Citation

@inproceedings{jeong2026cohe,
  title = {COHE: Auditing Non-Transitivity in Sample Difficulty Proxies for Vision Models},
  author = {Jeong, Dayena and Choi, Sunglok},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2026},
  note = {Evaluations \& Datasets Track},
  url = {https://neurips.cc/virtual/2026/poster/139659}
}