Jev vs. traditional classifiers

Accuracy: all correct labels. Sensitivity: positives caught. Specificity: negatives correctly rejected. AUC: score ranking across cutoffs, not accuracy or calibration.

“Same 16 examples” matches task examples, not prior knowledge. “Full” gives classical models more training rows. Tuned Jev also uses validation information; classical thresholds were not tuned.

On Breast Cancer, validation selected a Choice cutoff of 0.95. False alarms changed from 20 to 3, and missed positives from 0 to 1. This is an educational dataset, not clinical validation.

Cutoffs were chosen on validation data, not test data. The displayed default is the highest recorded test accuracy among the four Jev configurations, with validation-tuned Choice preferred for ties. That selection is descriptive, not an unbiased estimate of a selection procedure.

Scikit-learn bundled datasets; binary targets as described above. Stratified splits: 25% test, then 25% of the remainder for validation, seed 18. Eight training examples per class, sampled with NumPy’s seeded generator. The 16-example classical variants fit those same rows; “full” fits the training pool only, not validation.

Jev received named numeric features rounded to four decimals, semantic class names, labeled examples, and both Noul and Choice questions in the same request. Classical models used the original numeric values. Requested model: jev-latest. The saved cutoff-selection responses report jev-1.13.0; per-row resolved model versions were not stored, so the exact version of every prediction cannot be independently confirmed.

Gradient boosting used scikit-learn defaults and seed 18. Extra Trees used 200 trees and seed 18. Logistic regression used training-fitted StandardScaler and max_iter=5000. No classical hyperparameter search or classical threshold tuning was run.

Choice cutoffs were selected on validation data from 0.05, 0.10, 0.20…0.90, 0.95. Code maximized validation accuracy, then sensitivity, then proximity to 0.50. Iris: 0.7; Wine: 0.9; Breast Cancer: 0.95. Noul and untuned Choice used 0.50. Threshold selection gives tuned Jev configurations extra validation information; they are not strictly equal-budget comparisons to the untuned baselines.

Accuracy, sensitivity, specificity and AUC are saved to four decimal places. Matrix counts are exact. No repeated-split uncertainty analysis, end-to-end latency measurement, or cost accounting is included. No clinical conclusions should be drawn.

A stronger next benchmark

Use fresh task-specific data, repeated stratified splits, fixed model versions, a majority-class baseline, matched tuning budgets, and a final untouched test set. Measure error costs, calibration, request latency including retries, and cost per successful decision. Repeat the test when the model or prompt changes.

Further reading: Dataset documentation · Probability calibration · Evaluation pitfalls · Jev question types

Small test sets, one random split, possible pretraining exposure, and unequal tuning budgets limit the conclusions. Choosing a model after inspecting these test results requires a fresh final test. Calibration, speed, cost, and statistical superiority have not been established.