Flutterby™! : COMPAS vs Mechanical Turk

Mechanical Turkers may have out-predicted the most popular crime- predicting algorithm.

The accuracy, fairness, and limits of predicting recidivism, Julia Dressel and Hany Farid:

Algorithms for predicting recidivism are commonly used to assess a criminal defendant’s likelihood of committing a crime. These predictions are used in pretrial, parole, and sentencing decisions. Proponents of these systems argue that big data and advanced machine learning make these analyses more accurate and less biased than humans. We shMechanical Turkers may have out-predicted the most popular crime-predicting algorithmow, however, that the widely used commercial risk assessment software COMPAS is no more accurate or fair than predictions made by people with little or no criminal justice expertise. We further show that a simple linear predictor provided with only two features is nearly equivalent to COMPAS with its 137 features.

The official Equivant response dodges the big question and focuses on the methodology that the researchers used to train the linear predictor, but I think gets it wrong, because the rebuttal complains that:

... The scale was likely developed and then tested on the same large data sample. This is often known as giving only a limited internal validation. This commonly leads to “over-fitting” that falsely heightens the apparent predictive accuracy. ...

But the paper notes that:

Each classifier was trained 1000 times on a random 80% training and 20% testing split; we report the average testing accuracy and bootstrapped 95% confidence intervals for these classifiers.</bloockquote>

The paper is Science Advances 17 Jan 2018: Vol. 4, no. 1, eaao5580 DOI: 10.1126/sciadv.aao5580