Reza Borhani, Yaeli Cohen, Onyi Lam, Hareem Naveed,
Kevin H. Wilson, Chad Kenney, Rayid Ghani
tl;dr: The authors created models that predicted whether students in the Milwaukee
Public School systems would interact with the criminal justice system. These models
could help facilitate early-intervention programs to prevent future interactions
with the criminal justice system.
The Problem
- Students that have significant interactions with the juvenile justice
system often have difficulties reintegrating back into society. Reintegration
struggles are correlated with a host of other issues, ranging from decreased
likelihood of graduation to higher mortality rates.
- In 2015, Milwaukee Public Schools had a graduation rate of 58%, compared to
the state-wide graduation rate of 88%. Meanwhile, juvenile arrest rates
in Milwaukee have increased 163% between 2011 and 2015, in contrast to
a steady decrease nationally.
- Milwaukee Public Schools (MPS) uses targeted interventions in an effort to prevent
future interactions with the criminal justice system. Their current rules-based
approach flags roughly 22,000 students which far exceeds their capacity
to intervene with only 5,000 students per year.
- This leads to the the task: can MPS improve on their ability to target
"at-risk" students by using a machine learning model to predict which
students are most likely to have future interactions with the criminal
justice system?
The Approach
- The authors considered two datasets.
- The first was data from the Milwaukee Public School (MPS) system
on enrolled students from 2004 to 2015. The features in the MPS
dataset included demographics, attendance records, disciplinary
events, test assessment data, and school programs that the student
was enrolled in.
- The second dataset, from the Milwaukee District Attorney's (MDA)
office, consisted of juvenile interactions with the criminal justice
system from 2009 to 2015. Importantly, this dataset only includes
offenses that are referred to the DA's office, which is only a
subset of all interactions with the criminal justice system.
The authors note that this implies that only serious crimes are
included in the dataset. The features include demographic
information as well as details on the offense.
- A great deal of effort went into cleaning the data and matching
individuals from each dataset. Ultimately, the authors ended up with
9,451 unique individuals in the MDA dataset, and linked 86% of them with
MPS records.
- The authors used standard approaches, ranging from logistic regression,
random forests, AdaBoost, etc. The feature generation was
straightforward, with a focus on demographic characteristics, history of
abuse, and incidents of truancy.
- Recall that the goal of the study was to aid MPS in selecting students
with the highest risk, as their current approach flagged too many students.
Thus, the authors prioritized precision in the top 1%, i.e., ensuring the
model is as accurate as possible in the 1% of students most likely to interact
with the criminal justice system. The authors also prioritized stability
of the algorithm over time (i.e., if it works in 2010, it should work in 2015).
Results and Analysis
- The best performing model was a Random Forest. For the top 1% of predicted risk scores, they obtained a precision of 0.3 and recall of 0.1. This implies that, in the top 1% of risk scores predicted by the model, 30% are students who actually ended up interacting with the justice system (precision), and these students constitute 10% of all students who interacted with the justice system (recall).
- A direct comparison of their model to the benchmark of the rules-based approach that MPS uses reveals a large reduction in false positives:
|
Flags |
Correctly Identifies |
Heuristic Model (MPS) |
22,000 |
1,310 |
Their Model |
12,000 |
1,630 |
It's unclear at what percentile the authors thresholded their risk scores for this comparison.
- The most important features in their model included (i) the number of "child in need of protective services" records, (ii) age, (iii) number of discipline incidents in the last 2 years, and (iv) the average number of absence days over the years. It is unclear how the authors determined these features were the most important.
Contextualizing the Study
The authors use of a random forest greatly exceeds MPS’s heuristic model at
predicting future interactions with the criminal
justice system. In particular, the reduction of false positives is
promising, as it would lead to a decrease in needless interventions. Presumably,
interventions were occurring in the time frames considered in the dataset. It would
be useful to extend the model such that it incorporates these interventions.
It is important to note that consideration of interactions with the criminal justice
system cannot occur without contextualizing those interactions with the systemic
and historical racial disparities exhibited by the criminal justice system. These
racial disparities extend beyond criminal justice, such as in
disciplinary actions within K-12 public schools
[1].
Riddle & Sinclair, PNAS, 2019.
The dataset
that was used to train this model will be impacted by the very biases that lead to
the racial disparities in the first place.
Therefore, an analysis on how the algorithm treats people of color specifically
is imperative before depoyment. While the authors state that race is not one of
the most important features used, there is an abundance of work demonstrating
that protected variables can still impact machine learning algorithms, even if
they don’t do so directly
[2].
Hardt, Price, & Srebro, NeurIPS, 2016.
Additionally, an assumption built into targeted interventions is that
the students must be changed rather than the criminal justice system
or the education system itself. This is not to say that targeted interventions
are not capable of preventing future interactions with the criminal justice system.
Rather, the impact of targeted interventions must be
assessed. First, do they in fact decrease future interactions with the criminal
justice system, and second, are they the most cost effective way of reducing
the interactions?