Research
Peer-Reviewed
- Predicting behavioral health-involved police encounters: A machine learning approach. (Forthcoming: Journal of Quantitative Criminology, 2025)
- With: Andrea Tentner, Xander Beberman, and Harold Pollack
- Abstract: Prevention or de-escalation of dangerous police encounters with community members experiencing behavioral health challenges are central public health and safety challenges. A complementary approach that leverages emergency response data to predict which individuals are at highest risk for these encounters may be an important tool in the implementation of targeted prevention and service interventions. In this study, we use emergency response data to show that behavioral health-indicated encounters with police can be reliably predicted. Both simple and complex modeling approaches are able to identify small populations at extreme risk, though a machine learning approach is more accurate, especially when targeting larger groups. And ML can also identify opportunities for prevention. Fully 28% of individuals identified in our top 1% risk-group had no prior BH-police-involved emergency encounters. Both approaches, when combined with rich administrative data may also aid in tailoring pertinent interventions.
- Machine Learning Can Predict Shooting Victimization Well Enough to Help Prevent It (Review of Economics and Statistics, 2024)
- Working Paper Version
- With: Sara Heller, Benjamin Jakubowski, and Max Kapustin
- Abstract: This paper shows that shootings are predictable enough to be preventable. Using arrest and victimization records for almost 644,000 people from the Chicago Police Department, we train a machine learning model to predict the risk of being shot in the next 18 months. We address central concerns about police data and algorithmic bias by predicting shooting victimization rather than arrest, which we show accurately captures risk differences across demographic groups despite bias in the predictors. Out-of-sample accuracy is strikingly high: of the 500 people with the highest predicted risk, 13 percent are shot within 18 months, a rate 130 times higher than the average Chicagoan. Although Black male victims more often have enough police contact to generate predictions, those predictions are not, on average, inflated; the demographic composition of predicted and actual shooting victims is almost identical. There are legal, ethical, and practical barriers to using these predictions to target law enforcement. But using them to target social services could have enormous preventive benefits.
- Political Language in Economics (Economic Journal, 2024)
- Working Paper Version
- With: Bruce Kogut and Suresh Naidu
- Abstract: Do empirical estimates in economics reflect the political orientation of economists? We show that policy-relevant parameters are correlated with economist partisanship as predicted from the text of published academic papers. Specifically, we predict observed political behavior of a subset of economists using the phrases from their academic articles, obtain good out-of-sample fit, and then predict partisanship for all economists. We show considerable sorting of economists into fields of research by predicted partisanship, and yet can detect differences in partisanship among economists even within a field, even across those estimating the same theoretical parameter. Using policy-relevant parameters collected from previous meta-analyses we then show that imputed partisanship is correlated with estimated parameters, such that the implied policy prescription is consistent with partisan leaning. For example, we find that going from the most left-wing authored estimate of the taxable top income elasticity to the most right-wing authored estimate decreases the optimal tax rate from 84% to 58%.
- No Ground Truth? No Problem: Improving Administrative Data Linking Using Active Learning and a Little Bit of Guile (PLOS ONE, 2023)
- With: Sarah Tahamont, Melissa McNeill, Shi Yan, Aaron Chalfin, and Benjamin Hansen
- Abstract: While linking records across large administrative datasets [“big data”] has the potential to revolutionize empirical social science research, many administrative data files do not have common identifiers and are thus not designed to be linked to others. To address this problem, researchers have developed probabilistic record linkage algorithms which use statistical patterns in identifying characteristics to perform linking tasks. Naturally, the accuracy of a candidate linking In this paper, we investigate the value of providing groundtruth examples via active learning for linking performance. We confirm popular intuition that data linking can be dramatically improved with the availability of ground truth examples. But critically, in many real-world applications, only a relatively small number of tactically-selected ground-truth examples are needed to obtain most of the achievable gains. With a modest investment in ground truth, researchers can approximate the performance of a supervised learning algorithm that has access to a large database of ground truth examples using a readily available off-the-shelf tool.
- Dude, Where’s My Treatment Effect? Errors in Administrative Data Linking and the Destruction of Statistical Power in Randomized Experiments (Journal of Quantitative Criminology, 2020)
- Working Paper Version
- With: Sarah Tahamont, Aaron Chalfin, Shi Yan, and Benjamin Hansen
- Abstract: The increasing availability of large administrative datasets has led to an exciting innovation in social policy research—using administrative data to measure experimental outcomes in lieu of costly primary data collection. We demonstrate that this type of randomized experiment can have an unfortunate consequence: the destruction of statistical power. Combining experimental data with administrative records to track outcomes of interest typically requires linking datasets without a common identifier. In order to minimize mistaken linkages, researchers often use stringent linking rules like “exact matching” to ensure that speculative matches do not lead to errors in an analytic dataset. We show that this, seemingly conservative, approach leads to underpowered experiments, leaves real treatment effects undetected, and can therefore have profound implications for entire experimental literatures.
- Detecting latent ideology in expert text: Evidence from academic papers in economics (EMNLP Proceedings, 2014)
- With: Bruce Kogut and Suresh Naidu
- Abstract: Previous work on extracting ideology from text has focused on domains where expression of political views is expected, but it’s unclear if current technology can work in domains where displays of ideology are considered inappropriate. We present a supervised ensemble n-gram model for ideology extraction with topic adjustments and apply it to one such domain: research papers written by academic economists. We show economists’ political leanings can be correctly predicted, that our predictions generalize to new domains, and that they correlate with public policy-relevant research findings. We also present evidence that unsupervised models can under-perform in domains where ideological expression is discouraged.
- Profiting from filesharing: A measurement study of economic incentives in cyberlockers (P2P Proceedings, 2012)
- With: Keith Ross
- Abstract: This paper explores the incentive schemes that have been used by cyberlockers, and examines to what extent they have helped to foster the current environment. This short paper also presents a preliminary economic analysis of incentives in cyberlockers based on measurement results from three different types of sites related to cyberlockers- a search engine, a discussion forum, and a popular cyberlocker site.
- Facebook users have become much more private: A large-scale study (PerCom Proceedings, 2012)
- With: Ratan Dey and Keith Ross
- Abstract: We investigate whether Facebook users have become more private by crawling the public profile pages of 1.4 million New York City (NYC) Facebook users in March 2010 and again in June 2011. We found that NYC users in our sample became dramatically more private during this period. For example, in March 2010 only 17.2% of users in our sample hid their friend lists, whereas in June 2011, just 15 months later, 52.6% of the users hid their friend lists. We explore privacy trends for several personal attributes including friend list, networks, relationship, high school name and graduation year, gender, and hometown.
Working Papers
- Crime and the City: Computational Approaches and the Future of Urban Crime Research (Revise and Resubmit at Nature)
- With: Gian Maria Campedelli, Bruno Lepri, Patrick Sharkey, Aaron Chalfin, Eric Piza, Ariadna Albors Zumel, and Daniel Semenza
- Abstract: Urban environments have long been a central focus for crime researchers across diverse disciplines. Over the past fifteen years, this heterogeneous field has undergone a profound transformation, driven by the emergence of novel datasets and the rise of powerful, flexible computational methods. In this Review, we take stock of this transition and explore the significant potential these developments hold for advancing urban crime research, while also addressing the persistent challenges that continue to shape the field. Building on this overview, we highlight the importance of integrating causal inference beyond mere prediction tasks and outline three key directions for future research to ensure that the promise of new data and computational tools is fully realized. In doing so, we aim to support the production of more rigorous research and the development of more effective policies for safer, more sustainable cities worldwide.
- Time is on your side: The benefits of time-sensitive factors in pretrial risk assessments
- With: Greg Stoddard, Melissa McNeill, and Alexandra Chouldechova
- Abstract: Risk assessments are used in millions of pretrial hearings per year to guide judges as they weigh public safety risks against the cost of restricting defendant liberties. We show that the time since a prior event matters for its relevance as a risk factor and, as a result, risk models can be significantly improved by using time-sensitive risk factors. Using data from New Jersey, we find that time-sensitive risk models are simultaneously more accurate and less racially disparate than standard time-insensitive risk models. For example, New Jersey’s violent risk predictor could be 5% more accurate and 20% less disparate if time-sensitive factors were used. These improvements generalize across six other pretrial risk scales. We present a simple time-sensitive modeling approach called “time-limiting” that is no more complex than time-insensitive risk models and can work in any practical setting. Our results suggest that considering the use of time-sensitive risk factors should be adopted as a best practice in risk assessment development and revalidation.
- Perils and Pitfalls in the Use of Synthetic Control Methods to Study Public Safety Interventions (Under review)
- With: Aaron Chalfin
- Abstract: The method of synthetic controls, pioneered by Abadie et al. (2010), has generated a paradigm shift in the analysis of case studies. The method selects an appropriate synthetic comparison group by identifying a weighted set of units that closely match the treated unit on the basis of pre-intervention levels and trends. Since Abadie’s seminal paper, there has been a proliferation of research expanding and refining the method and a corresponding litany of software packages that provide the means to estimate these models. We show that there can be a shocking lack of correspondence between the estimates produced by commonly used software packages. Even the seemingly innocent choice between using R or Stata to estimate SCM can lead to a meaningful difference in estimated treatment effects. We demonstrate this surprising finding, invoking a recent debate in criminological research concerning a paper on the effects of “de-prosecution” by Hogan (2022) which has been criticized by Kaplan et al. (2022).
- Synthetic Data for Systematic Evaluation: How Well Can LLMs Perform Deductive Coding of Economic Concepts?
- With: Zhuohui Chen
- Abstract: Identifying concepts, ideas, and topics in text is foundational for advancing knowledge, enabling researchers to trace the evolution of ideas, understand intellectual movements, and map conceptual relationships within and across disciplines. While Large Language Models (LLMs) offer unprecedented opportunities for automated concept detection at scale, evaluating their capabilities faces significant challenges due to scarce evaluation datasets and difficulties in aligning generated topics with ground truth. This study introduces a methodological framework using controlled synthetic data generation to evaluate LLMs’ deductive coding performance. We generate 2,880 economics abstracts using real-world topic mixtures, controlling for topic explicitness, stylistic diversity, and vocabulary restrictions. Through systematic experimentation across different codebook configurations and LLM architectures, we find that while recall is generally high, precision in coding is moderate. While apparent false positives initially suggest overprediction, semantic analysis reveals that they likely represent valid emergent topics. Our framework provides a replicable methodology for systematic LLM evaluation and highlights promising possibilities and future research avenues for evaluating LLMs’ abilities to trace ideas and concepts across diverse textual domains.
- The Impact of Targeted Interventions on High-Risk Domestic Violence Victims
- With: Ashna Arora, Xander Beberman, and Ashley Motta
- Abstract: Domestic violence (DV) accounts for 50% of female homicides in the U.S. The criminal justice system – with which the majority of DV homicide victims initiate contact in the years leading up to their deaths – may be uniquely suited to intervene and prevent the escalation of this violence. Identifying these victims, however, remains a needle in the haystack problem, as the vast majority of these victims will not go on to be homicide victims. Targeted approaches – which attempt to identify high-risk victims and intervene effectively – offer a solution, but have not been rigorously evaluated. We study this approach in Chicago, where DV victims gauged to be at highest risk by prosecutors are selected for additional outreach as well as prosecutorial and advocacy resources. Leveraging variation in prosecutors’ tendencies to classify cases as high risk, we show that this approach rapidly and persistently lowers the likelihood of homicide by 71% for those on the margin of inclusion. Additionally, prosecutors are efficient at targeting this resource, consistently outperforming standard machine learning algorithms that predict homicide risk.
- P-Hacking in Academic Clinical Trials
- With: Jorge Guzman and Bruce Kogut
- Abstract: A common belief in the history of science is that institutions matter. They incentivize individual scientists to commit to the norms of open science, e.g., transparency, rigor, and accountability in return for reputation. However, historically, the violation of these norms, e.g., the reproducibility of research findings, have raised questions about the effectiveness of institutions. Using both distribution and density tests, we document statistically significant p-hacking in U.S. academic clinical trials but not in the trials registered by commercial laboratories.We analyze the impact of the 2013 SPIRIT statement–a checklist defining the minimal requirements of clinical trial protocols which was adopted by leading journals–as a type of treatment that could potentially diminish the incentives for p-hacking in clinical trial research. Focusing then on academic trials, we ask whether p-hacking decreased after the introduction of SPIRIT. We show that the dissemination of protocols sharply increased after the introduction of SPIRIT. Clinical trials started after the introduction of SPIRIT show lower levels of p-hacking, with the effects stronger for those trials whose plan more closely resembles the textual language of the SPIRIT guidance. In contrast, SPIRIT did not reduce p-hacking in trials already underway when introduced. Our analysis emphasizes the significance of institutional innovations to counter the consequences of over-incentivizing academic scientists to promote effort that sometimes favors career concerns over normative goals of open science.
Other
