Augmenting public sector data-driven decision support systems with expert knowledge: case of OTT

Abstract

Public sector data-driven decision support systems are uniquely challenging to design due to the ramifications they have on the societal level. Accountability and ethical considerations require these systems to arrive at an equilibirium between accuracy and interpretability amid various implementation and data constraints. While these systems need to contribute to legitimate governance through reasoned and explainable decision-making, they also need to accurately model the policy outcomes they were designed to support. Inopportunely, inductive data-driven systems struggle to solve problems that rely on heuristic input. In this thesis, a particular knowledge engineering technique was adopted to augment a public sector Machine Learning decision support tool with domain expert knowledge. The case in question is OTT – a job-seeker profiling tool used by the Estonian Unemployment Insurance Fund to predict the long-term unemployment risks of their clients. Upon augmenting it with knowledge from caseworkers and data scientists associated with the project, some evidence was found that accounting for expert knowledge in probabilistic data-driven models can lead to a model that performs better on new out-of-sample data and is more in line with underlying domain rules. This yields important implications on the future of Machine Learning in the public sector as it opens up new potential use cases in avenues where 1) labelled training data is hard to come by, 2) a more generalizable model is preferred due to frequent changes in the surrounding context, 3) a model has to perfectly mimic domain logic for interpretability and explainability reasons.