A client in the hospitality industry with thousands of restaurants nationwide was seeking a solution to better detect fraudulent refund-related activity. Elder Research developed machine learning models that evaluate and rank restaurant refund transactions according to their estimated likelihood of being fraudulent. These models leverage business-driven transactional inputs and additional restaurant-based inputs to both (1) predict the fraud likelihood and (2) provide explanations that describe which characteristics led to the prediction. This new process allows stakeholders to review a wider breadth of refunds more promptly while still prioritizing the riskiest activity.
The Challenge
All of the client’s restaurants faced the threat of fraudulent activity related to refunds. While the client already had a process in place to analyze refunds and uncover potential theft, it was time-consuming and not comprehensive enough to capture harmful activity across all restaurants. Therefore, they sought a solution that could automate key elements of their process in such a way that refunds could be evaluated more efficiently.
The Elder Research team prioritized developing a solution that would easily identify potential fraud patterns and clearly explain them to stakeholders. This model interpretability would clarify why specific refunds were labeled as risky, increasing transparency and helping stakeholders understand model outputs during transaction review.
The Solution
Elder Research developed interpretable, predictive models that assign fraud probabilities to all restaurants’ refunds and rank them daily for stakeholder review. These models leverage dozens of transactional and restaurant-level characteristics, including the time of day at which the refund occurred and how often refunds have occurred recently at each restaurant. We identified these features through collaboration with client stakeholders to pinpoint known and suspected fraud-related patterns.
Additionally, since the models used were interpretable, we could provide model explanations that specify which of the engineered features were most important when generating the refunds’ predictions.
We leveraged the client’s historical records of confirmed fraud cases so that the models could learn the characteristics that have historically been associated with fraudulent activity. These records provided thousands of positive fraud cases on which the models could train; however, it was still an extremely rare occurrence relative to all refunds. While it’s very common to need to predict rare events, especially in the fraud detection space, it’s still a challenging machine learning task because it’s difficult to tease out the predictive signals for rare occurrences.
By employing data preprocessing strategies to resample the training data, experimenting with different ways of measuring predictive outcomes, and implementing a strong testing strategy to accurately capture model performance, we were able to produce high-performing models that stakeholders could incorporate directly into their processes.
Furthermore, the models were developed and deployed within the client’s existing machine learning operations (MLOps) framework, which allows for convenient model retraining as behaviors evolve and stakeholders gather more confirmed fraud cases over time. This feedback loop provides an ongoing opportunity for the model to improve by reinforcing confirmed patterns and learning new patterns from the additional data.
The Results
Our models have been responsible for generating over 4.4 million refunds since their original deployment. Generally, business stakeholders now review a total of 200–600 refunds per month, looking across up to 500 restaurants; previously, they were only able to irregularly review a total of 20–200 refunds per month across up to 60 restaurants. In the time since our models replaced the previous system, this large increase in review rate has led to identifying fraudulent activity associated with up to $100,000 in total lost sales.
The line graph below shows the amount of likely fraud cases identified by our stakeholders before and after the models were implemented. Both the magnitude (total dollars) and breadth (number of restaurants) of identified fraud have increased since adoption of the models.
