Data Engineering for Promotion Analysis

Ingesting, transforming, fusing, and enriching large varieties of big and heterogeneous datasets seems straightforward on paper, but in reality requires a tremendous amount of nuance, experience, and wisdom in order to achieve high value results. Many organizations succeed in building data pipelines, but fail to do so in a way that truly adds organizational value. One of Elder Research’s premier CPG clients needed help integrating disparate data sources including syndicated sales, promotion, and coupon data to enable their marketing team to analyze the effectiveness and return on investment (ROI) of historical marketing events.

Elder Research teams utilize a balanced multi-disciplinary approach for analytics problem solving–data strategy, data science, and data engineering. Our proven approach comes from over twenty years of experience in valuable data architecture and analytic solution design. We started from both ends of the pipeline (use cases and data sources) and determined the best architecture to transition raw data sources into robust structures that support understandable, usable data insights. Data should be filtered and organized into storage architecture that supports business use cases most effectively. For this project,  our team developed a modular and tech stack agnostic pipeline to import relevant data sources and merge them into a single source by aligning the various event dates. The key value proposition was that the data sources were expertly fused and enriched so as to support downstream analytics, exploration, and data products both now and into the future.

We developed a hand-crafted methodology that ingested data from disparate sources, manipulated and enriched the data for analytics, and calculated the ROI of historical events. This resulted in a dataset with baseline and promoted sales for over 5,000 promotion events and over 200 coupon events for hundreds of products over a two-year span. The resulting insight from the data has helped our client understand historical behaviors of promotions and coupon events, informing and optimizing future promotion and coupon cycles and events for greatest ROI.