AWS Data Pipeline Design Supports Natural Language Processing

The Challenge

A leader in the fast food industry hired Elder Research to assist in the design of a cloud-based Data Pipeline System Design to support Natural Language Processing (NLP) to gain business insights from their customer and organizational data assets. Natural Language Processing (NLP) is a branch of artificial intelligence in which computers analyze textual data to understand, interpret, and manipulate human language.

The Solution

The project began with an in-person discovery session with key stakeholders to understand the relevant AWS infrastructure, data environment, and priority use cases. This understanding enabled us to design an NLP pipeline to be deployed in the client’s Amazon Web Services (AWS) environment that would automatically ingest, integrate, transform, and serve textual data for NLP. We recommended a phased collaborative project plan for deployment for two priority use cases.

The NLP data pipeline design enables:

  • Data availability across the enterprise – Moving NLP data from vendor-driven silos to organization-curated central locations leads to shared, actionable business intelligence.
  • Data cleanliness and formatting – Defining and applying frequently used categories across data platforms will unify insights for each segment; enabling multi-attribution will allow data rich engagements to reflect all categories referenced in the language; and translating audio files from telephone interactions into text will accurately capture the engagement.
  • Tool enhancements – Real-time translation of audio to text and resulting analytics can proactively identify potential problems and solutions as the agents are on the line with an operator or customer. Using NLP for categorizing web-based inputs will provide more nuanced and consistent data.
  • Future readiness – Harnessing the power of NLP and applying what is learned across a variety of business units, the client can continue their history of innovation and evolve to meet the consumer and owner-operator expectations of the future.

Two priority use cases were identified for the NLP pipeline:

  • Customer feedback – Improve the current customer service and customer engagement survey data processes by implementing a query engine for analysts to enable nimble extraction and manipulation of relevant data, auto-classifying categories for customer compliments or complaints to capture more robust data in a consistent format and definitions, frequency analysis to help identify patterns and prioritize solutions, and sentiment analysis to understand how customers are feeling and determine the best type of intervention.
  • Franchise owner support – Through several means, improve support service processes for franchise owners reaching out to the internal help desk: integrating data across data stores, real-time transcription of phone calls, and providing better recommendations from the knowledge base to resolve the problem in real time using a Resolution Suggestion Model.

The NLP Data Pipeline design incorporated various AWS services:

  • AWS Simple Storage Service (S3) – Saving raw data in S3 allows new analytics to be performed on data that was not previously leveraged. S3 features direct integration into many of Amazon’s products including Amazon Transcribe, making it a practical intermediate storage step before enhancing the data.
  • AWS Relational Database Service (RDS) – Used to store the parsed and processed data for analytics. RDS is highly scalable to allow for growth in the future without a prohibitive upfront cost. Housing all the data in RDS allows queries and analytics to pull from one primary source, allowing for more complicated and potentially interesting questions to be answered.
  • Amazon Transcribe – Automatically recognizes the speech in audio files and creates a transcription allowing for real-time insight and channel identification during support calls.
  • Amazon SageMaker – Features real-time predictions during model deployment which allows the Resolution Suggestion Model to provide Knowledge Base article recommendations to agents in real-time.
  • AWS Glue – Serverless and fully managed extract, transform and load (ETL) service used to reshape and enrich Voice of the Customer data. Glue analyzes the data, builds a metadata library, and automatically generates Python code for recommended data transformations.
  • Amazon Kinesis – Using AWS Kinesis Video Streams for help desk call audio provides real-time collection, processing, and data analysis to allows insights to be generated as quickly as possible. All infrastructure underlying the streaming process can be managed by Amazon, enabling the client to focus on the use of the data, rather than managing the streaming process.
  • AWS Lambda – Used to run code in a serverless, scalable way to eliminate the need to provision or manage servers and reduce cost by only paying for active compute time.

The flow diagram below shows the AWS Cloud data pipeline design for the customer feedback use case.

 

Results

Understanding of nuanced situations through voice and text provided by Natural Language Processing is a potential market differentiator for our client. This NLP pipeline will enable the client to exceed customer and franchise owner expectations by providing insights that can be applied across business units. The solution will increase enterprise access to textual data and insights, streamline text data processing so resources can be reallocated to new efforts, minimize the number of assumptions made about text data, and generate the potential for using textual data in innovative applications.