Research article · 2019 · Full text
International Journal of Statistics and Applied MathematicsVoice call analytics using natural language processing
Volume 4(6) · Pages 133–136
Research areasTime Series Analysis and ForecastingTraffic Prediction and Management Techniques
Abstract
Author-writtenThe main challenge of call centres based on voice call is the quality monitoring, where a person has to listen to all the recorded audio files or a random sample of audio files, to check how the call centre representative has performed. It also helps in recording the feedback from the customer which is very useful in various fields of business analytics like marketing, sales, service etc. This process involves enormous amount of human effort and the time consumed is very high. With the rise of machine learning applications in almost every industry, in this paper we have tried to develop a software with the use of Machine Learning and Natural Language Processing, which helps in performance analytics of voice call agents.
Summary
Source-checkedPlain-language reading of the full paper, prepared with AI and checked against the text. Each point links to the authors' own words.
Why it matters
- Quality monitoring in voice-based call centers is challenging because it typically requires humans to manually listen to all or random samples of recorded audio to evaluate agent performance. View evidence ↗
- The manual process of monitoring call center audio is inefficient, consuming significant time and human effort. View evidence ↗
- Capturing customer feedback from these calls provides valuable data for business analytics in areas such as sales, marketing, and service. View evidence ↗
The question
- The authors aimed to develop software using Machine Learning and Natural Language Processing (NLP) to automate the performance analytics of voice call agents. View evidence ↗
What they did
- The study used voice call samples sourced from educational videos on youtube.com, which were converted into audio format. View evidence ↗
- To reduce data transfer latency, audio files were programmatically divided into small chunks of 10 seconds each. View evidence ↗
- The authors used the fuzzywuzzy module to compare manually transcribed text against text generated by various APIs, including Google Cloud, Google Voice, and Pocket Sphinx. View evidence ↗
- Transcription accuracy for Pocket Sphinx, Google Cloud Speech to Text, and Google was tested using the Harvard Open Speech repository. View evidence ↗
- Agent performance was evaluated using a score based on the detection of customer emotions, use of banned words, use of greeting words, and mentions of competitor names. View evidence ↗
What they found
- The wit.ai API achieved the highest speech-to-text accuracy at nearly 90%, while the Google speech-to-text API achieved 87%. View evidence ↗
- For detecting angry calls, both wit.ai and Google achieved 100% accuracy, whereas Pocket Sphinx only achieved 37%. View evidence ↗
- Wit.ai and Google perfectly detected the 33% of calls that were unsatisfied, while Pocket Sphinx failed to identify any. View evidence ↗
- Wit.ai and Google perfectly detected the 33% of calls that were happy, while Pocket Sphinx failed to identify any. View evidence ↗
- In terms of cumulative overall accuracy for detecting human emotions, wit.ai reached 100% and Google reached 80%. View evidence ↗
- The authors found that detecting competitor names (nouns) is difficult for most APIs, but wit.ai performed well in this task. View evidence ↗
What it means
- The authors concluded that they successfully developed software capable of measuring call center representative performance and providing business analytics insights. View evidence ↗
- The current work is limited to the analysis of call center conversations; the authors intend to develop their own automated speech recognition software in the future. View evidence ↗
- Future improvements include using fuzzy mathematics and deep learning to increase transcription accuracy and implementing voice pitch analysis for emotion detection. View evidence ↗
Visual summary
3 visualsIllustrations explain the paper; charts are replotted from the paper's own table values.
Agent Performance Scoring Framework
AI-generated conceptual illustration (A digital communication interface and data processing system). It explains the idea and does not show study data.
View source evidence ↗
Accuracy of speech to text: Sphinx vs Wit.ai vs Google
Replotted from Table 1 (page 4) of the original paper. Every bar is the paper's own table value.
View source evidence ↗
Accuracy of human emotions: Sphinx vs Wit.ai vs Google
Replotted from Table 1 (page 4) of the original paper. Every bar is the paper's own table value.
View source evidence ↗Figures & tables
From the original paper
Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Table 1
Figure 7
Figure 8
Figure 8Introduction
Modern organisations are increasingly deploying voice bots to streamline various workplace processes. Chances are that if you dial your bank's helpline, you may be prompted by a pleasing formal sounding voice to input your request. That voice may be a chatbot's, designed to handhold you in reaching your desired result in the most efficient manner. The basic solution has been around for a while but thanks to advancement in AI technology, over the years it has rapidly improved in terms of the quality of interaction and the customer outcomes. There are several variations of conversational bots around us. Even our phones have chatbots in the form of Siri and Google Assistant. Amazon Alexa is another example of a verbal or voice based chatbot found commonly in households today. These bots fuel the Internet of Things (IoT) ecosystem and help users to naturally engage with all the smart devices at home. It is expected that by the year 2020, 50% of searches will be voice based. If that is where customers are heading, companies need to be on that platform as well.AI can create a paradigm shift in workplace functioning enabling by voice bots. The old UI-based legacy apps will be replaced by voice bots. There will be no need for different apps for different tasks. The communication and action will become streamlined saving organizational time and other resources. Tech giants like amazon, google, Facebook, Microsoft are giving more importance in voice-based applications, newly released flagship smart phones have speech to text features, they have released many open source API's, for transcribing speech to text, this shows that the industry 4.0 is moving towards voice based applications and voicebots. Motivated by this very fact we started this project, of all the commercially available softwares our's is one of a kind we have used low latency speaker-independent speech recognition, however specific users voice can be trained. We have chosen the voice call samples from youtube.com which was uploaded for educational purpose. These video files are then converted into audio format. The major challenge in development of this software is to find the best API among commercially available speech to text API's such as: api.ai Wit.ai Pocket sphinx Google
Speech recognition
The audio calls of the particular agent is merged into one large file programmatically in order to optimise the transcribing process and in order to reduce latency (the delay before a transfer of data begins following an instruction for its transfer) we In have to chunk the file into small chunks (10secs). We used fuzzywuzzy module to compare the manual transcribed text file of the voice calls and the text data transcribed from different API's like sphinx, google cloud, google voice etc. The accuracy of transcribing was tested between pocket sphinx, google cloud speech to text and google for accuracy using Harvard Open Speech repository from the internet. The accuracy of wit.ai was more compared to other packages.This work only focuses on the analysis of call centre conversations and future work will include our own automated speech recognition software.The transcribed text is stored into data sets the entire code is done using python language. The pocket sphinx module is used for both offline and online conversion of speech to text and wit.ai is used for online conversion of speech to text hence the accuracy of wit.ai is high.
Call analytics metrics
The following metrics are widely used in the call centre industry, to perform analytics. Customer emotions detection: The various emotions of the customer such as whether he is satisfied or not, whether he is happy or angry with the agent is detected. This data can also be used to gather insights like review for new product launch, marketing, sales, service, etc. Banned Words: Use of banned words can be detected.
Words in conversation texts have been compared with the list of banned words and checked whether agent or customer has used any banned word or not. The number of matching banned words has recorded as number of banned words in database. Greeting words: In industries like call centre, where there is direct interaction with the customer, it is very much important to for the call centre agent to greet the customers. But many times because of the hectic schedule, the representative might miss greeting the customers. Words in conversation texts have been compared with the list of greeting words and checked whether agent or customer has used any greeting word or not. The number of matching greeting words has recorded as number of greeting words in database. Usage of competitor's name: The various cases under which the competitors name is used by the customer can be recorded. This can give deep insights in marketing. Performance score: A performance score based on the above mentioned parameters is devised, to evaluate the call centre representative's performance.
Accuracy of unsatisfied calls detected
The actual percentage of unsatisfied calls is 33% which is perfectly detected by wit.ai and Google, whereas sphinx couldn't identify unsatisfied calls.
Accuracy of happy calls detected:
The actual percentage of happy calls is 33% which is perfectly detected by wit.ai and google, whereas sphinx couldn't identify happy calls. Accuracy of banned words detected:
Conclusions
In this paper we have developed a software to measure performance of call centre representative and also to get useful insights for business analytics. We have compared different API's based on which we can use the API's for different conditions We are working on a speech to text convertor using fuzzy mathematics, and deep learning models, to increase the accuracy of the existing methods. We are also working on voice pitch based customer emotions.
Figures and tables from the paper
References
5 in the original paper- Kopparapu SK. Non-Linguistic Analysis of Call Enter Conversations: Springer, 2015.
- Chakraborty G, Pagolu M, Garla S. Text Mining and Analysis: Practical Methods, Examples, and Case Studies Using SAS: SAS Institute, 2014.
- Takeuchi H, Subramaniam LV, Nasukawa T, Roy S. Automatic Identification of Important Segments and Expressions for Mining of Business-Oriented Conversations at Contact Centers, in EMNLP-CoNLL, 2007, 458-467.
- Mishne G, Carmel G, Hoory R, Roytman A, Soffer A. Automatic analysis of call-center conversations, in Proceedings of the 14th ACM international conference on Information and knowledge management, 2005, 453-459.
- Garnier-Rizet M, Adda G, Cailliau F, Gauvain JL, Guillemin-Lanne S, Lamel L et al. CallSurf: Automatic Transcription, Indexing and Structuration of Call Center Conversational Speech for Knowledge Extraction and Query by Content, in LREC, 2008.
Cited by 12
From OpenAlex, OpenCitations and reference lists read by Scinovex
- Exploring data mining: facets and emerging trends ↗
Sumeer Gul, Shohar Bano, Taseen Shah · Digital Library Perspectives · 2021
Purpose Data mining along with its varied technologies like numerical mining, textual mining, multimedia mining, web mining, sentiment analysis and big data mining proves itself as an emerging field and manifests itself in the form of different techniques such as information mining; big data mining; big data mining and Internet of Things (IoT); and educational data mining. This paper aims…
❝ 23 - A Multimodal Approach to Improve Performance Evaluation of Call Center Agent ↗
Ahmed Abdelrahman, Khaled F. Shaalan, Sergio Luis Toral, Yasser Hifny · Sensors · 2021
The paper proposes three modeling techniques to improve the performance evaluation of the call center agent. The first technique is speech processing supported by an attention layer for the agent's recorded calls. The speech comprises 65 features for the ultimate determination of the context of the call using the Open-Smile toolkit. The second technique uses the Max Weights Similarity (MWS)…
❝ 11PDF - Data-driven subjective performance evaluation: An attentive deep neural networks model based on a call centre case ↗
Ahmed Abdelrahman, Uthayasankar Sivarajah, Zahir Irani, Kamran Mahroof, Vincent Charles · Annals of Operations Research · 2022
Abstract Every contact centre engages in some form of Call Quality Monitoring in order to improve agent performance and customer satisfaction. Call centres have traditionally used a manual process to sort, select, and analyse a representative sample of interactions for evaluation purposes. Unfortunately, such a process is marked by subjectivity, which in turn results in a distorted picture of agent…
❝ 9PDF - Agent Productivity Modeling in a Call Center Domain Using Attentive Convolutional Neural Networks ↗
Ahmed Abdelrahman, Sergio Luis Toral, Khaled F. Shaalan, Yasser Hifny · Sensors · 2020
Measuring the productivity of an agent in a call center domain is a challenging task. Subjective measures are commonly used for evaluation in the current systems. In this paper, we propose an objective framework for modeling agent productivity for real estate call centers based on speech signal processing. The problem is formulated as a binary classification task using deep learning…
❝ 5PDF - Natural language processing: A literature survey ↗
Amit Kumar Shrivastava, Jay Kumar Jain · International Journal of Communication and Information Technology · 2020
It is quite hard to imagine a smart system like a voice assistant or a chat-bot or a recommender system without Natural Language Processing (NLP). It all starts with an initial unit that first interprets the data (audio or text) provided and then start making sense of the data and after proper processing of the data the actual steps are…
❝ 4PDF - A Framework for Mining Speech-to-Text Transcripts of the Customer for Automated Problem Remediation ↗
Prateeti Mohapatra, Gargi Dasgupta · Proceedings of the AAAI Conference on Artificial Intelligence · 2024
Technical support services get several thousand voice calls every year. These calls vary across a range of technical issues or maintenance requests for a suite of hardware and software products. On receiving the call, a support agent creates a ser- vice request artifact that contains her interpretation of the customer’s problem. This service request goes through the life cycle of…
❝ 1PDF - Using NLP to analyze whether customer statements comply with their inner belief ↗
Fabian Thaler, Stefan Faußer, Heiko Gewald · arXiv (Cornell University) · 2021
Customers' emotions play a vital role in the service industry. The better frontline personnel understand the customer, the better the service they can provide. As human emotions generate certain (unintentional) bodily reactions, such as increase in heart rate, sweating, dilation, blushing and paling, which are measurable, artificial intelligence (AI) technologies can interpret these signals. Great progress has been made in…
❝ 1PDF - The Context Sets the Tone: A Literature Review on Emotion Recognition from Speech Using AI ↗
Fabian Thaler, Maximilian Haug, Heiko Gewald, Philipp Brüne · Lecture notes in information systems and organisation · 2024
❝ 0 - Visual Chatbot for Knowledge Transfer ↗
Sarra Bouzayane, Arezki Aberkane · International Journal of Knowledge-Based Organizations · 2022
This paper proposes an approach to design a visual chatbot to enhance the virtual knowledge sharing process. Existing chatbots are either textual or vocal whose performance has not exceeded 60%. However, in various fields a textual description is no longer sufficient, and it is so essential for users to exchange images to better express their preferences. This prevents them from…
❝ 0 - Put your money where your mouth is: Using AI voice analysis to detect whether spoken arguments reflect the speaker's true convictions. ↗
Fabian Thaler, Stefan Faußer, Heiko Gewald · arXiv (Cornell University) · 2021
Customers' emotions play a vital role in the service industry. The better frontline personnel understand the customer, the better the service they can provide. As human emotions generate certain (unintentional) bodily reactions, such as increase in heart rate, sweating, dilation, blushing and paling, which are measurable, artificial intelligence (AI) technologies can interpret these signals. Great progress has been made in…
❝ 0PDF - THE USE OF CALL ANALYTICS IN BUSINESS DEVELOPMENT; APPLICATION OF BUSINESS, MARKETING AND ARTIFICIAL INTELLIGENCE ↗
Reza Mousavi, Sama Sohrabi Tabesh · 2021
The artificial intelligence focuses on the learning behaviors of human on any specific issue and develop intelligent systems for tackling the issues in future. The call analytics is technique of artificial intelligence that is now being adopted in business and especially the marketing. This study has focused on the implications and applications of call analytics in business. Moreover, it aligned…
❝ 0 - SMS classification using machine learning algorithms for the risky messages identifying ↗
Venkatalakshmi Katta · International Journal of Computing Programming and Database Management · 2020
The bulk of SMS that the Quick Response Team and Rescue Agencies received during disasters made it hard for them to categorize responses based on priorities. This paper provides a method that classifies SMS received by the agency as Spam, Invalid, Alert 1 Alert 2, and Alert 3. This method allows proper response to be extended to those asking for…
❝ 0PDF