Nand Kishor is the Product Manager of House of Bots. After finishing his studies in computer science, he ideated & re-launched Real Estate Business Intelligence Tool, where he created one of the leading Business Intelligence Tool for property price analysis in 2012. He also writes, research and sharing knowledge about Artificial Intelligence (AI), Machine Learning (ML), Data Science, Big Data, Python Language etc... ...Full Bio
Nand Kishor is the Product Manager of House of Bots. After finishing his studies in computer science, he ideated & re-launched Real Estate Business Intelligence Tool, where he created one of the leading Business Intelligence Tool for property price analysis in 2012. He also writes, research and sharing knowledge about Artificial Intelligence (AI), Machine Learning (ML), Data Science, Big Data, Python Language etc...
Data science is the big draw in business schools
515 days ago
7 Effective Methods for Fitting a Liner
525 days ago
3 Thoughts on Why Deep Learning Works So Well
525 days ago
3 million at risk from the rise of robots
525 days ago
Top 10 Hot Artificial Intelligence (AI) Technologies
5 Metrics Every Chatbot Should Track
40% of a typical bots' users only engage in one conversation.
This statistic, calculated by Ilker Koksal, co-founder of Botanalytics, suggests bot makers need to invest more effort to measure bot metrics and performance and deliver value to users. Traditional metrics like DAU/MAU and analytics tools like Google Analytics or Mixpanel work well for websites and mobile apps, but the unique conversational nature of chatbots requires a different perspective on performance.
Some traditional metrics may even be misleading. Session length is often used as a proxy for user engagement on web and mobile. However, many chatbots are utilitarian and should be a functional shortcut compared to their app or website counterparts. Increased session length could mean users are confused or the conversational flow is inefficient.
With the skyrocketing popularity of chatbots, bot developers have collected enough data to learn what is and isn't working. Bot analytics companies like Dashbot and Botanalytics have collectively pushed close to 100 million messages and get a bird's eye view for what bot metrics are most useful for developers. Developers on their platforms have tried dozens of new measurements to identify the best ways to improve their bots.
Here's what we've learned are the 5 chatbot metrics that produce the most useful insights.
1. ACTIVE & ENGAGED RATES
Many users barely interact with a chatbot before churning off. 40% of a bot's users only interact one time. Given the high churn, identifying and nurturing active and engaged users is key to long-term success.
Dennis Yang, co-founder of Dashbot, recommends using Active and Engaged Rates to combat churn. When a user reads a message in a session, that session is considered "active". When a user responds with a message in a session, that session is considered "engaged".
Active rate = number of active sessions of a user / total number of sessions of that user.
Engaged rate = number of engaged sessions of a user / total sessions of that user.
How do you optimize active and engaged rates? Yang suggests you answer this question:
What are the top messages users send my chatbot?
Machaao, a popular Facebook Messenger chatbot for cricket fans, increased their user engagement by 300% by analyzing and adapting to how the most active and engaged users spoke to the bot. Users' messages reflect their expectations around how a bot should behave. Fitting their mental models is usually a winning strategy to boosting engagement.
"We figured out that the top message sent to our bot was the Like button," says Harshal Dhir, founder of Machaao. Inspired by watching active rates, engaged rates, and top messages, Machaao enabled easier expression of Likes and also prioritized news and schedule formats that matched the expectations of their users.
2. CONFUSION TRIGGERS
The nascent chatbot industry has yet to develop the optimal user experience with conversational UI. Challenges exist throughout the funnel: bringing users to a bot, communicating functionality, driving towards action, and handling inquiries and errors.
Given the huge range of possible user input, chatbots often misinterpret or can't understand what a user wants. Thus, the incidences when your bot says a version of "I don't understand" must be closely watched. Also useful is seeing what user inputs caused the bot's confusion.
StreakTrivia is a daily trivia bot that runs a massively multiplayer trivia game on Facebook Messenger every day. Their bot asks players True or False questions and then presents the users with quick reply buttons to answer with. By closely watching confusion rate, StreakTrivia's team was able to catch an issue they would have otherwise missed. Turns out when the bot was confused, this was often due to the user typing in "true" or "false" instead of using the provided buttons.
Tracking confusion rate also helps triage when human intervention is needed. Just as bad customer support associates ruin a customer's opinion of your brand, so will a bad chatbot experience. Honing in on high-risk scenarios and escalating to trained staff dramatically reduces churn and provides a critical opportunity to learn about user needs. Read More