Last week, we released an Alpha version of Monty, the first ever AI-powered chatbot to help people buy software that will help their business thrive. We can’t express the excitement we have heard from the community since our launch and wanted to provide some updates.
By the Numbers
First, here are just a couple of the data points we’ve captured in our first week of Monty’s Alpha launch. We’ll continue to measure his impact and adoption over time.
- Number of chats: ~2,000
- Number of NEW users signed up to use Monty: ~600
Three Key Takeaways
We’ve been learning a lot during this past week as we continue to receive feedback and examine the chat queries coming through. The main takeaways from Monty’s Alpha launch can be summarized as the following:
- 1) There is overwhelming excitement for Monty’s potential. Software buyers believe that Monty will change the way we discover and purchase software. Specifically, Monty received praise for how well he can discern the type of software users want and direct them to a relevant category.
- 2) This is bigger than a typical chatbot launch, spurring larger discussions both inside and outside of G2. These include questions around “Will AI chat disrupt search?” and “What will be the long term impact on G2’s business?” Although we don’t have all the answers yet, we look forward to being part of these exciting conversations. What are your thoughts?
- 3) Initial users provided great, constructive feedback. We knew Monty was not perfect and we intentionally released him in Alpha form to get your feedback so we can iterate quickly. Thank you and keep it coming! The majority of feedback can be summarized as:
- The links that lead back to G2 are sometimes broken. We are working on it. The good news is this has an easy target. The links either work or they don’t. The bad news is this is not as easy to fix as one might think. Currently, we are working to resolve this by injecting more G2 data into the conversation via something OpenAI calls embeddings.
- It is not clear why Monty is recommending the products he does. Solving this problem is a bit more challenging because there isn’t as clear of a target. What is a good recommendation, and what is a bad one? While this can be subjective, we believe Monty’s suggestions should be heavily based on the experience of real users of the software products. For years, we have used our G2 Score algorithm as a way to rank software. We continue to stand behind this methodology, and we could tell Monty to return the highest G2 Score products every time. However, we believe this reduces the value of what AI can provide. It may appear that Monty sometimes has a mind of his own and this is what defines AI and makes it valuable. Monty realizes that one size doesn’t fit all and tries to discern what may be best for you. He does this by not just using the G2 Score, but also the text in the reviews, the text in the product descriptions and the text that it finds elsewhere outside of G2. All of this being said, we believe that Monty can better utilize the G2 data when providing answers and it is our highest priority right now to improve this. In fact, we have already made some improvements. How can you help Monty continue to improve? If you disagree with a recommendation you receive, let him know by asking, “Why didn’t you recommend XYZ product instead?” By doing this, Monty will get closer and continue to learn. Together, we will improve and tune Monty over time.
What's next?
Of course, after we address the above feedback noted, there are many ideas to explore Monty’s potential. Here are a couple that are at the top of our list:
- Monty will be more pervasive on the G2 site. Currently Monty is just on the homepage, but what if he could be used by everyone on the site and knew the context of where you are on the site and where you have been to improve his recommendations?
- Upgrade to GPT4. This is currently in limited BETA and we are in the process of upgrading. We are excited to see how this new model will improve recommendations.
Further Reading
Here are some of the most common questions we are hearing about Monty:
- When is the next release?
- Early in G2 we had a mantra of “We do it live.” This means we iterate and change things on a daily basis and there is no release schedule. This is absolutely the approach we are using as we continue to develop Monty. We’ll continue to communicate updates as we have them, so keep your eyes peeled.
- Why doesn’t Monty recommend the product in a category that has the highest G2 Score everytime?
- As noted above, this is all about how Monty determines his recommendations, which are customized based on the data provided by users. He also uses a broader dataset than just G2 rankings. For example, in some cases he might use G2 review text, G2 product descriptions, and externally publicly available content, including vendor websites.
- Besides Monty, are there other things G2 is considering with AI?
- Yes, we believe AI will impact our business in many ways in the years to come. We are actively exploring other use cases. Stay tuned as we continue to innovate with AI technology.
Keep the chatting going and feedback coming!
If you haven’t chatted with Monty already to help you discover software for your business, or want to chat again, you can start here. As we mentioned, the more people that use Monty, the smarter he will get.
Tim Handorf is a G2 co-founder and Head of G2 Labs.
Mike Wheeler is a G2 co-founder and serves as the company’s Chief Technology Officer.
