How do users search, evaluate and choose a messaging solution?
IBM MQ is a 25-year-old product used by businesses around the world for reliable enterprise messaging. In 48 hours IBM MQ provides 16 million travelers with real-time flight statuses, and in three minutes IBM MQ transfers $4 billion in banks around the world.
IBM MQ is typically managed by highly experienced in-house MQ admins. Recently, the skills and responsibilities of MQ admins are being replaced by app developers who advocate for simple, less sophisticated messaging solutions.
Prompted by the Digital Marketing team, this research aimed to understand why so few developers are reaching the trial for IBM MQ in the Cloud, and why there were three significant drop-offs in marketing and product experience.
As a team of two researchers, we reviewed the entire digital marketing experience of IBM MQ in the Cloud from discovery to trial completion. By using a variety of methods to understand the perspective of an application developer, we were able to make informed recommendations to a multi-disciplinary team.
How do users search, evaluate and choose a messaging solution?
Why aren’t users starting and completing our trial experience?
What are the bottlenecks to a great experience?
To understand the immediate pain points in the common path of our digital experience we decided to conduct a heuristic evaluation.
It was a suitable method for detecting the obvious usability issues, ensuring the design and development teams had work to focus on early in the research project.
We identified 34 usability issues in our first review. As this approach only allows you to evaluate how users interact might with the current design, we needed to speak to real users to understand the reality of their experience, as well as their unmet needs.
We used tools for implicit user tracking, like Segment and Amplitude, to understand where users were dropping off throughout their digital experience.
These tools helped us identify three areas of concern in the digital journey, but couldn’t detect which users (developers, admins or other) were dropping off and why.
Generative interviews were the next best step, but it took time, and pressure on internal stakeholders, to understand which users
were the priority for this research.
The digital marketing team finally concluded that app developers were the greater priority, but we did recruit a small number of MQ admins to validate our assumptions about
We recruited developers with experience of IBM MQ or another messaging middleware.
We asked about the major use cases and motivations researching messaging middleware. We also probed into what developers thought of competitor products and the suitability of their marketing. To ensure our internal stakeholders completely trusted the insights, we made sure that none of the participants knew we were calling from IBM.
As expected, developers were searching for and comparing IBM MQ against other types of messaging technology – trying to build a mental modal for why one was better than the other, and which was most suitable for their use case.
The next step was to validate whether the developer needs identified during interviews were served in IBM MQ’s marketing pages.
We recruited a second group of developers and asked them to ‘find and demo a suitable IBM product’ for our messaging use case.
We used a ‘think-aloud’ method to understand what developers were pleased and deterred by throughout the experience.
A significant issue was the discovery of developers using search terms so different from we team expected, they would have never found our digital marketing pages to begin with.
Usability testing sparked an interesting debate over how our SEO terms should be sourced – via Google, or via persona based research. Unmoderated usability testing in a diary study format helped us verify whether the search terms used in IBM MQ SEO were appropriate.
This method worked well. Instead of prompting developers on a given use case which caused our internal stakeholders concern, developers were documenting their natural search
By showing our digital marketing team what developers were really searching for we were able to sell the value of persona-driven search terms.
Conduct an open ended search experience to find a solution for the following use case.
Review the suitability of three messaging products based on the needs of this use case.
Assess the trial experience of two products, one must be IBM MQ in the Cloud.
Competitor analysis on the marketing pages of three major competitors to IBM MQ in the Cloud.
We compared product features, the design of educational materials, and how major concepts were described.
This research led to recognition of
It informed
Upon reflection