Signals in the Noise – Insight Engines Magic Quadrant 2021

It appears that we are approaching the early stages of a return to (albeit a new) normalcy. Cancelled events and conferences from a year ago are now virtual and planning for in-person conferences is on the horizon. A similar sign in our industry is the release of the annual Gartner Magic Quadrant for Insight Engines. The report’s beginnings trace back to the Gartner Magic Quadrant of Enterprise Search report, which tried to capture the many facets (pun) of search technology capabilities available to the enterprise. Gartner skipped releasing the report last year, making this report a much-anticipated update to the previous version of the Quadrant released in late 2019. That report featured MC+A as a strategic partner and reseller of Attivio.

Every quadrant report generates questions related to how vendors plot to the quadrants. Typically, all vendors who spent significant time with Gartner place well. There is considerable pride for the vendors that do well and questions on ranking criteria for those that did not as the quadrant ranking reflects their management and vision. Additionally, customers will often use this report alone to shortlist any search-specific ‘search’ projects. So there are immediate benefits to a strong placement on the Quadrant. The Quadrant coming out later than usual did not reduce questions, even with our lower expectations for the report due to its short absence.

Firstly, having read our share of these reports, we provide our assessment of it as third-party validation, with the disclosure that we have an obvious bias towards our partners. That said, this report is challenging to validate as a third party because of our understanding of the products, what Gartner has stated the scoring weights to be, and what Gartner has commented on in its strengths and weaknesses. Customers who would like a detailed assessment should contact our offices for a search technology assessment. These assessments allow us to fully detail the impact of any of these technologies on your business.

This report alone shapes much of the sourcing for enterprises throughout the year. The technology vendors, many of whom are our partners, spend a great deal of time and care to ensure that the Gartner analysts and others are fully aware of their products, company, and partners’ capabilities. Given that attention, along with the nonpublic information shared with analysts, the report is supposed to be the most up-to-date snapshot that the industry has of itself. This information allows purchasing departments to shortlist vendors for their RFPs as they become prescreened quickly.

Buyers should also understand that participation in the Quadrant report requires a buy-in. Additionally, some have suggested that the Quadrant report is biased , with Gartner itself stating it is purely opinion , while the grid oversimplifies a complicated task of fitting technology to your use case. Lastly, it’s unclear how Gartner applies weighting to the report criteria. For example, how does Gartner score the Natural Language Processing (NLP) capabilities and relevancy of the system by default and over time? These are critical components that seem to be missing from the completeness of vision category.

We recommend that companies in the market for this technology look at sources outside of Gartner to validate the use case and advise them through this process.

Much has changed between the releases of the Magic Quadrant for Insight Engines. It is no surprise that there is a movement by vendors around the quadrant. For example, in 2019, IBM took the top of the scoring in the category of “ability to execute,” with Google surprisingly second. In that report, Lucidworks was first in “completeness of vision” with Coveo in second (both of which were not surprising). The rest of the field ended up clustered in the lower section of the upper quadrant. Some speculation in the industry “sewing circle” was that the clustering intended to give room for scoring improvements over time as this was a new system/field.

The movement in the quadrant should not mean that these vendors did not keep up with the field. Instead, it is more likely that Gartner’s scoring model, which is not open for review, is how redistribution occurs. A comparison in evaluation criteria between 2019 and 2020:

It’s unclear why these minimal scoring changes alone would reshuffle the vendors so much. What is clear is that Gartner is losing its understanding of the needs of the enterprise market. We will discuss briefly some examples.

For an example of the scoring question, let’s look at Mindbreeze, which scored incredibly well. While not a partner of MC+A, other agencies (aka competitors of MC+A) who represent Mindbreeze speak very favorably of them, and their continued presence in the quadrant reflects well on the company and its executive team. However, we noticed Gartner’s comment about Mindbreeze: “the greatest flexibility in deployment options.” Mindbreeze’s solutions are historically delivered in an appliance form factor. Given that it can run on-prem and in their cloud, it is more flexible than on-prem or in the cloud. However, if you search “Hipaa” (HIPAA is a privacy regulation for health care information in the United States and any cloud provider must accommodate for it) on Mindbreeze’s website, you see:

It is striking that there is no mention of HIPAA compliance is a significant gap in capabilities and specification with regard to their cloud deployment options. It is not what we should expect from a technology leader.

Additionally, there is not any mention of which public clouds Mindbreeze deploys in, its platform’s integrity, etc. These are particularly important and known to anyone who is ever filled out InfoSec security forms. The other items in the report that raise questions are the inclusion of Gartner’s Peer Insights scores and ‘incorporating third party technology.’ Gartner Peer Insights is the equivalent of Yelp or Amazon reviews, are not officially validated, and are ignored by anyone serious about the technology space.

Does this mean that Mindbreeze should not score well? No, quite the contrary. Our criticism is of Gartner and their scoring criteria. We are aware of many companies the have deployed Mindbreeze and are pleased with the technology’s business outcome. Mindbreeze is late to the cloud, but their adoption of the cloud is excellent for customer choice. Going to the cloud is incredibly complex, and everyone, including Google after the GSA, has struggled to do it well. The cloud’s inherent complexity (security, availability zones, multi-tenancy, K8, etc.) highlights that customers should perform their due diligence beyond just reading these reports.

We were stunned to see Google’s placement in the report in any meaningful way. Similar signs in the field suggest a similar fate for Google Cloud Search to that of the Google Search Appliance, particularly when you consider Google’s lack of focus on launching new products. (See Google Stadia, Google Wave, etc.) (Read: Spring cleaning is upon us) Our observation is that Google Cloud Search lacks marketing behind the product. We are experiencing customers coming to us for guidance, stating that “Google isn’t selling Google Cloud Search.” Is this product going to be 2021 Attivio? If this ends up the case, what does the “Overall Viability” category mean?

Google’s commercial search leadership provides a free (spending billions on Google.com has benefits) perception advantage with potentials customers. “Let’s just call Google” is still a thing. But we fail to see how Cloud Search supports data enrichment, delivers results to various touch points or allows for query input flexibility on its own. These gaps require you to rely on additional Google Cloud Services. Google Cloud provides impressive capabilities, but many other vendors can also tap into those APIs for their product. (See Mindbreeze for reference when product company leverages third parties)