Over the last few years, as my team started understanding more of Coveo and the applications within the workspace for Enterprise Search, we did some analysis around the progress in this space. In this blog, we would like to share our findings on the history and a brief overview of typical features in Insight Engines such as Coveo.
Based on our research, the Enterprise Search started getting more formalized in the 1970s with IBM’s STAIRS (Storage and Information Retrieval System). A quick detour but if you are interested in a paper around information retrieval from University of Michigan, refer to this link. It is interesting to see that fundamentals and importance of RELEVANCE remain the same even today.
Later, during the 1980s, another software called Muscat was developed and commercialized in the UK which later got acquired by Smartlogik. As data continued to grow exponentially within and outside the enterprise with WWW in the 1990s, many new platforms have continued to emerge throughout the 1990s and 2000s. While Enterprise search, the ability to provide employees seamless access to data and information across multiple data sources has been a key requirement of organizations, many solutions have struggled to keep up with the demands of modern organizations. Here are some of the most common issues and limitations of traditional enterprise search:
- Traditional enterprise search often fails to deliver relevant results. There could various factors that hinder the relevance, including inability to comprehend the organizations information and data needs, poor indexing, incorrect metadata etc.
- Traditional enterprise search is focused on data retrieval and data synthesis based on the organization’s enterprise search needs. However, the crucial aspect that is not emphasized is on the UI/UX design patterns that will make the user experience seamless while using the enterprise search. Hence, the Traditional enterprise search solutions are complex and difficult to use and require training and skills to derive the expected search results.
- Traditional enterprise search solutions are designed conservatively to meet the immediate need of the organization’s complex information search needs. This model of design and implementation cannot scale to the rapidly changing enterprise search needs and poses challenges to swiftly implement new features and capabilities.
- Traditional enterprise search solutions are built in isolation; the solutions are often built on data from few systems. The power of “Data Aggregation”, where organizations and can easily access and analyze the data, is not comprehensively leveraged. This short coming limit the capability of traditional enterprise search solutions in providing insightful and relevant search results.
The above challenges with traditional enterprise solutions have influenced organizations to transform from traditional search engines to modern and flexible enterprise search solutions that will enable the organizations to meet the following goals:
- Support for multiple connectors out of the box. These connectors should include cloud data sources, cloud based repositories, flexibility to configure and customize new data sources.
- Centralizing the indexing at one place across all the content from various data sources.
- Improve search accuracy
- Enhance user experience
- Streamline information access
- Facilitate collaboration
- Provide insights and analytics
From Gartner’s website these modern systems are Insight Engines. “Insight engines apply relevancy methods to describe, discover, organize and analyze data”. This allows existing or synthesized information to be delivered proactively or interactively, and in the context of digital workers, customers or constituents at timely business moments. Products in this market use connectors to crawl and index content from multiple sources. They index the full range of enterprise content, from unstructured content such as word processor and video files through to structured content, such as spreadsheet files and database records. Various “pipelines” are used to preprocess content according to type, and to derive from it data that can be indexed for query, extraction and use via a range of touchpoints. Insight engines differ from search engines in terms of capabilities that enable richer indexes, more complex queries, elaborated relevancy methods, and multiple touchpoints for the delivery of data (for machines) and information (for people).”
Below is a beautiful blog on Coveo that explains the features in more detail.