Darwin Discovery Engine Blog

KM World Sessions: Fundamentals of Enterprise Search

This is the first in a series of notes from the 2009 KM World. Please forgive typos as this isi real time or near real time. Fundamentals of Enterprise Search was a preconference workshop. It was led by Avi Rappoport, Principal - Search Tools Consulting Editor, SearchTools.com. She is an independent consultant not connected with a vendor. Here is the session description.

“Search engines, big and small, have certain standard elements and processes. The more you understand them, the easier to tune them to solve your real information needs. This practical overview provides a big picture view of how search fits within enterprise and websites, and a focused introduction to search technology and user experience. Elements of search covered include robot spiders, database connectors and other tools for locating content, indexing issues, query parsing, retrieval, relevance ranking, and designing usable search interfaces. The workshop addresses common search problems and solutions, security issues, languages, new interface elements, important (and unimportant) features as well as providing tools for choosing a search engine or evaluating an existing one.”

Avi began with similarities and differences of enterprise search and Web search. The differences include:  limited scope, fewer meaningful hyperlinks for link analysis. Security and access control issues, content in databases, more control (for specifying value ranking, etc.), and no search spam.

Next she covered text search vs. data base search. Text search indexes multiple content sources and uses simple search commands instead of SQL. There is flexible indexing and retrieval and relevance ranking (major issue). There are new features such as spell check, auto completes, and facets. It works in the real world (e.g., eBay, Google).

Then she covered how information architecture works with search.  Information architecture is the art and science of organizing information for access and use. It creates order and systems and provides standard vocabulary. Search can supplement information architecture through user vocabularies and dynamic changes with new content.   

KM and search are opposite ways of approaching content. KM organizes stuff, and search finds it.   There are two main types of search: known item with short queries and “good enough” answers and exploratory search for research purposes. This is what Darwin Ecosystems excels at.  It can help you discover relationships between content that you did know exits and were not explicitly looking for. Avi said that search is an iceberg and people often see it as magic.

It is useful to index everything as it is hard to know in advance what people want.  Twitter has changed expectations for real time indexing, even for intranets. Three minutes is a good expectation. Here is another impact of consumer Web on enterprise computing.

Index security is an issue. (see my post: Attivio Aligns with Traction and Releases New Features) Without the right security you can see stuff you should not see. Need to work with security people on search issues to avoid this and have capabilities in the search tool.  The first step requires knowing what needs to be controlled and then you can determine how to do this.  Be aware of privacy laws.

After you determine what to secure, then deal with access control. Best to keep access control info as part of document store (what Attivio does - see post mentioned above). There are four levels of access control. One – access to search engine, two – collection-level access control (to portions of the search engine), three – locked results for a teaser for subscription, four – hit-level access control – link to a access control database at the point of display - hardest to do, useful when constantly changing rules.

Robot spiders start with a base URL for all hosts. For each page they repeat this process: read text info into internet format, save document in cache, save words into index, extract all links and check for rules, if they are new URLs add them to the list. It can repeat process over and over.

There are common problems with robots that SEO tries to avoid. Spiders can be disallowed by robots.txt or robots meta. Also cannot handle URLs with ? and & (but all spiders should handle these now), Javascript, forms, and interactive dynamic links, session IDs that change, multiple views of same data (wikis and Lotus Notes). 

External sources that have APIs like Twitter can be brought into enterprise search. You might want to partition this so it does not clutter standard search.  This is one way you can use Darwin Ecosystem inside the enterprise to look outside it. Relevance is relative. (sounds obvious but need to remember this when creating relevance listing).

Indexing multimedia needs to be dealt with now. There can be internal and external metadata to support this.  Best to use human judgment rather than automated systems. Automated systems can be a starting point but they need to be fine tuned by people. Speech to text and other automated capabilities are still buggy.

Stop words are common terms or ubiquitous terms. Traditionally you excluded them but there are consequences – such as copyright mentions. Best to index everything, especially since storage is much cheaper now. Avi gave a good example of excluding stop words by searching for phrase “whatever well be,” a song title.  On the other hand you can lots of irrelevant stuff.  Another example, the rock band, The Who, Here is where relevance can help so you get a lot if you include stop words but only need to see best ten examples. I tired this on Google and it did work as the top results related to the band even though there were 457,000,000. Avi said that Google may have set up an exception for this term. She also said you might get different results on wordpress.com.

Dealing with duplicate documents can be complex. First you need to decide what is a duplicate and then what is the primary if there are some slight differences (e.g. typo corrections).  Exact match is easy but similarity is more useful, harder but worth it.  Best to remove duplicates from index and hide results unless requested. This is what Google does.  Can create rules for handling duplicates. This is a good idea. However, you need human supervision but it is worth it.

Avi went through the search process: search form – query parser – query engine (goes to inverted index and back)  – relevance ranker (goes to document store, get stuff and brings back – formatter – search results.  This all happens very fast now. Queries come from many sources, not simply search fields. There are alerts, saved searches, automated searches, geographic information systems, and others. You need to balance relevance and completeness. You cannot have both.

Relevance ranking algorithms work differently. The most common is TF-DF or term frequencies : inverse document frequency. How often is the query word in document and how often is word in the index? There are others but this is most efficient. Look at title, metadata, and top of document. Remember relevance is task specific. There is no such thing as objective relevance. You can never please everyone. More like berry picking than hunting, Try different stuff instead of locking on single goal. Here is where Darwin Ecosystem can help with correlations of different topics with target key words.

Be sure to limit the user interface complexity. Google is a great example. Use familiar use interface elements. Put search into navigation so it appears everywhere. With auto-complete, use a drop down menu of matching words. Base this on search logs and use 7 - 10 most popular in alphabetic order

In summary, with enterprise search you have much more control on the capabilities and decisions with your search capability than on the Web. Make good use of these decisions. 

Darwin Ecosystem’s computing architecture is mindful of reducing the carbon footprint.

Since the conception and first iteration of our Awareness Engine, we have considered the importance of leveraging Cloud Computing for our scalability and cost management.  One of the by-products of this decision is a direct benefit to our planet; reducing our computing carbon footprint. 

The global IT industry generates about 2 percent of all global carbon dioxide emissions, or about as much greenhouse gases as the world's airlines, according to a recent Gartner study.   As technology advances, and demand increases globally, we are all faced with the challenge of how to reduce the carbon imprint our industry contributes to global warming and climate change.

This Saturday, we had the honor of  being invited to the Maple Leaf Ball in Boston where Quebec’s Prime Minister, Jean Charest, made a compelling speech about clean energy and Quebec’s concrete initiatives to leverage their hydro-electricity and other clean tech to service Canadian provinces and the United States.  His speech, and recent actions in New Brunswick,  Prince Edward Island, and Vermont not only demonstrate that Quebec is committed to this mission, but it also reinforced our strategic decision to open our research and development in Montreal.  To be located in such a progressive and action-driven location regarding clean-tech will only enhance our ability to see that our technology becomes environmentally friendly as our research will clearly be supported with the will of government initiatives and technical talent.

According to recent research our decision to adopt Cloud Computing makes Darwin Ecosystem an environment-conscious company.  Nonetheless Thierry Hubert, co-founder and driving force behind our technology’s direction, is convinced that Cloud Computing alone is not enough to answer the growing need for near-real-time data processing power.  In fact he believes that computing energy is being wasted on idled computers and that the future of computing does not reside in data-centers (whether they be in Mountain View or on the Cloud - wouldn’t this be  going backwards to the era of mainframes?).  As such, Darwin Ecosystem explores new ways to leverage distributed/grid computing models to further reduce the carbon footprint as it delivers timely and interest-drive information to its users and communities.

Darwin Ecosystem Brings Awareness Engine™ to the Web and within the Enterprise

The introduction of Web 2.0 social media into the enterprise creates large amounts of potentially useful information about the social side of business processes. The key is gaining awareness and access to this social intelligence. Paula Thronton recent covers this issue nicely in The Context of “Intent.” She opens with, “Two differentiating attributes of 2.0 are adaptation and emergence. Adaptive systems rely on feedback loops for continuous assessment. Emergence is the result of self-organizing adaptation.”  

Paula later adds, “The goal is to bring together relevant facts to inform discovery (the possibilities) that then lead to design — especially adaptive design to support individuals interacting with or on behalf of a business. Such facts are often difficult to find and difficult to effectively interpret and leverage.” And then offers, “A more 2.0 approach would bring the facts into the context they’re related to, featuring (draw attention to via teasers) certain findings in tidbits, leading to more detail.” 

We think that enterprise 2.0 provides a wealth of social “data” as a byproduct of its use. Companies are only beginning to figure out how to harvest this content. The organizations that do this are truly working in the enterprise 2.0 space. of course the same opportunity for discovering new and useful content and the relationships between content applies to the Web. Increasing awareness in context and finding related content is the goal of Darwin Ecosystem. Its awareness engine™ gives users the ability to perceive and be conscious of events and patterns of activities captured in the enterprise and Web 2.0. The introduction of Web 2.0 tools into the enterprise also opens up possibilities for even more information silos if these tools do not connect. Darwin provides a way to see emerging patterns of related content across information generated through a variety of tools. There is both a Web edition and an enterprise edition. The Web edition is available on our web site through free registration. The enterprise edition is available through our adopters program. 

With the enterprise edition, Darwin reduces the effort of keeping-up with enterprise 2.0 content (Information Overload Management). It classifies and correlates patterns of the business activities trapped in your Web. Through its Scan Cloud™ Darwin makes visible and measurable the value of the enterprise 2.0 content (Awareness and Monitoring). You can see related items in a tag cloud like visualization and this relationship shifts as you move through the cloud. This dynamic nature makes it easy for users to see the correlation across the enterprise 2.0 content (Discovering and Sharing).

Using correlationmetrics based on Chaos Theory, Darwin looks for the emergence of correlated themes within chaotic content. This moves Darwin away from the voting and popularity rankings used by many information aggregators and search engines to rank information. By using correlation metrics, it does not require a known process or taxonomy to discover useful and related information. Darwin allows for the emergence that Paula discussed. You can set up ongoing filters or “attractors” to explore emerging themes in specific topics of interest or simply look at the broad picture. 

Darwin is a not a replacement of existing enterprise 2.0 technologies, dashboards, and document management tools. It is designed to complement and leverage these technologies by making their tacit knowledge more visible. Darwin looks at events, blogs and other Web 2.0 sources that may correlate with the enterprise actions, its competitors and the voice of its critics and customers. This allows the enterprise to be capable of more easily discovering its own knowledge assets as well as its market and competitive positioning.

Darwin’s Awareness Engine™ , a program manager in Marketing becomes aware of concerns in R&D, Sales or other initiatives that are emerging as he/she tries to promote a product (all without having to wait and depend the next meeting or coffee break encounter).  They can also see competitive moves that may correlate with their own plans or the efforts of their customers and prospects. Likewise management can have a high-level view of what topics are emerging across all divisions and initiatives to better steer the business or measure the effectiveness of the vision’s execution, in the process discovering emerging and stronger initiatives or employees that are noteworthy. 

It is a Web browser application (Scan Cloud™) or it can become a custom solution through API access. It is delivered through a Web server with services and a database correlating the different Web 2.0 sources. For the enterprise there is an on-premise solution running on Ruby on Rails and making use of RSS feeds. Its Virtual Cortex™ database can be set on Oracle, MS-SQL or mySQL according to scalability needs. You can go from inside out so an enterprise can correlate their initiatives, knowledge and business intelligence against the pulse of the Web. The application is currently in Alpha stage and available for use.

Let's take a look at how it operates by looking at an example from the Web edition. We set up a query on the term - social media - for the last few days and saw the word - gain - was correlated to it as shown in the cluster below. This relationship interested us and we looked to see why gain is associated with social media by selecting gain and highlighting the cluster of correlations between social media and gain.

Then we saw three articles that matched this correlation in the past few days as you see in the image below. 

There was a story about how NASDAQ has launched a social network site and we went to the story shown blow. This was something we had not heard about and was very interesting to us. 

We also saw an article that reported that in the UK social networking sites account for 25% of display ads. We had read recently that ad revenue on the Web had passed tV ad revenue in the UK so this was especially significant.  There was also an article on using Facebook traffic to drive brand loyalty as you can see above. None of these articles appeared on a Google search for social media plus gain. Nor did they appear on Google News on the topic. Below is screen shot of the complete Darwin interface so you can see where the details above appeared and their relationships. 

This is a sample of how you can discover relevant new information through correlation that is hard, if not impossible to find in traditional search. It takes a very different approach to discover than the traditional key word search as you can find content you did not know to search for.  

You can also find stories  about themes that are emerging in the Web (or breaking news), as well as images that correlate with content as shown in results about a very recent helicopter crash in the Pacific shown below. 

In addition, Darwin allows you to correlate content in traditional mainstream media with what is happening in blogs and other social media. All of these features can be applied to the Web or to information stored behind the firewall through your own dedicated server.

In 2008 Darwin won a Young Entrepreneurs Award from the Office for Science and Technology of the French Embassy in the United States. The Young Entrepreneurs Initiative (YEI) is a platform for mentoring and networking US based entrepreneurs who understand the importance of internationalizing their vision and their activity, and who wish to set up a technology venue in France.