Wednesday, August 8, 2007

Clustering with Search Engines

When I first read about the Clustering, I was confused about its utility and ability to work on limited set of search results. But over the period time, after reading lot of research material on search usability and taking to people, I realized that searcher do not go beyond few pages. In fact the study shows that 2/3 of searcher do not go beyond 2 pages of 10 results each. The searcher either find the information that they are looking for or change the search terms. I personally do not go beyond 2-3 pages unless I am not able to refined search phrase. Even with 2 pages of result set, users have filter and infer the context of search results to find for the relevant information that they are looking for. So actually the users are spending more time on search result windows rather than actually working on searched information. When I am searching, it is not only important how relevant the search results are, but also how much time it took to me to get to those results. Here I see value of clustering with search engines.

Search technology has been evolving and maturing over last few years. The search companies are completing with each other for creating a niche for themselves and attracting more traffic to get benefit from advertising, but ever increasing demand of end users are always overtaking them from behind. The users are getting interested in federated search, clustering and faceted search apart from the regular sequential search. Google web search provides clustering as indented search results when it find more search results from same site or site section. Some other search engines like Vivisimo Velocity provide Topic Clustering, or grouping results into topics/subjects, that help in refine searches.

Can we use clustering for anything else? We can use clustering to build applications like job sites, event sites, social networking etc. Also we can generate tag cloud to guide users navigate through popular topics. The list can grow as we see more need for applications.

I was working with Nutch and Google search appliance, when I got interested in the clustering and faceted search technologies. I wanted to integrate the clustering technology with both these search engines to show my clients the value of clustering technology. I gave them demo with Clusty web search and they see lot of value. But question was, can I integrate it with Google search appliance and is there any free or low cost tool that can provide the same functionality. I knew about Vivisimo, but client wanted a cheaper solution, so I started digging more in the open source arena.

Then I came across Carrot2, Open Source Search Results Clustering Engine. Search Carrot2 provides an architecture for acquiring search results from various sources (YahooAPI, GoogleAPI, MSNAPI, eTools Meta Search, Alexa Web Search, PubMed, OpenSearch, Lucene index, SOLR), clustering the results and visualising the clusters. Currently, 5 clustering algorithms are available that are suitable for different kinds of document clustering tasks. Carrot2 has been successfully used in a number of commercial and research applications and resulted in a number of interesting publications.

I found this tool interesting and wanted to use it with my existing search engines. The tool provides seamless integration with nutch and lucene search engines. The developer only needs to point to existing search indexes and customize the page layout. The carrot application is deployed as webapp which you can be drop in any web application server. The deployment was cake walk and results were fascinating. Then I also got it working with the Google search appliance. Here I had to use its Java APIs to build a clustered search interface from the results of Google appliance.

If anyone is looking for low cost clustering solution, one can use open source clustering engine which can be integrated with any existing web search engines and also with enterprise search engines like Google search appliance, lucene index (one can debate on its enterprise capability). It is easy to deploy and configure and does not impose any extra baggage.

I am sure Google must be working on this technology and would come with a solution which is a notch better than Vivisimo or Carrot2. I am eagerly waiting to hear from them.

Friday, August 3, 2007

Enterprise Search Done Right

When I was doing an assignment with a leading energy research company on knowledge management assessment, I learnt how important a search could be for the business and how much a company could go wrong in implementing a search solution. The key to success in research and consulting organizations is effective implementation of knowledge management practice and search plays a important role in it. Enterprise search has direct impact on the business and productivity of employees in these type of organization.

When I were there, I looked at their infrastructure and their business processes, and tried to assess the maturity level of their knowledge management program. The results were appalling. They did not have a knowledge management program. They did not have enterprise content and collaboration management system. They did not have internal search. Their external search engine despite of having two searches was redundant. They did not understand the significance of search for their employees as well as business with respect to productivity and knowledge access. The researcher procured information from various knowledge databases and repositories through a library services. There were not processes in place to measure the effectiveness and efficiency of these library services. The researchers and scientist were so dependent on Google and Yahoo's for information related to their work.

This is not the only story of, how organizations can go wrong in understanding the user needs and implementing ineffective search solutions. I have browsed through so many organization's websites that have great products and services but lack in providing effective search solutions for their users. I do and do not blame them. There is so much work done on external search engines like Google, Yahoo that the expectation of end users have gone up for all kind of searches. When they browse organization search sites, they expect similar user experience and relevance as they get on external web search engines. I do not mean to say that there aren't enterprise search engines who provide similar relevance and experience as external search engines. There are search products from Fast, Verity and Autonomy that leaders in enterprise search.

If I need to suggest a tool that would solve the research organization's search problem, first I would try to understand their business and user needs. Then I would list the features that I need in my product. Lets drill down on the requirements:
1. Ability to search all my repositories including websites, file systems, databases, enterprise applications like SharePoint, Documentum etc. They researcher will be able to access of information including past work done as well knowledge acquired from various sources.
2. Ability to search on external Corporate and academic libraries, journals, feeds etc. It is also called federated search. They do not need to go to various search engines to get information.
3. Ability to classify my documents and content in easy to browse categories. This would help me drill down on the information based on categories rather than pages with 10 results. No one actually browse beyond couple of pages on search.
4. Web administrative interface with advance linguistic capabilities including metadata, synonyms, antonyms, relative weighting for text fields and stemming.
5. Ability to perform secure searches.
6. Ability to scale and perform.

The rest of requirements are generic like relevance, supporting formats, summary extraction, metadata indexing and crawling configuration. If I have a tool that provides solution to all the requirements, certainly it is a candidate for my approval.

When I did the research, I found Vivísimo Velocity Search Engine as one of most powerful enterprise search that satisfy my set of requirements and a great relevance algorithm as compared to Google/Yahoo. There were other who did come close but not close enough. They were either not able to satisfy all requirements or not packaged as single solution.

Vivísimo, a difficult word to pronounce, provides innovative search solution that not only provides same search relevance as compared to the external web search engines but also provides similar user experience. You can access their public web search at http://vivisimo.com/ to get a glimpse. Their clustering solution is also popularly known as Clusty.

Integral components of the Vivísimo Velocity Search Platform:
  • Velocity Search Engine
  • Velocity Content Integrator
  • Velocity Clustering Engine

The external web search of Vivísimo looks impressive and I just hope the enterprise search is as promising as their web search. I think anyone looking for enterprise search engine should take a look at their offering.

I have done lot of research of federated search and clustering technology which are commercial as well open source. I will write on these technologies in coming posts.

Tuesday, July 31, 2007

ISYS:web - Enterprise Search for Small and Medium Business

Yesterday ISYS Search Software announced Enterprise Search Support for Interwoven WorkSite. Last month, they had announced integration of ISYS:web 8 into Microsoft Office SharePoint Server 2007 (MOSS). When I read the news release, I got curious to know more about the ISYS and its market in the Enterprise search.

ISYS Search Software is based out of Sydney, Australia and has been business since 1998. Its major focus has been in Small & Medium Business (SMB) and Government sector. Currently ISYS has over 10,000 customers on seven continents, including Antarctica and was selected as 'one of the 100 most significant Australian innovations of the twentieth century' by the Powerhouse Museum in 1999.

When I looked into its product portfolio, I discovered following products:
1. ISYS:Desktop - indexing and retrieving tool for desktop and laptop primary windows based.
2. ISYS:web - designed for indexing and searching on public and intranet websites.
3. ISYS:sdk - SDK toolkit to support development

What interested me most is their ISYS:web product, an Enterprise Search Engine, suited for mid-market segment. It is out-of-box search tool with small foot-print, an easy installer, and can be customized and configured to meet complex enterprise needs. It compares with Google Search Appliance from pricing perspective and meet features and functionality of full blown enterprise search engine like Verity and Fast.

From platform perspective, it is based on Windows and supports Windows XP, Windows 2000, Windows 2003 server, Vista operating systems. That is one of the reasons, it can easily provide integration points to Interwoven WorkSite and Microsoft SharePoint Server, both windows based products.

What impressed me most, is its rich set of features. It provides a range of search, navigation and discovery tools all bundled into a single platform. One can execute natural language queries or easily construct advanced Boolean and proximity queries via point-and-click operators. Navigate, drill down and instantly locate the right information via On-The-Fly Categorization, ‘search within’ queries, hit highlighting and hit-to-hit navigation. Discover associations and connections between your search terms and the entities ISYS has automatically identified within your search results with ISYS Entities. Administrators will also benefit from ISYS:web’s easy-to-deploy, web-based administration, which includes features like the ISYS Site Designer and Search Designer, federation of remote ISYS indexes, Best Bets, and ISYS SearchTrends for measuring and analyzing search patterns.

It can seamlessly search multiple disparate data sources, including Microsoft Office, WordPerfect, Open Office, HTML, ZIP files, all major email products, all SQL data sources, SharePoint, Lotus Notes and Lotus Domino.

I wanted to try it myself, ISYS Search Software website provides search through its own search engine and I also found some more organization using ISYS, including City of San Jose and Town of Frisco. ISYS also have impressive list of customers, including Boeing and Cisco Systems.

When I look at the ISYS:web, it has all required features of Enterprise search engine for SMB and with low cost of ownership. Even the pricing model is quite different from tools in same category. Most of search engines in this category are per document based licensing or appliance model. ISYS:web has base license cost and an additional per user ($100) licensing model. In addition, it claims to have provision for volume licensing. If I am company with 100 employees, my total cost of ownership will $1000' + $100*100 = $11000, which is significantly lower than other search engines.

Though I am impressed and want to install and run the Search engine myself, I am also skeptical about its performance and scalability. Whatever it turns out to be, but it still meets its objectives of rapid return and low cost of ownership.

I am going to explore more on its navigational (auto-classification, faceted search) and discovery features and share with you.

Monday, July 30, 2007

Google gets straight with Autonomy

There have been numerous debates on the capabilities of Google as an truly effective enterprise search engine. I have read also that some of the largest enterprises are thinking of replacing or actually replacing Google as their enterprise search. I came to know that Cisco is one of them. On the other side, I also have seen organizations replacing other enterprise search engines with Google Appliance citing relevance as one of their concerns. NASA is one to replace Verity with Google Appliance on their portal.

The reasons as I know, why Google has not been taken seriously in enterprise search were:
1. Google Appliance search is not as effective as their web search. People are used to of using Google web search.
2. Google enterprise search are only able to index web pages, not databases, file systems etc.
3. Google enterprise search is not secured

There may be other reasons, but these are the one that I have heard and read more frequently.

Autonomy blasted Google on some of the limitation that have been listed above in their white paper which was published sometime ago. Also read Google has 'dumbed-down search', says Autonomy boss. The reasons, Autonomy feels threatened by Google enterprise search capability which is growing at the phenomenon rate. I do not have exact figures, but it seems Google already has over 9000 customers and lead the enterprise search market segment.

In the response to the allegations, Google published Don't believe everything you read on their enterprise blog. I think it would clear doubts in minds of readers who read the white paper and more so believed in what was written. I do not blame the readers as not all of them get chance to try new technology as it evolves or progresses. They believe in what is printed. Google did great job in responding to all the issues in details.

I need to ask Autonomy that why they are targeting Google alone, why not compare it with our world class search engines. Secondly, not long time ago Autonomy refused to even consider Google in the same league, why now.

I think this war of words will do lot of good to Google and its credibility as enterprise search engine.

Monday, July 23, 2007

EU authorized Germany to give $165 million for research

The European Union on Thursday authorized Germany to give $165 million for research on Internet search-engine technologies that could someday challenge U.S. search giant Google Inc.

The project is named Theseus and it has been designed to create an advanced multimedia search engine for the next-generation Internet.

The EU would also be paying the funds to companies including Siemens AG, SAP AG, Deutsche Thomson oHG and EMPOLIS GmbH for this research purpose. European companies in general spend far less on research than those based in other parts of the world, and the EU said the project should help change that.

Google spokeswoman Katie Watson had this to say on the new development: "We welcome all efforts to help democratize access to information."

Google has been aggressively working on enhancing it's multimedia search by including books, video content. Google's Book search beta edition is live from May last year and provides searching and rendering of books online. Google Video and Youtube lets searches on the rich media content including TV shows, movies, music videos, documentaries, personal productions and more.

The funding will help in bringing new strategy and technology for searching the multi-media content. Thanks EU. Other countries should also funds the local researchers and universities so that they can help researching in technologies that is most suited to the local market. It will also help in stopping monopolization the search market.

Sunday, July 22, 2007

Microsoft: SingleView augments SharePoint

Microsoft announced a new search product developed in partnership with consulting giant BearingPoint at Microsoft's Worldwide Partner Conference here on July 10.

A new product, SingleView, that augments Microsoft SharePoint Server 2007. According to press release, the new product is already been used by dozens of Enterprise customers.

What is new in this product which is not supported in existing search of MOSS 2007? MOSS 2007 search provides full text indexing and IFilters interfaces to support meta data search in the Microsoft Office documents. It is still limited to searching the data stored in its SQL Server and Microsoft-related servers.

The SingleView will act as master search index for all kinds of enterprise data, including data that isn't stored on any Microsoft-related servers such as Oracle databases, document management archives, SAP and other CRM (customer relationship management) repositories, e-mails and other sources of information.

"Enterprise search is a very attractive business for us and our partners," said Jared Spataro, the group product manager for enterprise search at Redmond, Wash.-based Microsoft. The software company says SharePoint has more than 85 million seats sold to date, and continues to experience a strong demand for future growth.

Enterprise search is also more complex because unlike general Web search, it isn't just looking for keywords in a massive pile of data. The difference is that most enterprise searches are looking at the relationships among different—and disparate—pieces of data. Enterprise Search Engines have evolved over the period of time while trying to solve real world. The Enterprise Search Engines have matured while engaging in the assignments with the clients. None of search engines can claim to 100% out-of-box solution for any mid-size or large organization. It is usually 60-80% depending on the complexity of problem of client.

There are other companies like Coveo and other, that have built search services around MOSS 2007. This is clear indication that MOSS search engine is not matured and feature-rich enough to solve enterprise problems.

I am yet to see any collateral on SingleView on the Microsoft site. I think it's great initiative from Microsoft to compete with other Enterprise search companies like Google Search Appliance, Autonomy etc.

The question here is "whether SingleView will always be packaged with MOSS or can become a standalone Enterprise search tool".

Friday, July 20, 2007

Google adds Custom Search Business Edition

On Tuesday, Google added a new Customer Search Business Edition (CSBE) to its search offering. It is a lower-end complement to Google's enterprise-targeted appliances targeting small businesses who are not willing to pay even $2000 for Google Mini. The new Google search edition is just $100 for searching upto 500 pages and $500 for pages up to 50,000 pages. Higher volume is supported but the pricing model is not yet announced.

The key feature of Google CSBE:

1. Hosted search edition for small business at very nominal cost
2. Add-free search lets small business to do their own branding
3. Quick setup through online wizard
4. Customization feature on website for branding
5. Online reports and web analytics
about visitor behavior
6. $100 for upto 500 pages, $500 for upto 50,000 pages
7. XML feed for search results
8.
Make refinements to help categorize search results
9. Subscription
10. Internationalization
11. Payment integrated with Google checkout

Some of the limitations of this edition:

1. No control on indexed content - crawling as well as indexing
2. No scheduling or control on indexing of content - specially for events

If I have a small company with no or limited IT resources and budget, this is great option I have. The advantages are more than the limitations of Google CSBE. I still feel it is better option for small businesses with no or limited IT resources over IBM OmniFind Yahoo Edition.

But IBM OmniFind Yahoo Edition is not a hosted solution and can be alternative to overcome the limitations of Google CSBE. We are not trying to compare oranges to apples. Both products are trying to solve small and medium businesses but not in the same category.