Open Source IR with Xapian

Richard Boulton

Lemur Consulting Ltd

History

• Project started 7 years ago
• Replacement for Muscat software
• Released under the GPL in 2000
• Name changed to Xapian in 2001
• Hoping to relicense (probably as MIT or new BSD)

Overview

So, what is it?

• Toolkit for IR
• Index documents and search them
• Associated tools / applications
• C++ core, bindings for many languages
• Community, adoption increasing

Features

Various weighting schemes, for ranked searches:

• Traditional probabilistic weighting
• BM25 ("early adopter")
• Looking at supporting DfR weighting schemes
• Would like to support new research ideas
• Can also sort in other orders - eg, date
• All combined with structured boolean

Features

Lots of other features:

• Relevance feedback and query expansion
• Phrase and proximity searching
• Duplicate removal
• Stemming (snowball) and wildcards

Features

• Platform independent database formats
• Simultaneous update and searching
• Atomic, reliable, update
• Multiple database search
• Networked search

Features

Speed / Scalability

• Index of 500 million web pages (1.5Tb)
  • Small clusters of servers
  • Search results in under a second
• More recently, Gmane
  • 34 million mail messages

Who is using it?

• Gmane - mailing list archive
• BeebleX - PHP information archive
• Die Zeit - German Newspaper
• Tweakers.net - the largest Dutch ICT-website
• TheyWorkForYou.com - Hansard archives
• Recoll - Destop Search
• MoinMoin - wiki software
• ...

Who would we like to use it?

More IR researchers please...

• Open source
• Extensible
• Scalable to large collections
• Avoids reinventing the wheel
• We're interested in new research ideas

Developer team

• Active and helpful developer community
• Commercial support available
  • Oligarchy
  • Lemur Consulting
• Collaborate on email
• Shared, public, code repository

Web site search

Abstract:

"We give a detailed case study of Xapian, of how to create a web site search mechansim."

• So lets do that...

Omega

• Files in the filesystem
• Index with omindex --db MyDb --url http://mysite.org/ /home/www/mysite.org
• Search with Omega CGI

Dynamic sites

• Generated pages
• Need to crawl
• Often multiple, remote, servers
• Javascript / Flash - argh!

Security

• Some users cannot see some pages
• Crawler needs to be let in
• Can sometimes implement in Xapian
• But sometimes need to talk to authentication server
• Hybrid?

Speed?

• Used to aim for 1 second
• Now, "immediate response"
• 1/25 second would be good
• 1/10 second acceptable
• Disk access times around 1/100 second

Speed

HOW?

• Caching
• Just hold everything in memory
• Clustered / distributed architecture
• Memory bandwidth becomes an issue

Contact us

• Website: http://www.xapian.org/
• Email lists: xapian-discuss, xapian-devel
• Commercial support
  • Oligarchy Ltd
  • Lemur Consulting Ltd
• Or talk to us now!