|
摘自:
http://edn.embarcadero.com/article/38723
By: John Kaster
Abstract: A guide to implementing Lucene as a cross-platform, cross-language search engine
Implementing a Lucene search engine
If you are interested in implementing a search engine based on Lucene, you may be interested in these notes compiled during our research.
Lucene
The Lucene FAQ is an excellent resource for learning about integration options and determining the best approaches to take. Some of the relevant questions and answers are:
Reading this FAQ before starting on a Lucene implementation is highly recommended.
Indexing
Lucene requires a unique ID for each content record it indexes. It uses Java's native Unicode string support for parsing text. Analyzers are available for various languages, including support for CJK languages (Chinese, Japanese, Korean, and some Vietnamese).
Lucene can also be used to index source code with a custom Analyzer, but we modified YAPP to provide what we needed. You can also generate an AST that can be consumed by Lucene, similar to what was done with this solution for Java source code.
Here's an example from the Lucene JavaDoc (which, unfortunately does not have convenient direct URLs) showing a complete use case (or Unit test, in this example):
Analyzer analyzer = new StandardAnalyzer();
Directory directory = new RAMDirectory();
IndexWriter iwriter = new IndexWriter(directory, analyzer, true );
iwriter.setMaxFieldLength(25000);
Document doc = new Document();
String text = "This is the text to be indexed." ;
doc.add(new Field("fieldname" , text, Field.Store.YES,
Field.Index.TOKENIZED));
iwriter.addDocument(doc);
iwriter.optimize();
iwriter.close();
IndexSearcher isearcher = new IndexSearcher(directory);
QueryParser parser = new QueryParser("fieldname" , analyzer);
Query query = parser.parse("text" );
Hits hits = isearcher.search(query);
assertEquals(1, hits.length());
for (int i = 0; i < hits.length(); i++) {
Document hitDoc = hits.doc(i);
assertEquals("This is the text to be indexed." , hitDoc.get("fieldname" ));
}
isearcher.close();
directory.close();
Some other information I pulled directly from their JavaDoc intro is included here.
The Lucene API is divided into several packages:
- org.apache.lucene.analysis defines an abstract Analyzer API for converting text from a java.io.Reader into a TokenStream, an enumeration of Tokens. A TokenStream is composed by applying TokenFilters to the output of a Tokenizer. A few simple implemenations are provided, including StopAnalyzer and the grammar-based StandardAnalyzer.
- org.apache.lucene.document provides a simple Document class. A document is simply a set of named Fields, whose values may be strings or instances of java.io.Reader.
- org.apache.lucene.index provides two primary classes: IndexWriter, which creates and adds documents to indices; and IndexReader, which accesses the data in the index.
- org.apache.lucene.search provides data structures to represent queries (TermQuery for individual words, PhraseQuery for phrases, and BooleanQuery for boolean combinations of queries) and the abstract Searcher which turns queries into Hits. IndexSearcher implements search over a single IndexReader.
- org.apache.lucene.queryParser uses JavaCC to implement a QueryParser.
- org.apache.lucene.store defines an abstract class for storing persistent data, the Directory, a collection of named files written by an IndexOutput and read by an IndexInput. Two implementations are provided, FSDirectory, which uses a file system directory to store files, andRAMDirectory which implements files as memory-resident data structures.
- org.apache.lucene.util contains a few handy data structures, e.g., BitVector and PriorityQueue.
To use Lucene, an application should:
- Create Documents by adding Fields;
- Create an IndexWriter and add documents to it with addDocument();
- Call QueryParser.parse() to build a query from a string; and
- Create an IndexSearcher and pass the query to its search() method.
Some simple examples of code which does this are:
Fields to Index
Here is a list of common fields we index with Lucene. Certain applications may have other fields to add to their index, but most should be able to provide a value that represents these fields.
|
|
|
ID of the application being indexed. This could be "gp" for GetPublished, "cc" for CodeCentral, "qc" for QualityCentral, "blog" for Blogs, etc.
|
|
ID of the discrete item being indexed. This would be SNIPPET_ID in CodeCentral, VERSIONID in GetPublished, etc. The unique "LuceneID" will be <AppID>.<ID>.
|
|
The full name of the author of the content
|
|
|
|
Short description, summary, or abstract of item
|
|
Main contents of the item
|
|
Publication, or modified, timestamp of the item. There are trade-offs for date precision storage.
|
|
2 character code for the Human language used for the text of the item
|
|
Optional. Product(s) for which the item applies.
|
|
Optional. Version(s) of the product for which the item applies.
|
|
Optional. Social bookmarking tags entered for the item.
|
|
Optional. Category(ies) for the item.
|
|
|
We also have several custom fields for source code metadata in the index.
The social bookmarking tags field could be interesting to search on, but social bookmarking tags are really supposed to be an alternative to free text search, so this isn't something we've pursued for the current interface.
Searching
Analysis
We want to track user searches to find out what kinds of searches are most popular, and what our most popular keywords are, so we can offer suggestions to search engine users. This requires more research to learn how Lucene supports analysis and reporting of usage patterns, or we may need to just track it ourselves.
A very handy tool to use when working with Lucene indices is Luke, the Lucene Index Toolbox.
Application and Database Integration
Lucene can directly index database records with any available JDBC connection. Custom SQL statements are required for retrieving the fields and records to be indexed, but that appears to be pretty easy and flexible.
By default, Lucene indexes plain text content. We provide either HTML or plain text values to Lucene, which our Document Adapter already supports.
Our web service interface
Since Lucene is written in Java, we created a web service to call it from our non-Java applications. The Lucene for .NET project appears to have stagnated, and since JBuilder makes it so easy to create a web service, the web service is the best way to make it available for all of our platforms and languages.
These are some of the relevant functions of our web service:
- Index the specified content ID and text
- Index the specified content ID and html
- Support "extra data" such as QualityCentral workarounds, products, versions, etc.
- Perform a search from the specified search expression, returning all matching IDs and scoring
- Refine an existing search result with the specified search expression
We are hoping to make our search web service publicly available in the future, and also provide a browser plug-in and IDE integration to it in the future.
You can download the source code to our web service from CodeCentral.
Application interaction
There are only three interactions an application needs to have with the search web service: 1) indexing new or updated content, 2) removing an index entry for deleted content, and 3) retrieving and processing search results.
Indexing content
Whenever content is created or updated on one of our web sites, the new content is submitted to our indexing queue with a call to the web service. If the specified unique ID already exists in the index, the current entry is deleted, then the content is reindexed. Lucene is very fast at indexing search content, and most requests happen in real time. However, if many indexing requests are being made at the same time, they are put into a queue to be processed when less demand is being placed on the indexer.
To ensure content gets indexed, most of our web systems also maintain their own queue of items to be resubmitted later to the Lucene index if the indexing request fails for any reason.
Example: Indexing blogs
Here's a relevant PHP snippet from batch indexing our Wordpress-base blog content:
private function indexPost( $site, $blogID, $blogName, $postID, $authorName,
$body, $title, $modifiedDate )
{
echo ( " Indexing post $postID ($authorName): $title\r\n" );
$contentID = $site . '.' . $blogName . '.' . $postID;
$modifiedDate = str_replace( ' ' , '' , str_replace( ':' , '' ,
str_replace( '-' , '' , $modifiedDate ) ) );
$authorName = validateUtf8( $authorName );
$title = validateUtf8( $title );
$body = validateUtf8( $body );
$comments = validateUtf8( $this ->getPostComments( $blogID, $postID ) );
$startTime = microtime( true );
$resultObj = $this ->sc->indexHTMLContentEx( 'blog' , $contentID,
$authorName, $title, $title, $body, $modifiedDate, 'en' , $comments,
'' , '' , '' , '' , '' );
$endTime = microtime( true );
$result = $resultObj->result;
$message = $resultObj->message;
if ( !$result )
throw new Exception( "Indexing failed, error: " . $message );
if ( count( $this ->callTimes ) == 1000 )
array_shift( $this ->callTimes );
$this ->callTimes[] = ( $endTime - $startTime );
++$this ->postCount;
}
Example: Indexing CodeCentral
Here is a C# sample that calls our web service, based on CodeCentral's metadata:
private static long indexCC()
{
DbConnection con = DbUtils.CreateConnectionName("CodeCentral" );
DateTime started = DateTime.Now;
Console.WriteLine("Indexing CodeCentral at "
+ DateTime.Now.ToShortTimeString());
long recCount = 0;
try
{
using (DbCommand cmd = DbUtils.MakeCommand(con,
"select snippet_id, u.first_name, u.last_name, s.title, s.shortdesc,"
+ "s.description, s.updated, s.comments, p.description, s.low_version,"
+ "s.high_version, c.description "
+ "from snippet s, category c, product p "
+ "inner join users u on s.user_id = u.user_id "
+ "where s.category_id = c.category_id "
+ "and s.product_id = p.product_id" ))
{
using (DbDataReader reader = DbUtils.InitReader(cmd))
{
while (reader.Read())
{
try
{
using (StreamWriter sr = new StreamWriter(
string .Format("{0}\\current.txt" ,
Path.GetDirectoryName(Application.ExecutablePath))))
{
sr.Write(reader[0].ToString());
}
ZipFile file;
try
{
file = CCZipUtils.GetZipFile(con as TAdoDbxConnection,
reader.GetInt32(0),
ConfigurationSettings.AppSettings["cacheRootPath" ]);
}
catch
{
file = null ;
}
List<SearchResult> result = indexHtmlContent(
"cc" ,
reader[0].ToString(),
reader[1].ToString(),
reader[2].ToString(),
reader[3].ToString(),
reader[4].ToString(),
reader[5].ToString(),
reader.GetDateTime(6),
"en" ,
reader[7].ToString(),
reader[8].ToString(),
string .Format("{0} to {1}" , reader.GetDouble(9),
reader.GetDouble(10)),
"" ,
reader[11].ToString(),
"" ,
"" ,
"" ,
file,
recCount++
);
foreach (SearchResult item in result)
{
if (!item.Result)
{
badIDs.AppendFormat("cc:{0} - {1}{2}" ,
item.ContentID, item.Message,
Environment.NewLine);
}
}
}
catch (Exception e)
{
badIDs.AppendFormat("cc:{0} - {1}{2}" , reader[0],
e.Message, Environment.NewLine);
}
}
}
}
}
finally
{
con.Close();
}
showTiming(started, recCount);
return recCount;
}
Example: Indexing QualityCentral
The QualityCentral web service is built with Delphi native code. Although the automatic indexing has not been deployed for the QualityCentral web service yet, we do have a reindexing application that can be used to index all reports in QualityCentral. Here's the routine that submits the content to the search engine:
procedure TQCIndexDataModule.IndexSince(AFromDate: TDateTime);
var
TickStart,
TickTime : Cardinal;
SearchWS: Search;
FldID,
FldFirstName,
FldLastName,
FldTitle,
FldDescription,
FldSteps,
FldModified,
FldProject,
FldVersion,
FldDataType,
FldWorkaround : TField;
recs: Int64;
Name: UnicodeString;
Result : BooleanResult;
begin
SearchWS := GetSearch;
WriteLn('Started at ' + DateTimeToStr(Now));
Recs := 0;
TickStart := GetTickCount;
try
QCReports.Active := False;
QCReports.ParamByName('FromDate' ).asDateTime := AFromDate;
QCReports.Active := True;
FldID := QCReports.FieldByName('defect_no' );
FldFirstName := QCReports.FieldByName('first_name' );
FldLastName := QCReports.FieldByName('last_name' );
FldTitle := QCReports.FieldByName('short_description' );
FldDescription := QCReports.FieldByName('description' );
FldSteps := QCReports.FieldByName('steps' );
FldModified := QCReports.FieldByName('modified_date' );
FldProject := QCReports.FieldByName('project' );
FldVersion := QCReports.FieldByName('version' );
FldDataType := QCReports.FieldByName('data_type' );
FldWorkaround := QCReports.FieldByName('workaround' );
while not QCReports.EOF do
begin
Name := FullName(FldFirstname.AsWideString, FldLastName.AsWideString);
try
Result := SearchWS.indexContentEx('qc' , fldID.AsString, Name,
FldTitle.AsWideString, FldDescription.asWideString,
fldSteps.AsWideString, LuceneDateTime(FldModified.AsDateTime),
'en' , QCCommentText(FldId.AsInteger), FldProject.AsWideString,
FldVersion.AsWideString, '' , FldDataType.AsWideString,
'' , '' , FldWorkAround.AsWideString );
if Result.result then
begin
Inc(Recs);
if Recs Mod 30 = 0 then
WriteLn(Recs,'#' , FldId.AsInteger, ':' , Name, ':' ,
FldTitle.AsWideString);
end
else
WriteLn('#Error on QC#' + fldID.AsString + ':' + Result.message_);
except
on E: Exception do
WriteLn('#Error on QC#' + fldID.AsString + ':' + E.Message);
end ;
QCReports.Next;
end ;
finally
TickTime := GetTickCount - TickStart;
QCReports.Close;
WriteLn('Stopped at ' + DateTimeToStr(Now));
WriteLn('Records: ' , Recs, ' Elapsed time: ' + TicksToTime(TickTime));
end ;
end ;
The Search type is the Delphi class for calling the web service generated from Delphi's WSDL importer. This is a console application, so once every 30 records the progress is displayed. We also track the amount of time it takes to create the index, and report on any exceptions we encounter.
Removing an index entry
The same method that deletes an index entry that already exists when a reindex request is submitted can be called directly via the web service. All of our content systems tell Lucene to remove index entries from the index when content is deleted.
Example: Removing QualityCentral reports from the index
Here's Delphi code for removing all QC reports from the search index. All that needs to be specified is the application ID and the unique contentID, which in this case is the report id.
procedure TQCIndexDataModule.ClearIndex;
var
SearchWS: Search;
FldID: TField;
begin
SearchWS := GetSearch;
QCAllReports.Close;
try
QCAllReports.Open;
FldID := QCAllReports.FieldByName('defect_no' );
while not QCAllReports.EOF do
begin
SearchWS.deleteContent('qc' , FldID.AsString);
QCAllReports.Next;
end ;
finally
QCAllReports.Close;
end ;
end ;
Retrieving and processing search results
Of course, the purpose of indexing all this content is to be able to find it when you search for it., and Lucene makes that very easy. However, displaying the search result is a bit more complicated than that, due to the complexity of our data models and requirements to conform with visibility rules of the systems using our search engine.
Therefore, application-specific searches are resolved in 2 passes:
Applications submit keyword search expressions to Lucene, which would return the list of IDs whose keywords match the required criteria. The search expression can be customized with additional fields restrictions such as Title, Abstract, Code, Comments, Workarounds, etc.
The returned list of IDs from the search expression is then further filtered based on additional metadata values (some of these could be Lucene fields as well if you want to frequently update your index), such as products, versions, author, site, publish date, popularity, ratings, entitlement, and so on.
A look back in time
The previous search engine used by CodeCentral, QualityCentral, and SiteCentral (our web site presentation engine) is based on a Delphi component I wrote in 1998 for searching CodeCentral. The architecture of this search technology remained basically the same since that time. Keywords are persisted to a database, and searches are performed with nested SQL select statements that combine the keyword filtering results (AND, OR, and NOT) with other criteria for the various databases using this keyword search engine.
Because we know the metadata we're searching, the results were generally good, but the only options is for keywords are the AND, OR, NOT operations, and wildcards.
This search engine had only minor updates to it as we moved from native Delphi code to .NET. The "2.0" version was going to include frequency scoring, proximity searching, and Unicode. I was done with the general-purpose Unicode keyword breaking and persistence engine classes, and was starting to work on the updated query generation code when I decided to revisit Lucene. I'm really glad I did.
|