Here is a list of common fields we index with Lucene. Certain applications may have other fields to add to their index, but most should be able to provide a value that represents these fields.

We also have several custom fields for source code metadata in the index.

The social bookmarking tags field could be interesting to search on, but social bookmarking tags are really supposed to be an alternative to free text search, so this isn't something we've pursued for the current interface.

Whenever content is created or updated on one of our web sites, the new content is submitted to our indexing queue with a call to the web service. If the specified unique ID already exists in the index, the current entry is deleted, then the content is reindexed. Lucene is very fast at indexing search content, and most requests happen in real time. However, if many indexing requests are being made at the same time, they are put into a queue to be processed when less demand is being placed on the indexer.

To ensure content gets indexed, most of our web systems also maintain their own queue of items to be resubmitted later to the Lucene index if the indexing request fails for any reason.

Example: Indexing blogs

Here's a relevant PHP snippet from batch indexing our Wordpress-base blog content:

private function indexPost( $site, $blogID, $blogName, $postID, $authorName,
$body, $title, $modifiedDate )
{
echo( "  Indexing post $postID ($authorName): $title\r\n" );
$contentID = $site . '.' . $blogName . '.' . $postID;
// Convert to standardized date format.
$modifiedDate = str_replace( ' ', '', str_replace( ':', '',
str_replace( '-', '', $modifiedDate ) ) );
// Make sure that everything we're passing is valid UTF-8
$authorName = validateUtf8( $authorName );
$title = validateUtf8( $title );
$body = validateUtf8( $body );
$comments = validateUtf8( $this->getPostComments( $blogID, $postID ) );
$startTime = microtime( true );
$resultObj = $this->sc->indexHTMLContentEx( 'blog', $contentID,
$authorName, $title, $title, $body, $modifiedDate, 'en', $comments,
'', '', '', '', '' );
$endTime = microtime( true );
$result = $resultObj->result;
$message = $resultObj->message;
if( !$result )
throw new Exception( "Indexing failed, error: " . $message );
// Track call time.
if( count( $this->callTimes ) == 1000 )
array_shift( $this->callTimes );
$this->callTimes[] = ( $endTime - $startTime );
++$this->postCount;
}

Example: Indexing CodeCentral

Here is a C# sample that calls our web service, based on CodeCentral's metadata:

private static long indexCC()
{
DbConnection con = DbUtils.CreateConnectionName("CodeCentral");
DateTime started = DateTime.Now;
Console.WriteLine("Indexing CodeCentral at "
+ DateTime.Now.ToShortTimeString());
long recCount = 0;
try
{
using (DbCommand cmd = DbUtils.MakeCommand(con,
"select snippet_id, u.first_name, u.last_name, s.title, s.shortdesc,"
+ "s.description, s.updated, s.comments, p.description, s.low_version,"
+ "s.high_version, c.description "
+ "from snippet s, category c, product p "
+ "inner join users u on s.user_id = u.user_id "
+ "where s.category_id = c.category_id "
+ "and s.product_id = p.product_id"))
{
using (DbDataReader reader = DbUtils.InitReader(cmd))
{
while (reader.Read())
{
try
{
// recCount++;
// indexer.indexHTMLContentEx(
using (StreamWriter sr = new StreamWriter(
string.Format("{0}\\current.txt",
Path.GetDirectoryName(Application.ExecutablePath))))
{
sr.Write(reader[0].ToString());
}
ZipFile file;
try
{
file = CCZipUtils.GetZipFile(con as TAdoDbxConnection,
reader.GetInt32(0),
ConfigurationSettings.AppSettings["cacheRootPath"]);
}
catch
{
file = null;
}
List<SearchResult> result = indexHtmlContent(
"cc",  // appid
reader[0].ToString(), // contentid
reader[1].ToString(), //first name 
reader[2].ToString(), //last name
reader[3].ToString(), // title
reader[4].ToString(), // summary
reader[5].ToString(), // body
reader.GetDateTime(6), //pubdate
"en", //language
reader[7].ToString(), // comments
reader[8].ToString(), // product
string.Format("{0} to {1}", reader.GetDouble(9),
reader.GetDouble(10)), // version
"", // tags
reader[11].ToString(), // category
"", // extradata
"", // contenttype
"", // workaround
file,
recCount++
);
foreach (SearchResult item in result)
{
if (!item.Result)
{
badIDs.AppendFormat("cc:{0} - {1}{2}",
item.ContentID, item.Message,
Environment.NewLine);
}
}
}
catch (Exception e)
{
badIDs.AppendFormat("cc:{0} - {1}{2}", reader[0],
e.Message, Environment.NewLine);
}
}
}
}
}
finally
{
con.Close();
}
showTiming(started, recCount);
return recCount;
}

Example: Indexing QualityCentral

The QualityCentral web service is built with Delphi native code. Although the automatic indexing has not been deployed for the QualityCentral web service yet, we do have a reindexing application that can be used to index all reports in QualityCentral. Here's the routine that submits the content to the search engine:

procedure TQCIndexDataModule.IndexSince(AFromDate: TDateTime);
var
TickStart,
TickTime : Cardinal;
SearchWS: Search;
FldID,
FldFirstName,
FldLastName,
FldTitle,
FldDescription,
FldSteps,
FldModified,
FldProject,
FldVersion,
FldDataType,
FldWorkaround : TField;
recs: Int64;
Name: UnicodeString;
Result : BooleanResult;
begin
SearchWS := GetSearch;
WriteLn('Started at ' + DateTimeToStr(Now));
Recs := 0;
TickStart := GetTickCount;
try
QCReports.Active := False;
QCReports.ParamByName('FromDate').asDateTime := AFromDate;
QCReports.Active := True;
FldID := QCReports.FieldByName('defect_no');
FldFirstName := QCReports.FieldByName('first_name');
FldLastName := QCReports.FieldByName('last_name');
FldTitle := QCReports.FieldByName('short_description');
FldDescription := QCReports.FieldByName('description');
FldSteps := QCReports.FieldByName('steps');
FldModified := QCReports.FieldByName('modified_date');
FldProject := QCReports.FieldByName('project');
FldVersion := QCReports.FieldByName('version');
FldDataType := QCReports.FieldByName('data_type');
FldWorkaround := QCReports.FieldByName('workaround');
while not QCReports.EOF do
begin
Name := FullName(FldFirstname.AsWideString, FldLastName.AsWideString);
try
Result := SearchWS.indexContentEx('qc', fldID.AsString, Name,
FldTitle.AsWideString, FldDescription.asWideString,
fldSteps.AsWideString, LuceneDateTime(FldModified.AsDateTime),
'en', QCCommentText(FldId.AsInteger), FldProject.AsWideString,
FldVersion.AsWideString, '', FldDataType.AsWideString,
'', '', FldWorkAround.AsWideString );
if Result.result then
begin
Inc(Recs);
if Recs Mod 30 = 0 then
WriteLn(Recs,'#', FldId.AsInteger, ':', Name, ':',
FldTitle.AsWideString);
end
else
WriteLn('#Error on QC#' + fldID.AsString + ':' + Result.message_);
except
on E: Exception do
WriteLn('#Error on QC#' + fldID.AsString + ':' + E.Message);
end;
QCReports.Next;
end;
finally
TickTime := GetTickCount - TickStart;
QCReports.Close;
WriteLn('Stopped at ' + DateTimeToStr(Now));
WriteLn('Records: ', Recs, ' Elapsed time: ' + TicksToTime(TickTime));
end;
end;

The Search type is the Delphi class for calling the web service generated from Delphi's WSDL importer. This is a console application, so once every 30 records the progress is displayed. We also track the amount of time it takes to create the index, and report on any exceptions we encounter.

The same method that deletes an index entry that already exists when a reindex request is submitted can be called directly via the web service. All of our content systems tell Lucene to remove index entries from the index when content is deleted.

Example: Removing QualityCentral reports from the index

Here's Delphi code for removing all QC reports from the search index. All that needs to be specified is the application ID and the unique contentID, which in this case is the report id.

procedure TQCIndexDataModule.ClearIndex;
var
SearchWS: Search;
FldID: TField;
begin
SearchWS := GetSearch;
QCAllReports.Close;
try
QCAllReports.Open;
FldID := QCAllReports.FieldByName('defect_no');
while not QCAllReports.EOF do
begin
SearchWS.deleteContent('qc', FldID.AsString);
QCAllReports.Next;
end;
finally
QCAllReports.Close;
end;
end;

Of course, the purpose of indexing all this content is to be able to find it when you search for it., and Lucene makes that very easy. However, displaying the search result is a bit more complicated than that, due to the complexity of our data models and requirements to conform with visibility rules of the systems using our search engine.

Therefore, application-specific searches are resolved in 2 passes:

Applications submit keyword search expressions to Lucene, which would return the list of IDs whose keywords match the required criteria. The search expression can be customized with additional fields restrictions such as Title, Abstract, Code, Comments, Workarounds, etc.

The returned list of IDs from the search expression is then further filtered based on additional metadata values (some of these could be Lucene fields as well if you want to frequently update your index), such as products, versions, author, site, publish date, popularity, ratings, entitlement, and so on.