Sitecore Rant: Lucene Index of System Fields

Lucene Index of System Fields

On a current project, there is a need to index the system language fields.  By default, there is an index that takes care of all items in "/sitecore/system" called "system_index".  This contains everything.  I don't want that.  I just want a list of all the languages so I created a new index and added to the "Sitecore.ContentSearch.Lucene.Indexes.Sharded.Master.config" file.  Simple enough.  I also have to define the fields that I want to have indexed.  To do that we open the "Sitecore.ContentSearch.Lucene.DefaultIndexConfiguration.config" file and define the fields that we want to index.




Notice that we have two kinds of fields in this list.  We have the default fields on top and three custom fields at the bottom.  For all custom fields that are added post-install, the field names are as is.  For most system fields, there are usually underscores either at the beginning of the name or in the middle.  Sometimes there are two leading underscores.  This is used to indicate system fields that are part of the standard template. The field names for "code_page", "regional_iso_code" and "worldlingo_language_identifier" are default system fields and would make sense to have underscores connecting the words.  Also, they are obtained from the section of excluded fields down below:



Note: Keep in mind that unless you remove the entry from the list of excluded fields, the field will not get indexed even if you tell it to in the previous screenshot.

The regional iso code is a good example to explain my rant.  This basically says to the indexer to ignore this field with the ID.  If you inspect the template for the system language, you can indeed see the ID does match up with the field so you would naturally believe the field name is also correct.  WRONG!  If you run the indexer, the value for this field will always be null because there is no such field with this name.  After debugging and looping through all the fields using item.Fields, we can see that the field name is simply "Regional Iso Code" and not "Regional_iso_code". 

Why would Sitecore decide to use this kind of notation for this field and others while some default fields look like this?



I know that there cannot be spaces in the html element tags but if you use underscores for some fields and camel case for others, it would lead us to believe that the underscores are deliberate and indicate the real field names.  If not, then why not just use camel case for all fields definitions in the excluded list?

Also, there is no way to know what the field names are unless you iterate through item.Fields for all the field names.  If you try looking at the config file above, you will be misguided very often, at least in the section of excluded fields.  The correct index entry for all the fields would be:

Sitecore: Set Up New Lucene Index and Search Result Class

Background

Lucene is great when you don't want to implement Solr.  Sure, it is not as powerful as Solr nor does it do faceting, but for all simple searches that return results, Lucene does the job well.  Both utilize the concept of creating indexes and querying for results against the indexes.

Requirements

Out-of-the-box, Sitecore defined indexes for general items like the content node, the media library, and templates across all three databases.  I am sure you can query the content node of the master database to get anything you want but you don't have to.  A better way is to create indexes that are less general and more specific and tailored to your needs. 

Set Up New Index

The general master database index is defined here:

Sitecore.ContentSearch.Lucene.Index.Master.config

The smaller indexes are boken down and defined here:

Sitecore.ContentSearch.Lucene.Indexes.Sharded.Master.config

If you want to define a new smaller index for the master database, this is where it should be defined.

Add a new index definition like this:



Here you will see that a custom crawler has been used.  You can refer to an older posting for more details about setting up custom crawlers:

http://mrstevenzhao.blogspot.com/2014/04/sitecore-custom-item-crawler.html

At this point, if you go to Sitecore admin control panel, you will see this new index listed when you navigate to the section for rebuilding indexes.

Set Up New Search Result Class

Sitecore has a BuilInFields class that contains index field names of general fields.  We can use those index fields to create a base class that only contains fields we care about.  Lets create one right now:




Please remember that all search result classes have to inherit from the SearchResultItem class which gives you even more fields and methods already defined there.

Now, we can create more custom classes based on top of the base result item class we just created.  Let's create one:



Keep in mind that Fields is just a static mapping class that stores key-value pairs for index field names.  Since these two new fields are not standard fields, we have to define these two new fields in the configuration file, Sitecore.ContentSearch.Lucene.DefaultIndexConfiguration.config.  Add this to the <fieldNames> section:



Now, Lucene knows about these two new fields and where to store the indexed values under.  Rebuild the index and the values for these new fields will be available to query against.

Summary

It is quite easy and beneficial to add new indexes.  If you add custom fields in your templates that need to be indexed, you would have to create custom indexes and search result classes to handle the new fields.  Also, you can isolate your queries away from the main master database index which also speeds up your queries and speeds up index rebuilds as well.