Case insensitive search support

It is not possible to create a case insensitive index in mongodb yet. Because of this, our case-insensitive searches are collections scans, and it is slow.

A possible hack to deal with this is to add case insensitive flag to metadata index definitions, and if a field has a case insensitive index, have mongo back end create a new hidden field with the lower case value of the original field. We can rewrite queries for fields in case-insensitive indexes to use the hidden field.

As an example, if an entity has a case insensitive index on 'field1', all documents would be saved with:

   ...
   field1: "field1Value",
   field1__lc: "field1value"

and case insensitive regex queries using "field1" will be translated to use "field1__lc". field1__lc (we can find a better naming scheme) won't be translated for retrieved documents, so this scheme is transparent and safe except for direct mongo updates.

---

If you reply to this email, your message will be added to the discussion below:

http://dev.forum.lightblue.io/Case-insensitive-search-support-tp409.html

To start a new topic under lightblue-dev, email [hidden email]
To unsubscribe from lightblue-dev, click here.
NAML

Index keys now have a caseInsensitive flag:

"indexes": [
    {
        "unique":true,
        "fields": [
            {"field":"field1",
             "dir":"$asc",
             "caseInsensitive":"true"
            },
            {"field":"field2",
             "dir":"$asc"
            }
        ]
    },
    {
        "unique":true,
        "fields": [
            {"field":"field3",
             "dir":"$asc",
             "caseInsensitive":"true"
            }
        ]
    }
]

The difference in how this is handled between lightblue and mongo is hidden from the user, but important.  The above caseInsensitive indexes will actually create indexes on the following fields:

`@mongoHidden.field1`
`@mongoHidden.field3`

And in turn, updates to the caseInsensitive fields will also update corresponding hidden fields:

{
  "field1": "testField1",
  "field2": "testField2",
  "field3": "testField3",
  "@mongoHidden.field1": "TESTFIELD1",
  "@mongoHidden.field3": "TESTFIELD3"
}

This is all entirely hidden from the user, and they will only be privy to the fields they've actually defined in metadata.

---

If you reply to this email, your message will be added to the discussion below:

http://dev.forum.lightblue.io/Case-insensitive-search-support-tp409p412.html

To start a new topic under lightblue-dev, email [hidden email]
To unsubscribe from lightblue-dev, click here.
NAML