Case insensitive search support

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Case insensitive search support

bserdar
It is not possible to create a case insensitive index in mongodb yet. Because of this, our case-insensitive searches are collections scans, and it is slow.

A possible hack to deal with this is to add case insensitive flag to metadata index definitions, and if a field has a case insensitive index, have mongo back end create a new hidden field with the lower case value of the original field. We can rewrite queries for fields in case-insensitive indexes to use the hidden field.

As an example, if an entity has a case insensitive index on 'field1', all documents would be saved with:

   ...
   field1: "field1Value",
   field1__lc: "field1value"

and case insensitive regex queries using "field1" will be translated to use "field1__lc". field1__lc (we can find a better naming scheme) won't be translated for retrieved documents, so this scheme is transparent and safe except for direct mongo updates.
Reply | Threaded
Open this post in threaded view
|

Re: Case insensitive search support

jewzaam
Administrator
This sounds reasonable to me.

On Thu, Jan 7, 2016 at 12:48 AM bserdar [via lightblue-dev] <[hidden email]> wrote:
It is not possible to create a case insensitive index in mongodb yet. Because of this, our case-insensitive searches are collections scans, and it is slow.

A possible hack to deal with this is to add case insensitive flag to metadata index definitions, and if a field has a case insensitive index, have mongo back end create a new hidden field with the lower case value of the original field. We can rewrite queries for fields in case-insensitive indexes to use the hidden field.

As an example, if an entity has a case insensitive index on 'field1', all documents would be saved with:

   ...
   field1: "field1Value",
   field1__lc: "field1value"

and case insensitive regex queries using "field1" will be translated to use "field1__lc". field1__lc (we can find a better naming scheme) won't be translated for retrieved documents, so this scheme is transparent and safe except for direct mongo updates.


If you reply to this email, your message will be added to the discussion below:
http://dev.forum.lightblue.io/Case-insensitive-search-support-tp409.html
To start a new topic under lightblue-dev, email [hidden email]
To unsubscribe from lightblue-dev, click here.
NAML
Reply | Threaded
Open this post in threaded view
|

Re: Case insensitive search support

bvulaj
In reply to this post by bserdar
Reply | Threaded
Open this post in threaded view
|

Re: Case insensitive search support

bvulaj
Index keys now have a caseInsensitive flag:

"indexes": [
    {
        "unique":true,
        "fields": [
            {"field":"field1",
             "dir":"$asc",
             "caseInsensitive":"true"
            },
            {"field":"field2",
             "dir":"$asc"
            }
        ]
    },
    {
        "unique":true,
        "fields": [
            {"field":"field3",
             "dir":"$asc",
             "caseInsensitive":"true"
            }
        ]
    }
]

The difference in how this is handled between lightblue and mongo is hidden from the user, but important.  The above caseInsensitive indexes will actually create indexes on the following fields:

`@mongoHidden.field1`
`@mongoHidden.field3`

And in turn, updates to the caseInsensitive fields will also update corresponding hidden fields:

{
  "field1": "testField1",
  "field2": "testField2",
  "field3": "testField3",
  "@mongoHidden.field1": "TESTFIELD1",
  "@mongoHidden.field3": "TESTFIELD3"
}

This is all entirely hidden from the user, and they will only be privy to the fields they've actually defined in metadata.
Reply | Threaded
Open this post in threaded view
|

Re: Case insensitive search support

jewzaam
Administrator
Is it possible to get access to these by creating a field in lightblue metadata that matches the upper case field name?

On Tue, Feb 2, 2016 at 1:24 PM bvulaj [via lightblue-dev] <[hidden email]> wrote:
Index keys now have a caseInsensitive flag:

"indexes": [
    {
        "unique":true,
        "fields": [
            {"field":"field1",
             "dir":"$asc",
             "caseInsensitive":"true"
            },
            {"field":"field2",
             "dir":"$asc"
            }
        ]
    },
    {
        "unique":true,
        "fields": [
            {"field":"field3",
             "dir":"$asc",
             "caseInsensitive":"true"
            }
        ]
    }
]

The difference in how this is handled between lightblue and mongo is hidden from the user, but important.  The above caseInsensitive indexes will actually create indexes on the following fields:

`@mongoHidden.field1`
`@mongoHidden.field3`

And in turn, updates to the caseInsensitive fields will also update corresponding hidden fields:

{
  "field1": "testField1",
  "field2": "testField2",
  "field3": "testField3",
  "@mongoHidden.field1": "TESTFIELD1",
  "@mongoHidden.field3": "TESTFIELD3"
}

This is all entirely hidden from the user, and they will only be privy to the fields they've actually defined in metadata.


If you reply to this email, your message will be added to the discussion below:
To start a new topic under lightblue-dev, email [hidden email]
To unsubscribe from lightblue-dev, click here.
NAML
Reply | Threaded
Open this post in threaded view
|

Re: Case insensitive search support

bserdar
This case insensitive index implementation is mongo specific. Another
backend would implement it differently without hidden fields. So it
wouldn't be a good idea to expose those.

We can add a feature to define fields as upper/lower case.

On Mon, Feb 8, 2016 at 5:15 PM, jewzaam [via lightblue-dev]
<[hidden email]> wrote:

> Is it possible to get access to these by creating a field in lightblue
> metadata that matches the upper case field name?
>
> On Tue, Feb 2, 2016 at 1:24 PM bvulaj [via lightblue-dev] <[hidden email]>
> wrote:
>>
>> Index keys now have a caseInsensitive flag:
>>
>> "indexes": [
>>     {
>>         "unique":true,
>>         "fields": [
>>             {"field":"field1",
>>              "dir":"$asc",
>>              "caseInsensitive":"true"
>>             },
>>             {"field":"field2",
>>              "dir":"$asc"
>>             }
>>         ]
>>     },
>>     {
>>         "unique":true,
>>         "fields": [
>>             {"field":"field3",
>>              "dir":"$asc",
>>              "caseInsensitive":"true"
>>             }
>>         ]
>>     }
>> ]
>>
>> The difference in how this is handled between lightblue and mongo is
>> hidden from the user, but important.  The above caseInsensitive indexes will
>> actually create indexes on the following fields:
>>
>> `@mongoHidden.field1`
>> `@mongoHidden.field3`
>>
>> And in turn, updates to the caseInsensitive fields will also update
>> corresponding hidden fields:
>>
>> {
>>   "field1": "testField1",
>>   "field2": "testField2",
>>   "field3": "testField3",
>>   "@mongoHidden.field1": "TESTFIELD1",
>>   "@mongoHidden.field3": "TESTFIELD3"
>> }
>>
>> This is all entirely hidden from the user, and they will only be privy to
>> the fields they've actually defined in metadata.
>>
>> ________________________________
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://dev.forum.lightblue.io/Case-insensitive-search-support-tp409p412.html
>> To start a new topic under lightblue-dev, email [hidden email]
>> To unsubscribe from lightblue-dev, click here.
>> NAML
>
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://dev.forum.lightblue.io/Case-insensitive-search-support-tp409p413.html
> To start a new topic under lightblue-dev, email
> [hidden email]
> To unsubscribe from lightblue-dev, click here.
> NAML
Reply | Threaded
Open this post in threaded view
|

Re: Case insensitive search support

bvulaj
In reply to this post by jewzaam
There is a check to make sure that metadata doesn't contain any @mongoHidden prefixed fields.  @mongoHidden is now effectively a reserve word.

So, no, even if they wanted to, a user would not be able to expose these fields.