Associations

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Associations

bserdar
After much back and forth about how to implement ad-hoc queries and cascades, I gave up, and came up with a less ambitious version of association implementation for the fist cut:

Scope:
  - Reference fields in metadata
  - Association implementation for find() API

Things to consider for future:
  - Cascading updates
  - ad-hoc associations in query requests


Associations  between entities  are defined  in metadata  as reference
fields.  The contents of a  reference field is populated by projecting
a  view of  another entity.

field : {
  "type": "object" | "array",
  "reference" : {  
      "entity": entityName,  
      "versionValue": versionValue,  
      "projection": projection_expression,  
      "query": query_expression,  
      "sort": sort  
    }  
}

Field type must be an array or an object. An object field denoted a
1-to-1 association. An array field denotes a 1-n association.

projection and sort: These specify what parts of the referenced entity
should be injected into the referencing entity. If the association is
1-to-n, sort specifies in what order the result set should be
injected. Paths should be evaluated relative to the reference
field. That is, when the projection refers to field "X", that is the
top-level field "X" of the referenced entity.

query: Paths should be evaluated relative to reference field, and any
reference to the enclosing field should be through the use of $parent
paths.

TODO: When saving/updating metadata, make sure the referenced entity
and version exists, and enabled. Also, evaluate the projection, query,
and sort to make sure they are valid. These should be evaluated on the
composite metadata containing all referenced entities.

Metadata preparation: the metadata for the referenced entity is
attached to the field tree of the enclosing entity, forming a
composite metadata of entities. We have to make sure there are no
cyclic references. One can traverse the fields of the referenced
entity as if it is defined as part of the metadata of the enclosing
entity.

Association computation: If reference field type is object, there must
be only one item in the result set (1-1 association). If there are
multiple results, execution should fail. If field type is array,
result is an object array. No value arrays.

Associations will be computed by the Mediator. The outline of the algorithm is:
  * Compute composite metadata
  * Determine retrieval strategy using reference queries. This
    needs research and experimentation to find out how to do it.
  * Execute the strategy

Q: should we assume an association is always 1-n? (i.e. a reference is always an array)

Reply | Threaded
Open this post in threaded view
|

Re: Associations

jewzaam
Administrator
Why include projection in the reference?  The query for the entity containing the reference must provide a projection for the reference.

Regarding object vs array, it would be simpler to assume array always, but it isn't necessarily true that it will always be multiples coming back.  For example, an order will reference the user that placed the order.  There will only ever be one user that placed the order.  I like the thought of getting an object if there are not multiple but it would be tricky if it's done dynamically.  For example, if user has many addresses and the addresses are references.  What of the case when a user has only one address?  The client must then handle getting an object or getting an array, depending on what the specific user reference result set is.  I think then having the type defined as you have it is reasonable.
Reply | Threaded
Open this post in threaded view
|

Re: Associations

bserdar
On Thu, Aug 21, 2014 at 12:38 PM, jewzaam [via lightblue-dev]
<[hidden email]> wrote:
> Why include projection in the reference?  The query for the entity
> containing the reference must provide a projection for the reference.

To prevent cycles. If entity X has a field 'f" referencing Y, and Y
has a field "g" referencing X, you'd want to have "g" exclude "f" in
X.

>
> Regarding object vs array, it would be simpler to assume array always, but
> it isn't necessarily true that it will always be multiples coming back.  For
> example, an order will reference the user that placed the order.  There will
> only ever be one user that placed the order.  I like the thought of getting
> an object if there are not multiple but it would be tricky if it's done
> dynamically.  For example, if user has many addresses and the addresses are
> references.  What of the case when a user has only one address?  The client
> must then handle getting an object or getting an array, depending on what
> the specific user reference result set is.  I think then having the type
> defined as you have it is reasonable.

If we assume refs are always arrays, and if the relationship is in
fact 1-1, that array would always have at most 1 elements. In a way,
we're deferring the dealing of having multiple elements in a
supposedly 1-1 relationship to the caller. Having an array for refs
simplifies it for us that all associations are potentially 1-n.

>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://lightblue-dev.1011138.n3.nabble.com/Associations-tp91p92.html
> To start a new topic under lightblue-dev, email
> [hidden email]
> To unsubscribe from lightblue-dev, click here.
> NAML
Reply | Threaded
Open this post in threaded view
|

Re: Associations

jewzaam
Administrator
The projection restriction could be done automatically though.  If X references Y and Y then references X the association code could simply filter it out.  Downside is if the instance of X could be different.  Example:

User -> Address
Address -> Company
Company -> Users

In theory, a request for a user could ask for an address and that address's company plus users.  Anyway, when you get beyond a single layer of references things get complicated and it doesn't seem like something I would want to manage with projection.  And it may not be appropriate to filter out the original user in that final list of users on the company.  Key points I guess are I like simple and not restrictive, but understand protecting the client from their own mistakes.  The risk is probably with wildcard and/or recursive projections.  Can something be restricted with those for referenced data?

Re arrays:  Yes, just was pointing out as a client that would be annoying, but it would be consistent.  From a consistency and simplicity point of view going with arrays all the time makes sense.  In that case, a new type could be created named "reference" instead of using "array".
Reply | Threaded
Open this post in threaded view
|

Re: Associations

bserdar
Lets do these:
  1) New data type: reference. Implies array
  2) Projection in reference is optional. If not given, all fields are included.
  3) No cyclic references. If there is a cyclic reference, we break
the last edge of the object tree causing cycle. So, in your eample, if
you retrieve a user, we'll get the addresses and companies, but not
users of companies. If you retrieve a company, we retrieve its user
and address, but not the companies of the address.

Any objections?

On Fri, Aug 22, 2014 at 7:09 AM, jewzaam [via lightblue-dev]
<[hidden email]> wrote:

> The projection restriction could be done automatically though.  If X
> references Y and Y then references X the association code could simply
> filter it out.  Downside is if the instance of X could be different.
> Example:
>
> User -> Address
> Address -> Company
> Company -> Users
>
> In theory, a request for a user could ask for an address and that address's
> company plus users.  Anyway, when you get beyond a single layer of
> references things get complicated and it doesn't seem like something I would
> want to manage with projection.  And it may not be appropriate to filter out
> the original user in that final list of users on the company.  Key points I
> guess are I like simple and not restrictive, but understand protecting the
> client from their own mistakes.  The risk is probably with wildcard and/or
> recursive projections.  Can something be restricted with those for
> referenced data?
>
> Re arrays:  Yes, just was pointing out as a client that would be annoying,
> but it would be consistent.  From a consistency and simplicity point of view
> going with arrays all the time makes sense.  In that case, a new type could
> be created named "reference" instead of using "array".
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://lightblue-dev.1011138.n3.nabble.com/Associations-tp91p96.html
> To start a new topic under lightblue-dev, email
> [hidden email]
> To unsubscribe from lightblue-dev, click here.
> NAML
Reply | Threaded
Open this post in threaded view
|

Re: Associations

jewzaam
Administrator
What if the reference in an entity is to provide hierarchy?  Product X has parent product Y which has parent product Z.  With what you propose there's no way to get X with it's parent Y and grandparent Z.
Reply | Threaded
Open this post in threaded view
|

Re: Associations

bserdar
Good point. I can think of a few ways to deal with it:
  1) Don't allow cyclic references. In the product example, the
metadata maintainer should define parent relations as regular fields,
not as reference, and has to deal with it.
  2) Allow cyclic references, and evaluate them with the understanding
that they have to be explicitly projected in the request. Also, don't
let recursive projections cross entity boundaries, so if the
projection calls for recursive inclusion of all fields, we go only one
level deep.
    So, a recursive all inclusive projection in the request won't
retrieve any references. To retrieve references, you have to also
include a projection including the reference explicitly.
  3) A cleaner way of implementing #2 is to come up with another type
of projection, a "reference projection" expression that you have to
include if you want references retrieved.



On Fri, Aug 22, 2014 at 9:36 AM, jewzaam [via lightblue-dev]
<[hidden email]> wrote:

> What if the reference in an entity is to provide hierarchy?  Product X has
> parent product Y which has parent product Z.  With what you propose there's
> no way to get X with it's parent Y and grandparent Z.
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://lightblue-dev.1011138.n3.nabble.com/Associations-tp91p105.html
> To start a new topic under lightblue-dev, email
> [hidden email]
> To unsubscribe from lightblue-dev, click here.
> NAML
Reply | Threaded
Open this post in threaded view
|

Re: Associations

jewzaam
Administrator
4) don't restrict anything and fail if an instance of an entity is seen more than twice in a result set I say twice to allow for case of user->site->customer->users where the first top level entity user is also included in the customer's users.
5) don't restrict anything and let hystrix command timeouts to kill long running requests.  Would want to test that the request is killed on the server in that case.  Given that hystrix kills the thread running the command this should happen, but it would be very bad if it didn't.




On Fri, Aug 22, 2014 at 11:53 AM, bserdar [via lightblue-dev] <[hidden email]> wrote:
Good point. I can think of a few ways to deal with it:
  1) Don't allow cyclic references. In the product example, the
metadata maintainer should define parent relations as regular fields,
not as reference, and has to deal with it.
  2) Allow cyclic references, and evaluate them with the understanding
that they have to be explicitly projected in the request. Also, don't
let recursive projections cross entity boundaries, so if the
projection calls for recursive inclusion of all fields, we go only one
level deep.
    So, a recursive all inclusive projection in the request won't
retrieve any references. To retrieve references, you have to also
include a projection including the reference explicitly.
  3) A cleaner way of implementing #2 is to come up with another type
of projection, a "reference projection" expression that you have to
include if you want references retrieved.



On Fri, Aug 22, 2014 at 9:36 AM, jewzaam [via lightblue-dev]
<[hidden email]> wrote:

> What if the reference in an entity is to provide hierarchy?  Product X has
> parent product Y which has parent product Z.  With what you propose there's
> no way to get X with it's parent Y and grandparent Z.
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://lightblue-dev.1011138.n3.nabble.com/Associations-tp91p105.html
> To start a new topic under lightblue-dev, email
> [hidden email]
> To unsubscribe from lightblue-dev, click here.
> NAML



If you reply to this email, your message will be added to the discussion below:
http://lightblue-dev.1011138.n3.nabble.com/Associations-tp91p106.html
To start a new topic under lightblue-dev, email [hidden email]
To unsubscribe from lightblue-dev, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: Associations

bserdar
For #5: the only way to kill a thread is to interrupt it, If the
thread submitted a long running query, interruption won't close the
mongodb connection, because it is pooled by the driver, and does not
belong to the thread. So, I believe interrupting a thread will still
wait for the query to return.

On Fri, Aug 22, 2014 at 2:33 PM, jewzaam [via lightblue-dev]
<[hidden email]> wrote:

> 4) don't restrict anything and fail if an instance of an entity is seen more
> than twice in a result set I say twice to allow for case of
> user->site->customer->users where the first top level entity user is also
> included in the customer's users.
> 5) don't restrict anything and let hystrix command timeouts to kill long
> running requests.  Would want to test that the request is killed on the
> server in that case.  Given that hystrix kills the thread running the
> command this should happen, but it would be very bad if it didn't.
>
>
>
>
> On Fri, Aug 22, 2014 at 11:53 AM, bserdar [via lightblue-dev] <[hidden
> email]> wrote:
>>
>> Good point. I can think of a few ways to deal with it:
>>   1) Don't allow cyclic references. In the product example, the
>> metadata maintainer should define parent relations as regular fields,
>> not as reference, and has to deal with it.
>>   2) Allow cyclic references, and evaluate them with the understanding
>> that they have to be explicitly projected in the request. Also, don't
>> let recursive projections cross entity boundaries, so if the
>> projection calls for recursive inclusion of all fields, we go only one
>> level deep.
>>     So, a recursive all inclusive projection in the request won't
>> retrieve any references. To retrieve references, you have to also
>> include a projection including the reference explicitly.
>>   3) A cleaner way of implementing #2 is to come up with another type
>> of projection, a "reference projection" expression that you have to
>> include if you want references retrieved.
>>
>>
>>
>> On Fri, Aug 22, 2014 at 9:36 AM, jewzaam [via lightblue-dev]
>> <[hidden email]> wrote:
>>
>> > What if the reference in an entity is to provide hierarchy?  Product X
>> > has
>> > parent product Y which has parent product Z.  With what you propose
>> > there's
>> > no way to get X with it's parent Y and grandparent Z.
>> >
>> > ________________________________
>> > If you reply to this email, your message will be added to the discussion
>> > below:
>> > http://lightblue-dev.1011138.n3.nabble.com/Associations-tp91p105.html
>> > To start a new topic under lightblue-dev, email
>> > [hidden email]
>> > To unsubscribe from lightblue-dev, click here.
>> > NAML
>>
>>
>> ________________________________
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://lightblue-dev.1011138.n3.nabble.com/Associations-tp91p106.html
>> To start a new topic under lightblue-dev, email [hidden email]
>> To unsubscribe from lightblue-dev, click here.
>> NAML
>
>
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://lightblue-dev.1011138.n3.nabble.com/Associations-tp91p111.html
> To start a new topic under lightblue-dev, email
> [hidden email]
> To unsubscribe from lightblue-dev, click here.
> NAML