Lightblue Plugin Design

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Lightblue Plugin Design

dcrissman
In https://github.com/lightblue-platform/lightblue-rest/issues/162 a rough cut of plugin capability was added to lightblue. Essentially it is now possibly to add arbitrary jars to the lightblue runtime classpath, but the hooks, controllers, and datasources are still gated by the json configuration files (just as they have always been).

Notable problems with this implementation:
* No protection if duplicate plugins contain the same dependency jars
* No protection if a plugin includes a duplicate dependency of lightblue itself.
* No SecurityManager to protect system resources from plugins
* Is implemented in lightblue-rest, but probably belongs in lightblue-core
* Ran into a casting issue when attempting to run lightblue-mongo as a plugin: https://github.com/lightblue-platform/lightblue-mongo/issues/214 
** Regardless of if mongo should be run as a plugin or not, I want to make sure other plugins don't run into similar issues


I have been thinking about these issues for a little while now and the remainder of this post are my thoughts for solutions. I am looking for any feedback you might have.


The research I have done points me to a separate "parent last" ClassLoader per plugin. Some testing will be needed, but I think that this will protected us from the first two bullet points. To achieve this we will need to know how a plugin is structured on the file system. So lets say we assume each plugin comes packaged in it's own directory, therefore all jar files in a given plugin directory belong in the same ClassLoader. For jars not in a directory, then we would need to assume each is a self-contained plugin and would need to be given its own ClassLoader.

The problem this causes is that we then have many ClassLoaders, and no real indication which is the correct one to use for any given class. I think the best solution is ServiceLoader, we would just need to iterate over each ClassLoader and ask it to instantiate all implementations of some interface in that ClassLoader. I don't know the full extent of what this affects, but it might remove or alter the need for the json configuration files as no longer would there be a need for something to tell lb some class exists and needs to be instantiated.

As for the SecurityManager if we use a lb specific sub-class of URLClassLoader, then in our SecurityManager implementation we check if the ClassLoader is an instance of our custom ClassLoader and then slap additional permissions on it if it is. Any thoughts as to what permissions we would want?

Any thoughts in general would be appreciated.

Thanks,
Dennis
Reply | Threaded
Open this post in threaded view
|

Re: Lightblue Plugin Design

jewzaam
Administrator
We deploy in EAP, what about modules?  Have they been investigated for loading these resources?  If so, what is wrong with them?

On Tue, Feb 23, 2016 at 4:22 PM dcrissman [via lightblue-dev] <[hidden email]> wrote:
In https://github.com/lightblue-platform/lightblue-rest/issues/162 a rough cut of plugin capability was added to lightblue. Essentially it is now possibly to add arbitrary jars to the lightblue runtime classpath, but the hooks, controllers, and datasources are still gated by the json configuration files (just as they have always been).

Notable problems with this implementation:
* No protection if duplicate plugins contain the same dependency jars
* No protection if a plugin includes a duplicate dependency of lightblue itself.
* No SecurityManager to protect system resources from plugins
* Is implemented in lightblue-rest, but probably belongs in lightblue-core
* Ran into a casting issue when attempting to run lightblue-mongo as a plugin: https://github.com/lightblue-platform/lightblue-mongo/issues/214 
** Regardless of if mongo should be run as a plugin or not, I want to make sure other plugins don't run into similar issues


I have been thinking about these issues for a little while now and the remainder of this post are my thoughts for solutions. I am looking for any feedback you might have.


The research I have done points me to a separate "parent last" ClassLoader per plugin. Some testing will be needed, but I think that this will protected us from the first two bullet points. To achieve this we will need to know how a plugin is structured on the file system. So lets say we assume each plugin comes packaged in it's own directory, therefore all jar files in a given plugin directory belong in the same ClassLoader. For jars not in a directory, then we would need to assume each is a self-contained plugin and would need to be given its own ClassLoader.

The problem this causes is that we then have many ClassLoaders, and no real indication which is the correct one to use for any given class. I think the best solution is ServiceLoader, we would just need to iterate over each ClassLoader and ask it to instantiate all implementations of some interface in that ClassLoader. I don't know the full extent of what this affects, but it might remove or alter the need for the json configuration files as no longer would there be a need for something to tell lb some class exists and needs to be instantiated.

As for the SecurityManager if we use a lb specific sub-class of URLClassLoader, then in our SecurityManager implementation we check if the ClassLoader is an instance of our custom ClassLoader and then slap additional permissions on it if it is. Any thoughts as to what permissions we would want?

Any thoughts in general would be appreciated.

Thanks,
Dennis


If you reply to this email, your message will be added to the discussion below:
http://dev.forum.lightblue.io/Lightblue-Plugin-Design-tp425.html
To start a new topic under lightblue-dev, email [hidden email]
To unsubscribe from lightblue-dev, click here.
NAML
Reply | Threaded
Open this post in threaded view
|

Re: Lightblue Plugin Design

dcrissman
You and I had talked about a desire to keep the plugin architecture from being tied to EAP. Because of that I have not spent time investigating EAP modules. Aside from our previous conversation though, I am not aware of anything that would be wrong with them. I will spend some time taking a look at that option also.

jewzaam wrote
We deploy in EAP, what about modules?  Have they been investigated for
loading these resources?  If so, what is wrong with them?

On Tue, Feb 23, 2016 at 4:22 PM dcrissman [via lightblue-dev] <
[hidden email]> wrote:

> In https://github.com/lightblue-platform/lightblue-rest/issues/162 a
> rough cut of plugin capability was added to lightblue. Essentially it is
> now possibly to add arbitrary jars to the lightblue runtime classpath, but
> the hooks, controllers, and datasources are still gated by the json
> configuration files (just as they have always been).
>
> Notable problems with this implementation:
> * No protection if duplicate plugins contain the same dependency jars
> * No protection if a plugin includes a duplicate dependency of lightblue
> itself.
> * No SecurityManager to protect system resources from plugins
> * Is implemented in lightblue-rest, but probably belongs in lightblue-core
> * Ran into a casting issue when attempting to run lightblue-mongo as a
> plugin: https://github.com/lightblue-platform/lightblue-mongo/issues/214
> ** Regardless of if mongo should be run as a plugin or not, I want to make
> sure other plugins don't run into similar issues
>
>
> I have been thinking about these issues for a little while now and the
> remainder of this post are my thoughts for solutions. I am looking for any
> feedback you might have.
>
>
> The research I have done points me to a separate "parent last" ClassLoader
> per plugin. Some testing will be needed, but I think that this will
> protected us from the first two bullet points. To achieve this we will need
> to know how a plugin is structured on the file system. So lets say we
> assume each plugin comes packaged in it's own directory, therefore all jar
> files in a given plugin directory belong in the same ClassLoader. For jars
> not in a directory, then we would need to assume each is a self-contained
> plugin and would need to be given its own ClassLoader.
>
> The problem this causes is that we then have many ClassLoaders, and no
> real indication which is the correct one to use for any given class. I
> think the best solution is ServiceLoader, we would just need to iterate
> over each ClassLoader and ask it to instantiate all implementations of some
> interface in that ClassLoader. I don't know the full extent of what this
> affects, but it might remove or alter the need for the json configuration
> files as no longer would there be a need for something to tell lb some
> class exists and needs to be instantiated.
>
> As for the SecurityManager if we use a lb specific sub-class of
> URLClassLoader, then in our SecurityManager implementation we check if the
> ClassLoader is an instance of our custom ClassLoader and then slap
> additional permissions on it if it is. Any thoughts as to what permissions
> we would want?
>
> Any thoughts in general would be appreciated.
>
> Thanks,
> Dennis
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://dev.forum.lightblue.io/Lightblue-Plugin-Design-tp425.html
> To start a new topic under lightblue-dev, email
> [hidden email]
> To unsubscribe from lightblue-dev, click here
> <http://dev.forum.lightblue.io/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=amV3emFhbUBnbWFpbC5jb218MXwtNDg4MDA0Mzg4>
> .
> NAML
> <http://dev.forum.lightblue.io/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
Reply | Threaded
Open this post in threaded view
|

Re: Lightblue Plugin Design

jewzaam
Administrator
+1 for not tied to EAP, but if we're doing a lot of work to get the same functionality a solution tailored to that app server would be nice.  And could go with tailoring solutions for other servers later as needed.

On Wed, Feb 24, 2016 at 11:02 AM dcrissman [via lightblue-dev] <[hidden email]> wrote:
You and I had talked about a desire to keep the plugin architecture from being tied to EAP. Because of that I have not spent time investigating EAP modules. Aside from our previous conversation though, I am not aware of anything that would be wrong with them. I will spend some time taking a look at that option also.

jewzaam wrote
We deploy in EAP, what about modules?  Have they been investigated for
loading these resources?  If so, what is wrong with them?

On Tue, Feb 23, 2016 at 4:22 PM dcrissman [via lightblue-dev] <
[hidden email]> wrote:

> In https://github.com/lightblue-platform/lightblue-rest/issues/162 a
> rough cut of plugin capability was added to lightblue. Essentially it is
> now possibly to add arbitrary jars to the lightblue runtime classpath, but
> the hooks, controllers, and datasources are still gated by the json
> configuration files (just as they have always been).
>
> Notable problems with this implementation:
> * No protection if duplicate plugins contain the same dependency jars
> * No protection if a plugin includes a duplicate dependency of lightblue
> itself.
> * No SecurityManager to protect system resources from plugins
> * Is implemented in lightblue-rest, but probably belongs in lightblue-core
> * Ran into a casting issue when attempting to run lightblue-mongo as a
> plugin: https://github.com/lightblue-platform/lightblue-mongo/issues/214
> ** Regardless of if mongo should be run as a plugin or not, I want to make
> sure other plugins don't run into similar issues
>
>
> I have been thinking about these issues for a little while now and the
> remainder of this post are my thoughts for solutions. I am looking for any
> feedback you might have.
>
>
> The research I have done points me to a separate "parent last" ClassLoader
> per plugin. Some testing will be needed, but I think that this will
> protected us from the first two bullet points. To achieve this we will need
> to know how a plugin is structured on the file system. So lets say we
> assume each plugin comes packaged in it's own directory, therefore all jar
> files in a given plugin directory belong in the same ClassLoader. For jars
> not in a directory, then we would need to assume each is a self-contained
> plugin and would need to be given its own ClassLoader.
>
> The problem this causes is that we then have many ClassLoaders, and no
> real indication which is the correct one to use for any given class. I
> think the best solution is ServiceLoader, we would just need to iterate
> over each ClassLoader and ask it to instantiate all implementations of some
> interface in that ClassLoader. I don't know the full extent of what this
> affects, but it might remove or alter the need for the json configuration
> files as no longer would there be a need for something to tell lb some
> class exists and needs to be instantiated.
>
> As for the SecurityManager if we use a lb specific sub-class of
> URLClassLoader, then in our SecurityManager implementation we check if the
> ClassLoader is an instance of our custom ClassLoader and then slap
> additional permissions on it if it is. Any thoughts as to what permissions
> we would want?
>
> Any thoughts in general would be appreciated.
>
> Thanks,
> Dennis
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://dev.forum.lightblue.io/Lightblue-Plugin-Design-tp425.html
> To start a new topic under lightblue-dev, email

> To unsubscribe from lightblue-dev, click here



If you reply to this email, your message will be added to the discussion below:
To start a new topic under lightblue-dev, email [hidden email]
To unsubscribe from lightblue-dev, click here.
NAML
Reply | Threaded
Open this post in threaded view
|

Re: Lightblue Plugin Design

bserdar
In reply to this post by dcrissman
Here's a somewhat more portable idea for plugins:
  - We write a utility class in core/util to load classes/extensions. Say, LBExtension.load(name)
  - This utility looks at the name (we can come up with a convention for extension names to make parsing easier), and
     - if the name is a class, loads the class, instantiates it and returns the instance
     - if the name is a JNDI name, looks it up in JNDI and returns a remote/local interface to it
  - We deploy plugins as EJBs in jboss
  - The custom classloader is used for non-jboss deployments
 
Reply | Threaded
Open this post in threaded view
|

Re: Lightblue Plugin Design

dcrissman
bserdar wrote
  - We write a utility class in core/util to load classes/extensions. Say, LBExtension.load(name)
+1 to moving the functionality into core. The reason it was not initially done that way was to prevent a chicken/egg problem with the datasources.json which is also loaded in lightblue-rest. Likely not an issue for hooks, but a problem for (as an example) the ldap controller.

Reply | Threaded
Open this post in threaded view
|

Re: Lightblue Plugin Design

bserdar
The JNDI lookup idea won't work. It'll still need the remote interface
classes. Need to find another way...

On Tue, May 3, 2016 at 12:43 PM, dcrissman [via lightblue-dev]
<[hidden email]> wrote:

> bserdar wrote
>   - We write a utility class in core/util to load classes/extensions. Say,
> LBExtension.load(name)
>
> +1 to moving the functionality into core. The reason it was not initially
> done that way was to prevent a chicken/egg problem with the datasources.json
> which is also loaded in lightblue-rest. Likely not an issue for hooks, but a
> problem for (as an example) the ldap controller.
>
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://dev.forum.lightblue.io/Lightblue-Plugin-Design-tp425p433.html
> To start a new topic under lightblue-dev, email
> [hidden email]
> To unsubscribe from lightblue-dev, click here.
> NAML
Reply | Threaded
Open this post in threaded view
|

Re: Lightblue Plugin Design

dcrissman
Reply | Threaded
Open this post in threaded view
|

Re: Lightblue Plugin Design

bserdar
Deploy plugins as jar files, and access them as modules?

That requires that the module using the plugin has to include it in
its deployment-structure.xml, afaik, so that's not a clean solution.
You can't add a new plugin without deploying lightblue itself.

Is there another way to add modules?

On Mon, Jan 9, 2017 at 8:36 PM, dcrissman [via lightblue-dev]
<[hidden email]> wrote:

> Was pointed to this documentation as a potential EAP way to address the
> issue
>
> https://access.redhat.com/documentation/en/red-hat-jboss-enterprise-application-platform/7.0/paged/configuration-guide/chapter-6-jboss-eap-class-loading
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://dev.forum.lightblue.io/Lightblue-Plugin-Design-tp425p435.html
> To start a new topic under lightblue-dev, email
> [hidden email]
> To unsubscribe from lightblue-dev, click here.
> NAML