Publishing sonar metrics

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Publishing sonar metrics

jewzaam
Administrator
I created a script in the lightblue repo that will publish fresh sonar metrics to the sonar-lightblue.rhcloud.com instance.  It assumes:
* you have rhc setup
* your current server is openshift online
* you have access to the 'lightblue' namespace

The script will handle submodule initialization and updates, port forwarding, and building things.  All you have to do is check out the code and run the script.

git clone git@github.com:lightblue-platform/lightblue.git
cd lightblue
./sonar.sh
Reply | Threaded
Open this post in threaded view
|

Re: Publishing sonar metrics

jewzaam
Administrator
Ideally, btw, I'd just have travis do the sonar stuff.  But, I haven't figured out how to make that happen.  The key problem is access.  The mysql database for sonar in the diy app on openshift isn't publicly exposed.  To get access I've considered ssh tunnel but that requires an ssh key.  I haven't convinced myself it's ok to put a private ssh key to that gear somewhere, even encrypted, that travis can get access to.  It would give access to the whole domain (if not all domains for my account), not just mysql.

This problem goes away if mysql is available publically.  Databases on openshift are available publicly when the containing app is scalable.  The diy cart used for sonar is not scalable.

Some options:
* Create a downloadable cart that is scalable
* Create a gear with something that is scalable and include the msyql gear.  Use this for sonar's database.
* Wait for inclusion on nemo.sonarqube.org [1]

It's manual for now but at least it's just one script.  I'll go for the "wait" option and hope we get included.  If so, then almost every job that was on our internal jenkins host is going to be gone.

[1] http://sonarqube.15.x6.nabble.com/sonar-dev-Could-you-please-add-lightblue-project-to-nemo-sonarqube-org-td5026980.html#a5027056
Reply | Threaded
Open this post in threaded view
|

Re: Publishing sonar metrics

lcestari
And what if we create a custom cart that expose MySQL ports to DMZ and setup sonar to use that instance?
Reply | Threaded
Open this post in threaded view
|

Re: Publishing sonar metrics

jewzaam
Administrator
We could, I've got enough gears available.  Think it's worth spending the time?


On Wed, Aug 20, 2014 at 10:34 AM, lcestari [via lightblue-dev] <[hidden email]> wrote:
And what if we create a custom cart that expose MySQL ports to DMZ and setup sonar to use that instance?


If you reply to this email, your message will be added to the discussion below:
http://lightblue-dev.1011138.n3.nabble.com/Publishing-sonar-metrics-tp71p77.html
To start a new topic under lightblue-dev, email [hidden email]
To unsubscribe from lightblue-dev, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: Publishing sonar metrics

jewzaam
Administrator
I spent a little time on this.  Created a scalable app and added mysql.  Nothing I've done has given access to mysql without port forwarding.  It's possible to access it from other openshift nodes, but not from my machine directly.  There is probably some quirk with how the connection is created.  I hit a similar problem trying the same thing with a mongo cart on a scalable application.  The solution ended up being the connection string.

Doesn't work:
mongo --host <host dns> --port <port> -u admin -p

Does work:
mongo <host dns>:<port>/<database> -u admin -p



And turned out wasn't supplying the database name I think.

For this case I am supplying it, but it's mysql so something else may be broken.  And yes I have setup permission to the database to any host.

mysql -h <host dns> -P <port> -u sonar -p<password> sonar

I have also tried the admin user.  Maybe I'll open a ticket.
Reply | Threaded
Open this post in threaded view
|

Re: Publishing sonar metrics

jewzaam
Administrator
Ok, OpenShift Online prevents it.  It makes sense, but it's disappointing.

https://www.openshift.com/forums/express/external-access-to-mysql

Next level of workaround would be to run the sonar job on openshift, custom jenkins job.  The layers are getting a bit big so I'm less and less interested in this solution.  We would have:

1. sonar on a diy gear
2. mysql on a scaled gear
3. sonar kicked off from jenkins
4. build, test, deploy on travis-ci (intergrated with coveralls)

Other option may be another place to host the database, but it doesn't really simplify things.
Reply | Threaded
Open this post in threaded view
|

Re: Publishing sonar metrics

lcestari
About that MySQL discussion on openshift forum, I think it is about the MySQL cartridge which it made to be a simple database (it is a category which doesnt allow external network connection). I thought something different, I thought maybe to customize that cartridge or even build one from scratch that would have a public port open to MySQL (which is not the default case, probably we have to change some configuration ( specially for the category and other things  https://openshift.github.io/documentation/oo_cartridge_developers_guide.html )
Reply | Threaded
Open this post in threaded view
|

Re: Publishing sonar metrics

jewzaam
Administrator

There are some free dB options in the marketplace that could be explored before that. Limited disk and i/o though.

On Aug 20, 2014 4:54 PM, "lcestari [via lightblue-dev]" <[hidden email]> wrote:
About that MySQL discussion on openshift forum, I think it is about the MySQL cartridge which it made to be a simple database (it is a category which doesnt allow external network connection). I thought something different, I thought maybe to customize that cartridge or even build one from scratch that would have a public port open to MySQL (which is not the default case, probably we have to change some configuration ( specially for the category and other things  https://openshift.github.io/documentation/oo_cartridge_developers_guide.html )


If you reply to this email, your message will be added to the discussion below:
http://lightblue-dev.1011138.n3.nabble.com/Publishing-sonar-metrics-tp71p89.html
To start a new topic under lightblue-dev, email [hidden email]
To unsubscribe from lightblue-dev, click here.
NAML
Reply | Threaded
Open this post in threaded view
|

Re: Publishing sonar metrics

lcestari
Yeah, that can solve the problem. Also, any gear on openshift online will have a limit of space (not sure how much it is , I think it is sth like 10GB), so we would have to setup a policy to remove old information from our database (I think sonar have this option off the shelf
Reply | Threaded
Open this post in threaded view
|

Re: Publishing sonar metrics

jewzaam
Administrator
Disk is limited to 1GB I think.  More than enough for sonar, especially since we don't need all historical data.  Trends are nice, but not required for this.  If we're included in nemo then we can get that long term historical trending.


On Fri, Aug 22, 2014 at 11:06 AM, lcestari [via lightblue-dev] <[hidden email]> wrote:
Yeah, that can solve the problem. Also, any gear on openshift online will have a limit of space (not sure how much it is , I think it is sth like 10GB), so we would have to setup a policy to remove old information from our database (I think sonar have this option off the shelf


If you reply to this email, your message will be added to the discussion below:
http://lightblue-dev.1011138.n3.nabble.com/Publishing-sonar-metrics-tp71p100.html
To start a new topic under lightblue-dev, email [hidden email]
To unsubscribe from lightblue-dev, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: Publishing sonar metrics

lcestari
The lightblue projects were accepted in Nemo!

You can find it in the list of this link http://nemo.sonarqube.org/measures/filter/68?asc=true&cols%5B%5D=metric%3Aalert_status&cols%5B%5D=name&cols%5B%5D=date&cols%5B%5D=metric%3Ancloc&cols%5B%5D=metric%3Aviolations&cols%5B%5D=links&display=list&page=2&pageSize=100&qualifiers=TRK&sort=name

For example, lightblue-core: http://nemo.sonarqube.org/dashboard/index/717402 

I think we could now open some issues related to issues that sonar mention. What do you think guys?
Reply | Threaded
Open this post in threaded view
|

Re: Publishing sonar metrics

jewzaam
Administrator
Very nice.  Can we change the profile to match what we do for our sonar instances?


On Wed, Oct 1, 2014 at 12:20 PM, lcestari [via lightblue-dev] <[hidden email]> wrote:
The lightblue projects were accepted in Nemo!

You can find it in the list of this link http://nemo.sonarqube.org/measures/filter/68?asc=true&cols%5B%5D=metric%3Aalert_status&cols%5B%5D=name&cols%5B%5D=date&cols%5B%5D=metric%3Ancloc&cols%5B%5D=metric%3Aviolations&cols%5B%5D=links&display=list&page=2&pageSize=100&qualifiers=TRK&sort=name

For example, lightblue-core: http://nemo.sonarqube.org/dashboard/index/717402 

I think we could now open some issues related to issues that sonar mention. What do you think guys?


If you reply to this email, your message will be added to the discussion below:
http://dev.forum.lightblue.io/Publishing-sonar-metrics-tp71p165.html
To start a new topic under lightblue-dev, email [hidden email]
To unsubscribe from lightblue-dev, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: Publishing sonar metrics

lcestari
I can ask them about it, but I saw this http://nemo.sonarqube.org/profiles and I presume that every project hosted so far are using the default (even the java group have more than one profile, it is always using the default), so I don't expect that they will change for us. Do you think that there are useless rules there? I even thought that might be good if we could add the findbugs rules to make more asserts on the codes.
Reply | Threaded
Open this post in threaded view
|

Re: Publishing sonar metrics

jewzaam
Administrator
I know that we've moved some specific rules down in priority.  I wouldn't call them useless but some things are just not a concern for us.  It's something we'd be very flexible on.  It's great to see lightblue out there, that's for sure!  I'll continue to publish to sonar.lightblue.io as well for now until we decide to just run with the nemo stuff.  May be good to have direct links to each repo in various repo's readme's.

On Fri, Oct 3, 2014 at 9:45 AM, lcestari [via lightblue-dev] <[hidden email]> wrote:
I can ask them about it, but I saw this http://nemo.sonarqube.org/profiles and I presume that every project hosted so far are using the default (even the java group have more than one profile, it is always using the default), so I don't expect that they will change for us. Do you think that there are useless rules there? I even thought that might be good if we could add the findbugs rules to make more asserts on the codes.


If you reply to this email, your message will be added to the discussion below:
http://dev.forum.lightblue.io/Publishing-sonar-metrics-tp71p170.html
To start a new topic under lightblue-dev, email [hidden email]
To unsubscribe from lightblue-dev, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: Publishing sonar metrics

lcestari
Do you mean to write in each repo's readme file the link to the sonar page (the one on openshift and the nemo)?
Reply | Threaded
Open this post in threaded view
|

Re: Publishing sonar metrics

jewzaam
Administrator
That and in the 'lightblue' repo maybe add as column to table of repos.

On Tue, Oct 7, 2014 at 2:57 PM, lcestari [via lightblue-dev] <[hidden email]> wrote:
Do you mean to write in each repo's readme file the link to the sonar page (the one on openshift and the nemo)?


If you reply to this email, your message will be added to the discussion below:
http://dev.forum.lightblue.io/Publishing-sonar-metrics-tp71p182.html
To start a new topic under lightblue-dev, email [hidden email]
To unsubscribe from lightblue-dev, click here.
NAML