How to configure Groovy/Grails Tool Suite (GGTS) to import Git projects

Ok, i admit, this post seems to be very stupid, but i often struggle with this IDE to import already existing git commited Grails project and configure the IDE to be attached to the git repository.

So this is my short how-to.


First, from the repository view select "clone a git repository". Add all the needed informations and clone it.

Then right-click the repository and select Import projects. Now select Import existing projects and proceed with the setup (which now will be easy).

Now, your project will be imported correctly and already connected to the git repository.

If you directly clone the project from command line and then import it via GGTS IDE, your project will not be connected to the git repository.

Grails external configurations without reloading the app

Any time you want to change the configurations in Config.groovy you have to restart the application (if it is deployed). And if you need to add new configurations you have to redeploy the app even if you don't need to add additional code to manage them because you have already built a dynamic mechanism.

This is quite annoying. Fortunately Grails expose different way to solve this problem.

First of all have a look at the documentation: http://grails.org/doc/latest/guide/conf.html#configExternalized

There is only something that doesn't work for me: they say it is possible to perform change without restarting the app but that's not my case, i had always to restart the app.

So, let's have a look at the important code section:

grails.config.locations = [
   "classpath:${appName}-config.properties",
   "classpath:${appName}-config.groovy",
   "file:${userHome}/.grails/${appName}-config.properties",
   "file:${userHome}/.grails/${appName}-config.groovy" ]

Here you can specify two different ways to get an external configuration file which is via classpath or via file.

If you want to put a file directly under the app root folder, simply use this kind of syntax:

file:configfile.groovy

This way you can read properties from an external file.

But what if you want to read the file using an environment variable which specify the location of the file, so that you can have different environment configuration files without the need to know where the file is, as often happens in real world systems where developers doesn't know what is the production config file which is usually managed by an application manager?

Here is the solution:

if (System.getenv("TEST_CONF_GROOVY")) {
   grails.config.locations << "file:" + System.getenv("TEST_CONF_GROOVY")
}

Here you are simply saying to Grails to use, if defined, an environment variable which is the path to your configuration file.

At this point you are able to specify external configuration file but what if you want to update configuration at runtime?

Here it is: you can specify a code section (i placed it inside a controller) which refresh the external configuration:

import grails.util.Environment

def grailsApplication;

def refreshConfig() {

def files = grailsApplication.config.grails.config.locations
   
   files.each { fileName ->
   
      File configFile = new File(fileName.replace("file:", "")).absoluteFile
   
      try {
   
         ConfigSlurper configSlurper = new ConfigSlurper(Environment.getCurrent().getName())
         ConfigObject updatedConfig
         updatedConfig = configSlurper.parse(configFile.text)
         grailsApplication.config.merge(updatedConfig)
      } catch (Throwable e) {
         log.error("Failed parsing and merging config file ${configFile} changes", e)
      }
   }
}

It is super simple and this code is specific to groovy configurations and not properties file. It simply parse again any external file and merge the new configurations with the current one.

So you don't need anymore to restart or reload the server!

Promocodes management

Coupon, discounts, promotional codes or...better known as promocodes! Have you ever wanted to integrate them in your website or mobile app? Yes? Ever tried? Well, it should be easy and quick...ok you have to create tables on your db, implement date checking, usage count and so on...ok, seems too much code for such a feature!
Well, now you have an opportunity to simplify anything!

Appromocodes.com

This is a self-promotional post, i have to admit! Appromocodes is a webapps i built to automatize and manage promocode management and you can find it here!

Why should i use Appromocodes?

Because promocodes help obtaining more users and convert them into paying customers

Promocodes are a well known tool used to incentivate this kind of actions. Quite every site has fields to accept promocodes and the reason is that it really helps. You can create promocodes and quickly distribute them through blogs, email or even publishing them if you want to promote your business via printed papers.

But managing promocodes is not a so easy and immediate task. It requires you to create db tables, perform validity or usage checks and so on.

Now you can save time and have a fine dashboard to manage promocodes and analyze promocodes usage simply using appromocodes.com.

What you have to do is simply perform an http call to Appromocodes api to check the promocode user passed to your form.

The tool is really easy to use and integrate in your webapplication or mobile app.

So go to appromocodes.com and start creating promocodes for your projects! 

Apache, Tomcat and Grails! Virtualhosts, hosts and contexts configurations!

I was in need to deploy a Grails application as a war on my Digitalocean instance.

The architecture i wanted to have was an Apache which redirects some calls to Tomcat (while manages others directly) and have my grails application deployed as a war on tomcat, responding without context path.

You can see the final result here at www.appromocodes.com (at the moment of this article there is only a landing page open to the public!)

This post is strictly related to DigitalOcean setup and Ubuntu 14.04.

So, to be clear i want my app (which comes from a war called appromocodes.war) to respond to www.appromocodes.com and i want that phpmyadmin responds at www.appromocodes.com/phpmyadmin and i want to have some other virtualhosts possibilities open.

So here are the steps:

Install servers

First of all, i installed a LAMP using DO images (but i thin it's a common ubuntu LAMP installation) then i installed tomcat and in the end i installed phpmyadmin. DO has a very comprehensive list of tutorials on how to do this, so check out for them. They will guid you through standard ubuntu installations

Enable modjk

This guide gives you hint on how to connect Apache 2 and Tomcat 7 using (and installing) mod_jk but there are a few points such as modifiyng 000-default.conf which are not best practice so i'll give you my way. Install mod jk via

sudo apt-get install libapache2-mod-jk

then configure tomcat to receive mod jk incoming traffic

sudo vim /etc/tomcat7/server.xml

uncommenting

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

Now let's create a worker file for Apache via

sudo vim /etc/apache2/workers.properties

And set this content:

# Define 1 real worker using ajp13
worker.list=worker1
# Set properties for worker (ajp13)
worker.worker1.type=ajp13
worker.worker1.host=localhost
worker.worker1.port=8009

Now to ask Apache to use this worker

sudo vim /etc/apache2/mods-available/jk.conf

and change the JkWorkersFile property to

/etc/apache2/workers.properties

Now the guide linked before told you to modify 000-default.conf but this is not the correct way. We will create a specific conf and add it to apache via a2ensite command (this guide is very clear).

Configure Apache

Let's create a file. In my example it is called appromocodes.com.conf under the sites-available Apache folder and put this content within:

<VirtualHost *:80>
     
ServerAdmin admin@appromocodes.com
ServerName appromocodes.com
ServerAlias www.appromocodes.com
DocumentRoot /var/lib/tomcat7/webapps/appromocodes/
     
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined

JkUnMount /phpmyadmin* worker1
JkMount /* worker1
     
</VirtualHost>

As you can see this virtualhost responds to my www.appromocodes.com domain and pass any call to Tomcat via ajp. Only /phpmyadmin path is still controlled by Apache thanks to the JkUnMount directive.

Now

sudo a2ensite appromocodes.com.conf

and your site configuration will be enabled (a2dissite command to disable it).
Ah, don't forget

sudo service apache2 restart

to see changes enabled.
At this moment, apache serves correctly /phpmyadmin calls and redirect all the traffic for www.appromocodes.com to tomcat. If you want to create another virtualhost for another domain you will see that everything works correctly.

Configure Tomcat

Now that apache sends the traffic to tomcat, we must instruct tomcat on what to do with this traffic since any call to www.appromocodes.com will be redirected to www.appromocodes.com:8080 default site (even if we will not see the port number), but our app is deployed via war (or folder) with a specific context. At this moment (suppose i've already deployed my app on tomcat), my webapp will respond to www.appromocodes.com/appromocodes. We must still do something to achieve our final result
Ok, so we could still use apache and define rewrite rules but tomcat offers "virtual hosts" too. It is not as comfortable as apache but it's a very good solution.
You can find tons of information on tomcat docs regarding Host tag used to define virtual hosts but one thing you must be aware is that you must pay attention because it's easy to get multiple deploys of the same webapplication so after struggling a lot i found a good way to proceed and achieve my targets.
Modify tomcat' server.xml in its Engine section in this way (this is my example obviously):

<Engine name="Catalina" defaultHost="localhost">
<Realm className="org.apache.catalina.realm.LockOutRealm">
<Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase"/>
</Realm>
<Host name="localhost" appBase="webapps" unpackWARs="false" autoDeploy="false">
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="localhost_access_log." suffix=".txt" pattern="%h %l %u %t &quot;%r&quot; %s %b" />
</Host>
<Host name="appromocodes.com" unpackWARs="false" autoDeploy="false">
<Alias>www.appromocodes.com</Alias>
</Host>
 
</Engine>

I have bolded the different elements respect the original ones. This section is telling Tomcat that we have two different hosts: localhost and appromocodes.com host to manage.
We have disabled autodeploys and war unpacking so tomcat will not do anything like this automatically (this will help avoid autoredeploy of a war in a first moment as a war and in a second as a directory after war extraction).
We are saying that both share the webapps folder (we could have specified another folder for appromocodes.com but it was useless now  and remeber to give that folder tomcat7:tomcat7 ownership and privileges).
Now, as specifically stated in the Tomcat docs, if you want to have an app that responds without context, you must declare it in a folder called ROOT or in a war called ROOT.war.
But i don't want to rename it any time so this is what we can do.
Execute

sudo service tomcat7 restart

and have a look at conf/Catalina. You will find a new folder called appromocodes.com. We ca now create a file called ROOT.xml in conf/Catalina/appromocodes.com with this content:

<Context
docBase="/var/lib/tomcat7/webapps2/appromocodes.war"
path=""
reloadable="true"
/>

This file will specify the ROOT application context for our appromocodes.com host and as you can see you can address any war file you want (or folder if you prefer).
If you now restart tomcat

sudo service tomcat7 restart

you will see that it will deploy a ROOT folder under webapps folder (if we had specified a different one in the server.xml we would have find it there so we could have specific ROOT folder for any host) and that's it!
Now our app responds correctly to www.appromocodes.com and we won't find any /appromocodes context in any call!

It is fundamental to define an external ROOT.xml file otherwise you will be able to specify only a different context instead of the natural one (war filename or folder name) but not a ROOT context and you will be subject to multiple redeploys.

Ok but...this is a general process not a Grails specific one!

That's right! This guide is ok for any general war deploy and apache 2 plus tomcat 7 configuration, but i have specified Grails too since it shows you how you can start from a Grails application, package it as a war via grails war (or grails prod war) and then deploying it as a J2EE webapplication.

Somebody might say it is better to run-app directly. In this case you can also specify different port such as 80 and have grails running directly your app. But in this way you loose the possibility to use Apache applications such as phpmyadmin or wordpress combined with your java app. And really important: grails consume much more memory while tomcat requires a lot less for the same app.

That's all folk, hope you save time!

How to use HttpBuilder in Grails apps

Do you need to perform some http calls from your Grails application? The best way is to use the powerful HttpBuilder groovy library.

You can find a lot of example about how to use it, and generally all these examples will rely on Grab, but the best way to use within your projects, is to add it to your BuildConfig file.

Here it is the snippet you should use. Add this to BuildConfig dependency section:

compile "org.codehaus.groovy.modules.http-builder:http-builder:0.7"

Super easy! You can find these information simply looking at the Download section of HttpBuilder documentation. Then in your controller or service you can use the example code simply importing all the needed classes.

import groovyx.net.http.HTTPBuilder
import groovyx.net.http.Method
import groovyx.net.http.ContentType
import groovyx.net.http.AsyncHTTPBuilder
import static groovyx.net.http.ContentType.JSON

...

def http = new HTTPBuilder('http://www.google.com')
    http.get( path : '/search',
        contentType : ContentType.TEXT,
        query : [q:'Groovy'] ) { resp, reader ->
            println "response status: ${resp.statusLine}"
            println 'Headers: -----------'
            resp.headers.each { h ->
                println " ${h.name} : ${h.value}"
            }
            println 'Response data: -----'
            System.out << reader
            println '\n--------------------'
}

Be careful to specify ContentType.TEXT which is slightly different from what stated in the documentation.

Running new versions of Grails

Sometimes you write an app in development on your local machine. A few days after a new Grails version is released. You still keep the older version on your development machine, but decide to install Grails newer version on your remote machine.

You check out the project, run grails update (since 2.4.2 you use the set-grails-version command), launch compilations commands and then...an error happened and it is not a bug of yours (you'll have time to discover those!).

Today my error is

war script error: java.lang.NoClassDefFoundError: Config$_run_closure1_closure5

I had openjdk1.6_27 (ok ok, it should be time to update it) on my remote machine and both grails 2.3.7 (working correctley) and grails 2.4.2 making me crazy.

I solved my issue simply by creating a brand new project on my remote machine (i did it to exclude it was a java version error) and packaging it as a war file. This was very useful to Grails: doing this way it was able to download all the new dependencies and plugin it needed.

Then back again to my checked out project and now grails war works correctly!

That'all.

Changing Grails server port

This is just a little hint very fast and useful: if you want to change the default port (8080) to another one, the fastest way is to add this property to the BuildConfig.groovy file (you can modfy Config.groovy but then you will have to set the Dserver property when launching grails):

grails.server.port.http = 8090

That's all folk!

Grails and external processes

Often we need to call external scripts from our web application. As an example, i'm developing a web app in Grails which needs to get some data from external sources. In my specific case, i decided to create some php scripts (which i can modify and edit quicker in response to external changes) to gather some informations. So my webapp will call these scripts and these scripts will pass the results back to Grails, as a json params for a Grails controller call.

Grails provide some incredibly easy ways to create and launch external processes. Simply define a string and call execute on it and you will have started an external process!

def command = "php " + pathToPhpFolder + "/my_script.php $myParam1 $myParam2";
command.execute();

This is the simpler approach, but it is not perfect because you may have problems with the params you pass, especially if they contains spaces. So a more safer (but still very easy) way is this:

def command = ["php", pathToPhpFolder + "/uploader.php", myParam1, myParam2];
command.execute();

These way all the parameters will be passed exactly as parameters, even if they contains spaces.

Process control

You may want to manage the external process and, for example, kill it if it goes in timeout. Here is the easy way:

def process = command.execute();
process.waitForOrKill(MY_TIMEOUT);

These instructions will launch the process and wait for a MY_TIMEOUT time before killing the process. The only things not exactly nice is that if the process timeouts, you will not have any exception so you may need to implement a way to determine if the process completed correctly or if it was killed (i use a status variable).

That's all!

How to configure Grails and Geb as webscraping tool versus php simple html dom library

I went across Geb because i was looking for a smart way to webscrape pages using Grails. Geb is a browser automation tool.

This definition is really important: it is not a library to parse html, it is a browser automation tool. What does this mean? It means that it will run an external browser and it will execute all the operations you coded on the browser pages as a human would do.

So it will load pages, populate form fields with text, will simulate clicks and will do all the things you would do as a human. You will also be able to look at the browser while it automatically does all the operations because Geb will run an external browser.

It is a very sophisticated framework but i think it should be used primarily as a test automator. You can run tests with different browser such as explorer, chrome or safari and query for "visual stuffs" like div's height or width. Obviously you can query the page for specific elements (and this is the part where you may want to use it as a scraping tool).

This is the premise i wanted to share. I think that Geb is a powerful tool, but i think it can be really useful for automated tests and not for scraping. And this because it seems to me to be too much complex for common and batch operations like scraping, if you compare it to a html parser library easy to use such as simple html dom.

But if you want to use Geb for scraping, here are my two cents on how to configure it with Grails version > 2.3.5 (and this is good i think because a lot of good material out there is outdated).

Pay attention: i'm sure this is not the best way to configure it, but it took me a little to make it run because docs are not clear about how to configure it with Grails. This is just a basci configuration, to allow you run and experiment with it.
You can find a Geb plugin for Grails but it seems to be useless.

So, here is my recipe.

1) Download Chrome driver from here and save it locally.

2) Create a GebConfig file under conf folder. This should be the code:

import org.openqa.selenium.chrome.ChromeDriver
driver = {
 System.setProperty('webdriver.chrome.driver', 'path/to/your/downloaded/chromedriver')
 new ChromeDriver() 
}

3) In BuildConfig add this repository to repositories section

mavenRepo "https://oss.sonatype.org/content/repositories/releases/"

4) Add under dependencies:

compile "org.gebish:geb-core:0.9.2"
compile "org.seleniumhq.selenium:selenium-support:2.26.0"
compile "org.seleniumhq.selenium:selenium-chrome-driver:2.31.0"

5) Now you can create a controller to test Geb. For example:

package gt

import geb.Browser;

class GebController {

    def index() {
  
  Browser.drive {
   go "http://google.com/ncr"
   
   // make sure we actually got to the page
   assert title == "Google"
   
   // enter wikipedia into the search field
   $("input", name: "q").value("wikipedia")
   
   // wait for the change to results page to happen
   // (google updates the page dynamically without a new request)
   waitFor { title.endsWith("Google Search") }
   
   // is the first link to wikipedia?
   def firstLink = $("li.g", 0).find("a")
   assert firstLink.text().contains("Wikipedia")
   
   // click the link
   firstLink.click()
   
   // wait for Google's javascript to redirect to Wikipedia
   waitFor { 
    title.startsWith("Wikipedia")
    render "OK"
   }
   
   
  }
  
 }
}

6) Execute the controller: you will see a new Chrome browser running all the code.

PHP and simple html dom parsing library

I will not write a lot on how to use the php library, and this because it is super-simple to understand. But how to integrate php with Grails?

Well, starting an external process with groovy is really simple:

def process = command.execute();
process.waitForOrKill(MY_TIMEOUT);

Where command is a string. This will start the process with a fixed timeout.
It will be really easy to run an external php process which executes different scraping operations using the simple html dom library.

Then it will be easy to get the output of the php process and parse it.

As an example, in a project of mine, i run external php processes (scheduled by a grails application) which perform scraping operations and at the end, they do call a grails controller passing a json result. The Grails controller persists the object and executes different operations.

I think this is a faster and cleaner approach to webscraping instead of using Geb.
I think Geb may be useful if you have to compile a lot of forms and visit a lot of pages to perform the scraping. But if all the things you have to do are only scraping lists (for example "products lists") and navigate through pagination, then i think php parsing it's quicker and more productive.

Grails Spring Security Plugin manual login - registration - remember me

Spring security plugin is a powerful framework to manage authentication. It can be used out of the box following its super-easy documentation, but what if you want to implement manual login/registration/rememberMe functionalities?
I found that login and registration are quite easy to understand from the plugin guide, while rememberMe was just a little cryptic.

So this is the code i want to share: it's the server-side code and it is examples (so values are hard-coded but this is just an example). This code is for spring-security-core:2.0-RC2 plugin.

Please refer to plugin guide for install and setup configurations.

User registration

This code will create a User instance and save it to db with password encrypted (for Role creation and assignment please refer to step 7 in tutorial)

def user = new User();
user.username = "john";
user.password = "secretpassword";
user.save()

User login

In your controller define reference to springSecurityService and passwordEncoder beans via

def springSecurityService
def passwordEncoder

then in your code

def user = User.findByUsername("john");
def isLoggable = passwordEncoder.isPasswordValid(user.password, thepasswordtocheck, null)

//add this if you want to log the user then
springSecurityService.reauthenticate(user.username);

Is user logged?

def isLoggedIn = springSecurityService.isLoggedIn();

RememberMe functionality

NOTE: with spring-security-core:2.0-RC2 version, property namespaces has changed from grails.plugins.springsecurity to grails.plugin.springsecurity (look the 's' in the word plugin), a lot of examples out there refers to old plugin version.

Set in your Config.groovy:

grails.plugin.springsecurity.rememberMe.alwaysRemember = true

I would add also set

grails.plugin.springsecurity.rememberMe.cookieName = 'grails_remember_me'
grails.plugin.springsecurity.rememberMe.key = 'anewrandomkey'

to better secure the token, but that is optional.

Then in your code you have to define reference to rememberMeServices bean via

def rememberMeServices

Then you should call rememberMeServices.loginSuccess...well, that is the interface as specified in the code but that is not working!!! you have to call rememberMeServices.onLoginSuccess.
This was a critical step i had to dig a lot to find it!

So code is:

def user = User.findByUsername("john");
springSecurityService.reauthenticate(user.username);
rememberMeServices.onLoginSuccess(request, response, springSecurityService.getAuthentication());

That's all, hope it can helps someone!

Grails on Openshift

Openshift platform allows you to create an app very easily but i found quite difficult to achieve an obvious task: deploying a Grails application without having to redeploy every time the full war file.

I found a simple way to avoid committing the war file, any time.

A Grails application can be run on Openshift just installing a Tomcat7 cartridge and then deploying the war file on it.

You can put the war file on the openshift instance and commit the war file as if it was the only file in your git repository.

This works correctly, but it means that any time you want to deploy an app, you must upload an entire war file. This is unacceptable, so i tried to solve the problem.

I tried two different approaches:

1. installing Grails on openshift: you can ssh to your application via rhc ssh command and then you can download Grails in your $OPENSHIFT_DATA_DIR and after setting some environment variables you can build your app
2. configure Jenkins for CI to build automatically on any git push, by setting some "special files" to instruct Jenkins to download Grails and build your project with it

These two ways work, but with an unacceptable complexity level! But trying to use these approaches i learned a lot of things.

Here are two links to very useful approaches: run Grails on Openshift and deploy with Jenkins.

These two approaches let me learn a lot about Openshift but didn't work for me. And even if those worked, it seemed to me that they were too complex and difficult to mantain.

The good news is that a solution exists and it is super-easy and clean and do perfectly work!

How to deploy a Grails application on Openshift without redeploying a war file any time

Go create you application on Openshift: the best way to do it is to use rhc (for more informations have a look here on Openshift website).

Step 1 create app on Openshift

I will create an app called "grails" in a local folder called openshift so go to openshift folder and 

rhc create-app grails jbossews-2.0

Step 2 create a Grails project

here you can use your favourite tool to create a local Grails project (or clone it from somewhere) in a different folder

Step 3 remove defaults from openshift folder 

Now we remove some files from the grails folder automatically created by rhc create which is connected to git)

rm -fr pom.xml src

Step 4 perform some git operations:

git add -A

git commit -m "get rid of useless application"

Step 5 add exploded war configurations to your grails project's BuildConfig.groovy

grails.project.war.exploded.dir = "/yourlocalpathto/openshift/grails/webapps/ROOT"

grails.war.exploded=true

A note about this: this is the key of the process. We are saying Grails to create an exploded-war (which is a full war file that instead of being compressed as a single binary file, is exploded in a directory) so that the first time we will have to upload via git everything (even all the jars) but next git push will send only the differences, which is what we wanted to achieve

Step 6 create a dodeploy file

This is a file which instructs jboss that after the push it must take the ROOT folder and deploy it on Tomcat (which is emulated by JBoss) as if it was a war file

in /yourlocalpathto/openshift/grails/webapps execute

touch ROOT.war.dodeploy

Step 7 perform grails war

perform grails war and it will create the ROOT folder

Step 8 push the files to OpenShift

go to /yourlocalpathto/openshift/grails/ and execute:

git add -A

git commit -m "deploy application"

git push

That's all folks!

Now all your files will be deployed to Openshift. You can check operation log opening a new terminal and giving rhc tail.

Any time you want to deploy the app, execute step 7 and 8 and wait only for Tomcat redeploy time which is slooooow!

You will have to upload only the changed files. This approach have a secondary advantage: you can have your grails app commited on a different git repository. You can push files to the git without needing to deploy the app. Any time you want to deploy, you build a war file and manually push it from /yourlocalpathto/openshift/grails folder.