Thursday, May 17, 2012

Play2.0 and cloudfoundry database connectivity

Right at the outset, an honest confession - I am no good at shell script. When I tried to deploy my play2.0 - scala application in cloudfoundry - and connect to the provisioned database service - I faced the difficult task of parsing a JSON string(cloudfoundry provides details of the services in JSON format) - extract the database credentials - and set them as environment variables so that when I start my play2.0 netty server - the server can read those environment variables and connect to the database. Following snippet shows a sample database related info that cloudfoundry provides as environment variable:

{"postgresql-9.0":[{"name":"database","label":"postgresql-9.0","plan":"free","tags":["postgresql","postgresql-9.0","relational"],"credentials":{"name":"d59efd73d7a0d458da6b156ea0ae67b6b","host":"172.30.48.124","hostname":"172.30.48.124","port":5432,"user":"u6dacee22b17d4c9d8e4aa832b7cc2947","username":"u6dacee22b17d4c9d8e4aa832b7cc2947","password":"p9cfd7252af8d4af4beb3815b38cf184e"}}]}
 


So, I had basically two options before me:

a) I write a shell script to parse the JSON script and extract  the relevant data
b) Or I change the netty server start-up code itself - so that when launched - it reads the VCAP_SERVICES JSON string - takes out the relevant database credentials and connects to the provisioned database instance.

So, I started on my venture to find a nice little shell script parses a JSON string and dishes out the database settings when asked for. Googling gave me some scripts - but they were not very helpful - either they were too lousy or too specific to some other tasks - and I dared not to modify them for the obvious reason as I have mentioned at the outset - I am not good at shell script.


So, I decided to follow my second option. I got the play2.0 source code from github repository - modified the netty server scala code - so that it calls some other piece of scala code(which does the JSON parsing - and sets the database specific details nice and cool) - and does what it normally does. Following snippet shows the piece of code that does JSON parsing and environment variable setting.

package play.core.server

import play.api.libs.json._
import play.api.libs.json.Json._
import play.api.libs.json.Json
import java.util.{Map=>JMap}
import java.util._
import java.lang.reflect.Field

object VCAPJsonParser {
  def init_env(): Unit  = {
    val vcap_services = System.getenv("VCAP_SERVICES")
    if(vcap_services == null || "" == vcap_services){
      println("vcap_services env value is not set!")
      ()
    }else {
    println("The VCAP_SERVICES json  : "+ vcap_services)
    val services: JsValue = Json.parse(vcap_services)
    val postgresql = services \ "postgresql-9.0"
    val credentials = postgresql(0) \ "credentials"
    println("The credentials json : "+ credentials)
    val dbname = (credentials \ "name").as[String]
    val hostname = (credentials \ "hostname").as[String]
    val user = (credentials \ "user").as[String]
    val password = (credentials \ "password").as[String]
    val port = (credentials \ "port").as[Int]
    val database = "jdbc:postgresql://" +hostname+":"+port+"/" +dbname
    println("database url : "+ database + ", user : "+ user+ ", password : "+ password)
    val envObj = System.getenv()
    val env = envObj.asInstanceOf[JMap[String,String]]
    var newEnv: JMap[String,String] = new java.util.HashMap()
    newEnv.putAll(env)
    newEnv.put("postgres_database",database)
    newEnv.put("postgres_dbuser",user)
    newEnv.put("postgres_password",password)
    setEnv(newEnv)
    println("Env set: "+ System.getenv())
    println("******************")
    println("Database url : "+ System.getenv("postgres_database")+", user : "+ System.getenv("postgres_dbuser")+", password : "+ System.getenv("postgres_password"))
    ()
   }
  }
 
  def main(args: Array[String]): Unit = {
    init_env
  }


  def setEnv(newenv: JMap[String,String]): Unit = {
    try {
    val processEnvironmentClass : Class[_] = Class.forName("java.lang.ProcessEnvironment")
        val theEnvironmentField : Field  = processEnvironmentClass.getDeclaredField("theEnvironment")
        theEnvironmentField.setAccessible(true)   
    val env: JMap[String,String] = (theEnvironmentField.get(null)).asInstanceOf[JMap[String,String]]
        env.putAll(newenv)
        val theCaseInsensitiveEnvironmentField: Field = processEnvironmentClass.getDeclaredField("theCaseInsensitiveEnvironment")
        theCaseInsensitiveEnvironmentField.setAccessible(true)
        val cienv: JMap[String,String] =(theCaseInsensitiveEnvironmentField.get(null)).asInstanceOf[JMap[String,String]]
    cienv.putAll(newenv)
    }catch {
     case e1: NoSuchFieldException => try {
   
       val classes: Array[Class[_]] = (classOf[Collections]).getDeclaredClasses()   
       val env: JMap[String,String] = System.getenv()
       for(cl <- classes){
     if("java.util.Collections$UnmodifiableMap".equals(cl.getName())) {
       val field: Field = cl.getDeclaredField("m")
       field.setAccessible(true)
           val obj = field.get(env)
           val map = obj.asInstanceOf[JMap[String,String]]
       map.clear()
           map.putAll(newenv)
     }
       }
     }catch {
       case e11: Exception => e11.printStackTrace(System.err)
     }
     case e2: Exception => e2.printStackTrace(System.err)
    }

  }
 
}

You can put this piece of code in the proper directory - call it from the main method of the "NettyServer.scala" and build the play framework from the source - and then push your application to cloudfoundry - and it will faithfully connect to your database instance.

I have done this - this is how my application(ooki.cloudfoundry.com) was running till I wrote this post.

I was not very happy to modify the netty server code just to read database specific details. Hence I rolled up my sleeves - And decided to give the shell script
approach another go. Following is shell script - that I ended up writing. Needless to say - there must be more efficient way of writing it - but it does the job currently. You are welcome to modify and make it more generic.


#!/usr/bin/env bash

function processVCAP_SERVICE_JSON_and_setDBenv(){
  echo "*******************"
  echo "$VCAP_SERVICES"
  echo "*******************"
  db_settings=()
  counter=0
  found="n"
  IFS=',' read -ra SERVICE_SETTINGS <<< "$VCAP_SERVICES"
  length="${#SERVICE_SETTINGS[@]}"
  for ((i=0; i<${length}; i++ ));
   do
    curr_item=${SERVICE_SETTINGS[$i]}
    curr_item="$(echo ${curr_item} | tr -d '\"')"
    echo "position : "+ "$i" + "and item is : "+ "$curr_item"
    #Take care of password:pbd0e3f367a6c4a36bcc48a3c763a929e}}]} <--
    if [[ "$curr_item" == password* ]]; then
       pwd_len="${#curr_item}"
       pwd_len=$pwd_len-4
       curr_item="${curr_item:0:$pwd_len}"
    fi

    if [[ "$found" = "y" ]] || [[ "$curr_item" == credentials* ]]; then
     if [ "$found" = "n" ]; then
       #get rid of 'credentials:{' part
       len="${#curr_item}"
       len=$len-1
       db_settings[counter]="${curr_item:13:$len}"
     else
       db_settings[counter]="${curr_item}"
     fi
     counter=$counter+1
     found="y"
     echo counter is : "{$counter}"
    fi
   done
  
   echo "ratul your db settings are : "  
   echo "${#db_settings}"
   echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
  
   #prepare the database_url,database_user & database_password and set them in env
   database_url=jdbc:postgresql://
   #change above in case of MySQL
   db_name=
   host_name=
   port_num=
   database_user=
   database_password=
   echo "============================================="
   for setting in "${db_settings[@]}"
    do
      echo ##############"${#setting}"   
      set_len="${#setting}"
      set_len=$set_len-1;
      echo "$setting and setting length=$set_len"

      case "$setting" in
           name:*) db_name="${setting:5:$set_len}";
                  echo "DB name : $db_name" ;;
       hostname:*) host_name="${setting:9:$set_len}";
          echo "DB host : $host_name" ;;

       port:*) port_num="${setting:5:$set_len}";
          echo "DB port : $port_num" ;;
       username:*) database_user="${setting:9:$set_len}";
          echo "DB user : $database_user" ;;
       password:*) database_password="${setting:9:$set_len}";
          echo "DB password : $database_password" ;;
      esac
    done

    database_url=$database_url$host_name:$port_num/$db_name
    export database_url=$database_url
    export database_password=$database_password
    export database_user=$database_user
}
processVCAP_SERVICE_JSON_and_setDBenv

echo "database url : $database_url"

echo "password : $database_password"

echo "user : $database_user"

exec java $JAVA_OPTS -Dhttp.port=$VCAP_APP_PORT -DapplyEvolutions.default=true -cp "`dirname $0`/lib/*" play.core.server.NettyServer `dirname $0`



Be sure to set environment variables accordingly in application.conf like so:

db.default.driver=org.postgresql.Driver
db.default.url=${database_url}
db.default.user=${database_user}
db.default.password=${database_password}






Saturday, May 12, 2012

Deploy play 2.0 app on cloudfoundry

For last few days I was trying to find a to deploy my play 2.0 application. I have already deployed my app at heroku(http://ooki.herokuapp.com/). But heroku offers very limited resources for free account (max slug size can not exceed 100MB and only 5MB of shared database space. So you have to work within these constraints and it becomes difficult to play around. In fact, to reduce my slug size I had to use a custom build pack from github. Also, if your application is not accessed for a long - heroku idles out your app - next time you access your app - it takes a long time for the response to come back. So you need to setup some kind ping program to keep hitting your app at regular interval say, every 10 minutes.

With cloundfoundry these problems are not there - cloudfoundry free resources are quite good enough(2GB RAM, 2GB disk space) so that you can concentrate on what you are doing instead of thinking about resource constraint.

Prerequisite:

Before you can deploy your app in cloudfoundry.com - you need to open an account with cloudfoundry.com. After registering it takes a day or two to get your account activated. You also have to have either the vmc command line tool or sts(eclipse plugin) installed on your system - visit the http://docs.cloudfoundry.com/getting-started.html link and follow the instructions there.
r new application is ready.App
Once you have setup the pre-requisites and got your user name and password from cloudfoundry.com - you are all set to deploy your app on cloudfoundry.com Paas.

I am going to use the vmc command line tool to deploy a newly created play 2.0 application on cloudfoundry.

Follow the steps below to deploy your app.

1. Launch command prompt.

2. Type in -> play new myPlayApp

ratul@ubuntu:~$ play new myPlayApp
       _            _
 _ __ | | __ _ _  _| |
| '_ \| |/ _' | || |_|
|  __/|_|\____|\__ (_)
|_|            |__/
            
play! 2.0.1, http://www.playframework.org

The new application will be created in /home/ratul/myPlayApp

What is the application name?
> myPlayApp

Which template do you want to use for this new application?

  1 - Create a simple Scala application
  2 - Create a simple Java application
  3 - Create an empty project

> 1

OK, application myPlayApp is created.

Have fun!

3. Opne the newly created myPlayApp/app/controllers/Application.scala file in a text editor.

Change defintion of the index method like so:

 def index = Action {
    val port = System.getenv("VCAP_APP_PORT")
    val host = System.getenv("VCAP_APP_HOST")
    println("App port : "+ port+" and host: "+ host)
    Ok(views.html.index("Your new application is ready."+ "App port : "+ port+" and host: "+ host))
  }

  Note: VCAP_APP_PORT & VCAP_APP_HOST are port name host names cloudfoundry assigns when it runs your app.

4. Save the Application.scala.

5. Open the myPlayApp/app/views/index.scala.html in a text editor and change the line that reads @play20.welcome(message) to @message and save it.

6. On the command, cd myPlayApp -> ratul@ubuntu:~$ cd myPlayApp/

7. launch the play promt by tying 'play'.

ratul@ubuntu:~/myPlayApp$ play
[info] Loading project definition from /home/ratul/myPlayApp/project
[info] Set current project to myPlayApp (in build file:/home/ratul/myPlayApp/)
       _            _
 _ __ | | __ _ _  _| |
| '_ \| |/ _' | || |_|
|  __/|_|\____|\__ (_)
|_|            |__/
            
play! 2.0.1, http://www.playframework.org

> Type "help play" or "license" for more information.
> Type "exit" or use Ctrl+D to leave this console.

8. Next we will bundle the application. So type 'dist' on the play prompt.

[myPlayApp] $ dist
[info] Updating {file:/home/ratul/myPlayApp/}myPlayApp...
[info] Done updating.                                                                 
[info] Compiling 5 Scala sources and 1 Java source to /home/ratul/myPlayApp/target/scala-2.9.1/classes...
[info] Packaging /home/ratul/myPlayApp/target/scala-2.9.1/myPlayApp_2.9.1-1.0-SNAPSHOT.jar ...
[info] Done packaging.

Your application is ready in /home/ratul/myPlayApp/dist/myPlayApp-1.0-SNAPSHOT.zip

[success] Total time: 12 s, completed 12 May, 2012 2:56:34 PM

9. We want to push the application contents as folder structure and not as zip so that we want to be able to push deltas later on. Hence open the folder myPlayApp/dist in your explorer and extract the contents inside 'myPlayApp-1.0-SNAPSHOT.zip' in the same folder.

10. Delete the myPlayApp-1.0-SNAPSHOT.zip. Go inside the myPlayApp-1.0-SNAPSHOT folder and delete the README and start files as well - we do not need them.

11. Now we are ready to push our content to cloudfoundry. Let's logr new application is ready.Appin first.

12. On the command prompt come out from the play promt by typing 'exit'.

13. cd dist/myPlayApp-1.0-SNAPSHOT/

14. Type 'vmc login' on the command prompt.

ratul@ubuntu:~/myPlayApp/dist/myPlayApp-1.0-SNAPSHOT$ vmc login
Attempting login to [http://api.cloudfoundry.com]

Email: ratul75@hotmail.com
Password: ******************
Successfully logged into [http://api.cloudfoundry.com]

15. On the prompt type : vmc target api.cloudfoundry.com

ratul@ubuntu:~/myPlayApp/dist/myPlayApp-1.0-SNAPSHOT$ vmc target api.cloudfoundry.com
Successfully targeted to [http://api.cloudfoundry.com]

16. When prompted by vmc, answer with y or n as shown below.

ratul@ubuntu:~/myPlayApp/dist/myPlayApp-1.0-SNAPSHOT$ vmc push
Would you like to deploy from the current directory? [Yn]: y
Application Name: myPlayApp
Detected a Standalone Application, is this correct? [Yn]: y
1: java
2: node
3: node06
4: ruby18
5: ruby19
Select Runtime [java]: 1
Selected java
Start Command: java $JAVA_OPTS -Dhttp.port=$VCAP_APP_PORT -cp "`dirname $0`/lib/*" play.core.server.NettyServer `dirname $0`
Application Deployed URL [None]: myPlayApp.${target-base}   
Memory reservation (128M, 256M, 512M, 1G, 2G) [512M]: 256M
How many instances? [1]: 1
Create services to bind to 'myPlayApp'? [yN]: n
Would you like to save this configuration? [yN]: y
Manifest written to manifest.yml.
Creating Application: OK
Uploading Application:
  Checking for available resources: OK
  Processing resources: OK
  Packing application: OK
  Uploading (80K): OK  
Push Status: OK
Staging Application 'myPlayApp': OK                                                 
Starting Application 'myPlayApp': OK     

ratul@ubuntu:~/myPlayApp/dist/myPlayApp-1.0-SNAPSHOT$

17. In your browser you can now open your application - type in myPlayApp.cloudfoundry.com - and we are done!

18. The broswer will show something like :

Your new application is ready.App port : 61897 and host: 172.30.50.24