Episode 15: Using a CAPTCHA in your Google App Engine Application

February 22, 2010 4 comments

Welcome to Episode 15. In this episode, we shall cover how to incorporate a CAPTCHA in your application. The CAPTCHA solution that I will be demonstrating over here is the ReCAPTCHA project found here. To quote from its Wikipedia entry, a CAPTCHA is a type of challenge-response test used in computing to ensure that the response is not generated by the computer. Usually, one sees one or two words shown to us that we need to enter and they need to match before our request submission is accepted by the server.   

See it in Action

  

Let us first see the application that we shall be developing in its final form. The main screen is a dummy form that accepts one field and displays the CAPTCHA to the user as shown below:   

   

If the CAPTCHA response is entered correctly and the submit button is clicked, the response is sent to the server. The server side validates the CAPTCHA challenge-response and will display an “all is well” message as shown below:   

   

If the CAPTCHA response is not entered correctly, the server side will display an “all is not well” message as shown below:   

   

ReCAPTCHA Project

  

We shall be using the ReCAPTCHAProject present at http://recaptcha.net/. The ReCAPTCHAproject is comprehensive and provides support for a variety of server side platforms, hence I have gone with it. I do not have a particular preference for it, except that most people recommended this to me and hence I decided to try it out. I was able to integrate it easily and quickly into a recent App Engine application that is currently live, so I can vouch for its ease of integration and which is what I shall demonstrate in this experiment.
 
To get started with ReCAPTCHA, we need to do 2 things:   

1. Get your private/public keys for the site name hat your application will be running on.

  

Follow these steps:   

  1. First get yourself a ReCAPTCHAaccount. Go to http://recaptcha.net/whyrecaptcha.html and click on the Sign up Now! button.
  2. This will bring up a login form, but you click on the “New to reCaptcha? Sign up” link. Here you can register by giving a username/password/email . Click on the Sign up now button.
  3. On successful registration, you will signed in automatically and led to a Create a reCAPTCHA key page. Here you need to enter the site name that you will be using ReCAPTCHAon. Remember you can get ReCAPTCHAkeys for more than 1 site and you can manage it all from your account itself. For e.g. you can get a key for localhost since that will be what we need to test on first before deploying to the Google App Engine cloud. And then you will atleast need one based on the Application ID of your Google App Engine application. So for e.g. if my application ID is gaejexperiments, then my full site url is http://gaejexperiments.appspot.com. So you will need to enter gaejexperiments.appspot.com in the Domain form field.
  4.  Click on Create Key. For e.g. shown below is the series of screens that I got when I wanted keys for localhost. You will need to do the same if you wish to test the code first on your local system i.e. localhost.

  

    

Then when you click Create Key, you will be presented the Public Key and Private Key as shown below:   

   

Please note down the Public and Private Key since you will need to use them in your application. The usage is straightforward and mentioned on the screen and I will be repeat it here:   

  • Use the Public Key in your HTML page (Javascript) to communicate with the ReCAPTCHA server to get the challenge (image).
  • Use the Private Key in your server side Java code that will be used to communicate with the Server to verify the challenge + response submitted by the form in the above step.

2. Download the JAR file that we shall be using at the Server side to verify the CAPTCHA challenge and response.

  

To do the server side validation, you need to download a JAR  file that is available from the following url : http://code.google.com/p/recaptcha/downloads/list?q=label:java-Latest. Go to the URL and you shall be presented with a screen as shown below:   

   

The JAR file is required and it encapsulates all communication between our application and the ReCAPTCHA Servers.   

Download the zip file, expand it in  an appropriate folder and you will find the recaptcha4j-0.0.7.jar present in that.  We will need it later on to be referenced in your App Engine Eclipse Project.   

Developing our Application

  

The first thing to do is to create a New Google Web Application Project. Follow these steps:   

1. Either click on File –> New –> Other or press Ctrl-N to create a new project. Select Google and then Web Application project. Alternately you could also click on the New Web Application Project Toolbar icon as part of the Google Eclipse plugin.
2. In the New Web Application Project dialog, deselect the Use Google Web Toolkit and give a name to your project. I have named mine GAEJExperiments. I suggest you go with the same name so that things are consistent with the rest of the article, but I leave that to you. In case you are following the series, you could simply use the same project and skip all these steps altogether. You can go straight to the Servlet Development section.
3. Click on Finish.   

This will generate the project and also create a sample Hello World Servlet for you. But we will be writing our own Servlet.   

Add the recaptcha4j-0.0.7.jar file to your Project classpath

  

The recaptcha4j-0.0.7.jar file that we downloaded needs to be added to the runtime and compile time CLASSPATH of your App Engine Eclipse Project. Follow these steps:
a) Copy recaptcha4j-0.0.7.jar to the WEB-INF\lib folder of your project. This folder is present in the war sub folder of your main project structure.
b) Go to Project properties –> Java Build Path and reference the JAR file (recaptcha4j-0.0.7.jar) in the Java Build Path of your application as shown below:   

   

The Front end HTML form [captcha.html]

<html xmlns="<a href="http://www.w3.org/1999/xhtml">http://www.w3.org/1999/xhtml</a>" lang="en" xml:lang="en">
<head>
 <title>ReCaptcha Integration</title>
<script type="text/javascript" src="<a href="http://api.recaptcha.net/js/recaptcha_ajax.js%22%3E%3C/script">http://api.recaptcha.net/js/recaptcha_ajax.js"></script</a>>
<script type="text/javascript">
 function PreloadCaptcha() {
  showRecaptcha();
 }
   

function showRecaptcha() {
    Recaptcha.create("YOUR_PUBLIC_KEY", "dynamic_recaptcha_1", {
          theme: "white",
          callback: Recaptcha.focus_response_field
    });
  }  

 
 var xmlhttp;
 function submitFormData(name)
 {  

 //alert("Message");
 xmlhttp=null;
 if (window.XMLHttpRequest)
   {// code for IE7, Firefox, Opera, etc.
   xmlhttp=new XMLHttpRequest();
   }
 else if (window.ActiveXObject)
   {// code for IE6, IE5
   xmlhttp=new ActiveXObject("Microsoft.XMLHTTP");
   }
 if (xmlhttp!=null)
   {
   xmlhttp.onreadystatechange=state_Change;
   var url = "postformdata";
   var params = "name="+name+"&recaptcha_challenge_field=" + Recaptcha.get_challenge() + "&recaptcha_response_field="+Recaptcha.get_response();
   var status = document.getElementById("status");
   status.innerHTML = "<img src='img/ajax-loader.gif'><b>Submitting your data. Please wait...</b>";
   xmlhttp.open("POST",url,true);
   xmlhttp.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
   xmlhttp.setRequestHeader("Content-length", params.length);
   xmlhttp.setRequestHeader("Connection", "close");  
   xmlhttp.send(params);
   }
 else
   {
   alert("Your browser does not support XMLHTTP.");
   }
 }
 
 function state_Change()
 {
 if (xmlhttp.readyState==4)
   {// 4 = "loaded"
   if (xmlhttp.status==200)
     {
     // 200 = "OK"
    var status = document.getElementById("status");
    status.innerHTML = xmlhttp.responseText;;
    Recaptcha.reload();
    setTimeout(function() {
      status.innerHTML = "";
    }, 3000);   
     }
   else {
    var status = document.getElementById("status");
    status.innerHTML = xmlhttp.responseText;;
    Recaptcha.reload();
    setTimeout(function() {
      status.innerHTML = "";
    }, 3000);   
   }
   }
 }
 </script>
</head>  

<body onload="PreloadCaptcha()">
       <FORM NAME="dataform">
       <TABLE>
     <TR>
     <TD><b>Your name:</b></TD>
     <TD><INPUT NAME="txtName"/></TD>
     </TR>
     <TR>
     <TD colspan="2"><div id="dynamic_recaptcha_1"></div></TD>
     </TR>
     <TR>
                    <TD colspan="2"><INPUT type="button" value="Submit Data" name="btnSubmitData" onClick="submitFormData(txtName.value); return true"></TD>
                    </TR>
                    <TR>
     <TD colspan="2"><div id="status"/></TD>
     </TR>
       </TABLE>
       </FORM>
</body>
</html>  

Let us go through the important parts of the code:  

1. The first part to notice is that I have referenced the javascript file for the ReCAPTCHA code as shown :

<script type="text/javascript" src="http://api.recaptcha.net/js/recaptcha_ajax.js"></script>
 

2. There is a javascript function call being made on the load of the body (<body onload=”PreloadCaptcha()”>). This invokes the function showRecaptcha() that is shown below:

Recaptcha.create("YOUR_PUBLIC_KEY", "dynamic_recaptcha_1", {
          theme: "white",
          callback: Recaptcha.focus_response_field
    });

 

The Recaptcha class is available from the javascript file and we use the create method. The first parameter is the PUBLIC_KEY that you get when you registered at ReCaptcha for  your site name. If you are testing it locally, this will be the Public Key for your localhost site. The second parameter is a DIV element that is present in the form.

The create() method on successful invocation will populate the DIV element with the CAPTCHA challenge/response fields.

3. The rest of the code is standard AJAX stuff. The submit button invokes the submitDataForm()  that does the following: 

  • It will invoke the URL : /postformdata that will be our servlet that does the verification. The servlet code is discussed in the next section
  • It will form the request parameter string as shown below:
    var params = “name=”+name+”&recaptcha_challenge_field=” + Recaptcha.get_challenge() + “&recaptcha_response_field=”+Recaptcha.get_response();
  • Note the two fields : recaptcha_challenge_field and recaptcha_challenge_response_field. We get the two values from the Recaptcha class methods, get_challenge() and get_response() respectively. The get_challenge() is what was provided by the ReCAPTCHA Server and the get_response() is what was entered by the user.
  • Finally we do a POST to the server and collect the response. The response is that displayed in another DIV element “status”.

Please note that the code is just for demonstration purpose and may not represent the best practices or most efficient way of writing JavaScript, AJAX, etc.

The Servlet [PostFormDataServlet.java]

 

package com.gaejexperiments.captcha; 

import java.io.IOException; 

import javax.servlet.http.*; 

//RECAPTCHA
import net.tanesha.recaptcha.ReCaptchaImpl;
import net.tanesha.recaptcha.ReCaptchaResponse; 
@SuppressWarnings("serial")
public class PostFormDataServlet extends HttpServlet {
 
 public void doPost(HttpServletRequest req, HttpServletResponse resp)
   throws IOException {
  resp.setContentType("text/plain");
  String strResponse = "";
  try {
   
/*
 * CAPTCHA CHECK
 *
 *
 */
   //First validate the captcha, if not -- just get out of the loop
   String challenge = (String) req.getParameter("recaptcha_challenge_field");
   String response = (String) req.getParameter("recaptcha_response_field");
   if ((challenge == null) || (response == null)) {
    throw new Exception("Your words did not match. Please try submitting again.");
   }
   
      String remoteAddr = req.getRemoteAddr();
      ReCaptchaImpl reCaptcha = new ReCaptchaImpl();
     
      reCaptcha.setPrivateKey("YOUR_PRIVATE_KEY");
     
      ReCaptchaResponse reCaptchaResponse =
          reCaptcha.checkAnswer(remoteAddr, challenge, response);
     
      if (!reCaptchaResponse.isValid()) {
       //RECAPTCHA VALIDATION FAILED
       throw new Exception("Your words did not match. Please try submitting again.");
      }
   
      strResponse = "Your record has been accepted and you did a good job entering the two words. Thank you";
  }
  catch (Exception ex) {
   strResponse = "You record was not accepted. Reason : " + ex.getMessage();
  }
  resp.getWriter().println(strResponse);
 }
}

Let us go through the important parts in the code:  

1. Notice that we are importing  2 classes : net.tanesha.recaptcha.ReCaptchaImpl and net.tanesha.recaptcha.ReCaptchaResponse at the beginning. These are the implementation classes that encapsulate the verification with the ReCAPTCHA Server and the response from the ReCAPTCHA Server respectively.  

2. We first extract out the challenge and the response fields as passed by our form.  

String challenge = (String) req.getParameter("recaptcha_challenge_field");
String response = (String) req.getParameter("recaptcha_response_field");

3. We also need the Remote IP Address that is making the request.  

String remoteAddr = req.getRemoteAddr();

4. We then instantiate an instance of the ReCaptchaImpl class. We set the Private Key that we got during our registration for the site localhost. Please use your key over here. And then we make a call to their server by invoking the checkAnswer method. The checkAnswer method takes 3 parameters : challenge, response and the Remote Address.  

      ReCaptchaImpl reCaptcha = new ReCaptchaImpl();
     
      reCaptcha.setPrivateKey("YOUR_PRIVATE_KEY");
     
      ReCaptchaResponse reCaptchaResponse =
          reCaptcha.checkAnswer(remoteAddr, challenge, response);

5. We will receive an instance of the ReCaptchaResponse class. We simply use a utility method isValid() to determine if the response entered for the challenge was correct. And depending on that we send back an appropriate response back to the browser, which is then displayed to the user.  

Servlet Configuration

To complete our Servlet development, we will also need to add the <servlet/> and <servlet-mapping/> entry to the web.xml file. This file is present in the WEB-INF folder of the project. The necessary fragment to be added to your web.xml file are shown below. Please note that you can use your own namespace and servlet class. Just modify it accordingly if you do so.   

<servlet>
<servlet-name>PostFormDataServlet</servlet-name>
<servlet-class>com.gaejexperiments.captcha.PostFormDataServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>PostFormDataServlet</servlet-name>
<url-pattern>/postformdata</url-pattern>
</servlet-mapping>

Running the application locally

I am assuming that you have already created a new Google Web Application Project and have created the above Servlets, web.xml , etc. We shall be running this episode within our local development server only since the keys that I have used are for localhost. Before you deploy the application to the actual Google App Engine infrastructure using your Application ID, do remember to get yourself the appropriate public/private ReCAPTCHA keys for your site name.  

So assuming that all is well, we will run our application, by right-clicking on the project and selecting Run As –> Web Application. Launch the browser on your local machine and navigate to http://localhost:<YourLocalPort>/captcha.html  

Conclusion

In this episode, we saw how to incorporate a CAPTCHA in our Google App Engine application. Please remember that you will need to determine if you really need a CAPTCHA in your application. There are some good points mentioned in this article on various other techniques that can be used to address the same problems.   

Till the next episode ….

Advertisements
Categories: Uncategorized

Episode 14: Writing a Chrome Extension powered by App Engine

December 31, 2009 7 comments

Welcome to Episode 14. In this episode we shall cover how you can get started with writing Google Chrome Extensions. We shall also drive the Extension via our App Engine application. We have already seen in earlier episodes how your App Engine applications participated within Google Wave, Google Talk, etc. So this episode takes it further and shows how you can write a Google Chrome Extension whose data will be provided by our App Engine Application.

Google Chrome Extension

Google Chrome browser since its release has got a fair amount of developer mind share. Recently they addressed extensibility of the Chrome browser by announcing support for writing your own extensions. So the focus is now on the developer to take this extension mechanism, harness it and produce some great extensions that extend the capabilities of the browser. You can read here the official announcement for Extension support over here.

Google Chrome Dev Channel version

Note: The first thing that you need to do is download a copy of the Chrome Browser from their Dev Channel because extensions are currently supported there till they come into the mainstream release. So given that, make sure that you download that version of Google Chrome browser.

The Beta channel or the Dev channel is available at : http://www.chromium.org/getting-involved/dev-channel.

Depending on your Operation System, download the appropriate version and install it. To check that you have extension enabled correctly, launch the Chrome Browser and click on the Settings icon as shown below. You should see the Extensions menu item available.

It is a good idea to take a look at the existing Chrome Extensions gallery that lists several excellent extensions that you can start using today inside of your Google Chrome browser. The gallery is available at Gallery : https://chrome.google.com/extensions. This page also contains a link where you can submit your extension too for the world to use!

See it in Action

Let us see our Google Chrome Extension that we shall build in this episode. The extension is a simple Reminder application. It shows the current Reminders that you have set up for the month. All the data is saved and managed by our App Engine application and the Chrome Extension simply requests and fetches the data for the current month.

Shown below is the Chrome Browser screen with our Extension available as a little red icon in the Toolbar.

When we click the icon, it will fetch the reminders that I have setup in the current month (December) and display them as shown below:

Sounds good? We shall split up the details into two parts:

Part I : The first part will deal with developing/deploying our Server side application. This application will be deployed as a Google App Engine Java application. We shall keep the implementation simple and not do any database operations, etc. The focus is on the second part where we need to understand what it takes to get a simple Chrome Extension up and running, and which is powered by this Google App Engine application.

Part II: The second part will deal with how you can write a Google Chrome Extension. We will consider the bare minimum that you need to do to get your extension working. I will leave the design and look/feel aspects to you.

Alright then, let’s get started.

Developing our Java Web Application  : MyReminderApp

The first thing to do is to create a New Google Web Application Project. Follow these steps:

1. Either click on File –> New –> Other or press Ctrl-N to create a new project. Select Google and then Web Application project. Alternately you could also click on the New Web Application Project Toolbar icon as part of the Google Eclipse plugin.

2. In the New Web Application Project dialog, deselect the Use Google Web Toolkit and give a name to your project. I have named mine MyReminderApp. I suggest you go with the same name so that things are consistent with the rest of the article.

3. Click on Finish. This will generate the project and also create a sample Servlet named MyReminderAppServlet for you. We shall be modifying this Servlet code appropriately.

Design discussion

Now that the project template is ready, let me discuss in brief what we shall be building:

1. A simple Reminder App that contains a list of Reminders. Each Reminder is kept simple and has 3 attributed only: The day, month and the description. For e.g. [25,12,Christmas Day], [19,12,My Birthday], [1,1,New Year Day], [31,7,Pay Taxes] and so on.

2. So we will define a Reminder.java class that simply has getter/setters for the 3 attributes : day, month and description

3. We will then write a ReminderService.java class (A Singleton) that implements two simple functions:

  • ArrayList<Reminder> findRemindersByMonth(String month)
    This method returns the reminders for a particular month. We pass in “1” and we get reminders defined for January, etc.
  • ArrayList<Reminder> findCurrentMonthReminders()
    This method returns the reminders for the current month. The method determines which month is currently on and retrieves reminders for that month only. And this is precisely the method that will be invoked when our extension asks for the current reminders.

4. Finally, we shall modify the MyReminderAppServlet.java so that it implements a simple REST like interface for retrieving the reminders. If the request stream contains a parameter named month, then the reminders will be returned for that month by invoking the findRemindersByMonth method. If the request stream does not contain a parameter named month, then the current month reminders will be returned by invoking the findCurrentMonthReminders() method. The response will be returned in XML format.

Given next are the code listings, that should be straight forward to understand now:

Reminder.java

package com.reminderapp;

public class Reminder {

private String day;
private String month;
private String event;


public Reminder(String day, String month, String event) {
super();
this.day = day;
this.month = month;
this.event = event;
}
public String getDay() {
return day;
}
public void setDay(String day) {
this.day = day;
}
public String getMonth() {
return month;
}
public void setMonth(String month) {
this.month = month;
}
public String getEvent() {
return event;
}
public void setEvent(String event) {
this.event = event;
}
}

The Reminder class simply has 3 attributes : day , month and year. There are getter/setter for each of these attributes and a constructor that takes in all the 3 attributes. We are keeping at as String for now to keep things simple.

ReminderService.java

package com.reminderapp;

import java.util.ArrayList;
import java.util.Calendar;

public class ReminderService {

private static ReminderService _instance = null;

private ArrayList<Reminder> _reminderList = null;

private ReminderService() {
initializeData();
}

public static ReminderService getInstance() {
if (_instance == null) {
_instance = new ReminderService();
}
return _instance;
}

public ArrayList<Reminder> findRemindersByMonth(String month) {
ArrayList<Reminder> _results = new ArrayList<Reminder>();
for (Reminder reminder : _reminderList) {
if (reminder.getMonth().equals(month)) {
_results.add(reminder);
}
}
return _results;
}

public ArrayList<Reminder> findCurrentMonthReminders() {
ArrayList<Reminder> _results = new ArrayList<Reminder>();
String currentMonth = "" + (Calendar.getInstance().get(Calendar.MONTH)+1);
for (Reminder reminder : _reminderList) {
if (reminder.getMonth().equals(currentMonth)) {
_results.add(reminder);
}
}
return _results;
}

private void initializeData() {
_reminderList = new ArrayList<Reminder>();
Reminder R1 = new Reminder("1","1","New Year Day");
Reminder R2 = new Reminder("26","1","Republic Day");
Reminder R3 = new Reminder("15","8","Independence Day");
Reminder R4 = new Reminder("25","12","Christmas Day");
Reminder R5 = new Reminder("31","12","New Year Eve");

_reminderList.add(R1);
_reminderList.add(R2);
_reminderList.add(R3);
_reminderList.add(R4);
_reminderList.add(R5);

}

}

The ReminderService is a singleton class. And we are initializing some data here via the initializeData() call. We are not complicating things by using a database here but this is something that should ideally be there in a real application. As you can notice we add 5 sample records and for the current month i.e. December (12), we have two events. The two methods findRemindersByMonth and findCurrentMonthReminders are straightforward in which it goes through the reminderList and matches the records as appropriate.

MyReminderAppServlet.java

package com.reminderapp;

import java.io.IOException;
import java.util.ArrayList;

import javax.servlet.http.*;

@SuppressWarnings("serial")
public class MyReminderAppServlet extends HttpServlet {
public void doGet(HttpServletRequest req, HttpServletResponse resp)
throws IOException {
resp.setContentType("text/xml");
//resp.getWriter().println("Hello, world");

String month = (String)req.getParameter("month");
ArrayList<Reminder> _results = null;
if (month == null) {
_results = ReminderService.getInstance().findCurrentMonthReminders();
}
else {
_results = ReminderService.getInstance().findRemindersByMonth(month);
}

StringBuilder SB = new StringBuilder();
SB.append("<?xml version=\"1.0\"?>");
SB.append("<ReminderList>");
for (Reminder reminder : _results) {
SB.append("<Reminder>"+reminder.getDay()+"/"+reminder.getMonth()+" : " + reminder.getEvent()+"</Reminder>");
}
SB.append("</ReminderList>");
resp.getWriter().println(SB.toString());
}
}

Let us go through the code for the MyReminderAppServlet in brief:

1. It implements the doGet(…) method and inspects if a request parameter named month is passed or not.

2. If the month parameter is passed, its value is extracted and the findRemindersByMonth method is invoked. Else the findCurrentMonthReminders is invoked.

3. The result is then formatted into a simple XML structure and then it is returned back in the response stream.

Configuring the Servlet

We need to make sure that the entry for  <servlet/> and <servlet-mapping/> in the web.xml file. is there for MyReminderAppServlet. This file is present in the WEB-INF folder of the project. The necessary fragment to be added to your web.xml file are shown below.

<servlet>
<servlet-name>MyReminderApp</servlet-name>
<servlet-class>com.reminderapp.MyReminderAppServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>MyReminderApp</servlet-name>
<url-pattern>/myreminderapp</url-pattern>
</servlet-mapping>

Deploying the Application

To deploy the application, you will need to first create your Application ID. The Application Identifier can be created by logging in at http://appengine.google.com with your Google Account. You will see a list of application identifiers already registered under your account (or none if you are just getting started). To create a new Application, click on the Create Application button and provide the Application Identifier as requested. Please note this name down since you will be using it for deployment.

For e.g. I have registered an application identifier named itsareminder.

To deploy the application, follow these steps (they should be familiar to you now):

  1. Click on the Deploy Icon in the Toolbar.
  2. In the Deploy dialog, provide your Email and Password. Do not click on Deploy button yet.
  3. Click on the App Engine Project settings link. This will lead you to a dialog, where you need to enter your Application ID [For e.g. my Application Identifier itsareminder]
  4. Click on OK. You will be lead back to the previous screen, where you can click on theDeploy button. This will start deploying your application to the GAEJ cloud. You should see several messages in the Console window as the application is being deployed.
  5. Finally, you should see the message “Deployment completed successfully”.

Testing our Reminder Service

To test our Reminder Service, we can use the http://<AppId>.appspot.com/myreminderapp URL. You can replace it with your AppId and endpoint as appropriate to your application. For e.g. I have hosted it at http://itsareminder.appspot.com/myreminderapp and putting this in the browser returns me the Reminders that are set for the current Month i.e. Month of December. I get the following response XML:

<ReminderList><Reminder>25/12 : Christmas Day</Reminder><Reminder>31/12 : New Year Eve</Reminder></ReminderList>

Now that the development of the Reminder Service is complete, we can move on to the next part i.e. developing the Chrome Extension.

Developing the Reminder Service Chrome Extension

The official documentation on developing a Chrome Extension is quite good and you can refer it for more details. I will keep things to a minimum here and cover only what is required to get the Chrome Extension up and running. Follow these steps:

1. Create a folder on your machine. For e.g. I have created the following folder on my machine in C:\. It is c:\MyReminderExtension. All the necessary files that we create for the extension will reside in this folder.

2. To create an extension, we need at the minimum 3 files. The three files are an icon file for your extension. This icon will be visible in the toolbar as we saw earlier. The second file is a manifest file called manifest.json. The manifest provides important metadata about the extension like name,version, permissions, action, etc. And the final file is the html file in which you will keep the core of your extensions code written in Javascript.

3. So the first thing that I do is to create a manifest file for the extension. The manifest.json file for the Reminder Service is shown below:

{
"name": "My Reminder Extension",
"version": "1.0",
"description": "Reminders based on month",
"browser_action": {
"default_icon": "myreminder.png",
"popup": "reminder.html"
},
"permissions": [
"http://itsareminder.appspot.com/"
]
}

As you can see this is pretty simple. There is the standard name/version/description that you can give as per your preferences. The browser_action has two important attributes. The default_icon refers to the icon file that contains the icon for our extension. So this means that we will have the myreminder.png file present in the same folder. The popup attribute indicates which HTML file to be invoked when the icon is clicked. The permissions attribute indicates which domain the extension will be accessing to retrieve the data. In our case it is the http://itsareminder.appspot.com domain at which we hosted our application.

4. As shown above, we will create a file named myreminder.png in the same folder i.e. C:\MyReminderExtension.

5. Finally, we have the reminder.html file that will contain the code of the extension. It is listed below:


<style>
body {
min-width:200px;
overflow-x:hidden;
}

</style>

<script>
var req = new XMLHttpRequest();
req.open(
"GET",
"http://itsareminder.appspot.com/myreminderapp",
true);
req.onload = showReminders;
req.send(null);

function showReminders() {
var reminders = req.responseXML.getElementsByTagName("Reminder");

for (var i = 0, reminder; reminder = reminders[i]; i++) {
var reminderDIV = document.createElement("div");
reminderDIV.innerHTML = reminder.textContent;
document.body.appendChild(reminderDIV);
}
}

</script>

The code above is focussed around the <script> tag. It requires knowledge about AJAX and the XMLHttpRequest object along with some Javascript basics. The core pieces are mentioned below:

1. We use the XMLHttpRequest to make a call to our Reminder Service. The end point is where our servlet is present to serve the requests i.e. http://itsareminder.appspot.com/myreminderapp.

2. When we get the response XML, we call the showReminders() method. This method simply parses the <Reminder> tags and for each tag, it creates a <div> element in which the text content is the Reminder Event Text returned. Its appends each of the <div/> elements to the document.body as shown.

Simple isnt it? Now let us see how to install this extension into your Chrome browser to test it out.

Installing the Reminder Chrome Extension

At this point in time, I will assume that you have the following 3 files created in the C:\MyReminder folder:

  • manifest.json
  • myreminder.png
  • reminder.html

You could have created different files but just remember that you need the icon, manifest and your html file that implements the Chrome extension code.

Once you have that, follow these steps:

1. Launch Google Chrome Browser. You need to go the Extensions page that allows you to view the current extensions installed and from where you can either install new extensions/reload/uninstall,etc. There are two ways to go to that page:

  • In the address, type in chrome://extensions
  • From the settings option as mentioned in the start of this article, click on the “Extensions” menu option.

2. You should see a screen that looks something like this:

What this means is that there are no extensions currently installed in my Chrome browser. If there were any, they would be listed over here.

3. Click on the Load unpacked extension button. This will bring up a dialog box from where you can navigate to the folder where your extension files are present. In our case, it was the C:\MyReminderExtension and so I navigate to that and click on OK as shown below:

4. This will install the Chrome Extension in the browser as shown below:

5. Finally, we can test out the browser by clicking on the icon and it will display the list of Reminders for the current month.

If you have newer versions, all you need to do is simply click on Reload option for the extension. Alternately, you can also Uninstall and load it again.

Some points to note:

1. You have not completely installed this extension. What this means is that if you close the Chrome Browser and launch it again, this Extension will not be visible.

2. The next step would be to visit the official documentation at http://code.google.com/chrome/extensions/getstarted.html and also go to the gallery and submit your Extension. The gallery is present at : https://chrome.google.com/extensions and you will find a link to submit your extension. Click that and follow the steps to submit your extension. All you will need to provide from your side will be a ZIP file of the folder that we mentioned above.

Conclusion

This concludes the final episode of the year 2009. In this episode we covered how to write a Google Chrome Extension and power it via a Google App Engine application.

I hope that the new year takes you further into writing Cloud applications using Google App Engine. I will continue to explore tools/languages/APIs/extensions that work in coherence with the Google App Engine. The journey or might I say “Experiments” are far from over. Have a great 2010 and thanks for reading this blog. Its been a joy to share my experiments with all of you.

Categories: Uncategorized Tags:

Episode 13: Using the Blobstore Java API

December 18, 2009 7 comments

Welcome to Episode 13 of this series. It has been a while since the last episode and to compensate I thought why not cover something that has just been released. In keeping with that line, this episode will cover the recently released Blobstore API in the 1.3.0 release of the Google App Engine SDK.  

Using the Blobstore API, we shall cover how to build an application that mimics several sites that allow you to upload a picture and send a Tweet. The Tweet will then contain a link back to the image that you uploaded. Blobstore API now makes all this possible along with several other applications that you could envision like file sharing, document management, etc.  

What is the Blobstore API?

The Blobstore API allows your application to store blobs of information, where each blob can be upto 50MB in size. These blobs are typically large files like video and images. This is different from the datastore API that does not allow for saving objects upto this large size. The API documention for the Blobstore API is provided here.  

The Blobstore API provides two types of functions:  

  • An ability to upload and save the blob automaticallyThe BlobstoreService which is provided by the com.google.appengine.api.blobstore.BlobstoreService allows us to specify a URL where users can upload their large files. You can think of this url as the action element in the HTML form. The implementation at this URL is internal to the BlobstoreService. But what it does is significant. It will extract out the file contents that you uploaded and store it as a Blob in the database. Each blob that is stored in the database is associated with the a Blob Key. This Blob key is then provided to your url and you can then use the Blob Key to do anything within your application. In our case, we form a url that is tweeted to the users who can then view the picture that we uploaded.
  • An ability to serve or retrieve the blob.The BlobstoreService also provides an ability to serve or retrieve the blob that was saved successfully. It will provide the blob as a response that could then use as a source in an <img> element for example. All you need to do is provide it the Blob Key and the response stream and in return, it will provide the content properly encoded as per its type that you could then use.

It will be clear as we work out the application in the next few sections.  

Note: The Blobstore API is enabled in your application only if you enable “billing” for your applications. What this means is that you can try out the API with your local development server but if you deploy the same on the App Engine cloud, your application will not be able to use the API if it has not been enabled for “billing”. So keep that in mind before leaping with joy that your new application enabled with large file uploads, etc is now ready to start functioning in the cloud. In case you try to use any Blobstore APIs without enabling billing in your application, you will get a FeatureNotSupported exception and the text as: “The Blobstore API will be enabled for this application once billing has been enabled in the admin console.”  

Prerequisites

Google announced the release of version 1.3.0 of the App Engine SDK for both Python and Java. We need to set our development environment with this release of the SDK since the Blobstore API was released in this version.  

So the first thing that you need to do is to download the latest release 1.3.0 of the App Engine SDK and upgrade your Eclipse development environment with that. There are two ways you can do that:  

a. Use the Update features of Eclipse to download and install the latest SDK  

b. Download the SDK from the following link. Make sure you download the Java SDK. Further to the download, inside of your Eclipse IDE, go to Windows -> Preferences and then Google -> App Engine SDK. Click on Add and provide the path to the expanded 1.3.0 distribution on your machine.  

I had covered in detail how to upgrade your SDK for your Eclipse IDE when new releases come out. This was covered in Episode 5, in which we upgraded from 1.2.5 -> 1.2.6. Readers are advised to take a look at it in case they are not familiar. The process remains the same.  

Tweet My Picture in Action

The application that we are going to develop is called Tweet My Picture. There are several sites now that allow you to upload a picture to their site and then send a Tweet. The Tweet contains a simple description and the url to the picture. An example is TwitPic.  I thought the Blobstore API now actually makes an application of this kind way too easy to create, so why not demonstrate that. We will not be creating a polished looking application but rather demonstrating the API and showing that it works. Remember that all the instructions later on deploying and running this application will happen in the local development server and not the live App Engine site because I need to enable billing in order to utilize the Blobstore Service. But the best part is that the local development server works fine, so we can do our complete development and testing on it. Then you are the best judge of writing your own application that utilizes the Blobstore API and making it a billable application when you go live.  

Before we get down to code, we should see the end result.  

Assuming that I have started my local development server, I navigate to the index.jsp page that is shown below:  

This will display a simple page, where you provide the following information:  

  • Your Twitter UserId and Password
  • The image file that you wish to upload. Simply click the Choose File button and select a file to upload.

  

Then when you click the Upload Picture button, what happens is the following:  

1. The file contents that  you uploaded will be stored by the Blobstore API into the datastore automatically for you. It will provide you with the Blob Key for the blob that was saved.  

2. We use the Blob Key to create a simple Tweet message containing the link. And then we use Twitter4J to send the Tweet.  

3. Finally a page is displayed indicating the status of the Tweet, the Tweet contents and it also shows the image that you uploaded. This page is shown below:  

   

Now, switch to your Twitter client and you will notice that the Tweet has been posted. A sample screenshot is shown below.The url is pointing to the localhost but that is because I wish to do my testing locally and I have not enabled billing for my live application at the appspot.com domain.  

  

Clicking on the Tweet link, takes you back to an page that shows you the picture as below:  

  

Nice, isn’t it? Do not limit yourself only to this example. You can now start thinking on creating serious document management, file sharing, etc applications and use the App Engine infrastructure to host it. So let’s get coding now.  

Implementing Tweet My Picture

The first thing to do is to create a New Google Web Application Project. Follow these steps:  

1. Either click on File –> New –> Other or press Ctrl-N to create a new project. Select Google and then Web Application project. Alternately you could also click on the New Web Application Project Toolbar icon as part of the Google Eclipse plugin. Please make sure that you have installed the latest 1.3.0 version of the App Engine SDK and are using that as the API for this version. Shown below is the New Project screenshot where you should see an additional SDK 1.3.0 and need to select that.  

  

2. In the New Web Application Project dialog, deselect the Use Google Web Toolkit and give a name to your project. I have named mine GAEJExperiments. I suggest you go with the same name so that things are consistent with the rest of the article, but I leave that to you. In case you are following the series, you could simply use the same project and skip all these steps altogether. You can simply go to the next part i.e. the Servlet code.  

3. Click on Finish. This will generate the project and also create a sample Hello World Servlet for you. But we will be writing our own Servlet.  

Adding Twitter4J Jar to the Classpath and WEB-INF\lib folder

Since we will be using the excellent Twitter4J library for posting to Twitter, you need to download the latest version of the JAR file. Download it from here. I have used twitter4j-2.0.10.jar. You need to copy this file to the WEB-INF\lib folder of the project. Also add the JAR file to the Build Classpath of the Eclipse project.  

index.jsp

The first page is the index.jsp page. This is a simple page that will have a form that accepts the Twitter UserId/Password along with the file to be uploaded. The code is shown below. Just a quick note that all the JSP files will go into the war folder of the project.  

  

<%@ page import="com.google.appengine.api.blobstore.BlobstoreServiceFactory" %>
<%@ page import="com.google.appengine.api.blobstore.BlobstoreService" %>  

<%
    BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
%>
<html>
    <head>
        <title>Tweet My Picture</title>
    </head>
    <body>
        <img src="tweet.png"/>
        <h1>Tweet My Picture</h1>
        <hr/>
        <h2>Upload a picture and send an automatic Tweet</h2>
        <form action="<%= blobstoreService.createUploadUrl("/upload") %>" method="post" enctype="multipart/form-data">
            Twitter User Id : <input type="text" name="twitter_userid"/><br/>
            Twitter Password : <input type="p" name="twitter_password"/><br/>
            File :
            <input type="text" name="filename"/>
            <input type="file" name="myTweetPic"/>
            <input type="submit" value="Upload Picture"/>
        </form>
</html>  

 

Let us look at the main points in the code above:  

1. It has a HTML <form/> that accepts the twitter userid and password. We also have an  <input/> element of type file, that allows the user to select the file from his machine to be uploaded.  

2. The action attribute of the FORM is where you should pay attention. The Form action as you know submits the request parameters to a particular server side resource like a Servlet/JSP, etc. The action here is created via a helper function provided by the Blobstore API itself. Notice that first we are getting an instance of the Blobstore service via the BlobstoreServiceFactory as shown below:  

BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();  

Then we are using a helper function called blobstoreService.createUploadURL(“SUCCESS_PATH”) for the action. Let us spend some time on that since it is important and will help us understand what the BlobstoreService does behind the scenes.  

If you launch your local development server and navigate to the index.jsp page and do a view source on the HTML, you will find that the action that is generated by the createUploadURL(…) method looks something like this:  

action=”/_ah/upload/agp0d2VldG15cGljchsLEhVfX0Jsb2JVcGxvYWRTZXNzaW9uX18YEgw”. You may wonder where did our SUCCESS_PATH i.e. /upload go away. Let me explain:  

When you do an upload, the request hits an internal URL in the App Engine infrastructure that will do the following:  

1. The BlobstoreService implementation inside of AppEngine will extract out the Blob and save it for you automatically in the datastore. You do not have to do anything special. It does this for the all the Blobs that are present in the HTTP Request stream.  

2. On successfully saving these blobs into the stores, what it has with itself now are one or more name value pairs i.e. a Map. The name is the request parameter name for your blob. For e.g. in our index.jsp, the file that we uploaded had the name of myTweetPic i.e. the form element. And the value will be the Blob Key that you can use to retrieve the Blob. It will put this map into the Request stream i.e. it will augment the Request stream with this map and then invoke your SUCCESS_PATH url. The SUCCESS_PATH url is the end point or in our case the Servlet.  

This Servlet can then extract out each of the Blobs that were saved by inspecting the Map and do further action. In our case, we will be posting a link to Twitter. This ervlet is what we shall see next.   

Upload Servlet

The Upload Servlet code is listed below.  

package com.gaejexperiments.twitter;  

import java.io.IOException;
import java.util.Map;
import java.util.logging.Level;
import java.util.logging.Logger;  

import javax.mail.Session;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;  

import twitter4j.Status;
import twitter4j.Twitter;  

import com.google.appengine.api.blobstore.BlobKey;
import com.google.appengine.api.blobstore.BlobstoreService;
import com.google.appengine.api.blobstore.BlobstoreServiceFactory;  

@SuppressWarnings("serial")
public class Upload extends HttpServlet {
 private final static Logger _logger = Logger.getLogger(Upload.class.getName());
 private BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
   

    public void doPost(HttpServletRequest req, HttpServletResponse res)
        throws ServletException, IOException {  

        Map<String, BlobKey> blobs = blobstoreService.getUploadedBlobs(req);
        BlobKey blobKey = blobs.get("myTweetPic");  

        if (blobKey == null) {
            res.sendRedirect("/");
        }
        else {
         String strStatus = "";
         try {
          String twitter_userid   = (String)req.getParameter("twitter_userid");
          String twitter_password = (String)req.getParameter("twitter_password");
          _logger.info("UserId:"+twitter_userid + " , Password:"+twitter_password);
          Twitter twitter = new Twitter(twitter_userid,twitter_password);
          String strTweet = "Check out my picture at : " + "<a href="http://localhost:8888/viewpic.jsp?blob-key=&quot;+blobKey.getKeyString">http://localhost:8888/viewpic.jsp?blob-key="+blobKey.getKeyString</a>(); 
             Status status = twitter.updateStatus(strTweet);
             strStatus = "Successfully updated the status to [" + status.getText() + "].";
             _logger.info(strStatus);
         }
         catch (Exception ex) {
          strStatus = "Could not update Twitter Status : " + ex.getMessage();
          _logger.log(Level.WARNING,strStatus);
         }
         finally {
             res.sendRedirect("/submitpic.jsp?blob-key="+blobKey.getKeyString() + "&status="+strStatus);
         }
         }
    }
}

 

Let us go through the main points in the Upload Servlet code:  

1. The Servlet implementation is the doPost() method. This method was invoked by the Blobstore Service after the blobs were successfully saved in the datastore.  

2. As mentioned, the Blobstore Service augments the Request stream with a Map of successfully saved Blobs. It also provides a helper method that we can use to extract out this Map instance from the request stream. The code is shown below:  

Map<String, BlobKey> blobs = blobstoreService.getUploadedBlobs(req);  

3. The next thing to do is to get the Blob Key for our saved Blob. The Blob Key is unique and will be used to retrieve the Blob as we shall see later on in the viewpic.jsp code. The Key that we will use is the same as the input parameter name for our file blob that we provided in the FORM i.e. index.jsp.   

BlobKey blobKey = blobs.get(“myTweetPic”);  

4. We do a simple check to verify if the blobKey instance is not null. If not, we create a status update for twitter giving a url that points to the viewpic.jsp page which is passed the blob-key request parameter as shown below:  

String strTweet = “Check out my picture at : ” + “http://localhost:8888/viewpic.jsp?blob-key=”+blobKey.getKeyString();   

5. Finally, we using Twitter4J to post out tweet using the twitter id and password that we provided.  

The status of our upload and tweet is then displayed by navigating to the submitpic.jsp page that simply shows the status and the picture that we uploaded.   

submitpic.jsp

This JSP file is invoked by our Upload Servlet to display the status of our Tweet to twitter and the image that we uploaded as discussed in the previous section. You will notice an interesting thing over here and that is the <img/> element. The source attribute of the <img/> element is a servlet residing at an endpoint /serve. All it needs is the blob-key request parameter. We shall see this servlet in a moment.  

<%@ page import="com.google.appengine.api.blobstore.BlobstoreServiceFactory" %>
<%@ page import="com.google.appengine.api.blobstore.BlobstoreService" %>  

<html>
    <head>
        <title>Tweet My Picture - Submission</title>
    </head>
    <body>
        <img src="tweet.png"/>
        <h1>Tweet My Picture</h1>
        <hr/>
        <h3>Submission Result: <%=request.getParameter("status")%></h3>
        <%
           String blobKey = (String)request.getParameter("blob-key");
           if (blobKey != null) {%>
                You uploaded : <br/>
       <img width="200" height="150" src="<%="<a href="http://localhost:8888/serve?blob-key=&quot;+blobKey">http://localhost:8888/serve?blob-key="+blobKey</a> %>">
           <%}%>
    </body>
</html>  

  

Serve Servlet

This Servlet uses another helper function of the BlobStoreService. The method is called serve and it takes two parameters as shown in the listing below. The blob Key and the HTTP response stream. Note that we pass it the Blob Key that we got after saving our blob. What the serve method will do is that it will automatically retrieve the Blob from the Datastore using the Blob Key that was provided to it. Once it retrieves the Blob successfully, it will take the content of the Blob, set the correct MIME types and insert that into the HTTP Response stream. You can the use the Response stream to assign it as a source for the HTML <img/> element. We did exactly that in the submitpic.jsp file and we do the same in the viewpic.jsp file that will be invoked when someone clicks on the Twitter post.  

  

package com.gaejexperiments.twitter;  

import java.io.IOException;  

import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;  

import com.google.appengine.api.blobstore.BlobKey;
import com.google.appengine.api.blobstore.BlobstoreService;
import com.google.appengine.api.blobstore.BlobstoreServiceFactory;  

@SuppressWarnings("serial")
public class Serve extends HttpServlet {
 private BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
 
 public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
        BlobKey blobKey = new BlobKey(req.getParameter("blob-key"));
        blobstoreService.serve(blobKey, resp);
    }
}

  

viewpic.jsp

As discussed earlier, the Tweet that we post has the following format:  

Check out my picture at http://localhost:8888/viewpic.jsp?blob-key=A_BLOB_KEY  

This will invoke the viewpic.jsp file with a request parameter named blob-key which contains the Blob key value. So all we have to do i to create a <img/> element, whose source attribute will be served via the /serve endpoint as shown below. Simple isn’t it ?  

  

<%@ page import="com.google.appengine.api.blobstore.BlobstoreServiceFactory" %>
<%@ page import="com.google.appengine.api.blobstore.BlobstoreService" %>  

<%
    BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
    String blobKey = (String)request.getParameter("blob-key");
%>
<html>
    <head>
        <title>Tweet My Picture - View</title>
    </head>
    <body>
    <img width="200" height="150" src="<%="<a href="http://localhost:8888/serve?blob-key=&quot;+blobKey">http://localhost:8888/serve?blob-key="+blobKey</a> %>">
    </body>
</html>

   

Configuring the Servlets

We need to add both the Upload Servlet and the Serve Servlet with their appropriate entries for  <servlet/> and <servlet-mapping/> in the web.xml file. This file is present in the WEB-INF folder of the project. The necessary fragment to be added to your web.xml file are shown below.  

 <servlet>
  <servlet-name>Upload</servlet-name>
  <servlet-class>com.gaejexperiments.twitter.Upload</servlet-class>
 </servlet>
 <servlet>
  <servlet-name>Serve</servlet-name>
  <servlet-class>com.gaejexperiments.twitter.Serve</servlet-class>
 </servlet>
 <servlet-mapping>
  <servlet-name>Upload</servlet-name>
  <url-pattern>/upload</url-pattern>
 </servlet-mapping>
 <servlet-mapping>
  <servlet-name>Serve</servlet-name>
  <url-pattern>/serve</url-pattern>
 </servlet-mapping>

  

Deploying the application

Since we are going to be running this application locally itself, we simply need to validate it against the local development server. Assuming that the development is complete, simply run the Web Application from your Eclipse IDE and using a browser, navigate to http://localhost:8888/index.jsp. In my case, the port is 8888, but it could be different for you.  

Please make sure that you are connected to the Internet and do have a valid Twitter userid and password.   

Refer to the section above on Tweet My Picture in action for the flow. I will not repeat it here again.  

An interesting thing that you could look at is to navigate to http://localhost:8888/_ah/admin console. Go to the DataStore viewer. If you have uploaded several images successfully, you will find instances of the Entity : __BlobInfo__ getting created. Click on the List Entities button to see the entities that you have uploaded so far. The ID/Name column is the Blob Key for your Blob. A sample screenshot from my development server instance is shown below:  

  

Conclusion

This brings the episode to an end. In this episode we saw the Blobstore API service that has been introduced in version 1.3.0 of the App Engine SDK. The Blobstore API service is experimental at this point in time and could change but it gives us enough today to see how to create application that require blobs of large sizes like video and images. This episode showed a simple implementation called Tweet My Picture but the possibilities are endless. Do keep in mind that if you have to use Blobstore Service in a live application hosted at the appspot.com domain, then you will need to enable “billing” for your application.  

Till the next episode, have fun with the Blobstore Service!

P.S: Just in case you were wondering… the picture of the lion was taken at a distance of 2 feet away from it in the National Park, Mumbai.

Episode 12 : Writing an Advanced Google Wave Robot using WadRobotFramework

December 3, 2009 5 comments

Welcome to Episode 12 of the series. This episode is an extension of the earlier episode where we saw how to write Simple Google Wave Robots using the WadRobotFramework. I strongly recommend that  you have completed the earlier episode and have got comfortable with the WadRobotFramework since this episode builds on the earlier one.

To recap, the WadRobotFramework distinguishes between 2 kinds of Robots and I summarize it again over here.

Simple Robots : These robots we covered in the earlier episode and saw how you can write a simple robot to react to a blip by appending a new blip (BlipAppenderRobot) or even modify the Blip Text (BlipModifierRobot).

Advanced Robots : These are of main focus in this article and I reproduce from the earlier episode the text so that you understand what Advanced Robots are first. The definition of Advanced Robots is per the WadRobotFramework and it is not meant to indicate this is the final definition of it.

Advanced Robots are those that can react to instructions ( or commands) in the Blips. Here are some samples of Advanced Robots and how they would react to commands from Blips:

1. A character Counting Advanced Robot:

Your submitted Blip Text contains : “Here is some text in the blip. Please count the length of this message for me.” {mycountingrobot:count}.

You can write an advanced Robot (mycountingrobot) that knows how to count the length of the message. So it gets notified when it there is a instruction (count) in the Blip. Your Advanced Robot can then count out the characters and then either append or modify a blip as needed.

2. A Tweeting Robot:

Your submitted Blip Text contains the following text : “{mytweetingrobot:tweet} Some text to tweet”

You can write an advanced Robot (mytweetingrobot) that knows how to tweet the message to Twitter. So it gets notified when it there is a instruction (tweet) in the Blip. Your Advanced Robot can then append a blip or modify a blip saying that the message has been tweeted.

The best part of it all is that you could combine all of this into a single Robot that can respond to one or more commands. For example, take a look at Today’s Special Robot (see http://ppandhi.wordpress.com/2009/11/08/todays-special-robot/) that can respond to more than one command. It can give you the quotes, day in history, word of the day, cricket score, your daily horoscope by simply responding to the command that you type in.

So for example, you could write a robot and give it commands like this:

1. {myrobot:doCommand1}

2. {myrobot:doCommand2}

3. {myrobot:doCommandN} and so on.

In this article, we are going to see exactly how to achieve the above command Robot that will delegate its work to different workers who are responsible for executing the command i.e. doing the work.

Let us get a few definitions in place first:

1. The Command Robot: This is the main class of your Robot and you need to extend the org.wadael.waverobotfrmwrk.advanced.WithWorkersRobot. You need to have an identifier for your robot, which is a unique ID for your Robot. Let us call it GAEJRobot.

2. Each Command Robot is capable of following instructions or commands. These instructions are executed by the Workers.

3. A Worker is a Command implementation that performs a certain logic. For e.g. fetching a stock quote, getting a word of a day, sending a Tweet, sending an email, etc. Each worker will indicate what instruction or command it obeys.

As an example, say you want to write an Advanced Robot class (WithWorkersRobot) whose identifier is named GAEJRobot that can responds to the following two commands:

a. SendTweet
b. GiveWordOfTheDay

So, you will implement two Workers and register (add) them to the GAEJRobot class. The two worker classes will be :

  • SendTweetWorker which says that listens to an instruction named tweet and it will implement its core logic in the doWork() method.
  • GiveWordOfTheDayWorker which says that it listens to an instruction named wotd and it will implement its core logic in the doWork() method.

Now, in your Wave Conversation, you can give the following text in the Blip (of course after adding the Robot as a participant).

1. {GAEJRobot:tweet}

2. {GAEJRobot:wotd}

Voila! The WadRobotFramework will then do the heavy lifting for you. It roughly works as follows:

  • When you submit a Blip, it will scan the Blip Text for the identifier and see if it matches itself.
  • If yes, it will scan out the instructions and invoke the doWork() method of the Worker Robot that implements the instruction.

This is not all. The WadRobotFramework has thought about parameters that you may need to pass to your Robot. For e.g. consider the following fictitious instruction that you need to give to a Stock Quote Robot.

{StockQuoteRobot:getquote GOOG} or {StockQuoteRobot:getquote GOOG,MSFT,ADBE,ORCL,IBM}

In short the format is as follows:

{RobotIdentifier:Instruction<space>[parameters]}

So in the above two examples the parameter GOOG and the parameter string “GOOG,MSFT,ADBE,ORCL,IBM” will be passed to the doWork() method of your RobotWorker that has registered with the Advanced Robot and who implements the getquote instruction. Please read this last statement clearly with the highlighted words as the key pieces required to build out an Advanced Robot.

Simple yet powerful and it opens up a wide range of Robots that you can start writing today. So let me get started and demonstrate the basic flow to get an Advanced Robot up and running which can accept 1 or more instructions. The Robot does not do anything specific except for simply demonstrating the flow. Readers are expected to extend it with their ideas brimming inside their minds.

To understand what we will build, it helps to take a look at the running robot as shown below:

You will notice that I have added my robot called MyAdvancedRobot. The identifier for the Robot is GAEJRobot and the Robot has two workers (Worker1 and Worker2)  registered with it, which implement the instructions  command1 and command2 respectively.

Now when I submit the text {GAEJRobot:command1} , the doWork() method of the Worker1 is invoked. It simply accepts the command and prints out that it received the command with no parameters passed to it.

Similarly, look at the wave conversation below:

Here I give the command2 to the GAEJRobot and I am also passing a string of parameters. When I click the Done button, the doWork() method of the Worker2 is invoked. It simply accepts the command and prints out that it received the command with the parameter string. Now, it could have processed the parameters and performed its logic accordingly. This demonstrates how the wiring is done by WadRobotFramework to make your life easier in writing Advanced Google Wave Robots.

Let us start writing code now. I will assume that you are fairly conversant now with the typical project structure and requirements of writing a Google Wave Robot. If not, I suggest that you first go through these tutorials in the following order:

  • Episode 7 : Writing your First Google Wave Robot
  • Episode 11: Develop Simple Google Wave Robots using the WadRobotFramework

Project Setup

Create a New Project or use the same project MyGoogleWaveRobot that we used in the earlier Episode 11.

If you are creating a New project, then please make sure that you download all the JARs from the following two locations:

http://code.google.com/p/wave-robot-java-client/downloads/list

The web page when you navigate to the above URL is shown below:

ep7-6

Download all the above files to your machine.

The WadRobotFramework JAR file is available at the following location :

http://code.google.com/p/wadrobotframework/downloads/list

The web page when you navigate to the above URL is shown below:

Download the JAR file to your machine. Once you have downloaded the 5 JAR files, make sure that they are copied to the \WEB-INF\lib folder of your project and that the Project Build Path is also setup with the JAR files as shown below:

Writing the Advanced Robot : MyAdvancedRobot.java

The first step is to create a Java class. Call it MyAdvancedRobot.java. The source code is listed below:

package com.gaejexperiments.waverobot;

import org.wadael.waverobotfrmwrk.advanced.WithWorkersRobot;

public class MyAdvancedRobot extends WithWorkersRobot {

 public MyAdvancedRobot() {
 super();
 //This will process 'command1'
 addRobotWorker(new Worker1());

 //This will process 'command2'
 addRobotWorker(new Worker2());
 }

 @Override
 public String getRobotIdentifier() {
 return "GAEJRobot";
 }

 @Override
 protected String getUsage() {
 return "Advanced Robot commands : command1 and command2";
 }

 @Override
 protected String getRobotSelfIntroduction() {
 return "I am an Advanced Robot";
 }

}

Let us dissect the code now:

1. Our Advanced Robot class MyAdvancedRobot extends the WithWorkersRobot class, which is required for creating the Advanced Robots with the WadRobotFramework.

2. The WithWorkersRobot class constructor uses a method called addRobotWorker(…) to add one or more workers to it. Remember each worker will execute or do its job as per the instruction that it obeys. So we are going to have two workers : Worker1 and Worker2 which we are adding to our AdvancedRobot. We will get to the listing of the Worker1 and Worker2 later but it is sufficient to keep in mind, that Worker1 class will perform the work required with command1 instruction is given in the Blip and Worker2 class will perform the work required when command2 instruction is given in the Blip. To recap, as you add more workers, you will need to add them here in the constructor using the addRobotWorker method.

3. The getRobotIdentifier() method is used to return the string that is the unique identifier of your Robot. This is used to distinguish Robots in the same Wave. The identifier if you recollect is going to be used by the other participants in the wave to give instructions to your Robot. As mentioned, the format in the Blip to give an instruction to your robot will be like this:

{RobotIdentifier:Instruction<space>[parameters]}

Each Instruction is implemented by a Worker. For e.g. command1 will be implemented by Worker1 and command2 will be implemented by Worker2.

So to invoke Worker1, we have to say {GAEJRobot:command1} in the Blip and submit it. Hence we return GAEJRobot in the getRobotIdentifier() method and this will be the unique way in which we identify this Robot

4. The getUsage() method is used to return a string that will be displayed to the participant when they type /help in the start of the Blip. This is useful to give a list of possible instructions that your Advanced Robot implements. In our case here, we are implementing two instructions and hence we have returned some string. But you can give an elaborate help string stating detailed command text, sample parameters, etc.

5. Finally, we have the getRobotSelfIntroduction() method. This is not mandatory but it is nice to announce to the particpants when you (Robot) gets added to the wave as a participant. Simply return a String that you would like to announce to the other (existing) participants in the Wave.

Implementing the Workers

We are now going to implement the Workers i.e. Worker1 and Worker2. The code is identical for both of them and it is listed below:

Worker1.java

package com.gaejexperiments.waverobot;

import org.wadael.waverobotfrmwrk.advanced.RobotWorker;

import com.google.wave.api.Blip;
import com.google.wave.api.Event;
import com.google.wave.api.RobotMessageBundle;

public class Worker1 implements RobotWorker {

 public String getInstruction() {

 return "command1";
 }

 public boolean doWork(RobotMessageBundle bundle, Blip blip, Event evt, String params) {
 blip.getDocument().append("Robot Worker 1 got the command with parameter string : " + params);
 return true;
 }

 public String getDescription() {
 return "Robot Worker 1";
 }
}

Worker2.java

package com.gaejexperiments.waverobot;

import org.wadael.waverobotfrmwrk.advanced.RobotWorker;

import com.google.wave.api.Blip;
import com.google.wave.api.Event;
import com.google.wave.api.RobotMessageBundle;

public class Worker2 implements RobotWorker {

 public boolean doWork(RobotMessageBundle bundle, Blip blip, Event evt, String params) {
 blip.getDocument().append("Robot Worker 2 got the command with parameter string : " + params);
 return true;
 }

 public String getDescription() {
 return "Robot Worker 2";
 }

 public String getInstruction() {

 return "command2";
 }

}

Let us go through the code of one of them and you will be able to understand it:

1. To recap, a Worker implements an instruction or a single command. Each Worker class needs to implement the RobotWorker interface in the WadRobotFramework.

2. It needs to implement the getInstruction() method which returns a String. This is the instruction that the Worker will obey or perform. In our case, the command1 is being done by Worker1 class and the command2 is being done by Worker2 class respectively.  So when someone submits {GAEJRobot:command1} in the Blip, the doWork() implementation of the Worker1 class will be invoked and if they submit {GAEJRobot:command2} in the Blip, the doWork() implementation of the Worker2 class will be invoked.

3. It needs to implement the doWork() method. This method is the heart or the main implementation of the Worker. Here you will place all your processing logic. Notice that since this is an Advanced Robot, it is assumed that you would even like to make use of the Google Wave API classes directly. So you are passed in instances of RobotMessageBundle, Blip and Event classes. The last parameter passed is params and it represents any parameters passed to the robot.

You will notice in the implementation that we have done for the Worker, that we simply Append to the Blip Text saying that the Worker got its command and notice that we also print out the Parameter String. So if you logic depends on the values of the parameters passed, you can parse out the parameters here itself and perform your logic.

That is all we need to do as far as writing Java code is concerned. Of course we have the other mandatory files that we need to create, which we will cover quickly now:

Configuring the MyAppenderRobot in web.xml

We need to add the MyAdvancedRobot in the <servlet/> and <servlet-mapping/> entry to the web.xml file. This file is present in the WEB-INF folder of the project. The necessary fragment to be added to your web.xml file are shown below.

<servlet>
<servlet-name>MyAdvancedRobot</servlet-name>
<servlet-class>com.gaejexperiments.waverobot.MyAdvancedRobot</servlet-class>
</servlet>
<servlet>
<servlet-name>MyRobotProfileServlet</servlet-name>
<servlet-class>com.gaejexperiments.waverobot.MyRobotProfileServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>MyAdvancedRobot</servlet-name>
<url-pattern>/_wave/robot/jsonrpc</url-pattern>
</servlet-mapping>
<servlet-mapping>
<servlet-name>MyRobotProfileServlet</servlet-name>
<url-pattern>/_wave/robot/profile</url-pattern>
</servlet-mapping>

Notice that we also have the ProfileServlet configured here, which is a good and recommended thing to have for your Robot. The ProfileServlet class is covered in the next section.

ProfileServlet.java

The implementation is straightforward and contains the ApplicationId that I have used for my AdvancedRobot. You can replace it with your id.

package com.gaejexperiments.waverobot;

import com.google.wave.api.ProfileServlet;

public class MyRobotProfileServlet extends ProfileServlet {

@Override
public String getRobotAvatarUrl() {
return "http://myinfoagent.appspot.com/_wave/myimage.jpg";
}

@Override
public String getRobotName() {
return "MyAdvancedRobot";
}

@Override
public String getRobotProfilePageUrl() {
return "http://myinfoagent.appspot.com";
}

}

Creating the MyAdvancedRobot capabilities.xml files

We need an additional file to describe the capabilities of the Robot that we have written. This file is called the capabilities.xml and it needs to reside in a certain location. You need to create a _wave directory inside of the war directory of your project. The location of this file is  war/_wave directory.

You will need to create the _wave directory and create the capabilities.xml file over there. The capabilities file shown below is pretty straightforward and is shown below:

<?xml version="1.0" encoding="utf-8"?>
<w:robot xmlns:w="http://wave.google.com/extensions/robots/1.0">
 <w:capabilities>
 <w:capability name="BLIP_SUBMITTED" content="true" />
 </w:capabilities>
 <w:version>1</w:version>
</w:robot>

Deploying the Application

To deploy the application, you will need to first create your Application ID. The Application Identifier can be created by logging in at http://appengine.google.com with your Google Account. You will see a list of application identifiers already registered under your account (or none if you are just getting started). To create a new Application, click on the Create Application button and provide the Application Identifier as requested. Please note this name down since you will be using it for deployment.

For e.g. I have registered an application identifier named myinfoagent.

To deploy the application, follow these steps (they should be familiar to you now):

  1. Click on the Deploy Icon in the Toolbar.
  2. In the Deploy dialog, provide your Email and Password. Do not click on Deploy button yet.
  3. Click on the App Engine Project settings link. This will lead you to a dialog, where you need to enter your Application ID [For e.g. my Application Identifier myinfoagent]
  4. Click on OK. You will be lead back to the previous screen, where you can click on the Deploy button. This will start deploying your application to the GAEJ cloud. You should see several messages in the Console window as the application is being deployed.
  5. Finally, you should see the message “Deployment completed successfully”.

MyAdvancedRobot in Action

Your application is going to be available at the http://yourapplicationid.appspot.com. In my case, the application is available at http://myinfoagent.appspot.com.

You can test for the presence of your robot capabilities file by simply typing in the following:

http://yourapplicationid.appspot.com/_wave/capabilities.xml [Replace yourapplicationid with the Application ID that you have] and you should see the capabilities.xml file being served up.

To test out the Robot, you need to launch the Google Wave client and login in with your account by going to http://wave.google.com. On successful login, you will be inside the Wave client from where you can create a new wave by clicking on the New Wave link. I then add the myinfoagent@appspot.com, which is our AdvancedRobot to the Wave as a participant as shown below:

On addition, the AdvancedRobot will announce itself to the Participants. This is the method getRobotSelfIntroduction() that we implemented in the MyAdvancedRobot.java class. The output is shown below:

Now, we type in the message /help in the beginning of the Blip and submit it as shown below:

When submitted, the method getUsage() of the MyAdvancedRobot class is invoked and it displays the commands and any help instruction that you provided in there.

The next thing we do is give a command to the first Worker as shown below and click on the Done button.

This will invoke the doWork() method on the Worker1 class since this class has indicated that it implements command1 as mentioned in the getInstruction() method of the class. The response of the Worker is shown below:

As you can see we did not pass any parameters to it, so it printed out null.

Now, let us invoke the second Worker as shown below and notice that we are passing parameters to it now:

When we click on Done, the doWork() method on the Worker2 class since this class has indicated that it implements command2 as mentioned in the getInstruction() method of the class. The response of the Worker is shown below:

Notice that the parameter string was now passed and successfully displayed. You can easily parse out the String and perform your logic.

Conclusion

This concludes the 2-part tutorial on creating Simple and Advanced Robots using the WadRobotFramework. The framework takes away much of the API that you need to know to get going with the Wave API and instead lets you focus on the core of our Robot logic. The framework is still is its early stages and would do with some feedback from you.

Till the next episode, just Smile and Wave!

Categories: Uncategorized

Episode 11: Develop Simple Google Wave Robots using the WadRobotFramework

December 3, 2009 11 comments

Welcome to Episode 11 of this series. We saw in earlier Episode # 7, how you can write a Google Wave Robot. Please go through that tutorial to get familiar with the basics. This episode will be focused on writing a Google Wave Robot using a framework called the WadRobotFramework. This framework aims to help you develop your Robot quickly so that you can focus on the logic of your Robot, while letting the framework do the heavy lifting. Sure, you can do your programming with the Google Wave Robot API as explained in Episode 7, but this framework as we shall see wraps the details nicely so that you get to focus on your Robot logic.

What is the WadRobotFramework?

The WadRobotFramework is available at http://code.google.com/p/wadrobotframework. This framework is the work of  Jerome BATON, who has stated that the goal of the framework is to make writing Google Wave Robots in Java much simpler.

The framework goes ahead and defines two kinds of Robots: Simple and Advanced.

Simple Robots

Simple Robots are typically those that can react to a new Blip. This kind of Robots simply wait till a Blip is submitted. And on submission of the Blip, they will append a message to the Blip. It could be as simple as just analyzing the text of the blip and appending some new information or simply a fun sort of a Robot, which simply prints a random message.

An example of a this is shown below and which we will build in this episode:

In the above wave, there are two participants (myself and the Robot). The Robot announces itself when it is added to the Wave. Additionally, whenever I submit a Blip (for e.g. Hello), it simply appends a new blip with some text.

Simple Robots are also those that can modify the Blip Text instead of appending a new blip. For e.g. They could read the blip text and filter out the profanities, do automated spell checks or simply add some information to the existing Blip Text itself. In other words, it modifies the Blip Text.

Shown below is an example of a Simple Robot that modifies the Blip Text:

In the above wave, there are two participants (myself and the Robot). The Robot announces itself when it is added to the Wave. Additionally, whenever I submit a Blip (for e.g. Hello), it simply modifies the Blip Text by mentioning when I submitted the Blip text.

There is also a 3rd thing that the Simple Robots could do, which is very useful too. They can act in a stealth manner and simply do their work quietly. For e.g. whether you wish to write a Robot that appends a Blip or modifies a submitted Blip, you can simply chose not to append or modify but instead get the Blip text and do your work quietly. For e.g. count the words in the background, etc.

The WadRobotFramework also supports Advanced Robots, which are particularly interesting and will be covered in the Next Episode.

Advanced Robots are those that can react to commands in the Blips. Here are some samples of Advanced Robots and how they would react to commands from Blips:

1. A character Counting Advanced Robot:

Your submitted Blip Text contains : “Here is some text in the blip. Please count the length of this message for me. {mycountingrobot:count}”.

You can write an advanced Robot (mycountingrobot) that knows how to count the length of the message. So it gets notified when it there is a command (count) in the Blip. Your Advanced Robot can then count out the characters and then either append or modify a blip as needed.

2. A Tweeting Robot:

Your submitted Blip Text contains the following text : “{mytweetingrobot:tweet}

You can write an advanced Robot (mytweetingrobot) that knows how to tweet the message to Twitter. So it gets notified when it there is a command (tweet) in the Blip. Your Advanced Robot can then append a blip or modify a blip saying that the message has been tweeted.

The best part of it all is that you could combine all of this into a single Robot that can respond to one or more commands. For example, take a look at Today’s Special Robot (see http://ppandhi.wordpress.com/2009/11/08/todays-special-robot/) that can respond to more than one command. It can give you the quotes, day in history, word of the day, cricket score, your daily horoscope by simply responding to the command that you type in. With the WadRobotFramework, we shall see in the next episode how you can write such an Advanced Robot that responds to different commands. If you cannot wait, take a look at the documentation at the site (Advanced Robots).

Since this episode will be focusing on Simple Robots, let us recap what we know so far:  The WadRobotFramework supports Simple Robots in two flavours: The BlipAppenderRobot and the BlipModifierRobot. The BlipAppenderRobot is used to simply append to a Blip. The BlipModifierRobot is used to modify a submitted Blip.

Simple, isnt it? Lets develop and deploy our Simple Robots using the WadRobotFramework.

Create a New Project

We need to create a New Project first. Follow the steps below:

1. Either click on File –> New –> Other or press Ctrl-N to create a new project. Select Google and then Web Application project. Alternately you could also click on the New Web Application Project Toolbar icon as part of the Google Eclipse plugin.

2. In the New Web Application Project dialog, deselect the Use Google Web Toolkit and give a name to your project. I have named mine MyGoogleWaveRobot and I suggest you go with the same name so that things are consistent with the rest of the article (but if you wish to name it something else, that is fine too). The Screenshot is shown below:

3. Click on Finish. This will generate the project and also create a sample Hello World Servlet for you. But we will be writing our own Servlet. So I suggest that you can delete the Servlet Java class and the mappings made in the web.xml or you can leave it for now since we are going to write our own.

Adding Google Wave Robot JAR files and WadRobotFramework JAR file to your Project Path

Since we are going to be writing a Wave Robot, we need some additional files on the client side. These additional files (JAR files) are required for the additional Wave API’s and also for deployment in your WEB-INF\lib folder, so that they are correctly deployed and available to the run-time engine. These JAR files do not ship along with the Google Eclipse plugin, so you will need to download them for a website. The Google code website for the JAR files is:

http://code.google.com/p/wave-robot-java-client/downloads/list

The web page when you navigate to the above URL is shown below:

ep7-6

Download all the above files to your machine.

The WadRobotFramework JAR file is available at the following location :

http://code.google.com/p/wadrobotframework/downloads/list

The web page when you navigate to the above URL is shown below:

Once you have downloaded the files, follow these steps to setup your Project Build Path and runtime correctly.

1. Copy all the 5 JAR files to the WEB-INF\lib folder of your Eclipse Project. After copying you should see the files as shown in the project hierarchy below:

2. Right-click on the Project in the Project Hierarchy. Select Properties and then Java Build Path. Click on Add JARs and then select the 5 JAR files from your Project WEB-INF\lib folder.

3. Your Project Build Path should like the screenshot below.

Click on OK to proceed. This completes your Build Path setup with the Google Wave Robot JAR and WadRobotFramework JAR files.

Writing the Simple Robot: MyAppenderRobot : MyAppenderRobot.java

Let us first create our Simple Robot Java class based on the WadRobotFramework. This Simple Robot is the one that will react to a new Blip. So all we will do is to make the robot append to the blip, when the blip is submitted. The Robot demonstrated here will be straightforward and you can easily modify it to make it react in your own way.

As discussed, this is known as the BlipAppenderRobot and so all we need to do is to extend the BlipAppenderRobot class in the WadRobotFramework and provide our simple implementation.

The steps are straightforward and given below. All we need to do is write our class that extends the org.wadael.waverobotfrmwrk.simple.BlipAppenderRobot class and provide an implementation for the getTextToAppend method.

Follow these steps:

1. Create a new Java class within the same package. The New Java Class dialog is shown below. I have named the class MyAppenderRobot as shown below. Click on the Browse button to select a Super Class.

2. In the Superclass Selection dialog shown below, type the word BlipAppenderRobot (some part of it is also OK as the screenshot shows below) in the Choose a type field as shown. This will bring up the correct Matching items i.e. org.wadael.waverobotfrmwrk.simple.BlipAppenderRobot. Click on OK.


This will generate the code and you simply replace it with the following code listing given below:

package com.gaejexperiments.waverobot;

import org.wadael.waverobotfrmwrk.simple.BlipAppenderRobot;

public class MyAppenderRobot extends BlipAppenderRobot {

@Override
protected String getTextToAppend(String msg) {
return "I am appending some text too";
}

@Override
protected String getRobotSelfIntroduction() {
return "I am the Appender Robot";
}

}

Let us go through the code now:

1. We have extended the BlipAppenderRobot since all we want to do in this Robot is to react to the blip and append our own Blip

2. All we need to do as part of extending the BlipAppenderRobot class is to implement the getTextToAppend method. This method gets passed one parameter msg that is the blip text that was submitted. So in all essence, you could inspect what the text was in the blip submitted and then react to it.  The implementation simply returns the string that it wants to append. The WadRobotFramework’s BlipAppenderRobot will take care of creating a Blip and appending it to the current blip.

3. We also implement an optional method named getRobotSelfIntroduction. This method returns a string and  you can return something that identifies the addition of your Robot to the Wave. So when your Robot is added as a participant and if you have implemented the getRobotSelfIntroduction method, then it will display this message out. It is sort of announcing to the other wave participants about your presence.

That is all there is to implemented the MyAppenderRobot. If you would have studied the earlier tutorial on writing a Google Wave Robot, you would have noticed that the WadRobotFramework has done away with all the Event processing that you had to handle yourself and also shielded you from methods that you need to know to create a blip and append to it.

Configuring the MyAppenderRobot in web.xml

We need to add the MyAppenderRobot  in the <servlet/> and <servlet-mapping/> entry to the web.xml file. This file is present in the WEB-INF folder of the project. The necessary fragment to be added to your web.xml file are shown below.

 <servlet>
 <servlet-name>MyAppenderRobot</servlet-name>
 <servlet-class>com.gaejexperiments.waverobot.MyAppenderRobot</servlet-class>
 </servlet>
 <servlet-mapping>
 <servlet-name>MyAppenderRobot</servlet-name>
 <url-pattern>/_wave/robot/jsonrpc</url-pattern>
 </servlet-mapping>

In the above fragment, you will note that url-pattern /_wave/robot/jsonrpc has to be mapped to the Robot Servlet that you have written. This is because the Google Wave system will invoke this url to communicate with your Robot using its protocol.

Creating the MyAppenderRobot capabilities.xml files

We need an additional file to describe the capabilities of the Robot that we have written. This file is called the capabilities.xml and it needs to reside in a certain location. You need to create a _wave directory inside of the war directory of your project. The location of this file is  war/_wave directory.

You will need to create the _wave directory and create the capabilities.xml file over there. The capabilities file shown below is pretty straightforward and is shown below:

<?xml version="1.0" encoding="utf-8"?>
<w:robot xmlns:w="http://wave.google.com/extensions/robots/1.0">
  <w:capabilities>
    <w:capability name="BLIP_SUBMITTED" content="true" />
  </w:capabilities>
  <w:version>1</w:version>
</w:robot>

There are two elements of interest over here.

The <capability> element is of particular interest and we have registered for the BLIP_SUBMITTED event. This is important because it informs Google Wave that whenever a BLIP is submitted by a participated, then our Robot needs to be notified of it. The WadRobotFramework will notify the Robot by invoking the getTextToAppend method.

The other element is the <version> element. If you change any capabilities in your robot, then it is recommended that before you deploy, you change the version value over here, so that Google Wave can detect that there is a newer version and hence it can then query for your modified capabilities if any.

Writing our MyAppenderRobot Profile Servlet (not required but nice)

This is not a required step but it would be good practice to do so to make your Robot look more professional. A Profile Servlet is used to tell the following about your Robot:

1. A Name for your Robot

2. A custom image for  your Robot

3. A profile page for your Robot (a URL)

If you provide these, then the Google Wave client is able to retrieve them and set it for your Robot when it is added as a participant. This makes the Robot look more professional.

This profile information needs to be provided by you by writing a Profile Servlet. The Profile Servlet is nothing but extending the com.google.wave.api.ProfileServlet class and providing simple implementations for the overwritten methods.

Follow these steps to write the Profile Servlet:

1. Create a new Java class within the same package. Name the classs MyRobotProfileServlet and mention the Superclass as com.google.wave.api.ProfileServlet class. Click on OK.

2. This will generate a MyRobotProfileServlet.java file.

The simplest way to generate the stubs for the required methods would be to go to Source –> Override/Implement Methods. This will bring up the dialog box as shown below and you only need to select the 3 methods to override as shown:

ep7-14

Click on OK. This will generate the stubs, which you can then overwrite with the code shown below. The code is easy to understand, all we are doing is providing values for the Name, Avatar(Image) and the Profile Page URL. Note that for the Avatar, we are providing a file myimage.jpg present in the WAR/_wave folder. You will need to copy an appropriate image file for yourself and make sure that it is physically copied to the folder locally in your Eclipse project before you deploy your application.

I have used my AppId as myinfoagent but you can replace it with your AppId.

package com.gaejexperiments.waverobot;

import com.google.wave.api.ProfileServlet;

public class MyRobotProfileServlet extends ProfileServlet {

@Override
public String getRobotAvatarUrl() {
return "http://myinfoagent.appspot.com/_wave/myimage.jpg";
}

@Override
public String getRobotName() {
return "MyRobot";
}

@Override
public String getRobotProfilePageUrl() {
return "http://myinfoagent.appspot.com";
}

}

Configuring the Profile Servlet

We need to add the Profile Servlet <servlet/> and <servlet-mapping/> entry to the web.xml file. This file is present in the WEB-INF folder of the project. The necessary fragment to be added to your web.xml file are shown below.


 <servlet>
 <servlet-name>MyRobotProfileServlet</servlet-name>
 <servlet-class>com.gaejexperiments.waverobot.MyRobotProfileServlet</servlet-class>
 </servlet>
 <servlet-mapping>
 <servlet-name>MyRobotProfileServlet</servlet-name>
 <url-pattern>/_wave/robot/profile</url-pattern>
 </servlet-mapping>


In the above fragment, you will note that url-pattern /_wave/robot/profile has to be mapped to the Profile Servlet that you have written. This is because the Google Wave system will invoke this url to get hold of your profile.

Deploying the Application

To deploy the application, you will need to first create your Application ID. The Application Identifier can be created by logging in at http://appengine.google.com with your Google Account. You will see a list of application identifiers already registered under your account (or none if you are just getting started). To create a new Application, click on the Create Application button and provide the Application Identifier as requested. Please note this name down since you will be using it for deployment.

For e.g. I have registered an application identifier named myinfoagent.

To deploy the application, follow these steps (they should be familiar to you now):

  1. Click on the Deploy Icon in the Toolbar.
  2. In the Deploy dialog, provide your Email and Password. Do not click on Deploy button yet.
  3. Click on the App Engine Project settings link. This will lead you to a dialog, where you need to enter your Application ID [For e.g. my Application Identifier myinfoagent]
  4. Click on OK. You will be lead back to the previous screen, where you can click on the Deploy button. This will start deploying your application to the GAEJ cloud. You should see several messages in the Console window as the application is being deployed.
  5. Finally, you should see the message “Deployment completed successfully”.

MyAppenderRobot in Action

Your application is going to be available at the http://yourapplicationid.appspot.com. In my case, the application is available at http://myinfoagent.appspot.com.

You can test for the presence of your robot capabilities file by simply typing in the following:

http://yourapplicationid.appspot.com/_wave/capabilities.xml [Replace yourapplicationid with the Application ID that you have] and you should see the capabilities.xml file being served up.

To test out the Robot, you need to launch the Google Wave client and login in with your account by going to http://wave.google.com. On successful login, you will be inside the Wave client from where you can create a new wave by clicking on the New Wave link. When you do that, currently you are the only participant (myself) as shown in the screen below:

Click on the + sign next to your icon and you can add one or more participants as shown below:

NOTE : Your Google Wave Robot is going to be available at <YOURAPPLICATIONID>@appspot.com , hence I have added myinfoagent@appspot.com as that was my application id. But you can replace it with your application id.

If all goes well, you will see your Robot added as a participant (with the icon and all, since the Profile Servlet is invoked behind the scenes by the Google Wave system). It announces itself and this was done via the implementation that we provided in the getRobotSelfIntroduction method.

Now I type a message “Hello” as shown below and click on the Done button, the BLIP_SUBMITTED event is fired and our Robot gets the event i.e. the getTextToAppend method is invoked on our Robot Java class. The method simply appends a blip as shown below:

Writing another Simple Robot: MyBlipModifierRobot : MyBlipModifierRobot.java

We saw how easy it was to create our Simple Robot that simply appended a blip to the wave conversation when a blip was submitted. Let us now write another Simple Robot that will modify the Blip Text when the blip is submitted.

As discussed, this is known as the BlipModifierRobot and so all we need to do is to extend the BlipModifierRobot class in the WadRobotFramework and provide our simple implementation.

The steps are straightforward and given below. All we need to do is write our class that extends the org.wadael.waverobotfrmwrk.simple.BlipModifierRobot class and provide an implementation for the modifyBlipText method.

Follow these steps:

1. Create a new Java class within the same package. Name it  MyBlipModifierAppenderRobot as shown below and mention the superclass as org.wadael.waverobotfrmwrk.simple.BlipModifierRobot

This will create the MyBlipModifierRobot.java file and you can replace it with the following source:

package com.gaejexperiments.waverobot;

import java.util.Date;
import org.wadael.waverobotfrmwrk.simple.BlipModifierRobot;

public class MyBlipModifierRobot extends BlipModifierRobot {

@Override
protected String modifyBlipText(String originalBlipText) {
return originalBlipText + " [You typed this at :" + new Date().toString() + "]";
}

@Override
protected String getRobotSelfIntroduction() {
return "I am the Blip Modifier Robot";
}

}

Let us go through the code now:

1. We have extended the BlipModifierRobot since all we want to do in this Robot is to modify the Blip that was submitted.

2. All we need to do as part of extending the BlipModifierRobot class is to implement the modifyBlipText method. This method gets passed one parameter originalBlipText that is the blip text that was submitted. So in all essence, you could inspect what the text was in the blip submitted and then modify it as needed .  The implementation above appends the text “You typed this at [DateTimeStamp]” to the originalBlipText and simply returns that string. The WadRobotFramework’s BlipModifierRobot will take care of modifying the Blip Text.

3. We also implement an optional method named getRobotSelfIntroduction. This method returns a string and  you can return something that identifies the addition of your Robot to the Wave. So when your Robot is added as a participant and if you have implemented the getRobotSelfIntroduction method, then it will display this message out. It is sort of announcing to the other wave participants about your presence.

That is all there is to implemented the MyBlipModifierRobot. If you would have studied the earlier tutorial on writing a Google Wave Robot, you would have noticed that the WadRobotFramework has done away with all the Event processing that you had to handle yourself and also shielded you from methods that you need to know to create a blip and append to it.

Configuring the MyBlipModifierRobot in web.xml

We need to add the MyBlipModifierRobot  in the <servlet/> and <servlet-mapping/> entry to the web.xml file. This file is present in the WEB-INF folder of the project. The necessary fragment to be added to your web.xml file are shown below.

<servlet>
 <servlet-name>MyBlipModifierRobot</servlet-name>
 <servlet-class>com.gaejexperiments.waverobot.MyBlipModifierRobot</servlet-class>
 </servlet>
 <servlet-mapping>
 <servlet-name>MyBlipModifierRobot</servlet-name>
 <url-pattern>/_wave/robot/jsonrpc</url-pattern>
 </servlet-mapping>

If you wish to check out this Robot in action, you can do so with the same project. Simply replace the previous Robot’s (MyAppenderRobot) servlet entries with the one shown above. And deploy the application following the same steps in the Deploying the Application section. Leave the Profile Servlet and the capabilities.xml file as is because the same applies in this case too.

Once you deploy the MyBlipModifierRobot, you should see it in action as shown below. The first screen shows the MyBlipModifierRobot announcing itself and then the second screen shows how it modified the blip text that I submitted by appending the TimeStamp to it.

Conclusion

This concludes Episode 11 of the series. We saw how easy it was to write Simple Google Wave Robots using the WadRobotFramework. Several Google Wave Robots that are out there simply do one of the things that we have seen in this episode i.e. modify a blip’s contents or append a blip in reaction to a blip submitted. You can get going with such Robots by using this framework which hides you from the underlying Google Wave API.

In the next episode, we shall see how to write an Advanced Google Wave Robot that can implement several commands. Till then, unleash those Simple Robots!

In the Superclass Selection dialog shown below, type the word AbstractRobot (some part of it is also OK as the screenshot shows below) in the Choose a type field as shown. This will bring up the correct Matching items i.e. com.google.wave.api.AbstractRobotServlet. Click on OKep7-8

Categories: Uncategorized Tags: ,

Episode 10: Using the Task Queue Service

November 24, 2009 18 comments

Welcome to Episode 10. In this episode, we shall cover the experimental Task Queue Service in Google App Engine. This is an experimental service, which means that it could undergo change in its core functionality from all respects like methods, package names, etc. In any case, we can safely expect that it shall get confirmed in some form or the other in a future release of the Google App Engine.

This episode is sort of a natural progression on our last episode, which covered the Cron Service. To reiterate, the Cron Service was used for background jobs that you wish to perform outside of a user request. They are not typically initiated by a user but instead configured by your application to perform certain periodic tasks like summary reports at end of the day, daily backups, etc. The Task Queue Service provides a facility to process tasks in the background which are :

  • Typically initiated by the user.
  • They can be created internally by the application also to break down a background job into smaller tasks.

Some of the examples of performing a Task in the background include:

1. A user signs up at the website and submits his/her information. On receiving the request, a simple message is displayed back to the user that his/her subscription request has been received and they will hear soon about the status. At the same time, a task can be created that can process this users request, perform some verification and then send out an email to the user if the subscription is setup successfully. So you could have done either of the following:

  • Receive the user details and create a Task. This single task will take care of creating the required records in the database and send out an email.
  • Receive the user details and create a Task. This task will simply create a record in the database and then create another task to send out an email.
  • And so on.

2. An online shopping site could accept the order from a buyer. A task is then created in the system that will process the order. As part of the order processing, once shipping is done, another task is told to update the shipping status. Similarly when the logistics provider gives updates on the shipment locations, a task could be launched to update the shipment route of the package.

To summarize, any background (asynchronous) task can be launched when a user initiates the event or when an application or business event occurs. All these tasks are put into one or more user defined or default queues and are executed by the Google App Engine infrastructure on behalf of your application.

What does a Task constitute? What is a Queue ? Who executes it ?

The official documentation of Tasks is excellent and I suggest reading that in detail. In this episode I will cover just about enough for you to get started on Tasks and then dig deeper into them depending on your needs. So let us first understand what is the basic information that I need let the Google App Engine know about my task. I will take the liberty here to describe all the key concepts via an example that we will build in this application.

We wish to implement the following flow in our application:

1. A user wishes to sign up for your newsletter by providing an email id in a web form provided at our site.
2. The user enters the email id and clicks on sign up (a button).
3. The request is sent to a Servlet that accepts the email id and creates a Task. The response is sent back to the user thanking them for their interest.
4. The Task is then executed independently by the Google App Engine infrastructure and our code inside the Task i.e. checking if the email id is used, etc is verified and then an email is sent.

So, as you can see we have decoupled the background task (in step 4) from the sign up process (step 1, step 2 & step3).

It should now be straightforward to define the key elements:

1. Task : This is the unit of work that we wish to perform. In fact, the actor that will be performing this task is Google App Engine. So when we create a Task, we need to provide a standard way for the Google App Engine to invoke tasks and pass them their payload. It should now be straightforward to see that when we create a task, all we need to tell GAEJ is the URL (where to invoke the task) and the parameterized payload (data). This can be done in a standard fashion that you know. The URL is nothing but the servlet URL that will invoke the servlet that implements the Task. And the parameterized data is nothing but request parameters passed into your servlet so that it can execute accordingly. The data in this case will be nothing but the email id of the user who wants to sign up for your newsletter.

2. Queue : All Tasks when they are created are placed in a queue. They are then executed by the Google App Engine. To help us manage and categorize our tasks, you can define your queues by giving them appropriate names. For e.g. myqueue, emailqueue, etc. What this helps you to do is to place your tasks in the appropriate queue. Few points to note about queues (refer to the documentation for finer details):

  • All queues in your application are defined in a file named queue.xml that is present in the WEB-INF folder of your application.
  • Each queue has a unique name and you can control the rate at which tasks are executed in this queue by the Google App Engine. If you do not specify a rate, then the default rate is 5 tasks/second.
  • There is a default queue for your application named ‘default’ and if you can chose to add your tasks to the default queue. Alternately, you can add them to your application defined queue.

I believe this should be sufficient for us to begin developing a simple flow that will demonstrate how to define a task queue, create a task and then see it getting executed by the Google App Engine.

Task Queue in Action

The diagram below shows what we will implement in the next few sections. The use case is that of an user that wishes to subscribe to our newsletter and how we shall break up the process into the request processing and then the task processing.

Let us break down the above flow into the steps given below and what we shall be developing at each step:

1. In Step 1, the user visits a web page and enters his/her email id to sign up for the newsletter. We shall not be developing a web page over here to keep things simple. Instead we shall be testing it out by directly invoking a servlet that accepts the email id of the user. This is the same thing that you would have normally done  by hooking up the action of the HTML form to that of this servlet.

2. In Step 2, we will be looking at a Servlet whose existence is to do some verification and then create a Task for background processing. This servlet (GAEJCreateTaskServlet) will create a Task in a queue called subscription-queue. As we covered earlier, to create a Task, we need to provide two pieces of information to the Google App Engine so that it can execute it. They are the URL and the Data. The URL (/gaejsignupsubscriber) is going to be that of a Servlet (GAEJSignupSubscriberServlet) that shall be doing the core task. And the data will be what the servlet needs to complete the task. In our case, the data is the emailid request parameter.

3. In Step 3, Google App Engine automatically scans the queues for any tasks that are queued up and picks them up for execution. In our case, it will find a Task instance and execute it by invoking the URL (/gaejsignupsubscriber) and passing it the relevant data i.e. emailid

4. Finally in Step 4, our Servlet (GAEJSignupSubscriberServlet) is invoked and it will complete its task. To keep things simple in our example, it will currently only print out a message. But it should be obvious that core logic associated with the task would have gone into the Servlet here. For our specific case, it would have involved checking if the user has not signed up already, creating a database record and then sending off a welcome email.

Implementing the above flow

To summarize the above steps in terms of what we have to code, here is the list:

1. Code the GAEJCreateTaskServlet that will accept the request parameter and create a Task instance in the subscription-queue.

2. Code the GAEJSignupSubscriberServlet that will be invoked by Google App Engine automatically. We will currently only print out a log statement because the intent is to demonstrate the whole sequence.

3. Configure our queue (subscription-queue) in a file named queue.xml. This file needs to be placed in the WEB-INF folder of your application.

4. Configure our GAEJCreateTaskServlet and GAEJSignupSubscriberServlet in the web.xml file.

Finally, we can execute our application and use the local development server to see the application in action. Users can optionally even deploy it to the Google App Engine cloud if they wish.

So let us get started.

GAEJCreateTaskServlet.java

This servlet accepts our request for subscription. We shall invoke it via the following url : http://appurl/gaejcreatetask?emailid=XYZ.

package com.gaejexperiments.taskqueue;

import java.io.IOException;

import javax.servlet.ServletException;
import javax.servlet.http.*;

import com.google.appengine.api.labs.taskqueue.Queue;
import com.google.appengine.api.labs.taskqueue.QueueFactory;
import com.google.appengine.api.labs.taskqueue.TaskOptions;

@SuppressWarnings("serial")
public class GAEJCreateTaskServlet extends HttpServlet {
 public void doGet(HttpServletRequest req, HttpServletResponse resp)
 throws IOException {

 String strCallResult = "";
 resp.setContentType("text/plain");
 try {
 //Extract out the To, Subject and Body of the Email to be sent
 String strEmailId = req.getParameter("emailid");

 //Do validations here. Only basic ones i.e. cannot be null/empty

 if (strEmailId == null) throw new Exception("Email Id field cannot be empty.");

 //Trim the stuff
 strEmailId = strEmailId.trim();
 if (strEmailId.length() == 0) throw new Exception("Email Id field cannot be empty.");
 //Queue queue = QueueFactory.getDefaultQueue();
 Queue queue = QueueFactory.getQueue("subscription-queue");
 queue.add(TaskOptions.Builder.url("/gaejsignupsubscriber").param("emailid",strEmailId));
 strCallResult = "Successfully created a Task in the Queue";
 resp.getWriter().println(strCallResult);
 }
 catch (Exception ex) {
 strCallResult = "Fail: " + ex.getMessage();
 resp.getWriter().println(strCallResult);
 }
 }

 @Override
 public void doPost(HttpServletRequest req, HttpServletResponse resp)
 throws ServletException, IOException {
 doGet(req, resp);
 }
}

The code listed below is straightforward to understand. It does the following:

1. Extracts out the request parameter (emailid) and does some basic validation on it.

2. It gets a handle to the subscription-queue through the following statement:

Queue queue = QueueFactory.getQueue("subscription-queue");

3. It adds a Task to the above queue by providing a Task URL (/gaejsignupsubscriber) and Data (emailid parameter). It uses a helper class TaskOptions.Builder to help create the instance of the task. As you can see it provides a url and then the param. The task is created by invoking the add method on the queue.

queue.add(TaskOptions.Builder.url("/gaejsignupsubscriber").param("emailid",strEmailId));

4. For the readers information, I have shown a commented out line

//Queue queue = QueueFactory.getDefaultQueue();

which shows how to get the handle to the default queue in case you wish to place your tasks in the default queue itself.

5. Do not that all the Task Queue classes are experimental and are present in the com.google.appengine.api.labs.taskqueue package. This could change in the future.

GAEJSignupSubscriberServlet.java

This servlet contains the core task logic. This will be invoked by Google App engine if it finds any tasks present in the subscription-queue. If any tasks are there, it will pick them up and invoke the URL mentioned in the Task and pass to it the data present in the Task instance.  The code shown below is straightforward, it simply logs a statement saying that it got invoked and it also logs the email id.

package com.gaejexperiments.taskqueue;

import java.io.IOException;
import java.util.logging.Logger;

import javax.servlet.ServletException;
import javax.servlet.http.*;

@SuppressWarnings("serial")
public class GAEJSignupSubscriberServlet extends HttpServlet {
 private static final Logger _logger = Logger.getLogger(GAEJSignupSubscriberServlet.class.getName());
 public void doGet(HttpServletRequest req, HttpServletResponse resp)
 throws IOException {

 String strCallResult = "";
 resp.setContentType("text/plain");
 try {
 String strEmailId = req.getParameter("emailid");
 _logger.info("Got a Signup Subscriber Request for Email ID : " + strEmailId);
 //
 // PUT YOUR TASK CODE HERE
 //
 strCallResult = "SUCCESS: Subscriber Signup";
 _logger.info(strCallResult);
 resp.getWriter().println(strCallResult);
 }
 catch (Exception ex) {
 strCallResult = "FAIL: Subscriber Signup : " + ex.getMessage();
 _logger.info(strCallResult);
 resp.getWriter().println(strCallResult);
 }
 }

 @Override
 public void doPost(HttpServletRequest req, HttpServletResponse resp)
 throws ServletException, IOException {
 doGet(req, resp);
 }

}

queue.xml

All queues are configured in a file named queue.xml. Google App Engine provides a default queue. This queue is aptly named “default“. But in case you need to define your own queues, which is what we are going to do, we need to define them in a file called queue.xml. This file is placed in the WEB-INF directory of your application. You can also override settings of the default queue by defining it in the file and providing your own values.

Take a look at the queue.xml shown below:

<?xml version="1.0" encoding="UTF-8"?>
<queue-entries>
 <queue>
 <name>default</name>
 <rate>5/s</rate>
 </queue>
 <queue>
 <name>subscription-queue</name>
 <rate>5/s</rate>
 </queue>
</queue-entries>

In the above configuration, you will find that we have defined our own queue named “subscription-queue”. There is also another element that we have defined for the <queue> called <rate>. This element determines the rate at which you tell Google App Engine to execute tasks. If you do not specify a rate, then the default execution rate is 5 tasks per second. In the above file, we have provided the expression as “5/s”, which reads as 5 per second. Other examples of <rate> expressions are 1000/d (One thousand per day), etc. I suggest to read up the documentation for more examples.

You will also find that we have defined the default queue and we can change the rate if we want. But I have left it as is.

Please make sure that the above file (queue.xml) is present in the WEB-INF folder at the time of deploying the application.

Configuring the Servlets (web.xml)

We need to add the <servlet/> and <servlet-mapping/> entry to the web.xml file. This file is present in the WEB-INF folder of the project. The necessary fragment to be added to your web.xml file are shown below. Please note that you can use your own namespace and servlet class. Just modify it accordingly if you do so. We are defining here both our servlets.

<servlet>
 <servlet-name>GAEJCreateTaskServlet</servlet-name>
 <servlet-class>com.gaejexperiments.taskqueue.GAEJCreateTaskServlet</servlet-class>
 </servlet>
 <servlet>
 <servlet-name>GAEJSignupSubscriberServlet</servlet-name>
 <servlet-class>com.gaejexperiments.taskqueue.GAEJSignupSubscriberServlet</servlet-class>
 </servlet>
 <servlet-mapping>
 <servlet-name>GAEJSignupSubscriberServlet</servlet-name>
 <url-pattern>/gaejsignupsubscriber</url-pattern>
 </servlet-mapping>
 <servlet-mapping>
 <servlet-name>GAEJCreateTaskServlet</servlet-name>
 <url-pattern>/gaejcreatetask</url-pattern>
 </servlet-mapping>

Task Execution in Action

I am assuming that you have already created a new Google Web Application Project and have created the above Servlets, web.xml and queue.xml respectively. For a change, we shall be running this episode within our local development server only.

So assuming that all is well, we will run our application, by right-clicking on the project and selecting Run As –> Web Application. Once you see the following message shown below, then the local server is ready to accept requests.

Nov 24, 2009 5:11:33 AM com.google.apphosting.utils.jetty.JettyLogger info
INFO: jetty-6.1.x
Nov 24, 2009 5:11:40 AM com.google.apphosting.utils.jetty.JettyLogger info
INFO: Started SelectChannelConnector@127.0.0.1:8080
The server is running at http://localhost:8080/

Follow the steps given below:

1. Launch the browser on your local machine and navigate to http://localhost:8080/_ah/admin. This is the administrative console of the local development server.

2. Click on Task Queues to view the current task queues that are configured in your application. You should see the screen shown below, which shows that we have configured two task queues: default and subscription-queue.

You will notice that both of the queues currently do not have any tasks since we have not created any.

3. The next step is to create a task in the subscription-queue. To do that, all we need to do is invoke the following url :

http://localhost:8080/gaejcreatetask?emailid=romin@rocketmail.com

This invokes our GAEJCreateTaskServlet that we have configured. It will create the sample task in the subscription-queue and we will see a message as shown below in the browser:

“Successfully created a Task in the Queue”

4. Click on the Task Queues link again, you will find that there will now be 1 task listed in the subscription-queue as shown below:

5. Click on the subscription-queue link. This will display the task instance as shown below:

6. Since this is the development server, no automatic execution of tasks (this will occur in Google App Engine cloud) will take place and you have to run them manually, by clicking on the Run button. Click that. It will display that there are no more tasks present if the task executes successfully.

7. To check that our task has executed successfully, we can visit the Console tab in Eclipse and you will see the success log as shown below:

Moving on

In this episode, we saw how to split up a process into individual tasks and assign them to the Google App Engine for execution. What we have demonstrated here is a simple flow and the specifics of the configuration. I encourage you to try it out in your applications and as an exercise deploy it to the Google App Engine and monitor it there via the Administration Console.

Do keep in mind that this API is experimental and is likely to change drastically. At the same time, if you have any feedback, it would be nice to pass it along to the Google App Engine team.

Till the next episode, Happy Multitasking!

Categories: Uncategorized

Episode 9: Using the Cron Service to run scheduled tasks

November 16, 2009 7 comments

Welcome to Episode 9. In this episode, we shall be looking at how you can run background tasks in your GAEJ Application. By background Task, I mean any piece of code that you would like to run at a scheduled time and independent of the user interaction.There are several examples of such tasks. For e.g. :

  • Hourly/Daily/Weekly/Monthly backup of data
  • End of the day report generation to report any errors, transactions, etc.
  • Sending an email at the end of the day (or once a day) with some information to subscribers. For e.g. News sites.

If you have written a few web applications, you would definitely have come across more scenarios like that.

In this episode, we shall cover the following:

  1. What is a Cron Job?
  2. How to schedule a Cron Job?
  3. Write a simple Cron Job that prints a single statement
  4. Configure, execute and monitor the Cron Job execution

Let’s go!

What is a Cron Job? When would you need one?

I will use information liberally from Wikipedia over here to explain some of the core concepts. You can refer to the Cron page at Wikipedia if you want.

The word ‘cron’ is short for Chronograph. A Cron is a time-based job scheduler. It enables our application to schedule a job to run automatically at a certain time or date. A Job (also known as a Task) is any module that you wish to run. This module can perform system maintenance or administration, though its general purpose nature means that it can be used for other purposes, such as connecting to the Internet and downloading email.

Examples include:

  • Taking a daily backup of data via a scheduled task and moving the file to another server. (Runs once daily)
  • Sending an email every week to your subscribers. (Runs once weekly)
  • Clearing the log files at the end of every day (Runs once daily)
  • Remind yourself of a wonderful weekend coming up, every Friday at 5:00 PM (Runs once a week on a Friday at 5:00 PM)

The Google App Engine provides a service called the Cron Service that helps us do two fundamental things:

  1. Allows your application to schedule these tasks.
  2. Execute these tasks based on their schedule.

What does a Cron Job look like? And how do I schedule one?

A Cron Job is nothing but a URL that is invoked by the Google App Engine infrastructure at its scheduled execution time. To write a Cron Job, you need to do the following:

1. Write a Java Servlet and configure it in the web.xml. Note down the URL where the servlet can be invoked. The URL is the <url-pattern> mentioned in the <servlet-mapping> for your Servlet configuration in web.xml. For e.g.  the URL is the <url-pattern> element specified in the segment of the web.xml that is shown below:

<servlet>
 <servlet-name>GAEJCronServlet</servlet-name>
 <servlet-class>com.gaejexperiments.cron.GAEJCronServlet</servlet-class>
 </servlet>

<servlet-mapping>
 <servlet-name>GAEJCronServlet</servlet-name>
 <url-pattern>/cron/mycronjob</url-pattern>
 </servlet-mapping>

2. Create a cron.xml file that specifies one or more Cron Jobs (Scheduled Tasks) that you want to execute. A sample for the above Cron Job is shown below:

<?xml version="1.0" encoding="UTF-8"?>
<cronentries>
 <cron>
 <url>/cron/mycronjob</url>
 <description>Put your Cron Job description here</description>
 <schedule>Put Cron Job Schedule here</schedule>
 </cron>
</cronentries>

The cron.xml file tells Google App Engine about the Cron Jobs that are scheduled by your application. This file resides in the WEB-INF directory of your application and is copied to the App Engine cloud when you deploy the application. The following points are important about the cron.xml file:

  1. Each Cron Job configured in your application is defined in a <cron/> element. So there can be one or more <cron/> elements.
  2. The above <cron/> element has the following 3 elements that defines the Job.
    • <url/> specifies where the Google App Engine can invoke your Cron Job. This is nothing but the Servlet URL that you defined in the web.xml file that we saw earlier.The Servlet URL will point to your Servlet which contains the Cron Job implementation.
    • <description/> is a simple text based description of what your Cron Job does. It does not influence any aspect of the execution and is used for display purposes when you look at your application configuration via the App Console.
    • <schedule/> is the time when your Job has to be executed. This is where you specify if your job is to be run daily, once every hour, on Friday at 5:00 PM, etc. It is completely dependent on when you wish to execute this job. However, you must follow some rules and they are specified in the documentation on Scheduling Format. I strongly recommend you to read it up to understand various ways of specifying the schedule. Some of the examples are: “every 1 minute”, “every 12 hours”, “every friday 17:00” and so on.

Develop a simple Cron Job

The first thing to do is to create a New Google Web Application Project. Follow these steps:

1. Either click on File –> New –> Other or press Ctrl-N to create a new project. Select Google and then Web Application project. Alternately you could also click on the New Web Application Project Toolbar icon as part of the Google Eclipse plugin.

2. In the New Web Application Project dialog, deselect the Use Google Web Toolkit and give a name to your project. I have named mine GAEJExperiments. I suggest you go with the same name so that things are consistent with the rest of the article, but I leave that to you. In case you are following the series, you could simply use the same project and skip all these steps altogether. You can simply go to the next part i.e. the Servlet code.

3. Click on Finish. This will generate the project and also create a sample Hello World Servlet for you. But we will be writing our own Servlet.

GAEJCronServlet.java

Our Cron Job is going to be very simple. It is simply going to print out a statement in the log file that says that it is getting executed. The Cron Service of Google App Engine automatically will invoke this Servlet when its scheduled time to execute has arrived. So all we need to do is code out Servlet. The code is shown below:

package com.gaejexperiments.cron;

import java.io.IOException;
import java.util.logging.Logger;

import javax.servlet.ServletException;
import javax.servlet.http.*;

@SuppressWarnings("serial")
public class GAEJCronServlet extends HttpServlet {
 private static final Logger _logger = Logger.getLogger(GAEJCronServlet.class.getName());
 public void doGet(HttpServletRequest req, HttpServletResponse resp)
 throws IOException {

 try {
 _logger.info("Cron Job has been executed");

 //Put your logic here
 //BEGIN
 //END
 }
 catch (Exception ex) {
 //Log any exceptions in your Cron Job
 }
 }

 @Override
 public void doPost(HttpServletRequest req, HttpServletResponse resp)
 throws ServletException, IOException {
 doGet(req, resp);
 }
}

The code is straightforward to understand. It has doGet() and doPost() methods. And you will find in the doGet() method, that we simply log with an INFO level, that the Cron Job has been executed. In fact, your actual Job implementation should go in here as indicated by the comments. So whether you are invoking a backend database, or sending a consolidated email report, etc should all go in here.

All that remains is to now tell the App Engine via configuration about your Servlet (via web.xml) and create the cron.xml file in which you will mention your Cron Job.

Configure the Cron Job

As mentioned, we need to configure the Servlet in the web.xml and also specify it in the cron.xml file. Let us look at that now:

Configuring the Servlet

We need to add the <servlet/> and <servlet-mapping/> entry to the web.xml file. This file is present in the WEB-INF folder of the project. The necessary fragment to be added to your web.xml file are shown below. Please note that you can use your own namespace and servlet class. Just modify it accordingly if you do so.

 <servlet>
   <servlet-name>GAEJCronServlet</servlet-name>
   <servlet-class>com.gaejexperiments.cron.GAEJCronServlet</servlet-class>
 </servlet>

 <servlet-mapping>
   <servlet-name>GAEJCronServlet</servlet-name>
   <url-pattern>/cron/gaejcronjob</url-pattern>
 </servlet-mapping>

Specifying the Cron Job (cron.xml)

The cron.xml for our application will contain only one Cron Job. And here we specify the Servlet URL along with the schedule. Notice that I have chosen to execute this Cron job every 2 minutes. But you are free to experiment if you like with different Schedule Formats. This files needs to be created in the WEB-INF folder of your project.

<?xml version="1.0" encoding="UTF-8"?>
<cronentries>
 <cron>
 <url>/cron/gaejcronjob</url>
 <description>GAEJExperiments Cron Job that simply announces that it got invoked.</description>
 <schedule>every 2 minutes</schedule>
 </cron>
</cronentries>

Deploy the Application

To deploy the application, follow these steps (they should be familiar to you now. I am assuming that you already have the Application ID with you):

  1. Click on the Deploy Icon in the Toolbar.
  2. In the Deploy dialog, provide your Email and Password. Do not click on Deploy button yet.
  3. Click on the App Engine Project settings link. This will lead you to a dialog, where you need to enter your Application ID [For e.g. my Application Identifier gaejexperiments]
  4. Click on OK. You will be lead back to the previous screen, where you can click on the Deploy button. This will start deploying your application to the GAEJ cloud. You should see several messages in the Console window as the application is being deployed.
  5. Finally, you should see the message “Deployment completed successfully”.

We can now check if the Google App Engine got our Cron Job correctly configured and verify if it is getting executed at the schedule that we have configured it to.

Monitoring the Cron Job

You can use the App Engine console to verify if your Cron Job is executing well or not. To do that, perform the following steps:

  1. Go to http://appengine.google.com and log in with your account.
  2. You will see a list of applications registered. Click on the application that you just deployed. In my case, it is gaejexperiments.
  3. When you click on a particular application, you will be taken to the Dashboard for that application, which contains a wealth of information around the requests, quotas, logs, versions, etc.
  4. Verify that the Cron Jobs that you specified in the cron.xml have been configured successfully for the application by clicking Cron Jobs, visible under Main. For our application that we deployed, here is the screen shot from the App Engine console:ep9-1

You will notice that the Cron Job has not yet run as the console indicates. Every time that the job is executed, this column is updated with the last date time stamp that the Job executed along with its status. Since we have configured our Job to run every 2 minutes, I waited for 2 minutes and then the job executed itself and when I refreshed the Cron Jobs page, the status was updated as shown below:

ep9-2

You can also click on the Logs link. This will display the application log. And all your application log statements that you code using the Logger class can be visible here. By default, the severity level is set at ERROR and we can change that to INFO and you should be able your log statements that had the log level of INFO. This was the log level at which we had logged the statement in our Java Servlet (Cron Job). Shown below is a screen shot of the log when the Cron Job was fired once.

ep9-3

Conclusion

This concludes Episode 9 of this series in which you learn how to schedule tasks in your Google App Engine applications. These background tasks that can be scheduled at a certain time and which are executed by the Cron Service are an indispensable part of several web applications that are deployed today. If you ever wish to do repeated tasks in your application without any user intervention like sending emails, crawling web sites, taking database backups, etc, then writing a Cron Job and scheduling it for execution is a key feature that you can utilize while deploying your application to Google App Engine.

There is a lot more to Cron Jobs and I suggest to read up the documentation.

Till the next episode, Happy Scheduling!

Categories: Uncategorized Tags: ,