Episode 13: Using the Blobstore Java API

Welcome to Episode 13 of this series. It has been a while since the last episode and to compensate I thought why not cover something that has just been released. In keeping with that line, this episode will cover the recently released Blobstore API in the 1.3.0 release of the Google App Engine SDK.  

Using the Blobstore API, we shall cover how to build an application that mimics several sites that allow you to upload a picture and send a Tweet. The Tweet will then contain a link back to the image that you uploaded. Blobstore API now makes all this possible along with several other applications that you could envision like file sharing, document management, etc.  

What is the Blobstore API?

The Blobstore API allows your application to store blobs of information, where each blob can be upto 50MB in size. These blobs are typically large files like video and images. This is different from the datastore API that does not allow for saving objects upto this large size. The API documention for the Blobstore API is provided here.  

The Blobstore API provides two types of functions:  

• An ability to upload and save the blob automaticallyThe BlobstoreService which is provided by the com.google.appengine.api.blobstore.BlobstoreService allows us to specify a URL where users can upload their large files. You can think of this url as the action element in the HTML form. The implementation at this URL is internal to the BlobstoreService. But what it does is significant. It will extract out the file contents that you uploaded and store it as a Blob in the database. Each blob that is stored in the database is associated with the a Blob Key. This Blob key is then provided to your url and you can then use the Blob Key to do anything within your application. In our case, we form a url that is tweeted to the users who can then view the picture that we uploaded.

• An ability to serve or retrieve the blob.The BlobstoreService also provides an ability to serve or retrieve the blob that was saved successfully. It will provide the blob as a response that could then use as a source in an <img> element for example. All you need to do is provide it the Blob Key and the response stream and in return, it will provide the content properly encoded as per its type that you could then use.

It will be clear as we work out the application in the next few sections.  

Note: The Blobstore API is enabled in your application only if you enable “billing” for your applications. What this means is that you can try out the API with your local development server but if you deploy the same on the App Engine cloud, your application will not be able to use the API if it has not been enabled for “billing”. So keep that in mind before leaping with joy that your new application enabled with large file uploads, etc is now ready to start functioning in the cloud. In case you try to use any Blobstore APIs without enabling billing in your application, you will get a FeatureNotSupported exception and the text as: “The Blobstore API will be enabled for this application once billing has been enabled in the admin console.”  

Prerequisites

Google announced the release of version 1.3.0 of the App Engine SDK for both Python and Java. We need to set our development environment with this release of the SDK since the Blobstore API was released in this version.  

So the first thing that you need to do is to download the latest release 1.3.0 of the App Engine SDK and upgrade your Eclipse development environment with that. There are two ways you can do that:  

a. Use the Update features of Eclipse to download and install the latest SDK  

b. Download the SDK from the following link. Make sure you download the Java SDK. Further to the download, inside of your Eclipse IDE, go to Windows -> Preferences and then Google -> App Engine SDK. Click on Add and provide the path to the expanded 1.3.0 distribution on your machine.  

I had covered in detail how to upgrade your SDK for your Eclipse IDE when new releases come out. This was covered in Episode 5, in which we upgraded from 1.2.5 -> 1.2.6. Readers are advised to take a look at it in case they are not familiar. The process remains the same.  

Tweet My Picture in Action

The application that we are going to develop is called Tweet My Picture. There are several sites now that allow you to upload a picture to their site and then send a Tweet. The Tweet contains a simple description and the url to the picture. An example is TwitPic.  I thought the Blobstore API now actually makes an application of this kind way too easy to create, so why not demonstrate that. We will not be creating a polished looking application but rather demonstrating the API and showing that it works. Remember that all the instructions later on deploying and running this application will happen in the local development server and not the live App Engine site because I need to enable billing in order to utilize the Blobstore Service. But the best part is that the local development server works fine, so we can do our complete development and testing on it. Then you are the best judge of writing your own application that utilizes the Blobstore API and making it a billable application when you go live.  

Before we get down to code, we should see the end result.  

Assuming that I have started my local development server, I navigate to the index.jsp page that is shown below:  

This will display a simple page, where you provide the following information:  

• Your Twitter UserId and Password
• The image file that you wish to upload. Simply click the Choose File button and select a file to upload.

  

Then when you click the Upload Picture button, what happens is the following:  

1. The file contents that  you uploaded will be stored by the Blobstore API into the datastore automatically for you. It will provide you with the Blob Key for the blob that was saved.  

2. We use the Blob Key to create a simple Tweet message containing the link. And then we use Twitter4J to send the Tweet.  

3. Finally a page is displayed indicating the status of the Tweet, the Tweet contents and it also shows the image that you uploaded. This page is shown below:  

   

Now, switch to your Twitter client and you will notice that the Tweet has been posted. A sample screenshot is shown below.The url is pointing to the localhost but that is because I wish to do my testing locally and I have not enabled billing for my live application at the appspot.com domain.  

  

Clicking on the Tweet link, takes you back to an page that shows you the picture as below:  

  

Nice, isn’t it? Do not limit yourself only to this example. You can now start thinking on creating serious document management, file sharing, etc applications and use the App Engine infrastructure to host it. So let’s get coding now.  

Implementing Tweet My Picture

The first thing to do is to create a New Google Web Application Project. Follow these steps:  

1. Either click on File –> New –> Other or press Ctrl-N to create a new project. Select Google and then Web Application project. Alternately you could also click on the New Web Application Project Toolbar icon as part of the Google Eclipse plugin. Please make sure that you have installed the latest 1.3.0 version of the App Engine SDK and are using that as the API for this version. Shown below is the New Project screenshot where you should see an additional SDK 1.3.0 and need to select that.  

  

2. In the New Web Application Project dialog, deselect the Use Google Web Toolkit and give a name to your project. I have named mine GAEJExperiments. I suggest you go with the same name so that things are consistent with the rest of the article, but I leave that to you. In case you are following the series, you could simply use the same project and skip all these steps altogether. You can simply go to the next part i.e. the Servlet code.  

3. Click on Finish. This will generate the project and also create a sample Hello World Servlet for you. But we will be writing our own Servlet.  

Adding Twitter4J Jar to the Classpath and WEB-INF\lib folder

Since we will be using the excellent Twitter4J library for posting to Twitter, you need to download the latest version of the JAR file. Download it from here. I have used twitter4j-2.0.10.jar. You need to copy this file to the WEB-INF\lib folder of the project. Also add the JAR file to the Build Classpath of the Eclipse project.  

index.jsp

The first page is the index.jsp page. This is a simple page that will have a form that accepts the Twitter UserId/Password along with the file to be uploaded. The code is shown below. Just a quick note that all the JSP files will go into the war folder of the project.  

  

<%@ page import="com.google.appengine.api.blobstore.BlobstoreServiceFactory" %>
<%@ page import="com.google.appengine.api.blobstore.BlobstoreService" %>  

<%
    BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
%>
<html>
    <head>
        <title>Tweet My Picture</title>
    </head>
    <body>
        <img src="tweet.png"/>
        <h1>Tweet My Picture</h1>
        <hr/>
        <h2>Upload a picture and send an automatic Tweet</h2>
        <form action="<%= blobstoreService.createUploadUrl("/upload") %>" method="post" enctype="multipart/form-data">
            Twitter User Id : <input type="text" name="twitter_userid"/><br/>
            Twitter Password : <input type="p" name="twitter_password"/><br/>
            File :
            <input type="text" name="filename"/>
            <input type="file" name="myTweetPic"/>
            <input type="submit" value="Upload Picture"/>
        </form>
</html>  

 

Let us look at the main points in the code above:  

1. It has a HTML <form/> that accepts the twitter userid and password. We also have an  <input/> element of type file, that allows the user to select the file from his machine to be uploaded.  

2. The action attribute of the FORM is where you should pay attention. The Form action as you know submits the request parameters to a particular server side resource like a Servlet/JSP, etc. The action here is created via a helper function provided by the Blobstore API itself. Notice that first we are getting an instance of the Blobstore service via the BlobstoreServiceFactory as shown below:  

BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();  

Then we are using a helper function called blobstoreService.createUploadURL(“SUCCESS_PATH”) for the action. Let us spend some time on that since it is important and will help us understand what the BlobstoreService does behind the scenes.  

If you launch your local development server and navigate to the index.jsp page and do a view source on the HTML, you will find that the action that is generated by the createUploadURL(…) method looks something like this:  

action=”/_ah/upload/agp0d2VldG15cGljchsLEhVfX0Jsb2JVcGxvYWRTZXNzaW9uX18YEgw”. You may wonder where did our SUCCESS_PATH i.e. /upload go away. Let me explain:  

When you do an upload, the request hits an internal URL in the App Engine infrastructure that will do the following:  

1. The BlobstoreService implementation inside of AppEngine will extract out the Blob and save it for you automatically in the datastore. You do not have to do anything special. It does this for the all the Blobs that are present in the HTTP Request stream.  

2. On successfully saving these blobs into the stores, what it has with itself now are one or more name value pairs i.e. a Map. The name is the request parameter name for your blob. For e.g. in our index.jsp, the file that we uploaded had the name of myTweetPic i.e. the form element. And the value will be the Blob Key that you can use to retrieve the Blob. It will put this map into the Request stream i.e. it will augment the Request stream with this map and then invoke your SUCCESS_PATH url. The SUCCESS_PATH url is the end point or in our case the Servlet.  

This Servlet can then extract out each of the Blobs that were saved by inspecting the Map and do further action. In our case, we will be posting a link to Twitter. This ervlet is what we shall see next.   

Upload Servlet

The Upload Servlet code is listed below.  

package com.gaejexperiments.twitter;  

import java.io.IOException;
import java.util.Map;
import java.util.logging.Level;
import java.util.logging.Logger;  

import javax.mail.Session;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;  

import twitter4j.Status;
import twitter4j.Twitter;  

import com.google.appengine.api.blobstore.BlobKey;
import com.google.appengine.api.blobstore.BlobstoreService;
import com.google.appengine.api.blobstore.BlobstoreServiceFactory;  

@SuppressWarnings("serial")
public class Upload extends HttpServlet {
 private final static Logger _logger = Logger.getLogger(Upload.class.getName());
 private BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
   

    public void doPost(HttpServletRequest req, HttpServletResponse res)
        throws ServletException, IOException {  

        Map<String, BlobKey> blobs = blobstoreService.getUploadedBlobs(req);
        BlobKey blobKey = blobs.get("myTweetPic");  

        if (blobKey == null) {
            res.sendRedirect("/");
        }
        else {
         String strStatus = "";
         try {
          String twitter_userid   = (String)req.getParameter("twitter_userid");
          String twitter_password = (String)req.getParameter("twitter_password");
          _logger.info("UserId:"+twitter_userid + " , Password:"+twitter_password);
          Twitter twitter = new Twitter(twitter_userid,twitter_password);
          String strTweet = "Check out my picture at : " + "<a href="http://localhost:8888/viewpic.jsp?blob-key=&quot;+blobKey.getKeyString">http://localhost:8888/viewpic.jsp?blob-key="+blobKey.getKeyString</a>(); 
             Status status = twitter.updateStatus(strTweet);
             strStatus = "Successfully updated the status to [" + status.getText() + "].";
             _logger.info(strStatus);
         }
         catch (Exception ex) {
          strStatus = "Could not update Twitter Status : " + ex.getMessage();
          _logger.log(Level.WARNING,strStatus);
         }
         finally {
             res.sendRedirect("/submitpic.jsp?blob-key="+blobKey.getKeyString() + "&status="+strStatus);
         }
         }
    }
}

 

Let us go through the main points in the Upload Servlet code:  

1. The Servlet implementation is the doPost() method. This method was invoked by the Blobstore Service after the blobs were successfully saved in the datastore.  

2. As mentioned, the Blobstore Service augments the Request stream with a Map of successfully saved Blobs. It also provides a helper method that we can use to extract out this Map instance from the request stream. The code is shown below:  

Map<String, BlobKey> blobs = blobstoreService.getUploadedBlobs(req);  

3. The next thing to do is to get the Blob Key for our saved Blob. The Blob Key is unique and will be used to retrieve the Blob as we shall see later on in the viewpic.jsp code. The Key that we will use is the same as the input parameter name for our file blob that we provided in the FORM i.e. index.jsp.   

BlobKey blobKey = blobs.get(“myTweetPic”);  

4. We do a simple check to verify if the blobKey instance is not null. If not, we create a status update for twitter giving a url that points to the viewpic.jsp page which is passed the blob-key request parameter as shown below:  

String strTweet = “Check out my picture at : ” + “http://localhost:8888/viewpic.jsp?blob-key=”+blobKey.getKeyString();   

5. Finally, we using Twitter4J to post out tweet using the twitter id and password that we provided.  

The status of our upload and tweet is then displayed by navigating to the submitpic.jsp page that simply shows the status and the picture that we uploaded.   

submitpic.jsp

This JSP file is invoked by our Upload Servlet to display the status of our Tweet to twitter and the image that we uploaded as discussed in the previous section. You will notice an interesting thing over here and that is the <img/> element. The source attribute of the <img/> element is a servlet residing at an endpoint /serve. All it needs is the blob-key request parameter. We shall see this servlet in a moment.  

<%@ page import="com.google.appengine.api.blobstore.BlobstoreServiceFactory" %>
<%@ page import="com.google.appengine.api.blobstore.BlobstoreService" %>  

<html>
    <head>
        <title>Tweet My Picture - Submission</title>
    </head>
    <body>
        <img src="tweet.png"/>
        <h1>Tweet My Picture</h1>
        <hr/>
        <h3>Submission Result: <%=request.getParameter("status")%></h3>
        <%
           String blobKey = (String)request.getParameter("blob-key");
           if (blobKey != null) {%>
                You uploaded : <br/>
       <img width="200" height="150" src="<%="<a href="http://localhost:8888/serve?blob-key=&quot;+blobKey">http://localhost:8888/serve?blob-key="+blobKey</a> %>">
           <%}%>
    </body>
</html>  

  

Serve Servlet

This Servlet uses another helper function of the BlobStoreService. The method is called serve and it takes two parameters as shown in the listing below. The blob Key and the HTTP response stream. Note that we pass it the Blob Key that we got after saving our blob. What the serve method will do is that it will automatically retrieve the Blob from the Datastore using the Blob Key that was provided to it. Once it retrieves the Blob successfully, it will take the content of the Blob, set the correct MIME types and insert that into the HTTP Response stream. You can the use the Response stream to assign it as a source for the HTML <img/> element. We did exactly that in the submitpic.jsp file and we do the same in the viewpic.jsp file that will be invoked when someone clicks on the Twitter post.  

  

package com.gaejexperiments.twitter;  

import java.io.IOException;  

import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;  

import com.google.appengine.api.blobstore.BlobKey;
import com.google.appengine.api.blobstore.BlobstoreService;
import com.google.appengine.api.blobstore.BlobstoreServiceFactory;  

@SuppressWarnings("serial")
public class Serve extends HttpServlet {
 private BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
 
 public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
        BlobKey blobKey = new BlobKey(req.getParameter("blob-key"));
        blobstoreService.serve(blobKey, resp);
    }
}

  

viewpic.jsp

As discussed earlier, the Tweet that we post has the following format:  

Check out my picture at http://localhost:8888/viewpic.jsp?blob-key=A_BLOB_KEY  

This will invoke the viewpic.jsp file with a request parameter named blob-key which contains the Blob key value. So all we have to do i to create a <img/> element, whose source attribute will be served via the /serve endpoint as shown below. Simple isn’t it ?  

  

<%@ page import="com.google.appengine.api.blobstore.BlobstoreServiceFactory" %>
<%@ page import="com.google.appengine.api.blobstore.BlobstoreService" %>  

<%
    BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
    String blobKey = (String)request.getParameter("blob-key");
%>
<html>
    <head>
        <title>Tweet My Picture - View</title>
    </head>
    <body>
    <img width="200" height="150" src="<%="<a href="http://localhost:8888/serve?blob-key=&quot;+blobKey">http://localhost:8888/serve?blob-key="+blobKey</a> %>">
    </body>
</html>

   

Configuring the Servlets

We need to add both the Upload Servlet and the Serve Servlet with their appropriate entries for  <servlet/> and <servlet-mapping/> in the web.xml file. This file is present in the WEB-INF folder of the project. The necessary fragment to be added to your web.xml file are shown below.  

 <servlet>
  <servlet-name>Upload</servlet-name>
  <servlet-class>com.gaejexperiments.twitter.Upload</servlet-class>
 </servlet>
 <servlet>
  <servlet-name>Serve</servlet-name>
  <servlet-class>com.gaejexperiments.twitter.Serve</servlet-class>
 </servlet>
 <servlet-mapping>
  <servlet-name>Upload</servlet-name>
  <url-pattern>/upload</url-pattern>
 </servlet-mapping>
 <servlet-mapping>
  <servlet-name>Serve</servlet-name>
  <url-pattern>/serve</url-pattern>
 </servlet-mapping>

  

Deploying the application

Since we are going to be running this application locally itself, we simply need to validate it against the local development server. Assuming that the development is complete, simply run the Web Application from your Eclipse IDE and using a browser, navigate to http://localhost:8888/index.jsp. In my case, the port is 8888, but it could be different for you.  

Please make sure that you are connected to the Internet and do have a valid Twitter userid and password.   

Refer to the section above on Tweet My Picture in action for the flow. I will not repeat it here again.  

An interesting thing that you could look at is to navigate to http://localhost:8888/_ah/admin console. Go to the DataStore viewer. If you have uploaded several images successfully, you will find instances of the Entity : __BlobInfo__ getting created. Click on the List Entities button to see the entities that you have uploaded so far. The ID/Name column is the Blob Key for your Blob. A sample screenshot from my development server instance is shown below:  

  

Conclusion

This brings the episode to an end. In this episode we saw the Blobstore API service that has been introduced in version 1.3.0 of the App Engine SDK. The Blobstore API service is experimental at this point in time and could change but it gives us enough today to see how to create application that require blobs of large sizes like video and images. This episode showed a simple implementation called Tweet My Picture but the possibilities are endless. Do keep in mind that if you have to use Blobstore Service in a live application hosted at the appspot.com domain, then you will need to enable “billing” for your application.  

Till the next episode, have fun with the Blobstore Service!

P.S: Just in case you were wondering… the picture of the lion was taken at a distance of 2 feet away from it in the National Park, Mumbai.