Google AppEngine: Measuring Java and Python performance

Google has recently made available the hosting of Java applications on the Google AppEngine environment which is fantastic! It did get me thinking what would the performance of Java and Python be like when hosted in the Google environment? This is difficult to answer as one of the appealing aspects of AppEngine is that the developer does not need to concern themselves with the hardware environment. This also means that we don’t know the precise details as to how the applications are clustered or given allocation of CPU. However, one thing that is appealing about the AppEngine is that it is a scalable environment that can grow depending upon the application requirements. So… I thought that I’ll write a very primitive Java and Python script to run on a local machine and on AppEngine and see what happens.

Disclaimer!! – As with any metric testing, I need to state a disclaimer that a lot of factors can heavily influence the timing, such as other applications running locally, network performance, Java optimization, garbage collection, class loading, caching etc. etc. the list can go on and on. There isn’t an easy way that I’m aware of to get precise and fair timings across all environments. In particular as it’s a hosted environment Google could decide next week to upgrade the hardware thus giving a very different set of results!

I decided to create a bare minimum app that calls math functions that are available in Java and Python. Upload this app to AppEngine without any optimizations to the code or changing any of the default settings in AppEngine. I wanted the code in both apps to be as similar as possible, but another disclaimer Python and Java can’t be compared fully as they both have differences in their behavior. In particular when using the Math functions in Python it is dependent upon the OS that is hosting it. I.e. running ‘print math.pow(10,6)’ generates an error in the Google Python environment.

I decided to run three tests using Square Root, Power and Log. Run it 10 times and note the output. The local environment that I ran the tests is a single core Intel Pentium M 2Ghz, Win XP Service Pack 3 with 1Gb RAM.

What happened?

Before I go any further, the code that I used… yes, yes.. optimizations can be applied etc. but I wanted to use the most basic app out of the box.

Java:

import java.io.IOException;
import javax.servlet.http.*;
@SuppressWarnings("serial")
public class metric1Servlet extends HttpServlet
{
 public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException
 {
   resp.setContentType("text/plain");

  long start = System.nanoTime();
  int result = 0;
  for (int i = 0; i < Math.pow(10,5); i++)
  {  result += Math.sqrt(i);  }
  long duration = (System.nanoTime() - start) / 1000000;
  resp.getWriter().println("Test 1: " + duration + "ms");

  start = System.nanoTime();
  result = 0;
  for (int i = 0; i < Math.pow(10,5); i++)
  { result += Math.pow(Math.E, 2) }
  duration = (System.nanoTime() - start) / 1000000; <
  resp.getWriter().println("Test 2: " + duration + "ms"); 

  start = System.nanoTime();
  result = 0;
  for (int i = 0; i < Math.pow(10,5); i++) &
  {  result += Math.log(Math.random()); }
  duration = (System.nanoTime() - start) / 1000000;
  resp.getWriter().println("Test 3: " + duration + "ms");
 }
}

Python:

import time
import math
import random

print 'Content-Type: text/plain'
print ''

start = time.time()
result=0
for i in range(0,int(math.pow(10,5))):
result =+ math.sqrt(i)
duration = (time.time() - start)*1000
print "Test 1: ", duration ,"ms"

start = time.time()
result=0
for i in range(0,int(math.pow(10,5))):
result += math.pow(math.e, 2)
duration = (time.time() - start)*1000
print "Test 2: ", duration ,"ms"

start = time.time()
result=0
for i in range(0,int(math.pow(10,5))):
result += math.log(random.random())
duration = (time.time() - start)*1000
print "Test 3: ", duration ,"ms"

I should mention that the AppEngine has a request timeout to kill any page requests taking longer than 30 seconds. The tests had to be quite short thus most disappointingly couldn’t be used to calculate the ‘meaning of life’. Perhaps next time Google? Anyway, details about the request timer for Java can be found here under ‘Request Timer’.

The output:

Test 1:

Test 2:

Test 3:

The first thing from the results is that running it 10 times really isn’t enough… when I get the time, I need to run it on a larger scale.

From the data, it’s interesting to see that Java is fairly consistent overall and there is a performance boost running it on the Google environment which is what I would expect as my local machine isn’t exactly ‘state of the art’. Test 2 does show some discrepancy for running the Java locally but shows some sign of consistency towards the end.

Python is very different in that running it in the AppEngine environment doesn’t show anywhere near the consistency of Java. The chart with the averages below shows the differences clearer.

Could it be the timing of when I ran the tests, such that the Python servers have more workload and that the Java environment is on a limited rollout? Hard to say… also running Java locally is consistently slower whereas Python in the AppEngine doesn’t always seem to be faster. Are these results just a one off? I used a clean install of Python when testing locally thus I made no customizations to the Python environment or to the Python App Engine SDK. I think I’ve raised more questions to myself than answers…hmmm not what I was planning! I think the key thing that I take from these basic tests is that it has shown differences between the two environments and that I really need to do these again on a larger scale.

That lead to another question, is it worth developing everything in Java as it’s that much faster than Python? I wouldn’t say so, as the tests don’t really answer this question sufficiently as both languages have their own strengths and weaknesses. Also, if it is the case that Java is on a limited rollout, the results could be very different when the Java AppEngine is fully available to everyone.

A key point that I take from these tests, is that there are so many ‘factors’ involved when the application is being hosted, it’s got me thinking about how applications can be accurately stress tested and measured when the hardware environment could change any time along with the varying performance of the local internet connectivity. Somebody might already have the answer for this, if they have let me know!

Advertisement