Tuesday, May 12, 2009

Google App Engine runtime switching and the mysterious multiple of 9001

Following my previous posts (see #1, #2, #3) on JVM-switching on the Google App Engine platform I've done some additional testing to find out how requests are spread between GAE/J runtimes/JVMs.

These are my results from the latest round of testing:

During a period of 443 hours 1,132,310 requests were successfully served by a total of 13 runtimes (as identifed by Runtime.getRuntime().hashCode()). Approximately one request per second was sent during the duration of the test.

The requests were divided between the 13 runtimes as follows:
  • Runtime #: First request served - last request served (active during X hours), Y requests served
  • A: 2009-04-24 12:35:11 - 2009-05-12 20:30:31 (440 h), 205042 reqs
  • B: 2009-04-24 12:54:58 - 2009-05-12 07:01:00 (426 h), 36286 requests
  • C: 2009-04-24 17:41:18 - 2009-05-12 23:29:23 (438 h), 126793 requests
  • D: 2009-04-24 21:16:22 - 2009-05-12 21:00:07 (432 h), 93573 requests
  • E: 2009-04-25 02:25:25 - 2009-04-25 05:53:43 (3 h), 9001 requests
  • F: 2009-04-27 17:19:38 - 2009-05-06 23:04:39 (222 h), 180020 requests
  • G: 2009-04-27 23:44:52 - 2009-05-11 19:37:07 (332 h), 204317 requests
  • H: 2009-04-28 00:24:36 - 2009-04-28 03:53:25 (3 h), 9001 requests
  • I: 2009-04-28 14:36:22 - 2009-05-11 13:40:38 (311 h), 99011 requests
  • J: 2009-04-30 05:19:11 - 2009-05-11 02:49:43 (262 h), 135180 requests
  • K: 2009-04-30 19:30:01 - 2009-05-12 23:12:16 (292 h), 25084 requests
  • L: 2009-05-01 16:25:13 - 2009-05-01 19:54:14 (3 h), 9001 requests
  • M: 2009-05-11 08:11:53 - 2009-05-11 08:11:53 (0 h), 1 request
  • Total: 13 runtimes, 1,132,310 requests
As we can see there seem to be no upper limit on how long a runtime is kept alive. It seems like the runtimes are kept alive as long as new requests are coming in continuously.

At the time of doing this summary only two of the thirteen runtimes had served requests during the last two hours. These were runtimes C and K. Let's classify these runtimes as active.

This leaves us with eleven inactive runtimes. If we look at the request counts for these inactive runtime we find that five of the eleven runtimes have request counts that are multiples of 9001 (number of requests served % 9001 == 0). These are: runtime E with 9001 requests, F 180020, H 9001, I 99011 and L 9001.

It seems like each active runtime is given requests in increments of 9001 requests. My hypothesis is that one test request is passed to a newly introduced runtime, and that if this test request is successfully served within a pre-defined timeout period then the runtime is allowed to serve 9000 requests. The testing procedure is then repeated after those 9000 requests have been served. These are my speculations, so please comment if you can confirm or dismiss these speculations.


  1. As of last week, I'm getting the same instance (hash) back for all calls to Runtime.getRuntime().hashCode() -- even 10,000 or more of them. is it possible that they've hijacked this method to try and obfuscate what server is serving up the request?

  2. @Matthew: Thanks a lot for your comment! Nice to know that I'm not the only one tinkering with GAE runtime identification :-)

    I'm still seeing varying values of Runtime.getRuntime().hashCode(). My guess would be that you'll see different runtimes if you keep sending additional requests. At times I've seen 10,000 requests in a row being served by the same runtime.

    Please report your findings here if you do some additional testing.

  3. Hi there,

    One of the tricks employed in Android in order to make new JVM turn up _really damn fast_ is to keep a 'zygote' process ready for immediate cloning when a request comes in.

    I have no idea what the getHashCode() implementation is like for Runtime, but in many cases it will simply be a hash of the object's location in memory.

    Assuming Google are using modern kernels with ASLR (address space layout randomization), what you may actually be seeing is your requests being answered by 13 physical machines, each of which may have a different getHashCode() result for their Runtime instance, which would be allocated (and assigned a memory location) once, and duplicated for every call to fork() when starting up a 'new' JVM to handle a request.

    This is pure speculation, but there are very few sane ways to go from 'zero' to hot serving JVM without resorting to fork.

  4. Sorry, forgot to add, I mention this because Python interpreters don't stay alive for anywhere near the figures you're seeing (there were some good posts about this to the appengine-python group).

  5. I had no trouble navigating through all the tabs as well as related info. The site ended up being truly simple to access.
    Intraday Nifty Future Tips || Share market tips || Share and Earn.

  6. A good informative post that you have shared and thankful your work for sharing the information. Got some appealing information and would like to give it a try. Applaud your work and keep sharing your information.Really impressed! Everything is very open and very clear clarification of issues.
    Call Option || Option Tips || Stock Option Tips || Options || Nifty Options

  7. Wow, awesome blog layout! How long have you been blogging for? you make blogging look easy. The overall look of your website is great, let alone the content!
    Ncdex Tips || Indian Stock Market || Bullion Tips || Nifty Future Tips || Nifty Option Tips || Future Tips

  8. I’m satisfied that you simply shared this useful information with us.
    Commodity tips || Stock Market || Share Market || Nse Tips || Bse Tips.