| Benchmark Tests | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The purpose of these tests was not to obtain some absolute numbers but rather to compare perfomance of JGenerator and
Macromedia Generator in the same environment. Why ? Because Macromedia perfomed testing of their Generator
in some dedicated environment which obviously cannot be reproduced exactly, so any comparison would be rather pointless.
But if we compare two programs under exactly same conditions we can conclude that at least under these conditions
one program is faster (or whatever) than another.
Because of these reasons and because I did not have access to some dedicated network I decided to eliminate network issues from the tests at all. I used http connections of course but it was within a single computer, so that all connections were local. For these tests I wrote very simple client. It's a Java program which creates specified number of threads (users) and then starts them. Each thread (user) opens a connection to the given url and reads it to an internal buffer. Then collects some statistics, closes the connection and repeats these steps again. There is also another thread which is sleeping most of the time, but wakes up once in 5 seconds and prints the statistics on the screen. Here is the source code of this program. Since I had Macromedia Generator Developer Edition which has a limitation of no more than one thread (connection) at a time, I used only one thread (user) when perfomed the tests. So for all the tests the command used was: java com.iv.flash.SiteTest -users 1 http://hostname/testname.swtAfter starting the client I was waiting several iterations to pick the best result (the maximum number of templates per second). The tests were conducted on dual Pentium III 450Mz (Dell Precision 610 Workstation), 512Mb of RAM, Windows NT 4.0 (Service Pack 6), Web server IIS, servlet engine JRun 2.3. JRun was running under JDK 1.3 (Hotspot). See some observations about VMs' perfomance below. According to Windows Task Manager one CPU was loaded all the time, another was loaded from 70% to 100%. So I believe one CPU was running the servlet and another the client. Well, of course this assumption is not quite accurate, but it is ok for our tests. I used standard MM Generator examples for the testing. You can find them under examples/ folder in MM Generator installation. Since JGenerator does not support insert gif/png right now, I modified the example which inserts gif to do insert jpeg (which is supported). This test is named insertJpeg.swt in the table below. The results in the table are extractions from the output of the SiteTest program described earlier. For those who want to see the raw output check these links: MM Generator, JGenerator. The output from MM Generator is a little bit bigger because this developer edition I used adds the Macromedia logo to the resulting file. Note that it is not the case for insMovie test. Such a big difference (in 6 times) in size is not because of this logo :), but because of the way MM Generator generates an output file (really bad way!). So here are the results:
Well, as you can clearly see (and easily repeat this procedure if you want) JGenerator outperforms MM Generator in all the tests. Sometimes, as for example in case of insTicker (which is very important example!), JGenerator faster than MM Generator in 20 times! And this is despite the fact that MM Generator is written in C++, but JGenerator in pure Java! I let you draw the conclusion by yourself. Since version 1.1.4 JGenerator supports caching of media, requests and fonts. I did conduct tests with requests cache enabled and media cache enabled. The results are here:
|
||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Interesting to note that perfomance of JGenerator under Hotspot VM is about 1.5 times faster than under JDK 1.2, but perfomance of Macromedia Generator is almost the same under both of these VMs. This is very well understandable considering the fact that most of MM Generator code resides in native (C++) libraries. This is also, from my point of view, major MM Generator design flaw which results in such a poor perfomance. They have very fine-grained native API and use it very extensively, but it's a very known fact that JNI is VERY slow in almost all VM (at least those from Sun). |