Practical
Embedded Java
BenchMarks


A practical engineering approach to using embedded Java in real-world applications.

...

BenchMark 1.1e 2009 Sep 23 Android 1.5

2009Sep 23: HTC G1 Phone Hardware running Android 1.5 OS These results use an improved test which runs longer in some tests so that there is more elapsed time over which to calculate speed. Also, to my surprise, there was no consistent difference with phone in airplane mode and with WiFi and the phone enabled. Variation in results from run to run is on the order of at least 10%. This new version of the code breaks the TLE value, which was never meaningful anyway, so it's not reported in the new test. By this time I am believing the earlier simulator results, which agree pretty well with the G1 hardware results. Also there doesn't seem to be any significant change in these results between Android 1.1 and 1.5.

DON'T PAY MUCH ATTENTION TO THE TLE VALUES since they can be heavily influenced by background tasks. This can be a huge factor on Android, so don't assign much value to the TLE. The TLE is a non-weighted average of every part of a test run, even the parts with suspect values caused by background task intrusion. I know it's easier to compare one number, but in the case of Android that is misleading. Look at the major task times such as math divisions, object creations, and string operations. Those are more meaningful.

Benchmark changes contemplated: This benchmark should be rewritten to do some self-adjustment of iterations and test size, and then analyze the results statistically, with standard deviation, max and min over some large number of test runs. Then try turning off background tasks which can be controlled (email, GPS, WiFi, other background user apps) and see what the difference is.

These results are so good in part because the HTC G1 uses the Qualcomm MSM7201A chipset which includes 528 MHz ARM11 Jazelle™ Java® hardware acceleration. This was a really smart choice of Google and HTC.

2009 Aug 18: Android Emulator Possibly completely meaningless results for Android 1.1 in the emulator, which is claimed to be instruction-timing accurate. With Debugger running in order to use LogCat to see System.out messages. System.out messages using this helper code. Take these first results with a bag of salt. I'm just learning to program Android and there's a lot to know before I can stand by these results.

Source code (warning, it's pretty awful at this point) is available through SVN at the Java.net Android project at https://android.dev.java.net/ along with a lot of my take on Android. Also check out the mail list archives for important information on using the code.

Comments and Disclaimer:
Please read the notes below the results. We make no claims that these are exhaustive or suitable for any specific purpose.. No I/O tests are shown here, for example. The BenchMark test code is probably highly imperfect and could be improved. (Send us your suggestions!) These tests are probably a good representation of what real world code is like. For example, it does no good to claim a math operation in "XX time" if you don't also allow the time to fetch operands and store the results. This code is intended to be generic, it is not optimized for any specific hardware. YMMV. UAYOR. BYOB.

All values are operations per second, so higher numbers are better(faster). Best result per test is in bold face.
Due to integer rounding and granularity of system timers (1 msec) results will vary slightly from run to run.
module processor chip
product model
avail code memory
processor clock
firmware version
Java version
aJile aJ100
JStik LP2/4
4096K
103 MHz
3.16.07
J2ME/CLDC1.0
aJile aJ100
SaJe
4096K
103 MHz
3.16.07
J2ME/CLDC1.0
Sun eSPOT
Blue (Beta)
4096K
180 MHz
080609
J2ME/CLDC1.1

Android 1.1
In emulator
on IBM T43

java.net project

HTC G1 Dev Phone
528 MHz Qualcomm MSM7201A
256 MB ROM
192 MB RAM
Android 1.0 OS

HTC G1 Dev Phone
Android 1.5 OS
Airplane mode or with WiFi and phone enabled
Benchmark 1.1d
java.net project

32768 element byte array
262144 element accesses
879677
956,729
544,997
1,254,277
744,727
3,084,047 (9)
2,016,492
2,880,703
32768 element byte array
262144 element copies (note 5)
32,768,000
52,428,800
7,943,757
131,072,000
110,376,421 (10)
2,147,483,648 byte array copies: 17,165 ms
125,108,281 per sec
16383 element int array
131072 element accesses (note 1)
1008246
1,092,266
429,744
963,764
1,236,528
1,927,529
2,001,099
1,579,180
16383 element int array
131072 element copies (note 5)
7281777
10,922,666
1,956,298
too fast
note 7
26,214,400
32,768,000
(8)
1,073,676,288 int array copies: 28,875 ms
37,183,594 per sec
byte add
2702702
2,531,645
1,449,275
note 7
3,448,275
5,555,555
6,451,612
byte sub
2777777
2,597,402
1,449,275
note 7
3,448,275
5,555,555
6,060,606
byte mul
1369863
1,315,789
1,428,571
note 7
3,448,275
4,878,048
5,405,405
byte div
1360544
1,324,503
1,169,590
7,142,857
2,564,102
3,846,153
3,773,584
4,081,632
int add
2898550
2,985,074
1,724,137
10,000,000
6,451,612
11,111,111
10,526,315
int sub
2898550
2,985,074
1,724,137
6,666,666
6,666,666
11,111,111
10,000,000
int mul
1587301
1,639,344
1,639,344
8,695,652
6,060,606
10,526,315
9,523,809
int div
1351351
1,379,310
1,333,333
4,166,666
3,636,363
5,714,285
5,714,285
5,128,205
float add
2985074
2,941,176
1,015,228
2,380,952
2,298,850
3,636,363
3,636,363
float sub
2666666
2,631,578
1,025,641
2,083,333
2,247,191
3,448,275
3,225,806
float mul
1408450
1,379,310
1,098,901
2,531,645
2,597,402
4,081,362
3,921,568
float div
1379310
1,351,351
706,713
1,324,503
1,333,333
2,083,333
2,105,263
2,040,816
double add
1754385
1,724,137
508,905
1,980,198
2,061,855
3,125,000
3,030,303
double sub
1612903
1,587,301
502,512
1,834,862
1,941,747
2,941,176
2,941,176
double mul
581395
574,712
501,253
1,869,158
2,020,202
3,125,000
3,125,000
double div
574712
568,181
215,053
404,040
366,972
571,428
574,712
584,795
string concat
3711
4051
607
5208
4113
5100
5376
4314
string compare
270270
277,777
75,187
1,428,571

20,000,000 (8)
1,818,181
3,333,333 (8)

2,758,620
method calls
847457
854,700
1,000,000
1,694,915
1,666,666
2,439,024
2,857,142
2,777,777
object creations
36101
41,152
66,006
161,290
92,592
115,606
127,388
70,859
Total Loop Executions (TLE) Note 4a
146536
151,269
44,148
186,579
167,193
441,886
690,666
Price @10
$299
$399
100-$250
free software
$400 HTC G1
TLEs per $ (higher is better) Note 4b
490
379
177-441
???
whole phone not just processor module
Date of Test    
2008Oct28
2009Aug18
2009 Aug 19
2009 Sep 23
Result file    
not yet!
 
Website
Notes (see the source code for all the details of each test)
1-TINI arrays can be a max of 64 KBytes in size, so for ints this is a limit of ((2**16)/4) - 1 elements, so I limited all the test arrays to this size.
2- When applications are deployed, TINI executes from NVRAM, SNAP from PSRAM, JStamp, JStik, and SaJe from Flash.
3- Our test setup is still under development, so these results may change in future revisions. Your mileage may vary. Take any benchmark with a grain of salt, two aspirin, or a pint of carbonated malt beverage, as appropriate. No animals were harmed in these tests.
4a - TLE is the unweighted total number of test iterations divided by the total elapsed time from the start of the first test through the end of the last (including any GC, output generation, or anything else that might occur between individual tests). Individual test times do not include loop iteration (that is calculated with an empty loop and subtracted out), deliberate GC, output generation, or other tasks under program control.

4b- TLEs per $ is an almost completely arbitrary attempt to assign some "relative score" to each system. Take this with a bag of salt. Adding more memory to a system increases its price and therefore lowers its score (since the speed did not increase) but its value probably is actually higher since it can run larger programs and/or store more data. Another example: JStik's score is about the same as JStamp, but JStik has much more memory, ethernet, high speed I/O interface, etc. Also this value will be way off if any of the individual tasks have irrational results (on Android this can happen due to other background processes).

5- array copies use System.arraycopy which will usually include optimized array access native methods
6- In summer 2008 a kit of three (one base station and two free range) costs $750 or $300 with educational discount. SunSPOTs include an 802.15.4 radio and some basic sensors which are not present on other modules. The base station does not include a battery or eDemo sensor board, just the processor/radio.
7- Some tests had an elapsed time too small to use in a meaningful calculation. Note: these results may be complete nonsense!
8- Iteration time was zero or very low which may cast some suspicion on the result of this test
9 - this iteration invoked from phone app list. First test was invoked from Android Eclipse IDE.
10 - made byte array copy task 8 times larger

 
Systronix® 939 Edison St, Salt Lake City, Utah, USA 84111
Tel +1-801-534-1017, Fax +1-801-534-1019
contact us     Time Zone: MDT (UTC-6)
 

Java and all Java-based marks are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. and other countries.
Systronix is independent of Sun Microsystems, Inc.
TStik, JStik, JCX, JStamp, JSimm, JDroid, and JRealTime are trademarks of Systronix, Inc.
1-Wire, iButton and TINI are trademarks of Dallas Semiconductor
Simmstick is a trademark of Dontronics
LEGO® is a trademark of Lego A/S, Denmark