BenchMark 1.1d

Practical
Embedded Java
BenchMarks

A practical engineering approach to using embedded Java in real-world applications.

Buy It

...

BenchMark 1.1d 2009 Aug 18,19 - adding Android

2009 Aug 19: HTC G1 Phone Hardware running Android 1.0 OS This is the oldest Android version. Some tests were significantly slower than the emulator, but most were remarkably close. Did multiple runs. This processor is fast enough that the number of iterations in the test needs to be increased. Variation from test to test is probably due to the nature of Android, which has several processes running at once.

DON'T PAY MUCH ATTENTION TO THE TLE VALUES since they can be heavily influenced by background tasks. This can be a huge factor on Android, so don't assign much value to the TLE. The TLE is a non-weighted average of every part of a test run, even the parts with suspect values caused by background task intrusion. I know it's easier to compare one number, but in the case of Android that is misleading. Look at the major task times such as math divisions, object creations, and string operations. Those are more meaningful.

Benchmark changes contemplated: This benchmark should be rewritten to do some self-adjustment of iterations and test size, and then analyze the results statistically, with standard deviation, max and min over some large number of test runs. Then try turning off background tasks which can be controlled (email, GPS, WiFi, other background user apps) and see what the difference is.

These results are so good in part because the HTC G1 uses the Qualcomm MSM7201A chipset which includes 528 MHz ARM11 Jazelle™ Java® hardware acceleration. This was a really smart choice of Google and HTC.

2009 Aug 18: Android Emulator Possibly completely meaningless results for Android 1.1 in the emulator, which is claimed to be instruction-timing accurate. With Debugger running in order to use LogCat to see System.out messages. System.out messages using this helper code. Take these first results with a bag of salt. I'm just learning to program Android and there's a lot to know before I can stand by these results.

Source code (warning, it's pretty awful at this point) is available through SVN at the Java.net Android project at https://android.dev.java.net/ along with a lot of my take on Android. Also check out the mail list archives for important information on using the code.

Comments and Disclaimer:

Please read the notes below the results. We make no claims that these are exhaustive or suitable for any specific purpose.. No I/O tests are shown here, for example. The BenchMark test code is probably highly imperfect and could be improved. (Send us your suggestions!) These tests are probably a good representation of what real world code is like. For example, it does no good to claim a math operation in "XX time" if you don't also allow the time to fetch operands and store the results. This code is intended to be generic, it is not optimized for any specific hardware. YMMV. UAYOR. BYOB.

All values are operations per second, so higher numbers are better(faster). Best result per test is in bold face. Due to integer rounding and granularity of system timers (1 msec) results will vary slightly from run to run.
module processor chip product model avail code memory processor clock firmware version Java version	DS80C400 TStik.72.nb 1024K 29.491 MHz 1.11 JDK1.1.8	Imsys Cjip SNAP 8096K 80 MHz 0.7.1 J2ME/CLDC1.0	aJile aJ80 JStamp 512K 73 MHz 3.16.07 J2ME/CLDC1.0	aJile aJ100 JStik LP2/4 4096K 103 MHz 3.16.07 J2ME/CLDC1.0	aJile aJ100 SaJe 4096K 103 MHz 3.16.07 J2ME/CLDC1.0	Sun eSPOT Blue (Beta) 4096K 180 MHz 080609 J2ME/CLDC1.1	Android 1.1 In emulator on IBM T43 java.net project	HTC G1 Dev Phone 528 MHz Qualcomm MSM7201A 256 MB ROM 192 MB RAM Android 1.0 OS
32768 element byte array 262144 element accesses	4068	106519	206575	879677	956,729	544,997	1,254,277	744,727 3,084,047 (9) 2,016,492
32768 element byte array 262144 element copies (note 5)	1448309	23831272	4443118	32,768,000	52,428,800	7,943,757		131,072,000 110,376,421 (10)
16383 element int array 131072 element accesses (note 1)	3891	105109	233224	1008246	1,092,266	429,744	963,764	1,236,528 1,927,529 2,001,099
16383 element int array 131072 element copies (note 5)	360087	6241523	1092266	7281777	10,922,666	1,956,298	too fast note 7	26,214,400 32,768,000 (8)
byte add	13542	263504	793650	2702702	2,531,645	1,449,275	note 7	3,448,275 5,555,555
byte sub	13514	262812	803212	2777777	2,597,402	1,449,275	note 7	3,448,275 5,555,555
byte mul	12189	241545	561797	1369863	1,315,789	1,428,571	note 7	3,448,275 4,878,048
byte div	5859	105042	561797	1360544	1,324,503	1,169,590	7,142,857	2,564,102 3,846,153 3,773,584
int add	13793	90867	641025	2898550	2,985,074	1,724,137	10,000,000	6,451,612 11,111,111
int sub	13765	90579	641025	2898550	2,985,074	1,724,137	6,666,666	6,666,666 11,111,111
int mul	8228	88183	478468	1587301	1,639,344	1,639,344	8,695,652	6,060,606 10,526,315
int div	2827	60864	621118	1351351	1,379,310	1,333,333	4,166,666	3,636,363 5,714,285 5,714,285
float add	2149	143781	735294	2985074	2,941,176	1,015,228	2,380,952	2,298,850 3,636,363
float sub	1871	137268	847457	2666666	2,631,578	1,025,641	2,083,333	2,247,191 3,448,275
float mul	1528	125234	826446	1408450	1,379,310	1,098,901	2,531,645	2,597,402 4,081,362
float div	201	76804	561797	1379310	1,351,351	706,713	1,324,503	1,333,333 2,083,333 2,105,263
double add	3104	96993	595238	1754385	1,724,137	508,905	1,980,198	2,061,855 3,125,000
double sub	2657	93896	813008	1612903	1,587,301	502,512	1,834,862	1,941,747 2,941,176
double mul	1755	64040	377358	581395	574,712	501,253	1,869,158	2,020,202 3,125,000
double div	207	34506	311526	574712	568,181	215,053	404,040	366,972 571,428 574,712
string concat	46	247	879	3711	4051	607	5208	4113 5100 5376
string compare	1756	5390	64935	270270	277,777	75,187	1,428,571	20,000,000 (8) 1,818,181 3,333,333 (8)
method calls	5556	14729	271739	847457	854,700	1,000,000	1,694,915	1,666,666 2,439,024 2,857,142
object creations	496	3635	17331	36101	41,152	66,006	161,290	92,592 115,606 127,388
Total Loop Executions (TLE) Note 4a	492	11557	41106	146536	151,269	44,148	186,579	167,193 441,886 690,666
Price @10	$99?	$130?	$99	$299	$399	100-$250	free software	$400 HTC G1
TLEs per $ (higher is better) Note 4b	4.97	88.9	415	490	379	177-441	???	whole phone not just processor module
Date of Test						2008Oct28	2009Aug18	2009 Aug 19
Result file						eSpotBlue	not yet!
Website	www.tstik.com	www.imsys.se	www.jstamp.com	www.jstik.com	saje.systronix.com	sunspotworld.com	developer.android.com

Notes (see the source code for all the details of each test)

1-TINI arrays can be a max of 64 KBytes in size, so for ints this is a limit of ((2**16)/4) - 1 elements, so I limited all the test arrays to this size.

2- When applications are deployed, TINI executes from NVRAM, SNAP from PSRAM, JStamp, JStik, and SaJe from Flash.

3- Our test setup is still under development, so these results may change in future revisions. Your mileage may vary. Take any benchmark with a grain of salt, two aspirin, or a pint of carbonated malt beverage, as appropriate. No animals were harmed in these tests.

4a - TLE is the unweighted total number of test iterations divided by the total elapsed time from the start of the first test through the end of the last (including any GC, output generation, or anything else that might occur between individual tests). Individual test times do not include loop iteration (that is calculated with an empty loop and subtracted out), deliberate GC, output generation, or other tasks under program control.

4b- TLEs per $ is an almost completely arbitrary attempt to assign some "relative score" to each system. Take this with a bag of salt. Adding more memory to a system increases its price and therefore lowers its score (since the speed did not increase) but its value probably is actually higher since it can run larger programs and/or store more data. Another example: JStik's score is about the same as JStamp, but JStik has much more memory, ethernet, high speed I/O interface, etc. Also this value will be way off if any of the individual tasks have irrational results (on Android this can happen due to other background processes).

5- array copies use System.arraycopy which will usually include optimized array access native methods

6- In summer 2008 a kit of three (one base station and two free range) costs $750 or $300 with educational discount. SunSPOTs include an 802.15.4 radio and some basic sensors which are not present on other modules. The base station does not include a battery or eDemo sensor board, just the processor/radio.

7- Some tests had an elapsed time too small to use in a meaningful calculation. Note: these results may be complete nonsense!

8- Iteration time was zero or very low which may cast some suspicion on the result of this test
9 - this iteration invoked from phone app list. First test was invoked from Android Eclipse IDE.
10 - made byte array copy task 8 times larger

Systronix® 939 Edison St, Salt Lake City, Utah, USA 84111
Tel +1-801-534-1017, Fax +1-801-534-1019
contact us Time Zone: MDT (UTC-6)

Java and all Java-based marks are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. and other countries.
Systronix is independent of Sun Microsystems, Inc.
TStik, JStik, JCX, JStamp, JSimm, JDroid, and JRealTime are trademarks of Systronix, Inc.
1-Wire, iButton and TINI are trademarks of Dallas Semiconductor
Simmstick is a trademark of Dontronics
LEGO® is a trademark of Lego A/S, Denmark