BenchMark 1.1c

Practical
Embedded Java
BenchMarks

A practical engineering approach to using embedded Java in real-world applications.

Buy It

...

BenchMark 1.1c

2008 Oct 28- added SunSPOT Blue Pre-release

Blue shows some interesting and dramatic changes. Some array accesses and operations (byte and int math) are a bit slower. But float and double operations are up to 7X (float div) faster, with many ops 4X faster.

Purple shows small but steady improvement in speed in almost all areas. This is the result from the first run (two are available in the file, and are almost identical). Most notable differences: method call performance (purple vs orange) increased dramatically (1.5X) but object creation fell by about the same ratio(0.68X).

Note to everyone: the slightly more optimal byte array test #2 is in the source code and recently captured results and are the recent results reported here... so there may be some benchmarks which use the older version which isn't a level comparison with those (SPOT purple and blue, at least) which use the new version.

Note to self: add some thread context switching and serial I/O tests to the next version.

Orange (note the clever color coding in the table) is closer to the production release of SPOT runtime. The test results are the best of two runs, which can vary a bit due to time base granularity (msec) and rounding. Typically (it depends on the time the test takes to execute) the rounding uncertainty is less than 1% on math operations. Byte and int array copies take the least time so have the most uncertainty, about 3% for byte array copies and 2% for int array copies. So consider differences of 1% on any test insignificant. Overall, the Orange release is about 10% better than the Green. This possibly implies that 1) there's not a huge amount of optimization room remaining 2) the Green release was pretty optimal to begin with. Also, it appears that more wireless code is loaded on the Orange release, and could be using some ARM cycles.

Also note that due to a clerical error the Green release byte array copies were incorrectly reported as being about 10X slower than they really were. That has been corrected. The results files are posted -- you are welcome to check my results or, better yet, run the test yourself.

BenchMark 1.1a 2006 Sep 25 - added SunSPOT Beta "Green Release" results

SunSPOT Beta (CLDC 1.1) added. SaJe is now grayed out since it is a legacy product and will not be made available in a RoHS-compliant version, so it's results (which would otherwise be the fastest in many cases) are not highlighted. There is a coming new aJile release to CLDC1.1, which could affect results for JStamp, JStik, and SaJe. It will also be possible to add Float and Double object tests.

What's new in 1.1a (2003)

Math operations with byte primitives
System.arraycopy operating on byte and int arrays. Note how much faster this arraycopy is than the brute force array access. But now an out of memory exception occurs on the TINI390 512K module.
SaJe board is added.
Test iterations are doubled from 1.0f to provide a longer time over which to calculate operations per second.

Files

Here are the BenchMark.java build.xml prever.bat files and the raw results capture file.
For more technical details on these modules, see this feature comparison table.

Comments and Disclaimer:

Please read the notes below the results. We make no claims that these are exhaustive or suitable for any specific purpose.. No I/O tests are shown here, for example. The BenchMark test code is probably highly imperfect and could be improved. (Send us your suggestions!) These tests are probably a good representation of what real world code is like. For example, it does no good to claim a math operation in "XX time" if you don't also allow the time to fetch operands and store the results. This code is intended to be generic, it is not optimized for any specific hardware. YMMV. UAYOR. BYOB.

All values are operations per second, so higher numbers are better(faster). Best result per test is in bold face. Due to integer rounding and granularity of system timers (1 msec) results will vary slightly from run to run.
module processor chip product model avail code memory processor clock firmware version Java version	DS80C390 DSTINI 1024K 36.864 MHz 1.02e JDK1.1.8	DS80C390 DSTINI 1024K 36.864 MHz 1.11 JDK1.1.8	DS80C400 TStik.72.nb 1024K 29.491 MHz 1.11 JDK1.1.8	Imsys Cjip SNAP 8096K 80 MHz 0.7.1 J2ME/CLDC1.0	aJile aJ80 JStamp 512K 73 MHz 3.16.07 J2ME/CLDC1.0	aJile aJ100 JStik LP2/4 4096K 103 MHz 3.16.07 J2ME/CLDC1.0	aJile aJ100 SaJe 4096K 103 MHz 3.16.07 J2ME/CLDC1.0	Sun eSPOT Green (Beta) 4096K 180 MHz 2006Aug25 J2ME/CLDC1.1	Sun eSPOT Orange (Beta) 4096K 180 MHz 20070118 J2ME/CLDC1.1	Sun eSPOT Purple (Beta) 4096K 180 MHz 20071018 J2ME/CLDC1.1	Sun eSPOT Blue (Beta) 4096K 180 MHz 080609 J2ME/CLDC1.1
32768 element byte array 262144 element accesses	3532	3644	4068	106519	206575	879677	956729	479239	480998	544997	544997
32768 element byte array 262144 element copies (note 5)	748982	655360	1448309	23831272	4443118	32768000	52428800	6241523	8192000	8192000	7943757
16383 element int array 131072 element accesses (note 1)	3454	3031	3891	105109	233224	1008246	1092266	432580	436906	489074	429744
16383 element int array 131072 element copies (note 5)	179550	165913	360087	6241523	1092266	7281777	10922666	1927529	1956298	1956298	1956298
byte add	11534	11142	13542	263504	793650	2702702	2531645	1438848	1449275	1639344	1449275
byte sub	11709	11123	13514	262812	803212	2777777	2597402	1449275	1449275	1639344	1449275
byte mul	11293	10395	12189	241545	561797	1369863	1315789	1428571	1428571	1612903	1428571
byte div	751	6015	5859	105042	561797	1360544	1324503	1190476	1197604	1290322	1169590
int add	11661	11337	13793	90867	641025	2898550	2985074	1724137	1709401	1960784	1724137
int sub	11648	11318	13765	90579	641025	2898550	2985074	1724137	1724137	1960784	1724137
int mul	4283	7877	8228	88183	478468	1587301	1639344	1652892	1639344	1869158	1639344
int div	766	3225	2827	60864	621118	1351351	1379310	1369863	1369863	1481481	1333333
float add	1748	2525	2149	143781	735294	2985074	2941176	250000	251256	254777	1015228
float sub	1597	2220	1871	137268	847457	2666666	2631578	246002	248138	250626	1025641
float mul	864	1836	1528	125234	826446	1408450	1379310	199401	199401	201816	1098901
float div	246	254	201	76804	561797	1379310	1351351	100452	100452	101061	706713
double add	2152	3254	3104	96993	595238	1754385	1724137	226500	226244	228050	508905
double sub	1972	2859	2657	93896	813008	1612903	1587301	215517	215517	217155	502512
double mul	912	1989	1755	64040	377358	581395	574712	193236	193236	194931	501253
double div	252	259	207	34506	311526	574712	568181	70348	70422	70596	215053
string concat	49	45	46	247	879	3711	4051	533	602	659	607
string compare	1888	1320	1756	5390	64935	270270	277777	67340	67114	80000	75187
method calls	5224	5566	5556	14729	271739	847457	854700	706713	699300	1058201	1000000
object creations	572	558	496	3635	17331	36101	41152	86206	85106	58309	66006
Total Loop Executions (TLE)	440	555	492	11557	41106	146536	151269	34553	38115	40969	44148
Price @10	$67	$67	$99?	$130?	$99	$299	$399	$183 Note 6	$183 Note 6	$183 Note 6	100-$250
TLEs per $ (higher is better) Note 4	6.56	8.28	4.97	88.9	415	490	379	189	208	224	177-441
Date of Test								2007Jan29	2007Jan29	2007Nov19	2008Oct28
Result file								eSpotGreen	eSpotOrange	eSpotPurple	eSpotBlue
Website	www.ibutton.com/tini		www.tstik.com	www.imsys.se	www.jstamp.com	www.jstik.com	saje.systronix.com	sunspotworld.com	sunspotworld.com	sunspotworld.com	sunspotworld.com

Notes (see the source code for all the details of each test)

1-TINI arrays can be a max of 64 KBytes in size, so for ints this is a limit of ((2**16)/4) - 1 elements, so I limited all the test arrays to this size.

2- When applications are deployed, TINI executes from NVRAM, SNAP from PSRAM, JStamp, JStik, and SaJe from Flash.

3- Our test setup is still under development, so these results may change in future revisions. Your mileage may vary. Take any benchmark with a grain of salt, two aspirin, or a pint of carbonated malt beverage, as appropriate. No animals were harmed in these tests.

4- TLEs per $ is an almost completely arbitrary attempt to assign some "relative score" to each system. Take this with a grain of salt. Adding more memory to a system increases its price and therefore lowers its score (since the speed did not increase) but its value probably is actually higher since it can run larger programs and/or store more data. Another example: JStik's score is about the same as JStamp, but JStik has much more memory, ethernet, high speed I/O interface, etc.

5- array copies use System.arraycopy which will usually include optimized array access native methods

6- In summer 2008 a kit of three (one base station and two free range) costs $750 or $300 with educational discount. SunSPOTs include an 802.15.4 radio and some basic sensors which are not present on other modules. The base station does not include a battery or eDemo sensor board, just the processor/radio.

Systronix® 939 Edison St, Salt Lake City, Utah, USA 84111
Tel +1-801-534-1017, Fax +1-801-534-1019
contact us Time Zone: MDT (UTC-6)

Java and all Java-based marks are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. and other countries.
Systronix is independent of Sun Microsystems, Inc.
TStik, JStik, JCX, JStamp, JSimm, JDroid, and JRealTime are trademarks of Systronix, Inc.
1-Wire, iButton and TINI are trademarks of Dallas Semiconductor
Simmstick is a trademark of Dontronics
LEGO® is a trademark of Lego A/S, Denmark