Tiny Benchmark Test

[Aurel]

Mike
Yes perhaps...
BUT i have tested this programs 3 times on my both old computers
( hehe i love retro stuff) 
and results are the same...second more or second less...
by the way Borland BCC is little bit faster than gcc   .

I agree with John ,this benchmarks are really little bit stupid.

[Mike Lobanovsky]

Who cares if one BASIC runs a second faster than the other?

Where did you see me comparing BASIC's here, John? I was benchmarking gcc -O3, gcc w/o optimization and FBSL's DynC where all the three compilers are 100% ANSI C standard compliant. Their execution speed is a direct criterion of the quality of resultant machine code they generate. I'm also appending a screenshot of gcc optimized to compile the same script for the smallest exe size possible (-Os option).

I posted my original message in response to Ed Davis' C-related inquiry. And I have added the O2 snippet only in response to Aurel's message to give him an idea of how the O2 variant would look on my machine.

OTOH both O2 and FBSL are much more than just BASIC's. FBSL already has three distinct languages incorporated and interacting while O2 is nearing the same point with its beneficial capability of reading much of assembly and C as direct BASIC inlines.

I am not in the habit of comparing Atari BASIC against Sinclair BASIC. I'm not a necro-monger. I believe that future belongs to such complete development environments as O2 or FBSL or Terra exactly thanks to their omnivorous nature. And I like to know exactly where I am with my FBSL visions/technologies/implementations amongst my peers. I do not want to end up my days the way that unfortunate newly-appeared ClipperBASIC author did.

P.S. And if what I said above still sounds stupid to you, Aurel, then you may call me a fool. Time will show.

.

[Aurel]

Oh Mike...what a heck is wrong with you ?
Where you see that i say that your words are stupid 
I just think that all this to much benchmarking is little bit stupid and boring..ok 

Hm...about FBSL...i think that you know where you are.
Sorry man ..from technical point of view FBSL is ok and fine   but
what is all that worth if you don't have users.
And as we can see there is no interest for (not only ) FBSL ,i dont see to much
activity on FBSL forum and why FBSL is not much more popular?
And as you can see situation is almost the same anywhere ...
if you have 3-5 active users - what is that ? ..nothing
Wishes are one thing and situation is another thing.
but who care ...forget 

Ha clipperBasic is a what ?
What he espect on a rubbish site like is basicprogramming.org with
users ( read losers) oriented to *unix .
And to be honest ,what is clipperBasic ...unfinished attempt  which don't have nothing
with basic,

[Charles Pegge]

I remember Clipper as a DbaseIII compiler. It inspired me to write a replacement for DbaseIII, and avoid costly licences for the company network, where I was working.

Is there any connection with Clipper Basic?

PS:
After looking at this BenchMark, I decided to have another crack at optimising OxygenBasic's conditionals. I think I'm making progress this time.

[Aurel]

After looking at this BenchMark, I decided to have another crack at optimising OxygenBasic's conditionals. I think I'm making progress this time.

That would be good Charles...

[Mike Lobanovsky]

Hello community,

To finish off another round of "the integer intensive benchmarks" so popular at BP.org, here are my results for some of the native-code and JIT compilers as well as bytecode and pure interpreters that I checked out under XP Sp3 on the hardware depicted in my signature below.

The reason why I opted for my own table is that I couldn't confirm some of the data presented at BP.org. For instance, my results for such items as GCC, FreeBASIC or Euphoria differ significantly from that data though some other items are in harmony with it. Also, I thought it would be interesting for some of our members to see where FBSL** or thinBasic or Scriba** (that's an interpretative reincarnation of ScriptBASIC) should've stood among their competitors.

The zip attached at the very bottom of this message contains the respective scripts and some precompiled executables where applicable, so that you could try and verify the results on your own HW yourselves.
______________________________
** FBSL's BASIC and Scriba are bytecode interpreters with typeless (a.k.a. Variant-type) variables. They are included in the Interpreters category because 99% of the benchmark code runs in quadruply nested loops where byte coding doesn't help due to difficulties with efficient garbage collection.

Code: [Select]

Native Code Compilers
=====================

MS VC12 with full
optimization (-Ox) 1.094 sec Proprietary

GCC v4.3.3 with full
optimization (-O3) 1.312 sec Open source

FreeBASIC 3.209 sec Open source

MS VB6 console with
full optimization 62.00 sec Proprietary


Just-In-Time Compilers
======================

FBSL Dynamic C 3.203 sec Freeware,
closed source

Oxygen Basic 3.593 sec Open source

LuaJIT 3.890 sec Open source


Bytecode Interpreters
=====================

Euphoria 43 sec Open source

Lua 77 sec Open source

MS VB6 console p-code 147 sec Proprietary


     Interpreters
     ============

thinBasic 543 sec Freeware,
closed source

FBSL BASIC** 564 sec Freeware,
closed source

ScriptBASIC 625 sec Open source
(Scriba**)

LB Booster 1375 sec Freeware,
closed source

Native code compilers:




Just-in-time compilers:




Bytecode interpreters:




Interpreters:



.

[Charles Pegge]

Very interesting results, Mike. Interpreters always look bad on integer tests. When doing trig functions, the performance difference is far smaller.

I am intrigued as to how VC12 squeezes the time down to 1 second! With a few simple optimisations to the inner loop, I can get it down from 3.6 to about 2 secs (in terms of your PC performance).

Even hand coding the inner loop in assembler, I can't push it below 2 seconds!

[JRS]

Script BASIC is a P-CODE interpreter. SB surely takes a hit in repetitive loop benchmarks as its variables are like variants. (objects)

Even hand coding the inner loop in assembler, I can't push it below 2 seconds!

I don't know what to say. I'm crushed! 

[Ed Davis]

my results for such items as GCC, FreeBASIC or Euphoria differ significantly from that data

Running some .exe's from your supplied archive on my machine:


intmandelGCC:           1.342 seconds (yours was 1.312)
intmandelFB:            3.377 seconds (yours was 3.209)


My FreeBasic script used '/' for division, along with int().  Changing those, and now my version comes in at 3.36, which is almost the same. 

Running the Euphoria script from your archive:



Definitely no idea why these are so different, from 43 to 24 seconds is a big difference.

Compiling your version of intmandelGCC with a later version of gcc (4.8.1):



Here are the specs for my machine:

Windows 7, Service Pack 1, 64-bit
Intel Core i7-3720QM CPU @2.60GHz
16.0 GB (15.9 usable)

[Charles Pegge]

Could the optimised C code be running in 2 concurrent threads?

[JRS]

Nope. Single thread.

There is no difference in speed between VC12 32 bit and a 64 bit version of the benchmark.

[Mike Lobanovsky]

Script BASIC is a P-CODE interpreter. SB surely takes a hit in repetitive loop benchmarks as its variables are like variants. (objects) I don't know what to say. I'm crushed!

You shouldn't be so upset with that, John. FBSL's BASIC is also a bytecode interpreter (thinBasic isn't - it's a pure interpreter) and totally Variant-based; nonetheless it suffers similar heavy performance penalties in loop benchmarks. That's why I made a special reservation about FBSL by classifying it under Interpreters rather than Bytecode Interpreters for this particular benchmark. Perhaps we should add this remark also to Scriba. That's a natural trade-off for the benefits of having typeless variables.

The nature of a Variant variable may change any time within the loop. It may become a string or even a class instance that need to be freed or destroyed before the bunch of them pollute the process stack and heap to the extent of total exhaustion. Same goes about recursion. Where strong data type interpreters enjoy temp var allocation on the function stack that's cleared automatically on function exit, Variants need an explicit malloc'ing and freeing of heap memory chunks. Thus, a simple loop becomes a nightmare for the garbage collector that does the cleanup job plus memory defragmentation whenever possible. Garbage collection adds dozens of auxiliary function calls that slow down the loop dramatically.

OTOH Lua is also Variant-based and is exceptionally fast at that. So the problem can be solved somehow, however I don't know how to re-implement Lua's methodology in the existing implementation of FBSL's BASIC without major rework of its engine from the ground up.

[Mike Lobanovsky]

Thanks for your response, Ed.

My FreeBasic script used '/' for division, along with int().  Changing those, and now my version comes in at 3.36, which is almost the same.

Oh yes, that explains pretty much the difference in our results.

Definitely no idea why these are so different, from 43 to 24 seconds is a big difference.

I'm using Euphoria's latest build, v4.1 for Windows from their repository. Is that what you've used for the test?

Could the optimised C code be running in 2 concurrent threads?

No Charles, definitely not although Euphoria does seem to be employing some sort of code optimization. I think the difference may also be caused by the architectural differences of our CPU's. Ed's machine is i7-based while mine is built around an i5.

Another point: the script takes about 1 second or more to compile while the console is already on the screen before it starts to execute. I don't think their parser is so slow. I'm rather inclined to attribute the lag to some heavy optimization which may be the reason why Euphoria is the fastest bytecode interpreter of all indie implementations.

Compiling your version of intmandelGCC with a later version of gcc (4.8.1):

Thanks for pointing this out, Ed. I've been using GCC v4.3.3 to compile FBSL for many years because its output code proved to be up to 45% faster than later builds. I'm using a post-build script to fix the crooked PE headers and garbage that v4.3.3 generates in its output binaries under Windows. Evidently it's high time for me to try out v4.8.1 against the FBSL sources. Perhaps I'll be able to enjoy a 30% boost there too.

[Charles Pegge]

Mike, could you time this one on your PC. I think it goes almost twice the speed of mine, but I would like to see how fast it goes on yours with my optimisations. (3.95 secs on my PC)

The code is very raw and messy. Don't look at it if you are squeamish


A few ideas for interpreter optimisation: You may already have these:

String pools for efficiently recycling small strings, and mass allocation/disallocation.

Dynamic tokenisation of source code. Progressively convert source into tokens.

Jump tables for efficiently converting operator tokens into handler calls.



.

[Mike Lobanovsky]

The code is very raw and messy. Don't look at it if you are squeamish

I love any code that runs so fast as this!

I think it goes almost twice the speed of mine, but I would like to see how fast it goes on yours with my optimisations. (3.95 secs on my PC)

AMAZING! Please see the snapshot below taken on fresh reboot into XP Sp3. And do not try to beat Java and .NET. They are cheaters: they run the JIT source that's been heavily optimized when precompiled slowly for the first time and written into a disk file.

String pools for efficiently recycling small strings, and mass allocation/disallocation.

FBSL uses separate pool allocators for strings, static Variant variables,  temporary Variant variables, and class instances.

Dynamic tokenisation of source code. Progressively convert source into tokens.

FBSL uses three ^^and a half^^ successive levels of source code processing at app start:
-- Level 1: preprocessor (separate for BASIC/DynAsm and DynC)
-- Level 2: recursive descent parser and tokenizer for BASIC, JIT for DynAsm and DynC
-- Level 3: BASIC bytecode compiler
-- Level 3.5: BASIC bytecode optimizer.

Jump tables for efficiently converting operator tokens into handler calls.

That's what I'm currently busy with having read the article that Ed pointed Aurel to recently at BP.org.

.