"This test is ridiculous..."

For the Compleat Fan
Locked
xytroxon
Posts: 296
Joined: Tue Nov 06, 2007 3:59 pm
Contact:

"This test is ridiculous..."

Post by xytroxon »

Found this interesting programming language comparison being disgusted er. "discussed" on the Lua newsgroup.

Perl, Python, Ruby, PHP, C, C++, Lua, tcl, javascript and Java comparison
http://raid6.com.au/~onlyjob/posts/arena/

This comparison consists of three parts:
Part 1: Speed.
Part 2: Memory usage.
Part 3: Language features.

Roberto Ierusalimschy, the the co-creator of "slowpoke" Lua, was not amused by the results...
From: Roberto Ierusalimschy <roberto <at> inf.puc-rio.br>
Subject: Re: Perl, Python, Ruby, PHP, C, C++, Lua, tcl, javascript and Java comparison
Newsgroups: gmane.comp.lang.lua.general
Date: 2013-07-15 17:24:28 GMT (4 hours and 17 minutes ago)

> Maybe this should be of interest:
>
> http://raid6.com.au/~onlyjob/posts/arena/

This test is ridiculous. Whoever wrote the code does not know how
to program in Lua (or in Java or in Javascript).

-- Roberto
I would like to hear Lutz's expert opinion on the test code methods.

-- xytroxon
"Many computers can print only capital letters, so we shall not use lowercase letters."
-- Let's Talk Lisp (c) 1976

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: "This test is ridiculous..."

Post by Lutz »

Benchmarking is a complicated business. These people here http://benchmarksgame.alioth.debian.org/ really understand this.

What the people here http://raid6.com.au/~onlyjob/posts/arena/ do well, is comparing one specific algorithm over different languages. They measure not only speed but also problem-size and memory usage.

But their benchmark doesn't tell us much about the capabilities of the different languages in general. Different kind of problems have different kind of solutions in different programming languages. The alioth.debian.org people write about this somewhere on there site and for some benchmarks they allow language specific solutions and in other problems they ask for the solution to be as close as possible to the algorithm they are prescribing.

When I did these: http://www.newlisp.org/benchmarks/ , I remember that just moving from one platform or OS to another, results could look totally different. Even when staying on the same hardware, e.g. testing on OSX versus, Windows XP, versus Linux on the same machine results can vary dramatically.

They should do that same analysis on a bigger variety of problems and not only compare the same algorithm but also allow language specific solutions for the same problems. They should also investigate different platforms and OSs. Of course that is a lot of work.

xytroxon
Posts: 296
Joined: Tue Nov 06, 2007 3:59 pm
Contact:

Re: "This test is ridiculous..."

Post by xytroxon »

Thank you Lutz.

While impressive looking with source codes, tables, and graphs, this line gave me pause:
Unexpected result: somehow Perl5 managed to finish faster than C. This came as unforeseen surprise which I found difficult to explain.
Or to believe... (Unless you are a Perl fanboy writing questionably valid tests in other languages ;)

-- xytroxon
"Many computers can print only capital letters, so we shall not use lowercase letters."
-- Let's Talk Lisp (c) 1976

Locked