Skip to content

Judging Environment

July 5, 2012

The judges’ environment will be a Virtual Machine running Debian 6.0 “squeeze” for i386. The VM will have 1 single core CPU, 1 Gb of RAM and 30Gb virtual disk space. There will be no network connection available for submitted programs at runtime.

UPDATE: We have put together a virtual appliance for the Debian testing release for those who would prefer more up to date packages. You can get this from here

Each entry will be run by a general user (not the super user or any privileged account). We will impose a time limit for each submission, which will be announced along with the contest task.

The exact structure of submissions will be announced along with the contest task. However, you should plan to submit either:

  • A statically linked binary compiled for Debian 6.0 for x86
  • Source files for an interpreted language, together with a README giving a list of standard Debian packages which are required to run your program.
  • (In extreme cases only) Source files, a list of standard Debian packages which are required to compile your program, a build script, and a README explaining why you could not submit a binary. An example of a good reason would be: “I only have access to a Raspberry Pi”. We would really prefer it if you could submit binaries however.

In all cases, you should submit your source code, and we will do our best to run each submitted program. We may contact you by email after the contest has ended if we need help with this.

We have found VirtualBox to be an easy to use tool for installing a Virtual Machine on all of Mac OS X, Windows and Linux. You can find torrents for .iso images for Debian 6.0 here. Full details on how to get Debian can be found here.

We recommend preparing a Virtual Machine in advance. Don’t worry, it’s easier than you think, I (Edwin Brady) managed to install and prepare a Debian VM on Mac OS X in around an hour, despite only just having heard of VirtualBox when I started… If you entered last year, details for setting up the environment will be similar (with the exception that we are using Debian for i386).


From → News

  1. Frank permalink

    Could you please clarify if Debian 6.0.5 for i386, or for x86_64/amd64 is the target environment? Your post mentions x86, but you refer to ‘setting up the environment’ post from last year, which is for amd64/x86_64… Thanks.

  2. aliher permalink

    Will there be a set of preinstalled packages like for example jvm’s similar to last year’s setup or we’ll have to bundle literary everything in submission?

    • We’ll publish a list of pre-installed packages early next week so that you don’t have to list everything (this would be annoying for everyone after all). If anyone needs anything exotic, or any bleeding edge run time environments, then let us know and we’ll do everything we can to be able to run your programs.

  3. “Standard Debian packages” for Squeeze are extremely outdated (like everything in Debian). Our team is going to use Scala 2.9.2 at least (or Scala 2.10-M4 if we get lucky), but the latest Scala package in Squeeze is version 2.7.7 that was released few years ago (eternity in Scala world).

    We may bring our own Scala, as it is available as a runtime library for JVM, but users of other languages will be in less luck. How about Erlang? Or Perl, where latest modules from CPAN are normally required? SBCL? Racket? Functional programming community just loves exotic languages and living on the bleeding edge, so latest versions of everything will be required.

    Last time organizers built judging environment, allowing requests for some more exotic packages and environments to be preinstalled if teams required that. Limiting our options to well-dated packages available in Squeeze is, well, discriminatory. And not in a good sense.

  4. Hello,

    Please can you confirm or correct that:

    * Only one copy of the program will run in a VM at any given time.
    * There will be some swap space available or taken out of that 30GB. Linux tends to handle OOM with no swap partition very badly.
    * The home directory and/or /tmp will be writable at runtime, with a decent chunk of that 30GB available.

    For other Ruby folks, just checked that bundler should provide gems offline nicely. You’ll want to “bundle package –all” before you submit and “bundle install –local” in your setup instructions.



  5. KevinH permalink

    Hi Tim

    i) Yes, we’ll only run one copy in the VM (you might be able to script multiple copies yourself if you need this for concurrency)

    ii) I’ll check – it seems sensible to set up some swap

    iii) Yes, most of the 30GB will be available to the VM. The disk will be cleared after each test run so it probably doesn’t matter whether it’s /tmp or not.

    Note that the VMs will be mapping to an underlying host that may have different physical characteristics to the VM (e.g. more RAM, different CPU type) but we will keep the tests fair.

    To clarify, we will

    o start a new VM instance for each test that we run
    o execute your entry on each set of test data
    o record the result for each set of test data
    o stop the VM after each test and discard its current state

    Each test will run as a completely fresh instance of the VM starting from the same checkpoint.

    Best Wishes

  6. Joel Klein permalink

    Is the VM’s 30GB disk all in a single partition?

  7. startrader permalink

    I will make the same request this year that I did last, in hopes that the 2013 orginizers will take heed. One of the great strengths of functional programming is its ability to do parallel and concurrent tasks cleanly. I feel that with the pervasiveness of multi-core hardware, contest orginizers should provide at least two cores in their VMs. It is a great shame that the last two years thecontest orginizers have not seen fit to allow a demomstration of what is arguabally the most compellin reason for the adoption of functional languages by a larger audience.

    • Hi Startrader,

      We did originally consider a multi-CPU VM, but we decided to restrict to a single CPU. We had three reasons for doing this.

      1) Most entries will be sequential and we would waste half our CPUs if we allocated two physical CPUs to each VM.

      2) If we restrict to a single CPU, we can run more entries at the same time on our physical test machines, so taking advantage of parallelism. We’d rather not reduce the parallelism advantage to us by using a multiple-CPU VM! If we use a 2-CPU VM, we double the TOTAL time to run the tests (unless everyone submits a perfectly parallel example, which is highly unlikely).

      3) Some of the test machines that we expect to use report multiple-CPU VMs as sub-optimal. It’s quite likely that the multi-CPU VMs may run more slowly than single-CPU versions and could cause problems with the host system. Not using these machines restricts our flexibility and would increase the time taken to run the tests.

      So, allowing entrants to use a parallel VM could only increase the total time spent running the tests and this would be a bad thing. The issue here is total throughput for all tasks rather than time-to-complete for one task.

      If we had more CPUs than we could allocate to test cases, then the situation would be different of course, but we’re only using 8-16 core machines and we need to minimise the total amount of time we spend running tests.


      • Chris King permalink

        I understand the testing constraints, but isn’t the point of the ICFP to show off the benefits of functional programming? Languages like Mercury have such beautifully simple parallelizability that it’s a shame not to flaunt those in the name of saving some CPU time.

        Why not consider a compromise?

        Perhaps allow entrants to specify whether their entry is parallelizable or not. This way you can save VM CPU time by running those that aren’t in a single-CPU environment.

        Perhaps only run entries in a multi-CPU environment in the later rounds.

  8. Gold Wingman permalink

    Just spent all day creating a debian “squeeze” 6.0.5 i386 virtualbox vm… and then experimenting with installing various “latest stable” versions of FP and scripting languages. As previous poster (@temerev) mentioned, squeeze has some pretty out-of-date versions. Fortunately everything went pretty smoothly building from sources for the most part. Some things like racket 5.2.1 and Haskell 7.4.1 and Clozure CL 1.8 (x86) binaries installed “out-of-the-box”.

    Others required source builds such as Python 3.2.3, Perl 5.16.0, Ruby 1.9.3-p194, node-v0.8.1, HaskellPlatform 2012.2.0.0, mono 2.10.9 (with which I successfully compiled a simple F# app with mono fsc.exe and also ran mono fsi.exe using F# 2.0.0 Nov 2011 CTP). monodevelop- also builds & runs (but no F# support in this version). Which reminds me… make sure you use “–gui- –readline-” command line options with inferior fsharp fsi in emacs and follow these instructions as well for building/running:

    Had to install a bunch of apt-get packages (almost 100) to get the above to build from sources. Also installed Iced Tea 1.8.13 (apt-get install openjdk-6-jdk) as well as Oracle JDK 1.6.0_33 and Scala 2.9.2 and tried the latest leiningen 2.0.0preview7 with clojure 1.4.0. So far I don’t see any show stoppers if you invest the effort to build from sources.

    Squeeze comes with C/C++ 4.4.5 and sbcl 1.0.40 and clisp 2.48 and ocaml 3.11.2 which are slightly out of date but probably usable, so I didn’t bother updating these from sources.

    Would be glad to provide/post build instructions and/or package lists to the organizers or others if they would be interested. It would probably be useful to put most/all of the above in the judges build VM based on my re-reading of what people used for last year’s contest. In particular, I am guessing Mono + F# and Haskell are going to be very popular requests. Unfortunately I can’t provide a torrent of the VM I built since some of the above are not redistributable, plus I built my VM with 16GB instead of 30GB disk (oops!)

    Why did I do this? I haven’t decided which language to use just yet, which may depend on the task and my mood of the team. I also wanted to see what options were at my disposal given that “squeeze” is somewhat behind the curve for various language binaries.

    Thought I would share my findings so far, in case it was helpful to anyone.

  9. Gold Wingman permalink

    *** Update for the moderator about my prior posting from last night ***

    I read some more of the replies above, and saw that someone was asking about erlang, and I also happened to think that since someone asked for node.js that somebody may want to use Google Go as well… So I am building erlang R15B01… but don’t have an easy way to test it really since I’ve only ever written about 1000 lines of erlang in my lifetime.

    I also tried the latest go 1.0.2 binary package and it runs fine on squeeze and produced statically linked executables using go tool 8g program.go go tool 8l -o program program.go… which was nice to discover since I have been hearing a lot of buzz about Go lately.

    I also cloned and gparted-ed my VM disk to 30GB so it meets the contest guidelines. I was thinking about changing my username to ‘fpuser’ and then I would have a more generic VM, and I could remove those few proprietary things and give you a copy of my VM to download (and would be happy to contribute some build instructions as mentioned before).

    Of course switching to a later debian “testing” or a derivative such as ubuntu 12.04 LTS would give the FP’ers the latest versions of most things. The ruby and python and perl folks would probably still want their latest & greatest built and installed.

    Anyway, I have my VM built with a bunch of languages and I’m ready to go as long as you don’t change distros between now and next week. Let me know if you want access to any of my stuff.

    • Gold Wingman permalink

      One note of clarification. My offer to provide access to a copy of the stripped-down open-source-only-tools VM was more for the contest organizers to see what packages would (or not) build & install on squeeze successfully. I am sorry that I can’t provide pre-built VM to all of the contest participants. I don’t have the bandwidth. Plus it probably violates some licensing issue.

      I am still willing to post my build instructions somewhere if folks would find them useful.

      I am open to suggestions regarding where to post them? They are just a bunch of text files containing commands you can copy-and-paste. Not even real scripts at this point. Maybe in a github public repo of some sort? Bitbucket? Other ideas? I could setup a facebook “build your ICFP VM” page as well I guess… since they are sponsoring the event they could host some of the content… ahem.

      I figured I might as well put my extensive sys-admin “meta” skills to good use this weekend while I am waiting for contest to start 🙂

      • Hi Gold,

        Could you send your email address to me please: kh at cs dot st-andrews dot ac dot uk – this will help us get a good VM set up for those at the bleeding edge


  10. sergio garcia permalink

    Any limitations on the size of the executables that we can submit? (SBCL can produce stand-alone executables, but they are quite big)

    • Hi Sergio,

      Logically no, but a multi-gigabyte binary is likely to be very slow to load! There will be some limit on RAM size in the VM (probably in the 384MB-1GB range).


  11. Gold Wingman permalink

    To F# mono contest participants: please read this and reply if you want to vote for F# config on judges VM…

    I wrote up some instructions to contribute to the judges for building latest stable mono 2.10.9 and installing F# CTP November 2010 binaries in their VM environment. But then I installed latest mono MDK for Mac OSX this morning and I noticed they bundle fsharp-cc126f2, for which I found the sources on the mono site. So I am working on additional instructions for building/installing the fsharp-cc126f2… which went pretty smoothly, BTW.

    The Question:
    Which would be better to ask for the judges to install, given that only one set of FSharp.*.dll’s can be registered in the mono gac at any given time if I understand correctly?

    1. FS CTP November 2010 (binaries) – probably most friendly for VS2010 developers?

    2. (sources dated Aug 2011) – probably most friendly for MacOS developers using latest mono MDK installer?

    3. latest – which I see was updated as recently as 2 months ago. When I build this one it reports version — which is a version that I have never heard of since I have only ever used and — both on Windows. I assume this one would not be as friendly for either VS2010 or MacOS developers trying to deploy F# binaries onto the judges VM???

    Thanks for your input…!

    • Gold Wingman permalink

      Clarification: option #3 above really only reports version during build process. After installing fsharpc reports:

      Microsoft F# 2.0 Compiler build (private)
      Copyright (c) 2002-2011 Microsoft…

    • Giop permalink

      Hi Gold W

      one vote for f#!

      now i try to install it on my VM then, if i need help, i will post again crying for infomations 🙂

    • Frank permalink

      After encountering these issues in last year’s contest, I eventually figured out how to use mono’s mkbundle to create a self-contained executable that doesn’t depend on the Mono runtime being installed. So, you can use whatever debian mono, F#/ other CLR-based language, and utils (eg F# PowerPack) you want, and not have to worry about versions / contents of GAC on the contest VM…

      BTW, it seems that “truly” statically linked executables are more-or-less a thing of the past, based on the warnings I was getting when I was setting this up.

      (I do yearn for the old days when ICFP contest submissions were DATA, not binaries…)

      Heres a long-winded makefile that’ll build an F# prog, and make a standalone version. Use at your own risk/ no warranties etc 🙂

      # Helper makefile for building standalone F#/Mono binaries, by Frank
      # ——————————————————————————-
      # Choose tool suite.
      # If using the F# Power Pack library, must use the .net 2 versions of the commands
      # (which are simply copies of the scripts, but patched to use the lib/mono/2.0 version of .exe)




      hello.fs \

      #REFS=–reference:FSharp.PowerPack.dll –reference:System.Xml.dll –reference:System.Xml.Linq.dll

      # ——————————————————————————-
      # Default to building just .net exe; use ‘standalone’ to create static-linked exe

      all: $(APPNAME).exe

      standalone: $(APPNAME)

      # ——————————————————————————-
      # Produce a .net executable

      $(APPNAME).exe: $(SRCS)
      $(FSC) $(FFLAGS) $(REFS) $(SRCS) –out:$(APPNAME).exe

      # ——————————————————————————-
      # Produce a stand-alone executable, independent of the mono installation.
      # There are some issues with letting mkbundle directly run the compiler, some
      # bugs, and also it doesn’t 100% create a ‘static-linked’ executable, which
      # I need for some enviromnents (eg ICFP contest submission).
      # mkbundle has some inconsistencies we need to work around, but it works…
      # eg: we can’t stop it running the assembler, but we can stop it running the compiler.

      # For reference, heres an example of how mkbundle runs cc:
      # cc -static -Wall `pkg-config –cflags mono-2` __bundle-main.c -lz \
      # `pkg-config –libs-only-L mono-2` -Wl,-Bstatic -lmono-2.0 \
      # -Wl,-Bdynamic `pkg-config –libs-only-l mono-2 | sed -e “s/\-lmono-2.0 //”` \
      # __bundle-data.o -o $(APPNAME)
      # NOTE: With a static executable, beware of LGPL issues with libraries etc.
      # The linux build reports the following linker warning for several functions:
      # “warning: Using ‘X’ in statically linked applications requires at runtime the
      # shared libraries from the glibc version used for linking”
      # where X includes:
      # dlopen, getgrgid_r, getgrnam_r, getpwnam_r, getpwuid_r, socket_transport_connect,
      # getaddrinfo, gethostbyname, convert_sockopt_level_and_name, getprotobyname


      MB_CFLAGS=$(shell pkg-config –cflags mono-2)
      MB_LFLAGS=$(shell pkg-config –libs-only-L mono-2)
      MB_LIBS=$(shell pkg-config –libs-only-l mono-2)
      #MB_LIBS=$(shell pkg-config –libs-only-l mono-2 | sed -e “s/\-lmono-2.0 //”)

      ifeq ($(shell uname -s),Darwin)
      # OSX doesn’t support statically linking an executable!
      # Ssee:
      # and:

      # OSX Mono (up to 2.11.2) is 32-bit, on 64-bit host
      # Adjust ‘commands’, not ‘flags’
      MB_AS+=-arch i386
      MB_CC+=-arch i386

      ifeq ($(MB_COMPRESS), 1)

      ifeq ($(MB_STATIC), 1)
      # Try building “truly static” linux executable. Check with ‘ldd ‘,
      # and ‘file ‘

      $(APPNAME): $(APPNAME).exe
      AS=”$(MB_AS)” $(MB) $(MB_FLAGS) -c –keeptemp \
      -o $(MB_TMP_MAIN_C) -oo $(MB_TMP_DATA_O) $(APPNAME).exe
      rm temp.s
      -o $(APPNAME)
      strip $(APPNAME)
      rm $(MB_TMP_MAIN_C) $(MB_TMP_DATA_O)

      # ——————————————————————————-
      # Clean up local build, and also VStudio build

      [ ! -f $(APPNAME).exe ] || rm $(APPNAME).exe
      [ ! -f $(APPNAME) ] || rm $(APPNAME)
      [ ! -f $(MB_TMP_MAIN_C) ] || rm $(MB_TMP_MAIN_C)
      [ ! -f $(MB_TMP_DATA_O) ] || rm $(MB_TMP_DATA_O)
      [ ! -d $(APPNAME).dSYM ] || rm -rf $(APPNAME).dSYM
      [ ! -d bin ] || rm -rf bin
      [ ! -d obj ] || rm -rf obj

  12. Hi!
    I’d whant to use “The ECLiPSe Constraint Programming System”
    I’ts kind of scripting language (can be compiled to bytecode, but you still need ECLiPSe VM).
    They have Linux i386 binaries, but no Debian packages (and of course no “standard Debian packages”).
    What are my options?

    • If the binary works on the VM, then you can include that as part of your submission (assuming their licence allows it of course – it seems it’s okay). If necessary, we can install the binary in the testing environment ourselves.

  13. Do I really need all 50 something CD iso’s and update CD iso’s?

    Also for future contests. I am one of those that would like to use current versions like Perl 5.16.0. Along with current modules.

    • No, you just need a netinst CD image to install Debian base system into your VM. Afterwards you can install additional packages like your favorite programming languages by apt-get.

    • Phil Darnowsky permalink

      I too had a small heart attack when I saw that, but then used the net-installation disk image: debian-6.0.5-i386-netinst.iso. That has a minimal set of packages that will get you on the network and allow you to download a full system.

    • Kevin Hammond permalink

      Jeremy, we’re putting together a Virtual Appliance for the Debian Testing release. This should avoid having to install too much and still allow you to be reasonably up to date (it has perl 5.14.2) – obviously if you want the very latest versions of some packages you may have to install them yourself, but it should simplify the process.

      If you want to try this, you can access it at:

      • Gold Wingman permalink

        Is it OK to mention that the username/password for the user appears to be ‘icfp’ and that “sudo su -” gets you root access in the VM? Confirmed that perl is version 5.14.2 and Python is 2.7.3rc2 and very few extra CPAN or Python modules are pre-installed. Looks like a pretty slim Debian 7alpha1 install with no frills like X11. Didn’t see ruby, sbcl, clisp, node, JDK, scala, mono, F#, g++, ocaml, haskell or other binary packages pre-installed… so contestants would probably need to “bundle your own” as mentioned by others above. The lack of frills will work to our advantage since most of the 1GB of RAM is still available for our programs to run.

        I have a couple of question to the judges:
        1. Will we be allowed to submit an ./install script as well as a ./run script with our submissions much like last year?

        2. If the contest rankings are based on CPU time (or it comes down to “fastest” runtime being a tie breaker)… can we assume that the ./install script execution time be excluded from the project run time?

        Thanks for the preview of the VM… much better than “flying blind.”

      • 1. Yes, you will be able to submit an install script, which will be run exactly once.
        2. The install script will not count towards the evaluation.

      • Christoph Breitkopf permalink

        Now I’m getting confused – which Debian version should we target: 6.0.5 or 7alpha1?

      • Either is fine, as long as you’re clear which in your submission (we’ll say how to do this…).

      • Gold Wingman permalink

        Debian “testing” wheezy only appears to have 200MB of /tmp space.

        You may want to add the following to your /etc/fstab to avoid strange crashing problems due to running out of /tmp space.

        tmpfs /tmp tmpfs nodev,nosuid,relatime,size=2G 0 0

        for more details. Editing the /etc/default/tmpfs also works, but is deprecated in preference of the /etc/fstab entry if I understand “man rcS” correctly.

        This is apparently a well-know problem with debian “testing”.

      • George Bullis permalink

        Regarding the tmpfs change – are the judges going to make this change as well?

      • Kevin Hammond permalink

        Yes, I’ve made the change and I’m trying to get the new image uploaded (not easy when you’re travelling!)

  14. Igor permalink

    Supposing I’m going to use C++, is there any good reason to install a full-blown VM? For I’d rather prefer to set up a cross compiler and use it to compile the binaries for the submission. Would that be enough for this contest?

    • That would be fine, though we’d still recommend you test it in the VM, just in case.

      I expect in most cases people will do develop in the environments they’re accustomed to then just use the VM for testing the submission.

  15. Johan Nielsen permalink

    @Gold Wingman: Amazing work you’ve done for the rest of us! I would very much build instructions for getting the most recent F# working on the VM. I did an attempt yesterday but seemed to fail…

    • Gold Wingman permalink

      I posted some mono + F# build instructions for debian squeeze 6.0.5 that I had worked out earlier this week:

      I haven’t had time to update the instructions to 7 alpha 1… I plan to work out those details tonight on a copy of the judges VM. If I have success and come up with anything useful I will share.

      I make no guarantees that any of this will work but it “worked for me once on squeeze” 🙂

      Frank’s mkbundle makefile approach sounds like a great way to go too. I haven’t tried it yet.

      Feedback is welcome as it may save me some time getting my own install done this evening when I get home from work.

  16. Joel Croteau permalink

    Can you clarify exactly what instruction architecture the virtual machine will be running? Is it pure i386 or will it support i686 or later instruction sets? Will extentions such as MMX and SSE be available?

    • Kevin Hammond permalink

      Dear Joel,

      We’re using VirtualBox. This doesn’t provide a specific virtual processor architecture emulation – rather it uses the underlying host to actually execute the instructions.

      The main test machine for the VM will be an Intel Xeon 5355 with 8 physical cores.

      /proc/cpuinfo shows the following flags, which I think should give you an idea of the instruction set that’s supported.

      flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat
      pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc
      arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2
      ssse3 cx16 xtpr pdcm dca lahf_lm dts tpr_shadow

      We may also test on recent Intel-based laptops and desktops (so 686 instructions should be available).

      I hope that helps!

  17. Eric permalink

    Will it be possible to receive feedback on whether our submissions are valid at the time of submission, e.g., that it passed some trivial test case (or some similar simple test)?

    I only have my personal machine (x86, gentoo), which lacks the computational resources for a VM, so I will be submitting a executable compiled for my machine and hope it executes in Debian. It would be helpful to know if there is a problem with the submission before the end of the contest.

    • Kevin Hammond permalink

      If you share a binary with us before the contest end and send us an email, we can probably check this for you (we won’t have time to do it for everyone, though!). If the binary doesn’t work, you can submit sources.

  18. What should I write in ./PACKAGES-TESTING file to get sun-java6-jre package?

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: