ScriptParametricStudy

This page explains how to use yade-$version-multi scripts to run parametric studies relatively easily.

Example: scripts/multi.py

This example can be found in scripts/multi.py and scripts/multi.table.

Let's say I want to study the influence of parameters that I call "density" and "initialVelocity" on the position of a sphere after a few iterations. I create a text file, called e.g. multi.table. The first line names parameters (columns) and every line is different set of those parameters. Special parameter is "description" which can be used to put some human-readble comment (blank lines are skipped, comments using '#' allowed)

description density initial speed
reference   2400    10
hi_v        2400    20
lo_v        2400     5
hi_rho      5000    10
hi_rho_v    5000    20
hi_rh0_lo_v 5000     5

In the "multi.py" file, we need to read those parameters at one point; the name of the table and line number will be passed by the yade-multi script though the environment variable PARAM_TABLE. <source lang="python"> from yade import utils utils.readParamsFromTable(gravity=-9.81,density=2400,initialSpeed=20,noTableOk=True) # noTableOk means we will just use default values if no table is supplied

2. now we have global variables gravity, density, initialSpeed.
3. They have values from the table or, if not defined there, they are at their defaults passed to readParamsFromTable

print gravity,density,initialSpeed

1. ...

</source> Then I set up the simulation as usual, using the parameters I have already defined. Because I want to compare results, I will add a line at the end of the simulation that will put some interesting values to a file: <source lang="python"> file('multi.out','a').write('%s %g %g %g %g\n'%(o.tags['description'],gravity,density,initialSpeed,o.bodies[1].phys.pos[1])) </source> Every simulation will add a new line to this file.

Now I can run scene.py using the parameters in gravity.table with the following command:

yade-trunk-multi multi.table multi.py

Let's see what happens:

falx:~/yade/trunk/scripts/$ yade-trunk-multi multi.table multi.py
Will run `/usr/local/bin/yade-trunk' on `multi.py' with nice value 10, output redirected to `multi.%.log', 2 jobs at a time.
Will use table `multi.table', with available lines 1, 2, 3, 4, 5, 6.
Will use lines  1, 2, 3, 4, 5, 6.
Job summary:
   #0: PARAM_TABLE=multi.table:1 nice -n 10 /usr/local/bin/yade-trunk -N PythonUI -- -n -x multi.py > multi.1.log 2>&1
   #1: PARAM_TABLE=multi.table:2 nice -n 10 /usr/local/bin/yade-trunk -N PythonUI -- -n -x multi.py > multi.2.log 2>&1
   #2: PARAM_TABLE=multi.table:3 nice -n 10 /usr/local/bin/yade-trunk -N PythonUI -- -n -x multi.py > multi.3.log 2>&1
   #3: PARAM_TABLE=multi.table:4 nice -n 10 /usr/local/bin/yade-trunk -N PythonUI -- -n -x multi.py > multi.4.log 2>&1
   #4: PARAM_TABLE=multi.table:5 nice -n 10 /usr/local/bin/yade-trunk -N PythonUI -- -n -x multi.py > multi.5.log 2>&1
   #5: PARAM_TABLE=multi.table:6 nice -n 10 /usr/local/bin/yade-trunk -N PythonUI -- -n -x multi.py > multi.6.log 2>&1
#0 started on Tue Oct 28 10:56:05 2008
#1 started on Tue Oct 28 10:56:05 2008
#1 done    (exit status 0), duration 00:00:06
#2 started on Tue Oct 28 10:56:11 2008
#0 done    (exit status 0), duration 00:00:06
#3 started on Tue Oct 28 10:56:12 2008
#2 done    (exit status 0), duration 00:00:06
#4 started on Tue Oct 28 10:56:17 2008
#3 done    (exit status 0), duration 00:00:06
#5 started on Tue Oct 28 10:56:18 2008
#4 done    (exit status 0), duration 00:00:06
#5 done    (exit status 0), duration 00:00:06
All jobs finished, bye.

Yade-multi options

$ yade-trunk-multi --help

Usage: yade-trunk-multi [options] TABLE SIMULATION.py

  yade-trunk-multi runs yade simulation multiple times with different parameters.
  See http://yade.wikia.com/wiki/ScriptParametricStudy for details.

Options:
  -h, --help            show this help message and exit
  -j NUM                Maximum number of simultaneous jobs to run (default: number of cores, i.e. 2)
  --log=FORMAT          Format of log files -- must contain a %, which will be replaced by line number (default: SIMULATION.%.log)
  -l LIST, --lines=LIST Lines of TABLE to use, in the format 2,3-5,8,11-13 (default: all available lines in TABLE)
  --nice=NICE           Nice value of spawned jobs (default: 10)
  --executable=FILE     Name of the program to run (default: /usr/local/bin/yade-trunk)

Inspecting output

All output from yade goes to file; those were named multi.%.log in our case, where % is replaced by line number.

$ cat multi.1.log
INFO  core/yade.cpp:176 main: Loading /home/vaclav/.yade-trunk/logging.conf
INFO  core/yade.cpp:226 main: Loading /home/vaclav/.yade-trunk/preferences.xml
INFO  core/yade.cpp:228 main: Loading plugins
Python TCP server listening on localhost:9001, auth cookie is `syucdk'
Running script multi.py
-9.81 2400 10

You can use the command

$ tail -f multi.*.log

to see output of all instances as it is being updated.

Connecting to a running instance over network

To connect to yade instance, you need to know its port and cookie ("password"), which are both written in the log file:

$ grep TCP multi.1.log
Python TCP server listening on localhost:9000, auth cookie is `aksdys'
$ telnet localhost 9000
Trying 127.0.0.1...
Connected to falx.
Escape character is '^]'.
Enter auth cookie: aksdys
__   __    ____                 __  _____ ____ ____  
\ \ / /_ _|  _ \  ___    ___   / / |_   _/ ___|  _ \ 
 \ V / _` | | | |/ _ \  / _ \ / /    | || |   | |_) |
  | | (_| | |_| |  __/ | (_) / /     | || |___|  __/ 
  |_|\__,_|____/ \___|  \___/_/      |_| \____|_|     

(connected from 127.0.0.1:53462)
>>> o=Omega() 

>>> o.iter
134676
>>> o.stopAtIter
3000000
>>> o.bodies[1].phys.pos[2]
1.9999975344780854
>>> exit     # or ^-D

Connecting from remote computers (i.e. not from localhost) is not allowed for security reasons. You can always use regular ssh to the machine running yade and then connect to yade over telnet locally.

Caveats

Overwriting files

It is your responsibility to ensure that running instances will not overwrite each other's files (saved simulations, sqlite database etc). You can use

• Omega().tags['id'] to uniquely identify any yade instance (it contains second-precision time and process ID)
• Omega().tags['line'] to get the line number; if not using parameter table, it will be "l!"

<source lang="python">

o.save('final-'+o.tags['id']+'.xml.bz2')  # INFO  core/Omega.cpp:268 saveSimulation: Saving file final-20081028T074308p3704.xml.bz2
# or
o.save('final-'+o.tags['line']+'.xml.bz2') # INFO  core/Omega.cpp:268 saveSimulation: Saving file final-l!.xml.bz2

</source>

Saving temporary simulation state

If you only want to save intermediary result to be loaded in the same session later, use saveTmp/loadTmp, which saves in memory only, thus avoiding the danger of overwriting a file accidentally. <source lang="python">

o.saveTmp('initial')              # INFO  core/Omega.cpp:268 saveSimulation: Saving file :memory:initial
#... do some calculation here
# now load the initial state again
o.loadTmp('initial)               # INFO  core/Omega.cpp:232 loadSimulation: Loading file :memory:initial

</source>