Imported from Bitbucket issue 7, created by apdavison on 06-26-2010 at 20:44, last modified: 06-30-2010 at 13:52

I am storing my simulation variables in a Python dict. For readability, the parameter file separates each variable on a new line. When trying to run

{{{
smt run --executable=python --main=main.py default_params.py
}}}

I get

{{{
File "/usr/local/bin/smt", line 30, in
main(sys.argv[2:])
File "/Library/Python/2.6/site-packages/sumatra/commands.py", line 177, in run
parameters = build_parameters(parameter_file, cmdline_parameters)
File "/Library/Python/2.6/site-packages/sumatra/parameters.py", line 227, in build_parameters
parameters = SimpleParameterSet(filename)
File "/Library/Python/2.6/site-packages/sumatra/parameters.py", line 63, in init
self.values[name] = eval(value)
File "", line 1
{
^
SyntaxError: unexpected EOF while parsing
}}}

Here is my parameter file.

{{{

-- coding: utf-8 --

"""
Default params for simple simulation to test project workspace.

AUTHORS:

• THOMAS MCTAVISH (2010-08-01): initial version
  """

SIMULATION PARAMETERS

sim_var = {
'tstop' : 40., # Duration of simulation
'asyn_onset' : 20, # Delay of the alpha synapse
'asyn_gmax' : 1, # Maximal conductance of the synapse \

OUTPUT PARAMETERS

'output_file' : 'plot', # Name of the graphic file without the extension
'graphic_format' : '.png', # May be '.jpg', '.gif', '.pdf'
'figsize' : (3, 2) # Size of our graphic, in (w, h) inches.
}
}}}

I have put the variables in a dict so that they can be more accessible -- they do not necessarily need to be global parameters, for example, and I can reference this dict for the pool of user-defined variables needed throughout the simulation, which are easily hidden in the globals() dict.

Imported from Bitbucket issue 47, created by apdavison on 08-09-2010 at 16:32, last modified: 08-24-2010 at 13:29

If there are multiple people working on a project, what should the most_recent()methods of Project and of RecordStore return? The label of the most recent record overall, or the most recent from that user?

Imported from Bitbucket issue 42, created by apdavison on 07-19-2010 at 20:36, last modified: 05-15-2013 at 11:43

This will probably require adding a most_recent() method to RecordStore.

Imported from Bitbucket issue 36, created by apdavison on 07-02-2010 at 13:51, last modified: 07-19-2010 at 20:56

Currently there are several cases where the version info is in the path to the module/package, but the dependency finder reports unknown.

Imported from Bitbucket issue 43, created by apdavison on 07-20-2010 at 18:32, last modified: 05-15-2013 at 09:57

Imported from Bitbucket issue 25, created by apdavison on 06-26-2010 at 21:29, last modified: 07-02-2010 at 12:54

Should print "The simulation results match", or the diff, if they don't

Imported from Bitbucket issue 11, created by apdavison on 06-26-2010 at 21:05, last modified: 07-02-2010 at 11:38

Imported from Bitbucket issue 26, created by apdavison on 06-26-2010 at 21:29, last modified: 06-30-2010 at 13:50

Imported from Bitbucket issue 28, created by apdavison on 06-26-2010 at 21:33, last modified: 04-27-2014 at 19:40

The aim is to support both Python 2 (>=2.6) and Python 3 (>=3.2) with a single codebase.

Imported from Bitbucket issue 37, created by apdavison on 07-11-2010 at 09:49, last modified: 08-30-2014 at 16:37

Imported from Bitbucket issue 19, created by apdavison on 06-26-2010 at 21:23, last modified: 06-30-2010 at 08:18

<main_file> <parameter_file> is not the only way to run scripts, e.g. <main_file> <secondary_file>, or <secondary_file> <main_file>, multiple parameter files, arguments, options

Imported from Bitbucket issue 12, created by apdavison on 06-26-2010 at 21:08, last modified: 02-14-2011 at 15:30

Imported from Bitbucket issue 20, created by apdavison on 06-26-2010 at 21:23, last modified: 02-25-2014 at 09:44

Imported from Bitbucket issue 15, created by apdavison on 06-26-2010 at 21:17, last modified: 04-04-2013 at 09:40

{{{
$ wget 'http://pypi.python.org/packages/source/S/Sumatra/Sumatra-0.2.1.tar.gz#md
5=696a2dd0e0128c33cc5169e25a87b805'
$ tar zxf Sumatra-0.2.1.tar.gz
$ cd Sumatra-0.2.1/
$ ls src/web/media/
smt.css
}}}

Imported from Bitbucket issue 45, created by apdavison on 08-03-2010 at 23:08, last modified: 02-14-2011 at 12:47

Imported from Bitbucket issue 35, created by apdavison on 07-02-2010 at 11:06, last modified: 03-09-2011 at 14:32

"smt help" by itself produces an error, but arguably it should do the same as "smt --help"

Imported from Bitbucket issue 31, created by apdavison on 06-28-2010 at 14:28, last modified: 12-10-2012 at 12:54

Imported from Bitbucket issue 29, created by apdavison on 06-26-2010 at 21:34, last modified: 05-14-2013 at 14:34

Although a vcs is necessary for almost all of Sumatra's functionality, it should not be a dependency for the help system.

Imported from Bitbucket issue 30, created by apdavison on 06-28-2010 at 14:27, last modified: 07-20-2010 at 09:19

The aim is to support both Python 2 (>=2.6) and Python 3 (>=3.2) with a single codebase.

Imported from Bitbucket issue 37, created by apdavison on 07-11-2010 at 09:49, last modified: 08-30-2014 at 16:37

If Sumatra is targeted at any command-line driven simulator or analysis tool, we can't assume people already have Python and easy_install/pip available, or that they are familar with Python.

Therefore, we need to revise the docs with this in mind.

Imported from Bitbucket issue 40, created by apdavison on 07-15-2010 at 15:21, last modified: 07-20-2010 at 07:43

At least one more is needed soon, to give an idea whether the DataStore interface is general enough.

Imported from Bitbucket issue 22, created by apdavison on 06-26-2010 at 21:25, last modified: 02-22-2013 at 14:11

Options used when compiling numpy would be a classic example

Imported from Bitbucket issue 16, created by apdavison on 06-26-2010 at 21:20, last modified: 04-04-2013 at 09:40

Imported from Bitbucket issue 39, created by apdavison on 07-15-2010 at 11:10, last modified: 07-20-2010 at 10:06

The web interface uses Django template system to render the simulation results. Up to now, the templates are defined globally in the sumatra source tree. However, it would be useful to allow users customize the templates for each project separately, so that the presentation can be fitted to the simulation.

A possible implementation would modify the Django TEMPLATE_DIRS setting:

[http://docs.djangoproject.com/en/dev/ref/templates/api/#loading-templates]

Imported from Bitbucket issue 4, created by apdavison on 05-12-2010 at 13:35, last modified: 07-19-2010 at 21:19

Imported from Bitbucket issue 27, created by apdavison on 06-26-2010 at 21:33, last modified: 05-14-2013 at 14:34

Imported from Bitbucket issue 27, created by apdavison on 06-26-2010 at 21:33, last modified: 05-14-2013 at 14:34

Imported from Bitbucket issue 32, created by apdavison on 06-28-2010 at 14:30, last modified: 06-28-2010 at 14:52

The above terms are used at various places in the code and documentation with overlapping meanings.

For the sake of clarity, I think we should retain just one term and one attribute. "group" is unnecessary, since grouping can be achieved with tagging. I think a single "label" attribute is best, which could be made up of a prefix plus timestamp, but could be some other string entirely. It is fine for smt to use prefix + timestamp, but this should not be fixed in the API.

Imported from Bitbucket issue 6, created by apdavison on 06-25-2010 at 15:40, last modified: 06-30-2010 at 13:49

e.g.

• the toy examples in the test directory

• https://bitbucket.org/apdavison/ircr2013

• https://bitbucket.org/apdavison/destexhe_jcns_2009

  Imported from Bitbucket issue 17, created by apdavison on 06-26-2010 at 21:21, last modified: 05-15-2013 at 11:41

If more than one simulation is launched and they will finish all together, then smt get confused and stores more files it should do.

{{{
Data key is ['29-04-2010/Sim_2/storage.sqlite',
'29-04-2010/Sim_2/plot_cai.png', '29-04-2010/Sim_2/plot_cali.png',
'29-04-2010/Sim_2/plot_ica.png', '29-04-2010/Sim_2/plot_v.png',
'29-04-2010/Sim_8/storage.sqlite', '29-04-2010/Sim_8/plot_cai.png',
'29-04-2010/Sim_8/plot_cali.png', '29-04-2010/Sim_8/plot_ica.png',
'29-04-2010/Sim_8/plot_v.png', '29-04-2010/Sim_5/storage.sqlite',
'29-04-2010/Sim_5/plot_cai.png', '29-04-2010/Sim_5/plot_cali.png',
'29-04-2010/Sim_5/plot_ica.png', '29-04-2010/Sim_5/plot_v.png',
'29-04-2010/Sim_1/storage.sqlite', '29-04-2010/Sim_1/plot_cai.png',
'29-04-2010/Sim_1/plot_cali.png', '29-04-2010/Sim_1/plot_ica.png',
'29-04-2010/Sim_1/plot_v.png', '29-04-2010/Sim_7/storage.sqlite',
'29-04-2010/Sim_7/plot_cai.png', '29-04-2010/Sim_7/plot_cali.png',
'29-04-2010/Sim_7/plot_ica.png', '29-04-2010/Sim_7/plot_v.png',
'29-04-2010/Sim_9/storage.sqlite', '29-04-2010/Sim_9/plot_cai.png',
'29-04-2010/Sim_9/plot_cali.png', '29-04-2010/Sim_9/plot_ica.png',
'29-04-2010!
/Sim_9/plot_v.png', '29-04-2010/Sim_6/storage.sqlite',
'29-04-2010/Sim_6/plot_cai.png', '29-04-2010/Sim_6/plot_cali.png',
'29-04-2010/Sim_6/plot_ica.png', '29-04-2010/Sim_6/plot_v.png',
'29-04-2010/Sim_4/storage.sqlite', '29-04-2010/Sim_4/plot_cai.png',
'29-04-2010/Sim_4/plot_cali.png', '29-04-2010/Sim_4/plot_ica.png',
'29-04-2010/Sim_4/plot_v.png', '29-04-2010/Sim_3/storage.sqlite',
'29-04-2010/Sim_3/plot_cai.png', '29-04-2010/Sim_3/plot_cali.png',
'29-04-2010/Sim_3/plot_ica.png', '29-04-2010/Sim_3/plot_v.png']
}}}

Imported from Bitbucket issue 3, created by mattions on 05-06-2010 at 12:44, last modified: 07-02-2010 at 10:53

The function may_have_working_copy() in _git.py calls git.repo.is_git_dir(), which is gone in the newest GitPython.

Possible fixes:

• Rewrite Sumatra to use the new API and change check_version() accordingly

• Require version 0.2.X of GitPython via check_version()

  Imported from Bitbucket issue 38, created by apdavison on 07-14-2010 at 08:57, last modified: 07-14-2010 at 10:20

Imported from Bitbucket issue 18, created by apdavison on 06-26-2010 at 21:21, last modified: 02-25-2014 at 09:45

Imported from Bitbucket issue 21, created by apdavison on 06-26-2010 at 21:24, last modified: 05-15-2013 at 09:55

to:

• RecordDifference

• programs.register_executable()

• launch.py

• datastore.py

  Imported from Bitbucket issue 13, created by apdavison on 06-26-2010 at 21:10, last modified: 07-02-2010 at 13:38

One issue is that the browser call must take place after starting the server, but the server call does not return. Use multiprocessing or threads?

Imported from Bitbucket issue 23, created by apdavison on 06-26-2010 at 21:27, last modified: 07-20-2010 at 10:33

I create the db on one computer. Then I started the smtweb from another one.
The files are all there, but I've got this error:

{{{
OSError at /default_20100429-190321/

(2, 'No such file or directory')

Request Method: GET
Request URL: http://127.0.0.1:8000/default_20100429-190321/
Exception Type: OSError
Exception Value:

(2, 'No such file or directory')

Exception Location: /homes/mattioni/work/compneur_rep/sumatra/datastore.py in init, line 122
Python Executable: /ebi/research/software/Linux_x86_64/opt/stow/python-2.6.1/bin/python
Python Version: 2.6.1`
}}}

{{{
Environment:

Request Method: GET
Request URL: http://127.0.0.1:8000/default_20100429-190321/
Django Version: 1.1.1
Python Version: 2.6.1
Installed Applications:
['sumatra.recordstore.django_store',
'django.contrib.contenttypes',
'tagging',
'sumatra.web']
Installed Middleware:
()

Traceback:
File "/homes/mattioni/work/compneur_rep/sumatra/web/views.py" in simulation_detail

1. datafiles = data_store.list_files(eval(record.data_key))
   

   File "/homes/mattioni/work/compneur_rep/sumatra/datastore.py" in list_files

2.     return [DataFile(path, self) for path in key]
   

   File "/homes/mattioni/work/compneur_rep/sumatra/datastore.py" in init

3.     stats = os.stat(self.full_path)
   

Exception Type: OSError at /default_20100429-190321/
Exception Value: (2, 'No such file or directory')
}}}

Files with db

{{{
mattioni@login-svr2:TimeScales(local)$ ls -a
. libs
.. mathhelper.pyx
biochemical_circuits mod
branch_dist module_finder_output.txt
Cython benchmark.ods neuronControl
Data nrnvisio.py
default_20100429-135514.param plotterBio.py
default_20100429-135952.param replot_with_interval.py
default_20100429-152632.param results_analizer.py
default_20100429-152638.param sandbox
default_20100429-152649.param setup.py
default_20100429-152653.param .smt
default_20100429-190321.param.orig spineIntegration.py
default.param storage2data.py
ecellControl test_spine.py
.git timecoursesImporter.py
.gitignore visioStart.py
helpers visualizerBiochemical.py
hoc visualizer.py
}}}

{{{
Label : default_20100429-190321
Reason : Delivering of the inputs. kir qfactor scan
Outcome :
Duration : 19629.149194
Repository : GitRepository at
/net/isilon4/research/lenovere/work/mattioni/compneur_rep/TimeSca
les
Main_File : spineIntegration.py
Version : d961f5b688325487f517ed61ddbec81049a8bc50
Executable : Python (version: 2.6.1) at
/ebi/research/software/Linux_x86_64/opt/stow/python-2.6.1/bin/pyt
hon
Timestamp : 2010-04-29 19:03:21.208508
Tags : inputs, all, kir
}}}

Imported from Bitbucket issue 2, created by mattions on 05-06-2010 at 12:42, last modified: 05-30-2013 at 21:31

A longer description of a project

Imported from Bitbucket issue 14, created by apdavison on 06-26-2010 at 21:15, last modified: 06-28-2010 at 15:08

Imported from Bitbucket issue 44, created by apdavison on 07-20-2010 at 19:09, last modified: 06-28-2011 at 13:24

Imported from Bitbucket issue 9, created by apdavison on 06-26-2010 at 20:46, last modified: 06-29-2010 at 12:18

Imported from Bitbucket issue 7, created by apdavison on 06-26-2010 at 20:44, last modified: 06-30-2010 at 13:52

Imported from Bitbucket issue 10, created by apdavison on 06-26-2010 at 20:47, last modified: 06-29-2010 at 12:19

Imported from Bitbucket issue 8, created by apdavison on 06-26-2010 at 20:45, last modified: 06-28-2010 at 15:33

Imported from Bitbucket issue 24, created by apdavison on 06-26-2010 at 21:28, last modified: 02-15-2011 at 12:17

I tried to run two simulations at the same time with sumatra (smt run --executable=python --main=main.py default.param). I got this error when I run the second simulation:

{{{
Traceback (most recent call last):
File "/_/bin/smt", line 30, in
main(sys.argv[2:])
File "/__/lib/python2.6/site-packages/sumatra/commands.py", line 195, in run
launch_mode=launch_mode)
File "/__/lib/python2.6/site-packages/sumatra/projects.py", line 117, in launch
reason)
File "/__/lib/python2.6/site-packages/sumatra/projects.py", line 108, in new_record
record.register()
File "/__/lib/python2.6/site-packages/sumatra/records.py", line 57, in register
self.dependencies = dependency_finder.find_dependencies(self.main_file, self.executable, self.on_changed)
File "/__/lib/python2.6/site-packages/sumatra/dependency_finder/init.py", line 30, in find_dependencies
return python.find_dependencies(filename, on_changed)
File "/__/lib/python2.6/site-packages/sumatra/dependency_finder/python.py", line 172, in find_dependencies
packages = find_imported_packages(filename)
File "/*_*/lib/python2.6/site-packages/sumatra/dependency_finder/python.py", line 162, in find_imported_packages
os.remove("module_finder_output.txt")
OSError: [Errno 2] No such file or directory: 'module_finder_output.txt'
}}}

Imported from Bitbucket issue 46, created by apdavison on 08-03-2010 at 23:44, last modified: 03-14-2011 at 13:48

This will allow Sumatra to detect when a datafile has been overwritten, and can also be used to speed-up comparisons between records.

My one concern is speed. How much time will this take? I guess that since in general any simulation/analysis that generates very large files will take a long time to run anyway, this should be acceptable. Need to do some timing/benchmarking though.

Imported from Bitbucket issue 41, created by apdavison on 07-19-2010 at 16:51, last modified: 06-27-2011 at 13:41

When a record is deleted from the db also the file should be deleted

Imported from Bitbucket issue 1, created by mattions on 05-06-2010 at 12:21, last modified: 07-01-2010 at 11:51

Although Sumatra was developed in the context of simulation-based computational neuroscience, it is clear that it could be useful in computational sciences more generally, and for computational analyses as well as for simulations. The documentation and class names should be modified accordingly.

Imported from Bitbucket issue 5, created by apdavison on 06-25-2010 at 15:33, last modified: 06-30-2010 at 15:20

It is possible that data files may be produced in different places in the filesystem that don't have a common root directory (or that are very far from it such that it would waste time searching all the intermediate directories). For this reason, it should be possible to supply a list of root directories, not just a single one.

Imported from Bitbucket issue 33, created by apdavison on 06-28-2010 at 14:33, last modified: 05-15-2013 at 11:16

Imported from Bitbucket issue 34, created by apdavison on 07-01-2010 at 15:42, last modified: 07-02-2010 at 10:46