pynbody / tangos

The Agile Numerical Galaxy Organisation System

pynbody

Pynbody is an analysis framework for N-body and hydrodynamic astrophysical simulations supporting PKDGRAV/Gasoline, Gadget, Gadget4/Arepo, N-Chilada and RAMSES AMR outputs. It supports Python 3 only (versions prior to 1.0 are still available on PyPI for Python 2). Minor version support adheres roughly to SPEC0.

Written in Python, the core tools are accompanied by a library of publication-level analysis routines. For a quick tour of some of the features, have a look at this IPython notebook.

Getting started

If python and the standard pip package manager is installed and properly configured, you can simply do:

$ pip install pynbody

If this fails, you may need some more detailed installation instructions. Once you have the package installed, try the introductory tutorials. The full documentation can be found here.

Contributing

Help us make pynbody better! As you develop analysis for your science with pynbody, consider making your code available for everyone else to use. You can do this by creating a tutorial or cookbook or by adding your code to the relevant sub-module and submitting a pull request (make a fork first -- see https://help.github.com/articles/using-pull-requests).

Acknowledging the code

When using pynbody, please acknowledge it by citing the Astrophysics Source Code Library entry. Optionally you can also cite the Zenodo DOI for the specific version of pynbody that you are using, which may be found here.

Support and Contact

If you have trouble with Pynbody or you have feature requests/suggestions you can submit an issue, and/or send us an email on the Usergroup mailing list.

tangos's Issues

Is it possible to add halos to an existing simulation ?

This is more a question than an issue.

Let's say I have a database with an already built simulation with halos containing more than 500 particles in each timestep.

I am now suddenly interested in haloes with less 500 particles. Is there a way to add these halos to the existing database without starting over ?

Rerunning tangos add does nothing since all timesteps already exists so adding new halos is skipped.

at() live property

Hi Andrew,

I found myself confused by the behaviour of the live property at() today. The calculation reconstructs the range of x values of an array, based on the plot_x0() and plot_xdelta() methods:

tangos/tangos/properties/live_profiles.py

Lines 13 to 20 in 2f2039d

	def live_calculate(self, halo, pos, ar):
	x0 = self._array_info.plot_x0()
	delta_x = self._array_info.plot_xdelta()
	# linear interpolation
	i0 = int((pos-x0)/delta_x)
	i1 = i0+1

However, it then assumes that the x value are linearly spaced by xdelta to locate the value in the array. This assumptions holds for the tangos builtins properties but not necessarily for some of custom defined properties which have e.g. logarithmic spacing in radius.

For my custom properties, I override the plot_x_values() method to ensure that plots are generated with the correct x range, here in the base class:

tangos/tangos/properties/__init__.py

Lines 166 to 169 in 2f2039d

	def plot_x_values(self, for_data):
	"""Return a suitable array of x values to match the
	given y values"""
	return np.arange(self.plot_x0(), self.plot_x0() + self.plot_xdelta() * (len(for_data) - 0.5), self.plot_xdelta())

Since this latter method is a public method of the PropertyCalculation class, wouldn't it make sense for the at() method to use plot_x_values directly rather than generating it?

Martin

Broken earliest() live calculation

Hi,

While investigating issue #123, I realised that I am unable to use the earliest() live calculation from a halo. It consistently returns a NoResultsError whenever called, from the beginning, end or middle of the merger tree:

In [1]: halo = tangos.get_halo("Halo1445_DMO/output_00101/halo_1")                                                                               

In [2]: halo.calculate('earliest()')                                                                                                             
---------------------------------------------------------------------------
NoResultsError                            Traceback (most recent call last)
<ipython-input-2-a1f61cad7e1b> in <module>
----> 1 halo.calculate('earliest()')

~/Documents/tangos/tangos/core/halo.py in calculate(self, calculation, return_description)
    143         from .. import live_calculation
    144         calculation = live_calculation.parser.parse_property_name_if_required(calculation)
--> 145         (value,), description = calculation.values_sanitized_and_description([self], Session.object_session(self))
    146         if len(value)==1:
    147             retval = value[0]

~/Documents/tangos/tangos/live_calculation/__init__.py in values_sanitized_and_description(self, halos, load_into_session)
    130         See values_sanitized for the definition of sanitized"""
    131         values, desc = self.values_and_description(halos)
--> 132         return self._sanitize_values(values, load_into_session), desc
    133 
    134     def values(self, halos):

~/Documents/tangos/tangos/live_calculation/__init__.py in _sanitize_values(self, unsanitized_values, load_into_session, no_results_raises)
    175         for x in output_values:
    176             if len(x)==0 and no_results_raises:
--> 177                 raise NoResultsError("Calculation %s returned no results" % self)
    178 
    179         if load_into_session is not None:

NoResultsError: Calculation earliest() returned no results

In [3]: halo = tangos.get_halo("Halo1445_DMO/output_00041/halo_1")                                                                               

In [4]: halo.calculate('earliest()')                                                                                                             
---------------------------------------------------------------------------
NoResultsError                            Traceback (most recent call last)
<ipython-input-4-a1f61cad7e1b> in <module>
----> 1 halo.calculate('earliest()')

~/Documents/tangos/tangos/core/halo.py in calculate(self, calculation, return_description)
    143         from .. import live_calculation
    144         calculation = live_calculation.parser.parse_property_name_if_required(calculation)
--> 145         (value,), description = calculation.values_sanitized_and_description([self], Session.object_session(self))
    146         if len(value)==1:
    147             retval = value[0]

~/Documents/tangos/tangos/live_calculation/__init__.py in values_sanitized_and_description(self, halos, load_into_session)
    130         See values_sanitized for the definition of sanitized"""
    131         values, desc = self.values_and_description(halos)
--> 132         return self._sanitize_values(values, load_into_session), desc
    133 
    134     def values(self, halos):

~/Documents/tangos/tangos/live_calculation/__init__.py in _sanitize_values(self, unsanitized_values, load_into_session, no_results_raises)
    175         for x in output_values:
    176             if len(x)==0 and no_results_raises:
--> 177                 raise NoResultsError("Calculation %s returned no results" % self)
    178 
    179         if load_into_session is not None:

NoResultsError: Calculation earliest() returned no results

In [5]: halo = tangos.get_halo("Halo1445_DMO/output_00007/halo_1")                                                                               

In [6]: halo.calculate('earliest()')                                                                                                             
---------------------------------------------------------------------------
NoResultsError                            Traceback (most recent call last)
<ipython-input-6-a1f61cad7e1b> in <module>
----> 1 halo.calculate('earliest()')

~/Documents/tangos/tangos/core/halo.py in calculate(self, calculation, return_description)
    143         from .. import live_calculation
    144         calculation = live_calculation.parser.parse_property_name_if_required(calculation)
--> 145         (value,), description = calculation.values_sanitized_and_description([self], Session.object_session(self))
    146         if len(value)==1:
    147             retval = value[0]

~/Documents/tangos/tangos/live_calculation/__init__.py in values_sanitized_and_description(self, halos, load_into_session)
    130         See values_sanitized for the definition of sanitized"""
    131         values, desc = self.values_and_description(halos)
--> 132         return self._sanitize_values(values, load_into_session), desc
    133 
    134     def values(self, halos):

~/Documents/tangos/tangos/live_calculation/__init__.py in _sanitize_values(self, unsanitized_values, load_into_session, no_results_raises)
    175         for x in output_values:
    176             if len(x)==0 and no_results_raises:
--> 177                 raise NoResultsError("Calculation %s returned no results" % self)
    178 
    179         if load_into_session is not None:

NoResultsError: Calculation earliest() returned no results

The links between snapshots are there and working -- I can successfully visualise or construct merger trees for various halos. I can also click on the "earliest" button of the web interface and successfully land on the earliest halo, but am unable to do so from the python interface or through the live-calculation system in the web interface.

This is not specific to this database or this simulation either, I verified the issue is reproduced for other simulations in the same database, for other databases produced with the same pipeline and for databases produced with independent pipelines.

I think this points towards a bug in the live calculation property, as the button for the web interface uses halo.earliest rather than halo.calculate('earliest()').

Martin

Tangos link finding 0 links

Hello,

I've tried running Tangos link for RAMSES simulations run using GenetIC initial conditions and halos found using HOP, but Tangos is finding 0 links and so no merger history is built.

An example of the output looks like this:

2018-09-27 09:52:24,982 : Gathering links for <TimeStep u'Halo1445/output_00008' z=13.27 t=0.32 Gyr> and <TimeStep u'Halo1445/output_00009' z=11.50 t=0.39 Gyr>
2018-09-27 09:52:24,982 : Identified 0 links between <TimeStep u'Halo1445/output_00008' z=13.27 t=0.32 Gyr> and <TimeStep u'Halo1445/output_00009' z=11.50 t=0.39 Gyr>
2018-09-27 09:52:24,982 : Identified 0 links between <TimeStep u'Halo1445/output_00009' z=11.50 t=0.39 Gyr> and <TimeStep u'Halo1445/output_00008' z=13.27 t=0.32 Gyr>
2018-09-27 09:52:24,982 : Preparing to commit links for <TimeStep u'Halo1445/output_00008' z=13.27 t=0.32 Gyr> and <TimeStep u'Halo1445/output_00009' z=11.50 t=0.39 Gyr>
2018-09-27 09:52:24,985 : Finished committing total of 0 links for <TimeStep u'Halo1445/output_00008' z=13.27 t=0.32 Gyr> and <TimeStep u'Halo1445/output_00009' z=11.50 t=0.39 Gyr>

There are no errors and the Tangos command will run through all the outputs finding 0 links.

TANGOS currently requires pynbody

the tutorial states that it is agnostic in terms of what one wishes to use (yt, pynbody,etc) but the profiles module in properties imports pynbody. So, importing the database fails if pynbody is not installed

match() live calculation and cross linking

Hi,

I am having trouble with the match() live calculation command. I am trying to match halos between different simulations that have been cross linked.

The links physically exist and are stored in the database (In [2] in the following python code). The match() command works as expected to find an earlier or later halo within the same simulation (In [3] in the code). When trying to match() across simulations, the live calculation returns no result (In [4] and [5]).

I am also seeing this behaviour in the web server, for which the "Find in another simulation feature also fails".

Any ideas on what is causing this?

Martin

In [1]: halo = tangos.get_halo("Halo1459_DMO_lowres/%101/halo_1")

In [2]: halo.all_links
Out[2]: 
[<HaloLink ptcls_in_common Halo1459_DMO_lowres/output_00101/halo_1 to Halo1459_DMO_lowres/output_00100/halo_1 weight=0.99>,
 <HaloLink ptcls_in_common Halo1459_DMO_lowres/output_00101/halo_1 to Halo1459_DMO_lowres_variancex09/output_00101/halo_1 weight=1.00>,
 <HaloLink ptcls_in_common Halo1459_DMO_lowres/output_00101/halo_1 to Halo1459_DMO_lowres_variancex11/output_00101/halo_1 weight=1.00>]

In [3]: halo.calculate('match("Halo1459_DMO_lowres/output_00100")')
Out[3]: <Halo 'Halo1459_DMO_lowres/output_00100/halo_1' | NDM=997808 Nstar=0 Ngas=0>

In [4]: halo.calculate('match("Halo1459_DMO_lowres_variancex11/output_00101")')
---------------------------------------------------------------------------
NoResultsError                            Traceback (most recent call last)
<ipython-input-4-d65a1a9e3b96> in <module>()
----> 1 halo.calculate('match("Halo1459_DMO_lowres_variancex11/output_00101")')

~/Documents/tangos/tangos/core/halo.py in calculate(self, calculation, return_description)
    143         from .. import live_calculation
    144         calculation = live_calculation.parser.parse_property_name_if_required(calculation)
--> 145         (value,), description = calculation.values_sanitized_and_description([self], Session.object_session(self))
    146         if len(value)==1:
    147             retval = value[0]

~/Documents/tangos/tangos/live_calculation/__init__.py in values_sanitized_and_description(self, halos, load_into_session)
    130         See values_sanitized for the definition of sanitized"""
    131         values, desc = self.values_and_description(halos)
--> 132         return self._sanitize_values(values, load_into_session), desc
    133 
    134     def values(self, halos):

~/Documents/tangos/tangos/live_calculation/__init__.py in _sanitize_values(self, unsanitized_values, load_into_session, no_results_raises)
    175         for x in output_values:
    176             if len(x)==0 and no_results_raises:
--> 177                 raise NoResultsError("Calculation %s returned no results" % self)
    178 
    179         if load_into_session is not None:

NoResultsError: Calculation match("Halo1459_DMO_lowres_variancex11/output_00101") returned no results


In [5]: halo.calculate('match("Halo1459_DMO_lowres_variancex11")')
---------------------------------------------------------------------------
NoResultsError                            Traceback (most recent call last)
<ipython-input-6-f201d26cb9ac> in <module>()
----> 1 halo.calculate('match("Halo1459_DMO_lowres_variancex11")')

~/Documents/tangos/tangos/core/halo.py in calculate(self, calculation, return_description)
    143         from .. import live_calculation
    144         calculation = live_calculation.parser.parse_property_name_if_required(calculation)
--> 145         (value,), description = calculation.values_sanitized_and_description([self], Session.object_session(self))
    146         if len(value)==1:
    147             retval = value[0]

~/Documents/tangos/tangos/live_calculation/__init__.py in values_sanitized_and_description(self, halos, load_into_session)
    130         See values_sanitized for the definition of sanitized"""
    131         values, desc = self.values_and_description(halos)
--> 132         return self._sanitize_values(values, load_into_session), desc
    133 
    134     def values(self, halos):

~/Documents/tangos/tangos/live_calculation/__init__.py in _sanitize_values(self, unsanitized_values, load_into_session, no_results_raises)
    175         for x in output_values:
    176             if len(x)==0 and no_results_raises:
--> 177                 raise NoResultsError("Calculation %s returned no results" % self)
    178 
    179         if load_into_session is not None:

NoResultsError: Calculation match("Halo1459_DMO_lowres_variancex11") returned no results

Stars go missing in halo properties

I've made a custom Tangos property which makes calculations involving the stars of a halo from a Ramses simulation. I set up the property like:

class HaloProperty(SphericalRegionHaloProperties):
    @centred_calculation
    def calculate(self, halo, existing_properties):

Up until now this has worked as expected, and calling 'halo.s' returns a dataset containing the star particles. However, in a recent simulation things have changed. Up until output 31 it works as normal, but afterwards it doesn't seem to find any stars.

If I load the simulation into Pynbody and centre on the desired halo then I find plenty of stars.
What could cause Tangos to miss these star particles?

python3.6 pyramid_debugtoolbar

Hi,

I finally played around with the database. Creating and filling works fine.
However, the web interface seems not to work for me under python3.6 the error below is what I get if I type
pserve development.ini

in the subfolder web.

2017-12-07 00:13:26,752 INFO [sqlalchemy.engine.base.Engine:1243][MainThread] SELECT CAST('test plain returns' AS VARCHAR(60)) AS anon_1
2017-12-07 00:13:26,753 INFO [sqlalchemy.engine.base.Engine:1244][MainThread] ()
2017-12-07 00:13:26,755 INFO [sqlalchemy.engine.base.Engine:1243][MainThread] SELECT CAST('test unicode returns' AS VARCHAR(60)) AS anon_1
2017-12-07 00:13:26,755 INFO [sqlalchemy.engine.base.Engine:1244][MainThread] ()
2017-12-07 00:13:26,756 INFO [sqlalchemy.engine.base.Engine:1140][MainThread] PRAGMA table_info("dictionary")
2017-12-07 00:13:26,756 INFO [sqlalchemy.engine.base.Engine:1143][MainThread] ()
2017-12-07 00:13:26,758 INFO [sqlalchemy.engine.base.Engine:1140][MainThread] PRAGMA table_info("creators")
2017-12-07 00:13:26,758 INFO [sqlalchemy.engine.base.Engine:1143][MainThread] ()
2017-12-07 00:13:26,758 INFO [sqlalchemy.engine.base.Engine:1140][MainThread] PRAGMA table_info("simulations")
2017-12-07 00:13:26,758 INFO [sqlalchemy.engine.base.Engine:1143][MainThread] ()
2017-12-07 00:13:26,759 INFO [sqlalchemy.engine.base.Engine:1140][MainThread] PRAGMA table_info("simulationproperties")
2017-12-07 00:13:26,759 INFO [sqlalchemy.engine.base.Engine:1143][MainThread] ()
2017-12-07 00:13:26,759 INFO [sqlalchemy.engine.base.Engine:1140][MainThread] PRAGMA table_info("trackdata")
2017-12-07 00:13:26,759 INFO [sqlalchemy.engine.base.Engine:1143][MainThread] ()
2017-12-07 00:13:26,759 INFO [sqlalchemy.engine.base.Engine:1140][MainThread] PRAGMA table_info("timesteps")
2017-12-07 00:13:26,759 INFO [sqlalchemy.engine.base.Engine:1143][MainThread] ()
2017-12-07 00:13:26,760 INFO [sqlalchemy.engine.base.Engine:1140][MainThread] PRAGMA table_info("halos")
2017-12-07 00:13:26,760 INFO [sqlalchemy.engine.base.Engine:1143][MainThread] ()
2017-12-07 00:13:26,760 INFO [sqlalchemy.engine.base.Engine:1140][MainThread] PRAGMA table_info("haloproperties")
2017-12-07 00:13:26,760 INFO [sqlalchemy.engine.base.Engine:1143][MainThread] ()
2017-12-07 00:13:26,760 INFO [sqlalchemy.engine.base.Engine:1140][MainThread] PRAGMA table_info("halolink")
2017-12-07 00:13:26,760 INFO [sqlalchemy.engine.base.Engine:1143][MainThread] ()
2017-12-07 00:13:26,761 INFO [sqlalchemy.engine.base.Engine:1140][MainThread] PRAGMA table_info("arrayplotoptions")
2017-12-07 00:13:26,761 INFO [sqlalchemy.engine.base.Engine:1143][MainThread] ()
Traceback (most recent call last):
File "/Users/buck/anaconda3/bin/pserve", line 6, in
sys.exit(main())
File "/Users/buck/anaconda3/lib/python3.6/site-packages/pyramid-1.5.7-py3.6.egg/pyramid/scripts/pserve.py", line 58, in main
return command.run()
File "/Users/buck/anaconda3/lib/python3.6/site-packages/pyramid-1.5.7-py3.6.egg/pyramid/scripts/pserve.py", line 328, in run
global_conf=vars)
File "/Users/buck/anaconda3/lib/python3.6/site-packages/pyramid-1.5.7-py3.6.egg/pyramid/scripts/pserve.py", line 363, in loadapp
return loadapp(app_spec, name=name, relative_to=relative_to, **kw)
File "/Users/buck/.local/lib/python3.6/site-packages/PasteDeploy-1.5.2-py3.6.egg/paste/deploy/loadwsgi.py", line 247, in loadapp
return loadobj(APP, uri, name=name, **kw)
File "/Users/buck/.local/lib/python3.6/site-packages/PasteDeploy-1.5.2-py3.6.egg/paste/deploy/loadwsgi.py", line 272, in loadobj
return context.create()
File "/Users/buck/.local/lib/python3.6/site-packages/PasteDeploy-1.5.2-py3.6.egg/paste/deploy/loadwsgi.py", line 710, in create
return self.object_type.invoke(self)
File "/Users/buck/.local/lib/python3.6/site-packages/PasteDeploy-1.5.2-py3.6.egg/paste/deploy/loadwsgi.py", line 146, in invoke
return fix_call(context.object, context.global_conf, **context.local_conf)
File "/Users/buck/.local/lib/python3.6/site-packages/PasteDeploy-1.5.2-py3.6.egg/paste/deploy/util.py", line 55, in fix_call
val = callable(*args, **kw)
File "/Users/buck/.local/lib/python3.6/site-packages/tangos-1.0.dev0-py3.6.egg/tangos/web/init.py", line 8, in main
config = Configurator(settings=settings)
File "/Users/buck/anaconda3/lib/python3.6/site-packages/pyramid-1.5.7-py3.6.egg/pyramid/config/init.py", line 301, in init
exceptionresponse_view=exceptionresponse_view,
File "/Users/buck/anaconda3/lib/python3.6/site-packages/pyramid-1.5.7-py3.6.egg/pyramid/config/init.py", line 412, in setup_registry
self.include(inc)
File "/Users/buck/anaconda3/lib/python3.6/site-packages/pyramid-1.5.7-py3.6.egg/pyramid/config/init.py", line 727, in include
c = self.maybe_dotted(callable)
File "/Users/buck/anaconda3/lib/python3.6/site-packages/pyramid-1.5.7-py3.6.egg/pyramid/config/init.py", line 825, in maybe_dotted
return self.name_resolver.maybe_resolve(dotted)
File "/Users/buck/anaconda3/lib/python3.6/site-packages/pyramid-1.5.7-py3.6.egg/pyramid/path.py", line 320, in maybe_resolve
return self._resolve(dotted, package)
File "/Users/buck/anaconda3/lib/python3.6/site-packages/pyramid-1.5.7-py3.6.egg/pyramid/path.py", line 327, in _resolve
return self._zope_dottedname_style(dotted, package)
File "/Users/buck/anaconda3/lib/python3.6/site-packages/pyramid-1.5.7-py3.6.egg/pyramid/path.py", line 376, in _zope_dottedname_style
found = import(used)
ModuleNotFoundError: No module named 'pyramid_debugtoolbar'<

any suggestions?

cheers
Tobias

Unable to access Pynbody particle data

Hello,

I have just recently updated Tangos and am now getting an error when writing properties with 'tangos write'. It seems to occur whenever there is an attempt to reference the halo particle data, however there is no such error if halo.properties['something'] is called. This error occurs independently of any arguments used alongside 'tangos write'.

The traceback looks like this:

Traceback (most recent call last):
  File "/scratch/mo00101/anaconda2/lib/python2.7/site-packages/tangos/tools/property_writer.py", line 382, in _get_property_value
    result = property_calculator.calculate(snapshot_data, db_data)
  File "/scratch/mo00101/anaconda2/lib/python2.7/site-packages/tangos/properties/pynbody/centring.py", line 61, in new_fn
    with _recenter(halo.ancestor, existing_properties['shrink_center']):
  File "/scratch/mo00101/anaconda2/lib/python2.7/site-packages/pynbody/snapshot/__init__.py", line 583, in ancestor
    return self.base.ancestor
  File "/scratch/mo00101/anaconda2/lib/python2.7/site-packages/pynbody/snapshot/__init__.py", line 583, in ancestor
    return self.base.ancestor
  File "/scratch/mo00101/anaconda2/lib/python2.7/site-packages/pynbody/snapshot/__init__.py", line 583, in ancestor
    return self.base.ancestor

<repeated many times>

  File "/scratch/mo00101/anaconda2/lib/python2.7/site-packages/pynbody/snapshot/__init__.py", line 583, in ancestor
    return self.base.ancestor
  File "/scratch/mo00101/anaconda2/lib/python2.7/site-packages/pynbody/snapshot/__init__.py", line 583, in ancestor
    return self.base.ancestor
RuntimeError: maximum recursion depth exceeded
2018-05-15 14:49:01,652 : Further errors from this calculation on this timestep will be counted but not individually reported.
2018-05-15 14:49:01,652 : Done with <TimeStep u'Halo332_DMonly/output_00067' z=0.52 t=8.50 Gyr>
2018-05-15 14:49:01,678 :             Succeeded: 0 property calculations
2018-05-15 14:49:01,678 :               Errored: 1 property calculations
2018-05-15 14:49:01,678 :   Errored during load: 0 property calculations
2018-05-15 14:49:01,678 :        Already exists: 1 property calculations
2018-05-15 14:49:01,678 : Missing pre-requisite: 0 property calculations
2018-05-15 14:49:01,678 : Attempting to commit 1 halo properties...
2018-05-15 14:49:01,678 : 1 properties were committed
2018-05-15 14:49:01,678 : CUMULATIVE RUNNING TIMES (just this node)
2018-05-15 14:49:01,678 :  ExampleHaloProperty 0.0s | 100.0%

All the best, -Matt

Support for slashes in path to simulation

Steps to reproduce

Add a simulation with a path containing slashes. The internal links used by the webserver incorrectly strips them away or do so inconsistently. This results in broken link on the web interface and plots do not show up.

Temporary work around

This can be fixed temporarily by changing the path to the simulation to another one without slashes (assuming the path exists).

sqlite3 database.db
> update "simulations" set basename="NEW_PATH_WITHOUT_SLASHES" where basename="OLD_PATH/WITH/SLASHES";

Abs() live calculation

Hi,

I found surprising behaviour when using the abs() live calculation today. On a scalar, negative number, the calculation fails due to an array shape issue:

In [3]: halo['mean_iron_hydrogen_ratio']
Out[3]: -2.5548620088265874

In [4]: halo.calculate('abs(mean_iron_hydrogen_ratio)')
---------------------------------------------------------------------------
AxisError                                 Traceback (most recent call last)
<ipython-input-4-64253b3ef2ad> in <module>()
----> 1 halo.calculate('abs(mean_iron_hydrogen_ratio)')

~/Documents/tangos/tangos/core/halo.py in calculate(self, calculation, return_description)
    143         from .. import live_calculation
    144         calculation = live_calculation.parser.parse_property_name_if_required(calculation)
--> 145         (value,), description = calculation.values_sanitized_and_description([self], Session.object_session(self))
    146         if len(value)==1:
    147             retval = value[0]

~/Documents/tangos/tangos/live_calculation/__init__.py in values_sanitized_and_description(self, halos, load_into_session)
    129 
    130         See values_sanitized for the definition of sanitized"""
--> 131         values, desc = self.values_and_description(halos)
    132         return self._sanitize_values(values, load_into_session), desc
    133 

~/Documents/tangos/tangos/live_calculation/__init__.py in values_and_description(self, halos)
    403             input_descriptions.append(id)
    404 
--> 405         calculator, results = self._evaluate_function(halos, input_descriptions, input_values)
    406 
    407         return results, calculator

~/Documents/tangos/tangos/live_calculation/__init__.py in _evaluate_function(self, halos, input_descriptions, input_values)
    520         else:
    521             inherited_description = None
--> 522         return inherited_description, self._as_1xn_array(self._func(halos, *input_values))
    523 
    524 

~/Documents/tangos/tangos/live_calculation/builtin_functions/arithmetic.py in abs(halos, vals)
      7 @BuiltinFunction.register
      8 def abs(halos, vals):
----> 9     return arithmetic_unary_op(vals, functools.partial(np.linalg.norm, axis=-1))
     10 
     11 @BuiltinFunction.register

~/Documents/tangos/tangos/live_calculation/builtin_functions/arithmetic.py in arithmetic_unary_op(vals1, op)
     90         if v1 is not None:
     91             v1 = np.asarray(v1, dtype=float)
---> 92             result = op(v1)
     93         else:
     94             result = None

~/anaconda3/lib/python3.6/site-packages/numpy/linalg/linalg.py in norm(x, ord, axis, keepdims)
   2479             # special case for speedup
   2480             s = (x.conj() * x).real
-> 2481             return sqrt(add.reduce(s, axis=axis, keepdims=keepdims))
   2482         else:
   2483             try:

AxisError: axis -1 is out of bounds for array of dimension 0

When used on an actual array, it returns a single number rather than the absolute value of individual elements that I would have expected:

In [5]: halo.calculate('abs(dm_density_profile)')
Out[5]: 10607776652.113852

I tracked to the use numpy.linalg.norm here:

tangos/tangos/live_calculation/builtin_functions/arithmetic.py

Lines 7 to 9 in 76ddb30

	@BuiltinFunction.register
	def abs(halos, vals):
	return arithmetic_unary_op(vals, functools.partial(np.linalg.norm, axis=-1))

Is this the expected behaviour and I am misusing the function? Or should the live calculation simply be numpy.abs rather than the full vector norm on the last axis?

Hope this is clear,
Martin

Tangos write for a halo and its progenitors

Hello,

Currently 'Tangos write' allows the option to calculate properties for specific halo numbers. This is useful for when I only want to calculate properties for a single main halo, as it saves time and avoids making the database file unnecessarily large.

If I used the argument --hmax=0 then I would only create properties for the largest halo, which would be exactly what I want except that the main halo in a simulation may not be the largest halo at very early time steps.

Is there a way to calculate halo properties for a specific halo based on the results of 'Tangos link' instead? IE, I would only have properties for the halo that is halo_0 at the final output and all of its progenitor halos.

EDIT:
I've tried altering the property_writer.py file in Tangos, but I don't know enough about how the code works to effect the changes I want. Ideally I would run something similar to this:

        if self.options.hmax is not None:
            query = sqlalchemy.and_(query, core.halo.Halo.halo_number<=self.options.hmax)

Instead of halo_number, I would be sorting by the halo_number of a linked halo. Instead of hmax, I would have an argument 'hlink' which would be the halo number at z=0. I've found the properties Halo.next, Halo.previous, Halo.links, etc etc. However, they don't return an integer which can be used in the sqlalchemy query. I've tried to change the source code such that they do, but to no avail.

After finding the appropriate halo, I would then update the hlink argument so that it assumed the value of the current halo halo_number. Again, I've no idea how to actually do this in Tangos. Assuming Tangos writer is run in time order, this would then build up the properties for all the progenitors of Halo_0.

EDIT 2:
I'm currently using a workaround, where I manually paste the results of halo.calculate_for_progenitors("halo_number()") into an argument in Tangos write. This isn't very pretty but it does what I need.

reverse_property_cascade sometimes returns IndexError instead of NoResultsError

Hi Andrew,

I found that when querying for many different properties, reverse_property_cascade will sometimes give me an IndexError instead of a NoResultsError if the property with no results is followed by other properties. Below I've copy-pasted the error message starting from lines relevant to you.

Thanks,
Angelo

  File "/home/fas/natarajan/arr56/.local/lib/python2.7/site-packages/tangos/core/halo.py", line 331, in reverse_property_cascade
    return self.calculate_for_progenitors(*args, **kwargs)
  File "/home/fas/natarajan/arr56/.local/lib/python2.7/site-packages/tangos/core/halo.py", line 327, in calculate_for_progenitors
    return self.calculate_for_descendants(*plist, **kwargs)
  File "/home/fas/natarajan/arr56/.local/lib/python2.7/site-packages/tangos/core/halo.py", line 316, in calculate_for_descendants
    return property_description.values_sanitized(results, Session.object_session(self))
  File "/home/fas/natarajan/arr56/.local/lib/python2.7/site-packages/tangos/live_calculation/__init__.py", line 160, in values_sanitized
    unsanitized_values = self.values(halos)
  File "/home/fas/natarajan/arr56/.local/lib/python2.7/site-packages/tangos/live_calculation/__init__.py", line 138, in values
    values, _ = self.values_and_description(halos)
  File "/home/fas/natarajan/arr56/.local/lib/python2.7/site-packages/tangos/live_calculation/__init__.py", line 295, in values_and_description
    values, description = c.values_and_description(mask.mask(halos))
  File "/home/fas/natarajan/arr56/.local/lib/python2.7/site-packages/tangos/live_calculation/__init__.py", line 677, in values_and_description
    description_class = properties.providing_class(self._name, sim.output_handler_class, silent_fail=True)
  File "/home/fas/natarajan/arr56/.local/lib/python2.7/site-packages/tangos/util/consistent_collection.py", line 17, in __getattr__
    return self._ensure_consistent(values, item)
  File "/home/fas/natarajan/arr56/.local/lib/python2.7/site-packages/tangos/util/consistent_collection.py", line 26, in _ensure_consistent
    return values[0]
IndexError: list index out of range

add_bh crashes when using Grp files

when using the grp files there will be no distinction between Central and non-Central BHs (halos only contain particles that are not in sub-halos). So, the bh_halo_assign function will return None for bh_halo, which will throw an error on line 119 in add_bh.py

Potential bug in latest() live calculation

Hi all,

I am filtering calculations such that they are performed solely on the progenitors of a final halo (i.e. across its full merger tree) using a syntax like include-only='latest().halo_number() == 1'.

However, latest() returns no results when applied to the final halo of the merger tree. Below is an example where output 101 is the final snapshot.

In [1]: halo = tangos.get_halo("Halo1445_DMO/%101/halo_1")                                                                                       

In [2]: halo.calculate('latest()')                                                                                                               
---------------------------------------------------------------------------
NoResultsError                            Traceback (most recent call last)
<ipython-input-2-a7c37d7dfaf7> in <module>
----> 1 halo.calculate('latest()')

~/Documents/tangos/tangos/core/halo.py in calculate(self, calculation, return_description)
    143         from .. import live_calculation
    144         calculation = live_calculation.parser.parse_property_name_if_required(calculation)
--> 145         (value,), description = calculation.values_sanitized_and_description([self], Session.object_session(self))
    146         if len(value)==1:
    147             retval = value[0]

~/Documents/tangos/tangos/live_calculation/__init__.py in values_sanitized_and_description(self, halos, load_into_session)
    130         See values_sanitized for the definition of sanitized"""
    131         values, desc = self.values_and_description(halos)
--> 132         return self._sanitize_values(values, load_into_session), desc
    133 
    134     def values(self, halos):

~/Documents/tangos/tangos/live_calculation/__init__.py in _sanitize_values(self, unsanitized_values, load_into_session, no_results_raises)
    175         for x in output_values:
    176             if len(x)==0 and no_results_raises:
--> 177                 raise NoResultsError("Calculation %s returned no results" % self)
    178 
    179         if load_into_session is not None:

NoResultsError: Calculation latest() returned no results

In [3]: halo = tangos.get_halo("Halo1445_DMO/%100/halo_1")                                                                                       

In [4]: halo.calculate('latest()')                                                                                                               
Out[4]: <Halo 'Halo1445_DMO/output_00101/halo_1' | NDM=1626215 Nstar=0 Ngas=0>

When applying this filtering to a tangos write command, the NoResultsError is systematically caught and no property calculations are ever performed on the final snapshot.

It isn't clear to me whether raising a NoResultsError is a wanted behaviour, or if latest() on the latest halo should rather return itself. Let me know whether you think this requires a fix, or if there is a more clever way to compute properties across the merger tree of a halo.

Martin

Missing last halo of HOP catalogue

Hi,

It is not entirely clear to me whether this is a Pynbody or Tangos issue, but I'll open it here anyway.

I just noticed that Tangos is consistently missing the last halo of the RAMSES+HOP catalogue. I tracked it to an inconsistency between the len() method applied on Pynbody halo catalogues and their actual index structure.

The following notebook is an example showing that the length of the catalogue returns 1825, but that halo indices run from 0 to 1825 (hence a length of 1826):

In [28]: s = pynbody.load("./output_00009/")

In [29]: h = s.halos()

In [30]: len(h)
Out[30]: 1825

In [31]: h[1825]
Out[31]: <SimSnap "./output_00009/:halo_1825" len=37>

In [32]: h[1824]
Out[32]: <SimSnap "./output_00009/:halo_1824" len=39>

In [33]: h[1826]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-33-f44840f7f2aa> in <module>()
----> 1 h[1826]

/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/halo/__init__.pyc in __getitem__(self, item)
     99             return res
    100         else:
--> 101             return self.calc_item(item)
    102 
    103     @property

/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/halo/__init__.pyc in calc_item(self, i)
     82                 return self._halos[i]
     83         else:
---> 84             h = self._get_halo(i)
     85             self._halos[i] = h  # weakref.ref(h)
     86             return h

/scratch/dp101/shared/python/anaconda2/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/halo/__init__.pyc in _get_halo(self, i)
    245         x = Halo(i, self, self.base, self._get_halo_indices(i))
    246         if len(x) == 0:
--> 247             raise ValueError("Halo %s does not exist" % (str(i)))
    248         x._descriptor = "halo_" + str(i)
    249         return x

ValueError: Halo 1826 does not exist

When enumerating the halo catalogue with the pynbody handler here:

tangos/tangos/input_handlers/pynbody.py

Lines 236 to 242 in 48e134a

	for i in range(istart, len(h)+istart):
	try:
	hi = h[i]
	if len(hi.dm) > min_halo_particles:
	yield i, len(hi.dm), len(hi.star), len(hi.gas)
	except (ValueError, KeyError) as e:
	pass

the range loop caps one index before the last and Tangos ends up missing the last halo.

It feels to me that evaluating len(halo_catalogue) with pynbody should return one more than the last index. However, it might be easier to adapt the Tangos loop to this issue in order to avoid breaking people's legacy code.

Hope this is clear,
Martin

trouble loading in user defined properties

I just updated my TANGOS version and now for some reason it does not seem to be reading my custom properties folder. As far as I can tell it is attempting to import it, but then when I attempt a live calculation that is defined in this folder, I get that the property has no providing class.

I did notice that a change in config.py in defining gathering property_modules fails to get tangos_nbodyshop_properties. Once I reverted back to the old way it was fine (currently the issue is with the following line... there is no "tangos_nbodyshop_properties" to get from the environment)

property_modules = os.environ.get("TANGOS_PROPERTY_MODULES","tangos_nbodyshop_properties") (line 35)

Still, I find that my live properties are not found, despite the following line being called without an ImportError in properties/__init__.py

importlib.import_module(pm) (line 461)

I double checked that, in fact, it is attempting to import the right names to apparently no errors. Confused as to what is happening. I have checked and my folder is being compiled with the rest of tangos and exists in my library.

Unable to load in derivable arrays in server mode

When attempting to run a calculation that uses load-mode=server, the process crashes on a step where it is attempting to load in a derivable array (in this case, "ne" for gas particles). This does not occur when running with load-mode=partial. The error comes as a KeyError for gas family.

a5310> applied to <Halo u'h1.cosmo50/h1.cosmo50PLK.1536gst1bwK1BH.003935/halo_1' | NDM=356536379 Nstar=95801512 Ngas=83888973>
Traceback (most recent call last):
  File "/u/mtremmel/anaconda/lib/python2.7/site-packages/tangos-1.0.dev0-py2.7.egg/tangos/tools/property_writer.py", line 370, in _get_prope
rty_value
    result = property_calculator.calculate(snapshot_data, db_data)
  File "/u/mtremmel/anaconda/lib/python2.7/site-packages/tangos-1.0.dev0-py2.7.egg/michaels_properties/profiles.py", line 94, in calculate
    rho_e = ps['rho_e']
  File "/u/mtremmel/anaconda/lib/python2.7/site-packages/pynbody-0.41-py2.7-linux-x86_64.egg/pynbody/analysis/profile.py", line 434, in __ge
titem__
    return self._get_profile(name)
  File "/u/mtremmel/anaconda/lib/python2.7/site-packages/pynbody-0.41-py2.7-linux-x86_64.egg/pynbody/analysis/profile.py", line 346, in _get
_profile
    self._profiles[name] = Profile._profile_registry[x[0]](self, *args)
  File "/u/mtremmel/anaconda/lib/python2.7/site-packages/tangos-1.0.dev0-py2.7.egg/michaels_properties/profiles.py", line 50, in rho_e
    n_e[i] = np.sum(subs.g['ne'][use] * subs.g['mass'][use].in_units('m_p'))/self._binsize.in_units('cm**'+str(int(self.ndim)))[i]
  File "/u/mtremmel/anaconda/lib/python2.7/site-packages/pynbody-0.41-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py", line 263, in __g
etitem__
    return self._get_array_with_lazy_actions(i)
  File "/u/mtremmel/anaconda/lib/python2.7/site-packages/pynbody-0.41-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py", line 358, in _ge
t_array_with_lazy_actions
    return self._get_array(name)
  File "/u/mtremmel/anaconda/lib/python2.7/site-packages/pynbody-0.41-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py", line 1909, in _g
et_array
    return self.base._get_family_array(name, self._unifamily, index, always_writable)
  File "/u/mtremmel/anaconda/lib/python2.7/site-packages/pynbody-0.41-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py", line 1852, in _g
et_family_array
    sl = self._family_indices[fam]
KeyError: <Family gas>```

How to use filter/mask with calculate_all() ?

Hi all,

I am in situation where I would like to calculate a property for all halos in a timestep that satisfy a condition on a different property (all halos in a mass range for example).

It feels like this should be feasible since the web interface allows you to filter halos on a given condition very easily but I can't find the right syntax to make it work within python with calculate_all().

An example of this use case is actually provided here

tangos/tangos/examples/mergers.py

Lines 31 to 42 in 7fb6052

	def most_major_mergers_since(ts, Mvir_min=0.8e12, Mvir_max=2e12, z_merger_max=4.0, no_merger_value=None):
	"""Given a timestep, calculate the most major merger ratio since a given redshift for all halos in a specified mass range
	:parameter ts - The (end) timestep to take halos from
	:parameter Mvir_min, Mvir_max - the minimum and maximum halo masses to consider
	:parameter z_merger_max - the maximum redshift to search for mergers
	:parameter no_merger_value - the value to insert for a halo if no merger was found (default None)
	:type ts tangos.core.TimeStep
	"""
	dbid, Mvir = ts.calculate_all("dbid()","Mvir")
	mask = (Mvir>Mvir_min)*(Mvir<Mvir_max)

but in a wasteful way (e.g. calculate for all halos and then mask, rather than calculate just for the masked halos)

Is this something that is easily achievable ? Where is the definition of the underlying filtering command used by the web interface ?

Failure test_web

Running test_web unit test requires the webtest package. Else nose outputs:

ModuleNotFoundError: No module named 'webtest'

One line should be added in the documentation to add this dependency. I do not know yet if this is a requirement purely for tests or for tangos.

lots of output generated by "at()" function

when using the "at()" live property calculation, I get a long output that looks something like the following

(340, 341, 0.271240234375, 0.728759765625)
(340, 341, 0.271240234375, 0.728759765625)
(448, 449, 0.81884765625004263, 0.18115234374995737)
(460, 461, 0.642333984375, 0.357666015625)
(470, 471, 0.60205078125, 0.39794921875)
(472, 473, 0.46582031250002842, 0.53417968749997158)
(468, 469, 0.73828125000004263, 0.26171874999995737)
(448, 449, 0.81884765625004263, 0.18115234374995737)
(460, 461, 0.642333984375, 0.357666015625)
(470, 471, 0.60205078125, 0.39794921875)
(472, 473, 0.46582031250002842, 0.53417968749997158)
(468, 469, 0.73828125000004263, 0.26171874999995737)
(501, 502, 0.62670898437501421, 0.37329101562498579)
(525, 526, 0.273681640625, 0.726318359375)

Issue when trying to remove a simulation

When trying to remove a simulation a simulation form the database (command tangos rm tutorial_ramses), I get the following error :

File "/Users/mrey/Documents/tangos/tangos/scripts/manager.py", line 189, in rem_simulation_timesteps
print(term.RED + "Delete simulation", sim, term.NORMAL)
TypeError: can't concat str to bytes

from https://github.com/pynbody/tangos/blob/master/tangos/scripts/manager.py#L189

I am struggling to understand exactly what is going on.

Check_delete and hanging references

I often get this message in the log :
check_deleted failed
gc reports hanging references: [(<SimSnap "/Users/mrey/Documents/simulations/tutorial_changa_blackholes/pioneer50h128.1536gst1.bwK1BH.000640" len=32962208>,), <cell at 0x10b1c0678: TipsySnap object at 0x10c256be0>]

• Is this worrying in term of memory usage ? Does this point to a misuse on my part ?

• In the case of tipsy, the reference is one line but in the case of HOP/Ramses, the print statement is the entire description of the pynbody Subsnap, leading to massive outputs (attached for only one halo) :
  screen shot 2018-01-30 at 10 22 50

Pseudo language, mpi and "&"

I often run into an annoying problem when running more complex --include-only with mpi. For example:
mpirun -np 6 tangos write dm_density_profile --with-prerequisites --for zoom_halo740 --backend mpi4py --include-only="contamination_fraction<0.01 & NDM()>500"

produces:
usage: tangos [-h] [--db-filename database_file.sqlite3] [--db-verbose]
[--verbose]
{add,recent-runs,rm,flag-duplicates,remove-duplicates,import,rollback,dump-iord,list-possible-properties,write,link,crosslink,serve}
...
tangos: error: unrecognized arguments: & NDM()>500

This error goes away if running the command without mpi as
tangos write dm_density_profile --with-prerequisites --for zoom_halo740 --include-only="contamination_fraction<0.01 & NDM()>500"

or if getting ride of the spaces between the two conditions as :
--include-only="contamination_fraction<0.01&NDM()>500"

Somehow, the mini language does not link the two conditions in the case of the first mpi command, resulting at the second being passed as an overall tangos argument.

The error also occurred when building the changa tutorial.

Are you meant to never use spaces for conditions like this ? is this a misuse on my part ?

Jobs assignment in server mode

I keep hitting the same error when writing a property with mpi in server mode as such:
mpirun -np 9 tangos write dm_density_profile--for mysim --load-mode=server --backend mpi4py

[ 1] 2018-03-02 10:45:50,881 : Gathering existing properties...
[ 1] 2018-03-02 10:45:50,890 : Successfully gathered existing properties; calculating halo properties now...
[ 1] 2018-03-02 10:45:50,890 : 5 halos to consider; 1 property calculations for each of them
Traceback (most recent call last):
File "/Users/mrey/anaconda3/bin/tangos", line 11, in
load_entry_point('tangos', 'console_scripts', 'tangos')()
File "/Users/mrey/Documents/tangos/tangos/scripts/init.py", line 57, in main
args.func(args)
File "/Users/mrey/Documents/tangos/tangos/scripts/init.py", line 14, in run
parallel_tasks.launch(obj.run_calculation_loop, 2, [])
File "/Users/mrey/Documents/tangos/tangos/parallel_tasks/init.py", line 54, in launch
backend.launch(_exec_function_or_server, num_procs, [function, args])
File "/Users/mrey/Documents/tangos/tangos/parallel_tasks/backends/mpi4py.py", line 55, in launch
function(*args)
File "/Users/mrey/Documents/tangos/tangos/parallel_tasks/init.py", line 84, in _exec_function_or_server
_server_thread()
File "/Users/mrey/Documents/tangos/tangos/parallel_tasks/init.py", line 111, in _server_thread
obj.process()
File "/Users/mrey/Documents/tangos/tangos/parallel_tasks/jobs.py", line 19, in process
self.contents, self.source, num_jobs))
RuntimeError: Number of jobs (5) expected by rank 1 is inconsistent with 0

This is generated by the Start Iteration message the first time any actual calculation is to be done:

tangos/tangos/parallel_tasks/jobs.py

Lines 16 to 18 in e53d577

	if num_jobs != self.contents:
	raise RuntimeError("Number of jobs (%d) expected by rank %d is inconsistent with %d" % (
	self.contents, self.source, num_jobs))

but I am struggling to understand why and/or to find a test for server mode to look at. Any insight ?

restarting a tangos write slower than first run

I'm doing a "tangos write shrink_center max_radius SFR_histogram --include-only="NDM()>1000"
--for h82651gs" on a zoom group run, and it got killed by the job scheduler because it ran out of queue time.
I then reran, hoping that it would pick up where it left off, but found out that just as much time, if not slightly more is taken if the properties already exist. How can I get the later timesteps processed in this simulation?

Also, I'm getting a lot of messages of the form:
/home/u14266/.local/lib/python2.7/site-packages/sqlalchemy/ext/declarative/clsregistry.py:129: SAWarning: This declarative base already contains a class with the same class name and module name as tangos.relation_finding.multi_hop.MultiHopHaloLink_vndj, and will be replaced in the string-lookup table.
Is this a problem?

TypeError in add_bh.py:scan_for_BHs()

I get the following error when running "tangos_add_bh":
File "/home/u14266/.local/lib/python2.7/site-packages/tangos/scripts/add_bh.py", line 119, in scan_for_BHs
bh_halos = bh_halos[bh_order_by_mass]
TypeError: 'NoneType' object has no attribute 'getitem'

Later in the scan_for_BHs() function there is a test "bh_halos is not None", so I assume that "None" is a valid return from bh_halo_assign(). Hence, it seems there should be a test for "bh_halos is not None on line 119.

ensure that property importing and calculations use consistent catalogue

Currently, when importing halo properties from halo stat files generated by a halo finder, the HaloStatFile class iterates through all possibilities until it lands a subclass that works. However, if multiple catalog formats exist this can be a problem. An example is if AHF was run and the IDL post processing also exists (creating simname.amiga.grp, simname.amiga.stat, etc). The IDL version will often omit halos that exist in the full catalog for a zoom run. Yet, if the AHF stat file still exists, the HaloStatFile class will use this file to import halos. Then, when properties are calculated with, for example, pynbody and the user has it set to read the IDL files instead, it will attempt to run the calculations on halos that do not exist in that catalog. Either allowing the user to specify which catalog to use at runtime or including an option to set this in config.py would be useful to avoid these errors.

Tangos breaking with python 3.7

Hi,

Similarly to (pynbody/pynbody#498), I found myself with failing unit tests for TANGOS and python 3.7.

Specifically, the tests failing are test_consistent_trees, test_gadget_rockstar, test_load_timestep, test_load_halo, test_tracking and test_yt.

An example error trace for the yt test follows:

======================================================================
ERROR: test suite for <module 'test_yt' from '/home/ucapmre/tangos/tests/test_yt.py'>
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ucapmre/anaconda3/lib/python3.7/site-packages/nose/suite.py", line 209, in run
    self.setUp()
  File "/home/ucapmre/anaconda3/lib/python3.7/site-packages/nose/suite.py", line 292, in setUp
    self.setupContext(ancestor)
  File "/home/ucapmre/anaconda3/lib/python3.7/site-packages/nose/suite.py", line 315, in setupContext
    try_run(context, names)
  File "/home/ucapmre/anaconda3/lib/python3.7/site-packages/nose/util.py", line 471, in try_run
    return func()
  File "/home/ucapmre/tangos/tests/test_yt.py", line 15, in setup
    add.SimulationAdderUpdater(output_manager).scan_simulation_and_add_all_descendants()
  File "/home/ucapmre/tangos/tangos/tools/add_simulation.py", line 40, in scan_simulation_and_add_all_descendants
    self.add_objects_to_timestep(ts, core.halo.Group)
  File "/home/ucapmre/tangos/tangos/tools/add_simulation.py", line 99, in add_objects_to_timestep
    min_halo_particles=self.min_halo_particles):
  File "/home/ucapmre/tangos/tangos/tools/add_simulation.py", line 86, in adapted
    for result in enumerate_fn(*args, **kwargs):
RuntimeError: generator raised StopIteration

It is hard to know whether some of these errors comes intrinsically from pynbody or from Tangos, because Tangos also relies on StopIteration for the enumerate_objects method of handlers:

tangos/tangos/input_handlers/pynbody.py

Lines 212 to 223 in 7f36946

	def enumerate_objects(self, ts_extension, object_typetag="halo", min_halo_particles=config.min_halo_particles):
	if self._can_enumerate_objects_from_statfile(ts_extension, object_typetag):
	for X in self._enumerate_objects_from_statfile(ts_extension, object_typetag):
	yield X
	else:
	logger.warn("No halo statistics file found for timestep %r",ts_extension)
	try:
	h = self._construct_halo_cat(ts_extension, object_typetag)
	except:
	logger.warn("Unable to read %ss using pynbody; assuming step has none", object_typetag)
	raise StopIteration

This behavior seems to have changed in python 3.7 (https://www.python.org/dev/peps/pep-0479/) and now generates a proper RuntimeError, likely the cause of these crashes.

Hope this is clear
Martin

--force and --with-prerequisites

Hi,

I am seeing an unexpected behaviour when running the following command:
mpirun -np 8 tangos write dm_density_profile --load-mode=server --with-prerequisites --for my_simulation --backend mpi4py

The calculation does not seem to check if prerequisites or the calculated property already exist in the database. It recalculates and store all the properties again as if using the --force keyword. I was not seeing this behaviour with the same command before updating my tangos architecture.

Is this a change of behaviour or a bug?
Martin

Unable to connect to tangos within web browser

When trying to open the tangos web browser with the command
tangos serve
the browser loads to a page which simply reads 'unable to connect' or 'problem loading page'.

The TANGOS_DB_CONNECTION path is set correctly, I am keeping the terminal open whilst trying to access the web browser, and all the functionality works from Python, but I am unable to open the web browser.

The text returned by the serve command is as follows:

Starting server in PID 12992.
Serving on http://localhost:6543
Serving on http://localhost:6543

Feature request: maximum of a property along the past merger history for all haloes in a snapshot

Hi,

I like to calculate the maximum of a property, say V_max or M_vir, along the past merger history for all haloes in a snapshot. Currently tangos provides this feature for a single halo using the syntax
Vpeak = halo.calculate("Vmax", "max")
but not for all haloes in a snapshot.

eagle branch fails to recognize existing properties

I'm currently using the eagle branch as there are some optimizations for gathering existing properties. However, this seems to make it so that TANGOS no longer recognizes when an asked-for property already exists and calculates it anyway (I did not force the calculation)

Notifying implementation inconsistency in user properties

Hi,

I am getting an unexpected error following my calculation of the SFR_histogram for a single halo at the latest timestep: 'tangos write SFR_histogram SFR_10Myr --for Halo600 --latest --hmax=0'.

The error reads:

Out[10]: ---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/anaconda3/lib/python3.6/site-packages/IPython/core/formatters.py in __call__(self, obj)
    691                 type_pprinters=self.type_printers,
    692                 deferred_pprinters=self.deferred_printers)
--> 693             printer.pretty(obj)
    694             printer.flush()
    695             return stream.getvalue()

~/anaconda3/lib/python3.6/site-packages/IPython/lib/pretty.py in pretty(self, obj)
    378                             if callable(meth):
    379                                 return meth(obj, self, cycle)
--> 380             return _default_pprint(obj, self, cycle)
    381         finally:
    382             self.end_group()

~/anaconda3/lib/python3.6/site-packages/IPython/lib/pretty.py in _default_pprint(obj, p, cycle)
    493     if _safe_getattr(klass, '__repr__', None) is not object.__repr__:
    494         # A user-provided repr. Find newlines and replace them with p.break_()
--> 495         _repr_pprint(obj, p, cycle)
    496         return
    497     p.begin_group(1, '<')

~/anaconda3/lib/python3.6/site-packages/IPython/lib/pretty.py in _repr_pprint(obj, p, cycle)
    691     """A pprint that just redirects to the normal repr function."""
    692     # Find newlines and replace them with p.break_()
--> 693     output = repr(obj)
    694     for idx,output_line in enumerate(output.splitlines()):
    695         if idx:

~/Documents/tangos/tangos/core/halo_data/property.py in __repr__(self)
     47             x = "<HaloProperty "
     48         if self.data_float is not None:
---> 49             return (x + self.name.text + "=%.2e" % self.data) + " of " + self.halo.short() + ">"
     50         elif self.data_array is not None:
     51             return x + self.name.text + " (array) of " + self.halo.short() + ">"

~/Documents/tangos/tangos/core/halo_data/property.py in data(self)
     65     @property
     66     def data(self):
---> 67         return self.get_data_with_reassembly_options()
     68 
     69     def get_data_with_reassembly_options(self, *options):

~/Documents/tangos/tangos/core/halo_data/property.py in get_data_with_reassembly_options(self, *options)
     68 
     69     def get_data_with_reassembly_options(self, *options):
---> 70         return extraction_patterns.HaloPropertyValueWithReassemblyOptionsGetter(*options).postprocess_data_objects([self])[0]
     71 
     72 

~/Documents/tangos/tangos/core/extraction_patterns.py in postprocess_data_objects(self, outputs)
    112 
    113     def postprocess_data_objects(self, outputs):
--> 114         return [self._postprocess_one_result(o) for o in outputs]
    115 
    116     def _setup_data_mapper(self, property_object):

~/Documents/tangos/tangos/core/extraction_patterns.py in <listcomp>(.0)
    112 
    113     def postprocess_data_objects(self, outputs):
--> 114         return [self._postprocess_one_result(o) for o in outputs]
    115 
    116     def _setup_data_mapper(self, property_object):

~/Documents/tangos/tangos/core/extraction_patterns.py in _postprocess_one_result(self, property_object)
    132         if hasattr(self._providing_class, 'reassemble'):
    133             instance = self._providing_class(property_object.halo.timestep.simulation)
--> 134             return instance.reassemble(property_object, *self._options)
    135         else:
    136             self._setup_data_mapper(property_object)

~/Documents/tangos/tangos/properties/pynbody/SF.py in reassemble(self, *options)
     35 
     36     def reassemble(self, *options):
---> 37         reassembled = super(StarFormHistogram, self).reassemble(*options)
     38         return reassembled/1e9 # Msol per Gyr -> Msol per yr
     39 

~/Documents/tangos/tangos/properties/__init__.py in reassemble(self, property, reassembly_type)
    248 
    249         if reassembly_type=='major':
--> 250             return self._reassemble_using_finding_strategy(property, strategy = rfs.MultiHopMajorProgenitorsStrategy)
    251         elif reassembly_type=='major_across_simulations':
    252             return self._reassemble_using_finding_strategy(property, strategy = rfs.MultiHopMajorProgenitorsStrategy,

~/Documents/tangos/tangos/properties/__init__.py in _reassemble_using_finding_strategy(self, property, strategy, strategy_kwargs)
    276         for t_i, hist_i in zip(t, stack):
    277             end = self.bin_index(t_i)
--> 278             start = end - len(hist_i)
    279             valid = hist_i == hist_i
    280             if t_i != previous_time:

TypeError: object of type 'numpy.float64' has no len()

I think the problem is generated because there is only one SFR_histogram property available across the merger tree of this halo, and hence a number (which has no length) is retruned rather than an array.

Am I misusing this property (in the sense that it should always be generated across the full merger tree) or is this a bug?

The error also generates a catastrophic failure of the web server preventing from accessing the page of this halo (but not the timestep page or other halo pages). I think this is because the webserver tries by default to reassemble histogram properties which fail with the same error as above.

Thanks
Martin

tangos won't recognise new property module

I am trying to add a custom property to (a local copy of) a database. The code lives here:
https://github.com/milchada/tangos_properties_uchadaya/
The property of interest is described in vdisp.py. iPython successfully imports this module. We added the following to tangos/properties/init.py:

def _import_configured_property_modules():
    from ..config import property_modules
    for pm in property_modules:
        if pm=="": continue
        print(pm)

So tangos prints the names of the modules it tries to import, i.e. TANGOS_PROPERTY_MODULES. It prints my module name.
However, when I ask for the property class of the property I have defined, I get:
NameError: No providing class for property v_disp_profile
This error regardless of which directory I try the import from.

What am I missing?

Crosslinking simulations with different number of timesteps

I am trying to crosslink two simulations that do not have the same time sampling. This is currently allowed by tangos.

The calculation fails because in the case that sim2 has 100 snapshots against 50 for sim1, the lines

tangos/tangos/tools/crosslink.py

Lines 150 to 152 in 0054bbe

	back_cat = output_handler_2.match_objects(ts2.extension, ts1.extension, halo_min, halo_max, dmonly, threshold,
	core.halo.Halo.object_typetag_from_code(object_typecode),
	output_handler_for_ts2= output_handler_1)

assumes that handler 2 can read snapshot 100 in sim1 which rightfully throws:
'OSError: File '/Users/mrey/Documents/sim1/output_00100': format not understood or does not exist'

If the cross linking tools was not designed to work with unequal number fo snapshots, a check at the beginning of the calculation would do.

Contamination fraction is inconsistent between load-modes

Hi all,

I noticed today that the result of the calculation "contamination_fraction" is inconsistent between being done with different load-modes.

Currently, "contamination_fraction" looks into the loaded particles, find the minimum mass of those and calculate the fraction of particles more massive than this minimum mass.

1. In a load-mode in which the entire snapshot is loaded, the minimum mass is always the overall minimum mass in the simulation (zoom resolution) so the calculation returns 1.0 for unzoomed halo, 0.X for contaminated halos and 0.0 for uncontaminated zoomed halos.

2. In a load-mode in which only halo data is loaded (e.g. server), the minimum mass is calculated locally, not necessarily knowing that there exists a zoom region elsewhere in the simulation. For example, a pure unzoomed halo will have minimum_mass 64 and zero particles above this value, while a pure zoomed halo will have minimum_mass 8.0 and zero particles above. The calculation then returns 0.0 for unzoomed halos, 0.X for contaminated halos and 0.0 for uncontaminated zoomed halos.

I am not sure which one should be the correct behaviour (number 1 for me) but the fact that it is inconsistent between load-mode is highly confusing.

Hope this is clear
Martin

Include only"Stored property" fails when no halos in timestep

When executing a write command using --include-only="whatever condition on a stored property previously calculated", I get :
File "/Users/mrey/Documents/tangos/tangos/live_calculation/init.py", line 138, in values
values, _ = self.values_and_description(halos)
File "/Users/mrey/Documents/tangos/tangos/live_calculation/init.py", line 389, in values_and_description
iv, id = self._input_value_and_description(input, halos)
File "/Users/mrey/Documents/tangos/tangos/live_calculation/init.py", line 500, in _input_value_and_description
return super(BuiltinFunction, self)._input_value_and_description(input_id, halos)
File "/Users/mrey/Documents/tangos/tangos/live_calculation/init.py", line 399, in _input_value_and_description
val, desc = input.values_and_description(halos)
File "/Users/mrey/Documents/tangos/tangos/live_calculation/init.py", line 678, in values_and_description
description_class = properties.providing_class(self._name, sim.output_handler_class, silent_fail=True)
File "/Users/mrey/Documents/tangos/tangos/util/consistent_collection.py", line 17, in getattr
return self._ensure_consistent(values, item)
File "/Users/mrey/Documents/tangos/tangos/util/consistent_collection.py", line 26, in _ensure_consistent
return values[0]

This is due to the fact that the first time steps of the simulation have no halos, hence no stored properties, hence bad access at 0. This is surprising because these time steps are successfully skipped for live properties.

The write command used is :
tangos write dm_density_profile --with-prerequisites --for zoom_halo740 --include-only="contamination_fraction<0.01"

Pynbody handler always crosslinks with itself

Hi,
I am running again in an issue fixed by 239c522: crosslinking two simulations always map halo 12 of simulation1/timestep5 with halo 12 of simulation2/timestep5, i.e. linking with itself and not taking into account particle ids.

I believe this bug was introduced following commit 50bd7c0:

• This line checks whether some snapshots are stored in memory, based on the timestep extension.
  

  tangos/tangos/input_handlers/__init__.py

  Lines 117 to 118 in 854edc7

  	ts_hash = hash((ts_extension,mode,type(self)))
  	stored_timestep = _loaded_timesteps.get(ts_hash, None)

• In the case of crosslinking the extension of simulation1/timestep5 and simulation2/timestep5 are actually the same, for example "output_0005" in my case.

• Rather than loading simulation2/timestep5, the handler then reuses the memory stored simulation1/timestep5, hence ending up linking with itself.

The easiest is probably for the hash to check with _extension_to_filename() rather than just extension but I don't want to break what you tried to achieve with these commits.

Let me know what you think!

Failure import_dependencies

From a fresh install, I tried to run the tests to find errors in the import/include tests namely:

sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string '/Users/mrey/Science//tangos_data.db'

The default config.py adds /Science/ for mac systems which does not exists in my case.
Editing my own config solved the problem but the default should check for existence before adding the path.

finder_id problem for MPI AHF runs

the finder_id is meant to map to the original order of the halo catalog, not the unique ID number assigned by the halo finder necessarily. For MPI runs (e.g. with AHF but also similar for others) the halo ID is a very large number not associated with its final position within the halo catalog (i.e. there is no 1, but rather very large identifying numbers). In pynbody, the halo catalog is read in as a list of halos and halos are extracted based on their position within that list (h.load_copy(5) will load in the 5th halo, for example). A while ago, the definition of finder_id was changed such that it was associated directly with the ID number given in the catalog. This works fine for normal AHF where the order = ID number. However, this will create problems for catalogs created with MPI, where this is not the case.

importing subhalos from AHF files

It would be great if tangos could import the subhalo information from the AHF files and store them similarly to pynbody under a key "children"

Changa+BH tutorial

A few issues arose when going through the changa BH tutorial:

• Missing --for in the command "tangos_add_bh tutorial_changa_blackholes". Otherwise, one gets an Unrecognised argument error. This is purely a documentation problem.

• Failure of "tangos_add_bh --for tutorial_changa_blackholes" when run with mpi producing:

Traceback (most recent call last):
  File "/Users/mrey/anaconda3/bin/tangos_add_bh", line 11, in <module>
    load_entry_point('tangos', 'console_scripts', 'tangos_add_bh')()
  File "/Users/mrey/Documents/tangos/tangos/scripts/add_bh.py", line 335, in main
    parallel_tasks.launch(run)
  File "/Users/mrey/Documents/tangos/tangos/parallel_tasks/__init__.py", line 54, in launch
    backend.launch(_exec_function_or_server, num_procs, [function, args])
  File "/Users/mrey/Documents/tangos/tangos/parallel_tasks/backends/mpi4py.py", line 55, in launch
    function(*args)
  File "/Users/mrey/Documents/tangos/tangos/parallel_tasks/__init__.py", line 86, in _exec_function_or_server
    function(*args)
  File "/Users/mrey/Documents/tangos/tangos/scripts/add_bh.py", line 331, in run
    timelink_bh(args.sims, session)
  File "/Users/mrey/Documents/tangos/tangos/scripts/add_bh.py", line 308, in timelink_bh
    generate_halolinks(session, fname, pairs)
  File "/Users/mrey/Documents/tangos/tangos/scripts/add_bh.py", line 216, in generate_halolinks
    for ts1, ts2 in parallel_tasks.distributed(pairs):
  File "/Users/mrey/Documents/tangos/tangos/parallel_tasks/jobs.py", line 48, in parallel_iterate
    MessageStartIteration(len(task_list)).send(0)
TypeError: object of type 'generator' has no len()

Numpy compatibility

I faced an error today which I am posting here as a reference/documentation.

I (by mistake) generated a database storing numpy arrays with numpy 1.16. When retrieving the properties through both python and the web server on a different computer with numpy 1.15, I received:

Traceback (most recent call last):
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/tweens.py", line 12, in _error_handler
    response = request.invoke_exception_view(exc_info)
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/view.py", line 756, in invoke_exception_view
    raise HTTPNotFound
pyramid.httpexceptions.HTTPNotFound: The resource could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/anaconda3/lib/python3.6/site-packages/waitress/channel.py", line 338, in service
    task.service()
  File "/anaconda3/lib/python3.6/site-packages/waitress/task.py", line 169, in service
    self.execute()
  File "/anaconda3/lib/python3.6/site-packages/waitress/task.py", line 399, in execute
    app_iter = self.channel.server.application(env, start_response)
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/router.py", line 270, in __call__
    response = self.execution_policy(environ, self)
  File "/anaconda3/lib/python3.6/site-packages/pyramid_retry-0.5-py3.6.egg/pyramid_retry/__init__.py", line 134, in retry_policy
    return request.invoke_exception_view(reraise=True)
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/view.py", line 755, in invoke_exception_view
    reraise_(*exc_info)
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/compat.py", line 148, in reraise
    raise value
  File "/anaconda3/lib/python3.6/site-packages/pyramid_retry-0.5-py3.6.egg/pyramid_retry/__init__.py", line 114, in retry_policy
    response = router.invoke_request(request)
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/router.py", line 249, in invoke_request
    response = handle_request(request)
  File "/anaconda3/lib/python3.6/site-packages/pyramid_tm-2.2-py3.6.egg/pyramid_tm/__init__.py", line 171, in tm_tween
    reraise(*exc_info)
  File "/anaconda3/lib/python3.6/site-packages/pyramid_tm-2.2-py3.6.egg/pyramid_tm/compat.py", line 36, in reraise
    raise value
  File "/anaconda3/lib/python3.6/site-packages/pyramid_tm-2.2-py3.6.egg/pyramid_tm/__init__.py", line 136, in tm_tween
    response = handler(request)
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/tweens.py", line 41, in excview_tween
    response = _error_handler(request, exc)
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/tweens.py", line 16, in _error_handler
    reraise(*exc_info)
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/compat.py", line 148, in reraise
    raise value
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/tweens.py", line 39, in excview_tween
    response = handler(request)
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/router.py", line 156, in handle_request
    view_name
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/view.py", line 642, in _call_view
    response = view_callable(context, request)
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/viewderivers.py", line 410, in viewresult_to_response
    result = view(context, request)
  File "/anaconda3/lib/python3.6/site-packages/pyramid-1.9.1-py3.6.egg/pyramid/viewderivers.py", line 148, in _requestonly_view
    response = view(request)
  File "/Users/martinrey/Documents/tangos/tangos/web/views/halo_data.py", line 287, in array_plot
    val, property_info = halo.calculate(name, True)
  File "/Users/martinrey/Documents/tangos/tangos/core/halo.py", line 145, in calculate
    (value,), description = calculation.values_sanitized_and_description([self], Session.object_session(self))
  File "/Users/martinrey/Documents/tangos/tangos/live_calculation/__init__.py", line 131, in values_sanitized_and_description
    values, desc = self.values_and_description(halos)
  File "/Users/martinrey/Documents/tangos/tangos/live_calculation/__init__.py", line 688, in values_and_description
    values = self.values(halos)
  File "/Users/martinrey/Documents/tangos/tangos/live_calculation/__init__.py", line 683, in values
    ret[0, i] = self._extraction_pattern.get_from_cache(h, self._name_id)[0]
  File "/Users/martinrey/Documents/tangos/tangos/core/extraction_patterns.py", line 61, in get_from_cache
    return self.postprocess_data_objects(return_vals)
  File "/Users/martinrey/Documents/tangos/tangos/core/extraction_patterns.py", line 114, in postprocess_data_objects
    return [self._postprocess_one_result(o) for o in outputs]
  File "/Users/martinrey/Documents/tangos/tangos/core/extraction_patterns.py", line 114, in <listcomp>
    return [self._postprocess_one_result(o) for o in outputs]
  File "/Users/martinrey/Documents/tangos/tangos/core/extraction_patterns.py", line 137, in _postprocess_one_result
    return self._mapper.get(property_object)
  File "/Users/martinrey/Documents/tangos/tangos/core/data_attribute_mapper.py", line 112, in get
    return self.unpack(getattr(db_object, self._attribute_name))
  File "/Users/martinrey/Documents/tangos/tangos/core/data_attribute_mapper.py", line 173, in unpack
    return self._unpack_compressed(packed)
  File "/Users/martinrey/Documents/tangos/tangos/core/data_attribute_mapper.py", line 161, in _unpack_compressed
    return pickle_loads(zlib.decompress(packed[2:]))
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

I tracked this thanks to this issue (numpy/numpy#12825) to unpickling a numpy array with a numpy version different to the one it was pickled with. I can confirm that adapting my numpy version to the one which I used to save the array into the database solves this issue.

I suspect this is more of an issue on the side of numpy, to ensure that versions are backward compatible. I am unsure whether there are active actions to be taken on the TANGOS side?

Martin

Generating images as tangos properties

I have been trying to generate dark matter projected images around my halos of interest (with pynbody handler). Following the baryonic example given in

tangos/tangos/properties/pynbody/images.py

Line 4 in 811444e

class BaryonicImages(SphericalRegionPropertyCalculation):

I derived the following class:

class DarkMatterCloseUpImages(PynbodyPropertyCalculation):
    names = "dm_closeup_image_x", "dm_closeup_image_x"

    @classmethod
    def plot_extent(cls):
        return 1000.0

    @classmethod
    def plot_xlabel(cls):
        return "x/a kpc"

    @classmethod
    def plot_ylabel(cls):
        return "y/a kpc"

    def plot_clabel(self):
         return r"M$_{\odot}$ kpc$^{-2}$"

    def requires_property(self):
        return ["shrink_center", "max_radius"]

    def region_specification(self, db_data):
        import pynbody
        side = pynbody.array.SimArray(4 * db_data['max_radius'], "kpc")
        xcenter = db_data['shrink_center'][0]
        ycenter = db_data['shrink_center'][1]
        zcenter = db_data['shrink_center'][2]

        x1 = xcenter - side/2
        y1 = ycenter - side/2
        z1 = zcenter - side/2
        x2 = xcenter + side/2
        y2 = ycenter + side/2
        z2 = zcenter + side/2
        return pynbody.filt.Cuboid(x1, y1=y1, z1=z1, x2=x2, y2=y2, z2=z2)

    def _render_projected(self, f, size):
        import pynbody.plot
        im = pynbody.plot.sph.image(f, qty='rho', width=size, units="Msol kpc^-2", noplot=True)
        return im

    @centred_calculation
    def calculate(self, particle_data, existing_properties):
        plot_size = 4 * existing_properties['max_radius']

        # This is the key. Pynbody tries to calculate the smooth array on the full snapshot for some reason ....
        particle_data.d['smooth']

        im_z = self._render_projected(particle_data.dm, plot_size)
        with particle_data.rotate_y(90):
            im_x = self._render_projected(particle_data.dm, plot_size)
        return im_z, im_x

My first question is on plot_extent(). Let's say I want to make a plot of the surroundings of the halo up to 4 radii. How can this be incorporated in the plot_extent that is a class method ?

The second and main problem is that this calculation fails every time throwing:

Traceback (most recent call last):
  File "/Users/mrey/Documents/tangos/tangos/tools/property_writer.py", line 382, in _get_property_value
    result = property_calculator.calculate(snapshot_data, db_data)
  File "/Users/mrey/Documents/tangos/tangos/properties/pynbody/centring.py", line 60, in new_fn
    return fn(self, halo, existing_properties)
  File "/Users/mrey/Documents/tangos/tangos/properties/pynbody/images.py", line 103, in calculate
    particle_data.d['smooth']
  File "/Users/mrey/anaconda3/lib/python3.6/site-packages/pynbody-0.43-py3.6-macosx-10.7-x86_64.egg/pynbody/snapshot/__init__.py", line 264, in __getitem__
    return self._get_array_with_lazy_actions(i)
  File "/Users/mrey/anaconda3/lib/python3.6/site-packages/pynbody-0.43-py3.6-macosx-10.7-x86_64.egg/pynbody/snapshot/__init__.py", line 357, in _get_array_with_lazy_actions
    self.__derive_if_required(name)
  File "/Users/mrey/anaconda3/lib/python3.6/site-packages/pynbody-0.43-py3.6-macosx-10.7-x86_64.egg/pynbody/snapshot/__init__.py", line 372, in __derive_if_required
    self._derive_array(name)
  File "/Users/mrey/anaconda3/lib/python3.6/site-packages/pynbody-0.43-py3.6-macosx-10.7-x86_64.egg/pynbody/snapshot/__init__.py", line 1942, in _derive_array
    self.base._derive_array(array_name, self._unifamily)
  File "/Users/mrey/anaconda3/lib/python3.6/site-packages/pynbody-0.43-py3.6-macosx-10.7-x86_64.egg/pynbody/snapshot/__init__.py", line 1774, in _derive_array
    self.base._derive_array(array_name, fam)
  File "/Users/mrey/anaconda3/lib/python3.6/site-packages/pynbody-0.43-py3.6-macosx-10.7-x86_64.egg/pynbody/snapshot/__init__.py", line 1471, in _derive_array
    result = fn(self[fam])
  File "/Users/mrey/anaconda3/lib/python3.6/site-packages/pynbody-0.43-py3.6-macosx-10.7-x86_64.egg/pynbody/sph/__init__.py", line 137, in smooth
    self.kdtree.populate('hsm', config['sph']['smooth-particles'])
  File "/Users/mrey/anaconda3/lib/python3.6/site-packages/pynbody-0.43-py3.6-macosx-10.7-x86_64.egg/pynbody/sph/kdtree.py", line 114, in populate
    smx = kdmain.nn_start(self.kdtree, int(nn), self.boxsize)
ValueError: The particles span a region larger than the specified boxsize

I tracked it down to a problem when calculating the smooth pynbody array rather than the actual plotting itself. I think pynbody or tangos tries to calculate the smooth array on the ancestor snapshot rather than the particle data. Any insight would be much appreciated.

issues with using more than one mpiproc per node

When I run a calculation using 50 processes, with 1 process per node things seem to work alright. However, running the same calculation on the same simulation steps/halos with 2 processes per node and 25 nodes (rather than 50), I quickly get a database error. I'm not sure if this is a problem with this specific filesystem or not.... The following is the database error that comes up. It happens quickly enough that it seems to be simply when the code is attempting to read in the existing database properties. This is all done on nobackupp2 on pleiades and using the most up-to-date master branch (except the very latest updates in the past 12 hours or so).

sqlalchemy.exc.DatabaseError: (sqlite3.DatabaseError) database disk image is malformed [SQL: u'SELECT creators.id AS creators_id, creators.command_line AS creators_command_line, creators.dtime AS creators_dtime, creators.host AS creators_host, creators.username AS creators_username, creators.cwd AS creators_cwd \nFROM creators \nWHERE creators.id = ?\n LIMIT ? OFFSET ?'] [parameters: (234, 1, 0)]

TypeError error when linking

I am using tangos + pynbody + RAMSES and running into the following error:

2020-01-17 15:37:55,143 : generating pairs of timesteps
2020-01-17 15:38:27,150 : Linking <TimeStep 'delete_me/output_00101' z=0.34 t=9.58 Gyr> and <TimeStep 'delete_me/output_00102' z=0.33 t=9.69 Gyr>              
2020-01-17 15:38:27,181 : Gathering halo information for <TimeStep 'delete_me/output_00101' z=0.34 t=9.58 Gyr> and <TimeStep 'delete_me/output_00102' z=0.33 t=9.69 Gyr>                                                                                                                                                      
2020-01-17 15:43:07,808 : Exception during attempt to crosslink timesteps <TimeStep 'delete_me/output_00101' z=0.34 t=9.58 Gyr> and <TimeStep 'delete_me/output_00102' z=0.33 t=9.69 Gyr>                                                                                                                                     
Traceback (most recent call last):                                                                                                                             
  File "/home/ccadiou/codes/tangos/tangos/tools/crosslink.py", line 138, in crosslink_ts                                                                       
    output_handler_for_ts2=output_handler_2)                                                                                                                   
  File "/home/ccadiou/codes/tangos/tangos/input_handlers/pynbody.py", line 322, in match_objects                                                               
    only_family=pynbody.family.dm, groups_1=h1, groups_2=h2)                                                                                                   
  File "/home/ccadiou/.conda/envs/python3/lib/python3.7/site-packages/pynbody/bridge/__init__.py", line 124, in fuzzy_match_catalog                            
    transfer_matrix = self.catalog_transfer_matrix(min_index,max_index,groups_1,groups_2,use_family,only_family)                                               
  File "/home/ccadiou/.conda/envs/python3/lib/python3.7/site-packages/pynbody/bridge/__init__.py", line 180, in catalog_transfer_matrix                        
    g1 = groups_1.get_group_array(family=only_family)[restriction_start_indices]                                                                               
TypeError: get_group_array() got an unexpected keyword argument 'family'                                                                                       
2020-01-17 15:43:07,864 : Linking <TimeStep 'delete_me/output_00102' z=0.33 t=9.69 Gyr> and <TimeStep 'delete_me/output_00103' z=0.32 t=9.77 Gyr>              
2020-01-17 15:43:07,896 : Gathering halo information for <TimeStep 'delete_me/output_00102' z=0.33 t=9.69 Gyr> and <TimeStep 'delete_me/output_00103' z=0.32 t=9.77 Gyr>    ```

Unable to populate AHF properties

When trying to import AHF properties to a dark matter only Romulus database using
tangos_import_from_ahf [property]
Tangos spits out lines saying that it processes each timestep in order, but no properties are written to the database.

Recent-runs does not recognize tangos_import_from_ahf was run. It does recognize when tangos link and tangos write are run.

AHF files exist in the directory for each timestep, and are all populated. When running:
tangos add [simulation]
Tangos spits out that the AHFCatalogue was loaded, so it presumably is able to read in the AHF files.

Running the same commands for tutorial_changa correctly loads in AHF properties.

The DMO database was created using the --quicker flag.
@emapple

Tangos cannot load output

Hello,

I've tried producing Tangos databases for some star-forming RAMSES simulations, only I am getting the "Could not load /sim_path/output_#####" error when I run this command:

tangos add simulation --min-particles 1000 --no-renumber

I've not met this error on previous simulations using the exact same Tangos distribution, so I think the problem is that either the RAMSES or HOP outputs are using some long-integers rather than standard integers. Would this cause this sort of problem?

Halo #1 in HOP+Ramses

When importing the tutorial_ramses (and one of my own ramses simulation), halo #1 in the database is always every unbound particle of the simulation. It is obviously the most massive object in the simulation.

Is there an option to specify this behaviour and avoid this ?