dnarayanan / caesar Goto Github PK
View Code? Open in Web Editor NEWLicense: Other
caesar's People
Contributors
Stargazers
Watchers
Forkers
saethlin smlower lq3552 michaeljwilson rennehan currodri weiguangcui astro-kevin kyleaoman ewanbjones98 jszpila314caesar's Issues
Unyt_array does not have attribute 'to_octree'
yt : [INFO ] 2022-02-28 12:47:40,006 Calculating time from 1.667e-01 to be 3.711e+16 seconds
yt : [INFO ] 2022-02-28 12:47:40,008 Assuming length units are in kpc/h (comoving)
yt : [INFO ] 2022-02-28 12:47:40,197 Parameters: current_time = 3.71136025935506e+16 s
yt : [INFO ] 2022-02-28 12:47:40,197 Parameters: domain_dimensions = [1 1 1]
yt : [INFO ] 2022-02-28 12:47:40,197 Parameters: domain_left_edge = [0. 0. 0.]
yt : [INFO ] 2022-02-28 12:47:40,198 Parameters: domain_right_edge = [6000. 6000. 6000.]
yt : [INFO ] 2022-02-28 12:47:40,198 Parameters: cosmological_simulation = 1
yt : [INFO ] 2022-02-28 12:47:40,198 Parameters: current_redshift = 4.999988038637683
yt : [INFO ] 2022-02-28 12:47:40,198 Parameters: omega_lambda = 0.6911
yt : [INFO ] 2022-02-28 12:47:40,198 Parameters: omega_matter = 0.3089
yt : [INFO ] 2022-02-28 12:47:40,199 Parameters: omega_radiation = 0.0
yt : [INFO ] 2022-02-28 12:47:40,199 Parameters: hubble_constant = 0.6774
Generating md5 hash: 2776it [00:01, 2033.38it/s]
yt : [INFO ] 2022-02-28 12:47:42,034 Allocating for 3.361e+07 particles
Loading particle index:
100%|███████████████████████████████████████████████████████| 72/72 [00:00<00:00, 493.22it/s]
yt : [INFO ] 2022-02-28 12:47:50,735 member_search() running on 8 cores
yt : [INFO ] 2022-02-28 12:47:52,916 No black holes found, RAM=0.5382 GB
The particle types will be loaded: ['PartType0', 'PartType4', 'PartType1']
yt : [INFO ] 2022-02-28 12:47:52,960 Calculated Omega_Baryon=0.0486007 and 256^3 effective resolution
yt : [INFO ] 2022-02-28 12:47:52,967 Using b=0.2 for halos
yt : [INFO ] 2022-02-28 12:47:52,968 Running 3D FOF to get Halo IDs, LL=6.91889, RAM=0.4757 GB
Traceback (most recent call last):
File "do_caesar.py", line 41, in <module>
obj.member_search(haloid='fof', b_halo=0.2, dust=False, blackholes=False, nproc=8)
File "/home/local/Anaconda3-2020.02/envs/py3samir/lib/python3.7/site-packages/caesar-0.2b0-py3.7-linux-x86_64.egg/caesar/main.py", line 340, in member_search
fubar_halo(self)
File "/home/local/Anaconda3-2020.02/envs/py3samir/lib/python3.7/site-packages/caesar-0.2b0-py3.7-linux-x86_64.egg/caesar/fubar_halo.py", line 64, in fubar_halo
halos.load_haloid()
File "/home/local/Anaconda3-2020.02/envs/py3samir/lib/python3.7/site-packages/caesar-0.2b0-py3.7-linux-x86_64.egg/caesar/fof6d.py", line 48, in load_haloid
self.run_caesar_fof(self.obj)
File "/home/local/Anaconda3-2020.02/envs/py3samir/lib/python3.7/site-packages/caesar-0.2b0-py3.7-linux-x86_64.egg/caesar/fof6d.py", line 86, in run_caesar_fof
haloid_all = fof(self.obj, pos, LL, group_type='halo') # run FOF
File "/home/local/Anaconda3-2020.02/envs/py3samir/lib/python3.7/site-packages/caesar-0.2b0-py3.7-linux-x86_64.egg/caesar/fubar.py", line 67, in fof
ot = pos.to_octree()
AttributeError: 'unyt_array' object has no attribute 'to_octree'
This is occuring when I try to run the member_search() method.
obj.member_search(haloid='fof', b_halo=0.2, dust=False, blackholes=False, nproc=8)
H5py v3 update breaks caesar usage
Discovered on ilifu - the newest v3 of h5py causes caesar to fail to load files; e.g.:
In [1]: import caesar
...: infile = '/idia/data/laduma/SIMBA/m100n1024/s50/Groups/m100n1024_151.hdf5'
...: infile2 = '/idia/data/laduma/SIMBA/m50n512/s50/Groups/m50n512_151.hdf5'
...: sim = caesar.load(infile2)
yt : [INFO ] 2020-12-10 11:19:57,929 Opening /idia/data/laduma/SIMBA/m50n512/s50/Groups/m50n512_151.hdf5AttributeError Traceback (most recent call last)
in
3 infile2 = '/idia/data/laduma/SIMBA/m50n512/s50/Groups/m50n512_151.hdf5'
4 #infile = '/idia/data/laduma/SIMBA/m100n1024/s50/Groups/m100n1024_151.hdf5'
----> 5 sim = caesar.load(infile2)
/anaconda3/lib/python3.6/site-packages/caesar-0.2b0-py3.6-linux-x86_64.egg/caesar/loader.py in load(filename)
586
587 def load(filename):
--> 588 return CAESAR(filename)
/anaconda3/lib/python3.6/site-packages/caesar-0.2b0-py3.6-linux-x86_64.egg/caesar/loader.py in init(self, filename)
205
206 self.unit_registry = UnitRegistry.from_json(
--> 207 hd.attrs['unit_registry_json'].decode('utf8'))
208
209 # Load the information about the simulation itself
AttributeError: 'str' object has no attribute 'decode'
As the error mentions, it is because decode('utf8') does not have support in python3, so the update for h5py likely breaks the support for it.
computation with nproc=2 report an error, while nproc=1 does not
Please see the attache error report:
In [6]: obj.member_search(haloid='snap',fof6d_file='/home2/weiguang/data6/CL300_Caesar/NewMDCLUSTER_0001/FoF6D/fof6d_snap_055.hdf5',nproc=1,lowres=[2,3])
yt : [INFO ] 2020-10-12 12:15:17,565 member_search() running on 1 cores
yt : [INFO ] 2020-10-12 12:15:17,771 Calculated Omega_Baryon=0.0476929 and 3842^3 effective resolution
yt : [INFO ] 2020-10-12 12:15:17,772 Using FOF Halo ID from snapshots, RAM=2.245 GB
yt : [INFO ] 2020-10-12 12:15:18,127 Initializing member search, loading particles, RAM=2.348 GB
yt : [INFO ] 2020-10-12 12:15:20,331 Loaded particle data, RAM=2.42 GB
yt : [INFO ] 2020-10-12 12:15:21,333 Loaded baryon data, RAM=2.406 GB
yt : [INFO ] 2020-10-12 12:15:29,799 Found 1918 valid halos, loaded indexes, RAM=0.8747 GB
yt : [INFO ] 2020-10-12 12:15:29,884 Calculating properties for 1918 halos (nproc=1), RAM=0.8747 GB
yt : [INFO ] 2020-10-12 12:15:35,515 Computed properties for 1918 halos, RAM=0.9176 GB
yt : [INFO ] 2020-10-12 12:15:39,335 Reading galaxy membership from fof6d file /home2/weiguang/data6/CL300_Caesar/NewMDCLUSTER_0001/FoF6D/fof6d_snap_055.hdf5
yt : [INFO ] 2020-10-12 12:15:39,852 Found 642 valid galaxies, loaded indexes, RAM=0.9318 GB
yt : [INFO ] 2020-10-12 12:15:39,881 Calculating properties for 642 galaxies (nproc=1), RAM=0.9318 GB
yt : [INFO ] 2020-10-12 12:15:41,339 Computed properties for 642 galaxies, RAM=0.9451 GB
yt : [INFO ] 2020-10-12 12:15:42,655 Doing HI/H2 calculation for 642 galaxies in 1918 halos, RAM=0.9498 GB
yt : [INFO ] 2020-10-12 12:15:42,893 Doing aperture mass calculation R=30 kpc, RAM=0.951 GB
yt : [INFO ] 2020-10-12 12:15:42,949 filled galaxy_lists, RAM=0.9515 GB
yt : [INFO ] 2020-10-12 12:15:44,164 Assigning galaxies to halos
yt : [INFO ] 2020-10-12 12:15:44,192 Linking galaxies and halos
yt : [INFO ] 2020-10-12 12:15:44,194 Assigning central galaxies
yt : [INFO ] 2020-10-12 12:15:44,200 Creating sublists
yt : [INFO ] 2020-10-12 12:15:44,202 Checking all objects for contamination. Lowres Types: [2, 3]
yt : [INFO ] 2020-10-12 12:15:44,202 Gathering low-res particles and constructing tree
/home2/weiguang/.local/lib/python3.6/site-packages/caesar-0.2b0-py3.6-linux-x86_64.egg/caesar/group.py:703: RuntimeWarning: invalid value encountered in double_scalars
self.contamination = lrmass / halo.virial_quantities['m200c'].d
In [7]: obj.member_search(haloid='snap',fof6d_file='/home2/weiguang/data6/CL300_Caesar/NewMDCLUSTER_0001/FoF6D/fof6d_snap_055.hdf5',nproc=2,lowres=[2,3])
yt : [INFO ] 2020-10-12 12:20:39,576 member_search() running on 2 cores
yt : [INFO ] 2020-10-12 12:20:39,577 Using FOF Halo ID from snapshots, RAM=2.415 GB
yt : [INFO ] 2020-10-12 12:20:39,901 Initializing member search, loading particles, RAM=2.518 GB
yt : [INFO ] 2020-10-12 12:20:41,879 Loaded particle data, RAM=2.561 GB
yt : [INFO ] 2020-10-12 12:20:42,792 Loaded baryon data, RAM=2.572 GB
yt : [INFO ] 2020-10-12 12:20:50,877 Found 1918 valid halos, loaded indexes, RAM=2.576 GB
yt : [INFO ] 2020-10-12 12:20:50,959 Calculating properties for 1918 halos (nproc=2), RAM=2.576 GB
yt : [INFO ] 2020-10-12 12:20:56,233 Computed properties for 1918 halos, RAM=2.638 GB
yt : [INFO ] 2020-10-12 12:21:00,675 Reading galaxy membership from fof6d file /home2/weiguang/data6/CL300_Caesar/NewMDCLUSTER_0001/FoF6D/fof6d_snap_055.hdf5
yt : [INFO ] 2020-10-12 12:21:01,183 Found 642 valid galaxies, loaded indexes, RAM=2.633 GB
yt : [INFO ] 2020-10-12 12:21:01,211 Calculating properties for 642 galaxies (nproc=2), RAM=2.633 GB
yt : [INFO ] 2020-10-12 12:21:02,361 Computed properties for 642 galaxies, RAM=2.633 GB
yt : [INFO ] 2020-10-12 12:21:03,705 Doing HI/H2 calculation for 642 galaxies in 1918 halos, RAM=2.633 GB
yt : [INFO ] 2020-10-12 12:21:03,954 Doing aperture mass calculation R=30 kpc, RAM=2.633 GB
yt : [INFO ] 2020-10-12 12:21:04,005 filled galaxy_lists, RAM=2.633 GB
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
<ipython-input-7-4655b4a029b6> in <module>
----> 1 obj.member_search(haloid='snap',fof6d_file='/home2/weiguang/data6/CL300_Caesar/NewMDCLUSTER_0001/FoF6D/fof6d_snap_055.hdf5',nproc=2,lowres=[2,3])
~/.local/lib/python3.6/site-packages/caesar-0.2b0-py3.6-linux-x86_64.egg/caesar/main.py in member_search(self, *args, **kwargs)
338 else:
339 from caesar.fubar_halo import fubar_halo
--> 340 fubar_halo(self)
341 assign.assign_galaxies_to_halos(self)
342 assign.assign_clouds_to_galaxies(self)
~/.local/lib/python3.6/site-packages/caesar-0.2b0-py3.6-linux-x86_64.egg/caesar/fubar_halo.py in fubar_halo(obj)
113
114 # reset particle lists to have original snapshot ID's; must do this after all group processing is finished
--> 115 reset_global_particle_IDs(obj)
116 # load global lists
117 load_global_lists(obj)
~/.local/lib/python3.6/site-packages/caesar-0.2b0-py3.6-linux-x86_64.egg/caesar/fubar_halo.py in reset_global_particle_IDs(obj)
160 part_list = 'group.%s'%plist_dict[p]
161 mylist = eval(part_list)
--> 162 mylist = obj.data_manager.indexes[mylist+offset[ip]]
163 if p == 'gas': group.glist = mylist
164 if p == 'star': group.slist = mylist
IndexError: index 1771876 is out of bounds for axis 0 with size 1530257
Caesar takes too long to write a very large dataset
Caesar took forever to write 5928066 halos and 180565 galaxies catalogue to the hdf5 file...
I think this is a similar problem showing in this issue:
h5py/h5py#1055
Yes, it is a limitation/problem of hdf5, but could we do alternative ways to structure the data? Such as saving the same property together in one single NumPy array?
obj.galinfo() fails
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-25-a8efb84b93b4> in <module>
----> 1 obj.galinfo()
~/miniconda3/envs/pd3env/lib/python3.6/site-packages/caesar-0.2b0-py3.6-linux-x86_64.egg/caesar/main.py in galinfo(self, top)
405 """
406 from caesar.utils import info_printer
--> 407 info_printer(self, 'galaxy', top)
408
409 def haloinfo(self, top=10):
~/miniconda3/envs/pd3env/lib/python3.6/site-packages/caesar-0.2b0-py3.6-linux-x86_64.egg/caesar/utils.py in info_printer(obj, group_type, top)
156 output += ' %04d %0.2e %0.2e %0.2e %0.2e %0.3f %0.2e %s\t| %0.2e %d \n' % \
157 (o.GroupID, o.masses['stellar'], o.masses['gas'],
--> 158 o.sfr, o.radii['total'], o.gas_fraction,
159 o.local_number_density, o.central,
160 phm, phid)
KeyError: 'total'
default halo finding options for member search
it looks like obj.member_search() works for a simba snapshot, but fails with a ```SystemExit: No Halo IDs found in snapshot -- please specify a source (haloid='fof' or 'snap')
/home/desika.narayanan/miniconda3/envs/pd3env/lib/python3.6/site-packages/IPython/core/interactiveshell.py:3334: UserWarning: To exit: use 'exit', 'quit', or Ctrl-D.
warn("To exit: use 'exit', 'quit', or Ctrl-D.", stacklevel=1)
if used on a non-`simba` snapshot. i suspect this is because it is defaulting to `snap`. this should probably default to `fof` so that it works in the more general sense by default, and has to be manually set for specific situations such as `simba`
Misplaced print statement causing install to fail
When I clone and install caesar I get
File "/home/fjennings/.conda/envs/analysis_xrays/lib/python3.11/site-packages/caesar-0.2b0-py3.11-linux-x86_64.egg/caesar/group.py", line 853
elif part_type == 'dust': suffix='dlist'
^^^^
SyntaxError: invalid syntax
due to the print statement on line 852 splitting the elifs
caesar won't install on OSX
it chokes looking for gcc. not sure if this is related to the fact that its using gcc from conda?
(base) planetearth:caesar_dn_fork desika.narayanan$ python setup.py install
running install
running bdist_egg
running egg_info
creating caesar.egg-info
writing caesar.egg-info/PKG-INFO
writing dependency_links to caesar.egg-info/dependency_links.txt
writing entry points to caesar.egg-info/entry_points.txt
writing requirements to caesar.egg-info/requires.txt
writing top-level names to caesar.egg-info/top_level.txt
writing manifest file 'caesar.egg-info/SOURCES.txt'
reading manifest file 'caesar.egg-info/SOURCES.txt'
writing manifest file 'caesar.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.7-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.macosx-10.7-x86_64-3.6
creating build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/property_manager.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/fubar_halo.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/particle_list.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/fof6d.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/simulation_attributes.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/progen.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/command_line.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/__init__.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/vtk_funcs.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/zoom_funcs.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/__version__.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/periodic_kdtree.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/saver.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/old_loader.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/utils.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/treecool_data.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/loader.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/linking.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/fubar.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/group.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/assignment.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/data_manager.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/vtk_vis.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/progen_rad.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/main.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
copying caesar/driver.py -> build/lib.macosx-10.7-x86_64-3.6/caesar
creating build/lib.macosx-10.7-x86_64-3.6/caesar/pyloser
copying caesar/pyloser/pyloser.py -> build/lib.macosx-10.7-x86_64-3.6/caesar/pyloser
copying caesar/pyloser/__init__.py -> build/lib.macosx-10.7-x86_64-3.6/caesar/pyloser
running build_ext
cythoning caesar/group_funcs/group_funcs.pyx to caesar/group_funcs/group_funcs.c
/Users/desika.narayanan/miniconda3/lib/python3.6/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /Users/desika.narayanan/caesar_dn_fork/caesar/group_funcs/group_funcs.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
building 'caesar.group_funcs' extension
creating build/temp.macosx-10.7-x86_64-3.6
creating build/temp.macosx-10.7-x86_64-3.6/caesar
creating build/temp.macosx-10.7-x86_64-3.6/caesar/group_funcs
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/desika.narayanan/miniconda3/include -arch x86_64 -I/Users/desika.narayanan/miniconda3/include -arch x86_64 -I/Users/desika.narayanan/miniconda3/include/python3.6m -I/Users/desika.narayanan/miniconda3/lib/python3.6/site-packages/numpy/core/include -c caesar/group_funcs/group_funcs.c -o build/temp.macosx-10.7-x86_64-3.6/caesar/group_funcs/group_funcs.o -fopenmp
In file included from /Users/desika.narayanan/miniconda3/lib/gcc/x86_64-apple-darwin11.4.2/4.8.5/include-fixed/syslimits.h:7:0,
from /Users/desika.narayanan/miniconda3/lib/gcc/x86_64-apple-darwin11.4.2/4.8.5/include-fixed/limits.h:34,
from /Users/desika.narayanan/miniconda3/include/python3.6m/Python.h:11,
from caesar/group_funcs/group_funcs.c:4:
/Users/desika.narayanan/miniconda3/lib/gcc/x86_64-apple-darwin11.4.2/4.8.5/include-fixed/limits.h:168:61: fatal error: limits.h: No such file or directory
#include_next <limits.h> /* recurse down to the real one */
^
compilation terminated.
error: command 'gcc' failed with exit status 1
(base) planetearth:caesar_dn_fork desika.narayanan$ which gc
(base) planetearth:caesar_dn_fork desika.narayanan$ which gcc
/Users/desika.narayanan/miniconda3/bin/gcc
(base) planetearth:caesar_dn_fork desika.narayanan$
v0.2b 6D FOF hard-coded
Hello,
While looking around, I noticed that in caesar/caesar/fof6d.py lines 143 and 144 that the linking lengths are hard-coded for position to be b_galaxy * MIS and linking length for velocity is 1 code_velocity_units. I was confused if this is intentional, since the fof6d_LL_ arguments can be defined by the user.
NotImplementedError for simulations with no SMBHs
Hi, I've been trying to use caesar on simulations with no supermassive black holes and found a little circular bug that occurs in the property manager.
Specifically, on line 146 a NotImplementedError is raised in the get_ptype_name() function:
def get_ptype_name(self, requested_ptype):
"""Gets the correct ptype name for this dataset.
Parameters
----------
requested_ptype : str
Typically 'gas','dm','star','bh'
Returns
-------
str
The proper ptype name for a given dataset.
"""
if not self.has_ptype(requested_ptype):
raise NotImplementedError('Could not find %s ptype!' % requested_ptype)
return self.ptype_aliases[requested_ptype.lower()]Here is a link:
caesar/caesar/property_manager.py
Line 146 in cd92546
However, get_ptype_name() is used in has_property() but the exception is not caught! So if the property does not exist the execution halts at the exception rather than continuing without that particle type. That occurs here:
def has_property(self, requested_ptype, requested_prop):
"""Returns True/False if requested property/field is present.
Parameters
----------
requested_ptype : str
Typically 'gas','dm','star','bh'
requested_prop : str
Requested property/field.
Returns
-------
boolean
True if property/field is present, False otherwise.
"""
prop = self.get_property_name(requested_ptype, requested_prop)
ptype = self.get_ptype_name(requested_ptype)
if ptype in self.ds.particle_fields_by_type:
fields = self.ds.particle_fields_by_type[ptype]
if prop in fields:
return True
fields = self.ds.derived_field_list
for f in fields:
if f[0] == ptype and f[1] == prop:
return True
return FalseI put a try...except... statement in has_property() that returns False on the exception, as a temporary fix. That seems to work fine but maybe there is a more elegant solution.
sys not imported
Line 91 in 442949d
The sys package has not been imported in this code block. It seems like there are a lot of cases where this happens, perhaps just a single global import would work fine?
code fails on arepo sims
this has only been tested on an idealized simulation, but we get:
/home/pg3552/anaconda3/lib/python3.7/site-packages/caesar-0.2b0-py3.7-linux-x86_64.egg/caesar/fubar.py:347: RuntimeWarning: divide by zero encountered in true_divide
efres = int(obj.simulation.boxsize.d / mips.d)
Traceback (most recent call last):
File "/home/pg3552/anaconda3/bin/caesar", line 11, in <module>
load_entry_point('caesar==0.2b0', 'console_scripts', 'caesar')()
File "/home/pg3552/anaconda3/lib/python3.7/site-packages/caesar-0.2b0-py3.7-linux-x86_64.egg/caesar/command_line.py", line 78, in run
run_caesar(args.input, var_dict)
File "/home/pg3552/anaconda3/lib/python3.7/site-packages/caesar-0.2b0-py3.7-linux-x86_64.egg/caesar/command_line.py", line 130, in run_caesar
obj.member_search(**args)
File "/home/pg3552/anaconda3/lib/python3.7/site-packages/caesar-0.2b0-py3.7-linux-x86_64.egg/caesar/main.py", line 337, in member_search
fubar_halo(self)
File "/home/pg3552/anaconda3/lib/python3.7/site-packages/caesar-0.2b0-py3.7-linux-x86_64.egg/caesar/fubar_halo.py", line 50, in fubar_halo
halos.MIS = get_mean_interparticle_separation(obj).d # also computes omega_baryon and related quantities
File "/home/pg3552/anaconda3/lib/python3.7/site-packages/caesar-0.2b0-py3.7-linux-x86_64.egg/caesar/fubar.py", line 347, in get_mean_interparticle_separation
efres = int(obj.simulation.boxsize.d / mips.d)
OverflowError: cannot convert float infinity to integer```
Zoom-in simulation halo offsets
Hello!
I am running a zoom-in simulation of a cluster and using caesar for the halo analysis. For some reason, the halo positions in the output caesar file do not correspond to any real positions of matter in the simulation or to the associated galaxies.
Here is an image that should clarify what I mean. As you can see, the position of the most massive halo does not even correspond to its associated most massive galaxy, nor is the halo position within the zoom region. I noticed this when trying to make projections of all of the halos, and they were all outside of the zoom region.
I am not sure what is going on, but hopefully someone has some insight! Thanks :)
Number of black holes given by caesar does not conform with that of yt ; stellar metallicity string not displayed
In the following sample code, output 1 gives 210 black holes. output 2 gives 0. output1 and output2 should match as they both correspond to number of black holes recovered using yt and caesar respectively.
output 3 gives length of metallicity string (=number of stars) = 2181832. output 4 results in an error KeyError: 'stellar'. This code uses snapshot 150 of the s25_512 SIMBA run and its respective caesarfile.
import yt
import caesar
snap = '/orange/narayanan/desika.narayanan/gizmo_runs/simba/m25n512/output/snapshot_150.hdf5'
ds = yt.load(snap)
obj = caesar.load('/orange/narayanan/desika.narayanan/gizmo_runs/simba/m25n512/output/Groups/caesar_0150_z2.262.hdf5')
ad = ds.all_data()
print("Number of particles of type 5:",len(ad['PartType5','ParticleIDs'])) #getting number of type5 particles by checking length of the ParticleID string using yt - output 1
print("Number of black holes:",obj.simulation.nbh) #simply printing number of black holes using the caesarfile - output 2
print(len(ad['PartType4','Metallicity_00']),ad['PartType4','Metallicity_00'][0:10]) #printing number of stellar metallicity values present using yt and also showing some sample values - output 3
for i in range(0,10):
print(obj.galaxies[i].metallicities['stellar']) #printing number of stellar metallicities using caesar for some sample galaxies - output 4
Caesar BH masses are incorrect
I am on the most recent yt and caesar hashes and attempted to run caesar to generate a catalog. However the catalog's BH information is not accurate; the black hole masses do not make sense as they are all listed as less than 1 solar mass. Even if this is a conversion error between code mass and solar masses, it does not make sense because the masses would not match up with the black hole masses pulled from yt. Below is the script I used to demonstrate this.
import yt
import caesar
import numpy as np
ds = yt.load('/orange/narayanan/desika.narayanan/gizmo_runs/simba/m25n512/output/snapshot_305.hdf5')
proc = 16
# making caesar file
testObj = caesar.CAESAR(ds)
testObj.member_search(haloid = 'fof',blackholes=True,fof6d = True,nproc = proc)
testObj.save('test_caesar.hdf5')
# compare BH information with new caesar file
caes_test = caesar.load('test_caesar.hdf5')
yt_dat = ds.all_data()
# direct BH masses
bh_masses_yt = yt_dat["PartType5","Masses"]
bh_sorted_yt = np.sort(bh_masses_yt)[::-1] # large to small
# caesar stored BH masses
bh_masses_caes = [gal.masses['bh'].to('Msun') for gal in caes_test.galaxies]
bh_masses_caes = np.array(bh_masses_caes)
# print out comparable results - masses are not consistent with each other, even with potential factors of 10^10
print(bh_sorted_yt.to('Msun'))
print(bh_masses_caes)
# number of black holes also not close
print(len(bh_sorted_yt))
print(np.sum([gal.nbh for gal in caes_test.galaxies]))Error Compiling Cython file while Installing Caesar Package
I have been trying to install the Caesar package, but I'm encountering a Cython compilation error.
Here are the steps I've taken:
I activated my conda environment, I tried with base and myenv
I tried to run python setup.py install in the Caesar package directory.
However, the installation fails with the error below
Environment:
OS: Ubuntu 20.04
Python version: 3.11 (from Anaconda)
Cython version: ( 3.0.0 py311h5eee18b_0)
Additional Context:
I have installed all the dependencies according to the package's installation instructions, and I am still encountering this issue. I have also tried updating Cython, but the issue persists.
OUTPUT
(base) herman@ThinkPad:~/Downloads/caesar-master$ sudo python3 setup.py install
running install
/usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
/usr/lib/python3/dist-packages/setuptools/command/easy_install.py:158: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
/usr/lib/python3/dist-packages/pkg_resources/init.py:116: PkgResourcesDeprecationWarning: 1.1build1 is an invalid version and will not be supported in a future release
warnings.warn(
running bdist_egg
running egg_info
writing caesar.egg-info/PKG-INFO
writing dependency_links to caesar.egg-info/dependency_links.txt
writing entry points to caesar.egg-info/entry_points.txt
writing requirements to caesar.egg-info/requires.txt
writing top-level names to caesar.egg-info/top_level.txt
reading manifest file 'caesar.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'caesar.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
Compiling caesar/group_funcs/group_funcs.pyx because it changed.
[1/1] Cythonizing caesar/group_funcs/group_funcs.pyx
Error compiling Cython file:
...
cdef part_struct *grp_partinfo
# get radius of each particle, sort by radii
grp_partinfo = <part_struct *> malloc((nparticles)*sizeof(part_struct)) # allocate particle info
nogil_load_partinfo(mass, pos, vel, ptype, cent_pos, cent_vel, grp_partinfo, Lbox, istart, iend, ndim)
qsort(<void*>grp_partinfo, <size_t>(nparticles), sizeof(part_struct), mycmp)
^
caesar/group_funcs/group_funcs.pyx:404:74: Cannot assign type 'int (const_void *, const_void ) except? -1 nogil' to 'int ()(const_void *, const_void *) noexcept'
Traceback (most recent call last):
File "/home/herman/Downloads/caesar-master/setup.py", line 69, in
setup(
File "/usr/lib/python3/dist-packages/setuptools/init.py", line 153, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python3.10/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.10/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.10/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/lib/python3/dist-packages/setuptools/command/install.py", line 74, in run
self.do_egg_install()
File "/usr/lib/python3/dist-packages/setuptools/command/install.py", line 116, in do_egg_install
self.run_command('bdist_egg')
File "/usr/lib/python3.10/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.10/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/lib/python3/dist-packages/setuptools/command/bdist_egg.py", line 164, in run
cmd = self.call_command('install_lib', warn_dir=0)
File "/usr/lib/python3/dist-packages/setuptools/command/bdist_egg.py", line 150, in call_command
self.run_command(cmdname)
File "/usr/lib/python3.10/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.10/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/lib/python3/dist-packages/setuptools/command/install_lib.py", line 23, in run
self.build()
File "/usr/lib/python3.10/distutils/command/install_lib.py", line 109, in build
self.run_command('build_ext')
File "/usr/lib/python3.10/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.10/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/lib/python3/dist-packages/setuptools/command/build_ext.py", line 79, in run
_build_ext.run(self)
File "/usr/lib/python3.10/distutils/command/build_ext.py", line 340, in run
self.build_extensions()
File "/usr/lib/python3.10/distutils/command/build_ext.py", line 449, in build_extensions
self._build_extensions_serial()
File "/usr/lib/python3.10/distutils/command/build_ext.py", line 474, in _build_extensions_serial
self.build_extension(ext)
File "/usr/lib/python3/dist-packages/setuptools/command/build_ext.py", line 202, in build_extension
_build_ext.build_extension(self, ext)
File "/usr/local/lib/python3.10/dist-packages/Cython/Distutils/build_ext.py", line 122, in build_extension
new_ext = cythonize(
File "/usr/local/lib/python3.10/dist-packages/Cython/Build/Dependencies.py", line 1134, in cythonize
cythonize_one(*args)
File "/usr/local/lib/python3.10/dist-packages/Cython/Build/Dependencies.py", line 1301, in cythonize_one
raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: caesar/group_funcs/group_funcs.pyx
Cython Compilation Fails
Just pulled and tried to build Caesar, and it fails w/ the following output;
Error compiling Cython file:
------------------------------------------------------------
...
cdef extern from "math.h":
double sqrt(double x)
double M_PI
cdef extern from "stdlib.h":
ctypedef void const_void "const void"
void qsort(void *base, int nmemb, int size, int(*compar)(const_void *, const_void *) noexcept nogil) nogil
^
------------------------------------------------------------
caesar/group_funcs/group_funcs.pyx:22:89: Expected ')', found 'noexcept'
Compilation error in build
My daily CI tests for swiftgalaxy failed this morning with an error building caesar. Yesterday's build passed and neither caesar nor swiftgalaxy have any commits in the past day, so I expect that this is some kind of version conflict with a caesar dependency or similar. However, it crashes with a Cython compiler error that looks like it could be due to an actual bug in the caesar code (?):
2023-10-18T03:26:37.0550792Z
2023-10-18T03:26:37.0551088Z Error compiling Cython file:
2023-10-18T03:26:37.0551620Z ------------------------------------------------------------
2023-10-18T03:26:37.0552068Z ...
2023-10-18T03:26:37.0552327Z
2023-10-18T03:26:37.0552697Z # reset order to order of galaxies in caesar file
2023-10-18T03:26:37.0553220Z gmass = np.array(galaxy_mass)
2023-10-18T03:26:37.0553676Z for ipt in range(len(quants)):
2023-10-18T03:26:37.0554151Z for ih,ig in enumerate(g_indexes):
2023-10-18T03:26:37.0554670Z galaxy_mass[ipt,ig] = gmass[ipt,ih]
2023-10-18T03:26:37.0555131Z ^
2023-10-18T03:26:37.0555726Z ------------------------------------------------------------
2023-10-18T03:26:37.0556180Z
2023-10-18T03:26:37.0556842Z caesar/hydrogen_mass_calc/hydrogen_mass_calc.pyx:929:23: Compiler crash in AnalyseExpressionsTransform
And with more context:
2023-10-18T03:26:06.0196686Z Building wheels for collected packages: caesar, progress, future
2023-10-18T03:26:06.0207691Z Building wheel for caesar (pyproject.toml): started
2023-10-18T03:26:37.0100707Z Building wheel for caesar (pyproject.toml): finished with status 'error'
2023-10-18T03:26:37.0335413Z error: subprocess-exited-with-error
2023-10-18T03:26:37.0336123Z
2023-10-18T03:26:37.0337343Z × Building wheel for caesar (pyproject.toml) did not run successfully.
2023-10-18T03:26:37.0338052Z │ exit code: 1
2023-10-18T03:26:37.0338440Z ╰─> [238 lines of output]
2023-10-18T03:26:37.0338810Z running bdist_wheel
2023-10-18T03:26:37.0339151Z running build
2023-10-18T03:26:37.0339467Z running build_py
2023-10-18T03:26:37.0339790Z creating build
2023-10-18T03:26:37.0340233Z creating build/lib.linux-x86_64-cpython-39
2023-10-18T03:26:37.0340848Z creating build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0341643Z copying caesar/zoom_funcs.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0342544Z copying caesar/fof6d.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0343412Z copying caesar/assignment.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0344291Z copying caesar/loader.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0345147Z copying caesar/fubar_halo.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0346021Z copying caesar/linking.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0346892Z copying caesar/progen_rad.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0347745Z copying caesar/fubar.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0348607Z copying caesar/old_loader.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0349446Z copying caesar/main.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0350318Z copying caesar/treecool_data.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0351184Z copying caesar/saver.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0352010Z copying caesar/utils.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0352905Z copying caesar/property_manager.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0353808Z copying caesar/__init__.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0354658Z copying caesar/driver.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0355511Z copying caesar/__version__.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0356414Z copying caesar/command_line.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0357289Z copying caesar/progen.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0358161Z copying caesar/data_manager.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0359428Z copying caesar/simulation_attributes.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0360417Z copying caesar/particle_list.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0361457Z copying caesar/vtk_funcs.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0362311Z copying caesar/group.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0363206Z copying caesar/periodic_kdtree.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0369593Z copying caesar/vtk_vis.py -> build/lib.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0370404Z creating build/lib.linux-x86_64-cpython-39/caesar/pyloser
2023-10-18T03:26:37.0371322Z copying caesar/pyloser/atten_laws.py -> build/lib.linux-x86_64-cpython-39/caesar/pyloser
2023-10-18T03:26:37.0372375Z copying caesar/pyloser/__init__.py -> build/lib.linux-x86_64-cpython-39/caesar/pyloser
2023-10-18T03:26:37.0373440Z copying caesar/pyloser/pyloser.py -> build/lib.linux-x86_64-cpython-39/caesar/pyloser
2023-10-18T03:26:37.0374134Z running build_ext
2023-10-18T03:26:37.0375303Z performance hint: caesar/group_funcs/group_funcs.pyx:62:5: Exception check on 'nogil_CoM_quants' will always require the GIL to be acquired.
2023-10-18T03:26:37.0376522Z Possible solutions:
2023-10-18T03:26:37.0377528Z 1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
2023-10-18T03:26:37.0378802Z 2. Use an 'int' return type on the function to allow an error code to be returned.
2023-10-18T03:26:37.0380194Z performance hint: caesar/group_funcs/group_funcs.pyx:116:5: Exception check on 'nogil_load_partinfo' will always require the GIL to be acquired.
2023-10-18T03:26:37.0381190Z Possible solutions:
2023-10-18T03:26:37.0382171Z 1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
2023-10-18T03:26:37.0384008Z 2. Use an 'int' return type on the function to allow an error code to be returned.
2023-10-18T03:26:37.0385398Z performance hint: caesar/group_funcs/group_funcs.pyx:224:5: Exception check on 'nogil_angular_quants' will always require the GIL to be acquired.
2023-10-18T03:26:37.0386404Z Possible solutions:
2023-10-18T03:26:37.0387393Z 1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
2023-10-18T03:26:37.0388648Z 2. Use an 'int' return type on the function to allow an error code to be returned.
2023-10-18T03:26:37.0390135Z performance hint: caesar/group_funcs/group_funcs.pyx:329:5: Exception check on 'nogil_virial_quants' will always require the GIL to be acquired.
2023-10-18T03:26:37.0391123Z Possible solutions:
2023-10-18T03:26:37.0392108Z 1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
2023-10-18T03:26:37.0393359Z 2. Use an 'int' return type on the function to allow an error code to be returned.
2023-10-18T03:26:37.0394692Z performance hint: caesar/group_funcs/group_funcs.pyx:360:5: Exception check on 'nogil_rotator' will always require the GIL to be acquired.
2023-10-18T03:26:37.0395653Z Possible solutions:
2023-10-18T03:26:37.0396641Z 1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
2023-10-18T03:26:37.0397882Z 2. Use an 'int' return type on the function to allow an error code to be returned.
2023-10-18T03:26:37.0399242Z performance hint: caesar/group_funcs/group_funcs.pyx:395:5: Exception check on 'nogil_radial_quants' will always require the GIL to be acquired.
2023-10-18T03:26:37.0400231Z Possible solutions:
2023-10-18T03:26:37.0401482Z 1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
2023-10-18T03:26:37.0402740Z 2. Use an 'int' return type on the function to allow an error code to be returned.
2023-10-18T03:26:37.0403875Z performance hint: caesar/group_funcs/group_funcs.pyx:284:17: Exception check will always require the GIL to be acquired.
2023-10-18T03:26:37.0404756Z Possible solutions:
2023-10-18T03:26:37.0405729Z 1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
2023-10-18T03:26:37.0406975Z 2. Use an 'int' return type on the function to allow an error code to be returned.
2023-10-18T03:26:37.0408104Z performance hint: caesar/group_funcs/group_funcs.pyx:404:23: Exception check will always require the GIL to be acquired.
2023-10-18T03:26:37.0408982Z Possible solutions:
2023-10-18T03:26:37.0409974Z 1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
2023-10-18T03:26:37.0411229Z 2. Use an 'int' return type on the function to allow an error code to be returned.
2023-10-18T03:26:37.0412449Z performance hint: caesar/group_funcs/group_funcs.pyx:429:28: Exception check will always require the GIL to be acquired.
2023-10-18T03:26:37.0413322Z Possible solutions:
2023-10-18T03:26:37.0414297Z 1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
2023-10-18T03:26:37.0415532Z 2. Use an 'int' return type on the function to allow an error code to be returned.
2023-10-18T03:26:37.0416660Z performance hint: caesar/group_funcs/group_funcs.pyx:433:27: Exception check will always require the GIL to be acquired.
2023-10-18T03:26:37.0417521Z Possible solutions:
2023-10-18T03:26:37.0418493Z 1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
2023-10-18T03:26:37.0419735Z 2. Use an 'int' return type on the function to allow an error code to be returned.
2023-10-18T03:26:37.0420854Z performance hint: caesar/group_funcs/group_funcs.pyx:783:24: Exception check will always require the GIL to be acquired.
2023-10-18T03:26:37.0421725Z Possible solutions:
2023-10-18T03:26:37.0422737Z 1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
2023-10-18T03:26:37.0423970Z 2. Use an 'int' return type on the function to allow an error code to be returned.
2023-10-18T03:26:37.0425099Z performance hint: caesar/group_funcs/group_funcs.pyx:787:31: Exception check will always require the GIL to be acquired.
2023-10-18T03:26:37.0466131Z Possible solutions:
2023-10-18T03:26:37.0467154Z 1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
2023-10-18T03:26:37.0468393Z 2. Use an 'int' return type on the function to allow an error code to be returned.
2023-10-18T03:26:37.0469514Z performance hint: caesar/group_funcs/group_funcs.pyx:789:31: Exception check will always require the GIL to be acquired.
2023-10-18T03:26:37.0470391Z Possible solutions:
2023-10-18T03:26:37.0471366Z 1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
2023-10-18T03:26:37.0472605Z 2. Use an 'int' return type on the function to allow an error code to be returned.
2023-10-18T03:26:37.0473431Z Compiling caesar/group_funcs/group_funcs.pyx because it changed.
2023-10-18T03:26:37.0474111Z [1/1] Cythonizing caesar/group_funcs/group_funcs.pyx
2023-10-18T03:26:37.0474720Z building 'caesar.group_funcs' extension
2023-10-18T03:26:37.0475299Z creating build/temp.linux-x86_64-cpython-39
2023-10-18T03:26:37.0476055Z creating build/temp.linux-x86_64-cpython-39/caesar
2023-10-18T03:26:37.0522770Z creating build/temp.linux-x86_64-cpython-39/caesar/group_funcs
2023-10-18T03:26:37.0525388Z gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/opt/hostedtoolcache/Python/3.9.18/x64/include/python3.9 -I/tmp/pip-build-env-7wnhqbws/normal/lib/python3.9/site-packages/numpy/core/include -c caesar/group_funcs/group_funcs.c -o build/temp.linux-x86_64-cpython-39/caesar/group_funcs/group_funcs.o -fopenmp
2023-10-18T03:26:37.0528310Z In file included from /tmp/pip-build-env-7wnhqbws/normal/lib/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1929,
2023-10-18T03:26:37.0529895Z from /tmp/pip-build-env-7wnhqbws/normal/lib/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
2023-10-18T03:26:37.0531354Z from /tmp/pip-build-env-7wnhqbws/normal/lib/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:5,
2023-10-18T03:26:37.0532320Z from caesar/group_funcs/group_funcs.c:1200:
2023-10-18T03:26:37.0534211Z /tmp/pip-build-env-7wnhqbws/normal/lib/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
2023-10-18T03:26:37.0536155Z 17 | #warning "Using deprecated NumPy API, disable it with " \
2023-10-18T03:26:37.0536707Z | ^~~~~~~
2023-10-18T03:26:37.0538997Z gcc -shared -Wl,--rpath=/opt/hostedtoolcache/Python/3.9.18/x64/lib -Wl,--rpath=/opt/hostedtoolcache/Python/3.9.18/x64/lib build/temp.linux-x86_64-cpython-39/caesar/group_funcs/group_funcs.o -L/opt/hostedtoolcache/Python/3.9.18/x64/lib -o build/lib.linux-x86_64-cpython-39/caesar/group_funcs.cpython-39-x86_64-linux-gnu.so -fopenmp
2023-10-18T03:26:37.0541939Z performance hint: caesar/hydrogen_mass_calc/hydrogen_mass_calc.pyx:448:5: Exception check on '_get_galaxy_hydrogen_masses' will always require the GIL to be acquired.
2023-10-18T03:26:37.0543054Z Possible solutions:
2023-10-18T03:26:37.0544053Z 1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
2023-10-18T03:26:37.0545336Z 2. Use an 'int' return type on the function to allow an error code to be returned.
2023-10-18T03:26:37.0546839Z performance hint: caesar/hydrogen_mass_calc/hydrogen_mass_calc.pyx:1142:5: Exception check on 'get_galaxy_aperture_masses' will always require the GIL to be acquired.
2023-10-18T03:26:37.0547950Z Possible solutions:
2023-10-18T03:26:37.0548933Z 1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
2023-10-18T03:26:37.0550163Z 2. Use an 'int' return type on the function to allow an error code to be returned.
2023-10-18T03:26:37.0550792Z
2023-10-18T03:26:37.0551088Z Error compiling Cython file:
2023-10-18T03:26:37.0551620Z ------------------------------------------------------------
2023-10-18T03:26:37.0552068Z ...
2023-10-18T03:26:37.0552327Z
2023-10-18T03:26:37.0552697Z # reset order to order of galaxies in caesar file
2023-10-18T03:26:37.0553220Z gmass = np.array(galaxy_mass)
2023-10-18T03:26:37.0553676Z for ipt in range(len(quants)):
2023-10-18T03:26:37.0554151Z for ih,ig in enumerate(g_indexes):
2023-10-18T03:26:37.0554670Z galaxy_mass[ipt,ig] = gmass[ipt,ih]
2023-10-18T03:26:37.0555131Z ^
2023-10-18T03:26:37.0555726Z ------------------------------------------------------------
2023-10-18T03:26:37.0556180Z
2023-10-18T03:26:37.0556842Z caesar/hydrogen_mass_calc/hydrogen_mass_calc.pyx:929:23: Compiler crash in AnalyseExpressionsTransform
2023-10-18T03:26:37.0557606Z
2023-10-18T03:26:37.0558132Z ModuleNode.body = StatListNode(hydrogen_mass_calc.pyx:1:0)
2023-10-18T03:26:37.0558872Z StatListNode.stats[23] = StatListNode(hydrogen_mass_calc.pyx:761:0)
2023-10-18T03:26:37.0559704Z StatListNode.stats[0] = CompilerDirectivesNode(hydrogen_mass_calc.pyx:761:0)
2023-10-18T03:26:37.0560708Z CompilerDirectivesNode.body = StatListNode(hydrogen_mass_calc.pyx:761:0)
2023-10-18T03:26:37.0561504Z StatListNode.stats[0] = DefNode(hydrogen_mass_calc.pyx:761:0,
2023-10-18T03:26:37.0562171Z defaults_struct = '__pyx_defaults',
2023-10-18T03:26:37.0562920Z doc = ' Compute aperture masses in various quantities; standalone version ',
2023-10-18T03:26:37.0563578Z is_cyfunction = True,
2023-10-18T03:26:37.0563972Z modifiers = [...]/0,
2023-10-18T03:26:37.0564415Z name = 'get_aperture_masses',
2023-10-18T03:26:37.0564853Z np_args_idx = [...]/0,
2023-10-18T03:26:37.0565232Z num_required_args = 3,
2023-10-18T03:26:37.0565629Z outer_attrs = [...]/2,
2023-10-18T03:26:37.0566048Z py_wrapper_required = True,
2023-10-18T03:26:37.0566518Z reqd_kw_flags_cname = '0',
2023-10-18T03:26:37.0566916Z used = True)
2023-10-18T03:26:37.0567760Z File 'Nodes.py', line 397, in analyse_expressions: StatListNode(hydrogen_mass_calc.pyx:765:4,
2023-10-18T03:26:37.0568486Z is_terminator = True)
2023-10-18T03:26:37.0569252Z File 'Nodes.py', line 397, in analyse_expressions: StatListNode(hydrogen_mass_calc.pyx:927:4)
2023-10-18T03:26:37.0570343Z File 'Nodes.py', line 7556, in analyse_expressions: ForInStatNode(hydrogen_mass_calc.pyx:927:4)
2023-10-18T03:26:37.0571450Z File 'Nodes.py', line 397, in analyse_expressions: StatListNode(hydrogen_mass_calc.pyx:928:8)
2023-10-18T03:26:37.0572537Z File 'Nodes.py', line 7556, in analyse_expressions: ForInStatNode(hydrogen_mass_calc.pyx:928:8)
2023-10-18T03:26:37.0573636Z File 'Nodes.py', line 397, in analyse_expressions: StatListNode(hydrogen_mass_calc.pyx:929:39)
2023-10-18T03:26:37.0574816Z File 'Nodes.py', line 6026, in analyse_expressions: SingleAssignmentNode(hydrogen_mass_calc.pyx:929:39)
2023-10-18T03:26:37.0575994Z File 'Nodes.py', line 6164, in analyse_types: SingleAssignmentNode(hydrogen_mass_calc.pyx:929:39)
2023-10-18T03:26:37.0577140Z File 'ExprNodes.py', line 4085, in analyse_target_types: IndexNode(hydrogen_mass_calc.pyx:929:23,
2023-10-18T03:26:37.0577875Z is_subscript = True,
2023-10-18T03:26:37.0578260Z result_is_used = True,
2023-10-18T03:26:37.0578647Z use_managed_ref = True)
2023-10-18T03:26:37.0579497Z File 'ExprNodes.py', line 4123, in analyse_base_and_index_types: IndexNode(hydrogen_mass_calc.pyx:929:23,
2023-10-18T03:26:37.0580274Z is_subscript = True,
2023-10-18T03:26:37.0580655Z result_is_used = True,
2023-10-18T03:26:37.0581044Z use_managed_ref = True)
2023-10-18T03:26:37.0581872Z File 'ExprNodes.py', line 4349, in analyse_as_buffer_operation: IndexNode(hydrogen_mass_calc.pyx:929:23,
2023-10-18T03:26:37.0582635Z is_subscript = True,
2023-10-18T03:26:37.0583012Z result_is_used = True,
2023-10-18T03:26:37.0583399Z use_managed_ref = True)
2023-10-18T03:26:37.0584227Z File 'ExprNodes.py', line 4968, in analyse_types: MemoryViewIndexNode(hydrogen_mass_calc.pyx:929:23,
2023-10-18T03:26:37.0584995Z is_memview_index = True,
2023-10-18T03:26:37.0585384Z memslice_index = True,
2023-10-18T03:26:37.0585768Z result_is_used = True,
2023-10-18T03:26:37.0586148Z use_managed_ref = True,
2023-10-18T03:26:37.0586535Z writable_needed = True)
2023-10-18T03:26:37.0586887Z
2023-10-18T03:26:37.0587224Z Compiler crash traceback from this point on:
2023-10-18T03:26:37.0599871Z File "/tmp/pip-build-env-7wnhqbws/normal/lib/python3.9/site-packages/Cython/Compiler/ExprNodes.py", line 4968, in analyse_types
2023-10-18T03:26:37.0601906Z performance_hint(index.pos, "Index should be typed for more efficient access")
2023-10-18T03:26:37.0602965Z TypeError: performance_hint() missing 1 required positional argument: 'env'
2023-10-18T03:26:37.0603853Z Compiling caesar/hydrogen_mass_calc/hydrogen_mass_calc.pyx because it changed.
2023-10-18T03:26:37.0604694Z [1/1] Cythonizing caesar/hydrogen_mass_calc/hydrogen_mass_calc.pyx
2023-10-18T03:26:37.0605316Z Traceback (most recent call last):
2023-10-18T03:26:37.0606639Z File "/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
2023-10-18T03:26:37.0607706Z main()
2023-10-18T03:26:37.0608818Z File "/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
2023-10-18T03:26:37.0610036Z json_out['return_val'] = hook(**hook_input['kwargs'])
2023-10-18T03:26:37.0611392Z File "/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
2023-10-18T03:26:37.0612674Z return _build_backend().build_wheel(wheel_directory, config_settings,
2023-10-18T03:26:37.0614060Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 434, in build_wheel
2023-10-18T03:26:37.0615037Z return self._build_with_temp_dir(
2023-10-18T03:26:37.0616194Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 419, in _build_with_temp_dir
2023-10-18T03:26:37.0617182Z self.run_setup()
2023-10-18T03:26:37.0618158Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 507, in run_setup
2023-10-18T03:26:37.0619283Z super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
2023-10-18T03:26:37.0620543Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 341, in run_setup
2023-10-18T03:26:37.0621466Z exec(code, locals())
2023-10-18T03:26:37.0621924Z File "<string>", line 69, in <module>
2023-10-18T03:26:37.0623009Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 103, in setup
2023-10-18T03:26:37.0623940Z return distutils.core.setup(**attrs)
2023-10-18T03:26:37.0625035Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
2023-10-18T03:26:37.0625966Z return run_commands(dist)
2023-10-18T03:26:37.0627046Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
2023-10-18T03:26:37.0627987Z dist.run_commands()
2023-10-18T03:26:37.0629043Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
2023-10-18T03:26:37.0629989Z self.run_command(cmd)
2023-10-18T03:26:37.0630986Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 989, in run_command
2023-10-18T03:26:37.0631896Z super().run_command(command)
2023-10-18T03:26:37.0632982Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
2023-10-18T03:26:37.0633917Z cmd_obj.run()
2023-10-18T03:26:37.0634800Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 364, in run
2023-10-18T03:26:37.0635653Z self.run_command("build")
2023-10-18T03:26:37.0636705Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
2023-10-18T03:26:37.0637701Z self.distribution.run_command(command)
2023-10-18T03:26:37.0638854Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 989, in run_command
2023-10-18T03:26:37.0639745Z super().run_command(command)
2023-10-18T03:26:37.0640965Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
2023-10-18T03:26:37.0642005Z cmd_obj.run()
2023-10-18T03:26:37.0643037Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 131, in run
2023-10-18T03:26:37.0644008Z self.run_command(cmd_name)
2023-10-18T03:26:37.0645083Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
2023-10-18T03:26:37.0646073Z self.distribution.run_command(command)
2023-10-18T03:26:37.0647131Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 989, in run_command
2023-10-18T03:26:37.0648044Z super().run_command(command)
2023-10-18T03:26:37.0649125Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
2023-10-18T03:26:37.0650170Z cmd_obj.run()
2023-10-18T03:26:37.0651128Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 88, in run
2023-10-18T03:26:37.0652038Z _build_ext.run(self)
2023-10-18T03:26:37.0653102Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
2023-10-18T03:26:37.0654086Z self.build_extensions()
2023-10-18T03:26:37.0655260Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
2023-10-18T03:26:37.0656340Z self._build_extensions_serial()
2023-10-18T03:26:37.0657630Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
2023-10-18T03:26:37.0658737Z self.build_extension(ext)
2023-10-18T03:26:37.0659865Z File "/tmp/pip-build-env-7wnhqbws/overlay/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 249, in build_extension
2023-10-18T03:26:37.0660902Z _build_ext.build_extension(self, ext)
2023-10-18T03:26:37.0662057Z File "/tmp/pip-build-env-7wnhqbws/normal/lib/python3.9/site-packages/Cython/Distutils/build_ext.py", line 130, in build_extension
2023-10-18T03:26:37.0663037Z new_ext = cythonize(
2023-10-18T03:26:37.0664072Z File "/tmp/pip-build-env-7wnhqbws/normal/lib/python3.9/site-packages/Cython/Build/Dependencies.py", line 1154, in cythonize
2023-10-18T03:26:37.0665013Z cythonize_one(*args)
2023-10-18T03:26:37.0666077Z File "/tmp/pip-build-env-7wnhqbws/normal/lib/python3.9/site-packages/Cython/Build/Dependencies.py", line 1321, in cythonize_one
2023-10-18T03:26:37.0667114Z raise CompileError(None, pyx_file)
2023-10-18T03:26:37.0667872Z Cython.Compiler.Errors.CompileError: caesar/hydrogen_mass_calc/hydrogen_mass_calc.pyx
2023-10-18T03:26:37.0668589Z [end of output]
2023-10-18T03:26:37.0668894Z
2023-10-18T03:26:37.0669429Z note: This error originates from a subprocess, and is likely not a problem with pip.
2023-10-18T03:26:37.0670174Z ERROR: Failed building wheel for caesar
Compatibility problems with HDFView 3.3.1
There seems to be a compatibility problem between the hdf5 files generated by Caesar and the latest (3.3.1) version of HDFView. Have a look at this loom video for the description of the problem: https://www.loom.com/share/31df471fb82d406a88ee5207a33dbfef
contamination check for zooms broken
on an arepo zoom:
## Largest 10 halos
## from: /blue/narayanan/desika.narayanan/arepo_runs/cosmic_sands/run0_gal0_ml10_baryon/output/Groups/caesar_snapshot_020.hdf5
## 142771 @ z=4.498
ID Mdm Mstar Mgas r fgas nrho | CentralGalMstar
---------------------------------------------------------------------------------
0000 2.78e+13 1.99e+12 4.17e+12 6.60e+03 3.10e-09 | 1.39e+11
0001 6.95e+12 4.78e+11 1.02e+12 2.29e+03 1.89e-08 | 7.56e+10
0002 4.25e+12 2.73e+11 6.62e+11 4.87e+03 3.34e-09 | 5.10e+10
0003 3.42e+12 2.37e+11 5.34e+11 2.32e+03 1.38e-08 | 4.07e+10
0004 2.59e+12 1.45e+11 4.46e+11 1.37e+03 2.58e-08 | 1.59e+10
0005 1.99e+12 1.24e+11 3.06e+11 1.52e+03 2.67e-08 | 3.05e+10
0006 1.54e+12 1.07e+11 2.48e+11 1.93e+03 1.79e-08 | 2.28e+10
0007 1.37e+12 8.09e+10 2.26e+11 1.18e+03 1.58e-08 | 1.07e+10
0008 1.25e+12 8.16e+10 1.95e+11 6.93e+02 1.72e-08 | 2.30e+10
0009 1.17e+12 6.61e+10 1.77e+11 8.10e+02 1.96e-08 | 1.78e+10
In [3]: obj.halos[0].contamination_check()
yt : [INFO ] 2023-11-06 12:13:12,509 Gathering low-res particles and constructing tree
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[3], line 1
----> 1 obj.halos[0].contamination_check()
File /blue/narayanan/desika.narayanan/conda/envs/py38/lib/python3.8/site-packages/caesar-0.2b0-py3.8-linux-x86_64.egg/caesar/loader.py:411, in Group.contamination_check(self, lowres, search_factor, printer)
405 def contamination_check(self,
406 lowres=[2, 3, 5],
407 search_factor=2.5,
408 printer=True):
409 from caesar.zoom_funcs import construct_lowres_tree
--> 411 construct_lowres_tree(self, lowres)
413 if self.obj_type == 'halo':
414 halo = self
File /blue/narayanan/desika.narayanan/conda/envs/py38/lib/python3.8/site-packages/caesar-0.2b0-py3.8-linux-x86_64.egg/caesar/zoom_funcs.py:205, in construct_lowres_tree(obj, lowres)
202 lr_pos = np.empty((0,3))
203 lr_mass = np.empty(0)
--> 205 pos_unit = obj.halos[0].pos.units
206 mass_unit = obj.halos[0].masses['total_r20'].units
208 for p in lowres: # all low res particles to only build tree once
File /blue/narayanan/desika.narayanan/conda/envs/py38/lib/python3.8/site-packages/caesar-0.2b0-py3.8-linux-x86_64.egg/caesar/loader.py:527, in Halo.__getattr__(self, attr)
523 if attr in self.obj._halo_dicts:
524 return LazyDict(
525 self.obj._halo_dicts[attr].keys(),
526 lambda d: self.obj._halo_dicts[attr][d][self._index])
--> 527 raise AttributeError("'{}' object as no attribute '{}'".format(
528 self.__class__.__name__, attr))
AttributeError: 'Halo' object as no attribute 'halos'
obj.galaxies doesn't have halo mass
Out[5]: unyt_quantity(8.57460861e+13, 'Msun')
In [6]: obj.galaxies[0].masses['dm']
Out[6]: unyt_quantity(0., 'Msun')```
gas masses are confusing
Currently the key obj.galaxies.masses['gas'] is not the same thing as the sum of the HI and H2 gas masses. this is because the former is only the dense gas mass that is included in the 6DFOF galaxy finding, while the latter is all of the HI and H2 gas associated within the galaxy's radius. This means:
MHI + MH2 != obj.galaxies.masses['gas']
i suggest:
[1] re-defining obj.galaxies.masses['gas'] to be the sum of the HI and H2 gas masses.
[2] defining a new field like obj.galaxies.masses['dense_gas'] or something to be equivalent to the old value of obj.glaxies.masses['gas']
Move to the quick-loader by default
Right now the quick-loader is opt-in by calling caesar.quick_load or caesar -q, but it's so stupidly faster that it should be the default. We don't want people hacking around the caesar loader because it's slow and shooting themselves in the foot with unit mistakes.
I've tried to pave the way for switching the default by making the the CAESAR and associated group classes in caesar/quick_load.py present as identical an interface as possible, and even included a huge regression test in the form of test/quick_loader_info.py which should ensure that the intended-public interface is the same (that is, things that start with a _ may be broken or missing). The only thing that this currently trips over is the simulation densities which I've seen be slightly different at load time because they're recomputed using a different order of floating-point operations. If such load-time extensions are desired in the future, they could be done using an @property in the quick-loader, following the implementation of Halo.galaxies.
AFAIK the only thing that's missing is obj.vtk_vis, but does anyone still use that?
I think the best path forward is to put a deprecation warning in caesar/loader.py's load function, try to prompt the community to upgrade, let that stew for a few months, then make the quick-loader the default. Such a warning could look like:
import warnings
warnings.warn('''In an upcoming release the caesar loader will be replaced by the
implementation currently called caesar.quick_load, or caesar -q at the command
line. It is much faster but has a fundamentally different implementation.
Please test your workflow with it now to prepare for the upcoming change,
and open an issue at https://github.com/dnarayanan/caesar if you find a
problem with it.''')Thoughts?
Cannot run caesar in v0.1 member search
Hello,
I have found an issue with the 6D FOF algorithm in v0.2 (I'll post in another issue thread to keep topics concise) and decide to run in v0.1 mode. However, I get this error output:
Traceback (most recent call last):
File "do_caesar.py", line 39, in <module>
obj.member_search(fof6d=True, fof6d_LL_factor=LL_pos, fof6d_mingrp=4, fof6d_velLL=LL_vel, nproc=8, v01_member_search=True)
File "/home/local/Anaconda3-2020.02/envs/py3samir/lib/python3.7/site-packages/caesar-0.2b0-py3.7-linux-x86_64.egg/caesar/main.py", line 334, in member_search
self.data_manager._member_search_init()
File "/home/local/Anaconda3-2020.02/envs/py3samir/lib/python3.7/site-packages/caesar-0.2b0-py3.7-linux-x86_64.egg/caesar/data_manager.py", line 30, in _member_search_init
self.load_particle_data(select=select)
File "/home/local/Anaconda3-2020.02/envs/py3samir/lib/python3.7/site-packages/caesar-0.2b0-py3.7-linux-x86_64.egg/caesar/data_manager.py", line 90, in load_particle_data
pdata = get_particles_for_FOF(self.obj, self.ptypes, select)
File "/home/local/Anaconda3-2020.02/envs/py3samir/lib/python3.7/site-packages/caesar-0.2b0-py3.7-linux-x86_64.egg/caesar/property_manager.py", line 437, in get_particles_for_FOF
if obj.load_haloid:
AttributeError: 'CAESAR' object has no attribute 'load_haloid'
maximum particle distances in groups
This is a feature request to optimize integration of Caesar into swiftgalaxy.
That package facilitates retrieving a collection (e.g. halo, galaxy) of particles from a SWIFT snapshot and providing them in a class with various utilities to manipulate them. After chatting with Romeel, I've added integration of Caesar in the branch linked above, but currently it's difficult to implement use of swiftsimio's sub-region reading robustly because I don't know the total extent of the particle group. Could you add another radius to the ones already available that specifies the maximum radius of any particle in the group?
Memory leak?
Hello,
Caesar is failing to process some of my simulations at high resolution. At one point in the halo-finding process the memory usage shoots up rapidly to wherever whatever node I'm working on kills it. The maximum I've seen it is on a full compute node at ~187GB.
The problem seems to be related to resolution (or maybe something that only happens at high resolution) rather than particle count. The problem happens in a snapshot that is ~60GB and a snapshot that is ~4.4GB, both at approximately the same resolution of M_gas ~ 4e5 Msun. At first I thought maybe the 60GB snapshot just needed more memory, but then I was running at N = 256^3 and V = (6Mpccm/h)^3 volume which gives the 4.4 GB snapshot. I don't see how ~187GB of RAM would be needed for that.
I can provide the snapshots where Caesar fails if needs. Interestingly, Caesar doesn't always fail on these snapshots -- only on certain snapshots from both simulations. In my small volume tests, sometimes the parameter choices I make for the sub-grid models actually changes where Caesar will have the memory leak, leading me to believe something might be happening at that resolution causing Caesar to freak out.
Here is an example of the Caesar progression right before it quits. There are no other messages (besides some yt info about the configuration files being deprecated).
Generating md5 hash: 67315it [00:07, 9386.58it/s]
yt : [INFO ] 2021-12-30 09:43:10,971 member_search() running on 20 cores
yt : [INFO ] 2021-12-30 09:43:11,230 Calculated Omega_Baryon=0.0479991 and 256^3 effective resolution
yt : [INFO ] 2021-12-30 09:43:11,231 Using b=0.2 for halos
yt : [INFO ] 2021-12-30 09:43:11,234 Running 3D FOF to get Halo IDs, LL=7.17968, RAM=0.4577 GBThe particle types will be loaded: ['PartType0', 'PartType4', 'PartType1', 'PartType5']
/project/b/babul/rennehan/anaconda3/envs/yt_caesar/lib/python3.9/site-packages/numpy/core/_asarray.py:102: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
return array(a, dtype, copy=False, order=order)
yt : [INFO ] 2021-12-30 09:45:28,141 Initializing member search, loading particles, RAM=1.321 GB
yt : [INFO ] 2021-12-30 09:45:30,781 Loaded particle data, RAM=1.438 GB
yt : [INFO ] 2021-12-30 09:45:32,781 Loaded baryon data, RAM=1.438 GB
yt : [INFO ] 2021-12-30 09:45:41,132 Found 6572 valid halos, loaded indexes, RAM=1.525 GB
yt : [INFO ] 2021-12-30 09:45:41,289 Calculating properties for 6572 halos (nproc=20), RAM=1.595 GB
yt : [INFO ] 2021-12-30 09:45:48,645 Computed properties for 6572 halos, RAM=2.624 GB
yt : [INFO ] 2021-12-30 09:45:54,260 Using b=0.02 for galaxies
yt : [INFO ] 2021-12-30 09:45:54,262 Running fof6d on 6572 halos w/20 proc(s), LL=0.717968, RAM=2.365 GB
yt : [INFO ] 2021-12-30 09:45:54,413 9167067 halo particles, 11.06% eligible for galaxies, RAM=2.366 GB
TerminatedWorkerError Traceback (most recent call last)
/tmp/ipykernel_416698/1962533154.py in
3 else:
4 obj = caesar.CAESAR(ds)
----> 5 obj.member_search(haloid = 'fof', blackholes = True, nproc = 20)
6 obj.save('%s/%s/caesar_%s.hdf5' % (data_dir, data_folder, snap_str))/project/b/babul/rennehan/anaconda3/envs/yt_caesar/lib/python3.9/site-packages/caesar-0.2b0-py3.9-linux-x86_64.egg/caesar/main.py in member_search(self, *args, **kwargs)
338 else:
339 from caesar.fubar_halo import fubar_halo
--> 340 fubar_halo(self)
341 assign.assign_galaxies_to_halos(self)
342 assign.assign_clouds_to_galaxies(self)/project/b/babul/rennehan/anaconda3/envs/yt_caesar/lib/python3.9/site-packages/caesar-0.2b0-py3.9-linux-x86_64.egg/caesar/fubar_halo.py in fubar_halo(obj)
82 fof6d_flag = halos.load_fof6dfile() # load galaxy ID's from fof6d_file
83 if fof6d_flag:
---> 84 halos.run_fof6d('galaxy') # run fof6d on halos to find galaxies
85 halos.save_fof6dfile() # save fof6d info
86 #snapname = ('%s/%s'%(obj.simulation.fullpath,obj.simulation.basename))/project/b/babul/rennehan/anaconda3/envs/yt_caesar/lib/python3.9/site-packages/caesar-0.2b0-py3.9-linux-x86_64.egg/caesar/fof6d.py in run_fof6d(self, target_type, nHlim, Tlim, sfflag, minstars)
172 ngs+=tmg
173 else:
--> 174 tmg = Parallel(n_jobs=self.nproc)(delayed(fof6d_halo)(len(self.obj.halo_list[ih].global_indexes),len(g_inds[ih]),self.obj.data_manager.pos[g_inds[ih]],self.obj.data_manager.vel[g_inds[ih]],self.minstars,self.obj.simulation.boxsize.d,self.fof_LL,self.vel_LL,self.kerneltab) for ih in range(len(self.obj.halo_list)))
175 grp_tags,ngs = zip(*tmg)
176 grp_tags=list(grp_tags)/project/b/babul/rennehan/anaconda3/envs/yt_caesar/lib/python3.9/site-packages/joblib/parallel.py in call(self, iterable)
1054
1055 with self._backend.retrieval_context():
-> 1056 self.retrieve()
1057 # Make sure that we get a last message telling us we are done
1058 elapsed_time = time.time() - self._start_time/project/b/babul/rennehan/anaconda3/envs/yt_caesar/lib/python3.9/site-packages/joblib/parallel.py in retrieve(self)
933 try:
934 if getattr(self._backend, 'supports_timeout', False):
--> 935 self._output.extend(job.get(timeout=self.timeout))
936 else:
937 self._output.extend(job.get())/project/b/babul/rennehan/anaconda3/envs/yt_caesar/lib/python3.9/site-packages/joblib/_parallel_backends.py in wrap_future_result(future, timeout)
540 AsyncResults.get from multiprocessing."""
541 try:
--> 542 return future.result(timeout=timeout)
543 except CfTimeoutError as e:
544 raise TimeoutError from e/project/b/babul/rennehan/anaconda3/envs/yt_caesar/lib/python3.9/concurrent/futures/_base.py in result(self, timeout)
443 raise CancelledError()
444 elif self._state == FINISHED:
--> 445 return self.__get_result()
446 else:
447 raise TimeoutError()/project/b/babul/rennehan/anaconda3/envs/yt_caesar/lib/python3.9/concurrent/futures/_base.py in __get_result(self)
388 if self._exception:
389 try:
--> 390 raise self._exception
391 finally:
392 # Break a reference cycle with the exception in self._exceptionTerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.
The exit codes of the workers are {SIGKILL(-9)}
Any help would be appreciated! Thanks.
fof6d error in galaxy finding at high redshift
Here are the calculations with errors:
In [1]: import caesar, yt
In [2]: ds = yt.load('/home2/weiguang/data7/Gizmo-Simba/NewMDCLUSTER_0037/snap_022.hdf5')
In [3]: obj = caesar.CAESAR(ds,skip_hash_check=True)
In [4]: obj.member_search(haloid='snap',fof6d_file='/tmp/fof6d',nproc=1,lowres=[2,3])
yt : [INFO ] 2020-12-17 22:56:34,861 member_search() running on 1 cores
/home2/weiguang/caesar/caesar/data_manager.py:32: SyntaxWarning: "is" with a literal. Did you mean "=="?
if select is 'all': self._load_gas_data()
/home2/weiguang/caesar/caesar/data_manager.py:34: SyntaxWarning: "is" with a literal. Did you mean "=="?
if select is 'all': self._load_star_data()
/home2/weiguang/caesar/caesar/data_manager.py:37: SyntaxWarning: "is" with a literal. Did you mean "=="?
if select is 'all': self._load_bh_data()
/home2/weiguang/caesar/caesar/data_manager.py:153: SyntaxWarning: "is" with a literal. Did you mean "=="?
if select is 'all':
/home2/weiguang/caesar/caesar/data_manager.py:213: SyntaxWarning: "is" with a literal. Did you mean "=="?
if select is 'all':
/home2/weiguang/caesar/caesar/data_manager.py:245: SyntaxWarning: "is" with a literal. Did you mean "=="?
if select is 'all':
yt : [INFO ] 2020-12-17 22:56:35,141 No black holes found, RAM=0.2813 GB
yt : [INFO ] 2020-12-17 22:56:35,152 Calculated Omega_Baryon=0.0482017 and 3840^3 effective resolution
yt : [INFO ] 2020-12-17 22:56:35,152 Using FOF Halo ID from snapshots, RAM=0.2813 GB
/home2/weiguang/.local/lib/python3.8/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
return array(a, dtype, copy=False, order=order)
yt : [INFO ] 2020-12-17 22:56:35,740 Initializing member search, loading particles, RAM=0.367 GB
yt : [INFO ] 2020-12-17 22:56:35,740 No black holes found, RAM=0.367 GB
yt : [INFO ] 2020-12-17 22:56:37,971 Loaded particle data, RAM=0.3666 GB
yt : [INFO ] 2020-12-17 22:56:39,087 Loaded baryon data, RAM=0.3454 GB
yt : [INFO ] 2020-12-17 22:56:39,146 Found 1 valid halos, loaded indexes, RAM=0.3454 GB
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-4-7e59a27c3716> in <module>
----> 1 obj.member_search(haloid='snap',fof6d_file='/tmp/fof6d',nproc=1,lowres=[2,3])
~/caesar/caesar/main.py in member_search(self, *args, **kwargs)
338 else:
339 from caesar.fubar_halo import fubar_halo
--> 340 fubar_halo(self)
341 assign.assign_galaxies_to_halos(self)
342 assign.assign_clouds_to_galaxies(self)
~/caesar/caesar/fubar_halo.py in fubar_halo(obj)
71 mylog.warning('No valid halos found! Aborting member search')
72 return
---> 73 get_group_properties(halos,halos.obj.halo_list) # compute halo properties
74
75 # Find galaxies, or load galaxy membership info
~/caesar/caesar/group.py in get_group_properties(self, grp_list)
774 def get_group_properties(self,grp_list):
775
--> 776 from caesar.group_funcs import get_group_overall_properties,get_group_gas_properties,get_group_star_properties,get_group_bh_properties
777
778 get_group_overall_properties(self,grp_list)
ImportError: cannot import name 'get_group_overall_properties' from 'caesar.group_funcs' (unknown location)
### a little more details on the variable values:
---------------------------------------------------------------------------------------------------------------------------
ipdb> mygals
array([0, 1])
ipdb> len(grp_tags)
1
ipdb> len(self.obj.halo_list)
1
ipdb> grp_tags[ih]
array([-1, 1, 1, 1, 1, 1, -1, 1, 1, -1, -1, 1, 1, 1, 1, -1, -1,
-1, -1, 1, 1, 1, 1, 1, 1, 0, -1, -1, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, -1, -1, -1, -1, -1, -1, -1, -1, 0, 0, 0, 0, -1,
-1, -1, -1, 0, -1, 0, 0, 0, 0, 1, -1, -1, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1,
1, 1, -1, 1, -1, 1, 1, 1, 1])
ipdb> g_inds[0].size
94
ipdb> grp_tags[0].size
94
ipdb> len_hi
250
ipdb> len_gi
94
This seems to happen when there is only one halo. The tags in grp_tags don’t get removed properly.
Windows Installation
It can't be installed on Windows OS
Bug with cKDTree positions
I have recently ran into an odd bug when calling the scipy.spatial.cKDTree function within calculate_local_densities in caesar/utils.py. I have managed to fix this by swapping the code currently present for a functionally identical (as far as I can tell) numpy function. Just wanted to share this issue here as I am still unsure what has caused this to happen.
This is the relevant code snippet:
pos = np.array([i.pos for i in group_list])
mass = np.array([i.masses['total'] for i in group_list])
box = obj.simulation.boxsize
box = np.array([box,box,box])
for i in range(3):
pos[pos[:,i]>box[i], i] -= box[i]
pos[pos[:,i]<0, i] += box[i]
TREE = cKDTree(pos, boxsize=obj.simulation.boxsize)
The error I was getting was from cKDTree, in which it said that there were positions outside of the bounds of the box, and the code would crash. The loop seen in the code snippet just before the call to cKDTree is moving particles to be within the boxsize using the periodic boundary conditions. However, upon printing the maximum positions of particles in each coordinate axis after this loop, I noticed that the array slicing was not catching some particles in the i=2 coordinate axis which exceeded the boxsize. Due to this, the cKDTree function was throwing a ValueError and crashing the code.
I replaced this loop with the following:
np.mod(pos,box,out=pos)
which handles all three coordinate directions at once. Functionally this achieves the same goal as the previous implementation, however it does not miss these particles which are exceeding the boxsize. My Caesar code has ran fine on the offending snapshot with this change implemented.
I am unsure what is wrong with the original implementation such that these coordinates are missed with the array slicing method but not by using the numpy.mod function.
failure with yt4.x
running a code like:
snapfile = '/orange/narayanan/s.lower/simba/m25n256_dm/output/run0/snapshot_000.hdf5'
caesarfile = '/orange/narayanan/s.lower/simba/m25n256_dm/output/run0/caesar_000.hdf5'
#fof6d_file =
ds = yt.load(snapfile)
obj = caesar.CAESAR(ds)
obj.member_search(haloid='fof')
obj.save(caesarfile)
returns:
yt : [INFO ] 2021-06-30 11:28:58,772 Calculating time from 9.100e-02 to be 1.516e+16 seconds
yt : [INFO ] 2021-06-30 11:28:58,773 Assuming length units are in kpc/h (comoving)
yt : [INFO ] 2021-06-30 11:28:58,958 Parameters: current_time = 1.5157411453791386e+16 s
yt : [INFO ] 2021-06-30 11:28:58,958 Parameters: domain_dimensions = [1 1 1]
yt : [INFO ] 2021-06-30 11:28:58,958 Parameters: domain_left_edge = [0. 0. 0.]
yt : [INFO ] 2021-06-30 11:28:58,959 Parameters: domain_right_edge = [25000. 25000. 25000.]
yt : [INFO ] 2021-06-30 11:28:58,959 Parameters: cosmological_simulation = 1
yt : [INFO ] 2021-06-30 11:28:58,959 Parameters: current_redshift = 9.989011006443139
yt : [INFO ] 2021-06-30 11:28:58,959 Parameters: omega_lambda = 0.7
yt : [INFO ] 2021-06-30 11:28:58,959 Parameters: omega_matter = 0.3
yt : [INFO ] 2021-06-30 11:28:58,959 Parameters: omega_radiation = 0.0
yt : [INFO ] 2021-06-30 11:28:58,959 Parameters: hubble_constant = 0.68
Generating md5 hash: 8193it [00:01, 5662.32it/s]
yt : [INFO ] 2021-06-30 11:29:00,532 Allocating for 1.678e+07 particles
Loading particle index: 100%|██████████████████████████████████████████████████████████████████| 64/64 [00:00<00:00, 40154.89it/s]
Traceback (most recent call last):
File "romeel_caesar_script.py", line 12, in <module>
obj.member_search(haloid='fof')
File "/home/s.lower/caesar/caesar/main.py", line 340, in member_search
fubar_halo(self)
File "/home/s.lower/caesar/caesar/fubar_halo.py", line 46, in fubar_halo
from caesar.fubar import get_mean_interparticle_separation
File "/home/s.lower/caesar/caesar/fubar.py", line 15, in <module>
from yt.data_objects.octree_subset import YTPositionArray
ModuleNotFoundError: No module named 'yt.data_objects.octree_subset'
haloinfo broken for DM only runs
see example below:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[1], line 1
----> 1 obj.haloinfo()
File /blue/narayanan/desika.narayanan/conda/envs/py38/lib/python3.8/site-packages/caesar-0.2b0-py3.8-linux-x86_64.egg/caesar/loader.py:366, in CAESAR.haloinfo(self, top)
365 def haloinfo(self, top=10):
--> 366 info_printer(self, 'halo', top)
File /blue/narayanan/desika.narayanan/conda/envs/py38/lib/python3.8/site-packages/caesar-0.2b0-py3.8-linux-x86_64.egg/caesar/utils.py:152, in info_printer(obj, group_type, top)
150 for o in group_list:
151 cgsm = -1
--> 152 if (hasattr(o,'central_galaxy')) & (hasattr(o.central_galaxy,'masses')): cgsm = o.central_galaxy.masses['stellar']
153 output += ' %04d %0.2e %0.2e %0.2e %0.2e %0.2e\t| %0.2e \n' % \
154 (o.GroupID, o.masses['dm'], o.masses['stellar'],
155 o.masses['gas'],o.radii['total_half_mass'],
156 o.local_number_density['1000'], cgsm)
157 cnt += 1
File /blue/narayanan/desika.narayanan/conda/envs/py38/lib/python3.8/site-packages/caesar-0.2b0-py3.8-linux-x86_64.egg/caesar/loader.py:527, in Halo.__getattr__(self, attr)
523 if attr in self.obj._halo_dicts:
524 return LazyDict(
525 self.obj._halo_dicts[attr].keys(),
526 lambda d: self.obj._halo_dicts[attr][d][self._index])
--> 527 raise AttributeError("'{}' object as no attribute '{}'".format(
528 self.__class__.__name__, attr))
AttributeError: 'Halo' object as no attribute 'central_galaxy'
docs are not up to date
the RTDs are tied to @rthompsonj's original caesar code, and not dnarayanan/caesar, but since the latter fork is the up to date version, the docs need to change to point to our fork.
search radius is hard coded
in simulation_attributes.py we have:
self.search_radius = ds.quan(1000.0, sru).to(obj.units['length'])
this is hard coded and we should make this a run time argument possibility
Installation Error unknown file type '.pyx'
'caesar.group_funcs' extension
error: unknown file type '.pyx' (from 'caesar/group_funcs/group_funcs.pyx')
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.