Comments (3)
Profiling information:
1137798 function calls (1114185 primitive calls) in 1133.420 CPU seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 74 539.657 7.293 539.657 7.293 {method 'isInsideV' of 'cPolygon.Polygon' objects} 75 225.521 3.007 297.129 3.962 bhpix.py:9(proj_healpix) 518 140.781 0.272 140.781 0.272 {method '_read_records' of 'tables.tableExtension.Table' objects} 75 101.655 1.355 402.557 5.367 bhpix.py:38(proj_bhealpix) 227 71.608 0.315 71.608 0.315 {numpy.core.multiarray.where} 74 11.384 0.154 11.384 0.154 {lsd.native.table_join} 965 9.384 0.010 9.412 0.010 colgroup.py:149(__getitem__) 1408 7.022 0.005 7.022 0.005 {method 'any' of 'numpy.ndarray' objects} 444 5.300 0.012 5.300 0.012 {numpy.core.multiarray.concatenate} 79 3.772 0.048 3.772 0.048 {method 'astype' of 'numpy.ndarray' objects} 74 3.366 0.045 962.203 13.003 join_ops.py:270(filter_space) 153 3.017 0.020 3.017 0.020 {numpy.core.multiarray.arange} 148 1.597 0.011 1131.155 7.643 join_ops.py:966(__iter__) 296 1.330 0.004 1.384 0.005 {method '_g_new' of 'tables.hdf5Extension.File' objects} 148/74 0.675 0.005 1111.929 15.026 join_ops.py:172(evaluate_join) 74 0.325 0.004 0.325 0.004 {method 'fill' of 'numpy.ndarray' objects} 148 0.322 0.002 1129.557 7.632 join_ops.py:511(__iter__) 18574 0.267 0.000 0.267 0.000 {method '_g_getAttr' of 'tables.hdf5Extension.AttributeSet' objects} 296 0.252 0.001 0.252 0.001 {method '_closeFile' of 'tables.hdf5Extension.File' objects}
from lsd.
The problem was in computation of whether stars fall within the given boundaries or not. Since the boundaries are arbitrary polygons, this becomes slow once a cell with lots of objects gets hit. Compounding the problem, when joining tables this calculation gets repeated for each cell that is joined, so about ~70x as many times than when there is no join (for PanSTARRS MDF fields).
I fixed it by introducing a "result cache", that remembers (on disk) the results of "heavy" functions and reuses them when those functions get called again. It's a general mechanism that will also allow caching of database files on local disk on the node, thus avoiding the slow network I/O to /n/panlfs (that feature will come later).
from lsd.
Fixed in commits 741f275 through a981340
from lsd.
Related Issues (19)
- Switch to file IO in pool2 instead of mmap, to support OSX
- MapReduce temporary files created in current directory
- Check for reserved words in column names
- Check for validity of table and column names
- Float-Array mismatch error on lsd-import fits? HOT 1
- Impossible to match a single detection to an object HOT 1
- "Exception: Another transaction is already ongoing" (even though there isn't) HOT 1
- bounds fails at specific galactic location
- bounds fails at specific galactic location HOT 1
- "AttributeError: Table instance has no attribute 'remote'" HOT 1
- NWORKERS keeps CPU down but not memory
- Replace pyfits dependency with astropy.io.fits HOT 1
- SFD builtin uses deprecated wcs_sky2pix
- lsd-admin: error message if .__dblock.lock already present in $LSD_DB
- Web Installer Broken HOT 1
- python 3 incompatible
- LSD @ Harvard directory not found
- Suppress stdout and stderr from workers
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lsd.