Benchmarking Tools about reproject

[astrofrog]

YES! We definitely need a small benchmark project. Of course, we can also include Montage in the benchmark. Do you want to create a repository for this?

[keflavich]

Yeah, sure, will do. Any thoughts on good input images to test? I was just using a single gaussian...

[keflavich]

Also, do you know how to use kapteyn well enough to include it in such a suite?

[astrofrog]

@keflavich - how about picking a cutout from a Spitzer survey? I don't use kapteyn, so not really.

[keflavich]

@astrofrog - OK, will try a cutout from Spitzer at some point. For now, starting with the ultra-trivial examples...

https://github.com/keflavich/reprojection_benchmarks

[astrofrog]

Awesome! This will be a great way to benchmark accuracy, speed, and approximations!

[keflavich]

yep. Right now, it looks like everything is wrong... hooray!

[astrofrog]

It's going to be interesting ;) If the cutout is small enough, can you include it in the repository?

[keflavich]

So it looks like starlink crops, reproject leaves nans as nans, and hcongrid turns nans to zeros (but there's a setting for that).

I think this indicates a problem in reproject, actually.

[keflavich]

@astrofrog - yes, will include any cutouts. For now, I'm using generated data, as I think it's a more reliable starting point. There might even be a True, Correct, Analytical "Right Answer" when using generated data.

[astrofrog]

@keflavich - just so I understand, is reproject the one in this package, or the wrapper around Montage?

[keflavich]

@astrofrog - this package. I'll rename it 'pyreproject' as I'm adding the montage-reproject now

[astrofrog]

I wouldn't be surprised that pyreproject is wrong :) But hopefully it will guide us towards the 'right'.

[keflavich]

Yep! Here it is with montage:

[keflavich]

Interesting - the packages clearly treat edge conditions differently. I think that's tunable in montage, using exact_size. Not sure about starlink. @dsberry, whenever you have a chance, could you comment on when/whether AST chooses to crop out NaN pixels along the edges?

[keflavich]

@astrofrog - if you have any tips or ideas on how to make a better grid without spending half an hour tuning the plot, I'd be interested. I've started writing wrappers of this kind of stuff for myself recently in https://github.com/keflavich/mpl_plot_templates; perhaps an aplpy-grid wrapper is called for

[keflavich]

Timing benchmarks now:

reproject_cmd               8              16              32              64             128
    pyreproject  0.208455085754  0.575993061066   2.24823093414   7.49773812294   29.2678470612
    montage_hdu  0.541980981827  0.587301015854  0.491894006729  0.515506029129  0.596334934235
   hcongrid_hdu  0.070965051651  0.076486825943 0.0782270431519 0.0865521430969  0.129561901093
       wcsalign  0.286320924759  0.232869148254  0.229321956635  0.230861902237  0.239408016205

[astrofrog]

cool - as I said, pyreproject has not at all been optimized yet :)

[keflavich]

Indeed, but now you can! Those nested for-loops look ripe for
cython/numba/pypy-ification.

[astrofrog]

@keflavich - indeed! Actually I first want to make sure the results are correct, which I think they are not. Then, I'll add a unit test that verifies the benchmark output. Then I'll optimize.

[dsberry]

The Ast.Resample method does not crop the output - it is created with the pixel bounds you request. If this area is larger than the area containing good input values, then the borders are filled with bad values. Likewise, any bad values in the input are propagated to the output if they fall within the requested output bounds. But starlink does not use NaNs to represent bad values, it uses a special value which you specify using the "badval" argument when calling Resample. See the section "Propagation of Missing Data" at http://starlink.jach.hawaii.edu/devdocs/sun211.htx/node384.html