BUG: test_discrete_basic fails on scipy-1.13.1-cp311-cp311-win_arm64 about scipy

[WarrenWeckesser]

Some history, FYI: I added that nudge, originally 10*np.spacing(cdf_supp), back in #17482. See my comment about it here: #17482 (comment) (starting at "Only one test had to be fixed...").

The nudge was increased to 15*np.spacing(cdf_supp) in #17432, with comments about it at https://github.com/scipy/scipy/pull/17432/files#r1090118460, and reference to the boost issue that is still open.

[mdhaber]

Ok, we can increase it to 30.

That doesn't annoy you?

Sure it does. But someone told me years ago that this was normal, and have been under that impression every since. I guess it wasn't you, then. But it sounded right because there are a lot of different machines out there.

I can't even run the full test suite on Windows because test_import_cycling hangs. But I'll disable that and file a report.

[mdhaber]

@cgohlke are you using nbinom in your application?

@tylerjereddy This was labeled stats, but I'm afraid I'd be unable to debug given that this is about Windows on ARM64. The test notes:

    # In very rare cases, the finite precision calculation of ppf(cdf(supp))
    # can produce an array in which an element is off by one.  We nudge the
    # CDF values down by 15 ULPs help to avoid this.

So what the user is reporting is consistent with a known issue in the test itself, and we do not typically adjust tests to ensure that they pass on all possible machines. I get more test failures on my own machine, TBH. Is this a wontfix?

[cgohlke]

are you using nbinom in your application?

No, at least not directly.

If this is a known issue with the test, I'm OK to close this issue.

[mdhaber]

I'll at least test whether it is close to the boundary. If so, I'll close. Thanks for understanding.

[mdhaber]

Can you try changing the 15 ULP used in the test to, say, 30? How far do you have to nudge it to make it pass?

Would you also run this:

from scipy import stats
import numpy as np
arg = (5, 0.5)
distfn = stats.nbinom

x = [5, 6, 7]
cdf = stats.nbinom.cdf(x, *arg)
print(cdf[1])  # 0.7255859375
ppf = stats.nbinom.ppf(cdf, *arg)
print(ppf[1])  # 6.0
ppf = stats.nbinom.ppf(cdf - 15 * np.spacing(cdf), *arg)
print(ppf[1])  # 6.0

x = 6
cdf = stats.nbinom.cdf(x, *arg)
print(cdf)  # 0.7255859375
ppf = stats.nbinom.ppf(cdf, *arg)
print(ppf)  # 6.0
ppf = stats.nbinom.ppf(cdf - 15*np.spacing(cdf), *arg)
print(ppf)  # 6.0

# use this import in 1.13.1
# from scipy.stats._boost import _nbinom_ppf, _nbinom_cdf
# use this import in main, instead
from scipy.special._ufuncs import _nbinom_ppf, _nbinom_cdf
cdf = _nbinom_cdf(6, 5, 0.5)
print(cdf)  # 0.7255859375
ppf = _nbinom_ppf(cdf, 5, 0.5)
print(ppf)  # 6.0

import matplotlib.pyplot as plt
x0 = 0.7255859375
p = x0 + np.arange(-30, 31) * np.spacing(x0)
x = _nbinom_ppf(p, 5, 0.5)
plt.plot(x)
plt.show()

And LMK the output?

It would also be helpful if you'd do the same with main. This problem could have been fixed with a big change to the way we get distribution special functions from Boost.

For me, the plot looks like:

[cgohlke]

try changing the 15 ULP used in the test to, say, 30

The test passes with 30 ULP.

Would you also run this:
And LMK the output?

0.7255859375
6.0
7.0
0.7255859375
6.0
7.0
0.7255859375
6.0

[mdhaber]

Thanks. It looks like Boost's _nbinom_ppf is bouncing around here, and it so happens that with -15 ULP in particular, it returns 7.0 instead of 6.0. the CDF and PPF are probably both quite accurate and reliable for most inputs, but the test is zooming in on exactly the area where we might have this sort of trouble. I'll let the scipy.special folks comment on whether there's anything to be done or whether it's something they'd like to report to Boost, but I'll go ahead and close the issue in the meantime.

[rgommers]

and we do not typically adjust tests to ensure that they pass on all possible machines.

Eh, this is incorrect. There are exceptions, like very niche platforms that don't implement C99 functions correctly. But beyond that, we consider failures as issues and accept fixes for them. We have countless examples of bumping tolerances when users or distro maintainers report test failures on configurations we don't test.

If the test passes with 30 ULP, we should update the test for that. Either unconditionally, or bump it only on Windows arm64. I haven't looked at the code, but I suspect the former is better since 30 ULP is still very tight.

If you close this kind of thing as wontfix, it just leaves it for the next person to rediscover. I'm reopening, because this is a valid bug report and can be fixed.

I get more test failures on my own machine, TBH.

That doesn't annoy you? I really need zero test failures on my configs, so I can review PRs and work on larger-scale topics without noise.

[rgommers]

Yes, let's get those things resolved.

there are a lot of different machines out there.

The idea is that our CI is representative enough of that very large set of machines/configs, and tolerances are set with reasonable margins so that small variations in how SciPy is built doesn't cause test failures. And if there are recurring/structural problems on some build config, we should add CI coverage for it if we can do so in a reasonable manner.

I think you're using conda envs? Our test suite passes in CI on Windows, but it's possible that we could make things more robust by adding a conda-forge based Windows job.

[mdhaber]

The OP also reported "which is actually an amazing result given that this platform is not supported yet". I tried to find out more information about officially supported platforms in our documentation, but I couldn't verify the statement using Toolchain Roadmap. What do we consider to be supported?

[rgommers]

What do we consider to be supported?

There is no hard line between supported/unsupported. A few thoughts:

• We have some hard requirements, like "must support C17 and C++17" and "all our dependencies work" that are covered in the toolchain roadmap.
• Beyond that, we have varying levels of support (a la PEP 11, just not written down like that - I may do so for NumPy in the near future):
  • tier 1: we release wheels for a platform on PyPI. That is a long-term commitment, we can't easily not publish such wheels for the next release.
  • tier 2: we test a platform in our CI but don't publish wheels (example: 32-bit Linux)
  • tier 3: we don't test a platform at all but do accept patches for it as long as they are reasonable (example: AIX, PowerPC, Windows on Arm, FreeBSD, Emscripten/Pyodide, etc.)

Windows on Arm64 support is clearly desirable, and we know it's going to become more important over time. It's now in tier 3, and once hosted and free CI support becomes available it'll move into tier 2. If it really takes off, at some point I'd expect we are going to publish wheels for it.