[BUG] Enabling limited API attempts to take the size of an opaque object about cython

[da-woods]

This'll be fixed in #5871 I think

[da-woods]

The current limited API PRs are sufficient to get Cython itself to compile so it'll actually be in a pretty usable state when those are all finished.

https://github.com/cython/cython/pulls?q=is%3Apr+is%3Aopen+sort%3Aupdated-desc+label%3A%22limited+api%22

[da-woods]

Duplicate of #5634

[vyasr]

Ah I'm sorry It took me long enough to zero in on the exact error that I didn't have the presence of mind to check for duplicates once I'd come up with a true MWE. Glad to see the progress, though! Looking forward to those PRs getting in. I'll look through them and see if all my local changes are covered; if not, I can submit a PR with what remains.

[da-woods]

Any local changes you have would be welcome - thanks. I'd largely stopped work on the limited API after getting Cython to compile itself because it was getting hard to work and test things in standalone branches.

[vyasr]

Could you elaborate a bit more on that? Were you running into problems with trying to create modular PRs that addressed specific gaps in the limited API coverage in Cython? I agree, that sounds difficult. IMO the best path forward is likely that once you get your PRs merged we go for a while where people make whatever contributions are needed to get their own projects to compile until coverage increases enough over time that a larger scale effort can be made. In the long run I think the only way for Cython to claim full support is probably running the whole test suite with the limited API on, but that is a heavy ask that is presumably a long ways away.

Speaking of which, let me know if there's anything I can do to help your PRs along. Unfortunately they are touching deeper parts of the Cython internals and using more of the Python C API than I have expertise with, but happy to test and I'm hopefully building some expertise by reading through them to let me help in the future.

[vyasr]

I opened #6061. I checked that it didn't overlap with any of yours (it only overlaps in one file the ModuleSetupCode.c file, but I'm changing different lines than all your PRs I think) so hopefully they can go in independently.

[da-woods]

Could you elaborate a bit more on that?

It was mostly about testing meaningfully and finding specific gaps.

If the initial work on it I'd created tests/run/isolated_limited_api.srctree as a way of testing the little bits that were working. I think the isolated tests were beginning to outlive their usefulness because it was getting harder to say "I've fixed this specific feature". And because some of the PRs depended on other PRs to actually be usable.

In the long run I think the only way for Cython to claim full support is probably running the whole test suite with the limited API on, [...]

Yes - we all agree on that. It wasn't really possible to use the full Cython test suite because it was crashing a bit unpredictably (there were/are almost certainly reference counting errors somewhere). Tests failing is OK, but if they crash the test runner doesn't detect it and ends up just timing out.

[...] but that is a heavy ask that is presumably a long ways away.

My plan for the next thing to do on the limited API was to start turning the tests on. Probably initially going one at a time with a known list of good tests.

Speaking of which, let me know if there's anything I can do to help your PRs along.

Any review would obviously be helpful. At some point I'm going to get a little more pro-active about just merging some of them myself.

A lot of it is pretty mechanical translation, but some of them have design decisions in that maybe need to be discussed (e.g. #5871 adds ways of conditionally compiling cdef functions. I think that's needed but there might be other ways to solve the underlying problems).

The other thing I'm considering is adding a "limited-api-preview" branch just containing all the unmerged limited API PRs. That would make it easier for anyone interested to test. (I wouldn't release anything, or keep it strictly updated with "master"; it'd just be a branch for anyone interested). I'll do that in a few days - just waiting for one other change before fixing some conflicts.