Remove dependency on scipy.stats for distributions about numpyro

[neerajprad]

I prefer to keep our current api (for rvs/logdf/frozen) and just rewrite the init method so that it only initializes things we need (AFAIK, we only need the parse_args method to automatically parse args).

I think this is worth discussing and fleshing out a bit. I used scipy to begin with as it was the choice of least resistance, but I am not convinced that following any of the design patterns from scipy will be a good long term plan. :)

My thought is that there is little benefit in maintaining frozen / unfrozen distribution objects, and significant maintenance cost. A lot of the convoluted code that we have inherited stems precisely from having to maintain both of these patterns.

I am not sure about why the scipy devs decided to support this, but I have not come across any use case for the non-frozen distribution instances, and our inference algorithms don't use that either. There aren't any perf benefits worth mentioning as well since sampling time would typically outweigh any minor savings from not having to create class instances to begin with. The initial version of Pyro actually followed this scipy pattern of frozen / non-frozen distributions but we changed to using distribution classes due to the tech debt that this decision led to!

I am not sure if we would need parse_args for the same reason. Instead, we could also just remove the loc, scale args from all the continuous distributions, and if users need to scale or translate their instances, using a TransformedDistribution is probably a better design pattern. I think if we stick closer to torch.distributions, we can just build off of the work that we know works well. If there are some deficiencies in the design of torch distributions, we shouldn't certainly overlook that and improve on it though.

[neerajprad]

I think this will require quite a few changes, so I'll try to figure out a way to do this more gradually, maybe by prototyping this in a separate module that can later be swapped out if we are happy with the design. For now, let us continue to use what we have, but I would like to address this soon rather than add more layers on top of an already convoluted design.

[neerajprad]

@fehiepsi - Let me know your thoughts. I think we should address this soon, so that we don't pile on more tech debt. I can take a look at it.

[fehiepsi]

I think to begin with we can start by only supporting sample, log_prob methods and adding more as we go along.

Agree. Currently, we only use rvs/logpdf for doing inference so there is no need to get inherited from scipy. I prefer to keep our current api (for rvs/logdf/frozen) and just rewrite the init method so that it only initializes things we need (AFAIK, we only need the parse_args method to automatically parse args). Other infos such as momtype=1, a=None, b=None, xtol=1e-14, badvalue=None, longname=None, extradoc=None, seed=None are not necessary right now. We can add more functionalities from scipy or from our own usage later.

[fehiepsi]

Hmm... so your proposal is to not use the api of scipy.stats. I misunderstood it. That makes sense to me! The repo is numpyro, not scipyro, so we are free to design for our convenience. 👍 :D Pls go for it!

[fehiepsi]

Previously, I put more weights on scipy users (so they just write a model with their familiarity). But now, I think that using pytorch/pyro dist interface will make transitions from pyro and numpyro easier for users (we only need to change the import torch.distributions by numpyro.distributions) without having to modify the model. It also makes it easier to debug what is going wrong by comparing to a corresponding pyro inference. I think that this aligns with your ideas so please feel free to build what you intend to do.