ferrine / hyrnn

Hyperbolic Neural Networks, pytorch

Hyperbolic RNN in Pytorch

Prerequisites

You need Python3.6 to run the code. The list of the dependencies are in requrements.txt. Run:

python -m pip install -r requirements.txt

Training

To reproduce the results from the Table 1 on the final report, run the following command:

cd ./experiment_run
python run.py --data_dir=./data --num_epochs=30 --log_dir=./logs --batch_size=1024 --num_layers=2 --cell_type=hyp_gru

where you can change the argument cell_type=hyp_gru to cell_type=eucl_gru if you want to run Euclidean version of GRU.

Note that training can take up to 12-15 hours, having the batch size of 1024. You can increase the batch size to get the results more quicker, but wait for some slight accuracy drop.

Results

Model	Value
Fully Euclidean GRU / B=64	93.25
Fully Hyperbolic GRU / B=1024	96.8

TODO

• Abstract interface for Riemannian manifolds, embed- ded in ambient real coordinate space.
• Compatible generic RSGD.
• Compatible generic RAdam.
• Compatible implementation of Poincare ball and Mo- bius arithmetics.
• Test coverage for optimization routines.
• GRU based on Mobius arithmetics, API-compatible with torch.nn.GRU.
• Layers parameterized by pivots of Log and Exp, as opposed to fixed pivot of 0 in Mobius arithmetics-based layers.
• Test coverage for “Mobius” layers and RNN loops.
• Numerical stability with float64.
• Numerical stability with float32.
• Investigation of possibility of using cudnn loop.
• C++ implementation of core operations and loops.

hyrnn's Issues

RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.

I'm sure that all the packedge and codes are up to date. But it cannot run using default settings, and when setting decision_type="hyp", the hyrnn.MobiusDist2Hyperplane module still has a problem as follows.

File "....../hyrnn/nets.py", line 180, in init
self.tangent = geoopt.ManifoldParameter(tangent, manifold=sphere).proj_()
File "...../geoopt/tensor.py", line 40, in proj_
return copy_or_set_(self, self.manifold.projx(self))
File "..../geoopt/utils.py", line 24, in copy_or_set_
return dest.set_(source)
RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.

Can't reproduce reported results

Hi, I ran the exact argument you provided in the README.md file
python run.py --data_dir=./data --num_epochs=30 --log_dir=./logs --batch_size=1024 --num_layers=2 --cell_type=hyp_gru
which yields a loss of 0.678 and precision of ~56%
I also ran
python run.py --data_dir=./data --num_epochs=30 --log_dir=./hyp_gru_hyp_decision --batch_size=1024 --num_layers=2 --cell_type=hyp_gru --decision_type hyp
which yields a loss of 0.693 and precision of ~50%
Do you know any potential reason for that?

Relu() between MobiusLinear

Hi, I am wondering if I can add a relu() between two MobiusLinear. Have you designed a hyperbolic relu()?

Update the using of newest packedges geoopt and catalyst.

The latest version of hyrnn uses: "geoopt.PoincareBall(c=c).set_default_order(order)" that encounters a problem: "AttributeError: 'PoincareBall' object has no attribute 'set_default_order'". And there are some other problems. So, the latest version of hyrnn should be updated to meet the newest packedges catalyst and geoopt.

Sphere manifold in MobiusDist2Hyperplane

Hi, I have a question regarding the implementation of the MobiusDist2Hyperplane class.

In the line that I link after this paragraph, the tangent parameter is a ManifoldParameter where the manifold is a Sphere manifold, and not a PoincareBall. Could you explain why that is the case?

hyrnn/hyrnn/nets.py

Line 181 in d53fa4b

self.tangent = geoopt.ManifoldParameter(tangent, manifold=sphere).proj_()

Thanks!

"project" in mobius_linear in nets.py

Hi. I find there is a "output = pmath.project(output, c=c)" in in mobius_linear() in nets.py. The reason is that the output of pmath.mobius_matvec can extend "c". I am confused about this because I believe all the operations can make the output still inside the poincare ball.

Full hyperbolic model

The sample command you give in the README.md file is
python run.py --data_dir=./data --num_epochs=30 --log_dir=./logs --batch_size=1024 --num_layers=2 --cell_type=hyp_gru
But I notice there are two arguments in run.py: embedding_type and decision_type, which are set to eucl by default. Does this mean the results you generated for hyp_gru performs logistic regression in the euclidean space?

ModuleNotFoundError: No module named 'geoopt.manifolds.poincare'

Hi, I meet a problem when I run the run.py. Does this error comes from the update of geoopt? I also find that there is a class named PoincareBall in geoopt.manifolds.stereographic, Is this and geoopt.manifolds.poincare the same?

test_model.py: ModuleNotFoundError

@CoderINusE

tests/test_model.py:5: in <module>
    from model import RNNBase
E   ModuleNotFoundError: No module named 'model'