iesl / box-embeddings Goto Github PK

1. Embedding wordnet like in gumbel box paper.

2. 2-class NLI using Multi-NLI.

Implement each one in Tensorflow and Pytorch in separate notebooks in an examples folder at the root level.

Add a RELEASE.md describing steps that need to be taken before making a release.

1. Update the version in setup.py
2. Check if the test coverage is above a threshold.
3. Check if the CHANGELOG is up-to-date.
4. Create a tag matching the version in the setup.py. Copy the "unreleased" section of the CHANGELOG into the description for the tag and release.

• Contributors
• Branches
• Tags

Examples of the following:

1. Train shallow box representations (can / should be on a simple toy dataset, eg. mammal, birds, etc. maybe do both BCE and max-margin)
2. How you use it on the output of BERT (maybe NLI?)
3. (Maybe more in the future)
   Probably do this in notebooks: https://mybinder.org/

Hello, what command is used to run the "examples" and what settings are required?

Currently we have gumbel_beta and also beta in softplus, this is confusing.

We should change to using intersection_temperature (taking the place of current gumbel_beta) and volume_temperature (where 1/volume_temperature takes the place of current beta in softplus calculation).

Clamping will kill the gradient. Adding a minimum eps will not.

Currently, the volume function is an approximation of the Bessel volume in this repo. However, I have tried to implement an exact version of the Bessel volume in the past. It was numerically not stable. I would like to request you to have a look at the code snippet and see how this could be appended to this repo.

The bessel function wrapper

class Bessel(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
        ctx.save_for_backward(input)
        dev = input.device
        with torch.no_grad():
            x = special.k0(input.detach().cpu()).to(dev)
            input.to(dev)
        return x

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_tensors
        dev = grad_output.device
        with torch.no_grad():
            grad_input = grad_output*(-special.k1(input.detach().cpu())).to(dev)
            input.to(dev)

        return grad_input

The volume function

    def _log_bessel_volume(cls,
                           z: Tensor,
                           Z: Tensor,
                           gumbel_beta: float=1.,
                           scale: Union[float, Tensor] = 1.) -> Tensor:
        eps = torch.finfo(z.dtype).tiny
        if isinstance(scale, float):
            s = torch.tensor(scale)
        else:
            s = scale
        element = (2*torch.exp((z-Z)/(2*gumbel_beta))).clamp_max(100)
        return (torch.sum(
            torch.log(2*gumbel_beta*Bessel.apply(element).clamp_min(eps)),
            dim=-1) + torch.log(s)
        )

It is a general library that can be used for NLP or graph learning. Calling it an NLP library might be too restrictive. @mboratko @ssdasgupta What do you think?

Once we have minus_Z and a parameterization that stores minuz_Z instead of Z directly, we can use z and minus_Z to compute intersection and volume.

Side-length based

1. L2 on side-length

2. L1 on side-length

3. Huber (smooth L1) on side lengths.

Volume-based

1. L1, L2 on box volumes.

Change the top level imports so that the tensorflow version can be imported without pytorch

The denominator can be zero and result in NaN output. Adding a small epsilon will help.

Storing raw notebook in VCS is messy and not recommended. Can we use jupytext instead? Will jupytext work with MyBinder?

https://mybinder.org/

1. Change theme (PyTorch? FastAPI? https://fastapi.tiangolo.com/)
2. Look into why inherented methods are appearing in subclasses (maybe a flag can turn this off)

One issue with subclassing will be the creation from zZ. We want to avoid the inverse-forward roundtrips whenever possible.

isinstance(tf.Variable([1]), tf.Tensor) returns False but returns True for tf.constant([1]). The variable implementation will be required for computing gradients as constants cannot be modified in tensorflow. Currently the code is functional for the tf.constant([1]) implementation

Personal branches (other than main and dev/main) seem to still require precommit hooks to pass.

Currently we need to have a separate embedding layer which we then pass to some box parameterization, it would be nice to wrap this so box embeddings can be created directly.

Current description mentions "Pytorch implementation for box..". Should be updated with Tensorflow too

We need this because of the numerical issues with pytorch's softplus. While creating conditional probability we need to concatenate, take volume, and split again.

Optional step: Find the failure points: on what shapes and values does this happen?
First step: Profile the difference between applying softplus on two tensors or applying softplus on their concatenation and split it. Profile both runtime and memory usage.

Capability to model boxes with fixed side-lengths.

Ref: https://github.com/dhruvdcoder/box-embeddings/blob/master/box_embeddings/parameterizations/center_fixed_delta_tensor.py

Hi,

I'm new to the box embeddings. I'm wondering if the current implementation has some sort of indexing method? E.g. if a box tensor contains 64 16-dimensional boxes, what's the best way to access specific boxes in this tensor? I think constructing a new box tensor with .from_zZ() could work but just wondering if we can do this more efficiently.

Thanks!

Currently, we stack up the two BoxTensors to compute intersection using logsumexp.

This does not support broadcasting. For instance, if we have two BoxTensors of box_shape (batch1, 1, box_dim) and (1, batch2, box_dim), all the other intersection ops can produce intersection box with box_shape (batch1, batch2, box_dim) but gumbel_intersection cannot. This operation is required for multilabel classification using boxes.

Plan:

1. Implement a logsumexp that works with two tensors and that is API consistent with torch.maximum(). Call this operator real_softmax because it is essentially a differentiable max operation.

2. Update the current gumbel_intersection to use real_softmax instead of stack + logsumexp.

3. Write boundary test-cases to check for numerical issues (underflow as well as overflow).

Follow the format from AllenNLP: https://github.com/allenai/allennlp/blob/main/CHANGELOG.md

https://www.iesl.cs.umass.edu/box-embeddings/main/index.html

Write a test either confirming the numerical issue or confirming the absence of it.

Line:

Change this to :

with open("README.md", "r", encoding="utf8") as fh:

This would be added to the base class, where we can perform it generically using zZ, but certain parameterizations may override this with more efficient implementations

I think it would make sense to consolidate all volume (and intersection) functions into a single one, where if the variance is zero we revert to the "hard" versions of these functions.

General code where you visualize multiple boxes.
2 functions:

1. One which can plot boxes in 2d
2. One which plots boxes in n dimensions, using a horizontal / vertical stack of one dimensional boxes

For the API, we should look into "grammar of graphics".

Add code examples in the docstrings for functions.
Use the tests as an example.