iesl / box-embeddings Goto Github PK
View Code? Open in Web Editor NEWBox Embeddings as Modules
Home Page: https://www.iesl.cs.umass.edu/box-embeddings
License: Apache License 2.0
Box Embeddings as Modules
Home Page: https://www.iesl.cs.umass.edu/box-embeddings
License: Apache License 2.0
Embedding wordnet like in gumbel box paper.
2-class NLI using Multi-NLI.
Implement each one in Tensorflow and Pytorch in separate notebooks in an examples
folder at the root level.
Add a RELEASE.md describing steps that need to be taken before making a release.
setup.py
setup.py
. Copy the "unreleased" section of the CHANGELOG into the description for the tag and release.Examples of the following:
Hello, what command is used to run the "examples" and what settings are required?
Currently we have gumbel_beta
and also beta
in softplus, this is confusing.
We should change to using intersection_temperature
(taking the place of current gumbel_beta
) and volume_temperature
(where 1/volume_temperature
takes the place of current beta
in softplus calculation).
Clamping will kill the gradient. Adding a minimum eps will not.
Currently, the volume function is an approximation of the Bessel volume in this repo. However, I have tried to implement an exact version of the Bessel volume in the past. It was numerically not stable. I would like to request you to have a look at the code snippet and see how this could be appended to this repo.
The bessel function wrapper
class Bessel(torch.autograd.Function):
@staticmethod
def forward(ctx, input):
ctx.save_for_backward(input)
dev = input.device
with torch.no_grad():
x = special.k0(input.detach().cpu()).to(dev)
input.to(dev)
return x
@staticmethod
def backward(ctx, grad_output):
input, = ctx.saved_tensors
dev = grad_output.device
with torch.no_grad():
grad_input = grad_output*(-special.k1(input.detach().cpu())).to(dev)
input.to(dev)
return grad_input
The volume function
def _log_bessel_volume(cls,
z: Tensor,
Z: Tensor,
gumbel_beta: float=1.,
scale: Union[float, Tensor] = 1.) -> Tensor:
eps = torch.finfo(z.dtype).tiny
if isinstance(scale, float):
s = torch.tensor(scale)
else:
s = scale
element = (2*torch.exp((z-Z)/(2*gumbel_beta))).clamp_max(100)
return (torch.sum(
torch.log(2*gumbel_beta*Bessel.apply(element).clamp_min(eps)),
dim=-1) + torch.log(s)
)
It is a general library that can be used for NLP or graph learning. Calling it an NLP library might be too restrictive. @mboratko @ssdasgupta What do you think?
Once we have minus_Z and a parameterization that stores minuz_Z instead of Z directly, we can use z and minus_Z to compute intersection and volume.
L2 on side-length
L1 on side-length
Huber (smooth L1) on side lengths.
Change the top level imports so that the tensorflow version can be imported without pytorch
The denominator can be zero and result in NaN output. Adding a small epsilon will help.
Storing raw notebook in VCS is messy and not recommended. Can we use jupytext instead? Will jupytext work with MyBinder?
One issue with subclassing will be the creation from zZ. We want to avoid the inverse-forward roundtrips whenever possible.
isinstance(tf.Variable([1]), tf.Tensor)
returns False but returns True for tf.constant([1])
. The variable implementation will be required for computing gradients as constants cannot be modified in tensorflow. Currently the code is functional for the tf.constant([1])
implementation
Personal branches (other than main
and dev/main
) seem to still require precommit hooks to pass.
def l2_side_regularizer(
box_tensor: BoxTensor, log_scale: bool = False
) -> Union[float, torch.Tensor]:
z = box_tensor.z # (..., box_dim)
Z = box_tensor.Z # (..., box_dim)
if not log_scale:
return torch.mean((Z - z) ** 2)
else:
return torch.mean(torch.log(torch.abs(Z - z) + eps))
Currently we need to have a separate embedding layer which we then pass to some box parameterization, it would be nice to wrap this so box embeddings can be created directly.
Current description mentions "Pytorch implementation for box..". Should be updated with Tensorflow too
We need this because of the numerical issues with pytorch's softplus. While creating conditional probability we need to concatenate, take volume, and split again.
Optional step: Find the failure points: on what shapes and values does this happen?
First step: Profile the difference between applying softplus on two tensors or applying softplus on their concatenation and split it. Profile both runtime and memory usage.
Capability to model boxes with fixed side-lengths.
Hi,
I'm new to the box embeddings. I'm wondering if the current implementation has some sort of indexing method? E.g. if a box tensor contains 64 16-dimensional boxes, what's the best way to access specific boxes in this tensor? I think constructing a new box tensor with .from_zZ()
could work but just wondering if we can do this more efficiently.
Thanks!
Currently, we stack up the two BoxTensors to compute intersection using logsumexp.
This does not support broadcasting. For instance, if we have two BoxTensors of box_shape (batch1, 1, box_dim) and (1, batch2, box_dim), all the other intersection ops can produce intersection box with box_shape (batch1, batch2, box_dim) but gumbel_intersection
cannot. This operation is required for multilabel classification using boxes.
Plan:
Implement a logsumexp that works with two tensors and that is API consistent with torch.maximum(). Call this operator real_softmax
because it is essentially a differentiable max operation.
Update the current gumbel_intersection
to use real_softmax
instead of stack + logsumexp.
Write boundary test-cases to check for numerical issues (underflow as well as overflow).
Follow the format from AllenNLP: https://github.com/allenai/allennlp/blob/main/CHANGELOG.md
Write a test either confirming the numerical issue or confirming the absence of it.
Line:
This would be added to the base class, where we can perform it generically using zZ, but certain parameterizations may override this with more efficient implementations
I think it would make sense to consolidate all volume (and intersection) functions into a single one, where if the variance is zero we revert to the "hard" versions of these functions.
General code where you visualize multiple boxes.
2 functions:
For the API, we should look into "grammar of graphics".
Add code examples in the docstrings for functions.
Use the tests as an example.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.