Some thoughts and question while trying out the hamming embedding idea about ungol

[kantholtz]

Hello - thanks for your message!

In my case, I would like to build up an embedding quantization module on top of an encoder, say BERT, hence it is hard to random sample embeddings as the initial values of codebook. I was wondering if there's any advice or thoughts from you for this case?

From the top of my head I would recommend you also use random encodings from the encoder, i.e. feed it some random text. (Maybe you already have a corpus you are working on?). Do you fine-tune the BERT or are you appending some downstream network before quantization/auto-encoding? If that does not work, and you use BERT directly, its token embeddings went through a tanh (if I remember correctly) - so maybe try sampling from some distribution between -1 and 1.

Getting the initialization right was one of the trickier parts when training this model.

Also, I'm considering to use the latent representation to compute loss and train directly (discard the reconstruction layer), since, in my case, the final goal is to preserve the similarity between two sentences as much as possible, and possibly preserve the ranking of the sentences similarity as much as possible. Thus the reconstruction loss seems not perfectly aligned with the goal.

This is true but I don't think that this is possible - at least unsupervised. Otherwise, you would need some knowledge of how the embeddings need to look like to guide the model while training. Do you have ground truth regarding the similarity? You could try to develop a regularizer to guide the codes in this direction. My insights were that although most codes translated their "geometric" similarity information into Hamming space, this was not really bettering the inherit difficulty to discriminate them - e.g. when you try to use them for clustering. Similarity information in itself is quite latent, too.

(Previously, I did binarization directly, which simply takes the sign of the embedding element,

Interesting - was that giving good results? I think I tried things like that as baseline experiments but don't remember that this worked particularly well.

and use STE to do BP).

What's that? It's been some time... maybe I forgot acronyms ;)

[nachtsky1077]

Thanks for your advice! I'll try out the codebook method.

Just a little background of my case: the data consists of query and document, and training data is formatted as something like (query, positive doc, negative doc1, negative doc2...).

The straight binarization method did give quite fair results in my case(that is to say, the binarized code could preserve the cosine similarity between query and documents quite well), which also surprised me. STE is short for Straight-Through Estimator, which is used to estimate the gradients of the sign operation during back-prop.

I tried out computing the similarity of query and docs in binary space, and train the model to push the similarity in binary space close to the similarity in the original continuous space, just like the idea of model distillation. In this way, I do get a model that could preserve the similarity quite well, but just as you mentioned, it's not discriminative enough. So maybe I would need to figure out a way to better discriminate the codes, instead of just preserving the similarity.

[kantholtz]

STE is short for Straight-Through Estimator, which is used to estimate the gradients of the sign operation during back-prop.

Makes sense, you would need to carry the gradient through the thresholding.

So maybe I would need to figure out a way to better discriminate the codes, instead of just preserving the similarity.

Usually, in that kind of setting, you would learn to rank (l2r) - a ranking loss objective such as a triplet margin loss would seem like the thing to do. Maybe you could train the autoencoder first with the reconstruction loss, and than fine-tune it with a l2r objective. It would be very interesting to observe how that would affect the codes.

Good luck!