Add support for more embedding models about langchain

[issam9]

Great. I'll work on it

[hwchase17]

if you have any questions about the right place to put it or the conventions, happy to take a look at any even half-implemented stuff! just lmk

[issam9]

Would like to work on adding support for HF models. I think it is more convenient to use sentence_transformers as they can handle both sentence transformers and models coming from HF by adding a mean pooling layer. What do you think?

[hwchase17]

i think thats a great idea!

[issam9]

I made an initial implementation where i tried to stay close to how the openai one is implemented. Please check in the link below. I don't like the part where i instiatiate SentenceTransformer class inside embed functions and would love to have some suggestions there. Thanks

https://github.com/issam9/langchain/blob/issam/hf_embeddings/langchain/embeddings/huggingface.py

[hwchase17]

yeah the init is a little tricky. i dont love the way we do the initialization for other things, may try to rework that in the future. In the meantime, what about adding a classmethod and making it like:

@classmethod
def from_model_name(cls, model_name):
      import sentence tranformers
      model = sentence_transformers.SentenceTransformer(model_name)
      return cls(model)

thoughts?

[hwchase17]

am thinking of changing to use pydantic dataclasses to take advantage of the postinit stuff https://stackoverflow.com/questions/66571079/alter-field-after-instantiation-in-pydantic-basemodel-class

[issam9]

I tried doing that by overriding init which makes it possible to keep using BaseModel:

def __init__(self, **kw):
    super().__init__(**kw)
    try:
          import sentence_transformers

          self.client = sentence_transformers.SentenceTransformer(self.model_name)
    except ImportError:
          raise ValueError(
              "Could not import sentence_transformers python package. "
              "Please it install it with `pip install sentence_transformers`."
          )

[hwchase17]

yeah i dont hate that implementation either. id be down to do something like that as well. need to figure out longer term plan and unify across codebsase, but in short term im down to do it that way and see how well it works

[issam9]

Hi, I created a pull request with init override implementation. please lmk if i need to apply any changes