add tests for notebooks about langchain

[ccurme]

I looked into a couple ways of doing this. Let me know if you had a preferred method in mind.

One is to add a dependency nbmake, which is a pytest plugin that lets you execute notebooks. You can see an implementation here: master...ccurme:langchain:cc/test_notebooks

The pro is that this is a simple solution built for this use-case. You can list all the notebooks you want to test, or use --ignore to exclude certain notebooks.

One con is that the error handling looked cumbersome to me. If you expect a specific error to be raised, for example, you need to add metadata to cells in the notebook JSON, which can get overwritten. I added an example on that branch.

Another option is to use nbclient, which is already included in dependencies. I added an example here: master...ccurme:langchain:cc/test_notebooks_nbclient

This is neat because you can write the tests in python and work with errors programmatically. But it's arguably more complicated than just passing --nbmake to pytest.

Let me know if you have thoughts or if I should open a PR.

[hwchase17]

hmmm adding the metadata is definitely cumbersome. but at the same time... how many notebooks do we have that expect errors?

[ccurme]

It doesn't look like many. My thought is to just to use nbmake and opt-in to including notebooks in the test. So we'd need to maintain a separate list of notebooks to test; we wouldn't discover them. Does this sound good, or is it important to discover notebooks so that we don't need to maintain a list?

[hwchase17]

that sounds perfect!

[hwchase17]

its probably possible to regexs, eg /* for everything in directory?

[ccurme]

We can enumerate directories, we can also go the other way and maintain a list of notebooks we don't test.

[dosubot]

Hi, @hwchase17! I'm Dosu, and I'm helping the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, the issue is about adding tests for notebooks to ensure they are kept up to date. There was a discussion with ccurme where two options were suggested: using nbmake, a pytest plugin, or using nbclient, which is already included in dependencies. The conclusion was to use nbmake and maintain a separate list of notebooks to test. There was also a suggestion to use regex to include all notebooks in a directory.

Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself or it will be automatically closed in 7 days.

Thank you for your contribution to LangChain!