aih / billsim Goto Github PK

• Add db table or fields (optional) in the model to hold bill text
• Add directory with SQL-based similarity calculation algorithms and README

For elasticsearch, see: https://github.com/elastic/elastic-github-actions/tree/master/elasticsearch

Also add Postgres (docker?) and save data to the Postgres db:
See https://github.com/microsoft/vscode-dev-containers/tree/main/containers/python-3-postgres

I tried running billsim with python3.8 and python3.9, and got two separate issues.

With 3.8

___________ ERROR collecting tests/constants_test.py ___________
tests/constants_test.py:3: in <module>
    from billsim.pymodels import BillPath
src/billsim/pymodels.py:36: in <module>
    class Section(SectionMeta):
src/billsim/pymodels.py:37: in Section
    similar_sections: list[SimilarSection]
E   TypeError: 'type' object is not subscriptable
_____________ ERROR collecting tests/utils_test.py _____________
tests/utils_test.py:9: in <module>
    from billsim.pymodels import BillPath
src/billsim/pymodels.py:36: in <module>
    class Section(SectionMeta):
src/billsim/pymodels.py:37: in Section
    similar_sections: list[SimilarSection]
E   TypeError: 'type' object is not subscriptable

With 3.9

from lxml import etree
ImportError: dlopen(/opt/homebrew/lib/python3.9/site-packages/lxml/etree.cpython-39-darwin.so, 2): no suitable image found.  Did find:
        /opt/homebrew/lib/python3.9/site-packages/lxml/etree.cpython-39-darwin.so: mach-o, but wrong architecture
        /opt/homebrew/lib/python3.9/site-packages/lxml/etree.cpython-39-darwin.so: mach-o, but wrong architecture

This latter one seems to be a issue with m1 macs from considering the error complains about architecture.

When finding a bill similarity from A to B, need to store that similarity in B to A so that looking up B brings up Bill A.

Add a table with run instance info. Each row include id, version and date.

Add for each bill2bill a run instance id.

See https://github.com/aih/billsim/blob/main/src/billsim/utils_db.py#L382

We currently use sqlalchemy for this batch save operation. However, in some cases, it causes errors:

Traceback (most recent call last):
  File "/home/ubuntu/.pyenv/versions/3.9.1/envs/py391/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1799, in _execute_context
    self.dialect.do_execute(
  File "/home/ubuntu/.pyenv/versions/3.9.1/envs/py391/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.StatementTooComplex: stack depth limit exceeded
HINT:  Increase the configuration parameter "max_stack_depth" (currently 2048kB), after ensuring the platform's stack depth limit is adequate.

We may try to increase max_stack_depth, but the statements also appear to be unnecessarily large and recursive.

Can we take advantage of psycopg3 improvements and use it directly for saves?

This issue is to create a github action to run tests on PR

Consider, e.g. sqlalchemy's create_async_engine() for async postgres calls

This function builds on the functions in this repository and the Go functions in aih/bills.

Assumptions:

• Each 'document' consists of an array of strings. The document has a unique id and each item in the array is also uniquely identified (either by an id or its ordinal position in the array).
• The length of each document array may vary

The generic similarity functions would:

1. Calculate a vocabulary of n-grams from the total corpus of documents (an array of documents).
2. Vectorize the documents so that they each document can be stored as a (sparse) array of the length of the vocabulary
3. Store the vectorized matrix of all documents in a pickle file (or eventually in Postgresql) (MOD- matrix of all documents)
4. Calculate the similarity between each item of each array and all other items in the MOD
5. Apply an item threshold to find similar items for each item in a document
6. Apply a document threshold to find similar documents
7. Return 5 and 6 in a model form that can be stored to a database (item-to-item and document-to-document similarity)

For a Library of Congress project, we need to connect to Postgres on port 5433, not port 5432.

The billsim code doesn't seem to use the values defined in the .env file at the top-level. I thought it did use those .env values, but potentially I hardcoded port 5433 into the billsim package and forgotten it, then reinstalled billsim and wiped that change away.

How can we best pass the postgres port and host into the billsim package? Potentially we should pass it at runtime rather than as an env variable? Or is there some way to have the billsim package use the .env file at the top level of the repo? This would appear to be an issue for any user of the billsim package, as generally you'd want to pass in username/password/etc and not use defaults.