hp0404 / speeches

Stackoverflow. Daily top questions

Automatically updates via GitHub Action, last update: 2024-05-06

rust What is the difference between ampmut transaction and ampmut transaction in SQLx - 4 votes Lifetime of temporary objects in if let statements - 2 votes How to handle quotreference to packed field is unalignedquot error - 1 votes Why does Tauri modify the parameter names of invoked functions - 1 votes Docopt only accepting extended format for option in Rust - 1 votes

fastapi FastAPI with SAM model Only a few workers load properly due to CUDA memory error - 3 votes Model Relationship for different User Roles What am I missing - 2 votes Does it make sense to specify the user id in the path if I already get it through the JWT token - 1 votes

pandas sort groups by size and drop smaller groups - 2 votes Why does filtering based on a condition results in an empty DataFrame in pandas - 2 votes Confusing conversion of types in pandas DataFrame - 2 votes Pandas read tab separated csv - 2 votes How to reverse axes when plotting a Series - 1 votes

Inspired by simonw

speeches's Issues

GET - sort logic

Current implementation sorts table by 'created_at' column:

speeches/app/main.py

Lines 32 to 37 in 0d97e4e

	statement = (
	select(Speeches)
	.order_by(Speeches.created_at.desc())
	.offset(offset)
	.limit(limit)
	)

I think we should first sort by 'date' DESC, then 'create_at' ASC

Refine Matcher's patterns

I've noticed a few issues so far:

• incorrect matching, e.g. special characters (dates, dashes, etc.) are treated as PoS and combined with proper PoS give 'false' phrases
• duplication, e.g. the same chunk of text might get matched with multiple patterns giving > 1 match - which might be fine as long as we have a clear approach to 'counting' unique phrases

This issue will be updated

separate tables?

I noticed that filtering full dataset takes a lot of time. This led me to think I might try moving 'texts' into a separate table, thus allowing fast filtering on metadata and then only joining relevant texts from another table.

Not sure if it helps but worth trying

validate config on assignment

Base settings class validates DATABASE_URI only once:

speeches/app/core/config.py

Lines 26 to 42 in c599bd7

	POSTGRES_SERVER: str
	POSTGRES_USER: str
	POSTGRES_PASSWORD: str
	POSTGRES_DB: str
	DATABASE_URI: Optional[Union[str, PostgresDsn]] = None
	@validator("DATABASE_URI", pre=True)
	def assemble_db_connection(cls, v: Optional[str], values: Dict[str, Any]) -> Any:
	if isinstance(v, str):
	return v
	return PostgresDsn.build(
	scheme="postgresql",
	user=values.get("POSTGRES_USER"),
	password=values.get("POSTGRES_PASSWORD"),
	host=values.get("POSTGRES_SERVER"),
	path=f"/{values.get('POSTGRES_DB') or ''}",
	)

If we change some of the fields after the settings class has been initialized, the validators won't run again. In regular usage it's fine as we don't expect settings to change unless we're directly testing different setups.

I'd like to have this fixed so that we could inject some overriding settings as a fixture shared across all tests and then maybe change fields again within the scope of some specific test functions.

We could set different POSTGRES_DB value for pytest or tweak SMTP_USER / SMTP_PASSWORD while testing email notifications

badly formed hexadecimal UUID string

Basically encountered this issue (tiangolo/sqlmodel#25) and went for int primary key instead.

TODO:

• go back to UUID as PK

tests

move enpoints from main.py to separate routers

This issues should be treated as a road-map:

SPEECHES:

• /speeches/
  • POST: insert new document + extract features on the fly
  • GET: query metadata from a database
• /speeches/{id}
  • GET: get document (probably metadata + joined text + features (optionally)
  • DELETE: delete {id} speech from all tables

FEATURES:

• /features/
  • POST: send text and receive extracted features, without writing to a database
• /features/{document_id}
  • GET: query all features associated with document_id (optionally filtering using feature_type)
  • DELETE: delete {document_id} features

auth

add auth for POST requests - to be sure that only a single user (cronjob) has write rights.

TODOs:

• try the simplest approach (without fastapi.security - as query)
• set up proper OAuth2 using fastapi docs (as bearer token)

alembic - turn off default indexing of all columns

When writing 'larger' texts to a PostgreSQL database, SQLModel might throw sqlalchemy.exc.OperationalError: (psycopg2.errors.ProgramLimitExceeded) - Values larger than 1/3 of a buffer page cannot be indexed.

This issue has been documented (tiangolo/sqlmodel#9)

change structure

I'm trying to think ahead (what's the easiest structure to maintain & build upon?)

• (maybe) move routes-specific code from main.py to separate routers/ module
• (maybe) move auth.py to core/ (purely because I've seen other people do it, but have no idea if it's the 'best practice')
• (maybe) merge database.py and core/config as database.py simply imports config's PostgreSQL connection string and creates sqlmodel engine

store embeddings

rename attributes