overhash / supermarket-tracker Goto Github PK

countdown api does not have ratelimits

also u should multithread it fr fr

Currently there is about a ~2000 non-unique products when retrieving all_products.

This is because categories may contain overlapping products. During get_all_products we should ensure to remove these duplicate products.

For a sample, sku 243682 with the following product information:

{
  "name": "angel bay veggie burger patties ",
  "barcode": "9419411000003",
  "sku": "243682",
  "per_unit_price": 1300
}

has duplicates under page 6 of /shop/browse/frozen (Frozen category) and also under page 14 of /shop/browse/christmas (Christmas category).

While this behavior doesn't have catastrophic consequences, it inflates the product count and means extra work later in the program.

Some basic suggestions like when an invalid parse option is found for a user passing the --supermarket option (e.g., --supermarket countdown when the user meant to type --supermarket Countdown) could be provided.

As described in #2 (comment), we need to investigate a cleaner way of handing out tasks.

Consider using https://docs.rs/tokio/latest/tokio/sync/struct.Semaphore.html for this.