osrs-tracker / osrs-tracker-aws

This repository contains the AWS code and configuration for OSRS Tracker.

OSRS Tracker AWS · ·

This repository contains the AWS code and configuration for OSRS Tracker, an application designed to track item prices and player information in the game Old School RuneScape.

Lambda functions

Schedule Driven

• osrs-tracker_refresh-items

  This function fetches all items and latest prices from prices.runescape.wiki/api/v1/osrs, merges them into an Item object and upserts all items to MongoDB every hour.

• osrs-tracker_queue-players

  This function fetches all players that have opted in for tracking at a particular UTC offset, and pushes them to a SQS queue where they will wait to be processed.

  Calculates current offset from the ScheduledEvent and queries MongoDB for players that have opted in for tracking at that specific offset.

• osrs-tracker_process-players

  This function fetches OSRS hiscores for the provided usernames, and prepends the hiscore to the player.hiscoreEntries array.

• osrs-tracker_hiscore-parser-validator

  This function fetches the hiscores every hour for a few test usernames and will run the parseHiscoreString function from the @osrs-tracker/hiscores package to test if the current parser is still valid, and no api changes have happened.

• osrs-tracker_clean-hiscores

  This function deletes all hiscores older then 60 days from the data base every day at UTC midnight.

API Gateway driven

• osrs-tracker_item-info

  When sending a GET /:id request, this function returns item details from MongoDB.

• osrs-tracker_item-search

  When sending a GET /:query request, this function returns items from MongoDB where the name partially matches the provided query. It also adds a confidence score property.

• osrs-tracker_player-hiscores

  When sending a GET /:username(?size=7&skip=0&scrapingOffset=0) request, this function returns scraped player hiscoreEntries for the provided username from MongoDB.

• osrs-tracker_player-info

  When sending a GET /:username(?scrapingOffset=0&hiscore=false) request, this function returns player information for the provided username from MongoDB, or attempts to scrape it if not found in MongoDB.

  The scrapingOffset will be added to the player.scrapingOffsets if it's not present yet, and will be used filter hiscoreEntries when hiscore=true. An initial hiscoreEntry will also be pushed if it's the first time the player was fetched with this specific scrapingOffset.

  By default, the returned player does not include hiscoreEntries. However, if the query parameter hiscore=true is included in the request, the function will also include the player's most recent hiscore entry matching the scrapingOffset.

  The function automatically refreshes the player info when information from MongoDB is older than 2 hours.

osrs-tracker-aws's Issues

Open mongodb connection at start of function

currently we get long duration because the connection is only opened after everything is scraped. We could make it a lot faster if we 'preheat' the mongodb connection.

No datapoints for 31 dec and 1 jan

Apparantly 31 dec and 1 jan weren't tracked because when we crossed the 100 tracked players treshold it caused the osrs-tracker_queue-players lambda function to throw an error.

Implement hiscore parser validation check

osrs-tracker/osrs-tracker-web#13

Add a way to fetch the latest hiscore entry for multiple players

This will be necessary for the XP Tracker player widgets. Where we can see how much overall XP our recent and favorite players have earned today.

We can create new lambda function, or update osrs-tracker_player-hiscores to also accept GET /?usernames= besides GET /:username.

TBD

Don't save unlimited amounts of scraped hiscores

Cap player hiscores to the 30 most recent per scraping offset.