edologgerbird / sfyr-data-pipeline Goto Github PK

Differentiate between Stocks, ETFs, DLCs, SWs, REITs, Business Trust, Company Warrants and ADRs.

Requirement: Ingest Data from the SGX Scrape and output the relevant information

Currently the scraped messages do not allow us to determine which stocks are mentioned in the posts.

To allow them to be stored and queried efficiently, we would need to automatically extract the mentioned stocks and store them in a format that is suitable for SQL query

As a user I should be able to call the function to scrape the latests posts
And it should automatically identify the stocks mentioned from those posts
Then the information should be uploaded into BigQuery
And be easily extracted out from BigQuery based on the stocks

Currently the scraped messages do not allow us to determine which stocks are mentioned in the telegram messages.

To allow them to be stored and queried efficiently, we would need to automatically extract the mentioned stocks and store them in a format that is suitable for SQL query

As a user I should be able to call the function to scrape the latest messages from the telegram groups
And it should automatically identify the mentioned stocks
Then the information should be uploaded into BigQuery
And be easily extracted out from BigQuery based on the mentioned stocks

Currently SGX Data Scrape output is stored in a csv file. However, the requirements of the assignment is to store all information in a data warehouse - in this case BigQuery.

The output should be automatically uploaded to BigQuery at the end of the extraction

• consumers

Currently the scraped messages do not allow us to determine which stocks are mentioned in the posts.

To allow them to be stored and queried efficiently, we would need to automatically extract the STI Movement and store them in a format that is suitable for SQL query

As a user I should be able to call the function to scrape the latests posts
And it should automatically identify the STI Movements from those posts
Then the information should be uploaded into BigQuery
And be easily extracted out from BigQuery

GBQ Ingest API should check to ensure data input is in the form of a pandas dataframe before calling the ingest API

• Get the names and tickers of SGX-listed companies.

Exploring if it is possible to ensure SBR scraper does not scrape information that occur before last scraped date

• Diagram
• Justifications
• Theorems
• Kappa/lambda?