ameynaik-9 / fetcherjs

DU Result Fetcher made with Node for concurrency.

FetcherJS

Node.js Version of DU Result fetcher. The aim is to improve the speed in comparison to python Version by using Concurrency.

Proof of Concept

Checkout the Proof of Concept.

comparison with the python version. It reduced the time to half for 50 students. For 800 students, the improvements should be even more.

Thoughts

• Concurrency is the MAJOR goal here
  • Promise.all() is the way to go.
• a way to fetch the URL with options, (fetch, request-promise) - Checkout wretch
• find a html parser, similar to beautifulsoup (jsdom, x-ray, ?)
• a in-memory database for when the data gets too large, (lokijs)