kshitij10496 / hercules Goto Github PK

Find the best solution to creating the URL required for scrapping Timetable data from ERP.

Refer: /data/scrapper/course_rooms.py

Add faculty website URL in https://hercules-10496.herokuapp.com/api/v1/faculty/info/all. wimp needs it too :D

As of now, we have to pass the cookie as an argument while running the extract data scripts.

We can use a web scraper /web driver ( selenium or robobrowser ) and login into erp while running the script. This will prompt the user to input the roll number and the password and the security question. Note that this won't be storing any password or other details that were input This will save us the trouble of having to log in and then copy paste the cookies.

This will make it easier for running the scripts to update every sem.

Writing HTTP Handlers before any unit tests might seem outrageously wrong.
I would agree with you on writing proper unit tests first, doing everything TDD, hands down.
This might not be ideal but it's the bare minimum which we should have before doing a public release.

If you are interested in writing unit tests/integration tests(with a mock DB), I would be happy to accommodate it and learn from you. 😃

The MVP of hercules used DATABASE_URL environment variable for the DB connection string.
Thanks to #6, we are now using envconfig for the management of configuration values.
Currently, the developer is required to set HERCULES_DATABASEURL for the same DB connection.

It would be aesthetically pleasing if someone can rename the variable to HERCULES_DATABASE instead.

@themousepotato Can you experiment with using hercules API to power wimp?
Given that the endpoints currently have bare minimum documentation, I would be happy to assist you. 😄

cc @icyflame @amrav

Add a cool and funny doodle! 🎨
It would be nice if the art could be associated/symbolise strength and power in an abstract way. 💪

This is an open doodle/drawing challenge rather than a technical one.
So, draw something up on a piece of paper and share it in the comments below.
Thus, you don't necessarily have to fork the repo and open a PR in order to participate.

Prize: If the winner is studying at KGP, they get a GitHub Octocat sticker and a dinner treat!

Voting: Use 👍 if you like a doodle. (Please do not use 👎 as this be discouraging.)

Deadline: Thursday, 25th October

We need to find a way to generate the API documentation.

Options:

• plicca/summerfish-swagger

Suggestions welcome!

See blackjack or Kronos.
https://github.com/metakgp/blackjack
No urgency on this one though.

Currently, the server.go program expects PORT and DATABASE_URL to be set in order to run the application. Use Kelsey Hightower's envconfig library for reading app configuration variables into the program rather than using os.Getenv.

Refer: server.go

@icyflame @amrav @DefCon-007 @athityakumar @thealphadollar

What is the best practice for providing read-access to the DB with anyone interesting in contributing to the project?
aka How do I share a replica of the production DB for local development?

From the top of my mind:

• Creating a credential user for every individual with read access.
  • Cons:
    • Proxy connection under Institute's LAN is blocked.
    • Tedious process on the part of the project maintainer.
• Storing a database dump in the repository.
  • Cons:
    • Regularly update the dump.
    • Bloats the size of the repo.

It would be wonderful if we can get the application to run via a single command.
This would involve the use of docker-compose for managing multiple containers.
Ideally, we should have 2 services - one for the Postgres DB and another for the API.

• Add development guidelines to setup a local environment in a host machine and a docker container.
• Add examples.

The idea is to enhance the security by restricting the clients to interact with the API only via GET requests. We need to add appropriate handler code and Access-Control-Allow-Methods response header.

https://hercules-10496.herokuapp.com/api/v1/department/info/all
Tried, various links, none works.

We now have a landing page listing all the API endpoints currently implemented in v1.
All thanks to #19 by @rasfa98 ! 😃

Next step would be to integrate it in the API and render the static HTML page on /api/v1.

References:
[1]: https://stackoverflow.com/questions/26559557/how-do-you-serve-a-static-html-file-using-a-go-web-server

Currently, I'm encoding the data scrapped from the timetable directly into a JSON. This is an ad-hoc way just to get things done. Ideally, we would want to store the data into a beautiful format, similar to what jq does.

Add indentation and formatting while encoding the scrapped data.
This would involve modifying the CourseEncoder class.

Refer: /data/scrapper/course_rooms.py

Add an API endpoint that contains current semester dates and previous historic dates.

Task: Implement input validation checks/methods on the parameters sent by the user.

Currently, all the endpoint handlers expectedly delegate the task of validation to the corresponding logic layer. Now, its the responsibility of the logic layer to perform input validation and return proper errors back to the handler.

In order to start with the fix, we need to discuss what constitutes a valid entity.
For example, what needs to be checked in order to assure that a department/course is valid?

P.S: This would form the basis of developing a better error handling mechanism. 😄

Data related to first year academics cannot be scraped from ERP. Possibly find another way to add data. May be hard-code :) cc @icyflame

See https://raw.githubusercontent.com/metakgp/chillzone/master/slots.1

In case this has been done, then update the README accordingly

At the moment, the JSON response from the API is raw and hinders human readability.
It would be ideal if we could prettify the responses.

Possible Fix: The task would probably require edits in the JSON Encoder present in RespondWithJSON.

References
[1]: https://stackoverflow.com/questions/19038598/how-can-i-pretty-print-json-using-go
[2]: https://godoc.org/encoding/json

Check this instance:

        {
            "course": {
                "name": "GRAPH THEORY AND ALGORITHMS",
                "code": "MA60041",
                "credits": 4
            },
            "slot": {
                "time": {
                    "day": "Monday",
                    "time": "12 PM"
                },
                "slot": "D4"
            },
            "rooms": [
                "NR223"
            ]
        },
        {
            "course": {
                "name": "GRAPH THEORY AND ALGORITHMS",
                "code": "MA61003",
                "credits": 4
            },
            "slot": {
                "time": {
                    "day": "Monday",
                    "time": "12 PM"
                },
                "slot": "D4"
            },
            "rooms": [
                "NR223"
            ]
        }
    ],

Here, MA60041 and MA61003 appears to be different but, under the same name GRAPH THEORY AND ALGORITHMS, same time 12 PM. Write validation conditions for such cases. @kshitij10496 WDYT?

This was discussed during Demo Day 10 on 9th November, 2018

Some of the data on the ERP has historical value. Some data (eg: grades) is very obviously useful historically, and should be a part of the present semester's data.

There is some other auxilliary information that would be useful for applications that might be developed later. Eg: Timetables, historical slot information.

Three components are required for the storage and retrieval of historical data to be frictionless:

1. A method to retrieve the data through an API endpoint that uses a unique semester identifier

There is no semester specific code. Giving the identifier (say) /2018FALL/timetable/MA20001 will get you the data from that semester. Whereas /timetable/MA20001 would get you the data from the ongoing semester.

2. A way to make present data historical and create empty present data

These (scripts?) make the movement of data between present and history easier. Once this is implemented, when 2018FALL is over, these two scripts will send the data in the "PRESENT" to "2018FALL" and create a new set of "PRESENT" data that is empty and would refer to the next semester "2019SPRING".

Because of (1), as soon as the data is moved to "2018FALL" specific data containers, the endpoints for historical data of that semester will magically start working! (Not magical as we have implemented it, but it would be magical because of how little you had to do to get it started)

3. A way to swap out the data container that stores historical data

As @kshitij10496 put aptly in the demo day call, Heroku's free plans have restrictions. So, we need to be flexible about where we store the historical data. A crude solution is to store them in a JSON file. Another solution is to put a MongoDB or ElasticSearch instance or Postgres server on the metakgp main server.

Whatever be the case, the data layer should be easily swappable to be anything.

When the above three pieces are in place, we can start reliably providing historical data. We will also be able to provide this data quickly when a semester ends (because of (2)). And we will be able to adapt to changing conditions, cheaper services, new technology (because of (3))

This can be sent as just another key in the endpoint for course -timetable ( see https://hercules-10496.herokuapp.com/api/v1/course/timetable/MA61023) or a separate endpoint. We can use this thread to discuss regarding the same.

As for how to extract syllabus, one may refer link :
( https://erp.iitkgp.ac.in/Acad/subject/get_syllabus_pdf.jsp?subno= <course_code_here> &year=20181 )

At the moment, all the responses from the API v1 have their Content-Type set as text/plain by default.

Task: Explicitly set the Content-Type header in successful HTTP responses to application/json.

update deps according to https://github.com/metakgp/wimp/blob/master/data/deps.4

Currently, landing on the homepage results in a 404.
It would be cool if we could list all the endpoints currently supported by the API populated from this section for the time being.

I would be happy with a skeleton page, mostly using HTML and CSS.

As of now the list of courses is not exhaustive. From what it shows, it only has the data on the courses offered in the current semester.

There are room numbers with value '0'. Replace those with 'In Dept'. Also, there are 'In Deptt'. Replace those with 'In Dept' ;)

Discussion related to the CI/CD development workflow

• Connect Heroku to the repo for automatic app deployments
• Choose CI tool: CircleCI, TravisCI, Semaphore (suggestions welcomed)
• Choose CQ/CR tool(s): Hound, Codacy, Coveralls, Codecov (suggestions welcomed)

Are there any other tools we should install (for monitoring/crash reporting etc)?

cc @icyflame @ghostwriternr