Coder Social home page Coder Social logo

skyward-rest's Introduction

Skyward REST

Build Status

Summary

Unofficial Rest API for Skyward

  • Queries data for the fastest output
  • Breaks down and parses complex responses
  • Handles edge cases with ease
  • Built functionally

Examples

Boilerplate

const skyward = require('skyward-rest')

const url = 'https://skyward.cooldistrict.net/...'

const scraper = skyward(url) // the scraper!

Scrape a user's course gradebook

scraper.scrapeGradebook(user, pass, options)
  .then(console.log) // => Large Object

Scrape a user's academic history

scraper.scrapeHistory(user, pass)
  .then(console.log) // => Array of Sizeable Objects

API

skyward( loginURL )

Function which returns an object containing the API.

  • loginURL string - URL to the login page of the specific district's Skyward domain. Note that the URL should not redirect.
const skyward = require('skyward-rest')

skyward('https://skyward.cooldistrict.net/scripts/wsisa.dll/WService=wsEAplus/seplog01.w')
// => { usable functions }

.scrapeReport( user, pass )

Fetches and parses a student's report card, returning a promise which results in an object that's data property is an array of Reports. Note that this differs from .scrapeGradebook in that individual assignments in a course are not scraped, only the bucket's score.

  • user string - the username or Login ID of the student who's grades will be retrieved
  • pass string - the password of the student
scraper.scrapeReport(user, pass)
  .then(({ data, raw }) => {
    console.log(data) // array of reports
    console.log(raw) // fetched html before parsing
  })

Report

An object that contains scores from a specific course over each bucket.

{
  course: 97776, // the five-digit course ID
  scores: [
    {
      bucket: 'TERM 1',
      score: 100
    },
    {
      bucket: 'TERM 2',
      score: 98
    },
    /* etc */
  ]
}

.scrapeGradebook( user, pass, options )

Fetches and parses user's a gradebook, returning a promise which results in an object that's data property is a Gradebook.

  • user string - the username or Login ID of the student who's gradebook will be retrieved
  • pass string - the password of the student
  • options object - information identifying which gradebook to scrape
    • course number - the five-digit course ID to scrape (e.g. 97776, 97674, etc. )
    • bucket string - the term to scrape (e.g. 'TERM 1', 'SEM 1', etc.)
scraper.scrapeGradebook(user, pass, { course: 97776, bucket: 'TERM 3' })
  .then(({ data, raw }) => {
    console.log(data) // gradebook
    console.log(raw) // fetched xml before parsing
  })

Gradebook

An object that contains information and assignments about a course at a specific bucket.

{
  course: 'PHYSICS 2 AP', // name of the course
  instructor: 'Jennifer Smith', // name of the instructor
  lit: { // information about the specific bucket
    name: 'S1', // bucket's alias
    begin: '08/20/2018', // bucket's begin date
    end: '12/20/2018' // bucket's end date
  },
  period: 1, // course's order in the day
  score: 99.5, // score recieved (usually contains a decimal)
  grade: 100, // score after rounding (always a whole number)
  gradeAdjustment: 1.5, // points added to average to get score (null if no adjustment)
  breakdown: [ // buckets which make up this bucket's score (null if no breakdown)
    {
      lit: 'Q2', // bucket's alias
      score: 95.5, // score recieved
      grade: 96, // score after rounding
      weight: 50, // part that this bucket's score makes up the parent bucket's score (out of 100)
    },
    {
      lit: 'Q1',
      grade: 100,
      score: 100,
      weight: 50,
    },
  ],
  gradebook: [ // grade categories which make up this bucket's score
    {
      category: 'Major', // category title
      breakdown: [ // buckets which make up this category (undefined if no breakdown)
        {
          lit: 'Q2', // bucket's alias
          weight: 70, // part that this bucket's score makes up this category's score (out of 100)
          dates: {
            begin: '10/22/2018', // bucket's begin date
            end: '12/20/2018', // bucket's end date
          },
          score: 96.5, // score recieved
          grade: 97, // score after rounding
          points: {
            earned: 965, // sum of all assignments' earned points
            total: 1000, // sum of all assignments' total points
          },
        },
        /* etc. */
      ],
      assignments: [ // assignments which make up this category
        {
          title: 'TEST IV',
          score: 100, // score recieved (null if no score)
          grade: 100, // score after rounding (null if no grade)
          points: {
            earned: 100, // earned points (null if no earned)
            total: 100, // total points (null if no total)
          },
          date: '09/07/18', // date the assignment is/was due
          meta: [ // assignment modifiers
            {
              type: 'absent', // modifier type (e.g. 'absent', 'noCount', or 'missing')
              note: 'Parent note received within 5d', // extra message
            }
          ],
        },
        /* etc. */
      ]
    },
    /* etc. */
  ]
}

.scrapeHistory( user, pass )

Fetches and parses user's a academic history, returning a promise which results in an object that's data property is an array of SchoolYears.

  • user string - the username or Login ID of the student who's academic history will be retrieved
  • pass string - the password of the student
scraper.scrapeHistory(user, pass)
  .then(({ data, raw }) => {
    console.log(data) // array of schoolYears
    console.log(raw) // fetched xml before parsing
  })

SchoolYear

An object that contins information, courses, and scores from a completed school year

{
  dates: {
    begin: '2018', // school year begin date
    end: '2019', // school year end date
  },
  grade: 12, // grade of student during the school year
  courses: [ // courses taken during the year
    {
      course: 'PHYSICS 2 AP', // course name
      scores: [
        {
          grade: 100, // grade recieved
          lit: 'S1', // bucket alias
        },
        /* etc. */
      ]
    },
    /* etc. */
  ]
}

skyward-rest's People

Contributors

kaelinator avatar shrey150 avatar unknownsilence avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

skyward-rest's Issues

undefined url in referrer ofscrape.js

C:\Users\Trent Hardy\Desktop\skywardtest\node_modules\skyward-rest\lib\scrape.js:53
const referer = url.split('/')
^

TypeError: Cannot read property 'split' of undefined
at Scraper (C:\Users\Trent Hardy\Desktop\skywardtest\node_modules\skyward-rest\lib\scrape.js:53:25)
at Object. (C:\Users\Trent Hardy\Desktop\skywardtest\node_modules\skyward-rest\index.js:14:27)
at Module._compile (internal/modules/cjs/loader.js:689:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
at Module.load (internal/modules/cjs/loader.js:599:32)
at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
at Function.Module._load (internal/modules/cjs/loader.js:530:3)
at Module.require (internal/modules/cjs/loader.js:637:17)
at require (internal/modules/cjs/helpers.js:22:18)
at Object. (C:\Users\Trent Hardy\Desktop\skywardtest\main.js:1:79)
at Module._compile (internal/modules/cjs/loader.js:689:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
at Module.load (internal/modules/cjs/loader.js:599:32)
at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
at Function.Module._load (internal/modules/cjs/loader.js:530:3)
at Function.Module.runMain (internal/modules/cjs/loader.js:742:12)

const skyward = require('skyward-rest')

const url = "https://skyward.kleinisd.net/scripts/wsisa.dll/WService=wsEAplus/seplog01.w"
const userId = '*********'
const password = '**********'

skyward(url)(userId, password)
    .then(student => {
        student.scrape('S1')
            .then(console.log) // => [Object Array]
            .then(() => student.close())
    })

User and password are correct, but censored for obvious reasons.

it appears to me that the scrape.js file of your module isn't taking in the url parameter from the main.js file that is calling the node module.

const Scraper = (url) =>
  (sId, pass) => {

    let loginInfo, secretCode, gradebook, browser, args

    const referer = url.split('/')
      .slice(0, -1)
      .concat('httploader.p?file=sfgradebook001.w')
      .join('/')

There also appears to be an unused import of Promise from bluebird in your index.js here:

const Promise = require('bluebird')

scrapeReportCard => scrapeReport.

The documentation for the API contains a naming error in the .scrapeReportcard function.

As shown:

.scrapeReportcard( user, pass )
Fetches and parses a student's report card, returning a promise which results in an object that's data property is an array of Reports. Note that this differs from .scrapeGradebook in that individual assignments in a course are not scraped, only the bucket's score.

user string - the username or Login ID of the student who's grades will be retrieved
pass string - the password of the student

scraper.scrapeReportcard(user, pass)
  .then(({ data, raw }) => {
    console.log(data) // array of reports
    console.log(raw) // fetched html before parsing
  })

=======================

.scrapeReportcard is a invalid function.

TypeError: scraper.scrapeReportcard is not a function
    at Object.<anonymous> (C:\Users\Trent Hardy\Desktop\skywardTest\main.js:10:9)
    at Module._compile (internal/modules/cjs/loader.js:689:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
    at Module.load (internal/modules/cjs/loader.js:599:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
    at Function.Module._load (internal/modules/cjs/loader.js:530:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:742:12)
    at startup (internal/bootstrap/node.js:283:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:743:3)

=======================

The documention should read .scrapeReport, which is the actual function defined in the API under the index.js:

As shown:

module.exports = skywardURL => ({
  scrapeReport: (user, pass) => (
    authenticate(skywardURL)(user, pass)
      .then(auth => reportcard.fetch(skywardURL)(auth))
      .then(response => ({
        raw: response.data,
        data: reportcard.getData(response),
      }))
  ),

Score decimal in earned

Example: 38.5 out of 50 currently returns { earned: 38, total: 5}, expected: { earned: 38.5, total: 50}

The first line is not working.

When I try the code
const skyward=require('skyward-rest')
, it says this:
IMG_5763
Do I need to do some git install or something to get this working and installed in the node modules?
Sorry, I am new to React-Native and I am still trying to figure things out.

* -> null

for earned points, * out of 0 returns { earned: '*', total: '0' }, expected: { earned: null, total: 0 }

Individual assignment `lit`s for semester scrapes

Current semester scrapes specify the points received out of the total for specific lits:

"score": [
  {"earned": 1114, "lit": "Q2", "total": 1200}, 
  {"earned": 951, "lit": "Q1", "total": 1000}
]

Using these totals, the assignment totals, and the order in which they were scored, it is possible to assign a lit value to each individual assignment.

Add explanation

Could you create a better README or create a wiki to explain how to use this?

Handle session expiration and prevent timeouts

After idling for about five minutes, an account will be prompted to stay logged in.
Pressing OK sends a GET request:
request
In case a specific user must be logged in for an extended amount of time, a request must be sent periodically.

In the case that the user logs in whilst the scraper is still scraping, the session expires and an unhandled exception is thrown.

Is this ever going to be put on npm?

Your README says that we can install the library from the npm repos using npm install skyward-rest. Are there plans to put it there?
For now, I would suggest changing it to

npm install https://github.com/FruitsNVeggies/skyward-rest

unsafe-eval' is not an allowed source of script

go this error while trying to scrape my gradebook: Uncaught (in promise) EvalError: Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: "script-src 'self' 'unsafe-inline'".

On src/reportcard/parse.js line 10:
return (results === null) ? {} : eval(`0 || ${results[1]}`);

My code:

// Skyward API things
const skyward = require('skyward-rest');
const url = 'https://skyward-lisdprod.iscorp.com/scripts/wsisa.dll/WService=wsedulewisvilletx/fwemnu01.w';
const scraper = skyward(url); // the scraper!

// login info
const user = 'user';
const pass = 'pass';

scraper.scrapeReport(user, pass)

I saw your comment on that line and tried to fix it myself, but no luck; it was way too complicated for me

Ability to scrape academic history

In order to calculate a student's GPA, their entire academic history must be accounted for.
All grades are featured under the Academic History tab in Skyward, and therefore can be accessed.

Smarter parameters for .scrape()

.scrape() should accept an array of lits, returning an array of all that was specified.
Also, instead of returning an empty array as the current .scrape() does when invoked without any arguments, it should scrape everything.

Accessing more information

How can I access more information on Skyward? For example when I reach this page with the url ../sfacademichistory001.w, I can only access the names of the courses that I've taken through the skyward-rest program. However, when I actually go to skyward, I am able to click on the class names for more information. When I click on the classname, I am not redirected to a new page with a link. It is just a pop-up with no link. How can I get the information in the pop-up?

When I do the line scrapeReport, I get a 5-digit number with key value "course". The comments in the README say that it is the 5-digit course number for the class, but I don't know how this course number was created. I can't find this number anywhere in skyward.

Thanks!!

Faas application

Puppeteer is huge and clunky compared to simply making requests. Eliminating puppeteer would reduce the dependency size and required runtime resources.

The goal is to make the library fit for a Functions-as-a-service (Faas) architecture.

stuGradesGrid not found

When running,

scraper.scrapeReport(username, password)
.then(({ data, raw }) => {
  //console.log(data) // array of reports
  console.log(raw) // fetched html before parsing
});

This error occurs:

(node:7867) UnhandledPromiseRejectionWarning: Error: stuGradesGrid not found

at module.exports (/home/alex/Documents/Programming/IA Ideas/node_modules/skyward-rest/src/reportcard/condense.js:28:39)
at Object.getData (/home/alex/Documents/Programming/IA Ideas/node_modules/skyward-rest/src/reportcard/index.js:13:19)
at authenticate.then.then.response (/home/alex/Documents/Programming/IA Ideas/node_modules/skyward-rest/index.js:13:26)
at process._tickCallback (internal/process/next_tick.js:68:7)

Information missing

When I go to the condense.js file in the history folder and I write console.log(data) under the export default (data) => { line, I get lines of data from the "Academic History" part of skyward. Then, it prints information about each class. It has values such as data-pStudent, data-pClassMeetRecID, etc. When I click on the information for each class, those values are not given. Where is it getting this information? In addition, the line console.log(data) also doesn't print some values that are available when I click on more information for each class. How can I evoke this information from the "Academic History" tab? I don't really understand what is happening with the code in the condense, scrape, and chunk files.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.