ucsd-historical-enrollment-data / webreg_scraper Goto Github PK

If the scraper is trying to log into WebReg (e.g., it was logged out due to maintenance) and you press CTRL+C, it'll only terminate after it's logged back in.

This way, for example, if we need enrollment data but WebReg isn't down, we can look up the data from the cache.

Hi Wang, I was using your webreg_scraper and I found that it was not collecting the days, start time, end time and some other attributes in the output file. I would like to get them in the result, how can I do so? I tried to go through the code but I know nothing about Rust. It seems like the webweg wrapper is imported somehow into this repo and its functions(SearchRequestBuilder) are used to get those data. Could you guide me through this?

Thank you

This can be made a configuration option. For each term, define a property "collectData" that can either be true if we want to collect enrollment data and false otherwise.

This is useful if we're only interested in exposing a term to the web server.

While this is not a problem with the program itself, I just wanted to see if you encountered the same problem when setting this up

npm i -g typescript
npm ERR! code ECONNREFUSED
npm ERR! syscall connect
npm ERR! errno ECONNREFUSED
npm ERR! FetchError: request to http://registry.npmjs.org/typescript failed, reason: connect ECONNREFUSED xxxx:xxxx::xxxx:xxxx:xx

I suspect its something to do with npm not able to connect via ipv6 but I'm not completely sure.

First, we should introduce a new feature called personal. This feature only works if the api feature is enabled. The idea is that this feature, if enabled, allows the user to access data specific to their account (hence the "personal" name). For example, the user will be able to get their schedule through an endpoint in the API if the personal feature is enabled. We can also use this opportunity to add endpoints like enrolling in classes automatically.

As mentioned above, we should give the user the ability to access their own schedule using the API. This endpoint's code is very similar to the code used by the other endpoints, the only difference being that there's a feature gate on the corresponding code.

That way, we can be more consistent with what the wrapper is returning.

/// Processes a GET response from the resulting JSON, if any.
///
/// # Parameters
/// - `res`: The initial response.
///
/// # Returns
/// The result of processing the response.
async fn _process_get_result<T: DeserializeOwned>(
    &self,
    res: Result<Response, Error>,
) -> Result<T, Cow<'a, str>> {
    match res {
        Err(e) => Err(e.to_string().into()),
        Ok(r) => {
            if !r.status().is_success() {
                return Err(r.status().to_string().into());
            }
            let text = match r.text().await {
                Err(e) => return Err(e.to_string().into()),
                Ok(s) => s,
            };
            match serde_json::from_str(&text) {
                Err(e) => return Err(e.to_string().into()),
                Ok(o) => o,
            }
        }
    }
}

Right now, whenever the scraper gets logged out, it has to wait 8 minutes. But, perhaps we can make this cooldown shorter when we know that the time is not 4AM Pacific.

Currently, there's no way for us to tell when a scraper last requested data. It would be nice if we could store when we last requested data so we can see, for example, if the scraper is "frozen."