bigfoott / scrapedduck

ScrapedDuck routinely scrapes LeekDuck.com and pushes the found data to a branch on this repository, for use by other external applications.

Hello! 👋

I'm Anthony, a software dev from New Jersey with a Bachelor's in Computer Science and an interest in web dev and virtual reality.

I'm currently focused on a few projects!

• VR Speedrunning Leaderboards is a project that makes it easy to view different VR games' leaderboards. Check out the various projects related to VRSR here.
• PoGOEvents is a project that allows you to add a list of current/upcoming Pokemon GO events to your iOS home screen! Uses data from ScrapedDuck.
• ScrapedDuck is a project that routinely scrapes LeekDuck.com and pushes the found data to a branch on the repository, to be used by other external applications.
• MMM-PokemonGOEvents is a project that displays Pokémon GO events on a MagicMirror².

View my programming portfolio or check out some of the 3D printed art I've made (which is also available on Thingiverse)!

scrapedduck's Issues

Add Rocket Lineup data

if possible, it would be nice to get an end point for the new rocket lineup page
https://leekduck.com/rocket-lineups/

Non event tasks with similar tasks text are getting pulled into the event task category

Duplicate pokemon encounters in research have same task instead of different

This occurs with Eevee and Onix currently in March's research.
Eevee is a reward for "Evolve a Pokémon" and "Catch 5 Pokémon", but the json returned by ScrapedDuck for research shows two Eevee rewards for just "Evolve a Pokémon" and no mention of "Catch 5 Pokémon".

    {
        "text": "Evolve a Pokémon",
        "type": "power",
        "rewards": [
            {
                "name": "Eevee",
                "image": "https://undefined",
                "canBeShiny": true,
                "combatPower": {
                    "min": 424,
                    "max": 459
                }
            },
            {
                "name": "Eevee",
                "image": "https://undefined",
                "canBeShiny": true,
                "combatPower": {
                    "min": 424,
                    "max": 459
                }
            }
        ]
    },

The same happens for Onix with "Battle in GO Battle League" and "Make 3 Great Throws in a row".

https://leekduck.com/research/

https://raw.githubusercontent.com/bigfoott/ScrapedDuck/data/research.json

<br> shows up in Community day disclaimer

A recent change to the format of the community day pages added HTML line breaks here.

They probably should be stripped by ScrapedDuck as the data already includes \n in it.

"bonusDisclaimers": [
  "* While most bonuses are only active during the three hours of the event, these bonuses will be active from 2:00 p.m. to 10:00 p.m. local time.<br>\n** The three-hour Incense bonus excludes Daily Adventure Incense.<br>\n*** The three-hour lure bonus excludes Golden Lure Modules."
],

Perhaps each disclaimer should also be their own entry in the bonsuDisclaimers array, rather than having \n.

Times are wrong?

It seems to not add the events (Community day) in Local Time to my calendar. The Community day is 2 PM - 5 PM local time, but it adds the April community day to 8-11 PM into my google calendar

Community Day extraData not populating

Will add a better description here once the next community day is announced :^)

Thank you and usage notification

A massive thank you for this project and having it auto maintained!

I just came to do a pet project I've been meaning to get around to for a while, which is an automated iCal calendar feed of the events page:

https://gocalendar.info/
https://othyn.github.io/go-calendar/
https://github.com/othyn/go-calendar

The events JSON resource came in perfectly for this and made a few days job into only a few short hours, massive thank you! Superb project.

Some of the start/end dates came up as null

Hello,

I am noticing some missing start/end dates. Example:

[
   {
      "eventID":"season-12-master-league_evolution-cup",
      "name":"Master League and Evolution Cup | Season of Light",
      "eventType":"go-battle-league",
      "heading":"GO Battle League",
      "link":"https://leekduck.com/events/season-12-master-league_evolution-cup/",
      "image":"https://leekduck.com/assets/img/events/gobattleleague-master-key.jpg",
      "start":null,
      "end":"2022-10-13T20:00:00.000Z",
      "extraData":null
   },
   {
      "eventID":"mega-manectric-in-mega-raids-october-2022",
      "name":"Mega Manectric in Mega Raids",
      "eventType":"raid-battles",
      "heading":"Raid Battles",
      "link":"https://leekduck.com/events/mega-manectric-in-mega-raids-october-2022/",
      "image":"https://leekduck.com/assets/img/events/mega-default.jpg",
      "start":null,
      "end":"2022-10-20T10:00:00.000",
      "extraData":{
         "raidbattles":{
            "bosses":[
               {
                  "name":"Mega Manectric",
                  "image":"https://leekduck.com/assets/img/pokemon_icons/pokemon_icon_310_51.png",
                  "canBeShiny":true
               }
            ],
            "shinies":[
               
            ]
         }
      }
   },
   {
      "eventID":"shiny-corsola-okinawa-shirt-pikachu",
      "name":"Pikachu wearing an Okinawan kariyushi shirt and Shiny Corsola",
      "eventType":"location-specific",
      "heading":"Location-specific",
      "link":"https://leekduck.com/events/shiny-corsola-okinawa-shirt-pikachu/",
      "image":"https://leekduck.com/assets/img/events/pikachu-okinawa.jpg",
      "start":null,
      "end":null,
      "extraData":null
   }
]

Community Day extraData: bonuses doesn't pull disclaimer

The bonuses data for the Community Day event show the asterisk for the disclaimer at the bottom of the bonuses, but the extraData doesn't pull the disclaimer itself
Disclaimer:

• While most bonuses are only active during the three hours of the event, these bonuses will be active from 11:00 a.m. to 7:00 p.m. local time.

Side note: I'm working on an iOS shortcut to quick reference some of this data. Leek Duck pointed me to this repo. I'd love to share progress with you. Let me know how to best contact you.

Add last updated metadata

Scrape pages for the last updated date and add that to the end point. this could be helpful to allow me to notify users of outdated information or cache information rather than needing to make an API request

Events can have multiple event types

The World of Wonders: Taken Over event is both an Event and Team Go Rocket. ScrapedDuck only notes the first of these two.

https://leekduck.com/events/world-of-wonders-taken-over/

   {
        "eventID": "world-of-wonders-taken-over",
        "name": "World of Wonders: Taken Over",
        "eventType": "event",
        "heading": "Event",
        "link": "https://leekduck.com/events/world-of-wonders-taken-over/",
        "image": "https://leekduck.com/assets/img/events/team-go-rocket-giovanni-shadow-groudon.jpg",
        "start": "2024-03-27T00:00:00.000",
        "end": "2024-03-31T23:59:00.000",
        "extraData": null
    },

Snapshot research tasks have different types

"Take 3 snapshots of wild Water-type Pokémon" has the type "catch",
But "Take a snapshot of a wild Pokémon" has the type "misc".

Both should likely be "misc".

Buddy & Friendship tasks missing "type"

https://raw.githubusercontent.com/bigfoott/ScrapedDuck/data/research.json

tasks for the Buddy & Friendship category are missing a "type".

All images for research rewards are undefined

[
    {
        "text": "Catch 7 Pokémon",
        "type": "catch",
        "rewards": [
            {
                "name": "Magikarp",
                "image": "https://undefined",
                "canBeShiny": true,
                "combatPower": {
                    "min": 99,
                    "max": 117
                }
            }
        ]
    }
]

Add extraData for general events

• Bonuses
• Event-specific field research
• (will put more if/when I think of more)

Showcases Extra Data

Add Pokemon names for PokeStop Showcases 🫡