kylart / malscraper Goto Github PK

View Code? Open in Web Editor NEW

175.0 175.0 50.0 627 KB

Scrape everything you can from MyAnimeList.net

License: MIT License

JavaScript 100.00%

anime anime-search api malscraper myanimelist myanimelist-api news scraper seasons

malscraper's People

Contributors

Stargazers

Watchers

Forkers

pwadesu jaruba yusufipek kadantte testinproduction rahmidaniel anifox-moe whosboosh raja1802 drouko nikxxx dowzhong inf3cti0n95 gulnazaki all-anime yobokata justalk khavitidala gabb-c chunkydonut21 maisans-maid sumonst21 nekirokankou nikhil9hac 0wx akashhdev cahyo224 platers unorphen nostaboo kanmikan htmai-880 davidwang1027 titbittyrant tractrick stavrosnik4 legendaryemoji damien111 arciven mrcanelas venkatlohithdasari sagnik1004 muunatic sawtaytoes luccamrofc dindin3515 johndoeantler timarques mastamaru

malscraper's Issues

Can't check credentials

What I'm trying to do:
I'm trying to check credentials of user trying to login

Error
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /api/account/verify_credentials.xml was not found on this server.</p> <p>Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.</p> </body></html>

Source Code
`const api = new malScraper.officialApi({
username: username,
password: password,
});

  try {
    const res = await api.checkCredentials();
    console.log(res);
  } catch (e) {
    console.log(e);
  }`

error

const malScraper = require('mal-scraper')
/home/pi/node_modules/mal-scraper/src/info.js:152
.then(async (items) => {
^

SyntaxError: Unexpected token (

Next Release ?

Hello,

I would like to know when the next release is planned?

Thanks.

Anime Search data model resulted wrong information

As you can see in the image, title, synopsis, and picture are not defined and wrong, this error probably comes from MAL, who may change something on the page. Please fix, thank you.

officialApi.search() method returns "Cannot read property 'entry' of undefined"

I suspect that the entire official API part is broken, as i noticed that at least some of the endpoints of the official API now returns 404 errors, and that the official API documentation itself returns a 404

Search.search() misplaced value in data model

Hello there, apparently there is some misplaced value in search.search() data model, here is the screenshot.

How to get titles without leftovers

how to getting title for new season without leftover (TV Continuing). please take a look here
spring 2018
https://pastebin.com/fXbsrzQZ
summer 2018
i have marked some leftover
https://pastebin.com/tkKAy5wQ

what i'm trying to achieve here were getting title without leftover (TV Continuing)
could we scaping season from this link?
https://www.livechart.me/summer-2018/tv?leftovers=false
summer for season 2018 for year

takes few seconds to fetch

Im trying to the api and takes around 10 seconds to fetch something. (ex: 2019 summer)
is there some kind of pre requisite for the api?

picture is undefined

picture property is undefined, happens when you try to get info from url and with all animes.

[DOCUMENTATION] Duplication of section

Hello,

You have a duplication of an anchor in the documentation. I am not sure how you wanna turn this. So I just lest you know.
Check for Anime reviews data model

This section appear twice in the Readme.md

high vulnerability

High Server-Side Request Forgery

Package axios

Patched in >=0.21.1

Dependency of mal-scraper

Path mal-scraper > axios

More info https://npmjs.com/advisories/1594

Missing package-lock.json

You should commit package-lock.json, this file keep accurate versions of installed dependencies and avoid some problems.

shortDescription

idk if this is an issue but the shortDescription is unfinished. Is there a way to get the full description?

Fetching user info in getWatchListFromUser

Hey! This is more of a question / feature request.. Is it possible to fetch user info like username, image_url and stuff like that in the getWatchListFromUser()? So it can be:

username: 'exampleUsername',
image_url: 'somelink',
otherStuff: 'As many as you want'
data: [/* This array contains the usual return */]

If that's not possible can we have a getUser or add a functionality to search for users in the search function.. I don't mind any work arounds and whatever you find suitable.

The way I use getWatchListFromUser is that people can see each other's lists but I cannot display their avatar and info upon fetching the list. I can only show the data.

Thank you in advance, I'm a huge fan of the work btw. Much love

Cannot read property 'forEach' of undefined

Getting this error when I try to get information about an anime by name using the function:
getInfoFromName()

Error Stack:

Cannot read property 'forEach' of undefined
   at node_modules/mal-scraper/src/info.js:193:23
   at runMicrotasks (<anonymous>)
   at processTicksAndRejections (internal/process/task_queues.js:93:5)

I hope it will be fixed soon!

Fix typo

Fix typo in repo's description.

Current Description

Scrap everything you can from MyAnimeList.net

Suggestion

Scrape everything you can from MyAnimeList.net

.getInfoFromName() shows a max 10 characters

Even though all the anime I search for has much more over 10 characters the listing shows only 10. Is their a way I can go around this to scrap all the characters listed under the anime on the my anime list website?

Update on .getInfoFromName()

Can you make the character array result in .getInfoFromName() grab from the characters in 'more characters' linked page on the main Mal anime page?

This is what the page displays:

from the looks of it you could also grab favourite count of the anime from this

Sorted search results

Is there a way to get the results of a search sorted by the total amount of people who have scored/watched it?

Can't use MalScraper because anything results 403 forbidden

Hello there, recently i am having this problem. MalScraper won't give the result, instead it returns '403 forbidden'.

I am using glitch.com server, i tried to my different project but the result is still the same, I don't know it's just server-side error or same problem to anyone, i hope you can provide help, thanks.

Here is the log
https://drive.google.com/file/d/1LhsB--hf-DKGyBjKZg06kbwSF3NurkLm/view?usp=drivesdk

Lack of information when searching for a manga

Hello:

I really have no idea if there is a template for bug reporting. My apologies.

The problem in question is searching for information from a manga using the search.seach() method.

Example of my code:

const malScraper = require('mal-scraper')

malScraper.getInfoFromURL("https://myanimelist.net/manga/21479/Sword_Art_Online").catch(err => undefined).then((data) => {
  console.log(data)
})

The expected answer is the result with all the information of the manga in question.

The result I receive:

{
  title: '',
  synopsis: '',
  picture: undefined,
  characters: [
    {
      link: 'https://myanimelist.net/character/36765/Kazuto_Kirigaya',
      picture: 'https://cdn.myanimelist.net/images/characters/7/204821.jpg',
      name: 'Kirigaya, Kazuto',
      role: 'Main',
      seiyuu: [Object]
    },
    {
      link: 'https://myanimelist.net/character/36828/Asuna_Yuuki',
      picture: 'https://cdn.myanimelist.net/images/characters/15/262053.jpg',
      name: 'Yuuki, Asuna',
      role: 'Main',
      seiyuu: [Object]
    },
    {
      link: 'https://myanimelist.net/character/55147/Shino_Asada',
      picture: 'https://cdn.myanimelist.net/images/characters/10/235939.jpg',
      name: 'Asada, Shino',
      role: 'Main',
      seiyuu: [Object]
    },      role: 'Main',
      seiyuu: [Object]
    },
    {
      link: 'https://myanimelist.net/character/75450/Alice_Zuberg',
      picture: 'https://cdn.myanimelist.net/images/characters/7/348874.jpg',
      name: 'Zuberg, Alice',
      role: 'Main',
      seiyuu: [Object]
    },
    {
      link: 'https://myanimelist.net/character/36831/Suguha_Kirigaya',
      picture: 'https://cdn.myanimelist.net/images/characters/9/192515.jpg',
      name: 'Kirigaya, Suguha',
      role: 'Main',
      seiyuu: [Object]
    },
    {
      link: 'https://myanimelist.net/character/64523/Yuuki_Konno',
      picture: 'https://cdn.myanimelist.net/images/characters/4/278171.jpg',
      name: 'Konno, Yuuki',
      role: 'Supporting',
      seiyuu: [Object]
    },{
      link: 'https://myanimelist.net/character/43892/Yui',
      picture: 'https://cdn.myanimelist.net/images/characters/15/264165.jpg',
      name: 'Yui',
      role: 'Supporting',
      seiyuu: [Object]
    },
    {
      link: 'https://myanimelist.net/character/106217/Bercouli_Synthesis_One',
      picture: 'https://cdn.myanimelist.net/images/characters/6/245565.jpg',
      name: 'Bercouli Synthesis One',
      role: 'Supporting',
      seiyuu: [Object]
    },
    {
      link: 'https://myanimelist.net/character/103297/Tiese_Shtolienen',
      picture: 'https://cdn.myanimelist.net/images/characters/10/239851.jpg',
      name: 'Shtolienen, Tiese',
      role: 'Supporting',
      seiyuu: [Object]
    }
  ],
  staff: [
    {
      link: 'https://myanimelist.net/character/36765/Kazuto_Kirigaya',
      picture: 'https://cdn.myanimelist.net/images/characters/7/204821.jpg',
      name: 'Kirigaya, Kazuto',
      role: 'Main'
    },{
      link: 'https://myanimelist.net/character/36828/Asuna_Yuuki',
      picture: 'https://cdn.myanimelist.net/images/characters/15/262053.jpg',
      name: 'Yuuki, Asuna',
      role: 'Main'
    },
    {
      link: 'https://myanimelist.net/character/55147/Shino_Asada',
      picture: 'https://cdn.myanimelist.net/images/characters/10/235939.jpg',
      name: 'Asada, Shino',
      role: 'Main'
    },
    {
      link: 'https://myanimelist.net/character/70899/Eugeo',
      picture: 'https://cdn.myanimelist.net/images/characters/4/367313.jpg',
      name: 'Eugeo',
      role: 'Main'
    },
    {  link: 'https://myanimelist.net/character/75450/Alice_Zuberg',
      picture: 'https://cdn.myanimelist.net/images/characters/7/348874.jpg',
      name: 'Zuberg, Alice',
      role: 'Main'
    },
    {
      link: 'https://myanimelist.net/character/36831/Suguha_Kirigaya',
      picture: 'https://cdn.myanimelist.net/images/characters/9/192515.jpg',
      name: 'Kirigaya, Suguha',
      role: 'Main'
    },
    {
      link: 'https://myanimelist.net/character/64523/Yuuki_Konno',
      picture: 'https://cdn.myanimelist.net/images/characters/4/278171.jpg',
      name: 'Konno, Yuuki',
      role: 'Supporting'
    },
    {
      link: 'https://myanimelist.net/character/43892/Yui',
      picture: 'https://cdn.myanimelist.net/images/characters/15/264165.jpg',
      name: 'Yui',
      role: 'Supporting'
    },
    {
      link: 'https://myanimelist.net/character/106217/Bercouli_Synthesis_One',
      picture: 'https://cdn.myanimelist.net/images/characters/6/245565.jpg',
      name: 'Bercouli Synthesis One',
      role: 'Supporting'
    },{
      link: 'https://myanimelist.net/character/106217/Bercouli_Synthesis_One',
      picture: 'https://cdn.myanimelist.net/images/characters/6/245565.jpg',
      name: 'Bercouli Synthesis One',
      role: 'Supporting'
    },
    {
      link: 'https://myanimelist.net/character/103297/Tiese_Shtolienen',
      picture: 'https://cdn.myanimelist.net/images/characters/10/239851.jpg',
      name: 'Shtolienen, Tiese',
      role: 'Supporting'
    }
  ],
  trailer: undefined,
  englishTitle: 'Sword Art Online',
  japaneseTitle: 'ソードアート・オンライン',
  synonyms: 'S.A.O, SAO',
  type: 'Novel',
  episodes: '',
  status: 'Publishing',
  aired: '',
  premiered: '',
  broadcast: '',
  producers: [ '' ],
  studios: [ '' ],
  source: '#490Popularity #168Members 47,694NovelBUNBUN (Art), Kawahara, Reki (Story)',
  genres: [ 'Action', 'Adventure', 'Fantasy', 'Game', 'Romance', 'Sci-Fi' ],
  duration: '',
  rating: '',
  score: '8.11',
  scoreStats: 'scored by 18,566 users',
  ranked: "#4902  2 based on the top manga page. Please note that 'R18+' titles are excluded",
  popularity: '#168',
  members: '47,694',
  favorites: '3,353',
  id: 21479
}

You can also see an error in the data shown, for example, the staff in charge of the study are the characters in the story themselves.
I have checked if it happens in other sleeves and indeed the error persists.

mal-scraper: v2.9.0
nodejs: v12.16.2
npm: v6.14.4

Install Error

npm ERR! code ENOSELF
npm ERR! Refusing to install package with name "mal-scraper" under a package
npm ERR! also called "mal-scraper". Did you name your project the same
npm ERR! as the dependency you're installing?
npm ERR!
npm ERR! For more information, see:
npm ERR! https://docs.npmjs.com/cli/install#limitations-of-npms-install-algorithm

npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\Berthens\AppData\Roaming\npm-cache_logs\2021-02-24T20_04_35_359Z-debug.log

Manga Details

Is there no way to retrieve manga details like getInfoFromName(name) for anime? Especially since the officialApi method is broken :(

[QUESTIONS] Refactoring and informations about files

Hello,

I have seen that in your directory src/search, you have a file name constants.js.
I was wondering if this file is exclusively for the directory search or for the whole application ?
I am asking that because there are duplicate constants like BASE_URL across multiple files and I was wondering if I can put everything here or I create a new one at the root of the src directory.

Thank you for your answer in advance.

Error: A 'declare' modifier cannot be used in an already ambient context.

With mal-scraper 2.12.0 and typescript 5.0.2 :

It seems that the declare at https://github.com/Kylart/MalScraper/blob/dev/src/index.d.ts#L4
declare let search: Search;
need to be remove.

Without it, the transpilation is working

getSeason returned an empty object

When I tryed getSeason('2020', 'summer') it returns an empty object in the array:

[BUG] Synonyme & staffs

Hello,

I have been increasing the tests before starting a refactoring and I have noticed this :

const data = await getInfoFromURL('https://myanimelist.net/anime/30654/Ansatsu_Kyoushitsu_2nd_Season')

response error :

synonyms: 'Ansatsu Kyoushitsu Season 2, Ansatsu Kyoushitsu Final Season'

It should be an array, not a string.

Also, I have notice an error on the novel scrap with the staffs and characters fields that are identical.

Bug

TypeError: Cannot read property 'url' of undefined
at C:\Users\7thghoul\node_modules\mal-scraper\src\info.js:222:33
at processTicksAndRejections (internal/process/task_queues.js:97:5)

getInfoFromName don't work

Node 10.6.0
Electron 2.0.5

TypeError: match is not a function
    at getResultsFromSearch.then (info.js?f8a3:154)
    at <anonymous>

Replace Request

Request has been deprecated since 2020 and has recently been marked with a moderate security vulnerability as seen here.

I see only the Official API section of the package uses request, so I'm assuming it can be replaced with axios or some alternative?

@Kylart sorry to mention you but I feel this is a relatively important issue as it shows several warnings on install.

Error pagination

Returns an error when trying to paginate an advanced search, which doesn't have as many elements
Code:

malScraper.search.search("anime", {
                "term": "12-sai.: Kiss, Kirai, Suki",
                "type": 2,
                "has": 50
            }).then((data) => console.log(data)).
                catch((err) => console.log(err));

Error:

TypeError: Cannot read property 'length' of null
    at hasNext (c:\Users\gonfe\Documents\GitHub\pruebas\node_modules\mal-scraper\src\search\index.js:145:27)
    at c:\Users\gonfe\Documents\GitHub\pruebas\node_modules\mal-scraper\src\search\index.js:163:22
    at processTicksAndRejections (internal/process/task_queues.js:97:5)

remove leftover

is it possible to hide/remove leftover title for new season

Support for episode titles

for example https://myanimelist.net/anime/31339/Drifters/episode

what i want to be in this api?

Episode Title
Aired

undefined in everything

so, my data is returning undefined when i put like data.title and etc

No data from .getInfoFromName(name)

title: '\n Top\n ',
synopsis: '',
picture: undefined,

search maxResults not working

Although I'm really enjoying this API for my little project there is a tiny issue that causes a lot of stuff to go slower,

in the search function the option maxResults isn't functional. No matter what is put there (1, 250, 40) it will always return 50.

const malScraper = require('mal-scraper')
const search = malScraper.search

var anime = search.search('anime', {
    maxResults: 100,
    term: argument
 })
    .catch(console.error)
    .then((results) => console.log(results.length))

always returns 50. no matter what you set maxResults to.

Data fetching incorrect

Some data in the anime JSON returns incorrect
some examples are "title" that return "Top"
and picture, that returns undefined

getRecommendationsList parameter object does not need name

malScraper.getRecommendationsList({ name: 'Sakura Trick', id: 20047 }) has the same result as
malScraper.getRecommendationsList({ name: 'anything', id: 20047 })
Looks like MAL doesn't care about the name. Ex https://myanimelist.net/anime/73/anything/userrecs works.

getRecommendationsList should be callable with just an id, no need to provide a name.

Timed out while running tests

Hello,

I have been interested by your scrapper for MAL but by running your test, I am facing this problem :
npm test

Timed out while running tests

After multiple try, I manage to see that it always appends on the same part of the code. By example, for the test checking that Naruto has 500 episode, you have a 5 recursive calls. You get all the pages one after the other. Same goes for Drifters. But Ava has a timeout error trigger if the connection is a bit too long.

False scoreStats

Hi, I found an error in the scoreStats's data : the number of users if printed twice.
For example : Dragon Ball Z is scored by 488778488,778 users. (We have 488,778 printed twice)
and it's the same for all animes.
I put some screenshots for you to see the issue.

I hope you will fix that quickly

Sincery yours

-LePtitMetalleux

[Feature Request]: Add Japanese fields

It doesn't look like the Japanese name for the field is currently supported for Japanese names, as seen here: https://myanimelist.net/manga/23390/Shingeki_no_Kyojin
進撃の巨人

Is it possible to add this data? Is it possible to search by Japanese title?

Support

Is there any way to add Nsfw / safe word filter to this

malScraper.getInfoFromName()

title and synopsis undefined

well i am using mal-scraper and it doesn't show the tittle of the anime also it doesn't show the synopsis

Idk if it's an issue. But it's bothering me.

When i use
.getinfofromname('Anohana')
If returns an irrevelant anime but when i use
.getresultsfromsearch('Anohana')
It returns exactly what i want in the first item in the array.. it's not only anohana, it's the same with many other animes as well. What do you think about it?
Shall we use getresultsfromsearch and then pick it's first name or url and use that value to search using getinfofromname or getinfofromurl?

Genres not parsed correctly

Example: Action, Adventure, Comedy, Super Power, Martial Arts, Shounen are parsed as

[
  'ActionAction',
  '    AdventureAdventure',
  '    ComedyComedy',
  '    Super PowerSuper Power',
  '    Martial ArtsMartial Arts',
  '    ShounenShounen'
]

Error search anime with type

If I add in the search parameters the key "type", it returns an incorrect result, example:

✔️Correct
Code:

var malScraper = require("mal-scraper")
malScraper.search.search("anime", {
                "term": "BanG Dream! 3rd Season"
            }).then((data) => console.log(data[0])).
                catch((err) => err);

Result:

{
  thumbnail: 'https://cdn.myanimelist.net/r/100x140/images/anime/1985/104767.jpg?s=a0e163e0c3ee85136e20d9ed15490b99',
  url: 'https://myanimelist.net/anime/37870/BanG_Dream_3rd_Season',
  video: 'https://myanimelist.net/anime/37870/BanG_Dream_3rd_Season/video',
  shortDescription: 'As the all-girl band Poppin’Party continues to rise in fame, they find a poster for an event at the legendary Nippon Budoukan. Now determined to participate in the "BanG 
Dream," Kasumi Toyama and the...read more.',
  title: 'BanG Dream! 3rd Season',
  type: 'TV',
  nbEps: '13',
  score: '8.09',
  startDate: '01-23-20',
  endDate: '04-23-20',
  members: '13,238',
  rating: 'PG-13'
}

❌ Error
Code:

var malScraper = require("mal-scraper")
malScraper.search.search("anime", {
                "term": "BanG Dream! 3rd Season",
                "type": 1
            }).then((data) => console.log(data[0])).
                catch((err) => err);

Result:

{
  thumbnail: 'https://cdn.myanimelist.net/r/100x140/images/anime/4/84177.jpg?s=a7c4462cf6f5458a8b28496a75863876',
  url: 'https://myanimelist.net/anime/25777/Shingeki_no_Kyojin_Season_2',
  video: 'https://myanimelist.net/anime/25777/Shingeki_no_Kyojin_Season_2/video',
  shortDescription: 'For centuries, humanity has been hunted by giant, mysterious predators known as the Titans. Three mighty walls—Wall Maria, Rose, and Sheena—provided peace and protection 
for humanity for over a hundre...read more.',
  title: 'Shingeki no Kyojin Season 2',
  type: 'TV',
  nbEps: '12',
  score: '8.40',
  startDate: '04-01-17',
  endDate: '06-17-17',
  members: '1,082,750',
  rating: 'R'
}

Cors problem on electron + webpack + vue

XMLHttpRequest cannot load https://myanimelist.net/search/prefix.json?type=anime&keyword=naruto. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:9080' is therefore not allowed access.

mal-scraper 2.2.1

electron 1.7.5

.getWatchListFromUser() issues

This issue addresses 2 bugs i ran into with the getWatchListFromUser() method :

First:

TypeError: Cannot read property '0' of undefined at parseString (D:\Felix\chino\indev\node_modules\mal-scraper\src\watchList.js:76:38) at Parser.<anonymous> (D:\Felix\chino\indev\node_modules\xml2js\lib\parser.js:303:18) at emitOne (events.js:115:13) at Parser.emit (events.js:210:7) at SAXParser.onclosetag (D:\Felix\chino\indev\node_modules\xml2js\lib\parser.js:261:26) at emit (D:\Felix\chino\indev\node_modules\sax\lib\sax.js:624:35) at emitNode (D:\Felix\chino\indev\node_modules\sax\lib\sax.js:629:5) at closeTag (D:\Felix\chino\indev\node_modules\sax\lib\sax.js:889:7) at SAXParser.write (D:\Felix\chino\indev\node_modules\sax\lib\sax.js:1436:13) at Parser.exports.Parser.Parser.parseString (D:\Felix\chino\indev\node_modules\xml2js\lib\parser.js:322:31)
Line 76 of the watchList file, when it resolves the promise, it tries reading the first element
of mal.myinfo (which i guess is supposed to be an array), however mal.myinfo doesn't seem to be
defined for all users.

By that i mean that it doesn't work with some users, and work with some others. For reference:

The bug happen with the user Paradoxalcorp (https://myanimelist.net/profile/Paradoxcorp)
Works as expected with the user niputi (https://myanimelist.net/profile/niputi)

Second:

Within the stats object that is in the object returned by getWatchListFromUser(), the property name of the property containing the username is undefined instead of username or name (see the screenshot)

Character info search

maybe I'm missing something, I wanted to know if there is any way we can find character info using their name just like we do to find anime malScraper.getInfoFromName(query)
if there is no way currently see if you can add that feature.

No get all anime function/method

I think there's no getAllAnimes function or anything similar that will get all animes that's in the API.

kylart / malscraper Goto Github PK

malscraper's People

Contributors

Stargazers

Watchers

Forkers

malscraper's Issues

Current Description

Suggestion

Recommend Projects

Recommend Topics

Recommend Org