kylart / malscraper Goto Github PK
View Code? Open in Web Editor NEWScrape everything you can from MyAnimeList.net
License: MIT License
Scrape everything you can from MyAnimeList.net
License: MIT License
What I'm trying to do:
I'm trying to check credentials of user trying to login
Error
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /api/account/verify_credentials.xml was not found on this server.</p> <p>Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.</p> </body></html>
Source Code
`const api = new malScraper.officialApi({
username: username,
password: password,
});
try {
const res = await api.checkCredentials();
console.log(res);
} catch (e) {
console.log(e);
}`
const malScraper = require('mal-scraper')
/home/pi/node_modules/mal-scraper/src/info.js:152
.then(async (items) => {
^
SyntaxError: Unexpected token (
Hello,
I would like to know when the next release is planned?
Thanks.
I suspect that the entire official API part is broken, as i noticed that at least some of the endpoints of the official API now returns 404 errors, and that the official API documentation itself returns a 404
how to getting title for new season without leftover (TV Continuing). please take a look here
spring 2018
https://pastebin.com/fXbsrzQZ
summer 2018
i have marked some leftover
https://pastebin.com/tkKAy5wQ
what i'm trying to achieve here were getting title without leftover (TV Continuing)
could we scaping season from this link?
https://www.livechart.me/summer-2018/tv?leftovers=false
summer for season 2018 for year
Im trying to the api and takes around 10 seconds to fetch something. (ex: 2019 summer)
is there some kind of pre requisite for the api?
picture property is undefined, happens when you try to get info from url and with all animes.
Hello,
You have a duplication of an anchor in the documentation. I am not sure how you wanna turn this. So I just lest you know.
Check for Anime reviews data model
This section appear twice in the Readme.md
High Server-Side Request Forgery
Package axios
Patched in >=0.21.1
Dependency of mal-scraper
Path mal-scraper > axios
More info https://npmjs.com/advisories/1594
You should commit package-lock.json
, this file keep accurate versions of installed dependencies and avoid some problems.
idk if this is an issue but the shortDescription is unfinished. Is there a way to get the full description?
Hey! This is more of a question / feature request.. Is it possible to fetch user info like username
, image_url
and stuff like that in the getWatchListFromUser()? So it can be:
username: 'exampleUsername',
image_url: 'somelink',
otherStuff: 'As many as you want'
data: [/* This array contains the usual return */]
If that's not possible can we have a getUser
or add a functionality to search for users in the search
function.. I don't mind any work arounds and whatever you find suitable.
The way I use getWatchListFromUser is that people can see each other's lists but I cannot display their avatar and info upon fetching the list. I can only show the data.
Thank you in advance, I'm a huge fan of the work btw. Much love
Getting this error when I try to get information about an anime by name using the function:
getInfoFromName()
Error Stack:
Cannot read property 'forEach' of undefined
at node_modules/mal-scraper/src/info.js:193:23
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
I hope it will be fixed soon!
Fix typo in repo's description.
Scrap everything you can from MyAnimeList.net
Scrape
everything you can from MyAnimeList.net
Even though all the anime I search for has much more over 10 characters the listing shows only 10. Is their a way I can go around this to scrap all the characters listed under the anime on the my anime list website?
Is there a way to get the results of a search sorted by the total amount of people who have scored/watched it?
Hello there, recently i am having this problem. MalScraper won't give the result, instead it returns '403 forbidden'.
I am using glitch.com server, i tried to my different project but the result is still the same, I don't know it's just server-side error or same problem to anyone, i hope you can provide help, thanks.
Here is the log
https://drive.google.com/file/d/1LhsB--hf-DKGyBjKZg06kbwSF3NurkLm/view?usp=drivesdk
Hello:
I really have no idea if there is a template for bug reporting. My apologies.
The problem in question is searching for information from a manga using the search.seach() method.
Example of my code:
const malScraper = require('mal-scraper')
malScraper.getInfoFromURL("https://myanimelist.net/manga/21479/Sword_Art_Online").catch(err => undefined).then((data) => {
console.log(data)
})
The expected answer is the result with all the information of the manga in question.
The result I receive:
{
title: '',
synopsis: '',
picture: undefined,
characters: [
{
link: 'https://myanimelist.net/character/36765/Kazuto_Kirigaya',
picture: 'https://cdn.myanimelist.net/images/characters/7/204821.jpg',
name: 'Kirigaya, Kazuto',
role: 'Main',
seiyuu: [Object]
},
{
link: 'https://myanimelist.net/character/36828/Asuna_Yuuki',
picture: 'https://cdn.myanimelist.net/images/characters/15/262053.jpg',
name: 'Yuuki, Asuna',
role: 'Main',
seiyuu: [Object]
},
{
link: 'https://myanimelist.net/character/55147/Shino_Asada',
picture: 'https://cdn.myanimelist.net/images/characters/10/235939.jpg',
name: 'Asada, Shino',
role: 'Main',
seiyuu: [Object]
}, role: 'Main',
seiyuu: [Object]
},
{
link: 'https://myanimelist.net/character/75450/Alice_Zuberg',
picture: 'https://cdn.myanimelist.net/images/characters/7/348874.jpg',
name: 'Zuberg, Alice',
role: 'Main',
seiyuu: [Object]
},
{
link: 'https://myanimelist.net/character/36831/Suguha_Kirigaya',
picture: 'https://cdn.myanimelist.net/images/characters/9/192515.jpg',
name: 'Kirigaya, Suguha',
role: 'Main',
seiyuu: [Object]
},
{
link: 'https://myanimelist.net/character/64523/Yuuki_Konno',
picture: 'https://cdn.myanimelist.net/images/characters/4/278171.jpg',
name: 'Konno, Yuuki',
role: 'Supporting',
seiyuu: [Object]
},{
link: 'https://myanimelist.net/character/43892/Yui',
picture: 'https://cdn.myanimelist.net/images/characters/15/264165.jpg',
name: 'Yui',
role: 'Supporting',
seiyuu: [Object]
},
{
link: 'https://myanimelist.net/character/106217/Bercouli_Synthesis_One',
picture: 'https://cdn.myanimelist.net/images/characters/6/245565.jpg',
name: 'Bercouli Synthesis One',
role: 'Supporting',
seiyuu: [Object]
},
{
link: 'https://myanimelist.net/character/103297/Tiese_Shtolienen',
picture: 'https://cdn.myanimelist.net/images/characters/10/239851.jpg',
name: 'Shtolienen, Tiese',
role: 'Supporting',
seiyuu: [Object]
}
],
staff: [
{
link: 'https://myanimelist.net/character/36765/Kazuto_Kirigaya',
picture: 'https://cdn.myanimelist.net/images/characters/7/204821.jpg',
name: 'Kirigaya, Kazuto',
role: 'Main'
},{
link: 'https://myanimelist.net/character/36828/Asuna_Yuuki',
picture: 'https://cdn.myanimelist.net/images/characters/15/262053.jpg',
name: 'Yuuki, Asuna',
role: 'Main'
},
{
link: 'https://myanimelist.net/character/55147/Shino_Asada',
picture: 'https://cdn.myanimelist.net/images/characters/10/235939.jpg',
name: 'Asada, Shino',
role: 'Main'
},
{
link: 'https://myanimelist.net/character/70899/Eugeo',
picture: 'https://cdn.myanimelist.net/images/characters/4/367313.jpg',
name: 'Eugeo',
role: 'Main'
},
{ link: 'https://myanimelist.net/character/75450/Alice_Zuberg',
picture: 'https://cdn.myanimelist.net/images/characters/7/348874.jpg',
name: 'Zuberg, Alice',
role: 'Main'
},
{
link: 'https://myanimelist.net/character/36831/Suguha_Kirigaya',
picture: 'https://cdn.myanimelist.net/images/characters/9/192515.jpg',
name: 'Kirigaya, Suguha',
role: 'Main'
},
{
link: 'https://myanimelist.net/character/64523/Yuuki_Konno',
picture: 'https://cdn.myanimelist.net/images/characters/4/278171.jpg',
name: 'Konno, Yuuki',
role: 'Supporting'
},
{
link: 'https://myanimelist.net/character/43892/Yui',
picture: 'https://cdn.myanimelist.net/images/characters/15/264165.jpg',
name: 'Yui',
role: 'Supporting'
},
{
link: 'https://myanimelist.net/character/106217/Bercouli_Synthesis_One',
picture: 'https://cdn.myanimelist.net/images/characters/6/245565.jpg',
name: 'Bercouli Synthesis One',
role: 'Supporting'
},{
link: 'https://myanimelist.net/character/106217/Bercouli_Synthesis_One',
picture: 'https://cdn.myanimelist.net/images/characters/6/245565.jpg',
name: 'Bercouli Synthesis One',
role: 'Supporting'
},
{
link: 'https://myanimelist.net/character/103297/Tiese_Shtolienen',
picture: 'https://cdn.myanimelist.net/images/characters/10/239851.jpg',
name: 'Shtolienen, Tiese',
role: 'Supporting'
}
],
trailer: undefined,
englishTitle: 'Sword Art Online',
japaneseTitle: 'ソードアート・オンライン',
synonyms: 'S.A.O, SAO',
type: 'Novel',
episodes: '',
status: 'Publishing',
aired: '',
premiered: '',
broadcast: '',
producers: [ '' ],
studios: [ '' ],
source: '#490Popularity #168Members 47,694NovelBUNBUN (Art), Kawahara, Reki (Story)',
genres: [ 'Action', 'Adventure', 'Fantasy', 'Game', 'Romance', 'Sci-Fi' ],
duration: '',
rating: '',
score: '8.11',
scoreStats: 'scored by 18,566 users',
ranked: "#4902 2 based on the top manga page. Please note that 'R18+' titles are excluded",
popularity: '#168',
members: '47,694',
favorites: '3,353',
id: 21479
}
You can also see an error in the data shown, for example, the staff in charge of the study are the characters in the story themselves.
I have checked if it happens in other sleeves and indeed the error persists.
mal-scraper: v2.9.0
nodejs: v12.16.2
npm: v6.14.4
npm ERR! code ENOSELF
npm ERR! Refusing to install package with name "mal-scraper" under a package
npm ERR! also called "mal-scraper". Did you name your project the same
npm ERR! as the dependency you're installing?
npm ERR!
npm ERR! For more information, see:
npm ERR! https://docs.npmjs.com/cli/install#limitations-of-npms-install-algorithm
npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\Berthens\AppData\Roaming\npm-cache_logs\2021-02-24T20_04_35_359Z-debug.log
Is there no way to retrieve manga details like getInfoFromName(name) for anime? Especially since the officialApi method is broken :(
Hello,
I have seen that in your directory src/search
, you have a file name constants.js
.
I was wondering if this file is exclusively for the directory search
or for the whole application ?
I am asking that because there are duplicate constants like BASE_URL across multiple files and I was wondering if I can put everything here or I create a new one at the root of the src directory.
Thank you for your answer in advance.
With mal-scraper 2.12.0 and typescript 5.0.2 :
It seems that the declare at https://github.com/Kylart/MalScraper/blob/dev/src/index.d.ts#L4
declare let search: Search;
need to be remove.
Without it, the transpilation is working
Hello,
I have been increasing the tests before starting a refactoring and I have noticed this :
const data = await getInfoFromURL('https://myanimelist.net/anime/30654/Ansatsu_Kyoushitsu_2nd_Season')
response error :
synonyms: 'Ansatsu Kyoushitsu Season 2, Ansatsu Kyoushitsu Final Season'
It should be an array, not a string.
Also, I have notice an error on the novel scrap with the staffs and characters fields that are identical.
TypeError: Cannot read property 'url' of undefined
at C:\Users\7thghoul\node_modules\mal-scraper\src\info.js:222:33
at processTicksAndRejections (internal/process/task_queues.js:97:5)
Node 10.6.0
Electron 2.0.5
TypeError: match is not a function
at getResultsFromSearch.then (info.js?f8a3:154)
at <anonymous>
Request has been deprecated since 2020 and has recently been marked with a moderate security vulnerability as seen here.
I see only the Official API section of the package uses request, so I'm assuming it can be replaced with axios or some alternative?
@Kylart sorry to mention you but I feel this is a relatively important issue as it shows several warnings on install.
Returns an error when trying to paginate an advanced search, which doesn't have as many elements
Code:
malScraper.search.search("anime", {
"term": "12-sai.: Kiss, Kirai, Suki",
"type": 2,
"has": 50
}).then((data) => console.log(data)).
catch((err) => console.log(err));
Error:
TypeError: Cannot read property 'length' of null
at hasNext (c:\Users\gonfe\Documents\GitHub\pruebas\node_modules\mal-scraper\src\search\index.js:145:27)
at c:\Users\gonfe\Documents\GitHub\pruebas\node_modules\mal-scraper\src\search\index.js:163:22
at processTicksAndRejections (internal/process/task_queues.js:97:5)
is it possible to hide/remove leftover title for new season
for example https://myanimelist.net/anime/31339/Drifters/episode
what i want to be in this api?
so, my data is returning undefined when i put like data.title and etc
title: '\n Top\n ',
synopsis: '',
picture: undefined,
Although I'm really enjoying this API for my little project there is a tiny issue that causes a lot of stuff to go slower,
in the search function the option maxResults isn't functional. No matter what is put there (1, 250, 40) it will always return 50.
const malScraper = require('mal-scraper')
const search = malScraper.search
var anime = search.search('anime', {
maxResults: 100,
term: argument
})
.catch(console.error)
.then((results) => console.log(results.length))
always returns 50. no matter what you set maxResults
to.
Some data in the anime JSON returns incorrect
some examples are "title" that return "Top"
and picture, that returns undefined
malScraper.getRecommendationsList({ name: 'Sakura Trick', id: 20047 })
has the same result as
malScraper.getRecommendationsList({ name: 'anything', id: 20047 })
Looks like MAL doesn't care about the name. Ex https://myanimelist.net/anime/73/anything/userrecs works.
getRecommendationsList should be callable with just an id, no need to provide a name.
Hello,
I have been interested by your scrapper for MAL but by running your test, I am facing this problem :
npm test
Timed out while running tests
After multiple try, I manage to see that it always appends on the same part of the code. By example, for the test checking that Naruto has 500 episode, you have a 5 recursive calls. You get all the pages one after the other. Same goes for Drifters. But Ava has a timeout error trigger if the connection is a bit too long.
Hi, I found an error in the scoreStats's data : the number of users if printed twice.
For example : Dragon Ball Z is scored by 488778488,778 users. (We have 488,778 printed twice)
and it's the same for all animes.
I put some screenshots for you to see the issue.
I hope you will fix that quickly
Sincery yours
-LePtitMetalleux
It doesn't look like the Japanese name for the field is currently supported for Japanese names, as seen here: https://myanimelist.net/manga/23390/Shingeki_no_Kyojin
進撃の巨人
Is it possible to add this data? Is it possible to search by Japanese title?
Is there any way to add Nsfw / safe word filter to this
malScraper.getInfoFromName()
When i use
.getinfofromname('Anohana')
If returns an irrevelant anime but when i use
.getresultsfromsearch('Anohana')
It returns exactly what i want in the first item in the array.. it's not only anohana, it's the same with many other animes as well. What do you think about it?
Shall we use getresultsfromsearch and then pick it's first name or url and use that value to search using getinfofromname or getinfofromurl?
Example: Action, Adventure, Comedy, Super Power, Martial Arts, Shounen are parsed as
[
'ActionAction',
' AdventureAdventure',
' ComedyComedy',
' Super PowerSuper Power',
' Martial ArtsMartial Arts',
' ShounenShounen'
]
If I add in the search parameters the key "type", it returns an incorrect result, example:
✔️Correct
Code:
var malScraper = require("mal-scraper")
malScraper.search.search("anime", {
"term": "BanG Dream! 3rd Season"
}).then((data) => console.log(data[0])).
catch((err) => err);
Result:
{
thumbnail: 'https://cdn.myanimelist.net/r/100x140/images/anime/1985/104767.jpg?s=a0e163e0c3ee85136e20d9ed15490b99',
url: 'https://myanimelist.net/anime/37870/BanG_Dream_3rd_Season',
video: 'https://myanimelist.net/anime/37870/BanG_Dream_3rd_Season/video',
shortDescription: 'As the all-girl band Poppin’Party continues to rise in fame, they find a poster for an event at the legendary Nippon Budoukan. Now determined to participate in the "BanG
Dream," Kasumi Toyama and the...read more.',
title: 'BanG Dream! 3rd Season',
type: 'TV',
nbEps: '13',
score: '8.09',
startDate: '01-23-20',
endDate: '04-23-20',
members: '13,238',
rating: 'PG-13'
}
❌ Error
Code:
var malScraper = require("mal-scraper")
malScraper.search.search("anime", {
"term": "BanG Dream! 3rd Season",
"type": 1
}).then((data) => console.log(data[0])).
catch((err) => err);
Result:
{
thumbnail: 'https://cdn.myanimelist.net/r/100x140/images/anime/4/84177.jpg?s=a7c4462cf6f5458a8b28496a75863876',
url: 'https://myanimelist.net/anime/25777/Shingeki_no_Kyojin_Season_2',
video: 'https://myanimelist.net/anime/25777/Shingeki_no_Kyojin_Season_2/video',
shortDescription: 'For centuries, humanity has been hunted by giant, mysterious predators known as the Titans. Three mighty walls—Wall Maria, Rose, and Sheena—provided peace and protection
for humanity for over a hundre...read more.',
title: 'Shingeki no Kyojin Season 2',
type: 'TV',
nbEps: '12',
score: '8.40',
startDate: '04-01-17',
endDate: '06-17-17',
members: '1,082,750',
rating: 'R'
}
XMLHttpRequest cannot load https://myanimelist.net/search/prefix.json?type=anime&keyword=naruto. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:9080' is therefore not allowed access.
mal-scraper 2.2.1
electron 1.7.5
This issue addresses 2 bugs i ran into with the getWatchListFromUser() method :
First:
TypeError: Cannot read property '0' of undefined at parseString (D:\Felix\chino\indev\node_modules\mal-scraper\src\watchList.js:76:38) at Parser.<anonymous> (D:\Felix\chino\indev\node_modules\xml2js\lib\parser.js:303:18) at emitOne (events.js:115:13) at Parser.emit (events.js:210:7) at SAXParser.onclosetag (D:\Felix\chino\indev\node_modules\xml2js\lib\parser.js:261:26) at emit (D:\Felix\chino\indev\node_modules\sax\lib\sax.js:624:35) at emitNode (D:\Felix\chino\indev\node_modules\sax\lib\sax.js:629:5) at closeTag (D:\Felix\chino\indev\node_modules\sax\lib\sax.js:889:7) at SAXParser.write (D:\Felix\chino\indev\node_modules\sax\lib\sax.js:1436:13) at Parser.exports.Parser.Parser.parseString (D:\Felix\chino\indev\node_modules\xml2js\lib\parser.js:322:31)
Line 76 of the watchList file, when it resolves the promise, it tries reading the first element
of mal.myinfo (which i guess is supposed to be an array), however mal.myinfo doesn't seem to be
defined for all users.
By that i mean that it doesn't work with some users, and work with some others. For reference:
The bug happen with the user Paradoxalcorp (https://myanimelist.net/profile/Paradoxcorp)
Works as expected with the user niputi (https://myanimelist.net/profile/niputi)
Second:
Within the stats object that is in the object returned by getWatchListFromUser(), the property name of the property containing the username is undefined instead of username or name (see the screenshot)
maybe I'm missing something, I wanted to know if there is any way we can find character info using their name just like we do to find anime malScraper.getInfoFromName(query)
if there is no way currently see if you can add that feature.
I think there's no getAllAnimes function or anything similar that will get all animes that's in the API.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.