drawrowfly / tiktok-scraper Goto Github PK

I keep getting this error when running the package and it stopped working for some reason. Reading from here that "node-uuid" dependency should be swapped out for "uuid"
jotform/jotform-api-nodejs#20

Also, is the trending function call supposed to be random? I'm just downloading 3 videos and download the same videos 2 weeks ago and today.

Command does not do anything. Am I correctly inserting the music id?

URL: https://www.tiktok.com/music/The-Box-6768646921029470981?lang=en

Tried
tiktok-scraper music The-Box-6768646921029470981 -n 50 -d

tiktok-scraper music 6768646921029470981 -n 50 -d

tiktok-scraper music The-Box -n 50 -d

none is downloading

this is the error:

the command:
tiktok-scraper hashtag tiktokchefs -d -n 30 -store

C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper\lib\helpers\multipleBar.js:42
        this.stream.clearLine();
                    ^

TypeError: this.stream.clearLine is not a function
    at Multibar.terminate (C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper\lib\help                                                                                           ers\multipleBar.js:42:21)
    at ProgressBar.bar.terminate (C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper\l                                                                                           ib\helpers\multipleBar.js:33:22)
    at ProgressBar.tick (C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper\node_modul                                                                                           es\progress\lib\node-progress.js:103:10)
    at Multibar.tick (C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper\lib\helpers\m                                                                                           ultipleBar.js:56:17)
    at ProgressBar.bar.tick (C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper\lib\he                                                                                           lpers\multipleBar.js:28:18)
    at Request.r.get.on.on.chunk (C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper\l                                                                                           ib\index.js:117:34)
    at Request.emit (events.js:198:13)
    at IncomingMessage.<anonymous> (C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper                                                                                           \node_modules\request\request.js:1073:12)
    at IncomingMessage.emit (events.js:198:13)
    at addChunk (_stream_readable.js:288:12)
    at readableAddChunk (_stream_readable.js:269:11)
    at IncomingMessage.Readable.push (_stream_readable.js:224:10)
    at HTTPParser.parserOnBody (_http_common.js:122:22)
    at TLSSocket.socketOnData (_http_client.js:442:20)
    at TLSSocket.emit (events.js:198:13)
    at addChunk (_stream_readable.js:288:12)
    at readableAddChunk (_stream_readable.js:269:11)
    at TLSSocket.Readable.push (_stream_readable.js:224:10)
    at TLSWrap.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)

It downloads a file(a corrupt zip file) with one video in it, but just only that.

can you update the signature? seems like not working again.

When using the module example, the collector returns an empty array.
Module not working:

(async () => {
    try {
        const posts = await TikTokScraper.user({ USER_ID: SOME_USER_ID_STRING }, { number: 20, by_user_id: true });
        console.log(posts); // { collector: [] }
    } catch (error) {
        console.log(error);
    }
})();

CLI works:

tiktok-scraper user lebeats_211

Any idea what could be causing the difference?

Suggestion

I'm getting following error:

Error: Data should not be empty or the "fields" option should be included
    at JSON2CSVParser.preprocessData (/Users/ehsan/.config/yarn/global/node_modules/json2csv/dist/json2csv.cjs.js:1484:15)
    at JSON2CSVParser.parse (/Users/ehsan/.config/yarn/global/node_modules/json2csv/dist/json2csv.cjs.js:1450:32)
    at Bluebird.fromCallback.cb (/Users/ehsan/.config/yarn/global/node_modules/tiktok-scraper/lib/index.js:225:102)
    at tryCatcher (/Users/ehsan/.config/yarn/global/node_modules/Bluebird/js/release/util.js:16:23)
    at Function.Promise.fromNode.Promise.fromCallback (/Users/ehsan/.config/yarn/global/node_modules/Bluebird/js/release/promise.js:209:30)
    at Promise (/Users/ehsan/.config/yarn/global/node_modules/tiktok-scraper/lib/index.js:225:44)
    at processTicksAndRejections (internal/process/next_tick.js:81:5)

Hi there,
Can you please add number of followers in your api reponse.

just tried today and it doesn't work - doesn't even create cvs file for trending.
It works perfectly for other API - get hashtag info etc.

It seems that the scraper doesn't work anymore, I tried to debug it and I received status code 200 in the body response of video requests (not 0), maybe the signature or something else has changed in tiktok system? hope it help

I've got Nodejs and installed from NPM, run the example in terminal but didn't work. Am i missing something?

Code app.js:
const TikTokScraper = require('tiktok-scraper');

let posts = TikTokScraper.hashtag("singing", { event: true, number: 200});

posts.on('data', (json) => {
console.log(json)
})

posts.on('done', () => {
console.log("done")
})

posts.on('error', (error) => {
console.log(error)
})

posts.scrape();

Version:
"dependencies": {
"tiktok-scraper": "^1.0.9"
}

Error:
TypeError: Cannot read property 'itemListData' of undefined
at TikTokScraper.mainLoop (/node_modules/tiktok-scraper/lib/index.js:394:53)
at async TikTokScraper.scrape (/node_modules/tiktok-scraper/lib/index.js:297:9)

Hi @drawrowfly

Currently, your node module supports for searching by hashtag, user, trend, music. Do you know a way to search by keyword? Are you planing to add "search by keyword" to this module?

Best,
-Young

Is it possible to use authorName for the downloaded file?

Tried a few http proxy and it will always fail. (I suspect it might be related to https over http).

I saw that online tiktok downloaders allows to download videos without a watermark. Is it possible with this lib?

In terminal:
https://www.tiktok.com/@maevaa.ghennam/video/6805591629471173894
Bad url format. Correct format: https://www.tiktok.com/@USERNAME/video/ID
In code regex:
if (!/^https://www.tiktok.com/@(\w+)/video/(\d+)$/.test(this.input)) {
throw Bad url format. Correct format: https://www.tiktok.com/@USERNAME/video/ID;
}
Part w+ equals [a-zA-Z0-9]+

Hey there, when downloading posts, how does the program handle new posts for an already downloaded user? Ideally, I'd be able to just run the script again for a user and it would download said user's new posts and their metadata without having to redownload everything. Is this a possibility?

Apologies for opening an issue for this, I couldn't see a way to contact you, drawrowfly.

This isn't really an issue, I'm just not sure where to put questions.

Is it possible to easily delete stored download history per user?

For instance, I want to re-download a user's videos and store the download history.

How do I delete their old download history?

I'm using ubuntu. I can navigate to the tmp folder but I'm not seeing how to identify which history file belongs to which user.

Thanks

Thank you for providing such a useful tool, but when I used it, I found that the videos are watermarked. There are two suggestions to see if it can be achieved:

1. Can you put the video link without watermark in it?
2. Can you provide a query to get the video information based on the video id (including the video download link without watermark)

is it possible to download videos without a watermark?

When viewing TikTok videos via the mobile app, sometimes there is a

This sound isn't available.

message above the video. I noticed that these videos do not get scraped. Is it possible to include them?

IDK if it helps in figuring out the issue, but I've been able to share them to Instagram via the mobile app and download/hear the sound that way.

Love the scraper, thanks!

This morning I tried to scrape a profile and the program immediately exited with no other message after displaying the "TikTok Scraper Started" message.

I updated the program and repeatedly receive the same message. Could it be that my IP is banned/limited?

Thanks

Hello drawrowfly,
I want to integrate your scraper into php website, can you give me your email? Are you available for a custom paid project? Thank you!

i run this project in RDP AWS
but i getting result

collector[]

and i test _getUserId() but getting result {"statusCode":200,"contentType":"text/html","content":""}
how to fix that?

Please add support to scrape multiple users from a text file

Any chance to get user posts by user id ?

Is it possible to use authorName for the downloaded file?

Trending hashtags are a good way to seed the scraper. These hashtags appear to be different than the hashtags present in the trending posts. The app itself seems to have an infinite scroll of these trending hashtags, but the web interface only loads a fixed number.

https://www.tiktok.com/discover

A feature request, something to investigate when you have time. :)

Hi i tested today,
i cant scraping profile, result collector length is 0

I`m just curious should we be worried from this warning we see every time we use the scrapper:

(node:61341) ExperimentalWarning: The fs.promises API is experimental

Is it possible to get a user’s following list?
I’ve seen this on github:
https://github.com/botsphp/tiktok_crawler

And didn’t understand how to use it correctly

As of 9:50am CET, I can't download video's anymore. What worked the day before, doesn't work now, I'm affraid.

command use: tiktop-scraper user thisisanexampleuser -n 9999 -d -store

commands used to download by music association does not work, either.

Firstly, thanks for all your awesome work on this!

I'm using the latest version (0.0.16), so I do not think it's a duplicate of this issue #44

CLI works just fine. e.g.

tiktok-scraper user USERNAME -n 20 -d -store

Using the node method returns 0 results (was previously working). Any ideas?

const posts = await TikTokScraper.user(USER_ID, {
      by_user_id: true,
      user_data: true,
      number: 10,
    });

Thank you for such amazing library.
is there any way to get user profile info ? Like following/followers count ?

The result does not include the likes count of a post.

The scraper doesn't seem to work when the keyword (e.g. the hashtag I search for) is in Hebrew script. The word does show up in Hebrew alphabet in Terminal and I have Unicode (UTF-8) text encoding enabled, so I don't think the issue is with Terminal. Can you advise? Thanks!

can someone explain to me how to fix this?

unrecognized arguments: fyp -n 100 -d -t all

I see the following message when I try to run the scraper and it exits without downloading any data.
⠋ TikTok Scraper Started(node:19861) ExperimentalWarning: The fs.promises API is experimental

Have you bumped into this?

Script
const TikTokScraper = require('tiktok-scraper');

(async () => {
try {
const posts = await TikTokScraper.hashtag('singing', { number: 100 });
console.log(posts);
} catch (error) {
console.log(error);
}
})();

Version:
"tiktok-scraper": "^1.0.13"

Error:
...\node_modules\jsdom\lib\jsdom\living\helpers\dates-and-times.js:235
} catch {
^

SyntaxError: Unexpected token {
at createScript (vm.js:80:10)
at Object.runInThisContext (vm.js:139:10)
at Module._compile (module.js:607:28)
at Object.Module._extensions..js (module.js:654:10)
at Module.load (module.js:556:32)
at tryModuleLoad (module.js:499:12)
at Function.Module._load (module.js:491:3)
at Module.require (module.js:587:17)
at require (internal/module.js:11:18)
at Object. (...\node_modules\jsdom\lib\jsdom\living\helpers\form-controls.js:18:5)
at Module._compile (module.js:643:30)
at Object.Module._extensions..js (module.js:654:10)
at Module.load (module.js:556:32)
at tryModuleLoad (module.js:499:12)
at Function.Module._load (module.js:491:3)
at Module.require (module.js:587:17)
at require (internal/module.js:11:18)
at Object. (...\node_modules\jsdom\lib\jsdom\living\helpers\focusing.js:5:24)
at Module._compile (module.js:643:30)
at Object.Module._extensions..js (module.js:654:10)
at Module.load (module.js:556:32)
at tryModuleLoad (module.js:499:12)

_Even when setting -n 0, i can only scrape 20-30 videos depending on each users. Redownloading the same user would get the same number of videos.
_I did scraped different users during 24h window, here's just an example:
Username: chiyuu
Total videos: 446.
Downloaded: 28.
In older version i've got 415.
Current version 1.0.12

When runnning with nodejs v10.18.1 and installed with Yarn or npm, I receive this error when trying $tiktok-scraper --help

internal/modules/cjs/loader.js:638
throw err;
^

Error: Cannot find module 'Bluebird'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
at Function.Module._load (internal/modules/cjs/loader.js:562:25)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Object. (/home/me/.config/yarn/global/node_modules/tiktok-scraper/lib/index.js:13:18)
at Module._compile (internal/modules/cjs/loader.js:778:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Object. (/home/me/.config/yarn/global/node_modules/tiktok-scraper/lib/instance.js:3:23)
at Module._compile (internal/modules/cjs/loader.js:778:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Object. (/home/me/.config/yarn/global/node_modules/tiktok-scraper/bin/cli.js:4:23)
at Module._compile (internal/modules/cjs/loader.js:778:30)

Hello, Drawrowfly,
Is there a way to get post info by the post id?

Hello,

my setup :
Ubuntu 18.04
NodeJS : I tried version 12 and 13
tiktok scraper : 1.0.12

I keep getting an error message when I'm trying to use a proxy-server . Tried different proxy-servers.
Only appears when I'm trying to use a proxy.

tiktok-scraper trend -d -n 5 -p https://115.23.28.168:8080
Error: Error: [object Object]
at TikTokScraper.extractTac (/usr/local/lib/node_modules/tiktok-scraper/lib/index.js:311:19)

Scraping started returning an empty collector. I'm guessing the signature is out of data again.

(Btw, how are you reverse engineering the signature? Happy to help maintain if you point me in the right direction.)

Code
const TikTokScraper = require('tiktok-scraper');

let posts = TikTokScraper.hashtag("singing", { event: true, number: 10});

posts.on('data', (json) => {
console.log(json)
})

posts.on('done', () => {
console.log("done")
})

posts.on('error', (error) => {
console.log(error)
})

posts.scrape();

Version:
"tiktok-scraper": "^1.0.14

Error:
Error: RequestError: Error: ESOCKETTIMEDOUT
at TikTokScraper.scrapeData (*/node_modules/tiktok-scraper/lib/index.js:616:19)
done

hi, i have a idea/solution for downloading video without watermark
i sniffing in app tiktok in android and i found new api.
but i dont know how to get video id

example :
https://www.tiktok.com/@vikaberk/video/6721248223035411718 >> video id right
but how to convert to
v09044310000bl3ammlqa7lrlgrvj9tg

this is old api to get no wm https://api2.musical.ly/aweme/v1/play/?video_id=v09044310000bl3ammlqa7lrlgrvj9tg&line=0&ratio=720p&watermark=0&media_type=4&vr_type=0&test_cdn=None&improve_bitrate=0&logo_name=tiktok

but right now, not work again with videoid 6721248223035411718