Wait's forever for cookie about soaper-dl

[jones2126]

I had the same issue and with the various debug steps above I think I worked out the correct steps for me to get this work. Posting here in case it helps someone else. Happy to be corrected.

Make sure https://soap2day.ac/ is accessable in your browser

$ sudo apt-get install jq.
$ sudo snap install go --classic
$ export GOBIN=/usr/local/bin
$ alias goSudo='\sudo -E go'
$ go install github.com/ericchiang/pup@latest
$ which pup   # statement not required; returned /usr/local/bin/pup
$ sudo apt install fzf
$ fzf –version  # statement not required; returns 0.20.0
$ sudo apt update
$ sudo apt install chromium-browser
$ sudo apt install git
$ git clone https://github.com/KevCui/soap2day-dl.git
$ cd ~/soap2day-dl/bin
$ npm i puppeteer-core puppeteer-extra puppeteer-extra-plugin-stealth
$ cd ~/soap2day-dl
$ ./soap2day-dl.sh -n 'game of'


possible debug steps
$ ./soap2day-dl.sh -d 'game of'
delete ~/soap2day-dl/user-agent
$ sudo nano  ~/soap2day-dl/bin/getCookie.js  #line 12:  change headless: true to  headless: false. 

@KevCui Thank you for sharing your work.

[KevCui]

Hey @EricMarcantonio, it might be caused by the invalid user agent. Could you please try to remove user-agent file in the script directory and run the script again? It will generate a new file with new user agent inside it. If it's still not working, please send me the generated user agent value and I will take another look.

[valtov]

I tried running the script on Ubuntu server with headless chrome

Mozilla/5.0 (X11; CrOS aarch64 13816.64.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4237.100 Safari/537.36

Same issue as above, although I'm guessing for headless the issue can't be helped

[KevCui]

Hey @valtov, thanks for the info. I tried the same user agent in your comment and it appears to be working for me.

Could you please check if you are able to visit the site https://soap2day.ac and access to any video content from Chrome browser on your Ubuntu machine? If yes, then please try to edit bin/getCookie.js file line 12: headless: true to headless: false. Save the change and run the script, it will start Chrome without headless and the cookies should be fetched.

[valtov]

Like I said I'm on Ubuntu server, all I see is a command line. I ran this
google-chrome-stable --headless --disable-gpu --screenshot --run-all-compositor-stages-before-draw --virtual-time-budget=10000 https://soap2day.ac
to see what's happening and its asking to click to 'verify human' which I don't think I can do. I'm guessing the only way forward (assuming I want it to work from my server) is to steal the cookies from my regular browser and paste them in somehow in your script.

[KevCui]

Hey @valtov, the script is using Puppeteer to run headless Chrome, and it should work on Ubuntu server as well, even there is only CLI.

You could run this command bin/getCookie.js "$(which google-chrome-stable)" https://soap2day.ac 'Mozilla/5.0 (X11; CrOS aarch64 13816.64.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4237.100 Safari/537.36' to see if it returns an error when fetching cookie.

If you'd like to investigate further, you could add await page.screenshot({ path: 'screenshot.png' }); in bin/getCookie.js file to capture a screenshot and see what's blocking the process.

[azeem39]

I am having the same issue on ubunut VPS and after i run "bin/getCookie.js "$(which google-chrome-stable)" https://soap2day.ac 'Mozilla/5.0 (X11; CrOS aarch64 13816.64.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4237.100 Safari/537.36'" I gee this :

internal/modules/cjs/loader.js:638
throw err;
^

Error: Cannot find module 'puppeteer-extra'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
at Function.Module._load (internal/modules/cjs/loader.js:562:25)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Object. (/home/soap2day-dl/bin/getCookie.js:4:19)
at Module._compile (internal/modules/cjs/loader.js:778:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)

[KevCui]

Hey @azeem39, from the error message in your comment Cannot find module 'puppeteer-extra', you are having a different issue. Please make sure the required npm packages are installed on your VPS: cd bin; npm i puppeteer-extra puppeteer-extra-plugin-stealth

[azeem39]

Thank you for that, when i run ./soap2day-dl.sh -p /tv_aTo2OTs.html -e 1.1 -d
i get this

• getopts :hlsdn❌p:e: opt
• set_var
  ++ command -v curl
• _CURL=/usr/bin/curl
  ++ command -v jq
• _JQ=/usr/bin/jq
  ++ command -v pup
• _PUP=/usr/local/bin/pup
  ++ command -v fzf
• _FZF=/usr/bin/fzf
  ++ command -v chromium
• _CHROME=/snap/bin/chromium
• _HOST=https://soap2day.ac
• _SEARCH_URL=https://soap2day.ac/search/keyword/
  +++ realpath ./soap2day-dl.sh
  ++ dirname /home/soap2day-dl/soap2day-dl.sh
• _SCRIPT_PATH=/home/soap2day-dl
• _SEARCH_LIST_FILE=/home/soap2day-dl/search.list
• _SOURCE_FILE=.source.html
• _EPISODE_LINK_LIST=.episode.link
• _EPISODE_TITLE_LIST=.episode.title
• _MEDIA_HTML=.media.html
• _SUBTITLE_LANG=English
• _GET_RESPONSE_JS=/home/soap2day-dl/bin/getResponse.js
• _FETCH_FILE_JS=/home/soap2day-dl/bin/fetchFile.js
• _COOKIE_FILE=/home/soap2day-dl/cookie.json
• _USER_AGENT_FILE=/home/soap2day-dl/user-agent
• _GET_COOKIE_JS=/home/soap2day-dl/bin/getCookie.js
• [[ -s /home/soap2day-dl/user-agent ]]
  ++ cat /home/soap2day-dl/user-agent
• _USER_AGENT='Mozilla/5.0 (X11; CrOS aarch64 13816.64.0) AppleWebKit/537.36 (KHTML, lik e Gecko) Chrome/92.0.4393.100 Safari/537.36'
  ++ get_cookie
  +++ is_file_expired /home/soap2day-dl/cookie.json 55
  +++ local o
  +++ o=yes
  +++ [[ -f /home/soap2day-dl/cookie.json ]]
  +++ echo yes
  ++ [[ yes == \y\e\s ]]
  ++ local cookie

[cyber-hunt7]

Good day, If I may ask where do you get this or how do you get this. aTo2OTs.html From: azeem39 ***@***.***> Sent: Friday, August 19, 2022 6:58 PM To: KevCui/soap2day-dl ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [KevCui/soap2day-dl] Wait's forever for cookie (Issue #22) Thank you for that, when i run ./soap2day-dl.sh -p /tv_aTo2OTs.html -e 1.1 -d i get this * getopts :hlsdn❌p:e: opt * set_var ++ command -v curl * _CURL=/usr/bin/curl ++ command -v jq * _JQ=/usr/bin/jq ++ command -v pup * _PUP=/usr/local/bin/pup ++ command -v fzf * _FZF=/usr/bin/fzf ++ command -v chromium * _CHROME=/snap/bin/chromium * _HOST=https://soap2day.ac * _SEARCH_URL=https://soap2day.ac/search/keyword/ +++ realpath ./soap2day-dl.sh ++ dirname /home/soap2day-dl/soap2day-dl.sh * _SCRIPT_PATH=/home/soap2day-dl * _SEARCH_LIST_FILE=/home/soap2day-dl/search.list * _SOURCE_FILE=.source.html * _EPISODE_LINK_LIST=.episode.link * _EPISODE_TITLE_LIST=.episode.title * _MEDIA_HTML=.media.html * _SUBTITLE_LANG=English * _GET_RESPONSE_JS=/home/soap2day-dl/bin/getResponse.js * _FETCH_FILE_JS=/home/soap2day-dl/bin/fetchFile.js * _COOKIE_FILE=/home/soap2day-dl/cookie.json * _USER_AGENT_FILE=/home/soap2day-dl/user-agent * _GET_COOKIE_JS=/home/soap2day-dl/bin/getCookie.js * [[ -s /home/soap2day-dl/user-agent ]] ++ cat /home/soap2day-dl/user-agent * _USER_AGENT='Mozilla/5.0 (X11; CrOS aarch64 13816.64.0) AppleWebKit/537.36 (KHTML, lik e Gecko) Chrome/92.0.4393.100 Safari/537.36' ++ get_cookie +++ is_file_expired /home/soap2day-dl/cookie.json 55 +++ local o +++ o=yes +++ [[ -f /home/soap2day-dl/cookie.json ]] +++ echo yes ++ [[ yes == \y\e\s ]] ++ local cookie — Reply to this email directly, view it on GitHub <#22 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIPD3QILP6XIHBN7AJUSDNTVZ64JDANCNFSM55IGIBDA> . You are receiving this because you are subscribed to this thread. <https://github.com/notifications/beacon/AIPD3QNVMX3XKIHFIIHI7ILVZ64JDA5CNFSM55IGIBDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOJDCVKOY.gif> Message ID: ***@***.*** ***@***.***> >

[azeem39]

that's the URL of the TV-series - i copied from the example section

[cyber-hunt7]

Is there a way you can showcase a demo of a series. Any series From: azeem39 ***@***.***> Sent: Friday, August 19, 2022 7:35 PM To: KevCui/soap2day-dl ***@***.***> Cc: Cybersecghost ***@***.***>; Comment ***@***.***> Subject: Re: [KevCui/soap2day-dl] Wait's forever for cookie (Issue #22) that's the URL of the TV-series - i copied from the example section — Reply to this email directly, view it on GitHub <#22 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIPD3QO5UBU2AEXA7YVLIH3VZ7AV7ANCNFSM55IGIBDA> . You are receiving this because you commented. <https://github.com/notifications/beacon/AIPD3QM2DABW5CGEXGEXVUTVZ7AV7A5CNFSM55IGIBDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOJDC5C2I.gif> Message ID: ***@***.*** ***@***.***> >

[KevCui]

Thank you for that, when i run ./soap2day-dl.sh -p /tv_aTo2OTs.html -e 1.1 -d i get this

• getopts :hlsdnxp:e: opt
• set_var
  ++ command -v curl
• _CURL=/usr/bin/curl
  ++ command -v jq
• _JQ=/usr/bin/jq
  ++ command -v pup
• _PUP=/usr/local/bin/pup
  ++ command -v fzf
• _FZF=/usr/bin/fzf
  ++ command -v chromium
• _CHROME=/snap/bin/chromium
• _HOST=https://soap2day.ac
• _SEARCH_URL=https://soap2day.ac/search/keyword/
  +++ realpath ./soap2day-dl.sh
  ++ dirname /home/soap2day-dl/soap2day-dl.sh
• _SCRIPT_PATH=/home/soap2day-dl
• _SEARCH_LIST_FILE=/home/soap2day-dl/search.list
• _SOURCE_FILE=.source.html
• _EPISODE_LINK_LIST=.episode.link
• _EPISODE_TITLE_LIST=.episode.title
• _MEDIA_HTML=.media.html
• _SUBTITLE_LANG=English
• _GET_RESPONSE_JS=/home/soap2day-dl/bin/getResponse.js
• _FETCH_FILE_JS=/home/soap2day-dl/bin/fetchFile.js
• _COOKIE_FILE=/home/soap2day-dl/cookie.json
• _USER_AGENT_FILE=/home/soap2day-dl/user-agent
• _GET_COOKIE_JS=/home/soap2day-dl/bin/getCookie.js
• [[ -s /home/soap2day-dl/user-agent ]]
  ++ cat /home/soap2day-dl/user-agent
• _USER_AGENT='Mozilla/5.0 (X11; CrOS aarch64 13816.64.0) AppleWebKit/537.36 (KHTML, lik e Gecko) Chrome/92.0.4393.100 Safari/537.36'
  ++ get_cookie
  +++ is_file_expired /home/soap2day-dl/cookie.json 55
  +++ local o
  +++ o=yes
  +++ [[ -f /home/soap2day-dl/cookie.json ]]
  +++ echo yes
  ++ [[ yes == \y\e\s ]]
  ++ local cookie

Hey @azeem39, unfortunately, the posted log appears to be incomplete. The log doesn't show the script bin/getCookie.js execution part. I cannot know what went wrong on your machine. If you have more logs to share, I can take another look for sure.

[WolfgangDpunkt]

Same here...I tried every solution advised. but always getting only endless lines of "[INFO] Wait a few seconds for fetching cookie..."
(Debian/Raspberry PI OS Lite 64-bit)

when i run ./soap2day-dl.sh -p /tv_aTo2OTs.html -e 1.1 -d i get this:

[INFO] Wait a few seconds for fetching cookie...
+++ /root/soap2day/soap2day-dl/bin/getCookie.js /usr/bin/chromium https://soap2day.ac 'Mozilla/5.0 (X11; CrOS aarch64 13816.64.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4406.100 Safari/537.36'
++ cookie=
++ [[ -z '' ]]
++ get_cookie
+++ is_file_expired /root/soap2day/soap2day-dl/cookie.json 55
+++ local o
+++ o=yes
+++ [[ -f /root/soap2day/soap2day-dl/cookie.json ]]
+++ echo yes
++ [[ yes == \y\e\s ]]
++ local cookie
++ print_info 'Wait a few seconds for fetching cookie...'
++ printf '%b\n' '\033[32m[INFO]\033[0m Wait a few seconds for fetching cookie...'
[INFO] Wait a few seconds for fetching cookie...
+++ /root/soap2day/soap2day-dl/bin/getCookie.js /usr/bin/chromium https://soap2day.ac 'Mozilla/5.0 (X11; CrOS aarch64 13816.64.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4406.100 Safari/537.36'
++ cookie=
++ [[ -z '' ]]
++ get_cookie
+++ is_file_expired /root/soap2day/soap2day-dl/cookie.json 55
+++ local o
+++ o=yes
+++ [[ -f /root/soap2day/soap2day-dl/cookie.json ]]
+++ echo yes
++ [[ yes == \y\e\s ]]
++ local cookie

[KevCui]

Hey @WolfgangDpunkt, could you please run /root/soap2day/soap2day-dl/bin/getCookie.js /usr/bin/chromium https://soap2day.ac 'Mozilla/5.0 (X11; CrOS aarch64 13816.64.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4406.100 Safari/537.36' and paste the logs? I need those logs for further investigations. Thanks!

[WolfgangDpunkt]

Thank you for your help!
I have entered the command, as you told.

First realisation, apparently puppeteer wasn't installed properly after all (actually I use for other purposes already, but ok).

So after running "npm i puppeteer", I tried your command again. This time I got an error message that it was not possible to execute this command as root user.

Well, then I tried as a normal user, now I could run the cookie thing without any problems. one step achieved. but when I tried to continue using the script itself now (./soap2day-dl.sh -n 'wire') , new permission errors came up , I guess it now needs root rights again for further steps.

This looks like it needs a long trial and error, so sorry, but I give up and I'll just use a browser extension for downloading.

But thanks for trying to solve the error and for your work.

[KevCui]

Hey @WolfgangDpunkt, 👌, no problem of choosing the tool you prefer the most.

[DutchCoder420]

same issue doesn't stop fetching cookie and all packages have been installed properly. used different user-agents but don't think that's the issue

[MagnusStrom]

Hello, I had this issue and installing puppeteer fixed it. However, Now it says media not found..

[MagnusStrom]

LOL, running .\soap2day-dl.sh doesn't work, but running soap2day-dl.sh does. Windows terimal is great.
Thank you for continuing to maintain this project!

[KevCui]

Hey @EricMarcantonio, @valtov, @azeem39, @DutchCoder420, I just made some changes to show the JS errors in the debug logs. If the same issue still happens, could you please pull the latest commit and run the command with -d flag? Please share the logs here and I will investigate further. Thanks!

[KevCui]

Hi all, I didn't receive any further debug logs after 3 months. Unfortunately I can't investigate the reported issue without debug logs. Therefore, I'm closing this issue for now.

If you have any questions, please feel free to leave comments below or open new issues.

Thanks @jones2126 for sharing the well-written debug steps 👍