How to implement Firefox Save As Complete Website? about automate-save-page-as

[abiyani]

What do you mean by "Save as complete website" ? Do you mean saving all webpages within a website ? This tool is not meant for that task, it just saves a single web page. You will have to crawl all the links and call this tool repeatedly on them yourself.

An example of such a tool is available in another repo of mine: https://github.com/abiyani/orkut-community-downloader - which is a tool I created to save all pages on my Orkut profile (and internally uses automate-save-page-as).

[bhuether]

Yeah, I am trying to write a script in PHP where I am using curl to login to a website, then get a page that has a table of contents, where each link takes me to a page that I want to then download using Firefox Save As Website Complete.

I am on a MAC, and I think the X11 setup is just proving to be too difficult for me to overcome. I can get the script to run and open Firefox, but after that it doesn't see a display, despite me using XQuartz on MAC.

So now I am looking into Firefox automation scripts, but so far nothing is recording the save as actions.

Thanks, Brian

[abiyani]

Yes I have never tested the script on Mac, and don't expect it to work either (due to the way it is intrinsically tied to X).

Btw, this tools was written before Chrome headless was available, so you should look into that for automating the task instead: https://developers.google.com/web/updates/2017/04/headless-chrome

[bhuether]

Hi, Funny you mention it. I just started looking into headless browsers today. But only Firefox Save As gets me all the needed resources. And need to figure out how to do Save As Website Complete using headless fFirefox browser but haven't seen yet how to do it. So this is quite the adventure in trying to figure things out! thanks, Brian

…

On Sun, Dec 16, 2018 at 1:20 AM Anurag Biyani ***@***.***> wrote: Yes I have never tested the script on Mac, and don't expect it to work either (due to the way it is intrinsically tied to X). Btw, this tools was written before Chrome headless was available, so you should look into that for automating the task instead: https://developers.google.com/web/updates/2017/04/headless-chrome — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#25 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AqqNYaSScIK06r7l20GZbQbbstlWZIqGks5u5XWrgaJpZM4ZUGNn> .

[bhuether]

Not sure about chrome headless, but I don't see anything in the documentation for Firefox headless about a saveas method. Strangely, it is only Firefox saveas that extracts videos from HTML5 elements from the site I am saving. Chrome, various extensions, no matter what settings, don't get the video.

…

On Dec 15, 2018 11:20 PM, "Anurag Biyani" ***@***.***> wrote: Yes I have never tested the script on Mac, and don't expect it to work either (due to the way it is intrinsically tied to X). Btw, this tools was written before Chrome headless was available, so you should look into that for automating the task instead: https://developers.google.com/web/updates/2017/04/headless-chrome — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#25 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AqqNYaSScIK06r7l20GZbQbbstlWZIqGks5u5XWrgaJpZM4ZUGNn> .