paulkinlan / domcurl Goto Github PK

cUrl-like utility for fetching a resource (in this case we will run JS and return after network is idle) - great for JS heavy apps

Home Page: https://www.npmjs.com/package/domcurl

License: Apache License 2.0

JavaScript 100.00%

pupeteer curl

domcurl's Introduction

domcurl

cUrl-like utility for fetching a resource (in this case we will run JS and return after network is idle) - great for JS heavy apps.

Installation

npm i domcurl

Usage

Basic usage

domcurl [url]

domcurl --url https://example.com

<!DOCTYPE html><html><head>
    <title>Example Domain</title>

    <meta charset="utf-8">
    <meta http-equiv="Content-type" content="text/html; charset=utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <style type="text/css">
    body {
        background-color: #f0f0f2;
        margin: 0;
        padding: 0;
        font-family: "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
        
    }
    div {
        width: 600px;
        margin: 5em auto;
        padding: 50px;
        background-color: #fff;
        border-radius: 1em;
    }
    a:link, a:visited {
        color: #38488f;
        text-decoration: none;
    }
    @media (max-width: 700px) {
        body {
            background-color: #fff;
        }
        div {
            width: auto;
            margin: 0 auto;
            border-radius: 0;
            padding: 1em;
        }
    }
    </style>    
</head>

<body>
<div>
    <h1>Example Domain</h1>
    <p>This domain is established to be used for illustrative examples in documents. You may use this
    domain in examples without prior coordination or asking for permission.</p>
    <p><a href="http://www.iana.org/domains/example">More information...</a></p>
</div>


</body></html>

Verbose output

Renders more details about the request and the response.

domcurl -v [url]

> GET / 
> Host: example.com
> upgrade-insecure-requests: 1
> user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/66.0.3347.0 Safari/537.36
< date: Wed, 28 Feb 2018 03:04:47 GMT
< content-encoding: gzip
< last-modified: Fri, 09 Aug 2013 23:54:35 GMT
< server: ECS (oxr/837E)
< etag: "1541025663+gzip"
< vary: Accept-Encoding
< x-cache: HIT
< content-type: text/html
< status: 200
< cache-control: max-age=604800
< content-length: 606
< expires: Wed, 07 Mar 2018 03:04:47 GMT
<!DOCTYPE html><html><head>
    <title>Example Domain</title>

    <meta charset="utf-8">
    <meta http-equiv="Content-type" content="text/html; charset=utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <style type="text/css">
    body {
        background-color: #f0f0f2;
        margin: 0;
        padding: 0;
        font-family: "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
        
    }
    div {
        width: 600px;
        margin: 5em auto;
        padding: 50px;
        background-color: #fff;
        border-radius: 1em;
    }
    a:link, a:visited {
        color: #38488f;
        text-decoration: none;
    }
    @media (max-width: 700px) {
        body {
            background-color: #fff;
        }
        div {
            width: auto;
            margin: 0 auto;
            border-radius: 0;
            padding: 1em;
        }
    }
    </style>    
</head>

<body>
<div>
    <h1>Example Domain</h1>
    <p>This domain is established to be used for illustrative examples in documents. You may use this
    domain in examples without prior coordination or asking for permission.</p>
    <p><a href="http://www.iana.org/domains/example">More information...</a></p>
</div>


</body></html>

Send output to a file

domcurl --url https://example.com -o test.txt

domcurl --url https://example.com --output test.txt

Set a custom header

domcurl --url https://example.com -H 'x-test:test1' -H 'x-test2:http://test.com'

Set a cookie

Sets a cookie on the request. It must be a valid Cookie string.

domcurl [url] -b "test=hello; Domain=airhorner.com; HttpOnly;"

Unlike cUrl you can multiple cookies by appending more -b arguments

domcurl [url] -b "test=hello; Domain=airhorner.com; HttpOnly;" -b "hello=world; Domain=airhorner.com; HttpOnly;"

Specify a request timeout

By default the command will timeout after 30 seconds. You can specify how long the command will wait before it errors.

domcurl --url https://example.com -m 60

domcurl --url https://example.com --max-time 60

Specify a trace file

Output a Chrome DevTools trace file (including screenshots.)

domcurl --url https://example.com --trace test.json

Using as a module

npm i domcurl

` const {domcurl} = require('domcurl');

domcurl(https://paul.kinlan.me/, {}); `

domcurl's People

Contributors

Stargazers

Watchers

Forkers

daveek deawx markchipman shafiahmed w4fz5uck5 xbl3 5l1v3r1 tortoisedoc andreivolt 1989shack sanix-darker emtee40

domcurl's Issues

feat(Add rate limit.... )

--limit-rate

Specify the maximum transfer rate you want curl to use - for both downloads and uploads. This feature is useful if you have a limited pipe and you'd like your transfer not to use your entire bandwidth. To make it slower than it otherwise would be.

The given speed is measured in bytes/second, unless a suffix is appended. Appending 'k' or 'K' will count the number as kilobytes, 'm' or 'M' makes it megabytes, while 'g' or 'G' makes it gigabytes. Examples: 200K, 3m and 1G.

If you also use the -Y, --speed-limit option, that option will take precedence and might cripple the rate-limiting slightly, to help keeping the speed-limit logic working.

If this option is used several times, the last one will be used.

feat(add support for methods other than GET)

Add better error logging for invalid urls

feat(add advanced filename mapping to output file)

-o, --output

Follow on from #4

Write output to instead of stdout. If you are using {} or [] to fetch multiple documents, you can use '#' followed by a number in the specifier. That variable will be replaced with the current string for the URL being fetched. Like in:

curl http://{one,two}.example.com -o "file_#1.txt"

or use several variables like:

curl http://{site,host}.host[1-5].com -o "#1_#2"

You may use this option as many times as the number of URLs you have. For example, if you specify two URLs on the same command line, you can use it like this:

curl -o aa example.com -o bb example.net

and the order of the -o options and the URLs doesn't matter, just that the first -o is for the first URL and so on, so the above command line can also be written as

curl example.com example.net -o aa -o bb

See also the --create-dirs option to create the local directories dynamically. Specifying the output as '-' (a single dash) will force the output to be done to stdout.

See also -O, --remote-name and --remote-name-all and -J, --remote-header-name.

feat(add output file support)

-o, --output

Support for complex file name substitution will be added in #19

========

curl http://{one,two}.example.com -o "file_#1.txt"

or use several variables like:

curl http://{site,host}.host[1-5].com -o "#1_#2"

You may use this option as many times as the number of URLs you have. For example, if you specify two URLs on the same command line, you can use it like this:

curl -o aa example.com -o bb example.net

and the order of the -o options and the URLs doesn't matter, just that the first -o is for the first URL and so on, so the above command line can also be written as

curl example.com example.net -o aa -o bb

See also the --create-dirs option to create the local directories dynamically. Specifying the output as '-' (a single dash) will force the output to be done to stdout.

See also -O, --remote-name and --remote-name-all and -J, --remote-header-name.

feat(add --trace to get a trace)

Should allow user define wait selector

In case of cUrl is confirm that all element is loaded before return. domcurl should do it same

Solution
allow user to pass selector that user want to wait with --wait-selector -w

feat(Add Max time)

cUrl has a max-time attribute (in seconds). hook it up.

feat(add support for multiple urls)

cUrl lets you add curl more than one url at a time.

curl http://example.com https://paul.kinlan.me

It also means that we have to fix multiple filename support.

feat(add stderr output file support)

--stderr

Redirect all writes to stderr to the specified file instead. If the file name is a plain '-', it is instead written to stdout.

If this option is used several times, the last one will be used.

See also -v, --verbose and -s, --silent.

feat(let the user add cookies.)

-b, --cookie

(HTTP) Pass the data to the HTTP server in the Cookie header. It is supposedly the data previously received from the server in a "Set-Cookie:" line. The data should be in the format "NAME1=VALUE1; NAME2=VALUE2".

If no '=' symbol is used in the argument, it is instead treated as a filename to read previously stored cookie from. This option also activates the cookie engine which will make curl record incoming cookies, which may be handy if you're using this in combination with the -L, --location option or do multiple URL transfers on the same invoke.

The file format of the file to read cookies from should be plain HTTP headers (Set-Cookie style) or the Netscape/Mozilla cookie file format.

The file specified with -b, --cookie is only used as input. No cookies will be written to the file. To store cookies, use the -c, --cookie-jar option.

Exercise caution if you are using this option and multiple transfers may occur. If you use the NAME1=VALUE1; format, or in a file use the Set-Cookie format and don't specify a domain, then the cookie is sent for any domain (even after redirects are followed) and cannot be modified by a server-set cookie. If the cookie engine is enabled and a server sets a cookie of the same name then both will be sent on a future transfer to that server, likely not what you intended. To address these issues set a domain in Set-Cookie (doing that will include sub domains) or use the Netscape format.

If this option is used several times, the last one will be used.

Users very often want to both read cookies from a file and write updated cookies back to a file, so using both -b, --cookie and -c, --cookie-jar in the same command line is common.

Refactor the runner in to a module.

feat(Let the user set the user agent)

-A, --user-agent

(HTTP) Specify the User-Agent string to send to the HTTP server. To encode blanks in the string, surround the string with single quote marks. This can also be set with the -H, --header option of course.

If this option is used several times, the last one will be used.

feat(Let the user set the viewport)

This is not mapped to curl but is interesting for DOM based queries.

feat(add ability to set referer)

-e, --referer

(HTTP) Sends the "Referrer Page" information to the HTTP server. This can also be set with the -H, --header flag of course. When used with -L, --location you can append ";auto" to the -e, --referer URL to make curl automatically set the previous URL when it follows a Location: header. The ";auto" string can be used alone, even if you don't set an initial -e, --referer.

If this option is used several times, the last one will be used.

See also -A, --user-agent and -H, --header.

cant use like a Node module

Hi.

Cool utility, but i have several problems when i use this like a Node module

npm init -y
npm i -g domcurl
npm i -S domcurl
touch index_1.js 
touch index_2.js

Example 1:

in this example func call run immediately

node index_1.js

index_1.js

const domcurl = require('domcurl');
console.log(domcurl);

output:

--url or default value is not a valid URL
{}

Example 2:

this is example from doc

node index_2.js

index_2.js

const {domcurl} = require('domcurl');
domcurl(`https://paul.kinlan.me/`, {});

output:

TypeError: domcurl is not a function

Sorry, I pointed out two problems in one ticket.
Thank you for your attention!🙄

feat(add url parameter support)

--url

Specify a URL to fetch. This option is mostly handy when you want to specify URL(s) in a config file.

If the given URL is missing a scheme name (such as "http://" or "ftp://" etc) then curl will make a guess based on the host. If the outermost sub-domain name matches DICT, FTP, IMAP, LDAP, POP3 or SMTP then that protocol will be used, otherwise HTTP will be used. Since 7.45.0 guessing can be disabled by setting a default protocol, see --proto-default for details.

This option may be used any number of times. To control where this URL is written, use the -o, --output or the -O, --remote-name options.

feat(add ability to set header)

-H, --header header/@file

(HTTP) Extra header to include in the request when sending HTTP to a server. You may specify any number of extra headers. Note that if you should add a custom header that has the same name as one of the internal ones curl would use, your externally set header will be used instead of the internal one. This allows you to make even trickier stuff than curl would normally do. You should not replace internally set headers without knowing perfectly well what you're doing. Remove an internal header by giving a replacement without content on the right side of the colon, as in: -H "Host:". If you send the custom header with no-value then its header must be terminated with a semicolon, such as -H "X-Custom-Header;" to send "X-Custom-Header:".

curl will make sure that each header you add/replace is sent with the proper end-of-line marker, you should thus not add that as a part of the header content: do not add newlines or carriage returns, they will only mess things up for you.

Starting in 7.55.0, this option can take an argument in @filename style, which then adds a header for each line in the input file. Using @- will make curl read the header file from stdin.

See also the -A, --user-agent and -e, --referer options.

Starting in 7.37.0, you need --proxy-header to send custom headers intended for a proxy.

Example:

curl -H "X-First-Name: Joe" http://example.com/

WARNING: headers set with this option will be set in all requests - even after redirects are followed, like when told with -L, --location. This can lead to the header being sent to other hosts than the original host, so sensitive headers should be used with caution combined with following redirects.

This option can be used multiple times to add/replace/remove multiple headers.

Getting the request headers is slow because we are waiting for the response

The puppeteer API doesn't make it easy to get the request that was issued for a 'goto' command.

feat(Allow the user to specifiy when they want the dom to be returned)

Follow the puppeteer API: 'load', 'domcontentloaded', 'networkidle0', 'networkidle1'

paulkinlan / domcurl Goto Github PK

domcurl's Introduction

domcurl

Installation

Usage

Basic usage

Verbose output

Send output to a file

Set a custom header

Set a cookie

Specify a request timeout

Specify a trace file

Using as a module

domcurl's People

Contributors

Stargazers

Watchers

Forkers

domcurl's Issues

Example 1:

Example 2:

Recommend Projects

Recommend Topics

Recommend Org