Coder Social home page Coder Social logo

http-request-fromcurl's Introduction

Travis Build Status AppVeyor Build Status

NAME

HTTP::Request::FromCurl - create a HTTP::Request from a curl command line

SYNOPSIS

my $req = HTTP::Request::FromCurl->new(
    # Note - curl itself may not appear
    argv => ['https://example.com'],
);

my $req = HTTP::Request::FromCurl->new(
    command => 'https://example.com',
);

my $req = HTTP::Request::FromCurl->new(
    command_curl => 'curl -A mycurl/1.0 https://example.com',
);

my @requests = HTTP::Request::FromCurl->new(
    command_curl => 'curl -A mycurl/1.0 https://example.com https://www.example.com',
);
# Send the requests
for my $r (@requests) {
    $ua->request( $r->as_request )
}

RATIONALE

curl command lines are found everywhere in documentation. The Firefox developer tools can also copy network requests as curl command lines from the network panel. This module enables converting these to Perl code.

METHODS

->new

my $req = HTTP::Request::FromCurl->new(
    # Note - curl itself may not appear
    argv => ['--user-agent', 'myscript/1.0', 'https://example.com'],
);

my $req = HTTP::Request::FromCurl->new(
    # Note - curl itself may not appear
    command => '--user-agent myscript/1.0 https://example.com',
);

The constructor returns one or more HTTP::Request::CurlParameters objects that encapsulate the parameters. If the command generates multiple requests, they will be returned in list context. In scalar context, only the first request will be returned. Note that the order of URLs between --url and unadorned URLs will be changed in the sense that all unadorned URLs will be handled first.

my $req = HTTP::Request::FromCurl->new(
    command => '--data-binary @/etc/passwd https://example.com',
    read_files => 1,
);

Options

  • argv

    An arrayref of commands as could be given in @ARGV.

  • command

    A scalar in a command line, excluding the curl command

  • command_curl

    A scalar in a command line, including the curl command

  • read_files

    Do read in the content of files specified with (for example) --data=@/etc/passwd. The default is to not read the contents of files specified this way.

GLOBAL VARIABLES

%default_headers

Contains the default headers added to every request

@option_spec

Contains the Getopt::Long specification of the recognized command line parameters.

The following curl options are recognized but largely ignored:

  • --disable

  • --dump-header

  • --include

  • --location

  • --progress-bar

  • --show-error

  • --fail

  • --silent

  • --verbose

  • --junk-session-cookies

    If you want to keep session cookies between subsequent requests, you need to provide a cookie jar in your user agent.

  • --next

    Resetting the UA between requests is something you need to handle yourself

  • --parallel

  • --parallel-immediate

  • --parallel-max

    Parallel requests is something you need to handle in the UA

METHODS

->squash_uri( $uri )

my $uri = HTTP::Request::FromCurl->squash_uri(
    URI->new( 'https://example.com/foo/bar/..' )
);
# https://example.com/foo/

Helper method to clean up relative path elements from the URI the same way that curl does.

LIVE DEMO

https://corion.net/curl2lwp.psgi

KNOWN DIFFERENCES

Incompatible cookie jar formats

Until somebody writes a robust Netscape cookie file parser and proper loading and storage for HTTP::CookieJar, this module will not be able to load and save files in the format that Curl uses.

Loading/saving cookie jars is the job of the UA

You're expected to instruct your UA to load/save cookie jars:

use Path::Tiny;
use HTTP::CookieJar::LWP;

if( my $cookies = $r->cookie_jar ) {
    $ua->cookie_jar( HTTP::CookieJar::LWP->new()->load_cookies(
        path($cookies)->lines
    ));
};

Different Content-Length for POST requests

Different delimiter for form data

The delimiter is built by HTTP::Message, and curl uses a different mechanism to come up with a unique data delimiter. This results in differences in the raw body content and the Content-Length header.

MISSING FUNCTIONALITY

  • File uploads / content from files

    While file uploads and reading POST data from files are supported, the content is slurped into memory completely. This can be problematic for large files and little available memory.

  • Mixed data instances

    Multiple mixed instances of --data, --data-ascii, --data-raw, --data-binary or --data-raw are sorted by type first instead of getting concatenated in the order they appear on the command line. If the order is important to you, use one type only.

  • Multiple sets of parameters from the command line

    Curl supports the --next command line switch which resets parameters for the next URL.

    This is not (yet) supported.

SEE ALSO

LWP::Curl

LWP::Protocol::Net::Curl

LWP::CurlLog

HTTP::Request::AsCurl - for the inverse function

The module HTTP::Request::AsCurl likely also implements a much better version of ->as_curl than this module.

https://github.com/NickCarneiro/curlconverter - a converter for multiple target languages

The cURL manpage

REPOSITORY

The public repository of this module is http://github.com/Corion/HTTP-Request-FromCurl.

SUPPORT

The public support forum of this module is https://perlmonks.org/.

BUG TRACKER

Please report bugs in this module via the Github bug queue at https://github.com/Corion/HTTP-Request-FromCurl/issues

AUTHOR

Max Maischein [email protected]

COPYRIGHT (c)

Copyright 2018-2023 by Max Maischein [email protected].

LICENSE

This module is released under the same terms as Perl itself.

http-request-fromcurl's People

Contributors

corion avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

http-request-fromcurl's Issues

Test suite fails: Accept-Encoding does not exist

Seen on CentOS7 smokers:

"my" variable $res masks earlier declaration in same scope at t/TestWgetIdentity.pm line 177.
Missing argument in sprintf at t/TestWgetIdentity.pm line 188.
# Testing with wget version '1.14'

#   Failed test '-O - --debug --cache $url --header X-Test: test (Logs)'
#   at t/TestWgetIdentity.pm line 255.
#     Structures begin differing at:
#          $got->{Accept-Encoding} = undef
#     $expected->{Accept-Encoding} = Does not exist
# $VAR1 = [
#           {
#             "User-Agent" => "Wget/1.14",
#             "X-Test" => "test",
#             "Accept" => "*/*",
#             "Connection" => "Keep-Alive",
#             "Accept-Encoding" => undef,
#             "Host" => "[::1]:46149"
#           },
#           {
#             "Connection" => "Keep-Alive",
#             "User-Agent" => "Wget/1.14",
#             "X-Test" => "test",
#             "Accept" => "*/*",
#             "Host" => "[::1]:46149"
#           }
#         ];

#   Failed test 'We create (almost) the same headers with LWP'
#   at t/TestWgetIdentity.pm line 177.
#     Structures begin differing at:
#          $got->[3] = 'Accept-Encoding: identity'
#     $expected->[3] = 'Host: [::1]:46149'
... etc ...

Test suite fails: content-length mismatch

On some systems a content-length check fails:

#   Failed test '--verbose -g -s --data-urlencode @$tempfile $url (reconstructed)'
#   at t/TestCurlIdentity.pm line 425.
#     Structures begin differing at:
#          $got->{headers}{Content-Length} = '34'
#     $expected->{headers}{Content-Length} = '26'
# Original command:
# $VAR1 = [
#           "--verbose",
#           "-g",
#           "-s",
#           "--data-urlencode",
#           "\@\$tempfile",
#           "\$url"
#         ];
# Original request:
# $VAR1 = {
#           "method" => "POST",
#           "headers" => {
#                          "User-Agent" => "curl/7.79.1",
#                          "Host" => "[::1]:30052",
#                          "Content-Length" => 26,
#                          "Content-Type" => "application/x-www-form-urlencoded",
#                          "Accept" => "*/*"
#                        },
#           "path" => "/",
#           "protocol" => "HTTP/1.1"
#         };
# Reconstructed command:
# $VAR1 = [
#           "--verbose",
#           "--silent",
#           "--request",
#           "POST",
#           "--header",
#           "Content-Length: 34",
#           "--header",
#           "Content-Type: application/x-www-form-urlencoded",
#           "--user-agent",
#           "curl/7.79.1",
#           "--data-raw",
#           "This%20is%20a%20test%0AMore%20test",
#           "http://[::1]:30052/"
#         ];
# Reconstructed request:
# $VAR1 = {
#           "protocol" => "HTTP/1.1",
#           "headers" => {
#                          "Accept" => "*/*",
#                          "Content-Type" => "application/x-www-form-urlencoded",
#                          "Host" => "[::1]:30052",
#                          "User-Agent" => "curl/7.79.1",
#                          "Content-Length" => 34
#                        },
#           "path" => "/",
#           "method" => "POST"
#         };

#   Failed test '--verbose -g -s --data-urlencode @$tempfile $url'
#   at t/TestCurlIdentity.pm line 276.
#     Structures begin differing at:
#          $got->{Content-Length} = '34'
#     $expected->{Content-Length} = '26'
# $VAR1 = [
#           {
#             "Content-Length" => 34,
#             "User-Agent" => "curl/7.79.1",
#             "Host" => "[::1]:30052",
#             "Accept" => "*/*",
#             "Content-Type" => "application/x-www-form-urlencoded"
#           },
#           {
#             "Host" => "[::1]:30052",
#             "User-Agent" => "curl/7.79.1",
#             "Accept" => "*/*",
#             "Content-Type" => "application/x-www-form-urlencoded",
#             "Content-Length" => 26
#           }
#         ];

#   Failed test 'We create (almost) the same headers with LWP'
#   at t/TestCurlIdentity.pm line 197.
#     Structures begin differing at:
#          $got->[5] = 'Content-Length: 34'
#     $expected->[5] = 'Content-Length: 26'
# Expected:
# Request:
# POST / HTTP/1.1
# Accept: */*
# Host: [::1]:30052
# User-Agent: curl/7.79.1
# Content-Length: 26
# Content-Type: application/x-www-form-urlencoded
# 
# This+is+a+test%0AMore+test
# 
# Got:
# Request:
# POST / HTTP/1.1
# Accept: */*
# Host: [::1]:30052
# User-Agent: curl/7.79.1
# Content-Length: 34
# Content-Type: application/x-www-form-urlencoded
# 
# This%20is%20a%20test%0AMore%20test
# 
#     use strict;
#      use LWP::UserAgent;
# 
#     my $ua = LWP::UserAgent->new('send_te' => '0');
#     my $r = HTTP::Request->new(
#         'POST' => 'http://[::1]:30052/',
#         [
#             'Accept' => '*/*',
#             'Host' => '[::1]:30052',
#             'User-Agent' => 'curl/7.79.1',
#             'Content-Length' => '34',
#             'Content-Type' => 'application/x-www-form-urlencoded'
#         ],
#         "This\x2520is\x2520a\x2520test\x250AMore\x2520test"
#     );
#     my $res = $ua->request( $r,  );
# 

#   Failed test 'We create (almost) the same headers with HTTP::Tiny'
#   at t/TestCurlIdentity.pm line 197.
#     Structures begin differing at:
#          $got->[4] = 'Content-Length: 34'
#     $expected->[4] = 'Content-Length: 26'
# Expected:
# Request:
# POST / HTTP/1.1
# Accept: */*
# User-Agent: curl/7.79.1
# Content-Length: 26
# Content-Type: application/x-www-form-urlencoded
# 
# This+is+a+test%0AMore+test
# 
# Got:
# Request:
# POST / HTTP/1.1
# Accept: */*
# User-Agent: curl/7.79.1
# Content-Length: 34
# Content-Type: application/x-www-form-urlencoded
# 
# This%20is%20a%20test%0AMore%20test
# 
#     use strict;
#      use HTTP::Tiny;
# 
#     my $ua = HTTP::Tiny->new('verify_SSL' => '1');
#     my $res = $ua->request(
#         'POST' => 'http://[::1]:30052/',
#         {
#           headers => {
#             'Content-Length' => '34',
#             'User-Agent' => 'curl/7.79.1',
#             'Accept' => '*/*',
#             'Content-Type' => 'application/x-www-form-urlencoded'
#           },
#           content =>  "This\x2520is\x2520a\x2520test\x250AMore\x2520test"
#         },
#     );
# 
# Looks like you failed 4 tests of 40.
t/curl-identity-data.t ....... 
Dubious, test returned 4 (wstat 1024, 0x400)
Failed 4/40 subtests 

Help!

$ curl2perl -?
Unknown option: ?
$ curl2perl --help
Unknown option: help
$ curl2perl -help
Unknown option: h

Test suite fails: Cache-Control without must-revalidate

On some systems the test suite fails like this:

#   Failed test '-O - --debug --no-cache $url --header X-Test: test (reconstructed)'
#   at t/TestWGetIdentity.pm line 390.
#     Structures begin differing at:
#          $got->{headers}{Cache-Control} = 'no-cache'
#     $expected->{headers}{Cache-Control} = 'no-cache, must-revalidate'
# Original command:
# $VAR1 = [
#           "-O",
#           "-",
#           "--debug",
#           "--no-cache",
#           "\$url",
#           "--header",
#           "X-Test: test"
#         ];
# Original request:
# $VAR1 = {
#           "method" => "GET",
#           "protocol" => "HTTP/1.1",
#           "path" => "/",
#           "headers" => {
#                          "Cache-Control" => "no-cache, must-revalidate",
#                          "Accept" => "*/*",
#                          "Host" => "[::1]:54342",
#                          "User-Agent" => "Wget/1.14 (linux-gnu)",
#                          "X-Test" => "test",
#                          "Connection" => "Keep-Alive",
#                          "Pragma" => "no-cache"
#                        }
#         };
# Reconstructed command:
# $VAR1 = [
#           "--debug",
#           "-O",
#           "-",
#           "--header",
#           "Cache-Control: no-cache",
#           "--header",
#           "Pragma: no-cache",
#           "--user-agent",
#           "Wget/1.14",
#           "--header",
#           "X-Test: test",
#           "http://[::1]:54342/"
#         ];
# Reconstructed request:
# $VAR1 = {
#           "method" => "GET",
#           "protocol" => "HTTP/1.1",
#           "path" => "/",
#           "headers" => {
#                          "Pragma" => "no-cache",
#                          "X-Test" => "test",
#                          "Connection" => "Keep-Alive",
#                          "Host" => "[::1]:54342",
#                          "User-Agent" => "Wget/1.14",
#                          "Cache-Control" => "no-cache",
#                          "Accept" => "*/*"
#                        }
#         };

Tests started failing, maybe srand is missing?

Last night my smoker sent the 26 PASS reports listed below and after that all following reports were FAILs. I guess there should be some srand somewhere. The manpage of File::Temp insists that this is necessary in forking environments. IIUC, the tests are forking? Here is the link to the paragraph in the File::Temp manpage: https://metacpan.org/module/File::Temp/source#L2542-2546

Just for the record: my /tmp/ directory has at the moment about 1500 files and directories matching m!^/tmp/\w{10}$!; so if you cannot reproduce my findings, please be more patient.

Here's a link to a sample fail reports: http://www.cpantesters.org/cpan/report/23dd792a-709b-11ee-98af-ea102c1f115a

137511985 | 5.3.0-2-amd64 | x86_64-linux-ld | 5.28.2 | linux | Andreas J. Koenig | 2023-10-22 05:09
137511707 | 5.16.0-5-amd64 | x86_64-linux-thread-multi | 5.28.3 | linux | Andreas J. Koenig | 2023-10-22 04:52
137511669 | 5.16.0-6-amd64 | x86_64-linux-thread-multi | 5.28.3 | linux | Andreas J. Koenig | 2023-10-22 04:50
137511631 | 5.16.0-5-amd64 | x86_64-linux-thread-multi | 5.30.0 | linux | Andreas J. Koenig | 2023-10-22 04:48
137511585 | 5.17.0-1-amd64 | x86_64-linux-thread-multi | 5.30.1 | linux | Andreas J. Koenig | 2023-10-22 04:45
137511535 | 5.16.0-5-amd64 | x86_64-linux-ld | 5.30.2 | linux | Andreas J. Koenig | 2023-10-22 04:42
137511497 | 5.10.0-14-amd64 | x86_64-linux | 5.30.3 | linux | Andreas J. Koenig | 2023-10-22 04:40
137511467 | 5.15.0-2-amd64 | x86_64-linux-ld | 5.32.1 | linux | Andreas J. Koenig | 2023-10-22 04:38
137511442 | 5.16.0-6-amd64 | x86_64-linux-ld | 5.32.1 | linux | Andreas J. Koenig | 2023-10-22 04:35
137511354 | 6.1.0-7-amd64 | x86_64-linux-thread-multi | 5.32.1 | linux | Andreas J. Koenig | 2023-10-22 04:29
137511239 | 5.16.0-6-amd64 | x86_64-linux-thread-multi | 5.34.0 | linux | Andreas J. Koenig | 2023-10-22 04:22
137511212 | 5.17.0-1-amd64 | x86_64-linux-multi-ld | 5.34.1 | linux | Andreas J. Koenig | 2023-10-22 04:21
137511171 | 5.10.0-14-amd64 | x86_64-linux-thread-multi-quadmath | 5.36.0 | linux | Andreas J. Koenig | 2023-10-22 04:18
137511134 | 5.10.0-14-amd64 | x86_64-linux-thread-multi-ld | 5.36.0 | linux | Andreas J. Koenig | 2023-10-22 04:16
137511108 | 5.10.0-14-amd64 | x86_64-linux | 5.36.0 | linux | Andreas J. Koenig | 2023-10-22 04:15
137511007 | 6.1.0-7-amd64 | x86_64-linux-thread-multi-ld | 5.36.1 | linux | Andreas J. Koenig | 2023-10-22 04:09
137510951 | 6.1.0-7-amd64 | x86_64-linux-ld | 5.37.11 | linux | Andreas J. Koenig | 2023-10-22 04:05
137510916 | 6.1.0-7-amd64 | x86_64-linux-thread-multi-ld | 5.37.11 | linux | Andreas J. Koenig | 2023-10-22 04:03
137510797 | 6.1.0-7-amd64 | x86_64-linux-thread-multi | 5.37.11 | linux | Andreas J. Koenig | 2023-10-22 03:56
137510753 | 6.1.0-7-amd64 | x86_64-linux-thread-multi | 5.39.1 | linux | Andreas J. Koenig | 2023-10-22 03:53
137510687 | 6.1.0-7-amd64 | x86_64-linux-quadmath | 5.39.2 | linux | Andreas J. Koenig | 2023-10-22 03:47
137510629 | 6.1.0-7-amd64 | x86_64-linux-multi | 5.39.2 | linux | Andreas J. Koenig | 2023-10-22 03:42
137510479 | 6.1.0-7-amd64 | x86_64-linux-thread-multi-ld | 5.39.2 | linux | Andreas J. Koenig | 2023-10-22 03:31
137510396 | 6.1.0-7-amd64 | x86_64-linux-quadmath | 5.39.3 | linux | Andreas J. Koenig | 2023-10-22 03:26
137510294 | 6.1.0-7-amd64 | x86_64-linux-thread-multi-ld | 5.39.3 | linux | Andreas J. Koenig | 2023-10-22 03:19
137510191 | 6.1.0-7-amd64 | x86_64-linux-thread-multi | 5.39.3 | linux | Andreas J. Koenig | 2023-10-22 03:12

improved translation of -F curl flag

Trying the following URL on https://corion.net/curl2lwp.psgi gives us some encoded form data

curl -X POST -F field=value -F name=Corion "https://httpbin.org/post" -H  "accept: application/json"

The following "translation" might be better:

use strict;
use warnings;

use LWP::UserAgent ();
use HTTP::Request::Common qw(POST);

my $ua = LWP::UserAgent->new(timeout => 10);
my $url = 'https://httpbin.org/post';
my %content = (
    field => 'value',
    name => 'Corion',
);

my $request = POST $url, 'Content-Type' => 'form-data', Content => \%content;
my $response = $ua->request($request);
print $response->decoded_content;

Test suite fails: user-agent mismatch

On almost all of my linux & freebsd smokers I see test failures like this:

#   Failed test '-O - --debug --no-cache $url --header X-Test: test (reconstructed)'
#   at t/TestWGetIdentity.pm line 390.
#     Structures begin differing at:
#          $got->{headers}{User-Agent} = 'Wget/1.18'
#     $expected->{headers}{User-Agent} = 'Wget/1.18 (freebsd10.3)'
...

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.