visola / go-http-cli

A powerful HTTP CLI client written in Go.

Hi, I'm Vini 👋

Welcome to my profile! I'm currently a Senior Software Engineer @ FOSSA and the technical and operational lead @ Revollu. Previously a SW engineer at Toast, HubSpot and IBM.

I've co-authored two books:

• Java Fundamentals - A fast-paced and pragmatic introduction to one of the world's most popular programming languages
• Professional Javascript - Develop your JavaScript programming skills by learning strategies and techniques commonly used in modern full-stack application development

I love learning together and sharing knowledge and I'd love to connect. Please send me an email if you'd like to connect.

About me

Some people think I have super powers but the truth is that I like being challenged. 🔥 And I have amazing friends and family that always supported me. ❤️

I like learning something new every day so I'm always finding new projects to work on. That means you may get lost in my repositories, so please first check the pinned repositories below. I'm an expert in Java and Javascript but I've studied a few other languages, including: Python and Go.

Besides learning programming languages I also love learning spoken and written languages. My native language is portuguese (🇧🇷) but I'm also fluent in english (🇺🇸). I've also studied: spanish (🇪🇸), german (🇩🇪), chinese (mandarin) (🇨🇳), french (🇫🇷) and I am currently studying japenese (🇯🇵).

I also run marathons. I've ran 3 of the 6 majors: Boston, Chicago and Berlin. I've also ran a few other marathons: Athens, Providence (RI), Cape Code (MA), Philadelphia and Narragansett (RI).

go-http-cli's Issues

Add command to kill the daemon

To kill the daemon it's necessary to run ps first, find the process, then run kill with the PID. It would be nice to have a feature where I could just run:

go-http-daemon --kill

And the daemon would kill itself.

Import one profile from another

Add an import element to the Profile yaml so that I can import one profile into another one. Good for sharing headers, auth and variables.

import: another-profile

or

import:
  - another-profile
  - yet-another

Add Request to Profile

In a Profile, the user would be able to configure a request to be executed by passing an argument like @login to execute a common request.

The YAML file could be configured like (profile.yml):

requests:
  - name: login
    url: /login
    headers:
       - Content-type: application-x-www-form-urlencoded
    method: POST
    body: username=MyUserName&password=MyPassword

  - name: create-employee
    url: /{orgId}/employees
    headers:
       - Content-type: application/json
    method: POST

Information passed in will be merged with the information provided. So for the second example, it would be possible to do the following:

$ http +profile @create-employee -d '{"name":"John Doe","birthdate":"1975-01-01"}'

Ability to express expected output.

Build release ZIP as part of the build

Build a ZIP file containing all binaries that can be downloaded as a release package.

Use `{ }` instead of `${ }` for variable replacement

When using ${ } from the command line, it needs to be inside single quotes or escaped, which is annoying. Switching to { } will avoid this issue and it's still a nice syntax.

Documentation, documentation, documentation

Documentation for a tool project like this has the following goals:

• Help users of the tool understand what the tool does and how
• Help contributors understand the architecture of the code and how it works

The first should include things like:

• how to download and install the tool
• a list of all the features and how to access them
• known erros and how to solve/workaround

The second should include:

• basic components and how they interact with each other
• how to build

The first set could live in the root README file and should include a link to the second set. The second set can live somewhere else, e.g.: ARCHITECTURE.md or something.

Add ability to express expected headers.

Ability to set a profile to be available for the test case.

User Story

As a developer I want to be able to test features from profiles.

Acceptance Criteria

• Integration test suite will setup a directory and environment variable where profiles will be put in
  • Each spec will have it's own directory where profiles will be put in
  • Path to load profiles from will be set automatically
  • If error, there should be a way to refer to the profiles that were created
• There's a way to declare one or more profiles in my test case

Add variables to session and enable override from CLI

When setting a profile, sometimes you want to set some default variables and override them to some other values to execute the same set of requests in some other context.

User story

As a user, I want to have an easy way to reuse my requests in a different context without having to create multiple profiles or changing profiles manually.

Acceptance criteria

• Daemon has a new endpoint that can be used to set variables in a global scope
• When using the -V, --variable option and not actually making a request (no request name or URL passed in), set the variable in a global session

Add ability to express expected path.

Easily build bodies and query strings

In HTTPie you can easily pass in key-value pairs and have them be set correctly in your body or query strings. This is a really nice feature that saves a lot of time.

User Story

As a user, when executing a request, I want to be able to easily build a query string or body to add to my request.

Acceptance criteria

Basic functionality is to add this behavior to CLI arguments and also to profiles:

• Internally, request.Request now has a Values field
  • If method is GET and there are key-value pairs, URL encode and add them to the query string
  • If method is not GET
    • If content-type is not set or is JSON mime type, build a JSON object, ensure content-type and set the JSON in the body
    • If content-type is application/x-www-form-urlencoded, URL encode values and set them in the body
• from the CLI:
  • Accept dangling key-value pairs from the CLI and parse them correctly
• from the profiles:
  • requests now has a new attribute values that reads as a map of strings or array of string

Default to POST if there is data to be sent

95% of the time, when you're sending data to the server, you want to execute an HTTP POST. So why not just default to it when data is set.

Refactor everything config to profile

Right now, it's hard to distinguish and name things because the concepts of request configuration, command line options and profile are mixed together. With more complex features like the daemon (#26) it will get harder to understand how things work.

This issue is to identify which parts should live where and refactor everything to go in the right place. So far, there are at least the following responsibilities:

• command line options: anything that the user can pass in the CLI as arguments
• profiles: where to find profile files, what is loaded from profiles
• request configuration: from the previous two replace variables, complete URLs and build a request

Failed to parse URL correctly

After #75, trying to pass a full URL will fail because it parse the : as key:value instead of a URL.

Add Basic Authentication

Add an auth option in the yaml file so that people don't have to base64 encode user and password or even understand how that type of authorization works.

This article provides a good explanation of all types of authentication that exist. The two most common are Basic and Bearer. Implementing those two would be a great start.

The idea is that in the yaml file the user could add something like:

auth:
  type: basic
  user: MyUsername
  password: myPassword

or

auth:
  type: bearer
  token: MyVeryLongTokenGoesHere

Create a redial function

After #34 is done, it would be nice to have a way to search and re-execute requests. An initial idea is to use something like https://github.com/gizak/termui to create a simple UI that can be used to search the history and re-execute previous requests.

Support header value with "=" on them

The algorithm that splits the value from the name doesn't support values that contains "=" on them. An example of such a header is:

Accept: text/html, application/xhtml+xml, application/xml;q=0.9, */*;q=0.8

Pass errors back correctly from the daemon

Because requests are now executing in the daemon, when there's an error, it needs to be passed back to the client somehow. Ideally, all requests to the daemon will be wrapped in some kind of payload/error carrier response that will contain all the information necessary to understand what happened on the daemon side.

Good examples of how to do this are: Flux Standard Actions or the JSON API standard.

Do not automatically follow redirects

cURL doesn't automatically follow redirects and that's a good thing. Such a low level tool like an HTTP client needs to let the user decide whether to follow redirects or not.

cURL has the following options that this issue aims to base off from:

 -L, --location      Follow redirects (H)
     --max-redirs NUM  Maximum number of redirects allowed (H)

Refactor Request Creation

Extract all the logic related to creating a http.Request from the configuration parsed.

Parsing the URL was already extracted (as part of #16) into its own function but setting all the headers, body and actually creating the request needs to be extracted into the request package so that it can be tested separated, further extended with variable replacement and used in new features I'm planning to add.

After this is done, main should only call a method and get the instance of http.Request and error. Check for error, output request information and run it.

Refactor Request and Response output logic

After #21 is done, the next step is to extract serializing Request and Response to be output on the console. Extracting this logic into its own package will make it simpler to test. It will also make it simpler extend with more advanced features like pretty printing and highlighting data going and coming based on content type headers.

Create a basic integration test framework that provides the following:

• Ability to run tests in a current build
• Test cases go in a directory specs and can be nested
• Specs are written in YAML and support:
  • Command to execute
  • Expected HTTP method
• Results are easily readable

Add integration tests

Add a framework using the Go test HTTP server to test that the whole infrastructure works correctly for all the different scenarios.

Add scripting framework to post-process request

User story

As a user, I want to execute some Javascript after a request to process the results.

Description

A very useful feature from Postman is the ability to execute a script after a request is executed to check the results or extract data from it. Read more here.

To make this work, it's required to add a javascript engine, possibly embed V8 or use an existing API bindings like go-v8.

Acceptance Criteria

• Add Javascript engine
• On the CLI, add a --post-process option that receives a file that will be used to post-process the executed request
• From profiles, have two ways to execute post process scripts:
  • add requests.postProcessFile loads a file to be executed
  • add requests.postProcessScript that can be used to inline javascript inside the YAML
• Post process scripts can access the following:
  • From the response:
    • status code
    • body as string
    • headers
  • From the Request
    • URL
    • body
    • headers
• Post process scripts needs to have the ability to:
  • set a session variable
  • set a global variable
  • [ ] set a cookie on the current session
  • add a new request to be executed, pushing one of the following to the requests queue:
    • by request name, loaded from the active profiles
    • by creating a request object, following the same structure as in the profile YAML

Make help actually helpful

Asking for help from the command line is not very helpful:

$ http --help
Usage of http:
  -c, --config keyValuePair     Path to configuration files to be used
  -d, --data string             Data to be sent as body
  -H, --header keyValuePair     Headers to include with your request
  -L, --location                Automatically follow redirects
      --max-redirs int          Maximum number of redirects to follow (default 50)
  -X, --method string           HTTP method to be used
  -o, --output string           File to save the response
  -T, --upload-file string      Path to the file to be uploaded
  -V, --variable keyValuePair   Variables to be used on substitutions
pflag: help requested

User Story

As a user, when I ask for help, I want to be reminded how to activate a profile, execute a named request or add values to a query string or body.

Acceptance Criteria

Typing help shows a message showing how to activate profiles, execute named requests and add values to a query string or body. It should also explain briefly what profiles are, where they should live and what they can do. A link to the official documentation (main GitHub page for now) would also be good.

Keep track of executed requests

Store the executed requests (and possibly their responses) somewhere. The initial idea is to store it under ~/.go-http-cli/history.yml, in yaml format so that it can be easily unmarshalled and used in future features.

Values in configured request aren't been URL encoded

If put a value like: start: 2018-03-13T21:00:00+01:00, the server is getting 2018-03-13T21:00:00 01:00, where the + is being replaced by a space.

These keys and values need to be correctly URL encoded before sent to server.

Pretty print responses

One of the most appealing features of HTTPie is it's (very) pretty and readable output.

User story

As a user, I want to easily read the response and understand the results when executing a request.

Acceptance Criteria

• Add a pluggable framework that enables simple addition of formatters based on mime-type
• Provide an initial implementation to process JSON responses

Add unit tests and increase coverage

After all the refactoring, I've been lazy and left the tests behind. Now that the code is getting a bit more stable, it's time to start adding tests again.

Add ability to express expected query strings.

Add autocomplete engine

Add an autocomplete processing that helps to complete parameters when typing in the CLI.

Add option to write output to a file

Add an option to write output to a file. From Curl help:

-o, --output FILE   Write to FILE instead of stdout

Add the concept of Session

It's necessary to keep track of sessions per host where the request was executed.

This is just a basic implementation and will only cover Cookie handling.

Load body from a file

There should be an option to load the body of the request from a file. Default method logic should work the same way as it does today. Something like:

$ http +contacts-db -f person.json /people

This would execute a POST to {URL_BASE_FROM_PROFILE}/people and the body would be the content from ./person.json.

Use built-in HTTP constants

Based on this comment.

Add a variables map to YAML configurations

A variables map could include variables that can be replaced in the URL, headers, parameters or even the body of the request.

But this issue is just to make the parser read this map.

Make build kill the running daemon

When developing locally, changing the daemon code requires you to find the process and kill it before testing, which is annoying and takes time. It would be great if every time I run ./gradlew build it would automatically kill de daemon for me.

Add response checks

User Story

As a user I want to be able to execute a request and ensure that it executed as expected.

Long description

Add response checks enables using this tool as a testing tool for APIs. You could write a profile with configured requests to the endpoints you want to test and write a shell script to execute the requests and make it fail based on the process exit code.

Acceptance Criteria

• Add a post process mechanism to check requests after execution
• Ensure process exit status matches good practices (zero for successfully matched responses, something else if not)
• Add a response attribute to request element in profile
  • response has a statusCode attribute that will be checked against the response from the server
  • response has a body attribute that will be used to validate the body, after converting it to a tree of values.
    • If response is application/x-www-form-urlencoded, it will be a map[string] of ([]string or string)
    • If response is application/json, it will be a JSON tree
    • Only values added to body will be compared, all others will be ignored. If value is an array in the response, it will check if the value(s) exist in the array, no matter the order

Replace variables inside the body and headers

For now, go-http-cli only replaces variables inside the URL. This feature needs to be extended to support headers and body.

Replace variables in the URL

Depends on #15.

When creating a request, try to replace variables in the path or URL.

First run fails with missing .pid file

When running it for the first time, the process panics because the daemon.pid file doesn't exist. The workaround is simple but this needs to be fixed.

Workaround:

mkdir -p ~/.go-http-cli
echo 10 > ~/.go-http-cli/daemon.pid

This is the error:

panic: open /home/appuser/.go-http-cli/daemon.pid: no such file or directory

goroutine 1 [running]:
github.com/visola/go-http-cli/daemon.KillDaemon()
	/Users/vinicius.isola/git/go-workspace/src/github.com/visola/go-http-cli/daemon/process.go:48 +0x89
github.com/visola/go-http-cli/daemon.EnsureDaemon(0x16, 0xc42006ae40)
	/Users/vinicius.isola/git/go-workspace/src/github.com/visola/go-http-cli/daemon/process.go:30 +0x59
main.main()
	/Users/vinicius.isola/git/go-workspace/src/github.com/visola/go-http-cli/binaries/http/main.go:15 +0x37

Ability to express expected exit code

Add the ability to request variable value from command line

User Story

As a user I want to be able to have the tool request a secret as a value for variables in a configured request from a profile.

Details

People shouldn't put their passwords in plain text files in their computer. But it's useful to have API definitions to login into systems to fetch authorization tokens for subsequent requests. So it would be nice to have a way to define a variable that should be asked during request execution instead of asking the user to put it in the file, setting it as an environment variable or pass it in as argument in the command line.

Acceptance Criteria

• Profiles and configured request are processed before sending it to the daemon
• Required variables are extracted from the calculated request to be executed
• When declaring a variable, a parameter could be added to make it dynamic and requested during execution, some syntax like {variableName:input}
  • For secrets or passwords, it would be good to have something like {variableName:secret}. When reading this information, the tool would hide the input like a password.
• If a variable with the specified marker is found, request them by name before sending the request to the daemon

Use a real build system

Move all the build logic into a real build system like Make or Gradle.

Ability to express expected body.

Add a daemon process to run requests from

The idea is to have a daemon process running on the background that can run the requests. There are many features that can use a daemon. A few ideas I have are:

• session management: Having a session management system makes it possible to have a login request initiating a session and keeping record of auth and CSRF tokens and cookies.
• avoid data in plain text files: storing passwords and tokens in plain text files is not a good idea, everybody knows that. Having a daemon allows to keep the password in memory for a specific amount of time, requesting it only once, instead of in every request.
• encrypt profiles: The profiles could store passwords and other important data, as long as they are securely encrypted. Unencrypted data could be kept inside the daemon so that the key only need to be typed once.
• cloud storage: Instead of storing your profiles in local directories, you could store them in the cloud and access them using a password. The daemon could cache all profiles from the cloud in memory and only request your password once.

Build is not updating coverage report

For some reason, when changing the code and building, coverage report is not getting updated. To see the latest version it's required to run ./gradlew clean build, which is not ideal.

Do not auto-follow redirects

Go's HTTP library automatically follows redirects but it's possible to make it not do that.

The idea is that the redirects will be followed but not by the http package, but by the same mechanism that executes the requests. That means that request.ExecuteRequest will start to return an array of request.ExecutedRequestResponse instead of only one. And for the majority of the requests (no redirects), it will output the same result. But when there's a redirect, it will show only the last one.

This change gives the opportunity to manage the intermediary states, passed by the redirects, using sessions. For example, if you login into a website using the tool, and the login page redirects and sets a cookie, this pattern will store the cookie and use it to the next requests, while the auto-redirect will ignore everything.

Support passing headers with `:` instead of just `=`

Right now go-http-cli only supports passing headers using =, like Content-type=application/json. This is annoying because:

1. HTTP headers are passed in as : in the protocol
2. curl accepts both formats.

It's an easy change and it's annoying when I try to use a header and forget to use =.