iq-scm / nigit

Web server that wraps around programs and shell scripts and exposes them as API

nigit

⚠️ This repository is no longer maintained by Lukas Martinelli.

Expose a program with a simple call to nigit <script> to the web. The small web server wraps around the program and exposes it as HTTP API. This comes in handy whenever you want to expose a legacy program without writing a web application and doing complicated subprocessing yourself.

Disclaimer

Please use goexpose. It is much more capable and mature than nigit. In the rush of building stuff I didn't realize that nigit is essentially a new take on the terrible CGI scripts we just escaped from. Be very careful where you use this, it is not intended for production usage in any means due to the security risks. I personally do use it for small home automation tasks.

Get Started

In this example we create a service to download the PDF version of websites using the wkhtmltopdf tool.

1. Create the bash script html2pdf.sh and make it executable with chmod +x html2pdf.sh.

#!/bin/bash
wkhtmltopdf "$URL" page.pdf > /dev/null 2>&1
cat page.pdf

2. Start up the server with nigit html2pdf.sh.
3. Download the PDF with curl -o google.pdf http://localhost:8000/html2pdf?url=http://google.com

And that's all you needed to do in order to expose wkhtml2pdf as useful webservice.

Install

You can download a single binary for Linux, OSX or Windows.

OSX

wget -O nigit https://github.com/lukasmartinelli/nigit/releases/download/v0.3/nigit_darwin_amd64
chmod +x nigit

./nigit --help

Linux

wget -O nigit https://github.com/lukasmartinelli/nigit/releases/download/v0.3/nigit_linux_amd64
chmod +x nigit

./nigit --help

Install from Source

go get github.com/lukasmartinelli/nigit

If you are using Windows or 32-bit architectures you need to download the appropriate binary yourself.

Examples

How you can use nigit to build small and concise services:

• PDF build service using pdflatex
• Convert DOCX files to Markdown with pandoc
• Image cropping with imagemagick
• Convert WAV to MP4 with avconf
• Transpile code with BabelJS
• Lint Shell scripts with shellcheck

Usage

Pass Arguments

Form arguments or query strings are passed as environment variables into the program.

#!/bin/bash
echo "$MY_ARGUMENT"

You can specify them as form values or alternatively post a JSON file which is more convenient in JavaScript.

# pass as form values
curl -X POST -F my_argument=test http://localhost:8000/
# pass as query string
curl http://localhost:8000/?my_argument=test
# pass as JSON
curl -X POST -H "Content-Type: application/json" \
  -d '{"envs": ["MY_ARGUMENT=test"]}' \
  http://localhost:8000/hlint

Upload File

Uploaded content is provided as stdin to the file. You can either specify it as file upload or form value. In both cases the field must be named stdin.

# set value of stdin in form
curl -X POST -F stdin="Ping" http://localhost:8000/
# for short input you can even use query strings
curl http://localhost:8000/?stdin=Ping
# post a file to the web api
curl -X POST -F [email protected] http://localhost:8000/

You can also specify a stdin field in JSON to pass something to the script.

curl -X POST -H "Content-Type: application/json" \
  -d '{"stdin": "Ping", "envs": ["MY_ARGUMENT=test"]}' \
  http://localhost:8000/hlint

If you are using a Bash script as wrapper you can read in the stdin with cat.

#!/bin/bash
greetings=$(cat)
echo "$greetings"

Serve Multiple Files

nigit can also serve multiple scripts under different paths if you append more programs as arguments.

nigit echo.sh curl.sh lint.sh

This will serve each script under a different HTTP route.

Handle /echo -> echo.sh
Handle /curl -> curl.sh
Handle /lint -> lint.sh

Mime Type

nigit serves the response either as text/plain if no Accept header is specified or exactly with the mime type specified by the Accept header.

If you wrap around a program that outputs valid JSON you need to set the Accept header and you are good.

curl -H "Accept: application/json" http://localhost:8000/

Authorization

nigit has authorization middleware which allows you to call an executable to approve requests. The Authorization http header gets passed as the AUTH env variable. Thanks to @pdxjohnny for the implementation.

./nigit --auth ./auth.sh ./echo.sh

Example auth.sh

#!/bin/bash
echo $AUTH
exit $AUTH

If it exits non-zero then the client with receive a http.Unauthorized and the stdout of auth.sh will be printed to nigit stdout.

curl -H "Authorization: 1" http://localhost:8000/echo?say=hi

Output: {"Error":"Unauthorized"}

curl -H "Authorization: 0" http://localhost:8000/echo?say=hi

Output: hi

Use together with Docker

nigit fits perfectly into the Docker ecosystem. You can install nigit into a Docker container and wrap around a program that requires complex dependencies.

Create a Dockerfile with nigit and your dependencies for the shell script. In this example we provide shellcheck as a service.

FROM debian:jessie

RUN apt-get update \
 && apt-get install -y --no-install-recommends shellcheck \
 && rm -rf /var/lib/apt/lists/* \

# install nigit
RUN wget --quiet -O /usr/bin/nigit https://github.com/lukasmartinelli/nigit/releases/download/v0.2/nigit_linux_amd64 \
 && chmod +x /usr/bin/nigit

# copy shell scripts
COPY . /usr/src/app/
WORKDIR /usr/src/app

EXPOSE 8000
CMD ["nigit", "--timeout", "5", "shellcheck.sh"]

Now create a bash script to wrap around shellcheck. We specify the json output formatter so that a web client could consume the API.

#!/bin/bash

function clone_repo() {
    local working_dir=$(mktemp -dt "lint.XXX")
    local git_output=$(git clone --quiet "$GIT_REPOSITORY" "$working_dir")
    echo "$working_dir"
}

function find_files() {
    local path="$1"
    local extension="$2"
    find "$path" -type f -name "*$extension"
}

function lint() {
    local repo_path=$(clone_repo)
    shellcheck --format=json $(find_files "$repo_path" "sh") || suppress_lint_error
    trap "rm -rf $repo_path" EXIT
}

lint

And now you can send links to Git repositories to your service to check them for Bash errors.

curl -H "Accept: application/json" \
http://localhost:8000/shellcheck?git_repository=https://github.com/lukasmartinelli/nigit.git

Security

You should never expose this script directly to the internet. Shell scripts that take input from external are always vulnerable to different attack vectors.

A better approach is to run nigit in a Docker container and access it from your other micro services and never exposing it to the public. This is still handy because you can isolate that old program and act as if it is an API to your other micro services.

Develop

You need a Go workspace to get started.

Install Dependencies

Several dependencies are required.

go get "github.com/codegangsta/cli"
go get "github.com/op/go-logging"

Build

Create a executable using the standard Go build tool.

go build

Cross Compile Release

We use gox to create distributable binaries for Windows, OSX and Linux.

docker run --rm -v "$(pwd)":/usr/src/nigit -w /usr/src/nigit tcnksm/gox:1.4.2-light