rhcarvalho / envdiff

Smart environment diffs

envdiff

Generates smart environment diffs.

The envdiff tool compares two environments, and tries to be smart where it can. For example, it can detect appends or prepends to lists like PATH.

Given two environments A and B, envdiff computes C such that:

(export A; export C) == export B

Examples here.

Installing

You will need to have the go tool installed, then:

go get -u github.com/rhcarvalho/envdiff

Now envdiff should be in your PATH.

Using

envdiff takes two arguments that should contain a list of newline or null-byte separated environment variables.

Examples:

1. Detecting new variables:

$ envdiff TERM=xterm $'TERM=xterm\nFOO=bar'
FOO=bar

2. Detecting removed variables:

$ envdiff $'TERM=xterm\nFOO=bar' TERM=xterm
FOO=

3. Detecting changing of a list:

$ envdiff PATH=/bin:/sbin PATH=/usr/bin/:/usr/sbin:/bin:/sbin
PATH=/usr/bin/:/usr/sbin:$PATH

How to reverse-engineer a Software Collection for Docker images

By detecting environment changes introduced by enabling certain Software Collections (SCL) and writing a Dockerfile ENV instruction that has the same effect, one can create Docker images that have the given collections enabled by default.

For example, if you install a Ruby collection in a CentOS image, you might as well want to have the ruby binary in your PATH. A user of your image need not to know about SCL or have to do any extra steps to be able to run a line like:

docker run --rm scl-enabled-ruby-image ruby -e 'puts "Hello World!"'

Great, how do I do that with envdiff? There's really only one required step!

1. Point envdiff at your target image:

$ envdiff -o docker scl openshift/ruby-20-centos7
# -- generated by https://github.com/rhcarvalho/envdiff, do not edit manually --
# The ENV instruction below is equivalent to:
#   scl enable nodejs010 ror40 ruby200 v8314
ENV MANPATH=/opt/rh/v8314/root/usr/share/man:/opt/rh/ruby200/root/usr/share/man:/opt/rh/ror40/root/usr/share/man:/opt/rh/v8314/root/usr/share/man:/opt/rh/nodejs010/root/usr/share/man: \
    LIBRARY_PATH=/opt/rh/v8314/root/usr/lib64:/opt/rh/v8314/root/usr/lib64 \
    X_SCLS=nodejs010 ror40 v8314  \
    LD_LIBRARY_PATH=/opt/rh/v8314/root/usr/lib64:/opt/rh/ruby200/root/usr/lib64:/opt/rh/ror40/root/usr/lib64:/opt/rh/v8314/root/usr/lib64:/opt/rh/nodejs010/root/usr/lib64 \
    CPATH=/opt/rh/v8314/root/usr/include:/opt/rh/v8314/root/usr/include \
    PATH=/opt/rh/v8314/root/usr/bin:/opt/rh/ruby200/root/usr/bin:/opt/rh/ror40/root/usr/bin:/opt/rh/v8314/root/usr/bin:/opt/rh/nodejs010/root/usr/bin:/usr/local/bin:/usr/bin \
    PYTHONPATH=/opt/rh/v8314/root/usr/lib/python2.7/site-packages:/opt/rh/v8314/root/usr/lib/python2.7/site-packages:/opt/rh/nodejs010/root/usr/lib/python2.7/site-packages \
    GEM_PATH=/opt/rh/ror40/root/usr/share/gems:/opt/app-root/src/.gem/ruby:/opt/rh/ruby200/root/usr/share/gems:/opt/rh/ruby200/root/usr/local/share/gems \
    PKG_CONFIG_PATH=/opt/rh/v8314/root/usr/lib64/pkgconfig:/opt/rh/ruby200/root/usr/lib64/pkgconfig:/opt/rh/ror40/root/usr/lib64/pkgconfig:/opt/rh/v8314/root/usr/lib64/pkgconfig
# -- end of generated instruction --

2. (Optional) You may want to redirect the output straight into a Dockerfile:

envdiff -o docker scl openshift/ruby-20-centos7 >> Dockerfile

If you want to print the environment in shell format, omit the -o docker flag:

$ envdiff scl openshift/python-34-centos7
# scl enable rh-python34
MANPATH=/opt/rh/rh-python34/root/usr/share/man:
X_SCLS=rh-python34
LD_LIBRARY_PATH=/opt/rh/rh-python34/root/usr/lib64
PATH=/opt/rh/rh-python34/root/usr/bin:/usr/local/bin:/usr/bin
XDG_DATA_DIRS=/opt/rh/rh-python34/root/usr/share
PKG_CONFIG_PATH=/opt/rh/rh-python34/root/usr/lib64/pkgconfig

How does the scl mode works?

What is envdiff doing to generate an environment that transforms an initial environment into one with collections enabled?
You can go read the code, it's all in the scl.go file, or follow along.

First, we need to discover what collections are installed. This can be accomplished with scl --list in a container.

Second, we need two environments: one without any collections enabled, and another one will all collections enabled.

To get a clean environment, we run bash -c env in a temporary container. We wrap the call to env in a Bash shell to be closer to what we get by enabling collections. To make sure that the environment is clean no matter what is in the image, we unset all environment variables set in the image when creating a container.

To get the environment with all collections enabled, it is scl enable collection1 collection2 ... env. In other words, we run env to dump the environment with all collections from the first step being enabled. Since scl has a shebang that points to /bin/bash, this time we don't wrap the env call like in the previous step.

The last step is to do what envdiff is made to do: compare two environments. Producing output as a Dockerfile ENV instruction makes it convenient to modify existing Dockerfiles to permanently enable collections.

envdiff's Issues

Avoid exporting PATH with empty path segment

See https://github.com/sclorg/s2i-python-container/pull/108/files#r58169176

For RPMs security team pointed out that enable scriptlet needs to avoid exporting path with empty path segment (usually a trailing ':'). It's done by appending envar only if it is already defined, e.g. export LD_LIBRARY_PATH=/opt/rh/ruby200/root/usr/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

IMO we should do the same in the case of docker images too.