heroku / heroku-buildpack-redis

An stunnel buildpack for Redis

Heroku buildpack: Redis Stunnel

This is a Heroku buildpack that allows an application to use an stunnel to connect securely to Heroku Redis. It is meant to be used in conjunction with other buildpacks.

Warning

This buildpack isn’t compatible with the heroku-24 stack and later. You don’t need this buildpack for Redis 6+, which supports native TLS.

For more information, see Securing Heroku Redis.

Usage

First, ensure your Heroku Redis addon is using a production tier plan. SSL is not available when using the hobby tier.

Then set this buildpack as your initial buildpack with:

$ heroku buildpacks:add -i 1 heroku/redis

Then confirm you are using this buildpack as well as your language buildpack like so:

$ heroku buildpacks
=== frozen-potato-95352 Buildpack URLs
1. heroku/redis
2. heroku/python

For more information on using multiple buildpacks check out this devcenter article.

Next, for each process that should connect to Redis securely, you will need to preface the command in your Procfile with bin/start-stunnel. In this example, we want the web process to use a secure connection to Heroku Redis. The worker process doesn't interact with Redis, so bin/start-stunnel was not included:

$ cat Procfile
web:    bin/start-stunnel bundle exec unicorn -p $PORT -c ./config/unicorn.rb -E $RACK_ENV
worker: bundle exec rake worker

We're then ready to deploy to Heroku with an encrypted connection between the dynos and Heroku Redis:

$ git push heroku main
...
-----> Fetching custom git buildpack... done
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/heroku/heroku-buildpack-redis.git
=====> Detected Framework: Redis-stunnel
       Using stunnel version: 5.02
       Using stack version: cedar
-----> Fetching and vendoring stunnel into slug
-----> Moving the configuration generation script into app/bin
-----> Moving the start-stunnel script into app/bin
-----> Redis-stunnel done
=====> Downloading Buildpack: https://github.com/heroku/heroku-buildpack-ruby.git
=====> Detected Framework: Ruby/Rack
-----> Using Ruby version: ruby-2.2.2
-----> Installing dependencies using Bundler version 1.7.12
...

Configuration

The buildpack will install and configure stunnel to connect to REDIS_URL over a SSL connection. Prepend bin/start-stunnel to any process in the Procfile to run stunnel alongside that process.

Stunnel settings

Some settings are configurable through app config vars at runtime:

• STUNNEL_ENABLED: Default to true, enable or disable stunnel.
• STUNNEL_LOGLEVEL: Default is notice, set to info or debug for more verbose log output.

Multiple Redis Instances

If your application needs to connect to multiple Heroku Redis instances securely, this buildpack will automatically create an Stunnel for each color Heroku Redis config var (HEROKU_REDIS_COLOR) and the REDIS_URL config var. If you have Redis urls that aren't in one of these config vars you will need to explicitly tell the buildpack that you need an Stunnel by setting the REDIS_STUNNEL_URLS config var to a list of the appropriate config vars:

$ heroku config:add REDIS_STUNNEL_URLS="CACHE_URL SESSION_STORE_URL"

Using the edge version of the buildpack

The heroku/redis buildpack points to the latest stable version of the buildpack published in the Buildpack Registry. To use the latest version of the buildpack (the code in this repository), run the following command:

$ heroku buildpacks:add https://github.com/heroku/heroku-buildpack-redis

heroku-buildpack-redis's Issues

Add compatibility w/ interactive commands (e.g. `rails console`)

The current buildpack appears to be incompatible with interactive commands like bin/start-stunnel rails console. I'd love to add support for this via an optional flag or environment variable, so that you can have secure interactive sessions when accessing a production environment.

I'll probably take a swing at this later and try to submit a PR – just wanted to open an issue first in case I was missing something :)

Retain exit code from wrapped process

When wrapping a process with start-stunnel, it seems to "eat" the exit code and always exit with 0, no matter what the wrapped processed exited with. This caused issues for us with a release script, since we use any release errors to abort the release.

When we added start-stunnel, this caused the release to continue even when our migrations failed.

Warning: `Update OpenSSL shared libraries or rebuild stunnel`

I'm trying this buildpack to connect to Heroku Redis using a premium-0 plan on a test app here, and it seems stunnel starts successfully, I can see the enabled output on my logs:

app web.1 - - buildpack=stunnel at=stunnel-enabled
...
app web.1 - - buildpack=stunnel at=stunnel-start

And the app seems to be working fine, however I also have the following on my logs:

2015.10.09 20:48:03 LOG5[21:139662156998464]: stunnel 4.53 on x86_64-pc-linux-gnu platform
2015.10.09 20:48:03 LOG5[21:139662156998464]: Compiled with OpenSSL 1.0.1e 11 Feb 2013
2015.10.09 20:48:03 LOG5[21:139662156998464]: Running with OpenSSL 1.0.1f 6 Jan 2014
2015.10.09 20:48:03 LOG5[21:139662156998464]: Update OpenSSL shared libraries or rebuild stunnel
2015.10.09 20:48:03 LOG5[21:139662156998464]: Threading:PTHREAD SSL:+ENGINE+OCSP Auth:LIBWRAP Sockets:POLL+IPv6
2015.10.09 20:48:03 LOG5[21:139662156998464]: Reading configuration from file vendor/stunnel/stunnel.conf

This tells me something is not right with the buildpack/stunnel installation, it seems to have been compiled against a different version of OpenSSL. Version 1.0.1f 6 Jan 2014 is the one available with heroku bash:

~ $ openssl version
OpenSSL 1.0.1f 6 Jan 2014

Can I test-drive running this using a free dyno, or do I need at least a professional 1X dyno? Is there anything I'm missing? I'd appreciate any pointers on this, thanks!

Remove support for Heroku-16

The Heroku-16 stack reached end-of-life on May 1st, 2021, and in a few days (June 1st, 2021) the Heroku-16 build system will be decommissioned, meaning no further builds will be possible:
https://help.heroku.com/0S5P41DC/heroku-16-end-of-life-faq

As such, support for Heroku-16 can shortly be dropped:
https://github.com/heroku/heroku-buildpack-redis/search?q=16

There are also some Cedar-14 references that can be removed :-)

Illegal TLS option on heroku-18

[ ] Clients allowed=4882
[.] stunnel 5.44 on x86_64-pc-linux-gnu platform
[.] Compiled/running with OpenSSL 1.1.0g  2 Nov 2017
[.] Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,OCSP,PSK,SNI Auth:LIBWRAP
[ ] errno: (*__errno_location ())
[.] Reading configuration from file /app/vendor/stunnel/stunnel.conf
[.] UTF-8 byte order mark not detected
[!] /app/vendor/stunnel/stunnel.conf:5: "options = NO_SSLv2": Illegal TLS option

It looks like the heart of the problem is that stunnel maps these strings to the associated constants in openssl, but because they are a part of a mask openssl makes them 0 when they are no longer used (https://github.com/openssl/openssl/blame/57be4444c645247d15428217e289ae36e5c3e6a8/include/openssl/ssl.h#L412).

Stunnel's option parsing uses 0 to indicate a parse error. Hence the process exits with Illegal TLS option.

As per the previous commit these were removed from OpenSSL in 2015 so probably the options can just be removed from the config.

Redis tunneling not working with Go programs?

Hi! Let me know if I'm doing something obviously wrong, but I've been trying to get a Go app up and running with Redis using this buildpack, and I'm never able to connect to REDIS_URL. At first I though it might be an issue with the redigo library, but it looks like it's actually lower level.

Steps to reproduce:

1. Create minimal Go app with code like https://play.golang.org/p/m8qfyWe3LZ and Procfile https://gist.github.com/488fb7dd761af5a4cdf3
2. Create heroku app with the heroku-redis addon
3. heroku buildpacks:set https://github.com/heroku/heroku-buildpack-redis
4. heroku buildpacks:add https://github.com/heroku/heroku-buildpack-go
5. heroku config:set STUNNEL_ENABLED=false (as a base case)

output:
https://gist.github.com/46c9a425dcad45a9f178 (expected)

1. heroku config:set STUNNEL_ENABLED=true

output:
https://gist.github.com/ee9ca9175030d1f6f0e4 (specifically "connection refused")

This was run on free-tier app and redis, but in production I was seeing the same error with Standard-1X worker dynos and Premium 5 heroku-redis.

Change detect name from "stunnel" to "Redis-stunnel"

On several occasions when looking at builds in a support ticket, I've been momentarily confused since the buildpacks URL list and the reported names in the build log don't match.

This buildpack reports its name as "stunnel" (which isn't entirely accurate, since it's not a generic stunnel implementation, even if one could get away with using it for some use-cases), but yet the URL only mentions redis (https://github.com/heroku/heroku-buildpack-redis).

As such I propose the detect name be changed from "stunnel" to say "Redis-stunnel".

Thoughts? :-)

heroku problem to conect from python app

Hi everyone. I recently upgraded the redis plan (free to Premium 0) of my staging app on heroku.
with this configuration:

celery==3.1.18
django==2.1.15
djangorestframework==3.9.0
gunicorn==19.8.1
kombu==3.0.37
redis==2.10.3

I am getting this error on sentry:

ConnectionError: null
File "django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "django/core/handlers/base.py", line 126, in _get_response
response = self.process_exception_by_middleware(e, request)
File "django/core/handlers/base.py", line 124, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "django/views/generic/base.py", line 68, in view
return self.dispatch(request, *args, **kwargs)
File "rest_framework/views.py", line 495, in dispatch
response = self.handle_exception(exc)
File "rest_framework/views.py", line 455, in handle_exception
self.raise_uncaught_exception(exc)
File "rest_framework/views.py", line 492, in dispatch
response = handler(request, *args, **kwargs)
File "rest_framework/generics.py", line 192, in post
return self.create(request, *args, **kwargs)
File "saf/apps/contratos/views.py", line 34, in create
self.perform_create(serializer)
File "rest_framework/mixins.py", line 26, in perform_create
serializer.save()
File "rest_framework/serializers.py", line 214, in save
self.instance = self.create(validated_data)
File "saf/apps/contratos/serializers.py", line 350, in create
club_lpf_destino=validated_data.get('club_lpf_destino'))
File "contextlib.py", line 52, in inner
return func(*args, **kwds)
File "saf/apps/contratos/service.py", line 81, in create
author=user, club=user.is_saf(), saf=user.is_club()
File "saf/apps/common/service.py", line 25, in _send_notifications
users=users_to_notify, title=title
File "saf/apps/notifications/service.py", line 34, in create
self.mobile_notificator.create(notification, mobile_users)
File "saf/apps/notifications/service.py", line 54, in create
self.send_notification.delay(payload)
File "celery/app/task.py", line 453, in delay
return self.apply_async(args, kwargs)
File "celery/app/task.py", line 559, in apply_async
**dict(self._get_exec_options(), **options)
File "celery/app/base.py", line 353, in send_task
reply_to=reply_to or self.oid, **options
File "celery/app/amqp.py", line 305, in publish_task
**kwargs
File "kombu/messaging.py", line 172, in publish
routing_key, mandatory, immediate, exchange, declare)
File "kombu/connection.py", line 470, in _ensured
interval_max)
File "kombu/connection.py", line 382, in ensure_connection
interval_start, interval_step, interval_max, callback)
File "kombu/utils/init.py", line 246, in retry_over_time
return fun(*args, **kwargs)
File "kombu/connection.py", line 250, in connect
return self.connection
File "kombu/connection.py", line 756, in connection
self._connection = self._establish_connection()
File "kombu/connection.py", line 711, in _establish_connection
conn = self.transport.establish_connection()
File "kombu/transport/virtual/init.py", line 809, in establish_connection
self._avail_channels.append(self.create_channel(self))
File "kombu/transport/virtual/init.py", line 791, in create_channel
channel = self.Channel(connection)
File "kombu/transport/redis.py", line 466, in init
self._disconnect_pools()
File "kombu/transport/redis.py", line 484, in _disconnect_pools
self._async_pool.disconnect()
File "redis/connection.py", line 898, in disconnect
connection.disconnect()
File "kombu/transport/redis.py", line 866, in disconnect
channel._on_connection_disconnect(self)
File "kombu/transport/redis.py", line 498, in _on_connection_disconnect
raise get_redis_ConnectionError()

Ability to provide a cert / PEM

Great buildback! Any plans to allow a PEM to be provided for stunnel via ENV so this buildpack could be used for things besides Heroku Redis without needing to fork it?

stunnel doesn't start fast enough

I'm currently having trouble setting this up, because my program connects to Redis right away, but it looks like stunnel is forked off into the background right away, and doesn't start in time for my program to try to connect:

buildpack=stunnel at=stunnel-enabled
buildpack=stunnel at=config-gen-start
Setting REDIS_URL_STUNNEL config var                                                                                                                                                                                                                         
buildpack=stunnel at=config-gen-end
buildpack=stunnel at=config-gen-override REDIS_URL
buildpack=stunnel at=stunnel-launched pid=18 signal=SIGINT
buildpack=stunnel at=stunnel-start
buildpack=stunnel at=app-launched pid=19
buildpack=stunnel at=app-start
time="2016-03-11T18:21:47Z" level=fatal msg="create worker crashed" err="dial tcp 127.0.0.1:6371: getsockopt: connection refused" pid=21 
2016.03.11 18:21:47 LOG5[20:140561750783808]: stunnel 4.53 on x86_64-pc-linux-gnu platform
buildpack=stunnel at=app-end
2016.03.11 18:21:47 LOG5[20:140561750783808]: Compiled with OpenSSL 1.0.1e 11 Feb 2013
buildpack=stunnel at=exit process=app
2016.03.11 18:21:47 LOG5[20:140561750783808]: Running  with OpenSSL 1.0.1f 6 Jan 2014
2016.03.11 18:21:47 LOG5[20:140561750783808]: Update OpenSSL shared libraries or rebuild stunnel
2016.03.11 18:21:47 LOG5[20:140561750783808]: Threading:PTHREAD SSL:+ENGINE+OCSP Auth:LIBWRAP Sockets:POLL+IPv6
2016.03.11 18:21:47 LOG5[20:140561750783808]: Reading configuration from file vendor/stunnel/stunnel.conf
2016.03.11 18:21:47 LOG5[20:140561750783808]: Configuration successful
buildpack=stunnel at=kill-app pid=19
bin/start-stunnel: line 49: kill: (-19) - No such process
buildpack=stunnel at=wait-app pid=19
buildpack=stunnel at=kill-aux name=stunnel pid=18 signal=SIGINT
buildpack=stunnel at=stunnel-end

Would it be a good idea to add a short sleep or somehow wait to make sure the stunnel is accepting connections before continuing with starting the actual app?

Consider overriding process entries

Should the buildpack automatically override process entries? This would eliminate any other changes than configuring the buildpack.

(I'm not sure if there is a significant penalty of always setting up the tunnel.)

Support Heroku Redis v6 rediss:// URLs

In new apps that use a premium-0 Redis addon (or higher) that is a Redis v6 addon, the url provided ends up being a rediss url. When combined with this addon, that should lead to a smooth connection. But the rediss url provides an accurate port number (not one that should be auto-bumped to the port one above, the behavior this buildpack currently expects and relies upon).

As seen in this PR:
opencounter@61c7ae6

It's possible to use the rediss url and not bump a URI port and in testing just now, I've confirmed that that works fine. But that commit isn't backwards compatible and so isn't likely to fit the needs of this buildpack.

For now, I'll be using the above buildpack (or a fork to fit our needs) but thought it worth writing up an issue as this snag had me scratching my head for a bit

Edited logs seen in debug mode when things are failing:

LOG7[ui]: Found 1 ready file descriptor(s)
LOG7[ui]: FD=4 events=0x2001 revents=0x0
LOG7[ui]: FD=9 events=0x2001 revents=0x1
LOG7[ui]: Service [REDIS_URL] accepted (FD=3) from 127.0.0.1:38934
LOG7[0]: Service [REDIS_URL] started
LOG7[0]: Option TCP_NODELAY set on local socket
LOG5[0]: Service [REDIS_URL] accepted connection from 127.0.0.1:38934
LOG6[0]: s_connect: connecting <host-ip>:13850
LOG7[0]: s_connect: s_poll_wait <host-ip>:13850: waiting 10 seconds
LOG3[0]: s_connect: connect <host-ip>:13850: Connection refused (111)
LOG3[0]: No more addresses to connect
LOG5[0]: Connection reset: 0 byte(s) sent to TLS, 0 byte(s) sent to socket
LOG7[0]: Local descriptor (FD=3) closed
LOG7[0]: Service [REDIS_URL] finished (0 left)

Edited logs seen when things go through properly using the above commit:

LOG7[ui]: Found 1 ready file descriptor(s)
LOG7[ui]: FD=4 events=0x2001 revents=0x0
LOG7[ui]: FD=9 events=0x2001 revents=0x1
LOG7[ui]: Service [REDIS_URL] accepted (FD=3) from 127.0.0.1:43782
LOG7[0]: Service [REDIS_URL] started
LOG7[0]: Option TCP_NODELAY set on local socket
LOG5[0]: Service [REDIS_URL] accepted connection from 127.0.0.1:43782
LOG6[0]: s_connect: connecting <host-ip>:13849
LOG7[0]: s_connect: s_poll_wait <host-ip>:13849: waiting 10 seconds
LOG5[0]: s_connect: connected <host-ip>:13849
LOG5[0]: Service [REDIS_URL] connected remote server from <remote-server-ip>:55492
LOG7[0]: Option TCP_NODELAY set on remote socket
LOG7[0]: Remote descriptor (FD=10) initialized
LOG6[0]: SNI: sending servername: <host-servername>
LOG6[0]: Peer certificate not required
LOG7[0]: TLS state (connect): before SSL initialization
LOG7[0]: TLS state (connect): SSLv3/TLS write client hello
LOG7[0]: TLS state (connect): SSLv3/TLS write client hello
LOG7[0]: TLS state (connect): SSLv3/TLS read server hello
LOG6[0]: Certificate verification disabled
LOG6[0]: Certificate verification disabled
LOG6[0]: Certificate verification disabled
LOG6[0]: Certificate verification disabled
LOG7[0]: TLS state (connect): SSLv3/TLS read server certificate
LOG7[0]: TLS state (connect): SSLv3/TLS read server key exchange
LOG6[0]: Client certificate not requested
LOG7[0]: TLS state (connect): SSLv3/TLS read server done
LOG7[0]: TLS state (connect): SSLv3/TLS write client key exchange
LOG7[0]: TLS state (connect): SSLv3/TLS write change cipher spec
LOG7[0]: TLS state (connect): SSLv3/TLS write finished
LOG7[0]: TLS state (connect): SSLv3/TLS write finished
LOG7[0]: TLS state (connect): SSLv3/TLS read server session ticket
LOG7[0]: TLS state (connect): SSLv3/TLS read change cipher spec
LOG7[0]: TLS state (connect): SSLv3/TLS read finished
LOG7[0]:      1 client connect(s) requested
LOG7[0]:      1 client connect(s) succeeded
LOG7[0]:      0 client renegotiation(s) requested
LOG7[0]:      0 session reuse(s)
LOG6[0]: TLS connected: new session negotiated
LOG7[0]: Peer certificate was cached (4616 bytes)

Support for Heroku-16 Stack

Hello,

We are using heroku-16 stack and the new change in REDIS_URL( dropping username 'h') left our apps broken once the scheduled redis maintenance is triggered.

Can we add it to the detect heroku-16 stack till it is still under EOL.

Enable certificate verification to prevent MITM attacks

Currently the TLS/SSL offering for Heroku Redis uses self-signed certificates, meaning the connection is still vulnerable to MITM attacks.

I've filed a ticket requesting that the certificates be signed (eg using a custom CA cert):
https://help.heroku.com/tickets/396267

If/when that's resolved, this buildpack needs to be updated, to:

• Specify the CA cert using the CAfile option
• Enable verification using verifyChain / verifyPeer options

This may also require a newer version of openssl/stunnel.

For more details, see:
https://www.stunnel.org/static/stunnel.html#SERVICE-LEVEL-OPTIONS

Document that the buildpack requires a production Heroku Redis plan

TLS isn't supported for the hobby/dev plans, and whilst this is mentioned here:
https://devcenter.heroku.com/articles/securing-heroku-redis

...it would be good to add this to the README too :-)