haproxytech / haproxy-consul-connect Goto Github PK
View Code? Open in Web Editor NEWHaProxy Connector for Consul Connect. Enables Service Mesh with Consul and HaProxy using TLS and Consul Discovery
License: Apache License 2.0
HaProxy Connector for Consul Connect. Enables Service Mesh with Consul and HaProxy using TLS and Consul Discovery
License: Apache License 2.0
I setup a test service in consul and tried starting this with the following command:
./haproxy-consul-connect -sidecar-for test_v1
HAProxy binary and dataplaneapi are in the path so it finds them. However, haproxy-consul-connect exits with this:
INFO[0000] consul: watching service test_v1
INFO[0000] consul: watching service test_v1-sidecar-proxy
INFO[0000] consul: leaf cert for service test changed, serial: 98:04, valid before: 2020-02-17 04:12:05 +0000 UTC, valid after: 2020-02-14 04:12:05 +0000 UTC
INFO[0000] consul: leaf cert for test ready
INFO[0000] consul: CA certs changed, active root id: 98:c3:ff:ca:a7:1a:f2:a5:ed:0b:04:3a:80:2c:62:a8:ec:35:3c:64
INFO[0000] consul: CA certs ready
INFO[0000] received consul config update
INFO[0000] handling new configuration
INFO[0000] haproxy: 045/195724 (15416) : New worker #1 (15417) forked
ERRO[0005] timeout waiting for dataplaneapi: error calling GET /v1/specification: response was 404: "{"code":404,"message":"path /v1/specification was not found"}"
INFO[0005] Shutting down because timeout waiting for dataplaneapi: error calling GET /v1/specification: response was 404: "{"code":404,"message":"path /v1/specification was not found"}"...
INFO[0005] cleaning config...
INFO[0005] killing dataplane-api with sig 15
INFO[0005] killing haproxy with sig 15
WARN[0005] haproxy: 045/195729 (15416) : Exiting Master process...
ERRO[0005] haproxy exited
ERRO[0005] dataplane-api exited
Noticed dataplaneapi now has /v2 instead of /v1. Also, it's now called dataplaneapi, not dataplane-api. Ofcourse, renamed dataplaneapi to dataplane-api go get this working.
Can anyone pls look into this?
Thanks!
The program is hard to deploy because it requires 2 additional binaries, ideally, it should:
Having to deploy 3 binaries is too boring
For now, the tests run at build time are limited to those not using dataplaneapi binary nor haproxy.
We need to test fully, so we need to modify Makefile to fetch HAProxy/dataplaneapi versions and to add it to the path to fully run tests.
@dcorbett-haproxy @aiharos How do you want to proceed?
Build from latest master.
When i run with -enable-intentions, this is what i see in the logs and connections are not going through. Works without using that flag but intentions are not honored.
ERRO[0018] error calling POST /v2/services/haproxy/configuration/filters?parent_type=frontend&parent_name=front_downstream&transaction_id=8eae2a31-e9c3-4d14-97c0-6a255c51c798: response was 422: "{"code":602,"message":"index in body is required"}"
INFO[0021] handling new configuration
ERRO[0021] error calling POST /v2/services/haproxy/configuration/filters?parent_type=frontend&parent_name=front_downstream&transaction_id=95c05b05-1380-47fb-9ca4-5ea7e7707e24: response was 422: "{"code":602,"message":"index in body is required"}"
INFO[0024] handling new configuration
ERRO[0024] error calling POST /v2/services/haproxy/configuration/filters?parent_type=frontend&parent_name=front_downstream&transaction_id=2d8d70f8-568c-409c-a555-f0c422bb5e5b: response was 422: "{"code":602,"message":"index in body is required"}"
INFO[0027] handling new configuration
ERRO[0027] error calling POST /v2/services/haproxy/configuration/filters?parent_type=frontend&parent_name=front_downstream&transaction_id=339c6bb3-14c2-487a-b092-75e234741fa6: response was 422: "{"code":602,"message":"index in body is required"}"
How do i get haproxy stats and logs? Do we need to use consul-template for those?
We very purposefully don't have a mode in Connect where you can choose to not enforce intentions. I'd strongly recommend not including one in this integration either. It's understandable if you needed a flag for initial testing or seeing what the overhead may be etc. but I would really prefer to keep away from adding more knobs that affect the security of any integration.
It's always possible to use permissive intentions that allow everything after all (that's even the default if you haven't setup restrictive ACLs). If you feel strongly that this is something you need to keep long term for testing the overhead of intention enforcement, could we at least make it on-by-default and Opt out (i.e. DisableIntentionChecking
) rather than opt -in?
Also, from what I can see the Certificate validation is being done in your SPOE handler and so presumably not in HAProxy at all which means that disabling SPOE is more than just disabling intention checks - it's disabling cert validation too. In other words it's a free for all, you don't even need a valid client cert, any old self-signed thing will do. That implies it should be named something more general like -disable-security
since it's not only intention enforcement but all of the mutual TLS benefits that go away.
If an option is kept for this I suggest that it needs very clear docs and sufficiently strong warnings about the precise way it weakens the threat model.
We should add some linting to the pipeline to help us catch issues with the code.
golangci-lint seems like a good choice, but I'm interested to hear if there are any other preferences before implementing this.
With Intentions enabled, the HAProxy configuration that's auto-generated contains this in the frontend
for the local service:
frontend front_downstream
mode http
bind 0.0.0.0:21000 name front_downstream_bind crt /tmp/haproxy-connect-002132058/e727ca4e64d85da74a3846094069b9bd80f41a92f36c5a16fd2417a1b2443f42 ca-file /tmp/haproxy-connect-002132058/6552a0a59b20ca0d1997c4e5e9afb8a5c3e31dc00aada13727733f742d79d4d5 ssl verify required
timeout client 30000
filter spoe engine intentions config /tmp/haproxy-connect-002132058/spoe.conf
tcp-request content accept if { var(sess.connect.auth) -m int eq 1 }
default_backend back_downstream
The tcp-request content accept
line will accept the connection if there is no "Deny" intention in Consul forbidding it. However, there is no rule that rejects the connection otherwise. So, all requests are accepted, no matter whether the are denied in Consul.
Workaround:
Editing the configuration so that tcp-request content accept if
is replaced with tcp-request content reject unless
fixes it:
tcp-request content reject unless { var(sess.connect.auth) -m int eq 1 }
I see some struct duplication from the ones in our api
package. Is there a good reason to need that vs just embedding our api package structs (say for upstreams or TLS configs) in your own wrapper that can add any additional context you want to pass through?
It might be reasonable as it is, I'm just thinking that as we add features especially with discovery chain stuff and some of our L7 config models it seems like wasted maintenance effort to keep up with copying more and more fields over to duplicate structs rather than just embedding ours.
Concretely, the biggest concerns there are the Config maps in Service.Proxy
and Upstream
and also the other fields in Upstream
like MeshGateway
. We'll be adding lots more config flags for things like timeouts, retries, circuit breaking etc. in to these places and as it stands now you'll have to explicitly add code to plumb them every time in the /consul package rather than just passing through and letting the /haproxy
package just map them to what it needs.
Introduced in Consul 1.6.
When Connect is enabled, the Consul UI has a tab that allows the user to create dynamic routes.
Could this be implemented in HAProxy-Consul-Connect?
https://www.consul.io/docs/connect/config-entries/service-router
Consul's Layer 7 routing feature: https://www.consul.io/docs/connect/l7-traffic
haproxy-consul-connect/main.go
Lines 57 to 59 in 55bff42
This is a naming convention only and there is nothing in Consul stopping users from naming proxies differently if they don't use the sidecar_service helper.
The canonical way to tell if something is a Proxy in Connect's data model is if service.Kind == "connect-proxy"
. If it's important that it's specifically a sidecar proxy (not say an ingress loadbalancer) which is true in this case you need to also check for service.Proxy.DestinationServiceID != ""
- The thing that canonically makes a Proxy service a sidecar is that field indicating that it is has exactly one app (destination) instance behind it.
So to correctly skip sidecar proxies in this loop the following should work:
// Skip all sidecar proxy registrations
if s.Service.Kind == api.ServiceKindConnectProxy && s.Service.Proxy.DestinationServiceID != "" {
continue
}
NOTE: In fact this whole loop might need to change making this issue moot. I'll explain more in the issue that I'll link below, but this was important to note to make sure the guarantees are understood here.
Is there a working example or a link that I can check? Need an example with sidecar proxy and HAProxy-consul-connect working together. Thanks for the help!
Hi!
Is HAProxy as a sidecar compatible with L7 traffic management features relatively recently added to Consul?
If not but it is possible, then do you plan to implement it?
Thanks!
Hi!
As you have noticed I have started to contribute to this project. I am doing that because I am POCing a service mesh solution for Egnyte that is using this project.
To make it work, apart from #51 and #59, I would also need a few more features, so I would like to ask how do they align with your plans - perhaps you are already working on some of them or on the contrary: you will not even accept PRs with them. Let's have a chat about it. :)
So here it goes, in no specific order:
So what do you think?
Currently, the SPOA that checks Intentions is giving me an error when a request is received:
echo_1 | time="2019-09-18T20:02:40Z" level=info msg="spoe: hello from @: map[capabilities:pipelining engine-id:54DB81E3-F4EA-4899-BE8D-5F6F1EC62A1A max-frame-size:16380 supported-versions:2.0]"
echo_1 | time="2019-09-18T20:02:40Z" level=error msg="spoe: error handling connection: hello: expected capabilities [async pipelining]"
echo_1 | 127.0.0.1:52874 | GET /
echo_1 | time="2019-09-18T20:02:40Z" level=info msg="spoe: hello from @: map[capabilities:pipelining engine-id:54DB81E3-F4EA-4899-BE8D-5F6F1EC62A1A max-frame-size:16380 supported-versions:2.0]"
While trying to add tests for Consul, I discovered that the client's binary API has been modified.
Apply the fixes done in hashicorp/consul#7637 to ensure it works with recent versions of Consul APIs
Added initial tests on Consul side: hashicorp/consul#7637
We need to find a way too sync both
Note if this is changed it might make #7 invalid.
haproxy-consul-connect/main.go
Lines 61 to 68 in 55bff42
Having skipped sidecars (assuming that works, see #7), this code attempts to find the first tag match on an actual service instance. But if it fails it returns an error about not finding the sidecar proxy which is confusing since it might have been there you just skipped it.
I think I understand the intent and it's rely on the existing sidecarFor
lookup code copied from Consul (great - was going to recommend that part), but as it is there are a lot of confusing ways this can break:
LookupProxyIDForSidecar
in the Consul command code does.I suggest it might be easier if all the lookup stuff was moved out of the /consul package and into the main.go so that you can handle all this resolving stuff in one place and just start the main body of the work with a proxy's ID already resolved or all argument errors already handled with messages that are clear to the user where there is ambiguity.
haproxy-consul-connect/haproxy/spoe.go
Lines 45 to 47 in 556c75e
Verifying certificates is not providing intermediates that may be present. That means it will only work in the Primary DC and even then it will only work for CA providers that choose to sign directly with their root key. (In the future this will likely be zero providers!).
The easiest thing to do is to follow the code in our SDK here: https://github.com/hashicorp/consul/blob/fd3c56ff68829821da4be139185c0a96938e1929/connect/tls.go#L265
At a high level it:
Can the build steps be documented pls?
I downloaded the zip from maser and i got this:
go test -timeout 30s ./...
go: github.com/go-openapi/[email protected] requires
github.com/globalsign/[email protected]: invalid version: git fetch --unshallow -f https://github.com/globalsign/mgo in /root/go/pkg/mod/cache/vcs/0eb69f2163bb2fb915dbc4b9c83a68ab02240cc5c1aaa829e77cb6f9fd51e98d: exit status 128:
fatal: git fetch-pack: expected shallow list
make: *** [test] Error 1
Am sure i'm doing something wrong. I'm just running "make"
go version go1.13.6 linux/amd64
https://github.com/haproxytech/haproxy-consul-connect/blob/master/consul/watcher.go
I see a couple of issues in the watcher:
removeUpstream
is setting a done bool under lock but the startUpstream loop reads that bool while not holding the lock which is a data race.TestAgent
is that this pattern leads to really inefficient test runs with tons of orphan goroutines by the end etc. I'd recommend plumbing a Stop mechanism or Context through every goroutine spawned right from the start so all resources can be cleaned up in tests and in case you ever need to support managing multiple proxies or decoupling process lifetime and proxy lifetime etc. It's way easier if it's just there already!This is partly a suggestion about clarifying security model docs and partly a suggestion to improve the overall security of this integration if I'm understanding the current model correctly.
The diagram in the README is fantastic for showing how this integration works. It's not currently clearly stating the threat model.
As far as I can see, the haproxy-connect process is the one that is authenticated by an ACL to fetch certificates. Which way does the dataplane API connection work though? And from what I can see it talks to HAProxy dataplane as a client which means it doesn't expose an open interface that another process on the box can connect to and get access to certificates/keys? If I'm right that sounds good.
One potential issue I see though is that it uses a hard-coded dataplane user/pass for HAProxy:
haproxy-consul-connect/haproxy/config.go
Lines 16 to 17 in 556c75e
This means that any other process on the box that can talk on that same unix socket and happens to know these credentials (it is open source after all) can completely reconfigure the proxy, for example to disable SPOE intention checking.
One possible simple solution would be to generate a strong crypto-random password for HAProxy on every startup so it's only in-memory in the consul-haproxy process and passed through to HAProxy. I guess the Config is written to disk so I'd suggest using the secure (hashed) password config to pass it through and hashing the random password in consul-haproxy. That way you can have high confidence that the only process able to configure the proxy is same consul-haproxy process that stated it.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.