kopeio / aws-es-proxy Goto Github PK
View Code? Open in Web Editor NEWSimple proxy that signs requests for AWS ElasticSearch service
Simple proxy that signs requests for AWS ElasticSearch service
Currently attempting to do the following:
elasticsearch-logging
service with this proxy. (this works)I had to also do the following:
4.5.4
is the version that I chose.server.basePath
to /api/v1/proxy/namespaces/kube-system/services/kibana-logging
. This setting is new(ish) to Kibana versions greater than 4.1.11
.At this point I have logs being sent to AWS ES service via fluentd, however, when I load the Kibana UI I get the following error:
Courier Fetch Error: unhandled courier request error: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
The Canonical String for this request should have been
'POST
/_mget
ignore_unavailable=true&preference=1473356469111&timeout=0
accept:application/json, text/plain, */*
accept-encoding:gzip, deflate, br
accept-language:en-US,en;q=0.8
content-length:62
content-type:application/json;charset=UTF-8
host:search-qa-test-xxxxxxxxxx.us-east-1.es.amazonaws.com
kbn-version:4.5.4
origin:https://10.x.x.x
referer:https://10.x.x.x/api/v1/proxy/namespaces/kube-system/services/kibana-logging/app/kibana
x-amz-date:20160908T174109Z
x-amz-security-token:xxxxxxxxx
x-forwarded-port:443
x-forwarded-uri:/api/v1/proxy/namespaces/kube-system/services/kibana-logging/elasticsearch/_mget
accept;accept-encoding;accept-language;content-length;content-type;host;kbn-version;origin;referer;x-amz-date;x-amz-security-token;x-forwarded-port;x-forwarded-uri
xxxxxxx'
The String-to-Sign should have been
'AWS4-HMAC-SHA256
20160908T174109Z
20160908/us-east-1/es/aws4_request
xxxxxxxx'
Hi Guys,
I would really want to use this proxy on my KOPS provisioned k8s cluster.
however, readme.md file will not help me there.
I see that people can confirm it works, but I couldn't hack it.
specifically:
I had created an ES cluster in AWS
changed elasticsearch-logging image in spec to be: kope/aws-es-proxy
added this to env:
- name: "AWS_REGION"
value: "us-east-1"
- name: "ES"
value: "http://search-[redacted].us-east-1.es.amazonaws.com/"
fluentd is running
proxy looks healthy (main.go:69] Listening on :9200)
no index on ES
what am I missing?
currently the latest version of dependencies will be pulled in.
use glide for version locking to make sure, aws-es-proxy can be built reproducable
When e.g. curl sends a request with content length greater than 1024, it also sends an Expect: 100-Continue header.
aws-es-proxy returns the 100 header, but probably does not pass it along to ES, causing a mismatch. Sample trace:
=> Send header, 201 bytes (0xc9)
0000: 50 55 54 20 2f 5f 74 65 6d 70 6c 61 74 65 2f 6b PUT /_template/k
[...]
0040: 30 30 0d 0a 55 73 65 72 2d 41 67 65 6e 74 3a 20 00..User-Agent:
0050: 63 75 72 6c 2f 37 2e 34 33 2e 30 0d 0a 41 63 63 curl/7.43.0..Acc
0060: 65 70 74 3a 20 2a 2f 2a 0d 0a 43 6f 6e 74 65 6e ept: */*..Conten
0070: 74 2d 4c 65 6e 67 74 68 3a 20 31 31 33 34 0d 0a t-Length: 1134..
0080: 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 61 70 Content-Type: ap
0090: 70 6c 69 63 61 74 69 6f 6e 2f 78 2d 77 77 77 2d plication/x-www-
00a0: 66 6f 72 6d 2d 75 72 6c 65 6e 63 6f 64 65 64 0d form-urlencoded.
00b0: 0a 45 78 70 65 63 74 3a 20 31 30 30 2d 63 6f 6e .Expect: 100-con
00c0: 74 69 6e 75 65 0d 0a 0d 0a tinue....
<= Recv header, 23 bytes (0x17)
0000: 48 54 54 50 2f 31 2e 31 20 31 30 30 20 43 6f 6e HTTP/1.1 100 Con
0010: 74 69 6e 75 65 0d 0a tinue..
=> Send data, 1134 bytes (0x46e)
0000: 7b 20 20 22 74 65 6d 70 6c 61 74 65 22 3a 20 22 { "template": "
Error:
The Canonical String for this request should have been
'PUT
/_template/k[...]
accept:*/*
content-length:1134
content-type:application/x-www-form-urlencoded
expect:
host:[...]
[...]
Hi,
I've used https://github.com/abutaha/aws-es-proxy before but it has some internal design issues and when I ran into problems it became hard to debug, so I found this here and find it much nicer.
Is this still maintained? Otherwise I'd fork it and commit to maintain it at least on a best effort basis.
If it's still maintained, can you provide an official docker image for this? That would be great!
The URL associated with the end point should be validated and a warning logged if its format is not correct.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.