Coder Social home page Coder Social logo

aws-es-proxy's People

Contributors

justinsb avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

aws-es-proxy's Issues

Kibana in Kubernetes cluster cannot talk to AWS ES using the proxy

Currently attempting to do the following:

  • Replace the existing elasticsearch-logging service with this proxy. (this works)
  • Have fluentd log directly to my AWS ES cluster using the proxy (this works)
  • Have an in-cluster install of Kibana talk to AWS ES cluster using this proxy (this seems to have issues)

I had to also do the following:

  • Upgrade Kibana to a version that supports ES 2.3 (what I'm running in AWS ES service). Kibana 4.5.4 is the version that I chose.
  • Configure Kibana to allow me to use it via Kubernetes proxy (this works by setting server.basePath to /api/v1/proxy/namespaces/kube-system/services/kibana-logging. This setting is new(ish) to Kibana versions greater than 4.1.11.

At this point I have logs being sent to AWS ES service via fluentd, however, when I load the Kibana UI I get the following error:

Courier Fetch Error: unhandled courier request error: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
The Canonical String for this request should have been
'POST
/_mget
ignore_unavailable=true&preference=1473356469111&timeout=0
accept:application/json, text/plain, */*
accept-encoding:gzip, deflate, br
accept-language:en-US,en;q=0.8
content-length:62
content-type:application/json;charset=UTF-8
host:search-qa-test-xxxxxxxxxx.us-east-1.es.amazonaws.com
kbn-version:4.5.4
origin:https://10.x.x.x
referer:https://10.x.x.x/api/v1/proxy/namespaces/kube-system/services/kibana-logging/app/kibana
x-amz-date:20160908T174109Z
x-amz-security-token:xxxxxxxxx
x-forwarded-port:443
x-forwarded-uri:/api/v1/proxy/namespaces/kube-system/services/kibana-logging/elasticsearch/_mget

accept;accept-encoding;accept-language;content-length;content-type;host;kbn-version;origin;referer;x-amz-date;x-amz-security-token;x-forwarded-port;x-forwarded-uri
xxxxxxx'

The String-to-Sign should have been
'AWS4-HMAC-SHA256
20160908T174109Z
20160908/us-east-1/es/aws4_request
xxxxxxxx'

documentation

Hi Guys,

I would really want to use this proxy on my KOPS provisioned k8s cluster.
however, readme.md file will not help me there.
I see that people can confirm it works, but I couldn't hack it.
specifically:
I had created an ES cluster in AWS
changed elasticsearch-logging image in spec to be: kope/aws-es-proxy
added this to env:
- name: "AWS_REGION"
value: "us-east-1"
- name: "ES"
value: "http://search-[redacted].us-east-1.es.amazonaws.com/"
fluentd is running
proxy looks healthy (main.go:69] Listening on :9200)

no index on ES

what am I missing?

use glide for dependency locking

currently the latest version of dependencies will be pulled in.

use glide for version locking to make sure, aws-es-proxy can be built reproducable

Doesn't handle Expect: 100-Continue

When e.g. curl sends a request with content length greater than 1024, it also sends an Expect: 100-Continue header.

aws-es-proxy returns the 100 header, but probably does not pass it along to ES, causing a mismatch. Sample trace:

=> Send header, 201 bytes (0xc9)
0000: 50 55 54 20 2f 5f 74 65 6d 70 6c 61 74 65 2f 6b PUT /_template/k
[...]
0040: 30 30 0d 0a 55 73 65 72 2d 41 67 65 6e 74 3a 20 00..User-Agent: 
0050: 63 75 72 6c 2f 37 2e 34 33 2e 30 0d 0a 41 63 63 curl/7.43.0..Acc
0060: 65 70 74 3a 20 2a 2f 2a 0d 0a 43 6f 6e 74 65 6e ept: */*..Conten
0070: 74 2d 4c 65 6e 67 74 68 3a 20 31 31 33 34 0d 0a t-Length: 1134..
0080: 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 61 70 Content-Type: ap
0090: 70 6c 69 63 61 74 69 6f 6e 2f 78 2d 77 77 77 2d plication/x-www-
00a0: 66 6f 72 6d 2d 75 72 6c 65 6e 63 6f 64 65 64 0d form-urlencoded.
00b0: 0a 45 78 70 65 63 74 3a 20 31 30 30 2d 63 6f 6e .Expect: 100-con
00c0: 74 69 6e 75 65 0d 0a 0d 0a                      tinue....
<= Recv header, 23 bytes (0x17)
0000: 48 54 54 50 2f 31 2e 31 20 31 30 30 20 43 6f 6e HTTP/1.1 100 Con
0010: 74 69 6e 75 65 0d 0a                            tinue..
=> Send data, 1134 bytes (0x46e)
0000: 7b 20 20 22 74 65 6d 70 6c 61 74 65 22 3a 20 22 {  "template": "

Error:

The Canonical String for this request should have been
'PUT
/_template/k[...]

accept:*/*
content-length:1134
content-type:application/x-www-form-urlencoded
expect:
host:[...]
[...]

Maintained?

Hi,

I've used https://github.com/abutaha/aws-es-proxy before but it has some internal design issues and when I ran into problems it became hard to debug, so I found this here and find it much nicer.

Is this still maintained? Otherwise I'd fork it and commit to maintain it at least on a best effort basis.
If it's still maintained, can you provide an official docker image for this? That would be great!

Validate end point URL

The URL associated with the end point should be validated and a warning logged if its format is not correct.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.