inreachventures / aws-signing-request-interceptor Goto Github PK
View Code? Open in Web Editor NEWRequest Interceptor for Apache Client that signs the request for AWS
License: MIT License
Request Interceptor for Apache Client that signs the request for AWS
License: MIT License
Adding :80
to the host in the ES server URL breaks signing. Amazon returns an error showing what they used to compute the signature and the port is missing, so the request fails.
This took me a while to pin down but glad it turned out to be something easy to fix (on my end at least).
Thanks for this helpful interceptor!
Sometimes (don't ask me about it, sorry), the Signage might break due to the "Connection" header:
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'PUT\n/user/user/aldrinleal2\n\naccept-encoding:gzip,deflate\nconnection:close\ncontent-length:107\ncontent-type:application/json; charset=UTF-8\nhost:search-wechase-dev-65phvknk3pxuhxrqzpr4hggmvq.us-east-1.es.amazonaws.com\nuser-agent:Apache-HttpClient/4.5.2 (Java/1.8.0_101)\nx-amz-date:20160823T135045Z\n\naccept-encoding;connection;content-length;content-type;host;user-agent;x-amz-date\n9192d051542ec4df30c816b8b59d9134d21f6d90878c4f45777b825c9bb9389f'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20160823T135045Z\n20160823/us-east-1/es/aws4_request\n77575171872045c07f741778e0b18f20121695938077b02519201f81b8187064'\n"}
The shortest change to allow this would be to add this line:
final LocalDateTime now = clock.get();
final AWSCredentials credentials = credentialsProvider.getCredentials();
final Map<String, Object> result = new TreeMap<>(String.CASE_INSENSITIVE_ORDER);
result.putAll(headers);
result.remove("Connection"); // THIS ONE
Hope it helps.
Is the below correct? The documentation is a bit vague...
What is the service value?
private static final String SERVICE = "es";
private String clusterUri;
@Bean
public JestClient jestClient() throws Exception {
final Supplier<LocalDateTime> clock = new Supplier<LocalDateTime>() {
public LocalDateTime get() {
return LocalDateTime.now();
}
};
DefaultAWSCredentialsProviderChain defaultAWSCredentialsProviderChain = new DefaultAWSCredentialsProviderChain();
Region region = Region.getRegion(Regions.fromName("ap-southeast-2"));
final AWSSigner awsSigner = new AWSSigner(defaultAWSCredentialsProviderChain, region.getName(),
SERVICE, clock);
final AWSSigningRequestInterceptor requestInterceptor = new AWSSigningRequestInterceptor(awsSigner);
final JestClientFactory factory = new JestClientFactory() {
@Override
protected HttpClientBuilder configureHttpClient(HttpClientBuilder builder) {
builder.addInterceptorLast(requestInterceptor);
return builder;
}
@Override
protected HttpAsyncClientBuilder configureHttpClient(HttpAsyncClientBuilder builder) {
builder.addInterceptorLast(requestInterceptor);
return builder;
}
};
factory.setHttpClientConfig(new HttpClientConfig
.Builder(clusterUri)
.multiThreaded(true)
.build());
return factory.getObject();
}
Hi @ericcitaire ,
I was using this library to connect to AWS ES using IAM role. I have added role to my application and then created JestClient using AWS ES endpoint, please refer below code to create JestClient.
public SElasticSearchClient(final String esEndpoint) { this.esEndpoint = esEndpoint; final Supplier<LocalDateTime> clock = () -> LocalDateTime.now(ZoneOffset.UTC); AWSCredentialsProviderChain awsCredentialsProvider = new DefaultAWSCredentialsProviderChain(); final AWSSigner awsSigner = new AWSSigner(awsCredentialsProvider, "us-east-1", "es", clock); final JestClientFactory factory = new JestClientFactory() { @Override protected HttpClientBuilder configureHttpClient(HttpClientBuilder builder) { builder.addInterceptorLast(new AWSSigningRequestInterceptor(awsSigner)); return builder; } }; this.jestClient = initJestClient(factory); }
When I deployed this code and run my application it says error 403 user doesn't have access.
Error Message:
{ "Message": "User: arn:aws:sts::*****:assumed-role/search.bulk_ingest/DiscoveryDeAggregator-test is not authorized to perform: es:ESHttpPost on resource: discovery-content2" }
Please let me know what I am missing in code to create JestClient.
Any help will really appreciated
Thanks,
Anish
I work in a limited environment where the latest Java I can use is 7 (actually, it's a C program calling into Java). Is it possible to port this lib back to Java 7?
When a URL like http://host/path?a=b+c
is signed, the signature is wrong.
aws-signing-request-interceptor version 0.0.15
jest version 2.0.4
Search via JestClient with index name of log-message-*
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "POST /log-message-*/log-message/_search HTTP/1.1[\r][\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Accept-Encoding: gzip,deflate[\r][\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKIAJFBO2C537SOTMKQQ/20170126/us-west-2/es/aws4_request, SignedHeaders=accept-encoding;content-length;content-type;host;user-agent;x-amz-date, Signature=fab5bd06f69d72bec5aa66d2e2da06123e8950d5a20f27ba0e7c52d0a45d0e86[\r][\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Connection: Keep-Alive[\r][\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Content-Length: 201[\r][\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Content-Type: application/json; charset=UTF-8[\r][\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Host: search-iscs-logging2-rbzonthuzd4o6ynunvaplwt4i4.us-west-2.es.amazonaws.com[\r][\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_45)[\r][\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "x-amz-date: 20170126T212513Z[\r][\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "[\r][\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "[\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "{[\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> " "query": {[\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> " "match_phrase": {[\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> " "logFile": "test:test:test-hostname:rest-test-hostname.api.2017-01-26.21-25-02.bbc65439-2e21-4ee6-a4fd-90c508e85086.txt:2017-01-26T21:25:04.636Z"[\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> " }[\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> " }[\n]"
15:25:13.502 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "}"
15:25:13.598 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "HTTP/1.1 403 Forbidden[\r][\n]"
15:25:13.598 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "Content-Type: application/json[\r][\n]"
15:25:13.598 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "x-amzn-RequestId: ec88e0db-e40d-11e6-bdf7-0be252654144[\r][\n]"
15:25:13.598 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "Content-Length: 877[\r][\n]"
15:25:13.598 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "Connection: keep-alive[\r][\n]"
15:25:13.598 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "[\r][\n]"
15:25:13.599 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'POST\n/log-message-%2A/log-message/_search\n\naccept-encoding:gzip,deflate\ncontent-length:201\ncontent-type:application/json; charset=UTF-8\nhost:search-iscs-logging2-rbzonthuzd4o6ynunvaplwt4i4.us-west-2.es.amazonaws.com\nuser-agent:Apache-HttpClient/4.5.2 (Java/1.8.0_45)\nx-amz-date:20170126T212513Z\n\naccept-encoding;content-length;content-type;host;user-agent;x-amz-date\nb279691bba767476d3f034826b94c64dafa3d888c775eda7582e9cb9d8a8932d'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20170126T212513Z\n20170126/us-west-2/es/aws4_request\n9d44dbc4f0ca90dc8f719077c1582dc27d73a3a5459c341872bf7c352f5559ed'\n"}"
Same search with index name of log-message-x
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "POST /log-message-x/log-message/_search HTTP/1.1[\r][\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Accept-Encoding: gzip,deflate[\r][\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKIAJFBO2C537SOTMKQQ/20170126/us-west-2/es/aws4_request, SignedHeaders=accept-encoding;content-length;content-type;host;user-agent;x-amz-date, Signature=fe965a263ae8164c0ea951a41b188dd127a05afd859caf3cb071508188501046[\r][\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Connection: Keep-Alive[\r][\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Content-Length: 201[\r][\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Content-Type: application/json; charset=UTF-8[\r][\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Host: search-iscs-logging2-rbzonthuzd4o6ynunvaplwt4i4.us-west-2.es.amazonaws.com[\r][\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_45)[\r][\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "x-amz-date: 20170126T212753Z[\r][\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "[\r][\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "[\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "{[\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> " "query": {[\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> " "match_phrase": {[\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> " "logFile": "test:test:test-hostname:rest-test-hostname.api.2017-01-26.21-27-36.a1726ebb-1845-4b49-82af-791662650e0c.txt:2017-01-26T21:27:38.901Z"[\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> " }[\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> " }[\n]"
15:27:53.251 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "}"
15:27:53.338 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "HTTP/1.1 404 Not Found[\r][\n]"
15:27:53.339 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "Access-Control-Allow-Origin: *[\r][\n]"
15:27:53.339 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "Content-Type: application/json; charset=UTF-8[\r][\n]"
15:27:53.339 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "Content-Length: 335[\r][\n]"
15:27:53.339 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "Connection: keep-alive[\r][\n]"
15:27:53.339 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "[\r][\n]"
15:27:53.339 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"log-message-x","index":"log-message-x"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"log-message-x","index":"log-message-x"},"status":404}"
When making requests to S3, the requests have to contain x-amz-content-sha256 in signed header
https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html
Hi nice work!
i'm using your library to check and initialise an index on a ES and i'm having the following cast exception:
Caused by: java.lang.ClassCastException: org.apache.http.client.methods.HttpHead cannot be cast to org.apache.http.HttpEntityEnclosingRequest
at vc.inreach.aws.request.AWSSigningRequestInterceptor.body(AWSSigningRequestInterceptor.java:59)
at vc.inreach.aws.request.AWSSigningRequestInterceptor.process(AWSSigningRequestInterceptor.java:32)
at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:182)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:48)
at com.draios.services.ElasticSearchService.initSchema(ElasticSearchService.java:99)
at com.draios.services.ElasticSearchService.<init>(ElasticSearchService.java:77)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
here is the code:
private void initSchema(JestClient esClient, String esIndex, String esEndpoint) throws IOException {
boolean indexExists = esClient.execute(new IndicesExists.Builder(esIndex).build()).isSucceeded();
if (indexExists) {
logger.info("Index {} is already configured on {}", esIndex,esEndpoint);
}else {
logger.info("Index {} is not configured on {} creating", esIndex,esEndpoint);
esClient.execute(new CreateIndex.Builder(esIndex).build());
}
}
using my a local elasticsearch node as endpoint it is working properly.
I have created repository with bug reproducing.
This repo contains a maven project with 2 java classes NonWorkingExampleStreamVersion, WorkingExampleNonStreamVersion. Both of them has main method. Both take file from s3 and try to store it in aws elasticsearch. But:
WorkingExampleNonStreamVersion - load whole file content into byte stream, creates base64 from it and then send file to elasticsearch
NonWorkingExampleStreamVersion - do the same thins as previous but instead of downloading whole file it wrap stream and uploads bytes to elaticsearch only when those byte arrives from s3. This version is does not work due to:"message": "The request signature we calculated does not match the signature you provided. Check > your AWS Secret Access Key and signing method. Consult the service documentation for details.
More details you can see on reaadme.md file. Feel free to ask me any questions.
All the dependencies of this project has security vulnerabilities to some degree.
We've been able to dependency manage most of these but guava is not one I think we can manage.
Can you upgrade the dependencies to the latest and re-release?
Our security scanner recommends us migrate to guava 31.0.1-jre which to follow semantic versioning would require a major version bump of your libary.
When a query parameter value contains an asterisks, the request signing fails and AWS returns an error:
The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
The reason is that AWS expects more characters in the URL to be encoded than what is necessary at the minimum. The Java SDK URLEncoder does not encode the asterisks. Here is the corresponding code from the AWS SDK: https://github.com/aws/aws-sdk-java/blob/7e064a48bee151ef222e0edea6157c1e65c7904a/aws-java-sdk-core/src/main/java/com/amazonaws/util/SdkHttpUtils.java#L66
Hi
I raised an issue in StackOverflow we follow your project for AWS Signing and we have issues in PutMapping and Create Records on AWS Signing.
We are creating index, put mapping and insert record and rollback by deleting index in exceptions case. we faced an 403 forbidden error in PUTMapping and insertrecords This 403 forbidden error not raised for Index create and Delete. it looks like weird behaviour.
We are executing this Java Program as AWS Lambda. We provide Full Permission to ES in AWS Lambda IAM ROLE.
We are using aws-java-sdk (1.11.33), vc.inreach.aws (0.0.15), Jest Client (2.0.2) and AWS ES 2.3.
Attached our Code Part
HttpClientConfig clientConfig = new HttpClientConfig.Builder(esEndPoint).multiThreaded(true).build();
JestClientFactory factory = new JestClientFactory();
factory.setHttpClientConfig(clientConfig);
JestClient jestClient = factory.getObject();
System.out.println("Final Es Endpoint " + esEndPoint + "/"+indexName);
//Check Index Exist in Elastic Search
boolean indexExists = jestClient.execute(new IndicesExists.Builder(indexName).build()).isSucceeded();
if (indexExists)
esUtils.insertESData(jestClient, esRecords, indexName, docType);
else
{
boolean isCreated = esUtils.createESIndex(jestClient, indexName, docType);
if(isCreated)
esUtils.insertESData(jestClient, esRecords,indexName,docType);
else
System.out.println("Record Insertion failure ");
}
**Esutils Class**
public class ESUtils {
public boolean createESIndex(JestClient jestClient, String indexName, String docType) throws Exception
{
System.out.println("ESUtils createESIndex " + indexName + "-" + docType);
String date_prop1 = "{\"type\" : \"date\", \"format\" : \"" + Constants.SYSTEM_DATE_TIME_FORMAT1 + "\"}";
String date_prop2 = "{\"type\" : \"date\", \"format\" : \"" + Constants.SYSTEM_DATE_TIME_FORMAT2 +
"||" + Constants.SYSTEM_DATE_TIME_FORMAT3 + "\"}";
String timestamp_prop1 = "{\"type\" : \"date\", \"format\" : \"epoch_millis\"}";
String timestamp_prop2 = "{\"type\" : \"date\", \"format\" : \"hour_minute_second_millis\"}";
String map_properties = "{ \"date\" : " + date_prop1 +
", \"messageDate\" : " + date_prop2 +
", \"timestamp\" : " + timestamp_prop1 + " } ";
/*", \"messageTime\" : " + timestamp_prop2 + */
boolean esIndexFlag;
try {
JestResult createResult = jestClient.execute(new CreateIndex.Builder(indexName).build());
if(!createResult.isSucceeded())
{
System.out.printf("ESUtils createESIndex : JestResult error message %s and response code %s",
createResult.getErrorMessage(), createResult.getResponseCode());
esIndexFlag = false;
}
else if(createResult.isSucceeded())
{
System.out.println("Create index"+createResult.getResponseCode());
PutMapping putMapping = new PutMapping.Builder(
indexName, docType, "{ \"" + docType + "\" : { \"properties\" : " + map_properties+"} }").build();
JestResult map_result = jestClient.execute(putMapping);
if(map_result.isSucceeded())
System.out.println("ESUtils createESIndex : Mapping Created");
else
{
System.out.println("ESUtils createESIndex : Mapping not Created" + map_result.getErrorMessage());
JestResult delete_result = jestClient.execute(new DeleteIndex.Builder(indexName).build());
if(delete_result.isSucceeded())
System.out.println("ESUtils createESIndex : Index Deleted");
else
System.out.println("ESUtils createESIndex : Index not Deleted"+delete_result.getErrorMessage());
}
esIndexFlag = true;
}
else
{
esIndexFlag = false;
System.out.println("ESUtils createESIndex : Index not created");
}
}
catch(Exception esExp)
{
esIndexFlag = false;
System.out.println("ESUtils createESIndex : Index create Exception :" + esExp);
}
return esIndexFlag;
}
public boolean insertESData(JestClient jestClient, Map<String, Object> esRecords, String indexName, String docType){
try {
Index index = new Index.Builder(esRecords).index(indexName).type(docType).build();
JestResult result = jestClient.execute(index);
if(result.isSucceeded())
{
System.out.println("ESUtils insertESData Success Message : " + result.getJsonString());
return true;
}
else
{
System.out.println("ESUtils insertESData Response code and Error message : "
+ result.getResponseCode() + " : " + result.getErrorMessage());
return false;
}
} catch (Exception exp) {
System.out.println("ESUtils insertESData Exception :" + exp);
return false;
}
}
}
The Error we are getting for PutMapping and InsertRecords are 403 forbidden error.
Here is the DEBUG-level logging from an example:
2016-08-07 19:08:49 DEBUG headers:135 - http-outgoing-0 >> POST /message/post/_search?search_type=scan&size=100&scroll=5m HTTP/1.1
2016-08-07 19:08:49 DEBUG headers:138 - http-outgoing-0 >> Accept-Encoding: gzip,deflate
2016-08-07 19:08:49 DEBUG headers:138 - http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAJU34EHB4XVGMOWVA/20160808/us-east-1/es/aws4_request, SignedHeaders=accept-encoding;connection;content-length;content-type;host;user-agent;x-amz-date, Signature=64d5917fa3d7905fb9852b55f3e7cf061ae6e2a5804cc303845790c97a3aab44
2016-08-07 19:08:49 DEBUG headers:138 - http-outgoing-0 >> Connection: Keep-Alive
2016-08-07 19:08:49 DEBUG headers:138 - http-outgoing-0 >> Content-Length: 26
2016-08-07 19:08:49 DEBUG headers:138 - http-outgoing-0 >> Content-Type: application/json; charset=UTF-8
2016-08-07 19:08:49 DEBUG headers:138 - http-outgoing-0 >> Host: search-hydra-mdl-hu7plzwptok3eip6jkx22r4nxq.us-east-1.es.amazonaws.com
2016-08-07 19:08:49 DEBUG headers:138 - http-outgoing-0 >> User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0)
2016-08-07 19:08:49 DEBUG headers:138 - http-outgoing-0 >> x-amz-date: 20160808T000849Z
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 >> "POST /message/post/_search?search_type=scan&size=100&scroll=5m HTTP/1.1[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKIAJU34EHB4XVGMOWVA/20160808/us-east-1/es/aws4_request, SignedHeaders=accept-encoding;connection;content-length;content-type;host;user-agent;x-amz-date, Signature=64d5917fa3d7905fb9852b55f3e7cf061ae6e2a5804cc303845790c97a3aab44[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 >> "Content-Length: 26[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 >> "Content-Type: application/json; charset=UTF-8[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 >> "Host: search-hydra-mdl-hu7plzwptok3eip6jkx22r4nxq.us-east-1.es.amazonaws.com[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 >> "User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0)[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 >> "x-amz-date: 20160808T000849Z[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 >> "[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:86 - http-outgoing-0 >> "{"query":{"match_all":{}}}"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 << "HTTP/1.1 403 Forbidden[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 << "Content-Type: application/json[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 << "Server: Jetty(8.1.12.v20130726)[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 << "x-amzn-RequestId: 4836bf0c-5cfc-11e6-970e-79636a7c0458[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 << "Content-Length: 918[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 << "Connection: keep-alive[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:72 - http-outgoing-0 << "[\r][\n]"
2016-08-07 19:08:49 DEBUG wire:86 - http-outgoing-0 << "{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'POST\n/message/post/_search\nscroll=5m&search_type=scan&size=100\naccept-encoding:gzip,deflate\nconnection:close\ncontent-length:26\ncontent-type:application/json; charset=UTF-8\nhost:search-hydra-mdl-hu7plzwptok3eip6jkx22r4nxq.us-east-1.es.amazonaws.com\nuser-agent:Apache-HttpClient/4.5.2 (Java/1.8.0)\nx-amz-date:20160808T000849Z\n\naccept-encoding;connection;content-length;content-type;host;user-agent;x-amz-date\nbaa6846b65b050d71831bb2e4cd6e6f1593902f6d82b16a6c1f9979d14cfcd12'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20160808T000849Z\n20160808/us-east-1/es/aws4_request\ne1a96e4e161e1a13e43eb5f0f5b25144d9921b797ed66fd5f08e88f14a29ab9c'\n"}"
2016-08-07 19:08:49 DEBUG headers:124 - http-outgoing-0 << HTTP/1.1 403 Forbidden
2016-08-07 19:08:49 DEBUG headers:127 - http-outgoing-0 << Content-Type: application/json
2016-08-07 19:08:49 DEBUG headers:127 - http-outgoing-0 << Server: Jetty(8.1.12.v20130726)
2016-08-07 19:08:49 DEBUG headers:127 - http-outgoing-0 << x-amzn-RequestId: 4836bf0c-5cfc-11e6-970e-79636a7c0458
2016-08-07 19:08:49 DEBUG headers:127 - http-outgoing-0 << Content-Length: 918
2016-08-07 19:08:49 DEBUG headers:127 - http-outgoing-0 << Connection: keep-alive
I rolled back to v0.0.12 and it appears to work just fine. Based on the nature of the change in v0.0.13, I suspect the issue has to do with moving to Keep-Alive connections.
Thanks for the work to get to v0.0.12, by the way -- v0.0.11 was choking on an "=" sign in a scroll ID! ๐
Super useful project!
Hi
I worked this library in elasticsearch versions 2.3 and 5.5 and run corretly. But I try to pass ElasticSearch 6.2 and when I executed http petition get mapping (GET schneider/user/_mapping).
I recived next error: "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details"
Is possible change security parameters in this version ES ? because in previous versions run correlty,
Thanks
It looks like 32f858a broke signing GETs.
I now get a 403 response with the following message (slightly sanitized):
The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
The Canonical String for this request should have been
'GET
/tetris_test/split_records/3A1accept-encoding:gzip,deflate
connection:close
content-length:
host:SANITIZED.us-west-2.es.amazonaws.com
user-agent:Apache-HttpClient/4.5.2 (Java/1.8.0_91)
x-amz-date:20160510T174638Zaccept-encoding;connection;content-length;host;user-agent;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'The String-to-Sign should have been
'AWS4-HMAC-SHA256
20160510T174638Z
20160510/us-west-2/es/aws4_request
a7d878202cc3ba4b24808058eb5f37c1d9b30c8eda9b12201bfc047201d48e23'
When I look at the Canonical String that was actually used for signing, it looks like this (note the "0" for content-length):
GET
/tetris_test/split_records/3A1accept-encoding:gzip,deflate
connection:close
content-length:0
host:SANITIZED.us-west-2.es.amazonaws.com
user-agent:Apache-HttpClient/4.5.2 (Java/1.8.0_91)
x-amz-date:20160510T174638Zaccept-encoding;connection;content-length;host;user-agent;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
When URL is like http://host/path?a=b%20%26%20c (a=b & c), the signature is wrong.
The message:
[HTTP/1.1 403 Forbidden]
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}
When I write a scroll query like this:
SearchScroll searchScroll = new SearchScroll.Builder(_scrollId, _scrollExpirationTime).build();
AWSSigningRequestInterceptor throw an exception:
Caused by: java.lang.IllegalArgumentException: Chunk [scroll_id=c2NhbjsxOzc3Mjo5WGljUUFNeVJGcVdDSzBjaUVQcDJ3OzE7dG90YWxfaGl0czo1NTg0Ow==] is not a valid entry at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145) at com.google.common.base.Splitter$MapSplitter.split(Splitter.java:512) at com.logicmonitor.lmes.elasticsearch.aws.AWSSigningRequestInterceptor.params(AWSSigningRequestInterceptor.java:)
I use JestClient 0.1.6
There is not error when query and insert data to AWS Elasticsearch service. But When I refresh an index like:
Refresh refresh = new Refresh.Builder().addIndex(index).build(); JestResult jestResult = _jestClient.execute(refresh);
And get an error return.
CONTEXT=index=newlmes_idx_28_1456704000000, jsonResult={"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'POST\n/newlmes_idx_28_1456704000000/_refresh\na=b\naccept-encoding:gzip,deflate\nconnection:close\ncontent-length:0\ncontent-type:application/json; charset=UTF-8\nhost:search-xxx.ap-northeast-1.es.amazonaws.com\nuser-agent:Apache-HttpClient/4.4.1 (Java/1.7.0_71)\nx-amz-date:20160405T091625Z\n\naccept-encoding;connection;content-length;content-type;host;user-agent;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20160405T091625Z\n20160405/ap-northeast-1/es/aws4_request\n87dc94bf640ae0c2e61071265eda687ee8c6dae9968b460687f836df717e798d'\n"}
If a query String contains "key=" without value, the signature fails because the "=" is URL-encoded.
Solution: In AWSSigningRequestInterceptor change line 64 from
if (index > 0 && pair.length() > index + 1) {
to
if (index > 0 && pair.length() > index) {
Hi, and thanks for you library! I have created a Scala port of the signer, available here: https://github.com/ticofab/aws-request-signer. I thought it would be cool if you linked it from your readme for any scala users that find your library first. Thanks in any case!
I was trying to delete indexes with pattern *_test
.
AWS signing with the following debugging info :
The request signature we calculated does not match the signature you provided
...
The Canonical String for this request should have been\n'DELETE\n/%2A_test\n\naccept-encoding:gzip
...
Update
I guess the bug is when encoding the CanonicalURI
Update
My bad, have not seen the discussion on #18
Closing this in favour of #18
Thanks,
Vinay
The comment here is incorrect:
Note: Double uri encoding is off simple before the signature does not match the expected signature of the test cases if it is enabled. This was a bit unexpected because AWSElasticsearchClient (AWS SDK Class) enabled double URI encoding in the signer by default. I can only assume that double encoding is needed when accessing the service but not when accessing elasticsearch.
S3 expects single encoding. ElasticSearch expects double encoding.
https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
Changing the AWS4Signer 'doubleUrlEncoding' constructor parameter to true in SkdSignerUtil.java (sic.) makes the unit tests pass, as the documents would suggest.
PR #28 (where the change was introduced) also notes that the double-encoding is, in-fact, correct. It appears that the change was not backed out of the unit test code.
Hi,
I'm not sure if it's the correct behavior to urlEncode 2 times for the Uri at line number 120 in AWSSigner.java?
SdkHttpUtils.urlEncode(SdkHttpUtils.urlEncode(uri, true), true) + RETURN +
I've downloaded it and fix it to encode only one time to be able to work with AWS Signing 4
Any suggestion?
Cheers,
Arnon K.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.