Comments (6)
Very good :) thank you. Therefore suspicion is confirmed. Proxy even does BOTH (HTTP 1.0 and Connection: close).
So gateleen should handle this case properly. Which he obviously does not currently -> Aka issue confirmed being a gateleen bug.
from gateleen.
Update from 2023-10-18 (10 months later):
Issue seems still to exists. Found 310 occurrences within time range of 12 hours for a single gateleen instance.
Occurrences greppd from log file: gateleen-prod-begin=20231018-030000-durationSec=43200.log (timestamps are in UTC+0200).
from gateleen.
Update 2024-05-27 (rougly 15 months later):
Issue still exists and now we see several thousand occurrences every day. Here an exmaple:
2024-05-27T02:15:16,367 prod houston ERROR ContextImpl - Unhandled exception
java.lang.IllegalStateException: You must set the Content-Length header to be the total size of the message body BEFORE sending any data if you are not using HTTP chunked encoding.
at org.swisspush.gateleen.core.http.LocalHttpServerResponse.write(LocalHttpServerResponse.java:221) ~[gateleen-core-2.0.2.jar:?]
at org.swisspush.gateleen.core.http.LocalHttpServerResponse.write(LocalHttpServerResponse.java:20) ~[gateleen-core-2.0.2.jar:?]
at org.swisspush.gateleen.logging.LoggingWriteStream.write(LoggingWriteStream.java:47) ~[gateleen-logging-2.0.2.jar:?]
at org.swisspush.gateleen.logging.LoggingWriteStream.write(LoggingWriteStream.java:39) ~[gateleen-logging-2.0.2.jar:?]
at org.swisspush.gateleen.logging.LoggingWriteStream.write(LoggingWriteStream.java:13) ~[gateleen-logging-2.0.2.jar:?]
at io.vertx.core.streams.impl.PumpImpl.lambda$new$1(PumpImpl.java:63) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.http.impl.HttpEventHandler.handleChunk(HttpEventHandler.java:51) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.http.impl.HttpClientResponseImpl.handleChunk(HttpClientResponseImpl.java:239) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$new$0(Http1xClientConnection.java:386) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:240) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:130) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleChunk(Http1xClientConnection.java:589) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:71) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.impl.DuplicatedContext.execute(DuplicatedContext.java:163) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.http.impl.Http1xClientConnection.handleResponseChunk(Http1xClientConnection.java:800) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.http.impl.Http1xClientConnection.handleHttpMessage(Http1xClientConnection.java:696) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.http.impl.Http1xClientConnection.handleMessage(Http1xClientConnection.java:663) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:156) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:153) ~[vertx-core-4.2.1.jar:4.2.1]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) ~[netty-codec-4.1.69.Final.jar:4.1.69.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) ~[netty-codec-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372) ~[netty-handler-4.1.69.Final.jar:4.1.69.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[netty-handler-4.1.69.Final.jar:4.1.69.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[netty-handler-4.1.69.Final.jar:4.1.69.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.69.Final.jar:4.1.69.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.69.Final.jar:4.1.69.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.69.Final.jar:4.1.69.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.69.Final.jar:4.1.69.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.69.Final.jar:4.1.69.Final]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
from gateleen.
We also sometimes have a lot of these exceptions. Source in our case were requests to an Openshift 3 instance that was shut down. The HAProxy before the pod answered with this HTML page:
<html>
<head>
<meta name="viewport" content="width=device-width, initial-scale=1">
<style type="text/css">
/*!
* Bootstrap v3.3.5 (http://getbootstrap.com)
* Copyright 2011-2015 Twitter, Inc.
* Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)
*/
/*! normalize.css v3.0.3 | MIT License | github.com/necolas/normalize.css */
html {
font-family: sans-serif;
-ms-text-size-adjust: 100%;
-webkit-text-size-adjust: 100%;
}
body {
margin: 0;
}
h1 {
font-size: 1.7em;
font-weight: 400;
line-height: 1.3;
margin: 0.68em 0;
}
* {
-webkit-box-sizing: border-box;
-moz-box-sizing: border-box;
box-sizing: border-box;
}
*:before,
*:after {
-webkit-box-sizing: border-box;
-moz-box-sizing: border-box;
box-sizing: border-box;
}
html {
-webkit-tap-highlight-color: rgba(0, 0, 0, 0);
}
body {
font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
line-height: 1.66666667;
font-size: 13px;
color: #333333;
background-color: #ffffff;
margin: 2em 1em;
}
p {
margin: 0 0 10px;
font-size: 13px;
}
.alert.alert-info {
padding: 15px;
margin-bottom: 20px;
border: 1px solid transparent;
background-color: #f5f5f5;
border-color: #8b8d8f;
color: #363636;
margin-top: 30px;
}
.alert p {
padding-left: 35px;
}
a {
color: #0088ce;
}
ul {
position: relative;
padding-left: 51px;
}
p.info {
position: relative;
font-size: 15px;
margin-bottom: 10px;
}
p.info:before, p.info:after {
content: "";
position: absolute;
top: 9%;
left: 0;
}
p.info:before {
content: "i";
left: 3px;
width: 20px;
height: 20px;
font-family: serif;
font-size: 15px;
font-weight: bold;
line-height: 21px;
text-align: center;
color: #fff;
background: #4d5258;
border-radius: 16px;
}
@media (min-width: 768px) {
body {
margin: 4em 3em;
}
h1 {
font-size: 2.15em;}
}
</style>
</head>
<body>
<div>
<h1>Application is not available</h1>
<p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>
<div class="alert alert-info">
<p class="info">
Possible reasons you are seeing this page:
</p>
<ul>
<li>
<strong>The host doesn't exist.</strong>
Make sure the hostname was typed correctly and that a route matching this hostname exists.
</li>
<li>
<strong>The host exists, but doesn't have a matching path.</strong>
Check if the URL path was typed correctly and that the route was created using the desired path.
</li>
<li>
<strong>Route and path matches, but all pods are down.</strong>
Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.
</li>
</ul>
</div>
</div>
</body>
</html>
Apparently, no Content-Length
response header was sent along.
from gateleen.
We also sometimes have a lot of these exceptions. Source in our case were requests to an Openshift 3 instance that was shut down. The HAProxy before the pod answered with this HTML page:
Is it possible to see the full, RAW HTTP header? I guess maybe the proxy is responding with either an HTTP/1.0 response or with a "Connection: close" header? If so, this would make the "Content-Length" header superfluous. And Gateleen should handle it appropriate (aka expecting the response to end as soon the TCP FIN arrives).
from gateleen.
Here's the full curl output:
[webermarca@xxxxx ~]$ curl -v -H 'Content-Type: application/json' -d '{"mailpieces":[],"deliveryOrganisation":"DOMICILE","deliveryDate":"2024-07-12T16:58:22.565+02:00","deliveryArea":201,"deliveryOfficePostcode6":307170,"deliveryAreaGroup":0}' -X POST https://xxx-int.xxx.xx.ch:443/api/sending/loadingscan
* About to connect() to xxx-int.xxx.xx.ch port 443 (#0)
* Trying 172.18.216.10...
* Connected to xxx-int.xxx.xx.ch (172.18.216.10) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=*.xxx.xx.ch,O=xxxxx
* start date: Nov 02 10:48:28 2023 GMT
* expire date: Nov 02 10:48:28 2024 GMT
* common name: *.xxx.xx.ch
* issuer: CN=SwissSign RSA TLS OV ICA 2022 - 1,O=SwissSign AG,C=CH
> POST /api/sending/loadingscan HTTP/1.1
> User-Agent: curl/7.29.0
> Host: xxx-int.xxx.xx.ch
> Accept: */*
> Content-Type: application/json
> Content-Length: 172
>
* upload completely sent off: 172 out of 172 bytes
* HTTP 1.0, assume close after body
< HTTP/1.0 503 Service Unavailable
< Pragma: no-cache
< Cache-Control: private, max-age=0, no-cache, no-store
< Connection: close
< Content-Type: text/html
<
<html>
<head>
<meta name="viewport" content="width=device-width, initial-scale=1">
<style type="text/css">
/*!
* Bootstrap v3.3.5 (http://getbootstrap.com)
* Copyright 2011-2015 Twitter, Inc.
* Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)
*/
/*! normalize.css v3.0.3 | MIT License | github.com/necolas/normalize.css */
html {
font-family: sans-serif;
-ms-text-size-adjust: 100%;
-webkit-text-size-adjust: 100%;
}
body {
margin: 0;
}
h1 {
font-size: 1.7em;
font-weight: 400;
line-height: 1.3;
margin: 0.68em 0;
}
* {
-webkit-box-sizing: border-box;
-moz-box-sizing: border-box;
box-sizing: border-box;
}
*:before,
*:after {
-webkit-box-sizing: border-box;
-moz-box-sizing: border-box;
box-sizing: border-box;
}
html {
-webkit-tap-highlight-color: rgba(0, 0, 0, 0);
}
body {
font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
line-height: 1.66666667;
font-size: 13px;
color: #333333;
background-color: #ffffff;
margin: 2em 1em;
}
p {
margin: 0 0 10px;
font-size: 13px;
}
.alert.alert-info {
padding: 15px;
margin-bottom: 20px;
border: 1px solid transparent;
background-color: #f5f5f5;
border-color: #8b8d8f;
color: #363636;
margin-top: 30px;
}
.alert p {
padding-left: 35px;
}
a {
color: #0088ce;
}
ul {
position: relative;
padding-left: 51px;
}
p.info {
position: relative;
font-size: 15px;
margin-bottom: 10px;
}
p.info:before, p.info:after {
content: "";
position: absolute;
top: 9%;
left: 0;
}
p.info:before {
content: "i";
left: 3px;
width: 20px;
height: 20px;
font-family: serif;
font-size: 15px;
font-weight: bold;
line-height: 21px;
text-align: center;
color: #fff;
background: #4d5258;
border-radius: 16px;
}
@media (min-width: 768px) {
body {
margin: 4em 3em;
}
h1 {
font-size: 2.15em;}
}
</style>
</head>
<body>
<div>
<h1>Application is not available</h1>
<p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>
<div class="alert alert-info">
<p class="info">
Possible reasons you are seeing this page:
</p>
<ul>
<li>
<strong>The host doesn't exist.</strong>
Make sure the hostname was typed correctly and that a route matching this hostname exists.
</li>
<li>
<strong>The host exists, but doesn't have a matching path.</strong>
Check if the URL path was typed correctly and that the route was created using the desired path.
</li>
<li>
<strong>Route and path matches, but all pods are down.</strong>
Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.
</li>
</ul>
</div>
</div>
</body>
</html>
* Closing connection 0
The interesting part is
HTTP 1.0, assume close after body
from gateleen.
Related Issues (20)
- Hooks on forwarded requests HOT 1
- LoggingHandler does not log JsonArray payloads HOT 1
- ContextImpl Unhandled java.lang.UnsupportedOperationException
- EventBusAppender instance pollution in Java Heap Space HOT 1
- Option to split items for a queue in multiple sub-queues HOT 6
- gateleen-hook-js build not portable to non-glibc systems
- Not implemented HttpClientRequest.connection() gets called
- Better handling for unmatched delegate executions HOT 1
- Json serialization blocking EventLoop for too long
- Wrong request logging for delta requests HOT 1
- MonitoringHandler blocking EventLoop HOT 1
- Some unknown actor blocks EventLoop by doing blocking IO
- QueueProcessor is blocking EventLoop
- KafkaProducerRecordBuilder is blocking EventLoop with JSON serialization
- IllegalStateException without any useful details when trying to register endHandler
- Router.createForwarders() blocks EventLoop
- Blocking IO in Forwarder?
- StackWalker takes suspiciously long during getLogger call
- Hook handling blocks EventLoop by searching for regex matches in a loop
- Analyze: Hook targets with wildcard values
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gateleen.