Comments (6)
hiddenalpha
commented on August 25, 2024
1
Very good :) thank you. Therefore suspicion is confirmed. Proxy even does BOTH (HTTP 1.0 and Connection: close).
So gateleen should handle this case properly. Which he obviously does not currently -> Aka issue confirmed being a gateleen bug.
from gateleen.
hiddenalpha
commented on August 25, 2024
Update from 2023-10-18 (10 months later):
Issue seems still to exists. Found 310 occurrences within time range of 12 hours for a single gateleen instance.
Occurrences greppd from log file: gateleen-prod-begin=20231018-030000-durationSec=43200.log (timestamps are in UTC+0200).
from gateleen.
hiddenalpha
commented on August 25, 2024
Update 2024-05-27 (rougly 15 months later):
Issue still exists and now we see several thousand occurrences every day. Here an exmaple:
2024-05-27T02:15:16,367 prod houston ERROR ContextImpl - Unhandled exception
java.lang.IllegalStateException: You must set the Content-Length header to be the total size of the message body BEFORE sending any data if you are not using HTTP chunked encoding.
at org.swisspush.gateleen.core.http.LocalHttpServerResponse.write(LocalHttpServerResponse.java:221) ~[gateleen-core-2.0.2.jar:?]
at org.swisspush.gateleen.core.http.LocalHttpServerResponse.write(LocalHttpServerResponse.java:20) ~[gateleen-core-2.0.2.jar:?]
at org.swisspush.gateleen.logging.LoggingWriteStream.write(LoggingWriteStream.java:47) ~[gateleen-logging-2.0.2.jar:?]
at org.swisspush.gateleen.logging.LoggingWriteStream.write(LoggingWriteStream.java:39) ~[gateleen-logging-2.0.2.jar:?]
at org.swisspush.gateleen.logging.LoggingWriteStream.write(LoggingWriteStream.java:13) ~[gateleen-logging-2.0.2.jar:?]
at io.vertx.core.streams.impl.PumpImpl.lambda$new$1(PumpImpl.java:63) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.http.impl.HttpEventHandler.handleChunk(HttpEventHandler.java:51) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.http.impl.HttpClientResponseImpl.handleChunk(HttpClientResponseImpl.java:239) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$new$0(Http1xClientConnection.java:386) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:240) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:130) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleChunk(Http1xClientConnection.java:589) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:71) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.impl.DuplicatedContext.execute(DuplicatedContext.java:163) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.http.impl.Http1xClientConnection.handleResponseChunk(Http1xClientConnection.java:800) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.http.impl.Http1xClientConnection.handleHttpMessage(Http1xClientConnection.java:696) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.http.impl.Http1xClientConnection.handleMessage(Http1xClientConnection.java:663) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:156) ~[vertx-core-4.2.1.jar:4.2.1]
at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:153) ~[vertx-core-4.2.1.jar:4.2.1]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) ~[netty-codec-4.1.69.Final.jar:4.1.69.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) ~[netty-codec-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372) ~[netty-handler-4.1.69.Final.jar:4.1.69.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[netty-handler-4.1.69.Final.jar:4.1.69.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[netty-handler-4.1.69.Final.jar:4.1.69.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.69.Final.jar:4.1.69.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.69.Final.jar:4.1.69.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-transport-4.1.69.Final.jar:4.1.69.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.69.Final.jar:4.1.69.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.69.Final.jar:4.1.69.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.69.Final.jar:4.1.69.Final]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
from gateleen.
mcweba
commented on August 25, 2024
We also sometimes have a lot of these exceptions. Source in our case were requests to an Openshift 3 instance that was shut down. The HAProxy before the pod answered with this HTML page:
<html>
<head>
<meta name="viewport" content="width=device-width, initial-scale=1">
<style type="text/css">
/*!
* Bootstrap v3.3.5 (http://getbootstrap.com)
* Copyright 2011-2015 Twitter, Inc.
* Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)
*/
/*! normalize.css v3.0.3 | MIT License | github.com/necolas/normalize.css */
html {
font-family: sans-serif;
-ms-text-size-adjust: 100%;
-webkit-text-size-adjust: 100%;
}
body {
margin: 0;
}
h1 {
font-size: 1.7em;
font-weight: 400;
line-height: 1.3;
margin: 0.68em 0;
}
* {
-webkit-box-sizing: border-box;
-moz-box-sizing: border-box;
box-sizing: border-box;
}
*:before,
*:after {
-webkit-box-sizing: border-box;
-moz-box-sizing: border-box;
box-sizing: border-box;
}
html {
-webkit-tap-highlight-color: rgba(0, 0, 0, 0);
}
body {
font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
line-height: 1.66666667;
font-size: 13px;
color: #333333;
background-color: #ffffff;
margin: 2em 1em;
}
p {
margin: 0 0 10px;
font-size: 13px;
}
.alert.alert-info {
padding: 15px;
margin-bottom: 20px;
border: 1px solid transparent;
background-color: #f5f5f5;
border-color: #8b8d8f;
color: #363636;
margin-top: 30px;
}
.alert p {
padding-left: 35px;
}
a {
color: #0088ce;
}
ul {
position: relative;
padding-left: 51px;
}
p.info {
position: relative;
font-size: 15px;
margin-bottom: 10px;
}
p.info:before, p.info:after {
content: "";
position: absolute;
top: 9%;
left: 0;
}
p.info:before {
content: "i";
left: 3px;
width: 20px;
height: 20px;
font-family: serif;
font-size: 15px;
font-weight: bold;
line-height: 21px;
text-align: center;
color: #fff;
background: #4d5258;
border-radius: 16px;
}
@media (min-width: 768px) {
body {
margin: 4em 3em;
}
h1 {
font-size: 2.15em;}
}
</style>
</head>
<body>
<div>
<h1>Application is not available</h1>
<p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>
<div class="alert alert-info">
<p class="info">
Possible reasons you are seeing this page:
</p>
<ul>
<li>
<strong>The host doesn't exist.</strong>
Make sure the hostname was typed correctly and that a route matching this hostname exists.
</li>
<li>
<strong>The host exists, but doesn't have a matching path.</strong>
Check if the URL path was typed correctly and that the route was created using the desired path.
</li>
<li>
<strong>Route and path matches, but all pods are down.</strong>
Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.
</li>
</ul>
</div>
</div>
</body>
</html>Apparently, no Content-Lengthresponse header was sent along.
from gateleen.
hiddenalpha
commented on August 25, 2024
We also sometimes have a lot of these exceptions. Source in our case were requests to an Openshift 3 instance that was shut down. The HAProxy before the pod answered with this HTML page:
Is it possible to see the full, RAW HTTP header? I guess maybe the proxy is responding with either an HTTP/1.0 response or with a "Connection: close" header? If so, this would make the "Content-Length" header superfluous. And Gateleen should handle it appropriate (aka expecting the response to end as soon the TCP FIN arrives).
from gateleen.
mcweba
commented on August 25, 2024
Here's the full curl output:
[webermarca@xxxxx ~]$ curl -v -H 'Content-Type: application/json' -d '{"mailpieces":[],"deliveryOrganisation":"DOMICILE","deliveryDate":"2024-07-12T16:58:22.565+02:00","deliveryArea":201,"deliveryOfficePostcode6":307170,"deliveryAreaGroup":0}' -X POST https://xxx-int.xxx.xx.ch:443/api/sending/loadingscan
* About to connect() to xxx-int.xxx.xx.ch port 443 (#0)
* Trying 172.18.216.10...
* Connected to xxx-int.xxx.xx.ch (172.18.216.10) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=*.xxx.xx.ch,O=xxxxx
* start date: Nov 02 10:48:28 2023 GMT
* expire date: Nov 02 10:48:28 2024 GMT
* common name: *.xxx.xx.ch
* issuer: CN=SwissSign RSA TLS OV ICA 2022 - 1,O=SwissSign AG,C=CH
> POST /api/sending/loadingscan HTTP/1.1
> User-Agent: curl/7.29.0
> Host: xxx-int.xxx.xx.ch
> Accept: */*
> Content-Type: application/json
> Content-Length: 172
>
* upload completely sent off: 172 out of 172 bytes
* HTTP 1.0, assume close after body
< HTTP/1.0 503 Service Unavailable
< Pragma: no-cache
< Cache-Control: private, max-age=0, no-cache, no-store
< Connection: close
< Content-Type: text/html
<
<html>
<head>
<meta name="viewport" content="width=device-width, initial-scale=1">
<style type="text/css">
/*!
* Bootstrap v3.3.5 (http://getbootstrap.com)
* Copyright 2011-2015 Twitter, Inc.
* Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)
*/
/*! normalize.css v3.0.3 | MIT License | github.com/necolas/normalize.css */
html {
font-family: sans-serif;
-ms-text-size-adjust: 100%;
-webkit-text-size-adjust: 100%;
}
body {
margin: 0;
}
h1 {
font-size: 1.7em;
font-weight: 400;
line-height: 1.3;
margin: 0.68em 0;
}
* {
-webkit-box-sizing: border-box;
-moz-box-sizing: border-box;
box-sizing: border-box;
}
*:before,
*:after {
-webkit-box-sizing: border-box;
-moz-box-sizing: border-box;
box-sizing: border-box;
}
html {
-webkit-tap-highlight-color: rgba(0, 0, 0, 0);
}
body {
font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
line-height: 1.66666667;
font-size: 13px;
color: #333333;
background-color: #ffffff;
margin: 2em 1em;
}
p {
margin: 0 0 10px;
font-size: 13px;
}
.alert.alert-info {
padding: 15px;
margin-bottom: 20px;
border: 1px solid transparent;
background-color: #f5f5f5;
border-color: #8b8d8f;
color: #363636;
margin-top: 30px;
}
.alert p {
padding-left: 35px;
}
a {
color: #0088ce;
}
ul {
position: relative;
padding-left: 51px;
}
p.info {
position: relative;
font-size: 15px;
margin-bottom: 10px;
}
p.info:before, p.info:after {
content: "";
position: absolute;
top: 9%;
left: 0;
}
p.info:before {
content: "i";
left: 3px;
width: 20px;
height: 20px;
font-family: serif;
font-size: 15px;
font-weight: bold;
line-height: 21px;
text-align: center;
color: #fff;
background: #4d5258;
border-radius: 16px;
}
@media (min-width: 768px) {
body {
margin: 4em 3em;
}
h1 {
font-size: 2.15em;}
}
</style>
</head>
<body>
<div>
<h1>Application is not available</h1>
<p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>
<div class="alert alert-info">
<p class="info">
Possible reasons you are seeing this page:
</p>
<ul>
<li>
<strong>The host doesn't exist.</strong>
Make sure the hostname was typed correctly and that a route matching this hostname exists.
</li>
<li>
<strong>The host exists, but doesn't have a matching path.</strong>
Check if the URL path was typed correctly and that the route was created using the desired path.
</li>
<li>
<strong>Route and path matches, but all pods are down.</strong>
Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.
</li>
</ul>
</div>
</div>
</body>
</html>
* Closing connection 0
The interesting part is
HTTP 1.0, assume close after body
from gateleen.
Related Issues (20)
- Hooks on forwarded requests HOT 1
- LoggingHandler does not log JsonArray payloads HOT 1
- ContextImpl Unhandled java.lang.UnsupportedOperationException
- EventBusAppender instance pollution in Java Heap Space HOT 1
- Option to split items for a queue in multiple sub-queues HOT 6
- gateleen-hook-js build not portable to non-glibc systems
- Not implemented HttpClientRequest.connection() gets called
- Better handling for unmatched delegate executions HOT 1
- Json serialization blocking EventLoop for too long
- Wrong request logging for delta requests HOT 1
- MonitoringHandler blocking EventLoop HOT 1
- Some unknown actor blocks EventLoop by doing blocking IO
- QueueProcessor is blocking EventLoop
- KafkaProducerRecordBuilder is blocking EventLoop with JSON serialization
- IllegalStateException without any useful details when trying to register endHandler
- Router.createForwarders() blocks EventLoop
- Blocking IO in Forwarder?
- StackWalker takes suspiciously long during getLogger call
- Hook handling blocks EventLoop by searching for regex matches in a loop
- Analyze: Hook targets with wildcard values
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gateleen.