wso2 / transport-http

Welcome to WSO2 HTTP Transport

This repository holds HTTP protocol handling implementations for C5 based products.

Introduction

WSO2 middleware products deals with many different transport protocols. HTTP, JMS, AMQP, MQTT are few of the highly used protocols. This repository contains the HTTP protocol handling implementations which are used by various WSO2 products built on top of C5 (Carbon 5) based products.

Features

• Provides the implementation for handling transport layer complexity of the message interaction
• Handles the concurrent connections from the source systems and connect with multiple target systems

transport-http's Issues

[WebSocket] Need test cases to test connection closure

Need to check whether the connections are closed according to the WebSocket specification.

Unknown channel option Warning from Server Bootstrap

Description:
Observe below warning when server startup. This is because SO_KEEPALIVE and SO_SNDBUF are not channel options for ServerSocketChannels. According to the java docs[1], ServerSocketChannel below options,

1. https://docs.oracle.com/javase/8/docs/api/java/nio/channels/ServerSocketChannel.html

Option Name	Description
SO_RCVBUF	The size of the socket receive buffer
SO_REUSEADDR	Re-use address

WARN Message

2017-12-07 12:32:57 WARN  ServerBootstrap:146 - Unknown channel option 'SO_KEEPALIVE' for channel '[id: 0xac0fb464]'
2017-12-07 12:33:03 WARN  ServerBootstrap:146 - Unknown channel option 'SO_SNDBUF' for channel '[id: 0xac0fb464]'

Need to remove setting those options in ServerConnectorBootstrap class.

Suggested Labels:
Bug

Suggested Assignees:
@shafreenAnfar

Affected Product Version:
6.0.51

OS, DB, other environment details and versions:
environment independent issue.

Steps to reproduce:
warning message appears when server startup.

Related Issues:
N/A

[WebSocket] Need to test binary message passthrough

Warning with exception occurs when accessing with untrusted certificate

Description:
When accessing a URL with an untrusted certificate following exception occurs.

[2019-03-29 17:56:27,922]  WARN {io.netty.channel.DefaultChannelPipeline} - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459)
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:392)
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:359)
	at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342)
	at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1010)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1354)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
	at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:917)
	at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)
	at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800)
	at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083)
	at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
	at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:292)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1248)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1159)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1194)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
	... 18 more

Suggested Labels:
Bug

Affected Product Version:
6.0.272 (WSO2 Stream Processor 4.4.0 - beta4)

OS, DB, other environment details and versions:
MacOS, Chrome

Steps to reproduce:

1. Start the dashboard profile of SP 4.4.0 - beta4
2. Access the https://localhost:9643/portal using Chrome.

[HTTP] No unit test cases are written to verify HTTP verb response behavior

Description:
No unit test cases are written to verify HTTP verb response behavior

Test cases are needed to verify HttpResponseStatusFuture functionality

Description:
Outbound response call is returning the status back to the user level. If the I/O operation failed, it returns the server connector error. Otherwise, the error is null. The notification is done using a response status future. Hence test cases are needed to verify the functionality in transport level.

Suggested Labels:
N/A

Suggested Assignees:
N/A

Affected Product Version:
ballerina 0.95 onwards versions

OS, DB, other environment details and versions:

Steps to reproduce:

Related Issues:
N/A

Upgrade carbon-messaging version to 3.0.2

Description:
Current transport version have carbon-messaging dependency for 2.3.7 version which is referring to carbon kernel milestone 3. And we have latest carbon-messaging version 3.0.2 which is referring to latest kernel GA version(5.2.0).

Suggested Labels:
task

Suggested Assignees:
@irunika

Affected Product Version:
6.0.51

Please open a security advisory

Hi WSO2 Team,

Please open a security advisory so I can report a security vulnerability to you.
I prefer GitHub security advisories over emails as it significantly simplifies the disclosure and CVE issuance process.

https://github.com/wso2/transport-http/security/advisories

[WebSocket] Need to Remove javax.websocket.session interface from Http-Transport

Description:
This interface is used originally for MSF4J implementation in the first place and now many products are using it which forces them to add javax.websocket as a dependency just to
work with transport-http WebSocket support. So it is better to have similar native transport interface rather than having a whole package for a single interface.

[WebSocket] Need to add test cases to test continuation frame handling

Description:
Need to add test cases to test continuation frame handling in the transport.

NPE observed when an idle connection is closed and not used

Description:
The following error being logged in the dashboard carbon logs, in WSO2 AM Analytics 2.6.0:

[2019-07-02 01:24:53,657] ERROR {org.wso2.msf4j.internal.MSF4JHttpConnectorListener} - Error in http connector listener java.lang.NullPointerException[2019-07-02 01:24:53,657] ERROR {org.wso2.msf4j.internal.MSF4JHttpConnectorListener} - Error in http connector listener java.lang.NullPointerException at org.wso2.transport.http.netty.listener.SourceHandler.handleIdleErrorScenario(SourceHandler.java:286) at org.wso2.transport.http.netty.listener.SourceHandler.userEventTriggered(SourceHandler.java:269) at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307) at io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:494) at io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:466) at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:745)

Affected Product Version:
WSO2 AM Analytics 2.6.0

Steps to reproduce:
This issue could not be recreated on a similar set up; and the cause is unknown.

Add method for adding a set of headers to HTTPCarbonMessage

Introduce Timeout error

Need to package the test cases in testng.xml

Description:
Seems we can refactor the test cases in testng.xml. All the test cases have been added as classes. But it is possible to cluster them into test packages after a careful inspection.

[WebSocket] Need to fix onError in WebSocketConnectorListner

Description:
Need to fix the way that the errors are notified to the listener since there is no clue on from which connection the error is thrown.

Add trailer header support

Description:
Current http2 implementation doesn't support passing trailer header frame at the end of the stream. We need to support this to support gRPC over http2. because in gRPC, it is mandated to have trailer header in server response. It carries request status and end of stream flag.

Suggested Labels:
improvement

Related Issues:
ballerina-platform/ballerina-lang#8460

[WebSocket] Need a way to block incoming messages until the connections are established in WebSocket proxy

Add trailing header support for HTTP/1.1

Description:
Currently this only supported in HTTP/2 version. Better if we can add trailing header support for HTTP/1.1 to maintain consistency.

HTTP host is wrongly logged

Description:
HTTP host is wrongly logged. Port no is logged for both host and port.

Related Issues:
wso2/product-sp#961

Screenshot:

[WebSocket] Need to add SSL configuration capability to WebSocket Client

Description:
Current client is doing the SSL handshake using InsecureTrustManagerFactory from netty. This need to be changed to our own SSL handler and give the capability to configure it.

Need to refactor SSL Configuration to improve reusability

Description:
In current transport, SSL Configuration is done in two steps

1. Add SSL related configurations to sender and listener configurations.
2. Generate SSL configuration from the related configuration instances.

But it seems they can be refactored more for more reusability and also getter methods in configurations related to SSL configurations can be moved to SSLConfig class. By doing like that code will become more reusable and also there will be only one place where we have to generate the SSLConfig instance.
It should be done as a separate task since, by refactoring this, most of the public APIs relate to SSL also can be refactored.

[WebSocket] Need Test Cases for WebSocket Basic Remote in Session

transport-http-6.0.289 we can't disable unwanted TLS versions.

Description:
In the transport-http-6.0.289 version, there is no option to disable unwanted TLS versions.
In the transport-http-6.0.163 version, we can disable unwanted TLS by enabling only the wanted TLS values as follows of the particular profile deployment.yaml file under the listenerConfigurations section.

parameters:
        - name: "sslEnabledProtocols"
          value: "TLSv1.2"

Since it's a security level risk not able to limit the TLS versions, it is better to add this feature to the transport-http-6.0.289 version as well.

Affected Product Version:
APIM-Analytics-3.0.0

When using the send method of the HttpClientConnector class, a memory leak is detected in the ByteBuf.

Description:
When using the send method of the HttpClientConnector class, a memory leak is detected in the ByteBuf.

Here is my code：

HttpResponseFuture responseFuture = httpClientConnector.send(request);
                              ... some bussiness code ...
httpMessage.getFullHttpCarbonMessage().addListener(new FullHttpMessageListener() {
                        @Override
                        public void onComplete(HttpCarbonMessage httpCarbonMessage) {
                            InputStream inputStream = null;
                            OutputStream outputStream = null;
                            try {
                                HttpMessageDataStreamer dataStreamer = new HttpMessageDataStreamer(httpCarbonMessage);
                                inputStream = dataStreamer.getInputStream();
                                outputStream = dataStreamer.getOutputStream();
                                byte[] content = IOUtils.toByteArray(inputStream);
                                ... some business code...
                          finally {
                                try {
                                    if (outputStream != null) {
                                        outputStream.close();
                                    }
                                    if (inputStream != null) {

                                        inputStream.close();
                                    }

I close the input and output stream in finally segment,but ByteBuf LEAK occurs.
The leak error is：

2023-04-27 01:25:17.158 1 [nioEventLoopGroup-29-4] ERROR io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
 io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:403)
 io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
 io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
 io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:140)
 io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120)
 io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:150)
 io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
 io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
 io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
 io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
 io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
 io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
 io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 java.lang.Thread.run(Thread.java:748)

No wso2 related class is found.
So, i add code to calculate the direct buffer,when one response is coming, direct buffer increase 4096KB, every time is same。
Anyone can help me？

[HTTP2] Register Idle Timeout Handler to Http2 server channel

Description:
Current HTTP2 server channel doesn't handle idle timeout and server waits forever. So idleTimeout configured in http2 service doesn't apply to server channel and server waits forever.

Disabled IdleTimeout test case in WebSocketServerTestCase

Description:
The mentioned test case was disabled since it kept failing intermittently. Need to look into why it is happening and fix it and enable the test case again.

Suggested Labels:
Type/Bug

Suggested Assignees:
Irunika

Affected Product Version:
6.0.74-SNAPSHOT

Why close() method always return false in HttpClientConnector

I am using HttpClientConnector to establish a connection and achieve HTTP communication，and after FullHttpMessageListener onComplete method has been called, i want to close the connection.But,the close() method always return false
@OverRide
public boolean close() {
return false;
}
Why design this?

[WebSocket] Cannot handle continuation frames

Description:
When an incoming message is not a final fragment of a large message WebSocket frame set a flag "fin" to indicate that it is not a final message. Then it sends continuation frames until the final message. This is not handled in current transport.

[Http] Harmless exception is logged if SSL verification fails

Description:
When we client initiates the connection over SSL, if the SSL verification fails, then if the connection timeout, transport will throw, IDLE_TIMEOUT_TRIGGERED_BEFORE_INITIATING_INBOUND_REQUEST
However, this can be safely ignored as in this case, we can close the connection.

Affected Product Version:
6.0.283

OS, DB, other environment details and versions:

Steps to reproduce:

Related Issues:

[WebSocket] Need to Test Cancelling WebSocket Upgrade

[WebSocket] Need to Test connection closing of client

[HTTP2] Possible message content lose when writing outbound message

Description:
When there are multiple Http content in one carbon message, part of the content can lose when writing outbound message.

Issue is in the Message Future code[1]. because of that every even content added in to the pendingPayload is ignored when writing outbound message.

1. https://github.com/wso2/transport-http/blob/master/components/org.wso2.transport.http.netty/src/main/java/org/wso2/transport/http/netty/message/MessageFuture.java#L53

Suggested Labels:
Bug

Suggested Assignees:
@isudana @shafreenAnfar