siddhi-io / siddhi-io-http Goto Github PK
View Code? Open in Web Editor NEWExtension that allows you to receive and publish events via HTTP and https transports
Home Page: https://siddhi-io.github.io/siddhi-io-http/
License: Apache License 2.0
siddhi-io-http's People
Contributors
Stargazers
Watchers
Forkers
ramindu90 niveathika sajithshn anugayan chaminda chathurikaa anoukh pcnfernando ksdperera minudika yasara123 gimantha senthuran16 test-coverage-enforce-bot charukak tishan89 suhothayan madurangasiriwardena nisalaniroshana janakamarasena rukshiw sinthuja pranavshah029 pradeepajey1 gomathyk thushaaanthan rolandhewage mayuravaani mohanvive ajkarki dnwick lasanthas raveensr buddhiwathsala slahirucd7 methma ajanthy janithcmw liaoweisharp inoshperera smahesh17 cabbage89 dilini-muthumala oshiwindsor sybernix cazci grainier nirothipan madushadhanushka kaantecik savindudimal rohankumardubeysiddhi-io-http's Issues
HTTP Source is removed when there multiple apps with same source hostname,port and context
Description:
Let's assume a scenario where we have siddhi app with HTTP source and we try to deploy another siddhi app with same HTTP source. In that case, it will remove the HTTP source from the first app also.
Suggested Labels:
@bug
Steps to reproduce:
- Create two siddhi apps with same HTTP source.
- Then add these siddhi apps one by one.
- It will give an error and does not allow to deploy the second app. It will also remove the HTTP source from the first app. So we cannot publish any events to that HTTP source.
HTTP sink GET request fails with HTTP 400
Description:
HTTP sink GET request fails with HTTP 400 because we are not sending in Host header. Need to follow up on this and investigate why we are not sending Host header and what to be expected as per spec
Affected Product Version:
SP 4.x.x
Steps to reproduce:
Can use below Siddhi App to reproduce
@App:name("SiddhiApp")
@App:description("Description of the plan")
-- Please refer to https://docs.wso2.com/display/SP400/Quick+Start+Guide on getting started with SP editor.
@sink(
type='http-request', sink.id='id1', publisher.url='http://api.worldbank.org/v2/countries/all/indicators/SP.POP.TOTL',
method='GET', headers = "'Host:api.worldbank.org'",
@map(type='json'))
define stream FooStream (payloadBody String);
@source(type='http-response', sink.id='id1', http.status.code='200',@map(type='text' ))
@sink(type = 'log', prefix = 'LOGGER')
define stream responseStream2xx(body string);
@source(type='http-response', sink.id='id1', http.status.code='400', @map(type='text'))
@sink(type = 'log', prefix = 'LOGGER')
define stream responseStream4xx(body string);
Netty resources leak when performing perf tests
Description:
When performing a perf test using http request-response flow, following LEAK error message is getting printed.
[2019-04-19 15:46:12,400] ERROR {io.netty.util.ResourceLeakDetector} - LEAK: ByteBuf.release() was not called before it's garbage-collected. Enable advanced leak reporting to find out where the leak occurred. To enable adv
anced leak reporting, specify the JVM option '-Dio.netty.leakDetection.level=advanced' or call ResourceLeakDetector.setLevel() See http://netty.io/wiki/reference-counted-objects.html for more information.
After setting the leak detection level to "ADVANCED", it provides following logs.
[2019-05-31 06:37:24,564] ERROR {io.netty.util.ResourceLeakDetector} - LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:273)
io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
#2:
io.netty.buffer.AdvancedLeakAwareByteBuf.getByte(AdvancedLeakAwareByteBuf.java:154)
io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:360)
io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
#3:
io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:803)
io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:603)
io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:227)
io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
#4:
io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:803)
io.netty.handler.codec.http.HttpObjectDecoder$LineParser.parse(HttpObjectDecoder.java:852)
io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:208)
io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
#5:
io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:160)
io.netty.handler.codec.http.HttpObjectDecoder.skipControlCharacters(HttpObjectDecoder.java:568)
io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:202)
io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
#6:
Hint: 'codec' will handle the message from this point.
io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
#7:
Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
Created at:
io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:634)
io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:345)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:126)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
: 6 leak records were discarded because they were duplicates
CORS handling not available
Description:
When calling endpoints directly from a web application; first an HTTP OPTIONS request is sent to figure permission to access selected resources from the server. Currently in the server, if any request other than an HTTP POST request is received the request is dropped with the status code 400[1].
And the following error gets logged
[2018-08-20 09:50:38,052] ERROR {org.wso2.transport.http.netty.listener.SourceHandler} - Error while notifying listeners org.wso2.extension.siddhi.io.http.source.exception.HttpSourceAdaptorRuntimeException: Request type is not a type of POST
at org.wso2.extension.siddhi.io.http.source.HTTPConnectorListener.onMessage(HTTPConnectorListener.java:75)
at org.wso2.transport.http.netty.contractimpl.HttpWsServerConnectorFuture.notifyHttpListener(HttpWsServerConnectorFuture.java:77)
at org.wso2.transport.http.netty.listener.SourceHandler.notifyRequestListener(SourceHandler.java:200)
at org.wso2.transport.http.netty.listener.SourceHandler.channelRead(SourceHandler.java:119)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at org.wso2.transport.http.netty.listener.WebSocketServerHandshakeHandler.channelRead(WebSocketServerHandshakeHandler.java:100)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at org.wso2.transport.http.netty.listener.UriAndHeaderLengthValidator.channelRead(UriAndHeaderLengthValidator.java:64)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
Suggested Labels:
Suggested Assignees:
Affected Product Version:
OS, DB, other environment details and versions:
Steps to reproduce:
Related Issues:
Incorrect testcase naming format.
Description:
Incorrect test case naming format which cause to skip test at building.
Suggested Labels:
Bug
Suggested Assignees:
@Yasara123
Refresh token grant is not considered in HTTP sink calls
Description:
In the HTTP sink, the refresh token is not considered even though it is specified.
Affected Product Version:
SI 1.1.0
Multiple http request sources
Description:
Multiple http request sources in the same file dont allow connection to the second named source (get connection refused).
@source(type = 'http', receiver.url = "http://0.0.0.0:8006/ctab",@map(type = 'json'))
define stream CtabUrl (url string);
@source(type='http-request', source.id='StreamSource', receiver.url='http://0.0.0.0:8007/url', connection.timeout='150000', @Map(type='json', @attributes(messageId='trp:messageId')))
define stream UrlStream (messageId string);
Affected Product Version:
Ubuntu
Can this extension be used in standalone Siddhi Library?/ Is there any workaround method to enable http receiver?
Description:
I noticed that it is written in Siddhi-IO-HTTP introduction
... This extension only works inside the WSO2 Data Analytic Server and cannot be run with standalone siddhi.
However, in the "How To Use" section below. It suggest the method of including it as a java library. I tried to include it in my siddhi project and it reported "NoSuchMethodError" in receiving http message.
(Detailed message is posted in Issues of siddhi library))
Does that mean if I just want to use siddhi as library then I cannot use this extension?
Is there any way I can do to establish http stream listener by using siddhi library?
Thank you in advance!
Suggested Labels:
Suggested Assignees:
Affected Product Version:
Siddhi - 4.1.17
HTTP Extention - 1.0.18
OS, DB, other environment details and versions:
IntelliJ IDEA 2017.3.5 (Community Edition)
Build #IC-173.4674.33, built on March 6, 2018
JRE: 1.8.0_152-release-1024-b15 amd64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
Windows 10 10.0
Steps to reproduce:
Related Issues:
Issue in the OSGI bundle activator exception handling/logging
Description:
In the existing implementation [1], if the exception throws while initializing the component, its bubble up to OSGi framework, but the problem is carbon logging uses pax logging as logging framework and it's not supported for Eclipse Equinox logging framework. Hence this OSGi bundle exception not captured by the logging framework.
A fix would be: wso2/carbon-secvault#55
Suggested Labels:
v1.x.x
Suggested Assignees:
Affected Product Version:
OS, DB, other environment details and versions:
Steps to reproduce:
Related Issues:
N/A
Adding a property to enable hostname verification
Description:
Hostname verification should be enabled by default. But, as per the current implementation, there isn't any defined property in the transport level that would trigger this behavior.
Suggested Labels:
wso2sp
Affected Product Version:
wso2sp-4.3.0
Error while notifying listeners and request type is not a type of POST
Description:
For the following Siddhi app deployment in K8s, I got the following error and the Siddhi runner did not come to the healthy state due to this error.
Siddhi App
@App:name("0.InputRoute-passthrough")
@source(type='http', receiver.url='http://0.0.0.0:9090/route', @map(type='json'))
define stream passthroughInputRouteStream(routeNo string);
@sink(type='nats',cluster.id='siddhi-stan',destination = '0.InputRoute_InputRouteStream', bootstrap.servers='nats://siddhi-nats:4222',@map(type='text'))
define stream InputRouteStream(routeNo string);
from passthroughInputRouteStream select * insert into InputRouteStream;Error Log
[2020-01-18 02:50:47,196] ERROR {org.wso2.transport.http.netty.contractimpl.listener.states.ReceivingHeaders} - Error while notifying listeners io.siddhi.extension.io.http.source.exception.HttpSourceAdaptorRuntimeException: Request type is not a type of POST
at io.siddhi.extension.io.http.source.HTTPConnectorListener.onMessage(HTTPConnectorListener.java:76)
at org.wso2.transport.http.netty.contractimpl.HttpWsServerConnectorFuture.notifyHttpListener(HttpWsServerConnectorFuture.java:72)
at org.wso2.transport.http.netty.contractimpl.listener.states.ReceivingHeaders.notifyRequestListener(ReceivingHeaders.java:98)
at org.wso2.transport.http.netty.contractimpl.listener.states.ReceivingHeaders.readInboundRequestHeaders(ReceivingHeaders.java:77)
at org.wso2.transport.http.netty.contractimpl.listener.states.ListenerReqRespStateManager.readInboundRequestHeaders(ListenerReqRespStateManager.java:38)
at org.wso2.transport.http.netty.contractimpl.listener.SourceHandler.channelRead(SourceHandler.java:130)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at org.wso2.transport.http.netty.contractimpl.listener.WebSocketServerHandshakeHandler.channelRead(WebSocketServerHandshakeHandler.java:135)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at org.wso2.transport.http.netty.contractimpl.listener.UriAndHeaderLengthValidator.channelRead(UriAndHeaderLengthValidator.java:64)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Unknown Source)Affected Product Version:
Siddhi runner 5.1.2
Redirection is not handled by http sink
Description:
When the sink receives a 301 response for a request, it does not redirect to the URL it receives.
Affected Product Version:
SP 4.0.0, 4.1.0, 4.2.0, 4.3.0
Proper error logging must be done when ports used in HTTP configurations are already in use
Description:
If the port we have used to make the HTTP(S) connection is already being used by a different program in the server, an error must be thrown. However, no such error is displayed on the console as of now. The only observation is, the connection messages such as 'Source Listener has created for url http://localhost:5005/inputStream' would not be displayed when the Siddhi application containing the relevant HTTP source configuration is run.
Steps to reproduce:
Consider the following HTTP source configuration (Here, 8005 is assumed to be a port already being used by a different program in the system). Add this configuration to a siddhi app, save it and run on the editor environment (or as worker) in WSO2 Stream Processor. You would notice that no connection messages are displayed on the console. However, no errors are displayed either. When attempting to publish data only, the user would know that the HTTP connection is not established.
@source(type = 'http', receiver.url='http://localhost:8005/inputStream', basic.auth.enabled='false',
@Map(type='text'))
define stream inputStream(name string, age int, country string);
This issue must be resolved by logging proper error messages, when the port used in HTTP source configuration is already used by a different program.
HTTP requests fail due to passing whole URL as the resource
Description:
When making a GET request for a resource, the whole URL sent as the resource that needs to be accessed. Therefore it fails with a 404.
The following is a correct GET request to the same resource.
Suggested Labels:
bug
Suggested Assignees:
@minudika
Affected Product Version:
WSO2SP-4.0.0, WSO2SP-4.1.0
Query params are ignored from publisher.url
Description:
The query params placed in publisher.url are not been sent to the server.
Here is the code snippet
@App:name("Http_trial")
@App:description("Description of the plan")
define stream SweetProductionStream (projectCode string, totalBid int );
@sink(type="http",
method="POST",
headers="'Content-Type:application/json','Accept:application/json','Host:192.168.43.79'",
follow.redirect='true',
publisher.url = "http://192.168.43.79:8080/path?abc=xyz",
@map(type='json', validate.json='true', @payload("""{"properties":[{"name":"amount","value":"100"}]}""")))
define stream LowProductionAlertStream (projectCode string, totalBid int);
@info(name = 'query1')
from SweetProductionStream
select *
insert into LowProductionAlertStream;Packet captured using Wireshark
The request uri seen in the packet details only contains /path and is missing ?abc=xyz
Is there any other way to specify query params ? Or am I missing something here ?
Affected Product Version:
v1.0.33
OS, DB, other environment details and versions:
Windows 10 and Ubuntu 16.04
Steps to reproduce:
Use WSO2 Stream Processor Studio to create the file.
Create a http sink having query params in publisher.url
‘application/x-www-form-urlencoded’ request header cannot be automatically added
Description:
When type='keyvalue', the ‘application/x-www-form-urlencoded’ request header cannot be automatically added
Suggested Labels:
Suggested Assignees:
Affected Product Version: all
OS, DB, other environment details and versions:
Steps to reproduce:
'io.siddhi.extension.io.http.sink.HttpSink#generateCarbonMessage' function, 'if (contentType.contains(mapType)) {
httpHeaders.set(HttpConstants.HTTP_CONTENT_TYPE, contentType);
}' logic vulnerability.
Related Issues:
Allow client credentials to be sent in the token request body for OAuth password grant
Description:
Some Oauth providers require the token request body to include the client credentials in the request body as seen in here, currently our implementation only support username and password and the grant_type in the message body.
Exception occurred in TargetHandler : Connection reset by peer
Description:
After posting to SharePoint Online, I received the following error messages - and can't post to SharePoint Online for a little while after that.
INFO {org.wso2.transport.http.netty.contractimpl.sender.TargetHandler} - Exception occurred in TargetHandler : Connection reset by peer
INFO {org.wso2.transport.http.netty.contractimpl.sender.TargetHandler} - Exception occurred in TargetHandler : Connection reset by peer
INFO {org.wso2.transport.http.netty.contractimpl.sender.TargetHandler} - Exception occurred in TargetHandler : Connection reset by peer
Suggested Labels:
Suggested Assignees:
Affected Product Version:
1.1.0
OS, DB, other environment details and versions:
Steps to reproduce:
Related Issues:
Siddhi HTTP Sink documentation not clear about SSL parameters to bypass SSL certification errors
Description:
Siddhi HTTP Sink documentation not clear about SSL parameters to bypass SSL certification errors (i.e: expired or self signed).
Suggested Labels:
Type/Docs, Type/Question
Affected Product Version:
WSO2 Stream Processor 4.3.0
OS, DB, other environment details and versions:
Red Hat Enterprise Linux Server release 7.5 (Maipo)
Steps to reproduce:
Tested using this sink configuration:
@sink(type='http',
publisher.url='https://xxx.xxx.xxx.xxx:5050/v1/executions',
headers="'content-type:application/json','Accept-Encoding:gzip, deflate','Connection:keep-alive','X-Auth-Token: be991689c1394444b4966ed400a7eb80'",
method='POST',
@map(type='json', @payload(""" {"action":"{{action}}","parameters":{"url":"{{managed_object}}"} } """)))
In this case the HTTPS endpoint has a self-signed certificate and the http sink produces this exception:
[2018-11-16 10:54:29,871] WARN {io.netty.channel.DefaultChannelPipeline} - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1478)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
... 16 more
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416)
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272)
... 20 more
Caused by: java.security.cert.CertificateException: No subject alternative names present
at sun.security.util.HostnameChecker.matchIP(HostnameChecker.java:145)
at sun.security.util.HostnameChecker.match(HostnameChecker.java:94)
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1501)
Related Issues:
Handle success range Http responses
Description:
Since only HTTP status code 200 is considered as success in current http sink implementation, all other 2XX based codes are considered as errors and retry happens.
Suggested Labels:
bug
Suggested Assignees:
Affected Product Version:
siddhi-io-http
2.2.0
OS, DB, other environment details and versions:
macOS
Steps to reproduce:
Call a backend API that returns a 2XX based status code other than 200.
Related Issues:
N/A
OAuth Token EP does not support scopes
Description:
When using OAuth Token EP, scopes cannot be specified. This needs to be modified.
Affected Product Version:
siddhi-io-http 2.2.0
XML response is clipped at certain value, and parsing fails
Description:
When a big XML response is received by @source(type='http-response'...), the response is clipped at certain length, and then the clipped response is tried to be parsed (which fails)
Suggested Labels:
bug
Suggested Assignees:
@minudika
Affected Product Version:
- siddhi-io-http-1.0.39
- latest SP
Steps to reproduce:
- Create a http service that responds with a big XML response. (In my case, big was ~1300 character long w/o whitespaces)
- Setup http-request / http-response. I have it like this:
--Sink stream - get http events
@sink(type = 'http-request', publisher.url = 'http://localhost:3000/lastevent',
sink.id = 'request-sink',
@map(type = 'json'))
define stream postHttpRequestStream(payloadBody String); -- payloadBody is currently not used
--Source stream - incoming http events
@source(type = 'http-response', sink.id = 'request-sink', http.status.code = '200',
@map(type = 'xml',
@attributes(eventName = 'reply/equipmentReplies/movementEvents/sightingEventCodeText')))
define stream httpResponseStream2xx(eventName string);
The response from above URL - localhost:3000/lastevent, in postman is:
- Run this code in stream processor, I get an error-
[2018-10-18_15-51-26_165] INFO {org.wso2.extension.siddhi.map.xml.sourcemapper.XmlSourceMapper} - Error parsing incoming XML event : trainIIAlphaCode>ICHR</trainIIAlphaCode><sightingEventCode>4050</sightingEventCode><sightingEventCodeText>JUNCTION RECEIVED</sightingEventCodeText><toMarkId>BNSF</toMarkId><trainInd>UP</trainInd></movementEvents><tripPlan><tripPlanId/></tripPlan><waybill><commodity/><destination><city/><splc/><state/></destination><grossWeight>0</grossWeight><netWeight>0</netWeight><tareWeight>0</tareWeight><weightCode>8</weightCode></waybill><requestType>LAST_EVENT</requestType></equipmentReplies></reply></ns2:TracingPlatformRequest>. Reason: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1,1] Message: Content is not allowed in prolog.. Hence dropping message chunk (Encoded)
Additional information:
If I keep resizing the XML output from server, by removing XML elements, I reach a point where the error is no more.
I have validated my XML via multiple XML validators online. This is the raw version of the XML output:
<ns2:TracingPlatformRequest xmlns:ns2="http://someUrl/someEndPoint"><TracingPlatformRequestHeader>TracingPlatformRequestHeaderVal</TracingPlatformRequestHeader><reply><equipmentReplies><groupSequenceNumber>0001</groupSequenceNumber><equipment><equipmentId><equipmentInitial>BNSF</equipmentInitial><equipmentNumber>0000000100</equipmentNumber></equipmentId></equipment><umlerDetails><owner>BNSF</owner></umlerDetails><movementEvents><eta><etaDate>Thu Oct 18 2018</etaDate><ediEventCode/><eventCode/><location><splc/></location><reportingScac/></eta><fromMarkId>UP</fromMarkId><loadEmptyStatus>E</loadEmptyStatus><location><splc>536870000</splc></location><reportingRailroadSCAC>BNSF</reportingRailroadSCAC><sightingDate>Thu Oct 18 2018</sightingDate><sightingClmCode>R</sightingClmCode><trainIIAlphaCode>ICHR</trainIIAlphaCode><sightingEventCode>4050</sightingEventCode><sightingEventCodeText>JUNCTION RECEIVED</sightingEventCodeText><toMarkId>BNSF</toMarkId><trainInd>UP</trainInd></movementEvents><tripPlan><tripPlanId/></tripPlan><waybill><commodity/><destination><city/><splc/><state/></destination><grossWeight>0</grossWeight><netWeight>0</netWeight><tareWeight>0</tareWeight><weightCode>8</weightCode></waybill><requestType>LAST_EVENT</requestType></equipmentReplies></reply></ns2:TracingPlatformRequest>
Thanks,
Ravi Pandey
File handle leak? （A large number of "pipe/eventpoll" leaks）
Description:
siddhi sql like this:
@app:name("fileHandleTest")
@app:description('test')
@app:statistics(reporter = 'jmx')
define stream testStream(triggered_time long);
@sink(type='http', publisher.url='http://192.168.1.23:7031/alertCenter/merge/mergeData', method='GET',@Map(type='json'))
define stream OutputStream (triggered_time long);
@info(name = "http_data_deal_with")
from testStream#log("---------dd---------------",true)
select *
insert into OutputStream;
Suggested Labels:
Suggested Assignees:
Affected Product Version:
siddhi-core : <= 4.2.17
siddhi-io-http: <=1.0.42
OS, DB, other environment details and versions:
CentOS Linux release 7.5.1804 (Core)
Steps to reproduce:
step1: start siddhi of this siddhi-sql appliaction,
execute cmd:
lsof -p 7921 | wc -l
321
step2: then stop siddhi;
lsof -p 7921 | wc -l
351
step3: start siddhi of this siddhi-sql appliaction,
execute cmd:
lsof -p 7921 | wc -l
381
step4: then stop siddhi;
execute cmd:
lsof -p 7921 | wc -l
381
lsof -p
the result:
java 7921 root 322u a_inode 0,10 0 6091 [eventpoll]
java 7921 root 323r FIFO 0,9 0t0 325541144 pipe
java 7921 root 324w FIFO 0,9 0t0 325541144 pipe
java 7921 root 325u a_inode 0,10 0 6091 [eventpoll]
java 7921 root 326r FIFO 0,9 0t0 325541145 pipe
java 7921 root 327w FIFO 0,9 0t0 325541145 pipe
java 7921 root 328u a_inode 0,10 0 6091 [eventpoll]
java 7921 root 329r FIFO 0,9 0t0 325541146 pipe
java 7921 root 330w FIFO 0,9 0t0 325541146 pipe
java 7921 root 331u a_inode 0,10 0 6091 [eventpoll]
java 7921 root 332r FIFO 0,9 0t0 325541147 pipe
java 7921 root 333w FIFO 0,9 0t0 325541147 pipe
java 7921 root 334u a_inode 0,10 0 6091 [eventpoll]
java 7921 root 335r FIFO 0,9 0t0 325541148 pipe
java 7921 root 336w FIFO 0,9 0t0 325541148 pipe
java 7921 root 337u a_inode 0,10 0 6091 [eventpoll]
java 7921 root 338r FIFO 0,9 0t0 325541149 pipe
java 7921 root 339w FIFO 0,9 0t0 325541149 pipe
java 7921 root 340u a_inode 0,10 0 6091 [eventpoll]
java 7921 root 341r FIFO 0,9 0t0 325541150 pipe
java 7921 root 342w FIFO 0,9 0t0 325541150 pipe
java 7921 root 343u a_inode 0,10 0 6091 [eventpoll]
NPE thrown when starting and stopping siddhi app with multiple HTTP sources
Description:
$subject
Stacktrace
[2020-01-09 13:26:16,561] ERROR {io.siddhi.core.stream.input.source.Source} - Error on 'JsonInputMapper'. Error while connecting at Source 'http' at 'PersonalDetailsDefaultMapping'. java.lang.NullPointerException
at io.siddhi.extension.io.http.source.HttpConnectorRegistry.createHttpServerConnector(HttpConnectorRegistry.java:218)
at io.siddhi.extension.io.http.source.HttpSource.connect(HttpSource.java:418)
at io.siddhi.core.stream.input.source.Source.connectWithRetry(Source.java:160)
at io.siddhi.core.SiddhiAppRuntimeImpl.startSources(SiddhiAppRuntimeImpl.java:502)
at io.siddhi.core.SiddhiAppRuntimeImpl.start(SiddhiAppRuntimeImpl.java:427)
at io.siddhi.distribution.editor.core.internal.DebugRuntime.start(DebugRuntime.java:93)
at io.siddhi.distribution.editor.core.internal.DebugProcessorService.start(DebugProcessorService.java:42)
at io.siddhi.distribution.editor.core.internal.EditorMicroservice.start(EditorMicroservice.java:761)
at io.siddhi.distribution.editor.core.internal.EditorMicroservice.startWithVariables(EditorMicroservice.java:781)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.wso2.msf4j.internal.router.HttpMethodInfo.invokeResource(HttpMethodInfo.java:187)
at org.wso2.msf4j.internal.router.HttpMethodInfo.invoke(HttpMethodInfo.java:143)
at org.wso2.msf4j.internal.MSF4JHttpConnectorListener.dispatchMethod(MSF4JHttpConnectorListener.java:218)
at org.wso2.msf4j.internal.MSF4JHttpConnectorListener.lambda$onMessage$58(MSF4JHttpConnectorListener.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Siddhi App:
/**
{
event : {
"name" : "siddhi.user",
"age" : "25"
"address" : "20, Palm Groove, Colombo, Sri Lanka",
"contact": "+9xxxxxxx1,+9xxxxxxx2"
}
}
**/
@Source(type = 'http', receiver.url = 'http://localhost:8006/PersonalDetails',
@map(type = 'json'))
define stream PersonalDetailsDefaultMapping (name string, age int, address string, contact string);
/**
{
"name" : "siddhi.user",
"age" : "25",
"address" : {
"number": "20",
"lane": "Palm Groove",
"city": "Colombo"
"country": "Sri Lanka",
},
"contact": ["+9xxxxxxx1, +9xxxxxxx2"]
}
**/
@Source(type = 'http', receiver.url = 'http://localhost:8006/PersonalDetailsCustom',
@map(type = 'json',
@attributes(name = 'name', age ='age', addressCountry = 'address.country', contact = 'contact[0]' ) ))
@sink(type='log')
define stream PersonalDetailsCustomMapping (name string, age int, addressCountry string, contact string);
/**
{
person : [
{
"name" : "siddhi.user",
"age" : "25",
"address" : {
"number": "20",
"lane": "Palm Groove",
"city": "Colombo"
"country": "Sri Lanka",
},
"contact": ["+9xxxxxxx1, +9xxxxxxx2"]
},
{
"name" : "siddhi.user",
"age" : "25",
"address" : {
"number": "20",
"lane": "Palm Groove",
"city": "Colombo"
"country": "Sri Lanka",
},
"contact": ["+9xxxxxxx1, +9xxxxxxx2"]
},
]
}
**/
@Source(type = 'http', receiver.url = 'http://localhost:8006/PersonalDetailsMultiple',
@map(type = 'json', enclosing.element = 'person',
@attributes(name = 'name', age ='age', addressCountry = 'address.country', contact = 'contact' ) ))
@sink(type='log')
define stream PersonalDetailsEnclosingElement (name string, age int, addressCountry string, contact string);
Affected Product Version:
Latest
OS, DB, other environment details and versions:
Siddhi tooling daily build 8/1/2020
Issue in accessing same boss and worker group in HTTP carbon transport from MSF4J and Siddhi together
It is workable without shutting down EventLoop Groups from ServerConnectorController as current implementation. But in that scenario both MSF4j and Siddhi uses same ServerConnectorController and it gives following issues when considering the use case for Siddhi.
- Cannot configure event group sizes (boss group size and worker group size) independently from Siddhi side.
- Cannot pause Siddi level transport since it will block the thread group as in . Even if we use a queue to add events, it will block the thread group when inserting to the queue. In that scenario thread group will also get blocked for someone else like MSF4j.
This issue is also reference to
wso2/carbon-transports#340
And following mail thread
[Dev][Architecture] Issue in accessing same boss and worker group in HTTP carbon transport from MSF4J and Siddhi together
Allow users to configure connection pools in http sink
Description:
Users should be able to configure connection pools in siddhi http sink in order to tune up performance.
Multiple HTTP Call Response Sources
When using HTTP call and call response scenario we allow 2 sources with http.status.code = 200 and [2|4]00 to register in Siddhi app. However, when the response arrives, according to our current logic in findAndGetResponseSource method we randomly pick a source from the map matching the regex and send a response. Please find a sample app to simulate this below.
define stream FooStream (message String, headers String);
@sink(type='http-request',publisher.url='http://localhost:8005/abc',
method='POST',
headers='{{headers}}',sink.id='source-1',
@map(type='json', @payload('{{message}}')))
Define stream BarStream (message String, headers String);
@source(type='http-response', sink.id='source-1', http.status.code='[2|4]00',
@map(type='json',@attributes(name='name', id='id')))
define stream responseStream1(name String, id int);
@source(type='http-response', sink.id='source-1', http.status.code='200',
@map(type='json',@attributes(name='name', id='id')))
define stream responseStream2(name String, id int);
We need to have a robust mechanism to handle such responses which need to be sent to multiple sources
Unable to receive the json responce to the source
Description:
I am trying to connect to the github api through a siddhi app using the bellow code,
@app:name("RequestResponseSample")
@app:description('This sample siddhi app demonstrates how to use transport properties along with request sink and response source')
define stream regStream (id String);
@sink(type='http-request', publisher.url='http://api.github.com/repos/wso2/product-sp/releases',
method='POST',
chunk.disabled='true',
sink.id='reg-company',
@Map(type='json', @payload("""{"id":"{{id}}"}""")))
define stream requestStream (id string);
-- source to receive successful responses (which have 2xx status codes)
@source(type='http-response' , sink.id='reg-company', http.status.code='200',
@Map(type='json', @attributes(tag_name='$.tag_name')))
define stream responseStream(tag_name string);
-- source to receive the responses with 4xx status codes
@source(type='http-response' , sink.id='reg-company', http.status.code='403' ,
@Map(type='text', regex.A='((.|\n)*)', @attributes(message='A[1]')))
define stream responseStream4xx(message string);
@sink(type='log')
define stream logStream(tag_name string);
from regStream
select id
insert into requestStream;
from responseStream
select *
insert into logStream;
But I am not getting a response. It displays an error message saying that the channel is inactive
Bellow is the response I received when I requested through "Postman"
NPE while invoking EP with Token EP
Description:
While invoking an endpoint with Token endpoint specified in the HTTP sink, the following exception occurs.
ERROR {io.siddhi.core.stream.StreamJunction} - Error in 'SiddhiApp' after consuming events from Stream 'FooStream', null. Hence, dropping event 'Event{timestamp=1598441762269, data=[***], isExpired=false}' java.lang.NullPointerException
at io.siddhi.extension.io.http.sink.updatetoken.HttpRequest.sendPostRequest(HttpRequest.java:127)
at io.siddhi.extension.io.http.sink.updatetoken.HttpsClient.getPasswordGrantAccessToken(HttpsClient.java:95)
at io.siddhi.extension.io.http.sink.HttpSink.getAccessToken(HttpSink.java:817)
at io.siddhi.extension.io.http.sink.HttpSink.setAccessToken(HttpSink.java:844)
at io.siddhi.extension.io.http.sink.HttpSink.sendOauthRequest(HttpSink.java:668)
at io.siddhi.extension.io.http.sink.HttpSink.publish(HttpSink.java:607)
at io.siddhi.core.stream.output.sink.Sink.publish(Sink.java:182)
at io.siddhi.extension.map.json.sinkmapper.JsonSinkMapper.mapAndSend(JsonSinkMapper.java:209)
at io.siddhi.core.stream.output.sink.SinkMapper.mapAndSend(SinkMapper.java:180)
at io.siddhi.core.stream.output.sink.SinkCallback.receive(SinkCallback.java:55)
at io.siddhi.core.stream.StreamJunction.sendEvent(StreamJunction.java:199)
at io.siddhi.core.stream.StreamJunction$Publisher.send(StreamJunction.java:474)
at io.siddhi.core.stream.input.InputDistributor.send(InputDistributor.java:34)
at io.siddhi.core.stream.input.InputEntryValve.send(InputEntryValve.java:45)
at io.siddhi.core.stream.input.InputHandler.send(InputHandler.java:78)
at org.wso2.carbon.siddhi.editor.core.internal.DebuggerEventStreamService.pushEvent(DebuggerEventStreamService.java:70)
at org.wso2.carbon.event.simulator.core.internal.generator.SingleEventGenerator.sendEvent(SingleEventGenerator.java:85)
at org.wso2.carbon.event.simulator.core.impl.SingleApiServiceImpl.runSingleSimulation(SingleApiServiceImpl.java:17)
at org.wso2.carbon.event.simulator.core.api.SingleApi.runSingleSimulation(SingleApi.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.wso2.msf4j.internal.router.HttpMethodInfo.invokeResource(HttpMethodInfo.java:187)
at org.wso2.msf4j.internal.router.HttpMethodInfo.invoke(HttpMethodInfo.java:143)
at org.wso2.msf4j.internal.MSF4JHttpConnectorListener.dispatchMethod(MSF4JHttpConnectorListener.java:218)
at org.wso2.msf4j.internal.MSF4JHttpConnectorListener.lambda$onMessage$0(MSF4JHttpConnectorListener.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Affected Product Version:
- WSO2 Streaming Integrator Tooling 1.0.0
- WSO2 Identity server 5.10.0
Steps to reproduce:
The sample Siddhi file is as follows.
@App:name("SiddhiApp")
@sink(type='http',
publisher.url='http://jsonplaceholder.typicode.com/posts',
method="POST",
headers="'Content-Type: application/json'",
oauth.username="***",
oauth.password="***",
consumer.key="***",
consumer.secret="***",
token.url='https://localhost:9443/oauth2/token',
ssl.verification.disabled="true",
@map(
type='json'
))
define stream FooStream (id int);
define stream BarStream(id int);
from BarStream
select *
insert into FooStream;
Misleading error when backend is not reachable.
Description:
In a sink of type 'http-request', if the backend specified in the 'publisher.url' parameter is not reachable, upon simulating an event, the following error is being printed even when a source of 'http-response' is specified to handle responses with 500 status code (using a regex).
ERROR {org.wso2.extension.siddhi.io.http.source.HttpResponseMessageListener} - No source of type 'http-response' for status code '500' has been defined. Hence dropping the response message.
For the same scenario, from SI 1.0.1 onwards, the correct error is being printed. Following is a sample.
ERROR {io.siddhi.core.stream.output.sink.Sink} - Error on 'test_http_status_code'. Dropping event at Sink 'http-request' at 'CreditCardStream' as its still trying to reconnect!, events dropped '{CardNumber=1111}'
Affected Product Version:
SP 4.3.0, SP 4.4.0, SI 1.0.0
Steps to reproduce:
- Specify an unreachable backend for 'publisher.url' in 'http-request' sink.
- Make sure a source of type 'http-response' is specified to handle responses with '500' status code, using a regex.
e.g. http.status.code = '[3|4|5]00' - Simulate an event.
Host header is missing when invoking an endpoint via http-request
Description:
When invoking an API via http-request sink, Host HTTP header is not getting set in the HTTP request. This causes 400 Bad Request error from the server.
This needs to be fixed to get the hostname from the URL and set into the header.
Affected Product Version:
1.0.39
Add testcases for OAuth cases.
Description:
This issue is created to keep track of the task: Testcases for OAuth cases.
Related Issues:
#100
Suggested Assignees:
https://github.com/thushaaanthan
Revisit HTTPSink implementation & clean the source code
Description:
It seems, there are a lot places that we could improve the HTTPSink implementation. It seems like some code segments are bit smelly. Need to check this out ASAP.
Rename http-request and http-response to more meaningful names
Description:
http-request and http-response are used for both incoming and outgoing calls. Hence this causes confusion among users. This need to be renamed to differentiate the call type.
Connections are not closed correctly in 1.0.44 and 1.0.45 release
Basic Authentication Is Not Working
Description:
The Basic Authentication mechanism is not working as expected, and this behaviour is observed in SI-4.X versions when tested locally.
For example, when we use the http-call sink method with Basic Auth enabled as shown below, the Base64 encoded username and password is getting malformed.
@sink(type='http-call', sink.id='D1', publisher.url='http://localhost:8281/foobar/1.0', method='POST', basic.auth.username='admin', basic.auth.password='admin',
headers='Content-Type: application/json', ...,
When we checked the wirelogs from the backend service, we observed that the Authorization header value being sent to the backend was incorrect,
"Authorization: Basic UnpooledHeapByteBuf(ridx: 0, widx: 11, cap: 11/11)[\r][\n]"
Seems like there is an issue with the Base64 encoding of the Authorization header at [1]. The encoded header is in bytes and it needs to be converted to string through byte methods which causes the incorrect value to get assigned as the Authorization header. Since the encoded username and password value are invalid, the http-call sink method fails.
[1]
Steps to reproduce:
- Create a backend with Basic Auth enabled
- Create a Siddhi App which calls the above backend using an
http-callsink method with Basic Auth. - When invoked, the Basic Auth mechanism will fail due to the improper encoded username and password.
Affected Product Version:
- Streaming-Integrator-4.X.X
Sometimes long responses are not completely received by http-response source.
Description:
When using a request sink and responses source, if the response is a long one, sometimes only a part of the response is captured by the responses source.
Wrong body send to get Oauth token
Description:
The payload send to get a token here looks like following in current implementation &username=abc&password=123&grant_type=password When this is send to /token end point it will throw an exception since the payload cannot be parced due to the first charactor being '&'" -m "Fixed by changing the payload generation
Suggested Labels:
bug
Suggested Assignees:
Affected Product Version:
siddhi-io-http
2.2.0
OS, DB, other environment details and versions:
macOS
Steps to reproduce:
Call an endpoint protected with WSO2 API manager via http sink. Provide username, password, client_id, client_secreat only and let the access token be generated by http sink. This fails
Related Issues:
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.