Coder Social home page Coder Social logo

siddhi-io / siddhi-io-http Goto Github PK

View Code? Open in Web Editor NEW
13.0 13.0 52.0 5.55 MB

Extension that allows you to receive and publish events via HTTP and https transports

Home Page: https://siddhi-io.github.io/siddhi-io-http/

License: Apache License 2.0

Java 100.00%
curl extension http https io request response rest siddhi soap

siddhi-io-http's People

Contributors

anoukh avatar anugayan avatar charukak avatar chathurikaa avatar dilini-muthumala avatar dnwick avatar gokul avatar grainier avatar inoshperera avatar ksdperera avatar lasanthas avatar madurangasiriwardena avatar madushadhanushka avatar maheshika avatar minudika avatar mohanvive avatar nisalaniroshana avatar niveathika avatar oshiwindsor avatar pcnfernando avatar pradeepajey1 avatar ramindu90 avatar rukshiw avatar sajithshn avatar senthuran16 avatar suhothayan avatar sybernix avatar thushaaanthan avatar tishan89 avatar wso2-jenkins-bot avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

siddhi-io-http's Issues

HTTP Source is removed when there multiple apps with same source hostname,port and context

Description:
Let's assume a scenario where we have siddhi app with HTTP source and we try to deploy another siddhi app with same HTTP source. In that case, it will remove the HTTP source from the first app also.

Suggested Labels:
@bug

Steps to reproduce:

  • Create two siddhi apps with same HTTP source.
  • Then add these siddhi apps one by one.
  • It will give an error and does not allow to deploy the second app. It will also remove the HTTP source from the first app. So we cannot publish any events to that HTTP source.

HTTP sink GET request fails with HTTP 400

Description:
HTTP sink GET request fails with HTTP 400 because we are not sending in Host header. Need to follow up on this and investigate why we are not sending Host header and what to be expected as per spec

Affected Product Version:
SP 4.x.x

Steps to reproduce:
Can use below Siddhi App to reproduce

@App:name("SiddhiApp")
@App:description("Description of the plan")

-- Please refer to https://docs.wso2.com/display/SP400/Quick+Start+Guide on getting started with SP editor. 

@sink(
    type='http-request', sink.id='id1', publisher.url='http://api.worldbank.org/v2/countries/all/indicators/SP.POP.TOTL', 
    method='GET', headers = "'Host:api.worldbank.org'",
    @map(type='json'))
define stream FooStream (payloadBody String);

@source(type='http-response', sink.id='id1', http.status.code='200',@map(type='text' ))
@sink(type = 'log', prefix = 'LOGGER')
define stream responseStream2xx(body string);

@source(type='http-response', sink.id='id1', http.status.code='400', @map(type='text'))
@sink(type = 'log', prefix = 'LOGGER')
define stream responseStream4xx(body string);

Netty resources leak when performing perf tests

Description:
When performing a perf test using http request-response flow, following LEAK error message is getting printed.

[2019-04-19 15:46:12,400] ERROR {io.netty.util.ResourceLeakDetector} - LEAK: ByteBuf.release() was not called before it's garbage-collected. Enable advanced leak reporting to find out where the leak occurred. To enable adv
anced leak reporting, specify the JVM option '-Dio.netty.leakDetection.level=advanced' or call ResourceLeakDetector.setLevel() See http://netty.io/wiki/reference-counted-objects.html for more information.

After setting the leak detection level to "ADVANCED", it provides following logs.

[2019-05-31 06:37:24,564] ERROR {io.netty.util.ResourceLeakDetector} - LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:273)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:748)
#2:
	io.netty.buffer.AdvancedLeakAwareByteBuf.getByte(AdvancedLeakAwareByteBuf.java:154)
	io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:360)
	io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
	io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:748)
#3:
	io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
	io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:803)
	io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:603)
	io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:227)
	io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
	io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:748)
#4:
	io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByte(AdvancedLeakAwareByteBuf.java:670)
	io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:803)
	io.netty.handler.codec.http.HttpObjectDecoder$LineParser.parse(HttpObjectDecoder.java:852)
	io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:208)
	io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
	io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:748)
#5:
	io.netty.buffer.AdvancedLeakAwareByteBuf.getUnsignedByte(AdvancedLeakAwareByteBuf.java:160)
	io.netty.handler.codec.http.HttpObjectDecoder.skipControlCharacters(HttpObjectDecoder.java:568)
	io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:202)
	io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
	io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:748)
#6:
	Hint: 'codec' will handle the message from this point.
	io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:748)
#7:
	Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
	io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:748)
Created at:
	io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:634)
	io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:345)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:126)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.lang.Thread.run(Thread.java:748)
: 6 leak records were discarded because they were duplicates

CORS handling not available

Description:
When calling endpoints directly from a web application; first an HTTP OPTIONS request is sent to figure permission to access selected resources from the server. Currently in the server, if any request other than an HTTP POST request is received the request is dropped with the status code 400[1].

[1] - https://github.com/wso2-extensions/siddhi-io-http/blob/master/component/src/main/java/org/wso2/extension/siddhi/io/http/source/HTTPConnectorListener.java#L61-L77

And the following error gets logged

[2018-08-20 09:50:38,052] ERROR {org.wso2.transport.http.netty.listener.SourceHandler} - Error while notifying listeners org.wso2.extension.siddhi.io.http.source.exception.HttpSourceAdaptorRuntimeException: Request type is not a type of POST 
	at org.wso2.extension.siddhi.io.http.source.HTTPConnectorListener.onMessage(HTTPConnectorListener.java:75)
	at org.wso2.transport.http.netty.contractimpl.HttpWsServerConnectorFuture.notifyHttpListener(HttpWsServerConnectorFuture.java:77)
	at org.wso2.transport.http.netty.listener.SourceHandler.notifyRequestListener(SourceHandler.java:200)
	at org.wso2.transport.http.netty.listener.SourceHandler.channelRead(SourceHandler.java:119)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at org.wso2.transport.http.netty.listener.WebSocketServerHandshakeHandler.channelRead(WebSocketServerHandshakeHandler.java:100)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
	at org.wso2.transport.http.netty.listener.UriAndHeaderLengthValidator.channelRead(UriAndHeaderLengthValidator.java:64)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	at java.lang.Thread.run(Thread.java:748)

Suggested Labels:

Suggested Assignees:

Affected Product Version:

OS, DB, other environment details and versions:

Steps to reproduce:

Related Issues:

Multiple http request sources

Description:
Multiple http request sources in the same file dont allow connection to the second named source (get connection refused).


@source(type = 'http', receiver.url = "http://0.0.0.0:8006/ctab",@map(type = 'json'))
define stream CtabUrl (url string);
@source(type='http-request', source.id='StreamSource', receiver.url='http://0.0.0.0:8007/url', connection.timeout='150000', @Map(type='json', @attributes(messageId='trp:messageId')))
define stream UrlStream (messageId string);



Affected Product Version:

Ubuntu

Can this extension be used in standalone Siddhi Library?/ Is there any workaround method to enable http receiver?

Description:

I noticed that it is written in Siddhi-IO-HTTP introduction

... This extension only works inside the WSO2 Data Analytic Server and cannot be run with standalone siddhi.

However, in the "How To Use" section below. It suggest the method of including it as a java library. I tried to include it in my siddhi project and it reported "NoSuchMethodError" in receiving http message.
(Detailed message is posted in Issues of siddhi library))

Does that mean if I just want to use siddhi as library then I cannot use this extension?
Is there any way I can do to establish http stream listener by using siddhi library?
Thank you in advance!

Suggested Labels:

Suggested Assignees:

Affected Product Version:
Siddhi - 4.1.17
HTTP Extention - 1.0.18
OS, DB, other environment details and versions:
IntelliJ IDEA 2017.3.5 (Community Edition)
Build #IC-173.4674.33, built on March 6, 2018
JRE: 1.8.0_152-release-1024-b15 amd64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
Windows 10 10.0
Steps to reproduce:

Related Issues:

Issue in the OSGI bundle activator exception handling/logging

Description:
In the existing implementation [1], if the exception throws while initializing the component, its bubble up to OSGi framework, but the problem is carbon logging uses pax logging as logging framework and it's not supported for Eclipse Equinox logging framework. Hence this OSGi bundle exception not captured by the logging framework.

[1] - https://github.com/wso2-extensions/siddhi-io-http/blob/master/component/src/main/java/org/wso2/extension/siddhi/io/http/source/internal/ServiceComponent.java#L49

A fix would be: wso2/carbon-secvault#55

Suggested Labels:
v1.x.x

Suggested Assignees:

Affected Product Version:

OS, DB, other environment details and versions:

Steps to reproduce:

Related Issues:
N/A

Adding a property to enable hostname verification

Description:
Hostname verification should be enabled by default. But, as per the current implementation, there isn't any defined property in the transport level that would trigger this behavior.

Suggested Labels:
wso2sp

Affected Product Version:
wso2sp-4.3.0

Error while notifying listeners and request type is not a type of POST

Description:
For the following Siddhi app deployment in K8s, I got the following error and the Siddhi runner did not come to the healthy state due to this error.

Siddhi App

@App:name("0.InputRoute-passthrough")

@source(type='http', receiver.url='http://0.0.0.0:9090/route', @map(type='json'))
define stream passthroughInputRouteStream(routeNo string);

@sink(type='nats',cluster.id='siddhi-stan',destination = '0.InputRoute_InputRouteStream', bootstrap.servers='nats://siddhi-nats:4222',@map(type='text'))
define stream InputRouteStream(routeNo string);

from passthroughInputRouteStream select * insert into InputRouteStream;

Error Log

[2020-01-18 02:50:47,196] ERROR {org.wso2.transport.http.netty.contractimpl.listener.states.ReceivingHeaders} - Error while notifying listeners io.siddhi.extension.io.http.source.exception.HttpSourceAdaptorRuntimeException: Request type is not a type of POST
        at io.siddhi.extension.io.http.source.HTTPConnectorListener.onMessage(HTTPConnectorListener.java:76)
        at org.wso2.transport.http.netty.contractimpl.HttpWsServerConnectorFuture.notifyHttpListener(HttpWsServerConnectorFuture.java:72)
        at org.wso2.transport.http.netty.contractimpl.listener.states.ReceivingHeaders.notifyRequestListener(ReceivingHeaders.java:98)
        at org.wso2.transport.http.netty.contractimpl.listener.states.ReceivingHeaders.readInboundRequestHeaders(ReceivingHeaders.java:77)
        at org.wso2.transport.http.netty.contractimpl.listener.states.ListenerReqRespStateManager.readInboundRequestHeaders(ListenerReqRespStateManager.java:38)
        at org.wso2.transport.http.netty.contractimpl.listener.SourceHandler.channelRead(SourceHandler.java:130)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at org.wso2.transport.http.netty.contractimpl.listener.WebSocketServerHandshakeHandler.channelRead(WebSocketServerHandshakeHandler.java:135)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
        at org.wso2.transport.http.netty.contractimpl.listener.UriAndHeaderLengthValidator.channelRead(UriAndHeaderLengthValidator.java:64)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
        at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Unknown Source)

Affected Product Version:
Siddhi runner 5.1.2

Redirection is not handled by http sink

Description:
When the sink receives a 301 response for a request, it does not redirect to the URL it receives.

Affected Product Version:
SP 4.0.0, 4.1.0, 4.2.0, 4.3.0

Proper error logging must be done when ports used in HTTP configurations are already in use

Description:
If the port we have used to make the HTTP(S) connection is already being used by a different program in the server, an error must be thrown. However, no such error is displayed on the console as of now. The only observation is, the connection messages such as 'Source Listener has created for url http://localhost:5005/inputStream' would not be displayed when the Siddhi application containing the relevant HTTP source configuration is run.

Steps to reproduce:
Consider the following HTTP source configuration (Here, 8005 is assumed to be a port already being used by a different program in the system). Add this configuration to a siddhi app, save it and run on the editor environment (or as worker) in WSO2 Stream Processor. You would notice that no connection messages are displayed on the console. However, no errors are displayed either. When attempting to publish data only, the user would know that the HTTP connection is not established.

@source(type = 'http', receiver.url='http://localhost:8005/inputStream', basic.auth.enabled='false',
@Map(type='text'))
define stream inputStream(name string, age int, country string);

This issue must be resolved by logging proper error messages, when the port used in HTTP source configuration is already used by a different program.

HTTP requests fail due to passing whole URL as the resource

Description:
When making a GET request for a resource, the whole URL sent as the resource that needs to be accessed. Therefore it fails with a 404.

httperror1

The following is a correct GET request to the same resource.

httperror2

Suggested Labels:
bug

Suggested Assignees:
@minudika

Affected Product Version:
WSO2SP-4.0.0, WSO2SP-4.1.0

Query params are ignored from publisher.url

Description:

The query params placed in publisher.url are not been sent to the server.
Here is the code snippet

@App:name("Http_trial")
@App:description("Description of the plan")

define stream SweetProductionStream (projectCode string, totalBid int );     


@sink(type="http",
        method="POST",
        headers="'Content-Type:application/json','Accept:application/json','Host:192.168.43.79'",
        follow.redirect='true',
        publisher.url = "http://192.168.43.79:8080/path?abc=xyz",
        @map(type='json', validate.json='true', @payload("""{"properties":[{"name":"amount","value":"100"}]}""")))
define stream LowProductionAlertStream (projectCode string, totalBid int);


@info(name = 'query1')
from SweetProductionStream 
select *
insert into LowProductionAlertStream;

Packet captured using Wireshark
siddhiql_packet

The request uri seen in the packet details only contains /path and is missing ?abc=xyz
Is there any other way to specify query params ? Or am I missing something here ?

Affected Product Version:
v1.0.33

OS, DB, other environment details and versions:
Windows 10 and Ubuntu 16.04

Steps to reproduce:
Use WSO2 Stream Processor Studio to create the file.
Create a http sink having query params in publisher.url

‘application/x-www-form-urlencoded’ request header cannot be automatically added

Description:
When type='keyvalue', the ‘application/x-www-form-urlencoded’ request header cannot be automatically added

Suggested Labels:

Suggested Assignees:

Affected Product Version: all

OS, DB, other environment details and versions:

Steps to reproduce:
'io.siddhi.extension.io.http.sink.HttpSink#generateCarbonMessage' function, 'if (contentType.contains(mapType)) {
httpHeaders.set(HttpConstants.HTTP_CONTENT_TYPE, contentType);
}' logic vulnerability.

Related Issues:

Exception occurred in TargetHandler : Connection reset by peer

Description:
After posting to SharePoint Online, I received the following error messages - and can't post to SharePoint Online for a little while after that.

INFO {org.wso2.transport.http.netty.contractimpl.sender.TargetHandler} - Exception occurred in TargetHandler : Connection reset by peer 
INFO {org.wso2.transport.http.netty.contractimpl.sender.TargetHandler} - Exception occurred in TargetHandler : Connection reset by peer 
INFO {org.wso2.transport.http.netty.contractimpl.sender.TargetHandler} - Exception occurred in TargetHandler : Connection reset by peer 

Suggested Labels:

Suggested Assignees:

Affected Product Version:
1.1.0

OS, DB, other environment details and versions:

Steps to reproduce:

Related Issues:

Siddhi HTTP Sink documentation not clear about SSL parameters to bypass SSL certification errors

Description:
Siddhi HTTP Sink documentation not clear about SSL parameters to bypass SSL certification errors (i.e: expired or self signed).

Suggested Labels:
Type/Docs, Type/Question

Affected Product Version:
WSO2 Stream Processor 4.3.0

OS, DB, other environment details and versions:
Red Hat Enterprise Linux Server release 7.5 (Maipo)

Steps to reproduce:

Tested using this sink configuration:

@sink(type='http',
      publisher.url='https://xxx.xxx.xxx.xxx:5050/v1/executions',
      headers="'content-type:application/json','Accept-Encoding:gzip, deflate','Connection:keep-alive','X-Auth-Token: be991689c1394444b4966ed400a7eb80'",
      method='POST',
      @map(type='json', @payload(""" {"action":"{{action}}","parameters":{"url":"{{managed_object}}"} } """)))

In this case the HTTPS endpoint has a self-signed certificate and the http sink produces this exception:

[2018-11-16 10:54:29,871]  WARN {io.netty.channel.DefaultChannelPipeline} - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
        at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1478)
        at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
        ... 16 more
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416)
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272)
        ... 20 more
Caused by: java.security.cert.CertificateException: No subject alternative names present
        at sun.security.util.HostnameChecker.matchIP(HostnameChecker.java:145)
        at sun.security.util.HostnameChecker.match(HostnameChecker.java:94)
        at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
        at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1501)

Related Issues:

Handle success range Http responses

Description:
Since only HTTP status code 200 is considered as success in current http sink implementation, all other 2XX based codes are considered as errors and retry happens.

Suggested Labels:
bug

Suggested Assignees:

Affected Product Version:
siddhi-io-http
2.2.0

OS, DB, other environment details and versions:
macOS

Steps to reproduce:
Call a backend API that returns a 2XX based status code other than 200.

Related Issues:
N/A

XML response is clipped at certain value, and parsing fails

Description:
When a big XML response is received by @source(type='http-response'...), the response is clipped at certain length, and then the clipped response is tried to be parsed (which fails)

Suggested Labels:
bug

Suggested Assignees:
@minudika

Affected Product Version:

  • siddhi-io-http-1.0.39
  • latest SP

Steps to reproduce:

  • Create a http service that responds with a big XML response. (In my case, big was ~1300 character long w/o whitespaces)
  • Setup http-request / http-response. I have it like this:
--Sink stream - get http events
@sink(type = 'http-request', publisher.url = 'http://localhost:3000/lastevent',
	sink.id = 'request-sink', 
	@map(type = 'json'))
define stream postHttpRequestStream(payloadBody String); -- payloadBody is currently not used

--Source stream - incoming http events
@source(type = 'http-response', sink.id = 'request-sink', http.status.code = '200', 
	@map(type = 'xml', 
		@attributes(eventName = 'reply/equipmentReplies/movementEvents/sightingEventCodeText')))
define stream httpResponseStream2xx(eventName string);

The response from above URL - localhost:3000/lastevent, in postman is:
postman response

  • Run this code in stream processor, I get an error-

[2018-10-18_15-51-26_165] INFO {org.wso2.extension.siddhi.map.xml.sourcemapper.XmlSourceMapper} - Error parsing incoming XML event : trainIIAlphaCode>ICHR</trainIIAlphaCode><sightingEventCode>4050</sightingEventCode><sightingEventCodeText>JUNCTION RECEIVED</sightingEventCodeText><toMarkId>BNSF</toMarkId><trainInd>UP</trainInd></movementEvents><tripPlan><tripPlanId/></tripPlan><waybill><commodity/><destination><city/><splc/><state/></destination><grossWeight>0</grossWeight><netWeight>0</netWeight><tareWeight>0</tareWeight><weightCode>8</weightCode></waybill><requestType>LAST_EVENT</requestType></equipmentReplies></reply></ns2:TracingPlatformRequest>. Reason: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1,1] Message: Content is not allowed in prolog.. Hence dropping message chunk (Encoded)


Additional information:
If I keep resizing the XML output from server, by removing XML elements, I reach a point where the error is no more.

I have validated my XML via multiple XML validators online. This is the raw version of the XML output:
<ns2:TracingPlatformRequest xmlns:ns2="http://someUrl/someEndPoint"><TracingPlatformRequestHeader>TracingPlatformRequestHeaderVal</TracingPlatformRequestHeader><reply><equipmentReplies><groupSequenceNumber>0001</groupSequenceNumber><equipment><equipmentId><equipmentInitial>BNSF</equipmentInitial><equipmentNumber>0000000100</equipmentNumber></equipmentId></equipment><umlerDetails><owner>BNSF</owner></umlerDetails><movementEvents><eta><etaDate>Thu Oct 18 2018</etaDate><ediEventCode/><eventCode/><location><splc/></location><reportingScac/></eta><fromMarkId>UP</fromMarkId><loadEmptyStatus>E</loadEmptyStatus><location><splc>536870000</splc></location><reportingRailroadSCAC>BNSF</reportingRailroadSCAC><sightingDate>Thu Oct 18 2018</sightingDate><sightingClmCode>R</sightingClmCode><trainIIAlphaCode>ICHR</trainIIAlphaCode><sightingEventCode>4050</sightingEventCode><sightingEventCodeText>JUNCTION RECEIVED</sightingEventCodeText><toMarkId>BNSF</toMarkId><trainInd>UP</trainInd></movementEvents><tripPlan><tripPlanId/></tripPlan><waybill><commodity/><destination><city/><splc/><state/></destination><grossWeight>0</grossWeight><netWeight>0</netWeight><tareWeight>0</tareWeight><weightCode>8</weightCode></waybill><requestType>LAST_EVENT</requestType></equipmentReplies></reply></ns2:TracingPlatformRequest>

Thanks,
Ravi Pandey

File handle leak? (A large number of "pipe/eventpoll" leaks)

Description:
siddhi sql like this:

@app:name("fileHandleTest")
@app:description('test')
@app:statistics(reporter = 'jmx')

define stream testStream(triggered_time long);
@sink(type='http', publisher.url='http://192.168.1.23:7031/alertCenter/merge/mergeData', method='GET',@Map(type='json'))
define stream OutputStream (triggered_time long);

@info(name = "http_data_deal_with")
from testStream#log("---------dd---------------",true)
select *
insert into OutputStream;

Suggested Labels:

Suggested Assignees:

Affected Product Version:
siddhi-core : <= 4.2.17
siddhi-io-http: <=1.0.42

OS, DB, other environment details and versions:
CentOS Linux release 7.5.1804 (Core)

Steps to reproduce:
step1: start siddhi of this siddhi-sql appliaction,
execute cmd:

lsof -p 7921 | wc -l
321
step2: then stop siddhi;
lsof -p 7921 | wc -l
351
step3: start siddhi of this siddhi-sql appliaction,
execute cmd:
lsof -p 7921 | wc -l
381
step4: then stop siddhi;
execute cmd:
lsof -p 7921 | wc -l
381

lsof -p
the result:
java 7921 root 322u a_inode 0,10 0 6091 [eventpoll]
java 7921 root 323r FIFO 0,9 0t0 325541144 pipe
java 7921 root 324w FIFO 0,9 0t0 325541144 pipe
java 7921 root 325u a_inode 0,10 0 6091 [eventpoll]
java 7921 root 326r FIFO 0,9 0t0 325541145 pipe
java 7921 root 327w FIFO 0,9 0t0 325541145 pipe
java 7921 root 328u a_inode 0,10 0 6091 [eventpoll]
java 7921 root 329r FIFO 0,9 0t0 325541146 pipe
java 7921 root 330w FIFO 0,9 0t0 325541146 pipe
java 7921 root 331u a_inode 0,10 0 6091 [eventpoll]
java 7921 root 332r FIFO 0,9 0t0 325541147 pipe
java 7921 root 333w FIFO 0,9 0t0 325541147 pipe
java 7921 root 334u a_inode 0,10 0 6091 [eventpoll]
java 7921 root 335r FIFO 0,9 0t0 325541148 pipe
java 7921 root 336w FIFO 0,9 0t0 325541148 pipe
java 7921 root 337u a_inode 0,10 0 6091 [eventpoll]
java 7921 root 338r FIFO 0,9 0t0 325541149 pipe
java 7921 root 339w FIFO 0,9 0t0 325541149 pipe
java 7921 root 340u a_inode 0,10 0 6091 [eventpoll]
java 7921 root 341r FIFO 0,9 0t0 325541150 pipe
java 7921 root 342w FIFO 0,9 0t0 325541150 pipe
java 7921 root 343u a_inode 0,10 0 6091 [eventpoll]

NPE thrown when starting and stopping siddhi app with multiple HTTP sources

Description:
$subject

Stacktrace

[2020-01-09 13:26:16,561] ERROR {io.siddhi.core.stream.input.source.Source} - Error on 'JsonInputMapper'. Error while connecting at Source 'http' at 'PersonalDetailsDefaultMapping'. java.lang.NullPointerException
	at io.siddhi.extension.io.http.source.HttpConnectorRegistry.createHttpServerConnector(HttpConnectorRegistry.java:218)
	at io.siddhi.extension.io.http.source.HttpSource.connect(HttpSource.java:418)
	at io.siddhi.core.stream.input.source.Source.connectWithRetry(Source.java:160)
	at io.siddhi.core.SiddhiAppRuntimeImpl.startSources(SiddhiAppRuntimeImpl.java:502)
	at io.siddhi.core.SiddhiAppRuntimeImpl.start(SiddhiAppRuntimeImpl.java:427)
	at io.siddhi.distribution.editor.core.internal.DebugRuntime.start(DebugRuntime.java:93)
	at io.siddhi.distribution.editor.core.internal.DebugProcessorService.start(DebugProcessorService.java:42)
	at io.siddhi.distribution.editor.core.internal.EditorMicroservice.start(EditorMicroservice.java:761)
	at io.siddhi.distribution.editor.core.internal.EditorMicroservice.startWithVariables(EditorMicroservice.java:781)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.wso2.msf4j.internal.router.HttpMethodInfo.invokeResource(HttpMethodInfo.java:187)
	at org.wso2.msf4j.internal.router.HttpMethodInfo.invoke(HttpMethodInfo.java:143)
	at org.wso2.msf4j.internal.MSF4JHttpConnectorListener.dispatchMethod(MSF4JHttpConnectorListener.java:218)
	at org.wso2.msf4j.internal.MSF4JHttpConnectorListener.lambda$onMessage$58(MSF4JHttpConnectorListener.java:129)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Siddhi App:


/** 
{
    event : {
        "name" : "siddhi.user",
        "age" : "25"
        "address" : "20, Palm Groove, Colombo, Sri Lanka",
        "contact": "+9xxxxxxx1,+9xxxxxxx2"
    }
}
**/
@Source(type = 'http', receiver.url = 'http://localhost:8006/PersonalDetails', 
	@map(type = 'json'))
define stream PersonalDetailsDefaultMapping (name string, age int, address string, contact string);


/** 
{
    "name" : "siddhi.user",
    "age" : "25",
    "address" : {
        "number": "20",
        "lane": "Palm Groove",
        "city": "Colombo"
        "country": "Sri Lanka",
    },
    "contact": ["+9xxxxxxx1, +9xxxxxxx2"]
}
**/
@Source(type = 'http', receiver.url = 'http://localhost:8006/PersonalDetailsCustom', 
	@map(type = 'json', 
	       @attributes(name = 'name', age ='age', addressCountry = 'address.country', contact = 'contact[0]' ) ))
	       @sink(type='log') 
define stream PersonalDetailsCustomMapping (name string, age int, addressCountry string, contact string);


/** 
{
    person : [
                {
                    "name" : "siddhi.user",
                    "age" : "25",
                    "address" : {
                        "number": "20",
                        "lane": "Palm Groove",
                        "city": "Colombo"
                        "country": "Sri Lanka",
                    },
                    "contact": ["+9xxxxxxx1, +9xxxxxxx2"]
                }, 
                {
                    "name" : "siddhi.user",
                    "age" : "25",
                    "address" : {
                        "number": "20",
                        "lane": "Palm Groove",
                        "city": "Colombo"
                        "country": "Sri Lanka",
                    },
                    "contact": ["+9xxxxxxx1, +9xxxxxxx2"]
                }, 
                ]
                
}
**/
@Source(type = 'http', receiver.url = 'http://localhost:8006/PersonalDetailsMultiple', 
	@map(type = 'json', enclosing.element = 'person',
	       @attributes(name = 'name', age ='age', addressCountry = 'address.country', contact = 'contact' ) ))
	       @sink(type='log') 
define stream PersonalDetailsEnclosingElement (name string, age int, addressCountry string, contact string);

Affected Product Version:
Latest

OS, DB, other environment details and versions:
Siddhi tooling daily build 8/1/2020

Issue in accessing same boss and worker group in HTTP carbon transport from MSF4J and Siddhi together

It is workable without shutting down EventLoop Groups from ServerConnectorController as current implementation. But in that scenario both MSF4j and Siddhi uses same ServerConnectorController and it gives following issues when considering the use case for Siddhi.

  1. Cannot configure event group sizes (boss group size and worker group size) independently from Siddhi side.
  2. Cannot pause Siddi level transport since it will block the thread group as in . Even if we use a queue to add events, it will block the thread group when inserting to the queue. In that scenario thread group will also get blocked for someone else like MSF4j.

This issue is also reference to
wso2/carbon-transports#340

And following mail thread
[Dev][Architecture] Issue in accessing same boss and worker group in HTTP carbon transport from MSF4J and Siddhi together

Multiple HTTP Call Response Sources

When using HTTP call and call response scenario we allow 2 sources with http.status.code = 200 and [2|4]00 to register in Siddhi app. However, when the response arrives, according to our current logic in findAndGetResponseSource method we randomly pick a source from the map matching the regex and send a response. Please find a sample app to simulate this below.

define stream FooStream (message String, headers String);
@sink(type='http-request',publisher.url='http://localhost:8005/abc', 
method='POST', 
headers='{{headers}}',sink.id='source-1', 
@map(type='json', @payload('{{message}}'))) 
Define stream BarStream (message String, headers String);

@source(type='http-response', sink.id='source-1', http.status.code='[2|4]00', 
@map(type='json',@attributes(name='name', id='id')))
define stream responseStream1(name String, id int);

@source(type='http-response', sink.id='source-1', http.status.code='200', 
@map(type='json',@attributes(name='name', id='id')))
define stream responseStream2(name String, id int);

We need to have a robust mechanism to handle such responses which need to be sent to multiple sources

Unable to receive the json responce to the source

Description:
I am trying to connect to the github api through a siddhi app using the bellow code,

@app:name("RequestResponseSample")
@app:description('This sample siddhi app demonstrates how to use transport properties along with request sink and response source')

define stream regStream (id String);

@sink(type='http-request', publisher.url='http://api.github.com/repos/wso2/product-sp/releases',
method='POST',
chunk.disabled='true',
sink.id='reg-company',
@Map(type='json', @payload("""{"id":"{{id}}"}""")))
define stream requestStream (id string);

-- source to receive successful responses (which have 2xx status codes)
@source(type='http-response' , sink.id='reg-company', http.status.code='200',
@Map(type='json', @attributes(tag_name='$.tag_name')))
define stream responseStream(tag_name string);

-- source to receive the responses with 4xx status codes
@source(type='http-response' , sink.id='reg-company', http.status.code='403' ,
@Map(type='text', regex.A='((.|\n)*)', @attributes(message='A[1]')))
define stream responseStream4xx(message string);

@sink(type='log')
define stream logStream(tag_name string);

from regStream
select id
insert into requestStream;

from responseStream
select *
insert into logStream;

But I am not getting a response. It displays an error message saying that the channel is inactive

Bellow is the response I received when I requested through "Postman"

image
@minudika

NPE while invoking EP with Token EP

Description:

While invoking an endpoint with Token endpoint specified in the HTTP sink, the following exception occurs.

ERROR {io.siddhi.core.stream.StreamJunction} - Error in 'SiddhiApp' after consuming events from Stream 'FooStream', null. Hence, dropping event 'Event{timestamp=1598441762269, data=[***], isExpired=false}' java.lang.NullPointerException
	at io.siddhi.extension.io.http.sink.updatetoken.HttpRequest.sendPostRequest(HttpRequest.java:127)
	at io.siddhi.extension.io.http.sink.updatetoken.HttpsClient.getPasswordGrantAccessToken(HttpsClient.java:95)
	at io.siddhi.extension.io.http.sink.HttpSink.getAccessToken(HttpSink.java:817)
	at io.siddhi.extension.io.http.sink.HttpSink.setAccessToken(HttpSink.java:844)
	at io.siddhi.extension.io.http.sink.HttpSink.sendOauthRequest(HttpSink.java:668)
	at io.siddhi.extension.io.http.sink.HttpSink.publish(HttpSink.java:607)
	at io.siddhi.core.stream.output.sink.Sink.publish(Sink.java:182)
	at io.siddhi.extension.map.json.sinkmapper.JsonSinkMapper.mapAndSend(JsonSinkMapper.java:209)
	at io.siddhi.core.stream.output.sink.SinkMapper.mapAndSend(SinkMapper.java:180)
	at io.siddhi.core.stream.output.sink.SinkCallback.receive(SinkCallback.java:55)
	at io.siddhi.core.stream.StreamJunction.sendEvent(StreamJunction.java:199)
	at io.siddhi.core.stream.StreamJunction$Publisher.send(StreamJunction.java:474)
	at io.siddhi.core.stream.input.InputDistributor.send(InputDistributor.java:34)
	at io.siddhi.core.stream.input.InputEntryValve.send(InputEntryValve.java:45)
	at io.siddhi.core.stream.input.InputHandler.send(InputHandler.java:78)
	at org.wso2.carbon.siddhi.editor.core.internal.DebuggerEventStreamService.pushEvent(DebuggerEventStreamService.java:70)
	at org.wso2.carbon.event.simulator.core.internal.generator.SingleEventGenerator.sendEvent(SingleEventGenerator.java:85)
	at org.wso2.carbon.event.simulator.core.impl.SingleApiServiceImpl.runSingleSimulation(SingleApiServiceImpl.java:17)
	at org.wso2.carbon.event.simulator.core.api.SingleApi.runSingleSimulation(SingleApi.java:63)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.wso2.msf4j.internal.router.HttpMethodInfo.invokeResource(HttpMethodInfo.java:187)
	at org.wso2.msf4j.internal.router.HttpMethodInfo.invoke(HttpMethodInfo.java:143)
	at org.wso2.msf4j.internal.MSF4JHttpConnectorListener.dispatchMethod(MSF4JHttpConnectorListener.java:218)
	at org.wso2.msf4j.internal.MSF4JHttpConnectorListener.lambda$onMessage$0(MSF4JHttpConnectorListener.java:129)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Affected Product Version:

  • WSO2 Streaming Integrator Tooling 1.0.0
  • WSO2 Identity server 5.10.0

Steps to reproduce:

The sample Siddhi file is as follows.

@App:name("SiddhiApp")

@sink(type='http', 
    publisher.url='http://jsonplaceholder.typicode.com/posts', 
    method="POST",
    headers="'Content-Type: application/json'",
    oauth.username="***",
    oauth.password="***",
    consumer.key="***",
    consumer.secret="***",
    token.url='https://localhost:9443/oauth2/token',
    ssl.verification.disabled="true",
    @map(
        type='json'
    ))
define stream FooStream (id int);

define stream BarStream(id int);

from BarStream 
select *
insert into FooStream;

Misleading error when backend is not reachable.

Description:
In a sink of type 'http-request', if the backend specified in the 'publisher.url' parameter is not reachable, upon simulating an event, the following error is being printed even when a source of 'http-response' is specified to handle responses with 500 status code (using a regex).

ERROR {org.wso2.extension.siddhi.io.http.source.HttpResponseMessageListener} - No source of type 'http-response' for status code '500' has been defined. Hence dropping the response message.

For the same scenario, from SI 1.0.1 onwards, the correct error is being printed. Following is a sample.

ERROR {io.siddhi.core.stream.output.sink.Sink} - Error on 'test_http_status_code'. Dropping event at Sink 'http-request' at 'CreditCardStream' as its still trying to reconnect!, events dropped '{CardNumber=1111}'

Affected Product Version:
SP 4.3.0, SP 4.4.0, SI 1.0.0

Steps to reproduce:

  1. Specify an unreachable backend for 'publisher.url' in 'http-request' sink.
  2. Make sure a source of type 'http-response' is specified to handle responses with '500' status code, using a regex.
    e.g. http.status.code = '[3|4|5]00'
  3. Simulate an event.

Host header is missing when invoking an endpoint via http-request

Description:
When invoking an API via http-request sink, Host HTTP header is not getting set in the HTTP request. This causes 400 Bad Request error from the server.

This needs to be fixed to get the hostname from the URL and set into the header.

Affected Product Version:
1.0.39

Basic Authentication Is Not Working

Description:
The Basic Authentication mechanism is not working as expected, and this behaviour is observed in SI-4.X versions when tested locally.
For example, when we use the http-call sink method with Basic Auth enabled as shown below, the Base64 encoded username and password is getting malformed.

@sink(type='http-call', sink.id='D1', publisher.url='http://localhost:8281/foobar/1.0', method='POST', basic.auth.username='admin', basic.auth.password='admin',
headers='Content-Type: application/json', ...,

When we checked the wirelogs from the backend service, we observed that the Authorization header value being sent to the backend was incorrect,
"Authorization: Basic UnpooledHeapByteBuf(ridx: 0, widx: 11, cap: 11/11)[\r][\n]"
Seems like there is an issue with the Base64 encoding of the Authorization header at [1]. The encoded header is in bytes and it needs to be converted to string through byte methods which causes the incorrect value to get assigned as the Authorization header. Since the encoded username and password value are invalid, the http-call sink method fails.

[1]

this.authorizationHeader = HttpConstants.AUTHORIZATION_METHOD + Base64.encode

Steps to reproduce:

  1. Create a backend with Basic Auth enabled
  2. Create a Siddhi App which calls the above backend using an http-call sink method with Basic Auth.
  3. When invoked, the Basic Auth mechanism will fail due to the improper encoded username and password.

Affected Product Version:

  • Streaming-Integrator-4.X.X

Wrong body send to get Oauth token

Description:
The payload send to get a token here looks like following in current implementation &username=abc&password=123&grant_type=password When this is send to /token end point it will throw an exception since the payload cannot be parced due to the first charactor being '&'" -m "Fixed by changing the payload generation

Suggested Labels:
bug

Suggested Assignees:

Affected Product Version:
siddhi-io-http
2.2.0

OS, DB, other environment details and versions:
macOS

Steps to reproduce:
Call an endpoint protected with WSO2 API manager via http sink. Provide username, password, client_id, client_secreat only and let the access token be generated by http sink. This fails

Related Issues:

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.