siddhi-io / siddhi-execution-string

Extension that provides basic string handling capabilities

Siddhi Execution String

The siddhi-execution-string extension is a Siddhi extension that provides basic string handling capabilities such as concat, length, replace all, etc.

For information on Siddhi and it's features refer Siddhi Documentation.

Download

• Versions 5.x and above with group id io.siddhi.extension.* from here.
• Versions 4.x and lower with group id org.wso2.extension.siddhi.* from here.

Latest API Docs

Latest API Docs is 5.0.13.

Features

• groupConcat (Aggregate Function)
  

  This function aggregates the received events by concatenating the keys in those events using a separator, e.g.,a comma (,) or a hyphen (-), and returns the concatenated key string.

• charAt (Function)
  

  This function returns the 'char' value that is present at the given index position. of the input string.

• charFrequency (Function)
  

  Gives the frequency of a char in input string.

• coalesce (Function)
  

  This returns the first input parameter value of the given argument, that is not null.

• concat (Function)
  

  This function returns a string value that is obtained as a result of concatenating two or more input string values.

• contains (Function)
  

  This function returns true if theinput.string contains the specified sequence of char values in the search.string.

• equalsIgnoreCase (Function)
  

  This returns a boolean value by comparing two strings lexicographically without considering the letter case.

• fillTemplate (Function)
  

  fillTemplate(string, map) will replace all the keys in the string using values in the map. fillTemplate(string, r1, r2 ..) replace all the entries {{1}}, {{2}}, {{3}} with r1 , r2, r3.

• hex (Function)
  

  This function returns a hexadecimal string by converting each byte of each character in the input string to two hexadecimal digits.

• length (Function)
  

  Returns the length of the input string.

• locate (Function)
  

  This function returns the position of one string within another. Optionally, the user can give the starting index to be search from.

• lower (Function)
  

  Converts the capital letters in the input string to the equivalent simple letters.

• regexp (Function)
  

  Returns a boolean value based on the matchability of the input string and the given regular expression.

• repeat (Function)
  

  Repeats the input string for a specified number of times.

• replaceAll (Function)
  

  Finds all the substrings of the input string that matches with the given expression, and replaces them with the given replacement string.

• replaceFirst (Function)
  

  Finds the first substring of the input string that matches with the given regular expression, and replaces itwith the given replacement string.

• reverse (Function)
  

  Returns the input string in the reverse order character-wise and string-wise.

• split (Function)
  

  Splits the input.string into substrings using the value parsed in the split.string and returns the substring at the position specified in the group.number.

• strcmp (Function)
  

  Compares two strings lexicographically and returns an integer value. If both strings are equal, 0 is returned. If the first string is lexicographically greater than the second string, a positive value is returned. If the first string is lexicographically greater than the second string, a negative value is returned.

• substr (Function)
  

  Returns a substring of the input string by considering a subset or all of the following factors: starting index, length, regular expression, and regex group number.

• trim (Function)
  

  Returns a copy of the input string without the leading and trailing whitespace (if any).

• unhex (Function)
  

  Returns a string by converting the hexadecimal characters in the input string.

• upper (Function)
  

  Converts the simple letters in the input string to the equivalent capital/block letters.

• tokenize (Stream Processor)
  

  This function splits the input string into tokens using a given regular expression and returns the split tokens.

Dependencies

There are no other dependencies needed for this extension.

Installation

For installing this extension on various siddhi execution environments refer Siddhi documentation section on adding extensions.

Support and Contribution

• We encourage users to ask questions and get support via StackOverflow, make sure to add the siddhi tag to the issue for better response.

• If you find any issues related to the extension please report them on the issue tracker.

• For production support and other contribution related information refer Siddhi Community documentation.

siddhi-execution-string's Issues

[Doc] Regex used for substr function is optional, but documentation it is marked as mandatory

Description:
$subject. Also, Description of the substr() function is unclear

Suggested Labels:
Type/Documentation

Suggested Assignees:
@RukshiW @YvonneW

Update mkdocs

Content of auto generated documentation site should be updated and refined.

ERROR {io.siddhi.core.stream.output.sink.Sink} - Error on 'APIM_ALERT_EMAIL_NOTIFICATION'

While running WSO2 gateway sending the mail below error is coming

ERROR {io.siddhi.core.stream.output.sink.Sink} - Error on 'APIM_ALERT_EMAIL_NOTIFICATION'. Error is encountered while connecting to the smtp server.Error is encountered while creating the connection using the session.Couldn't connect to host, port: smtp.gmail.com, 465; timeout -1, error while connecting at Sink 'email' at 'EmailNotificationStream', will retry in '30 sec'. io.siddhi.core.exception.ConnectionUnavailableException: Error is encountered while connecting to the smtp server.

Improve logging in GetStringJSONFunctionExtension

Description:
Improve logging in GetStringJSONFunctionExtension

INFO {io.siddhi.extension.execution.json.function.GetStringJSONFunctionExtension} - CollectorApp:query_2: Cannot find json element for the path 'field_462501'. Hence returning the default value 'null' (Encoded)

Adding new extensions for map and evaluate strings

Description:

Problem

Currently, Siddhi execution string does not support string manipulation for the following scenarios.

1. Let say I have a stringS=" {{prize}} > 100 " and the value of the prize coming from a different stream. Assume that, now the prize is 300. Then I need to get the string S'= "300 > 100"
2. The next scenario is get the string S' = "300 > 100" and evaluate it. The evaluation of that mathematical expression should return true.

Suggestion

1. Implement an extension called str: map(STRING, STRING...) and this will return a STRING.
2. Implement an extension called str: eval(STRING, STRING...) and this will return an Object.

Suggested Labels:
Feature improvement

Runtime exception testcases are not asserted

Description:
$subject
https://github.com/siddhi-io/siddhi-execution-string/blob/master/component/src/test/java/io/siddhi/extension/execution/string/CharAtFunctionExtensionTestCase.java#L169

This can be easily achieved through Siddhi Error Streams

str:concat() should support not string parameters.

Description:
Currently, only string params can be passed to str:concat(), but if we need to contact a string and int, then we have to first convert the int to string and then pass that to str:concat().

This can be improved if we can support str:concat() to accept other formats other than string.

Add jacoco plugin for test coverage report generation

Add a function to aggregate a string attribute of events in a stream and concatenate to a single string

Description:
Add a function that aggregate a specified string attribute of events in a stream and concatenate to them into a single string. This function should behave similar to MySQL GROUP_CONCAT function.

SELECT student_name,
GROUP_CONCAT(test_score)
FROM student
GROUP BY student_name;

Affected Product Version:
Siddhi v4.x.x

Cast string error in function groupConcat

Description:
The first parameter type in function groupConcat is String. So if the column I need to use in function groupConcat is not string, I need to cast the column type to string. But there still throw an exception "Integer cannot cast to string" even I have already cast the column type to string.

Affected Product Version:

4.0.24

Steps to reproduce:

1. The code:
   `@App:name("GroupByApp")
   @app:description("Description of the plan")

-- Please refer to https://docs.wso2.com/display/SP400/Quick+Start+Guide on getting started with SP editor.

define stream inputstream (id string,weight int);

@sink(type='log', prefix='LOGGER')
define stream OutputStream(id string,weight string);

@info(name='GroupbyQuery')
from inputstream#window.timeBatch(10 second)
select id,str:groupConcat(cast(weight,'string'),',',true) as weight
group by id
insert into OutputStream;`

2. The exeption:
   [2018-12-28_08-58-27_187] ERROR {org.wso2.siddhi.core.query.processor.stream.AbstractStreamProcessor} - Error on 'GroupByApp'. java.lang.Integer cannot be cast to java.lang.String Dropping event chunk EventChunk{first=StreamEvent{ timestamp=1545958697185, beforeWindowData=null, onAfterWindowData=[1], outputData=[1, null], type=CURRENT, next=null}}, error in processing org.wso2.siddhi.core.query.processor.stream.window.TimeBatchWindowProcessor. (Encoded) java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.String at org.wso2.extension.siddhi.execution.string.GroupConcatFunctionExtension.processAdd(GroupConcatFunctionExtension.java:167) at org.wso2.siddhi.core.query.selector.attribute.aggregator.AttributeAggregator.process(AttributeAggregator.java:82) at org.wso2.siddhi.core.query.selector.attribute.processor.executor.GroupByAggregationAttributeExecutor.execute(GroupByAggregationAttributeExecutor.java:92) at org.wso2.siddhi.core.query.selector.attribute.processor.AttributeProcessor.process(AttributeProcessor.java:41) at org.wso2.siddhi.core.query.selector.QuerySelector.processInBatchGroupBy(QuerySelector.java:293) at org.wso2.siddhi.core.query.selector.QuerySelector.process(QuerySelector.java:84) at org.wso2.siddhi.core.query.processor.stream.window.TimeBatchWindowProcessor.process(TimeBatchWindowProcessor.java:254) at org.wso2.siddhi.core.query.processor.stream.window.WindowProcessor.processEventChunk(WindowProcessor.java:66) at org.wso2.siddhi.core.query.processor.stream.AbstractStreamProcessor.process(AbstractStreamProcessor.java:125) at org.wso2.siddhi.core.query.input.stream.single.EntryValveProcessor.process(EntryValveProcessor.java:50) at org.wso2.siddhi.core.util.Scheduler.sendTimerEvents(Scheduler.java:197) at org.wso2.siddhi.core.util.Scheduler$EventCaller.run(Scheduler.java:249) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

Siddhi String Tokenizer return parameter in not clear in the documentation

Description:
I am trying to use the string tokenize method in Siddhi in my siddhi app, I tried to find the details in the documentation, but it is not clear. Following is a screenshot from the documentation.

What is the return value of the tokenizer ? When we tokenize a string, what is the attribute value of the tokenized string ?

Suggested Labels:
siddhi, siddhi-string-extension, string, tokenize, tokenizer

Suggested Assignees:

Affected Product Version:

OS, DB, other environment details and versions:

Steps to reproduce:

Related Issues:

Unable to get only distinct values from the tokenize method

Description:
In str:tokenize extension, all the splitted tokens are returned. There is no way to get only distinct tokens. Therefore a parameter needs to be introduced to get only distinct tokens.

Affected Product Version:
4.0.14

Improvements to documentation related to the default datasources of WSO2 SP

Description:
The following changes are required.

1. Add information on how to use/update these default data sources in a production environment (e.g., the Carbon_DB can be kept as the H2, The PERMISSION_DB can be shared between both worker and dashboard. etc.).
2. Change default DB of IdP Client to WSO2_OAUTH_APP_DB ? Currently, it is given as DB_AUTH_DB. (Doc space - 4.0.0 - 4.x.x).
3. Add information about the data sources configured by default for the solutions, which are as follows.

• 1. HTTP analytics
• 2. Twitter Analytics
• 3. Message Tracing
• 4. APIM Analytics
• 5. IS Analytics
• 6. EI Analytics
• 7. Geolocation
• 8. WSO2AM_MGW_ANALYTICS_DB

Please find the context in the config file[1]

[1] https://github.com/wso2/product-sp/blob/master/modules/distribution/carbon-home/conf/worker/deployment.yaml#L378