everwatchsolutions / json-data-generator Goto Github PK

Sometimes, we have a need to generate values that are based on some other source of data. For example, we might have an existing source of data that provides information about airports around the world. We might want to use the actual names, locations, or callsigns of those airports when generating events. So we would like to have a new Data Source that you could provide to the generator that could be pulled from when each event is generated. Then you should be able to reference the values within the items pulled from the data source in your generated events.

Hi there, 😸

I noticed that this project is not published in maven repositories. Is there a reason you have not published this project so far?

Best, Nick.

I am getting this error. Couldn't find much help on this.

Hi everyone,

I would like to create random GPS positions that in the end I plan to send to a Kafka topic. In a first step I have only use the "logger" producer which worked perfectly, but I am failing to fill the Kafka topic with the same data. I have created a new Kafka topic ("RandomGPS") for this aim and it remains empty.

Here are the producers settings

"producers": [{
"type": "kafka",
"broker.server": "172.17.0.3",
"broker.port": 9092,
"topic": "RandomGPS",
"flatten": true,
"sync": true
},{
"type":"logger"
}]

Is there something I am missing ? Thanks for your help.

2018-08-01 02:11:07,951 ERROR n.a.d.j.g.SimulationRunner [main] Error reading config: test
com.fasterxml.jackson.databind.JsonMappingException: Can not instantiate value of type [map type; class java.util.LinkedHashMap, [simple type, class java.lang.String] -> [simple type, class java.lang.Object]] from String value ('repeat(3)'); no single-String constructor/factory method
at [Source: java.io.BufferedInputStream@5b8dfcc1; line: 9, column: 23] (through reference chain: net.acesinc.data.json.generator.workflow.Workflow["steps"]->java.util.ArrayList[0]->net.acesinc.data.json.generator.workflow.WorkflowStep["config"]->java.util.ArrayList[0])
at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:148) ~[jackson-databind-2.5.2.jar:2.5.2]
at com.fasterxml.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:843) ~[jackson-datab

myWorkflow.json:

GNU nano 2.2.6 File: conf/myWorkflow.json

{
"eventFrequency": 4000,
"varyEventFrequency": true,
"repeatWorkflow": true,
"timeBetweenRepeat": 60000,
"varyRepeatFrequency": true,

    "steps":[{
        "config":["repeat(3)",      {
                "timestamp": "nowTimestamp()",
                "system": "BADGE",
                "actor": "bob",
                "action": "ENTER",
                "objects": ["Building 1"],
                "location": "45.5,44.3",
                "message": "Entered Building 1"
            }

],
"duration": 0

}
]
}

Can i get properties from previous step? this.prop , this.step[0].prop don't work..

Hi there!
Thanks a lot for the great tool!

I've changed exampleSimConfig.json to check JSON generating with multiple threads:

{
    "workflows": [{
            "workflowName": "test",
            "workflowFilename": "exampleWorkflow.json"
        }, {
            "workflowName": "test",
            "workflowFilename": "exampleWorkflow.json"
        },
        {
            "workflowName": "test",
            "workflowFilename": "exampleWorkflow.json"
        }
    ],
    "producers": [{
            "type": "logger"
        },
        {
            "type": "file",
            "output.directory": "test_data/test",
            "file.prefix": "test_",
            "file.extension": ".json"
        }
    ]
}

I've got many errors during simulation:

DEBUG n.a.d.j.g.RandomJsonGenerator [Thread-2] Error creating type [ random("PRINT","OPEN","COPY") ]. Prop [ action ] being ignored in output.
org.apache.commons.math3.exception.NumberIsTooLargeException: lower bound (0) must be strictly less than upper bound (-1)
	at org.apache.commons.math3.distribution.UniformIntegerDistribution.<init>(UniformIntegerDistribution.java:78) ~[commons-math3-3.5.jar:3.5]
	at org.apache.commons.math3.random.RandomDataGenerator.nextInt(RandomDataGenerator.java:198) ~[commons-math3-3.5.jar:3.5]
	at net.acesinc.data.json.generator.types.RandomType.getNextRandomValue(RandomType.java:48) ~[json-data-generator-1.2.2-SNAPSHOT.jar:1.2.2-SNAPSHOT]
	at net.acesinc.data.json.generator.RandomJsonGenerator.processProperties(RandomJsonGenerator.java:102) [json-data-generator-1.2.2-SNAPSHOT.jar:1.2.2-SNAPSHOT]
	at net.acesinc.data.json.generator.RandomJsonGenerator.processProperties(RandomJsonGenerator.java:133) [json-data-generator-1.2.2-SNAPSHOT.jar:1.2.2-SNAPSHOT]
	at net.acesinc.data.json.generator.RandomJsonGenerator.generateJson(RandomJsonGenerator.java:50) [json-data-generator-1.2.2-SNAPSHOT.jar:1.2.2-SNAPSHOT]
	at net.acesinc.data.json.generator.EventGenerator.generateEvent(EventGenerator.java:243) [json-data-generator-1.2.2-SNAPSHOT.jar:1.2.2-SNAPSHOT]
	at net.acesinc.data.json.generator.EventGenerator.executeStep(EventGenerator.java:182) [json-data-generator-1.2.2-SNAPSHOT.jar:1.2.2-SNAPSHOT]
	at net.acesinc.data.json.generator.EventGenerator.runSequential(EventGenerator.java:71) [json-data-generator-1.2.2-SNAPSHOT.jar:1.2.2-SNAPSHOT]
	at net.acesinc.data.json.generator.EventGenerator.runWorkflow(EventGenerator.java:53) [json-data-generator-1.2.2-SNAPSHOT.jar:1.2.2-SNAPSHOT]
	at net.acesinc.data.json.generator.EventGenerator.run(EventGenerator.java:257) [json-data-generator-1.2.2-SNAPSHOT.jar:1.2.2-SNAPSHOT]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]

I would appreciate any suggestion to solve this issue.

I am trying to produce message and publish it to secure end point. I started it with ssl parameters but somehow i don't see any data being published. During the starup it gave following message

Adding HTTP Post Logger with properties: {type=http-post, url=https://server-msrt-ft2.xxx.com:3905/events/com.message-router-EVENT-INPUT}

post this message I don't see anything in the log on the command line that what's happening. I only know that my consumer is not seeing any message. I added extra producer to see what it is producing and found it is creating json files but not sure why http-post is not working. Can someone suggest how to debug this ?
I have attached log file for reference. Following is my Simconfig file:

{
"workflows": [{
"workflowName": "newsyslog",
"workflowFilename": "ecnewsyslogWorkflow.json"
}, {
"workflowName": "newsnmp",
"workflowFilename": "ecnewsnmpWorkflow.json"
}
],
"producers": [ {
"type": "http-post",
"url": "https://server-msrt-ft2.xxx.com:3905/events/com.message-router-EVENT-INPUT"
}
]
}

I want to use http post which uses basic auth.
How do I add basic auth ?

Steps:

1. Create field random with subfields that contain closing parenthesis, e.g.
   "function_name": "random('function1()', 'function2()')"

Expected:
The field should have been filled with one of the provided values, e.g. function1() or function2()

Actual:
Parsing argument breaks after first closing parenthesis disregarding any quoting, .e.g. 'function1(

I keep seeing dates being generated outside of the max date range.

Hi,

the last released version is from May 25, 2016
Since then, a lot new features were added (appreciate that).
Don't you plan to make periodic releases any more, because they are quite nice to use in e.g. the test automation by just downloading it instead of cloning+building.

When trying to build with "mvn clean package" i get errors as follows

Hi, I have a a workflow containing one step with three config elements. It should generate events which are linked via id. Is there a way to use a value from message 1 from config in same config for message 2?

regards

I will like to tell the config file that i want say 1000 records and after that the execution should stop. Also is it possible to write all these 1000 records in a single file

Discussed in #89

Hi, I need to generate json messages like the ones produced by CollectD ( a metrics collector among many, but I should use that...).
Messages are like these two:

• [{"values":[0,0],"dstypes":["derive","derive"],"dsnames":["user","syst"],"time":1519314305.896,"interval":10.000,"host":"host1","plugin":"processes","plugin_instance":"pengine","type":"ps_cputime","type_instance":""}]
• [{"values":[696],"dstypes":["gauge"],"dsnames":["value"],"time":1519314305.896,"interval":10.000,"host":"host2","plugin":"processes","plugin_instance":"stonithd","type":"ps_stacksize","type_instance":""}]

After lot of configurations I wasn't able to produce a message with {"values":[0,0]......
Seems that are not supported array of values of primitives types.
Have you any suggestion ?
Thanks
R.

how to link one work flow to one producer but not all producers like it's happening currently .

First off, thanks so much for creating the json-data-generator. It's exactly what I needed at this point in my project, and it will be invaluable going forward. Thanks for your hard work!

Can you recommend the best way (i.e. the correct place in the code) to modify the workflows so as to write all events/iterations to a single file, as opposed to a single file for each workflow iteration? I'm not fluent in Java, so I'm not certain where/how to perform this change.

Thanks.

1. Clone repository: git clone https://github.com/everwatchsolutions/json-data-generator.git (1.4.1)
2. Compile using mvn clean package from command line (success), maven 3.3.9, Oracle Java 1.8.0_152, Linux Mint 18.2
3. Copy bin.tar and .jar files to separate directory
4. Modify classes/defaultSimConfig.json by removing nats producer section (only default logger is left)
5. Copy classes/defaultSimConfig.json and classes/normalUser1Workflow.json to above directory
6. Enter above directory and start program using command java -jar json-data-generator-1.4.1-SNAPSHOT.jar defaultSimConfig.json

Expected:
Demo project is run

Actual:
achamier@deathstar ~/git/json-data-generator/run $ java -jar json-data-generator-1.4.1-SNAPSHOT.jar defaultSimConfig.json
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/eclipse/paho/client/mqttv3/MqttException
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.eclipse.paho.client.mqttv3.MqttException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more

When generator is run from target directory it runs without an issue.
I googled some solution, there are several threads which ends in "oh it looks like you forgot about dependencies" but withour real solution, and as far as I can see mqttv3 is declared both in pom.xml and json-data-generator.iml. Any help?

Hi Folks,

I'd like to use json-data-generator to generate events that mimics users interacting with a content application. Due to the nature of the application however there are a few requirements to be met:

1-content has an ordered structure of 1-cover x N items that impose a given navigation condition (ie, in order to see the 2nd item of a given content you must see the cover and 1st item).
2-I need to simulate multiple users navigating at the same time (or in a given interval)

I noticed on the readme that there is some kind of cursor or references, but it was not clear how can I use them.

Can you give some directions on how to build those events?

Thanks in advance

Upgrade for security fixes

If i have an array with integer values, the generator is not keeping those values on output and and returning a null array.

e:g "doubleValues" : [10] is coming as "doubleValues" : [] on output

If i provide the sim config as specified in documentation, i get the error below.

When i looked into the source code looks like the sim config is read as stream so i used the command below and i was able to see the generator working.

I am using windows to run this generator. Let me know if this is a valid issue or not.

An HTTP POST logger would be another helpful logger.

Thanks for great tool. I'm trying to generate IP address, but the following don't work:

stringMerge(.,integer(100, 180),integer(50, 80),integer(100, 160),integer(180, 240))
integer(100, 180).integer(50, 80).integer(100, 160).integer(180, 240)

Is this possible without some big hacks?

Upgrade for security fixes

There have been multiple requests for the ability to reference values of previous events in new events, however we currently only maintain state within a single event. I'm creating this ticket to track interest in this feature. Unfortunately this isn't an easy feature to implement as we would have to track each event generated which would mean we would need a cache of some kind. There would also need to be a way to reference other events that I'm not sure how people would know which event they want to pull values from.

If you have any comments about this feature, please add them below.

Hi, I have just recently started using the json-data-generator, and it is a useful tool.
I noticed that the main thread never terminates, even after the simulation is finished. I might be missing something. I am wondering if this is on purpose or is it a bug?

From looking at the code, it seems that the main thread of the application is left waiting at line 162 of JsonDataGenerator forever (master branch), because the status of the "running" boolean within SimulationRunner, which is accessed using isRunning() is never updated once the simulation is finished.

The following is the while loop which seems to always last forever:

        while (gen.isRunning()) {
            try {
                Thread.sleep(1000);
            } catch (InterruptedException ex) {
                //wakie wakie!
            }
        }

There are "running" booleans in three classes: JsonDataGenerator, SimulationRunner and EventGenerator. It looks like the problem is in the SimulationRunner. It creates a group of threads, and starts them running. But when you call the isRunning() method on it, it should be checking the running status of all the threads that it started. If it did that, its isRunning() method would return false when all the threads had ended, and so the JsonDataGenerator would also know when the simulation had ended, and the main thread would not wait at line 162 forever.

Within EventGenerator, the running boolean in there is set to false when a particular workflow is finished. But at the level of SimulationRunner, it seems to never update its boolean.

The consequence of this is that when a user runs the jar from the command line they have to do a CTRL+C to kill it.

Please let me know if I am missing something. Thanks

Hi!

Running your jackieChan example sedngin it to kafka it returns next exception:

2016-12-06 18:42:33,202 WARN n.a.d.j.g.RandomJsonGenerator [Thread-2] Error creating type [ double(1.0,10.0) ]. Prop [ strength ] being ignored in output. Reason: For input string: "4,3396" 2016-12-06 18:42:33,202 DEBUG n.a.d.j.g.RandomJsonGenerator [Thread-2] Error creating type [ double(1.0,10.0) ]. Prop [ strength ] being ignored in output. java.lang.NumberFormatException: For input string: "4,3396" at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043) ~[?:1.8.0_111] at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110) ~[?:1.8.0_111] at java.lang.Double.parseDouble(Double.java:538) ~[?:1.8.0_111]

and the strenght value is ignored in output:

2016-12-06 18:42:33,219 DEBUG n.a.d.j.g.l.KafkaLogger [Thread-2] Sending event to Kafka: [ {"timestamp":"2016-12-06T18:42:33.163Z","style":"KUNG_FU","action":"JUMP","weapon":"ROPE","target":"HEAD"} ]

Thanks!

Hi,

I am trying to using the framework to generate the json output using the sample schema as below.

{
"$schema": "http://json-schema.org/draft-04/schema",
"title": "Basic Info",
"type": "object",
"properties": {
"firstName": {
"type": "string"
},
"age": {
"type": "number"
}
}
}

I am not able to convert or parse the above schema into json output. The code doesn't recognize it as a schema rather it just considers it as another json documents and parses the same. Meaning, it just converts the above schema as such into a json output. I could see from the jsondatagenerator.java classes, that we feed the sample json files like config-array-test.json and convert them. However, the schema i am trying to parse or convert doesn't work. Requesting your inputs on the same.

Below is the output:

{"$schema":"http://json-schema.org/draft-04/schema",
"title":"BasicInfo",
"type":"object",
"properties.firstName.type":"string",
"properties.age.type":"number"}

Hi there! thanks for this excellent work. I am very new to this kind of json response. I was using http-post to generate data. but I got 404 error.
floodConfig.json
{ "workflows": [{ "workflowName": "flood", "workflowFilename": "floodWorkflow.json" }], "producers": [{ "type": "http-post", "url": "http://localhost:80/ingest" }] }
floodWorkflow.json
{ "eventFrequency": 2000, "varyEventFrequency": true, "repeatWorkflow": true, "timeBetweenRepeat": 2000, "varyRepeatFrequency": true, "steps": [{ "config": [{ "id": 1, "w_lvl": "double(10.0,100.0)" }, { "id": 2, "w_lvl": "double(10.0,100.0)" }, { "id": 3, "w_lvl": "double(10.0,100.0)" }, { "id": 4, "w_lvl": "double(10.0,100.0)" }] }] }

After I run this configuration, I copy and paste the URL which is http://localhost:80/ingest and get 404 error. Please teach me senpai

Is there anyway to send a GET request to this tool to respond with test data based on the configuration?

It would be nice to be able to take two generated values and merge them together into a single field.

The json-data-generator should be able to write json documents to separate files. This would generate 1 json event per file.

Hello!
Congratulations on your code, it is helping me a lot!

I'm trying to connect the kafka and the data-stream, but it has not worked.
The zookeeper and Kafka are ok, I have a topic, consumer and producer.
Always get this error displayed below.
Can you help me please?

root@vinicius-vm-3:/home/vinicius/json-data-generator/files# java -jar json-data-generator-1.2.2-SNAPSHOT.jar producer.json 2016-10-06 15:27:01,636 INFO n.a.d.j.g.JsonDataGenerator [main] Overriding Simulation Config file from command line to use [ producer.json ] 2016-10-06 15:27:01,643 DEBUG n.a.d.j.g.JsonDataGenerator [main] Creating Simulation Runner using Simulation Config [ producer.json ] 2016-10-06 15:27:01,965 INFO n.a.d.j.g.JsonDataGenerator [main] Adding Kafka Producer with properties: {type=kafka, broker.server=127.0.0.1, broker.port=9092, topic=topico-json-data, sync=false} SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Exception in thread "main" java.lang.NullPointerException at net.acesinc.data.json.generator.log.KafkaLogger.<init>(KafkaLogger.java:49) at net.acesinc.data.json.generator.JsonDataGenerator.<init>(JsonDataGenerator.java:51) at net.acesinc.data.json.generator.JsonDataGenerator.main(JsonDataGenerator.java:107) root@vinicius-vm-3:/home/vinicius/json-data-generator/files#

Thanks
Sorry my english

Hi,

I have successfully used your tool to generate fake log entries that simulate logs generate by some web server. This includes header/response data URL, duration, exceptions, etc.

I would like to share my workflow/config because I believe that others might be interested to use it or enhance it.
I have submitted the files in pull request #38

Regards,
Jovan

It would be helpful to run workflow steps in different orders such as random or just have the generator pick one of the steps at random and continue on.

Hi

I'm running the generator, with this configuration:

"eventFrequency": 1, "steps": [{ "config":[{ "user_id": "uuid()", "page_id": "uuid()", "ad_id": "random('cmp1', 'cmp2', 'cmp3', 'cmp4', 'cmp5', 'cmp6', 'cmp7', 'cmp8')", "ad_type": "random('banner', 'modal', 'sponsored-search', 'mail', 'mobile')", "event_type": "random('view', 'click', 'purchase')", "event_time": "now()", "ip_address": "127.0.0.1" }], "duration": 100000 }] }

The events are sent to a Kafka topic.

At the end of the execution, it says that it generated 0.0 events/s. There is a way to fix that ?