Writing multiple iterations to a single file about json-data-generator

[AaronJessen]

Hi Andrew,

I did manage to solve this by adding all generated Events to an ArrayList (without committing them to a file/logger), and then performing the logEvent iteration outside of the while loop.

I also added a quantity parameter to each configuration step, so I could easily generate 1,000 or even a million instances of a given config step that all get added to a single file for easy importing into a NoSQL database.

// Store all events in this ArrayList for future processing by each logger type
ArrayList<String> events = new ArrayList<String>();

Iterator<WorkflowStep> it = workflow.getSteps().iterator();

    while (running && it.hasNext()) {
        WorkflowStep step = it.next();

        // custom "quantity" parameter added to each configuration step to allow for looping
        int stepQuantity = step.getQuantity();
        int stepQuantityRun = 0;

        while (stepQuantityRun < stepQuantity) {
            executeStep(step);
            stepQuantityRun++;
        }

        if (!it.hasNext() && workflow.isRepeatWorkflow()) {
            it = workflow.getSteps().iterator();

            try {
                performWorkflowSleep(workflow);
            } catch (InterruptedException ie) {
                //wake up!
                running = false;
                break;
            }
        }
    }
    
    // Log all events here, instead, by concatenating the Events ArrayList from above
    for (EventLogger l : eventLoggers) {
        l.logEvent(String.join(",", events)
                , null);
    }

Here's the executeStep method:

try {
                String event = generateEvent(wrapper);
               
                // Add the events to the Events Array
                events.add(event);

                try {
                    performEventSleep(workflow);
                } catch (InterruptedException ie) {
                    //wake up!
                    running = false;
                    break;
                }
            } catch (IOException ioe) {
                log.error("Error generating json event", ioe);
            }

I'm building a CRM system, and I'd like the mock data to have relationships built in, despite the records continuing to be selected randomly. My goal is to have an Account/Contact class that not only randomly pulls a Name from the list of companies/contacts, but when company "XYZ" is selected, an ID field of "123" would always be included as well (instead of a random alphaNumeric type). In that way, the mock data has relationships built into it and is much more useful for intricate systems, such as a CRM app.

I'm hacking around and am getting closer to a solution, but again, I'm completely new to Java, so the progress is slow. Any suggestions would be greatly appreciated.

[andrewserff]

Hey @AaronJessen did you figure this out? Just saw you closed it. I haven't had a chance to look at it yet, so let me know if you solved this.

[Nelly-Barret]

Hi Andrew,

I did manage to solve this by adding all generated Events to an ArrayList (without committing them to a file/logger), and then performing the logEvent iteration outside of the while loop.

I also added a quantity parameter to each configuration step, so I could easily generate 1,000 or even a million instances of a given config step that all get added to a single file for easy importing into a NoSQL database.

...

Hello @AaronJessen and @andrewserff ,

I'm very interested in your open source software to generate large JSON datasets to evaluate my own software. As far as I looked at it, it is very well coded (even if it lacks some comments)! Like @AaronJessen , I need to generate one file which contains all the generated objects and the quantity parameter would be very useful for me.

Unfortunately, I'm not able to use @AaronJessen pieces of code to do it on my own. Have you forked the project and made a version with your improvements? If you have, I will be very grateful if you can share it with me.

Sorry for re-opening this old issue but I need some insights.

[UPDATE] I finally managed to use the quantity parameter as an argument and managed to write the events in one file.

Best,

Nelly