Request Generator

Request Generator

0. Update Notes (10/28/2019)

10/28/2019

Changes to SmartRequestGenerator:

add saveDynamicTreeToFile method to store the dynamic tree to file
add sample usage code to example/RegularClient.java. You can also find the usage and explanation here

Bug fix:

fix the bug that changes to the configuration file does not take effect
fix generator illegal bounds error when all files are deleted

10/26/2019

Two components are added:

File propagator: basically a generator genrates only CREATE_FILE request.
Smart request generator: a generator that maintains a copy of the directoies that propagated to the file system, thus generating requests based on its knowledge of directories structure.

Configuration file changes:

request_ratio: enhancement of read_write_ratio with more predefined request types. The ratio should be declared in the order of READ, WRITE, DELETE, CREATE_FILE, RMDIR, LS, CREATE_DIR. For example: 0.25,0.25,0.25,0.25,0,0,0

Please make sure your configuration file is updated before using the updated code Things that are not important to the phase 2 project have been striked out in the following description.

You can download the data file in here

1. Test the Sample Code

1.1 Build project

Build the project by using mvn install. The artifact shall be generated under target

1.2 Run the jar

Make sure resources folder is placed at the same folder with the jar file.

#### Random request generation
The following command generate random request. You can specify the number of requests needs to be generated, otherwise the generator uses the value specified in the configurate file (resources/config.properties) NOTE: the number of requests is specified per thread.

java -jar RequestGenerator.jar -r <filename> [number of requests]

Request generation with fixed order

First, use the following command to generate a dataset.

java -jar RequestGenerator.jar -f <src> <dataset>

Then feed the generator with the dataset:

java -jar RequestGenerator.jar -s <dataset>

File propagator

The file propagator uses data file to generate CREATE_FILE requests. At the same time produces a rank file, which can be used later for request generator with zipf distribution.

java -jar RequestGenerator-1.0-SNAPSHOT.jar -p <data file> <output rank file>

Smart generator

The smart generator can take three parameters

static file: required. If only static file is given, the generator won't genrate DELETE request.
dynamic file: optional. dynamic file should be the same file that used for file propagator, and should be either the same as the static file or a portion of the static file.
rank file: optional. Rank file is the file generated by the file propagator

java -jar RequestGenerator-1.0-SNAPSHOT.jar -m <static file> [dynamic file] [rank file]

2. Use the Sample Code

Implement callback

The callback provides a generated request and the id of which thread generates it.

    RequestThread.RequestGenerateThreadCallBack callBack = new RequestThread.RequestGenerateThreadCallBack() {
        @Override
        public void onRequestGenerated(Request request, int threadId) {
            System.out.println("thread[" + threadId + "]: " +  request);
        }
    }

Declare a generator

~~There are two types of generator: ClientRequestGenerator and SequentialRequestGenerator, corresponding to what has been described on section 1.2.~~ The following code demostrates how to declare SmartRequestGenerator. You can implement your own generator by extending the abstract class RequestGenerator.

    RequestGenerator generator = new SmartRequestGenerator("staticFile", 
                                                            "dynamicFile", 
                                                            "rankFile");

Start generator service

Pass the callback and the generator to the service. Parameters numThreads, interarrivalRate and numOfRequests can be read from the configuration file

    RequestService service = new RequestService(numThreads,
            interarrivalRate,
            numOfRequests,
            generator,
            callBack);

    service.start();

Save Dynamic Tree to File

The dynamic tree file is the file that you used for FilePropagator to populate files to the file system. That means the dynamic tree file has all the data in the file system.

If you use SmartRequestGenerator with dynamic tree file, the generator keeps tracking the changes made to the file system. For example, when a DELETE or CREATE_FILE request is generated, the tree removes or adds the correspoding file. After runing for a while, the file system may be created a lot of new files or deleted a lot of exisitng files. You want to save the new tree for next time to use. Thus, you can use saveDynamicTreeToFile to do this.

    ((SmartRequestGenerator)generator).saveDynamicTreeToFile("dynamicTree.txt");

Declare and start file propagator

FSPropagator takes the same callback interface as parameter.

    FSPropagator propagator = new FSPropagator("src", "rankFile", callback);

    propagator.start();

3. The Configuration File

The configuration file contains the following parameters:

read_write_inter_arrival_rate: uses possion distribution. rate = number of requests / total time in ms. For example, 5000 requests in 18 minutes, the rate would be = 5000 / (18 * 60 * 1000) = 0.00463
request_distribution: value can be [uniform|zipf]
read_write_ratio: percentage of read and write, sum to 1
alpha: used for zipf
number_threads
num_of_requests: number of requests per thread to generate, use -1 for infinite requests
request_ratio: enhancement of read_write_ratio with more predefined request types. The ratio should be declared in the order of READ, WRITE, DELETE, CREATE_FILE, RMDIR, LS, CREATE_DIR. For example: 0.25,0.25,0.25,0.25,0,0,0

phanirajl / request-generator-cassandra Goto Github PK