This guide walks you through the process of creating a basic batch-driven solution.
You will build a service that imports data from a CSV spreadsheet, transforms it with custom code, and stores the final results in a database.
Typically, your customer or a business analyst supplies a spreadsheet. For this simple
example, you can find some made-up data in src/main/resources/sample-data.csv:
link:initial/src/main/resources/sample-data.csv[role=include]This spreadsheet contains a first name and a last name on each row, separated by a comma. This is a fairly common pattern that Spring can handle without customization.
Next, you need to write an SQL script to create a table to store the data. You can find
such a script in src/main/resources/schema-all.sql:
link:initial/src/main/resources/schema-all.sql[role=include]|
Note
|
Spring Boot runs schema-@@platform@@.sql automatically during startup. -all is the default for all platforms.
|
For all Spring applications, you should start with the Spring Initializr. The Initializr offers a fast way to pull in all the dependencies you need for an application and does a lot of the set up for you. This example needs the Spring Batch and HyperSQL Database dependencies. The following image shows the Initializr set up for this sample project:
|
Note
|
The preceding image shows the Initializr with Maven chosen as the build tool. You
can also use Gradle. It also shows values of com.example and
batch-processing as the Group and Artifact, respectively. You will use those
values throughout the rest of this sample.
|
The following listing shows the pom.xml file created when you choose Maven:
link:complete/pom.xml[role=include]
The following listing shows the build.gradle file created when you choose Gradle:
link:complete/build.gradle[role=include]
Now that you can see the format of data inputs and outputs, you can write code to
represent a row of data, as the following example (from
src/main/java/com/example/batchprocessing/Person.java) shows:
link:complete/src/main/java/com/example/batchprocessing/Person.java[role=include]You can instantiate the Person class either with first and last name through a
constructor or by setting the properties.
A common paradigm in batch processing is to ingest data, transform it, and then pipe it
out somewhere else. Here, you need to write a simple transformer that converts the names
to uppercase. The following listing (from
src/main/java/com/example/batchprocessing/PersonItemProcessor.java) shows how to do
so:
link:complete/src/main/java/com/example/batchprocessing/PersonItemProcessor.java[role=include]PersonItemProcessor implements Spring Batch’s ItemProcessor interface. This makes it
easy to wire the code into a batch job that you will define later in this guide. According
to the interface, you receive an incoming Person object, after which you transform it to
an upper-cased Person.
|
Note
|
The input and output types need not be the same. In fact, after one source of data is read, sometimes the application’s data flow needs a different data type. |
Now you need to put together the actual batch job. Spring Batch provides many utility classes that reduce the need to write custom code. Instead, you can focus on the business logic.
To configure your job, you must first create a Spring @Configuration class like the following example in
src/main/java/com/exampe/batchprocessing/BatchConfiguration.java:
link:complete/src/main/java/com/example/batchprocessing/BatchConfiguration.java[role=include]
...
}For starters, the @EnableBatchProcessing annotation adds many critical beans that
support jobs and save you a lot of leg work. This example uses a memory-based database
(provided by @EnableBatchProcessing), meaning that, when it is done, the data is gone. It also autowires a couple factories needed further below.
Now add the following beans to your BatchConfiguration class to define a reader, a processor, and a writer:
link:complete/src/main/java/com/example/batchprocessing/BatchConfiguration.java[role=include]The first chunk of code defines the input, processor, and output.
-
reader()creates anItemReader. It looks for a file calledsample-data.csvand parses each line item with enough information to turn it into aPerson. -
processor()creates an instance of thePersonItemProcessorthat you defined earlier, meant to converth the data to upper case. -
write(DataSource)creates anItemWriter. This one is aimed at a JDBC destination and automatically gets a copy of the dataSource created by@EnableBatchProcessing. It includes the SQL statement needed to insert a singlePerson, driven by Java bean properties.
The last chunk (from src/main/java/com/example/batchprocessing/BatchConfiguration.java)
shows the actual job configuration:
link:complete/src/main/java/com/example/batchprocessing/BatchConfiguration.java[role=include]The first method defines the job, and the second one defines a single step. Jobs are built from steps, where each step can involve a reader, a processor, and a writer.
In this job definition, you need an incrementer, because jobs use a database to maintain execution state. You then list each step, (though this job has only one step). The job ends, and the Java API produces a perfectly configured job.
In the step definition, you define how much data to write at a time. In this case, it writes up to ten records at a time. Next, you configure the reader, processor, and writer by using the bits injected earlier.
|
Note
|
chunk() is prefixed <Person,Person> because it is a generic method. This
represents the input and output types of each “chunk” of processing and lines up with
ItemReader<Person> and ItemWriter<Person>.
|
The last bit of batch configuration is a way to get notified when the job completes. The
following example (from
src/main/java/com/example/batchprocessing/JobCompletionNotificationListener.java) shows
such a class:
link:/complete/src/main/java/com/example/batchprocessing/JobCompletionNotificationListener.java[role=include]The JobCompletionNotificationListener listens for when a job is BatchStatus.COMPLETED
and then uses JdbcTemplate to inspect the results.
Although batch processing can be embedded in web apps and WAR files, the simpler approach demonstrated below creates a standalone application. You package everything in a single, executable JAR file, driven by a good old Java main() method.
The Spring Initializr created an application class for you. For this simple example, it
works without further modification. The folowing listing (from
src/main/java/com/example/batchprocessing/BatchProcessingApplication.java) shows the
application class:
link:complete/src/main/java/com/example/batchprocessing/BatchProcessingApplication.java[role=include]For demonstration purposes, there is code to create a JdbcTemplate, query the database,
and print out the names of people the batch job inserts.
The job prints out a line for each person that gets transformed. After the job runs, you can also see the output from querying the database. It should resemble the following output:
Converting (firstName: Jill, lastName: Doe) into (firstName: JILL, lastName: DOE)
Converting (firstName: Joe, lastName: Doe) into (firstName: JOE, lastName: DOE)
Converting (firstName: Justin, lastName: Doe) into (firstName: JUSTIN, lastName: DOE)
Converting (firstName: Jane, lastName: Doe) into (firstName: JANE, lastName: DOE)
Converting (firstName: John, lastName: Doe) into (firstName: JOHN, lastName: DOE)
Found <firstName: JILL, lastName: DOE> in the database.
Found <firstName: JOE, lastName: DOE> in the database.
Found <firstName: JUSTIN, lastName: DOE> in the database.
Found <firstName: JANE, lastName: DOE> in the database.
Found <firstName: JOHN, lastName: DOE> in the database.Congratulations! You built a batch job that ingested data from a spreadsheet, processed it, and wrote it to a database.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent