-
Spring Batch - Read from multiple Flat Files:

 In this example, we will show you how to read multiple flat files using MultiResourceItemReader and print data onto the console. You can write data into the XML, SQL DB or NOSQL DB as per your need. I am keeping the things simple here and only showing the purpose of multi resource file reads.

customer1.csv

 1,John,Doe,10-10-1952 10:10:10
2,Amy,Eugene,05-07-1985 17:10:00
3,Laverne,Mann,11-12-1988 10:10:10
4,Janice,Preston,19-02-1960 10:10:10
5,Pauline,Rios,29-08-1977 10:10:10
6,Perry,Burnside,10-03-1981 10:10:10
7,Todd,Kinsey,14-12-1998 10:10:10
8,Jacqueline,Hyde,20-03-1983 10:10:10
9,Rico,Hale,10-10-2000 10:10:10

10,Samuel,Lamm,11-11-1999 10:10:10

 customer2.csv

 11,Robert,Coster,10-10-1972 10:10:10
12,Tamara,Soler,02-01-1978 10:10:10
13,Justin,Kramer,19-11-1951 10:10:10
14,Andrea,Law,14-10-1959 10:10:10
15,Laura,Porter,12-12-2010 10:10:10
16,Michael,Cantu,11-04-1999 10:10:10
17,Andrew,Thomas,04-05-1967 10:10:10
18,Jose,Hannah,16-09-1950 10:10:10
19,Valerie,Hilbert,13-06-1966 10:10:10

20,Patrick,Durham,12-10-1978 10:10:10

 Customer.java

 @AllArgsConstructor
@NoArgsConstructor
@Builder
@Data
public class Customer implements ResourceAware{
private Long id;
private String firstName;
private String lastName;
private LocalDateTime birthdate;
private Resource resource;

}

 CustomerFieldSetMapper.java

 public class CustomerFieldSetMapper implements FieldSetMapper<Customer> {
private static final DateTimeFormatter DT_FORMAT = DateTimeFormatter.ofPattern("dd-MM-yyyy HH:mm:ss");

@Override
public Customer mapFieldSet(FieldSet fieldSet) throws BindException {
//@// @formatter:off
return Customer.builder()
.id(fieldSet.readLong("id"))
.firstName(fieldSet.readString("firstName"))
.lastName(fieldSet.readString("lastName"))
.birthdate(LocalDateTime.parse(fieldSet.readString("birthdate"), DT_FORMAT))
.build(); 
// @formatter:on
}

}

 JobConfig.java

 @Configuration
public class JobConfig {
@Autowired
private JobBuilderFactory jobBuilderFactory;

 @Autowired
private StepBuilderFactory stepBuilderFactory;

 @Value("classpath*:/data/customer*.csv")
private Resource[] inputFiles;

 @Bean
public MultiResourceItemReader<Customer> multiResourceItemreader() {
MultiResourceItemReader<Customer> reader = new MultiResourceItemReader<>();
reader.setDelegate(customerItemReader());
reader.setResources(inputFiles);

 return reader;
}

 @Bean
public FlatFileItemReader<Customer> customerItemReader() {
DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();
tokenizer.setNames(new String[] { "id", "firstName", "lastName", "birthdate" });

 DefaultLineMapper<Customer> customerLineMapper = new DefaultLineMapper<>();
customerLineMapper.setLineTokenizer(tokenizer);
customerLineMapper.setFieldSetMapper(new CustomerFieldSetMapper());
customerLineMapper.afterPropertiesSet();

 FlatFileItemReader<Customer> reader = new FlatFileItemReader<>();
reader.setLineMapper(customerLineMapper);

 return reader;
}

 @Bean
public ItemWriter<Customer> customerItemWriter(){
return items -> {
for (Customer customer : items) {
System.out.println(customer.toString());
}
};
}

@Bean
public Step step1() {
return stepBuilderFactory.get("step1")
.<Customer, Customer>chunk(10)
.reader(multiResourceItemreader())
.writer(customerItemWriter())
.build();
}

@Bean
public Job job() {
return jobBuilderFactory.get("job")
.start(step1())
.build();
}

}
MultipleFlatFilesApplication.java

@SpringBootApplication
@EnableBatchProcessing
public class MultipleFlatFilesApplication {

public static void main(String[] args) {
SpringApplication.run(MultipleFlatFilesApplication.class, args);
}
}

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.1.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.example</groupId>
<artifactId>multipleFlatFiles</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>multipleFlatFiles</name>
<description>Demo project for Spring Boot</description>

<properties>
<java.version>1.8</java.version>
</properties>

<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-batch</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>

<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.2</version>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.batch</groupId>
<artifactId>spring-batch-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>

Here no need to have application.properties file. In you need you can add and utilize it.

   .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.1.1.RELEASE)

 2018-12-19 22:45:01.750  INFO 5508 --- [           main] c.example.MultipleFlatFilesApplication   : Starting MultipleFlatFilesApplication on DESKTOP-NQ639DU with PID 5508 (F:\spring-boot-spring-batch-master\Spring-Batch-by-Michael-Minella\multipleFlatFiles\target\classes started by pc in F:\spring-boot-spring-batch-master\Spring-Batch-by-Michael-Minella\multipleFlatFiles)
2018-12-19 22:45:01.754  INFO 5508 --- [           main] c.example.MultipleFlatFilesApplication   : No active profile set, falling back to default profiles: default
2018-12-19 22:45:03.035  INFO 5508 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...
2018-12-19 22:45:03.294  INFO 5508 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed.
2018-12-19 22:45:03.302  INFO 5508 --- [           main] o.s.b.c.r.s.JobRepositoryFactoryBean     : No database type set, using meta data indicating: H2
2018-12-19 22:45:03.484  INFO 5508 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : No TaskExecutor has been set, defaulting to synchronous executor.
2018-12-19 22:45:03.755  INFO 5508 --- [           main] c.example.MultipleFlatFilesApplication   : Started MultipleFlatFilesApplication in 2.391 seconds (JVM running for 2.951)
2018-12-19 22:45:03.757  INFO 5508 --- [           main] o.s.b.a.b.JobLauncherCommandLineRunner   : Running default command line with: []
2018-12-19 22:45:03.845  INFO 5508 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [SimpleJob: [name=job]] launched with the following parameters: [{}]
2018-12-19 22:45:03.876  INFO 5508 --- [           main] o.s.batch.core.job.SimpleStepHandler     : Executing step: [step1]
Customer(id=1, firstName=John, lastName=Doe, birthdate=1952-10-10T10:10:10, resource=file [F:\spring-boot-spring-batch-master\Spring-Batch-by-Michael-Minella\multipleFlatFiles\target\classes\data\customer1.csv])
Customer(id=2, firstName=Amy, lastName=Eugene, birthdate=1985-07-05T17:10, resource=file [F:\spring-boot-spring-batch-master\Spring-Batch-by-Michael-Minella\multipleFlatFiles\target\classes\data\customer1.csv])
Customer(id=3, firstName=Laverne, lastName=Mann, birthdate=1988-12-11T10:10:10, resource=file [F:\spring-boot-spring-batch-master\Spring-Batch-by-Michael-Minella\multipleFlatFiles\target\classes\data\customer1.csv])
Customer(id=4, firstName=Janice, lastName=Preston, birthdate=1960-02-19T10:10:10, resource=file [F:\spring-boot-spring-batch-master\Spring-Batch-by-Michael-Minella\multipleFlatFiles\target\classes\data\customer1.csv])
Customer(id=5, firstName=Pauline, lastName=Rios, birthdate=1977-08-29T10:10:10, resource=file [F:\spring-boot-spring-batch-master\Spring-Batch-by-Michael-Minella\multipleFlatFiles\target\classes\data\customer1.csv])
Customer(id=6, firstName=Perry, lastName=Burnside, birthdate=1981-03-10T10:10:10, resource=file [F:\spring-boot-spring-batch-master\Spring-Batch-by-Michael-Minella\multipleFlatFiles\target\classes\data\customer1.csv])
Customer(id=7, firstName=Todd, lastName=Kinsey, birthdate=1998-12-14T10:10:10, resource=file [F:\spring-boot-spring-batch-master\Spring-Batch-by-Michael-Minella\multipleFlatFiles\target\classes\data\customer1.csv])
Customer(id=8, firstName=Jacqueline, lastName=Hyde, birthdate=1983-03-20T10:10:10, resource=file [F:\spring-boot-spring-batch-master\Spring-Batch-by-Michael-Minella\multipleFlatFiles\target\classes\data\customer1.csv])
Customer(id=9, firstName=Rico, lastName=Hale, birthdate=2000-10-10T10:10:10, resource=file [F:\spring-boot-spring-batch-master\Spring-Batch-by-Michael-Minella\multipleFlatFiles\target\classes\data\customer1.csv])
Customer(id=10, firstName=Samuel, lastName=Lamm, birthdate=1999-11-11T10:10:10, resource=file [F:\spring-boot-spring-batch-master\Spring-Batch-by-Michael-Minella\multipleFlatFiles\target\classes\data\customer1.csv])
Customer(id=11, firstName=Robert, lastName=Coster, birthdate=1972-10-10T10:10:10, resource=file [F:\spring-boot-spring-batch-master\Spring-Batch-by-Michael-Minella\multipleFlatFiles\target\classes\data\customer2.csv])
Customer(id=12, firstName=Tamara, lastName=Soler, birthdate=1978-01-02T10:10:10, resource=file [F:\spring-boot-spring-batch-master\Spring-Batch-by-Michael-Minella\multipleFlatFiles\target\classes\data\customer2.csv])
Customer(id=13, firstName=Justin, lastName=Kramer, birthdate=1951-11-19T10:10:10, resource=file [F:\spring-boot-spring-batch-master\Spring-Batch-by-Michael-Minella\multipleFlatFiles\target\classes\data\customer2.csv])
Customer(id=14, firstName=Andrea, lastName=Law, birthdate=1959-10-14T10:10:10, resource=file [F:\spring-boot-spring-batch-master\Spring-Batch-by-Michael-Minella\multipleFlatFiles\target\classes\data\customer2.csv])
Customer(id=15, firstName=Laura, lastName=Porter, birthdate=2010-12-12T10:10:10, resource=file [F:\spring-boot-spring-batch-master\Spring-Batch-by-Michael-Minella\multipleFlatFiles\target\classes\data\customer2.csv])
Customer(id=16, firstName=Michael, lastName=Cantu, birthdate=1999-04-11T10:10:10, resource=file [F:\spring-boot-spring-batch-master\Spring-Batch-by-Michael-Minella\multipleFlatFiles\target\classes\data\customer2.csv])
Customer(id=17, firstName=Andrew, lastName=Thomas, birthdate=1967-05-04T10:10:10, resource=file [F:\spring-boot-spring-batch-master\Spring-Batch-by-Michael-Minella\multipleFlatFiles\target\classes\data\customer2.csv])
Customer(id=18, firstName=Jose, lastName=Hannah, birthdate=1950-09-16T10:10:10, resource=file [F:\spring-boot-spring-batch-master\Spring-Batch-by-Michael-Minella\multipleFlatFiles\target\classes\data\customer2.csv])
Customer(id=19, firstName=Valerie, lastName=Hilbert, birthdate=1966-06-13T10:10:10, resource=file [F:\spring-boot-spring-batch-master\Spring-Batch-by-Michael-Minella\multipleFlatFiles\target\classes\data\customer2.csv])
Customer(id=20, firstName=Patrick, lastName=Durham, birthdate=1978-10-12T10:10:10, resource=file [F:\spring-boot-spring-batch-master\Spring-Batch-by-Michael-Minella\multipleFlatFiles\target\classes\data\customer2.csv])
2018-12-19 22:45:03.963  INFO 5508 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [SimpleJob: [name=job]] completed with the following parameters: [{}] and the following status: [COMPLETED]
2018-12-19 22:45:03.970  INFO 5508 --- [       Thread-2] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown initiated...
2018-12-19 22:45:03.988  INFO 5508 --- [       Thread-2] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown completed.









Comments

Post a Comment

Popular posts from this blog

Install Redis on Oracle Linux 7+

-
Install Redis on Oracle Linux 7+ Install Redis Permalink In this section you’ll add the  EPEL  repository, and then use it to install Redis. Add the EPEL repository, and update YUM to confirm your change: sudo yum install epel-release sudo yum update Install Redis: sudo yum install redis Start Redis: sudo systemctl start redis Optional : To automatically start Redis on boot: sudo systemctl enable redis Verify the Installation Permalink Verify that Redis is running with  redis-cli : redis-cli ping If Redis is running, it will return: PONG Configure Redis Permalink In this section, you’ll configure some basic persistence and tuning options for Redis. Persistence Options Permalink Redis provides two options for disk persistence: Point-in-time snapshots of the dataset, made at specified intervals (RDB). Append-only logs of all the write operations performed by the server (AOF). Each option has its own pros and cons whic...
Read more
-
Image
Docker Tutorial with Hello World Example Docker  is a developer tool to package applications along with their runtime environment, so anybody can deploy and run them in any other machine without facing runtime environment conflicts. It is very similar to virtual machine concept ( virtualization ), where you can get a VM image and run it on any supporting hardware. All internal programs in VM will function as they were packaged originally. Difference between a VM and a docker image  is that docker image does not package whole virtual operating system. It uses the OS resources just like other process in developer’s machine, only application and it’s runtime specific dependencies are packaged ( Containerization ). Docker allows users to publish docker images and consume those published by others in repositories like  Docker Hub . In this tutorial, learn to  install Docker container in windows , to  create docker image  and to  deploy Docker i...
Read more
-
Microservices Using Spring Boot and Spring Cloud Microservices is the hot buzzword in software development and many organizations prefer building their enterprise applications using microservices architecture. In the Java community, Spring Boot is the most widely used framework for building both monoliths and microservices. I am planning to write a series of articles covering how to build microservices using  Spring Boot  and  Spring Cloud . In this article, we are going to learn about the following: Monoliths What are microservices? Advantages of microservices Challenges with microservices Why Spring Boot and Spring Cloud are a good choice for microservices Introducing the application Traditionally, we are building large enterprise applications in a modularized fashion (?!) but finally deploy them together as a single deployment unit (EAR or WAR). These are called monolithic applications. There are some issues with the monolithic architecture, s...
Read more