Spring, FlatFileItemWriter write large file - java

I need to write a large file using FlatFileItemWriter.
When I call the write method, a OutOfMemory occurs.
Is there a way to write large files ?
My writer is defined like this :
FlatFileItemWriter<FieldSet> writer = new FlatFileItemWriter<>();
File mostRecent = <myFile>
FileSystemResource resource = new FileSystemResource(<myTmpFile>);
FileUtils.deleteQuietly(resource.getFile());
writer.setResource(resource);
DelimitedLineAggregator<FieldSet> delimitedLine = new DelimitedLineAggregator<>();
delimitedLine.setDelimiter(SEPARATOR);
writer.setLineAggregator(delimitedLine);
writer.setHeaderCallback(new FlatFileHeaderCallback() {
public void writeHeader(Writer writer) throws IOException
{
StringBuffer buffer = new StringBuffer();
for (Column column : Column.values())
{
buffer.append(column.getName());
buffer.append(SEPARATOR);
}
writer.write(CHAR_FOR_EXCEL + buffer.toString());
}
});
writer.open(new ExecutionContext());
writer.write(items);

Related

Spring Boot Controller export an Excel

I have a java/spring boot application where I want to build an API endpoint that creates and returns a downloadable excel file. Here is my controller endpoint:
#RestController
#RequestMapping("/Foo")
public class FooController {
private final FooService fooService;
#GetMapping("/export")
public ResponseEntity export() {
Resource responseFile = fooService.export();
return ResponseEntity.ok()
.header(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename="+responseFile.getFilename())
.contentType(MediaType.MULTIPART_FORM_DATA)
.body(responseFile);
}
}
Then the service class
public class FooService {
public Resource export() throws IOException {
StringBuilder filename = new StringBuilder("Foo Export").append(" - ")
.append("Test 1.xlsx");
return export(filename);
}
private ByteArrayResource export(String filename) throws IOException {
byte[] bytes = new byte[1024];
try (Workbook workbook = generateExcel()) {
FileOutputStream fos = write(workbook, filename);
fos.write(bytes);
fos.flush();
fos.close();
}
return new ByteArrayResource(bytes);
}
private Workbook generateExcel() {
Workbook workbook = new XSSFWorkbook();
Sheet sheet = workbook.createSheet();
//create columns and rows
return workbook;
}
private FileOutputStream write(final Workbook workbook, final String filename) throws IOException {
FileOutputStream fos = new FileOutputStream(filename);
workbook.write(fos);
fos.close();
return fos;
}
}
This code successfully creates the proper excel file using the Apache POI library. But this won't return it out of the controller properly because ByteArrayResource::getFilename always returns null:
/**
* This implementation always returns {#code null},
* assuming that this resource type does not have a filename.
*/
#Override
public String getFilename() {
return null;
}
What type of resource can I use to return the generated excel file?
Since you are using ByteArrayResource, you can use the below controller code assuming that the FooService is autowired in the controller class.
#RequestMapping(path = "/download_excel", method = RequestMethod.GET)
public ResponseEntity<Resource> download(String fileName) throws IOException {
ByteArrayResource resource = fooService.export(fileName);
return ResponseEntity.ok()
.headers(headers) // add headers if any
.contentLength(resource.contentLength())
.contentType(MediaType.parseMediaType("application/vnd.ms-excel"))
.body(resource);
}
Basically , there are few points that you first need to understand & then decide what you want to do ,
1.Is excel creation on disk needed or can you stream it from memory?
If its a download pop up, user might keep it open for long time & memory be occupied during that period ( disadvantage of in memory approach ) .
Secondly, if generated file has to be new for each request ( i.e. data to be exported is different ) then there is no point in keeping it at disk ( disadvantage of in disk approach ) .
Thirdly, it will be hard for an API code to do disk clean up because you never know in advance as when user will finish up his down load ( disadvantage of in disk approach ) .
Answer by Fizik26 is this In - Memory approach where you don't create a file on disk. . Only thing from that answer is that you need to keep track of length of array out.toByteArray() & that can easily be done via a wrapper class.
2.While downloading a file , your code needs to stream a file chunk by chunk - thats what Java streams are for.
Code like below does that.
return ResponseEntity.ok().contentLength(inputStreamWrapper.getByteCount())
.contentType(MediaType.parseMediaType("application/vnd.ms-excel"))
.cacheControl(CacheControl.noCache())
.header("Content-Disposition", "attachment; filename=" + "SYSTEM_GENERATED_FILE_NM")
.body(new InputStreamResource(inputStreamWrapper.getByteArrayInputStream()));
and inputStreamWrapper is like ,
public class ByteArrayInputStreamWrapper {
private ByteArrayInputStream byteArrayInputStream;
private int byteCount;
public ByteArrayInputStream getByteArrayInputStream() {
return byteArrayInputStream;
}
public void setByteArrayInputStream(ByteArrayInputStream byteArrayInputStream) {
this.byteArrayInputStream = byteArrayInputStream;
}
public int getByteCount() {
return byteCount;
}
public void setByteCount(int byteCount) {
this.byteCount = byteCount;
}
}
Regarding file name, if file name is not an input to end point - that means ..its system generated ( a combination of constant string plus a variable part per user ). I am not sure why you need to get that from resource.
You won't need this wrapper if use - org.springframework.core.io.ByteArrayResource
Letting controller know is always better what it is going to write using ReponseEntity. At service level just create and play around the objects. #RestController or #Controller doesn't matter here.
What you are looking forward for in your controller is somewhat like this (sample) -
#GetMapping(value = "/alluserreportExcel")
public ResponseEntity<InputStreamResource> excelCustomersReport() throws IOException {
List<AppUser> users = (List<AppUser>) userService.findAllUsers();
ByteArrayInputStream in = GenerateExcelReport.usersToExcel(users);
// return IO ByteArray(in);
HttpHeaders headers = new HttpHeaders();
// set filename in header
headers.add("Content-Disposition", "attachment; filename=users.xlsx");
return ResponseEntity.ok().headers(headers).body(new InputStreamResource(in));
}
Generate Excel Class -
public class GenerateExcelReport {
public static ByteArrayInputStream usersToExcel(List<AppUser> users) throws IOException {
...
...
//your list here
int rowIdx = 1;
for (AppUser user : users) {
Row row = sheet.createRow(rowIdx++);
row.createCell(0).setCellValue(user.getId().toString());
...
}
workbook.write(out);
return new ByteArrayInputStream(out.toByteArray());
and finally, somewhere, in your view -
<a href="<c:url value='/alluserreportExcel' />"
target="_blank">Export all users to MS-Excel</a>
For full example, take a peek -
here, here and here.
You have to set the file name to the response header using Content-disposition. Try this
#GetMapping("/export")
public ResponseEntity export(HttpServletResponse response) {
fooService.export(response);
}
Change your service method like this
public Resource export(HttpServletResponse response) throws IOException {
StringBuilder filename = new StringBuilder("Foo Export").append(" - ")
.append("Test 1.xlsx");
return export(filename, response);
}
private void export(String filename, HttpServletResponse response) throws IOException {
try (Workbook workbook = generateExcel()) {
FileOutputStream fos = write(workbook, filename);
IOUtils.copy(new FileInputStream(fos.getFD()),
servletResponse.getOutputStream());//IOUtils is from apache commons io
response.setContentType("application/vnd.ms-excel");
response.setHeader("Content-disposition", "attachment; filename=" + filename);
}catch(Exception e) {
//catch if any checked exception
}finally{
//Close all the streams
}
}
You can use this :
headers.add("Content-Disposition", "attachment; filename=NAMEOFYOURFILE.xlsx");
ByteArrayInputStream in = fooService.export();
return ResponseEntity
.ok()
.headers(headers)
.body(new InputStreamResource(in));
It will download the Excel file when you call this endpoint.
In your export method in your service, you have to return something like that :
ByteArrayOutputStream out = new ByteArrayOutputStream();
try {
workbook.write(out);
} catch (IOException e) {
e.printStackTrace();
}
return new ByteArrayInputStream(out.toByteArray());

Accessing an external file

I'm looking to reaccess a file created in android studio. I found the following code snippet on stack exchange
File root = android.os.Environment.getExternalStorageDirectory();
// See http://stackoverflow.com/questions/3551821/android-write-to-sd-card-folder
File dir = new File(root.getAbsolutePath() + "/download");
File file = new File(dir, "myData.txt");
From my understanding this creates a file called "myData.txt" in the download folder. What im looking to do is pass "file" into a function in another method like so
public void onLocationChanged(Location location) {
ArrayList<Location> list = new ArrayList<>();
list.add(location);
GPX.writePath(file,"hello",list);
}
How do I go about creating a file variable that acesses the txt file without actually creating a new file?
File root = android.os.Environment.getExternalStorageDirectory();
// See http://stackoverflow.com/questions/3551821/android-write-to-sd-card-folder
File dir = new File(root.getAbsolutePath() + "/download");
File file = new File(dir, "myData.txt");
Actually it does not create a physical file. It just creates an object in memory (referenced by file).
You can then use the reference in your method to write to a physical file, like so:
public void onLocationChanged(File location) throws IOException {
ArrayList<File> list = new ArrayList<>();
list.add(location);
writeToFile("Text", location);
}
private static void writeToFile(String text, File file)
throws IOException {
try(BufferedWriter writer = new BufferedWriter(new FileWriter(file))) {
writer.write(text);
}
}
Update 2
If you want to use the reference in your method, you can pass the file reference to your method which should have some code to write the text into the actual file:
GPX.writePath(file,"hello",list);
...
public static void writePath(File file, String n, List<Location> points) throws IOException {
...
try(BufferedWriter writer = new BufferedWriter(new FileWriter(file))) {
writer.write(n);
}
}

Create html file and add rows dynamically

I wanted to create log.html file using java wherein i'll dump errors into html table-row as and when errors are caught.
I created table columns but when i am passing anything to it then my data is overwriting column names instead of appending row.
My java code is as follows:
public static void main(String[] args) throws IOException
{
String month = getMonth(1);
System.out.println(month);
//table(month);
addrow("abc");
}
public static void table(String Update) throws IOException
{
//define a HTML String Builder
StringBuilder htmlStringBuilder=new StringBuilder();
htmlStringBuilder.append("<html><head><title>Packaging Issues</title></head>");
htmlStringBuilder.append("<body><h1> Files Received </h1>");
htmlStringBuilder.append("<table border=\"1\" bordercolor=\"#000000\">");
htmlStringBuilder.append("<tr><td><b>Cycle</b></td><td>"+Update+"</td></tr>");
htmlStringBuilder.append("<tr><td><b>Version</b></td><td></tr>");
newtable(htmlStringBuilder.toString());
}
public static void newtable(String a)
{
StringBuilder htmlStringBuilder=new StringBuilder();
htmlStringBuilder.append(a);
htmlStringBuilder.append("<h1> Issues Found </h1>");
htmlStringBuilder.append("<table border=\"1\" bordercolor=\"#000000\">");
htmlStringBuilder.append("<tr><td><b>Type</b></td>");
htmlStringBuilder.append("<td><b>Field</b></td>");
htmlStringBuilder.append("<td><b>Issue</b></td></tr>");
addrow(htmlStringBuilder.toString());
// WriteToFile(htmlStringBuilder.toString(),"log.html");
}
public static void addrow(String a)
{
try {
StringBuilder htmlStringBuilder=new StringBuilder();
htmlStringBuilder.append("<tr><td><b>"+a+"</b></td>");
htmlStringBuilder.append("<td><b>Field</b></td>");
htmlStringBuilder.append("<td><b>Issue</b></td></tr>");
htmlStringBuilder.append("</table></body></html>");
WriteToFile(htmlStringBuilder.toString(),"log.html");
} catch (IOException e) {
e.printStackTrace();
}
}
Here's the writetofile method:
public static void WriteToFile(String fileContent, String fileName) throws IOException {
String projectPath = "/home/abc";
String tempFile = projectPath + File.separator+fileName;
File file = new File(tempFile);
// if file does exists, then delete and create a new file
//write to file with OutputStreamWriter
OutputStream outputStream = new FileOutputStream(file.getAbsoluteFile());
Writer writer=new OutputStreamWriter(outputStream);
writer.write(fileContent);
writer.close();
}
The constructor FileOutputStream(file) overwrites the contents of the file. Use this constructor instead
FileOutputStream fos =new FileOutputStream(file, true) ;

Uploading a file using Spring MVC not working [duplicate]

This question already has answers here:
How to upload only csv files into the server in Java
(3 answers)
Closed 7 years ago.
I am trying to upload a csv file and i intend to write success to the uploaded csv if the upload goes successfully. below is my code(this is not working as of now)
#RequestMapping(value = "/submitUploadCarForm")
public String uploadCar(CarUploadForm uploadform,HttpServletRequest request, Model model, BindingResult result) {
try {
CommonsMultipartFile file = uploadform.getFileData();
// parse csv file to list
csvList = CarCSVUtil.getCSVInputList(file.getInputStream());
for (CarCSVFileInputBean inputRecord : csvList) {
Car carentity = new Car();
carentity.setId(inputRecord.getId());
carentity.setName(inputRecord.getcarName());
carentity.setShortName(inputRecord.getcarShortName());
carentity.setEnvironment(inputRecord.getEnvironment());
carentity = this.Service.saveCar(carentity);
CarConfig compcfg = new CarConfig();
compcfg.setCar(carentity);
compcfg.setCarType(pickCarType(carTypes,inputRecord.getcarType()));
this.Service.saveCompCfg(compcfg);
inputRecord.setuploadstatus("success");<--- This is where i need help
}
}
catch (Exception e) {
e.printStackTrace();
result.rejectValue("name", "failureMsg","Error while uploading ");
model.addAttribute("failureMsg", "Error while uploading ");
}
return "view";
}
I am using this code to import csv file and data is store into database.Add opencsv jar file to your build path.
public String importCSV(#RequestParam("file") MultipartFile file,
HttpServletRequest req) throws IOException {
CsvToBean csv = new CsvToBean();
CSVReader reader = new CSVReader(new InputStreamReader(
file.getInputStream()), ',', '\"', 1);
ColumnPositionMappingStrategy maping = new ColumnPositionMappingStrategy();
maping.setType(tbBank.class);
String[] column = { "bankid", "bankname", "bankbranch" };
maping.setColumnMapping(column);
List banklist = csv.parse(maping, reader);
for (Object obj : banklist) {
tbBank bank = (tbBank) obj;
projectservice.insertBank(bank);
}
return "redirect:/bankview";
}

Closing a file after writing recursively

I don't know why my code only writes in the file the first time and then nothing, I think the problem is salida.close() but i'm not sure. A bit of help will be appreciated. The method itself saves a binary tree in a file. If you need more information, ask me. Here is my code:
public boolean guardarAgenda() throws IOException
{
NodoAgenda raiz;
raiz = this.root;
guardar(raiz);
if(this.numNodes == 0)
return true;
else return false;
}
public void guardar(NodoAgenda nodo) throws IOException
{
FileWriter fich_s = new FileWriter("archivo.txt");
BufferedWriter be = new BufferedWriter(fich_s);
PrintWriter output = new PrintWriter(be);
Parser p = new Parser();
if (nodo != null)
{
guardar(nodo.left);
p.ponerPersona(nodo.info);
String linea = p.obtainLine();
salida.println(linea);
guardar(nodo.right);
this.numNodes--;
}
output.close();
}
You are opening the same file on multiple calls without even the option of appending content. So no wonder you are not finding what you expect in it once processing ended.
When working with recursion and using the same resource on each call it's better to have a second method that takes that result as argument, and does only the writing/adding to it, opening/instantiation should happen in a method which initiates the first call, something similar to this:
public void guardar(NodoAgenda nodo) throws IOException
{
FileWriter fich_s = new FileWriter("archivo.txt");
BufferedWriter be = new BufferedWriter(fich_s);
PrintWriter salida = new PrintWriter(be);
if (nodo != null)
{
guardar(nodo.left, salida);
}
output.close();
}
public void guardar(NodoAgenda nodo, PrintWriter salida) throws IOException
{
if (nodo != null)
{
Parser p = new Parser();
guardar(nodo.left);
p.ponerPersona(nodo.info);
String linea = p.obtainLine();
salida.println(linea);
guardar(nodo.right, salida);
this.numNodes--;
}
}

Categories

Resources