What is the best handle a csv Spring Multipartfile?
I have used something like this before:
public void handleFile(MultipartFile multipartFile){
try{
InputStream inputStream = multipartFile.getInputStream();
IOUtils.readLines(inputStream, StandardCharsets.UTF_8)
.stream()
.forEach(this::handleLine);
} catch (IOException e) {
// handle exception
}
}
private void handleLine(String s) {
// do stuff per line
}
As far as I know, this first loads the whole file into a list in memory before processing it, which will probably take quite some time for files with tens of thousends of lines.
Is there a way to handle it line by line without the overhead of implementing the iteration by hand (i.e. using stuff like read(), hasNext(), ...)?
I am looking for something concise similar to this example for files from the file system:
try (Stream<String> stream = Files.lines(Paths.get("file.csv"))) {
stream.forEach(this::handleLine);
} catch (IOException e) {
// handle exception
}
In cases when you have InputStream you can use this one:
InputStream inputStream = multipartFile.getInputStream();
new BufferedReader(new InputStreamReader(inputStream, StandardCharsets.UTF_8))
.lines()
.forEach(this::handleLine);
In other cases:
No matter whether it is multipart file or you have multiple independent files, there are many approaches to do it in Java 8 using Stream API:
Solution 1:
If your files are in different directories you can do it this way:
Imagine you have a List of String which contains paths of your files like below:
List<String> files = Arrays.asList(
"/test/test.txt",
"/test2/test2.txt");
Then you can read all lines of above files as below:
files.stream().map(Paths::get)
.flatMap(path -> {
try {
return Files.lines(path);
} catch (IOException e) {
e.printStackTrace();
}
return Stream.empty();
}).forEach(System.out::println);
Solution 2:
You can also read all lines of files that exist in /test/ehsan directory using Files.walk in the following way:
try (Stream<Path> stream = Files.walk(Paths.get("/test/ehsan"), 1)) {
stream.filter(Files::isRegularFile)
.flatMap(path -> {
try {
return Files.lines(path);
} catch (IOException e) {
e.printStackTrace();
}
return Stream.empty();
})
.forEach(System.out::println);
} catch (IOException e) {
e.printStackTrace();
}
And if you want to read all lines of files in /test/ehsan directory recursively you can do it this way:
try (Stream<Path> stream = Files.walk(Paths.get("/test/ehsan"))) {
stream.filter(Files::isRegularFile)
.flatMap(path -> {
try {
return Files.lines(path);
} catch (IOException e) {
e.printStackTrace();
}
return Stream.empty();
})
.forEach(System.out::println);
} catch (IOException e) {
e.printStackTrace();
}
As you can see the second parameter to Files.walk specifies the maximum number of directory levels to visit and if you don't pass it the default will be used which is Integer.MAX_VALUE.
Solution 3:
Lets not stop here, we can go further. what if we wanted to read all lines of files exist in two completely different directories for example /test/ehsan and /test2/ehsan1?
We can do it but we should be cautious, Stream should not be so long( because it reduces readability of our program) it will be better to break them in separate methods, However because it is not possible to write multiple methods here I will write in one place how to do that:
Imagine you have a List of String which contains paths of your directories like below
list<String> dirs = Arrays.asList(
"/test/ehsan",
"/test2/ehsan1");
Then we can do that this way:
dirs.stream()
.map(Paths::get)
.flatMap(path -> {
try {
return Files.walk(path);
} catch (IOException e) {
e.printStackTrace();
}
return Stream.empty();
})
.filter(Files::isRegularFile)
.flatMap(path -> {
try {
return Files.lines(path);
} catch (IOException e) {
e.printStackTrace();
}
return Stream.empty();
})
.forEach(System.out::println);
public static List<String> readCSV(String fileName) throws IOException {
List<String> records = new ArrayList<>();
try (BufferedReader br = new BufferedReader(
new FileReader(fileName))) {
String line;
while ((line = br.readLine()) != null) {
records.add(line);
}
}
return records;
}
Related
I'm learning about Java and exception handling and the try/catch blocks. And I'm doing an example from YouTube, and I want to ask you if this is a pattern or something when you use 2 try blocks and one catch block:
private List<User> parseCSVFile(final MultipartFile file) throws Exception {
final List<User> users = new ArrayList<>();
try {
try (final BufferedReader br = new BufferedReader(new InputStreamReader(file.getInputStream()))) {
String line;
while ((line = br.readLine()) != null) {
final String[] data = line.split(",");
final User user = new User();
user.setName(data[0]);
user.setEmail(data[1]);
user.setGender(data[2]);
users.add(user);
}
return users;
}
} catch (final IOException e) {
logger.error("Failed to parse CSV file {}", e);
throw new Exception("Failed to parse CSV file {}", e);
}
}
I try to understand why this approach is better than using a try and a catch block like this:
try (final BufferedReader br = new BufferedReader(new InputStreamReader(file.getInputStream()))) {
String line;
while ((line = br.readLine()) != null) {
final String[] data = line.split(",");
final User user = new User();
user.setName(data[0]);
user.setEmail(data[1]);
user.setGender(data[2]);
users.add(user);
}
return users;
} catch (final IOException e) {
logger.error("Failed to parse CSV file {}", e);
throw new Exception("Failed to parse CSV file {}", e);
}
Two nested try-blocks in the first code snippet are redundant. When an exception occurs, it would be propagated until the point where it can be handled (or otherwise the execution would terminate).
Regarding the usage of try-blocks, you need to understand that it doesn't make sense having try without catch or finally (and compiler will not allow that).
Note that in case of try-with-resources try(myResource){}, you do have an implicit finally-block, therefore even without a catch-block try-with-resources can be useful and perfectly valid from the compiler perspective of view.
For more information on exception-handling refer to the official tutorial provided by Oracle.
EDIT
Looks like java changed behavior of try-with-resource, so my answer is no longer valid.
It did explains why you can see double try in some old tutorials.
ORIGINAL POST
Try with resources is syntactic sugar.
try(A a = foo()){
bar();
} catch (E e) {
}
is equivalent of
A a = foo();
try {
bar();
} catch (E e) {
} finally {
a.close();
}
Which mean exception thrown from foo won't be caught. therefore you need another try ... catch.
I am working on a debuging csv output inside an event driven java application. I define my filewriter like this on init.
public File csvFile;
public FileWriter fileWriter;
then I initialies them
this.csvFile = new File("c:\\missingitems.csv");
this.fileWriter = null;
try {
this.fileWriter = new FileWriter(this.csvFile);
StringBuilder line = new StringBuilder();
line.append("Date, ItemId");
line.append("\n");
this.fileWriter.write(line.toString());
} catch (IOException e) {
e.printStackTrace();
}
then within my actual program logic this gets called for every timestep in my data
for(Long item : this.items) {
StringBuilder line = new StringBuilder();
line.append(event.getDateTime().toLocalDate().toString() + "," +item.intValue());
line.append("\n");
try {
this.fileWriter.write(line.toString());
} catch (IOException e) {
e.printStackTrace();
}
}
on exit of my program I call
try {
this.fileWriter.close();
} catch (IOException e) {
e.printStackTrace();
}
However this seems to be only working if i close the filewriter after each append. Is there a way of keeping the file open and just append to it, I would also like to not loose my data in case my application crashes. I am a python guy and not super familiar with java.
I have PVPStats objects stored in PlayerMeta.java:
public static Map <UUID, PVPstats> sPVPStats = new HashMap<>();
I know for sure the map is getting populated with objects that contain the expected vars for each uuid.
I'm trying to write these objects (converted to single lines of strings) into plugins/core/killstats.txt when the server calls onDisable() in Main.java
Along with the Map object, in PlayerMeta.java are also the methods to update and retrieve PVPStats objects from the Map. Those are all working.
The part that is not is working is the write method:
public static void writePVPStats() throws IOException {
BufferedWriter w = new BufferedWriter(new FileWriter("plugins/core/killstats.txt"));
sPVPStats.keySet().forEach(user -> {
try {
System.out.println(sPVPStats);
// stdout = {a6b6e3a1-a1ec-4fee-9d6d-f5e495c3e9d7=a6b6e3a1-a1ec-4fee-9d6d-f5e495c3e9d7:1:0}
w.write(user.toString() + "\n");
w.flush();
} catch (IOException e) {
throw new UncheckedIOException(e);
}
});
w.close();
}
kill.txt after onDisable() is done:
a6b6e3a1-a1ec-4fee-9d6d-f5e495c3e9d7
Instead it needs to be:
{a6b6e3a1-a1ec-4fee-9d6d-f5e495c3e9d7=a6b6e3a1-a1ec-4fee-9d6d-f5e495c3e9d7:1:0}
For reference, here is the complete PVPStats class.
Lastly, in case it matters / helps, the reader on server launch:
Files.readAllLines(killstats_user_database.toPath()).forEach(line -> {
PVPstats stats = PVPstats.fromString(line);
PlayerMeta.sPVPStats.put(stats.playerid, stats);
});
Source Code:
backend.FileManager.java
backend.PlayerMeta.java
backend.PVPstats.java
events.PVP.java
EDIT
I just tried this with killstats.txt file type nad killstats.txt doesnt have anything in it now.
public static void writePVPStats() throws IOException {
BufferedWriter w = new BufferedWriter(new FileWriter("plugins/core/killstats.txt"));
for (PVPstats object: sPVPStats.values()) {
try {
System.out.println(sPVPStats);
w.write(object.toString() + "\n");
w.flush();
} catch (IOException e) {
throw new UncheckedIOException(e);
}
};
w.close();
}
Ok so there were multiple issues. I wasn't controlling the flushing of the buffer, I declared the hasmap incorrectly, I wasn't accessing the value part of the hash map, and I wasn't correctly enforcing plain text.
SOLUTION
public static Map <UUID, PVPstats> sPVPStats = new HashMap<UUID, PVPstats>();
public static void writePVPStats() throws IOException {
BufferedWriter w = new BufferedWriter(new FileWriter("plugins/core/killstats.txt"));
for (PVPstats object: sPVPStats.values()) {
try {
System.out.println(sPVPStats);
System.out.println(object.toString());
w.write(object.toString() + "\n");
w.flush();
} catch (IOException e) {
throw new UncheckedIOException(e);
}
};
w.close();
}
I am trying write to a csv file. After the execution of the code bellow the csv file is still empty.
File is in folder .../webapp/resources/.
This is my dao class:
public class UserDaoImpl implements UserDao {
private Resource cvsFile;
public void setCvsFile(Resource cvsFile) {
this.cvsFile = cvsFile;
}
#Override
public void createUser(User user) {
String userPropertiesAsString = user.getId() + "," + user.getName()
+ "," + user.getSurname() +"\n";;
System.out.println(cvsFile.getFilename());
FileWriter outputStream = null;
try {
outputStream = new FileWriter(cvsFile.getFile());
outputStream.append(userPropertiesAsString);
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
outputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
#Override
public List<User> getAll() {
return null;
}
}
This is a part of beans.xml.
<bean id="userDao" class="pl.project.dao.UserDaoImpl"
p:cvsFile="/resources/users.cvs"/>
Program compiles and doesn't throw any exceptions but CSV file is empty.
If you're running your app in IDE, the /webapp/resources used for running app will differ from the /webapp/resources in your IDE. Try to log full path to file and check there.
try using outputStream.flush() as the final statement in the first of the try block.
I think you're looking at the wrong file. If you specify an absolute path /resources/users.cvs, then it probably won't be written into the a folder relative to the webapp. Instead, it will be written to /resources/users.cvs
So the first step is to always log an absolute path to make sure the file is where you expect it.
Try with this code, it will at least tell you where the problem lies (Java 7+):
// Why doesn't this method throw an IOException?
#Override
public void createUser(final User user)
{
final String s = String.format("%s,%s,%s",
Objects.requireNonNull(user).getId(),
user.getName(), user.getSurname()
);
// Note: supposes that .getFile() returns a File object
final Path path = csvFile.getFile().toPath().toAbsolutePath();
final Path csv;
// Note: this supposes that the CSV is supposed to exist!
try {
csv = path.toRealPath();
} catch (IOException e) {
throw new RuntimeException("cannot locate CSV " + path, e);
}
try (
// Note: default is to TRUNCATE the destination.
// If you want to append, add StandardOpenOption.APPEND.
// See javadoc for more details.
final BufferedWriter writer = Files.newBufferedWriter(csv,
StandardCharsets.UTF_8);
) {
writer.write(s);
writer.newLine();
writer.flush();
} catch (IOException e) {
throw new RuntimeException("write failure", e);
}
}
I am wondering why I get this warning with the new eclipse Juno despite I think I correctly closed everything. Could you please tell me why I get this warning in the following piece of code?
public static boolean copyFile(String fileSource, String fileDestination)
{
try
{
// Create channel on the source (the line below generates a warning unassigned closeable value)
FileChannel srcChannel = new FileInputStream(fileSource).getChannel();
// Create channel on the destination (the line below generates a warning unassigned closeable value)
FileChannel dstChannel = new FileOutputStream(fileDestination).getChannel();
// Copy file contents from source to destination
dstChannel.transferFrom(srcChannel, 0, srcChannel.size());
// Close the channels
srcChannel.close();
dstChannel.close();
return true;
}
catch (IOException e)
{
return false;
}
}
IF you're running on Java 7, you can use the new try-with-resources blocks like so, and your streams will be automatically closed:
public static boolean copyFile(String fileSource, String fileDestination)
{
try(
FileInputStream srcStream = new FileInputStream(fileSource);
FileOutputStream dstStream = new FileOutputStream(fileDestination) )
{
dstStream.getChannel().transferFrom(srcStream.getChannel(), 0, srcStream.getChannel().size());
return true;
}
catch (IOException e)
{
return false;
}
}
You won't need to explicitly close the underlying channels. However if you're not using Java 7, you should write the code in a cumbersome old way, with finally blocks:
public static boolean copyFile(String fileSource, String fileDestination)
{
FileInputStream srcStream=null;
FileOutputStream dstStream=null;
try {
srcStream = new FileInputStream(fileSource);
dstStream = new FileOutputStream(fileDestination)
dstStream.getChannel().transferFrom(srcStream.getChannel(), 0, srcStream.getChannel().size());
return true;
}
catch (IOException e)
{
return false;
} finally {
try { srcStream.close(); } catch (Exception e) {}
try { dstStream.close(); } catch (Exception e) {}
}
}
See how much better the Java 7 version is :)
You should always close in finally because if an exception rise, you won't close the resources.
FileChannel srcChannel = null
try {
srcChannel = xxx;
} finally {
if (srcChannel != null) {
srcChannel.close();
}
}
Note: even if you put a return in the catch block, the finally block will be done.
eclipse is warning you about the FileInputStream and FileOutputStream that you can no longer reference.