Develop Reference

Reading and writing file in ISO-8859-1 encoding?

I have file encoded in ISO-8859-1. I'm trying to read it in as a single String, do some regex substitutions on it, and write it back out in the same encoding.
However, the resulting file I get always seems to be UTF-8 (according to Notepad++ at least), mangling some characters.
Can anyone see what I'm doing wrong here?
private static void editFile(File source, File target) {
// Source and target encoding
Charset iso88591charset = Charset.forName("ISO-8859-1");
// Read the file as a single string
String fileContent = null;
try (Scanner scanner = new Scanner(source, iso88591charset)) {
fileContent = scanner.useDelimiter("\\Z").next();
} catch (IOException exception) {
LOGGER.error("Could not read input file as a single String.", exception);
return;
}
// Do some regex substitutions on the fileContent string
String newContent = regex(fileContent);
// Write the file back out in target encoding
try (BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(target), iso88591charset))) {
writer.write(newContent);
} catch (Exception exception) {
LOGGER.error("Could not write out edited file!", exception);
}
}

There is nothing actually wrong with your code. Notepad++ sees the file encoded in UTF-8 because on a basic level there is no difference between UTF-8 and the encoding you're trying to use. Only specific characters differ and some (a lot) are missing from ISO compared to UTF. You can read more here or by simply searching ISO-8859-1 vs UTF-8 in Google.
I've created a simple project with your code and tested it with characters that are different for the ISO encoding - the result is a file that IntelliJ (and probably Notepad++ as well - cannot easily check, I'm on Linux) recognizes as ISO-8859-1. Apart from that, I've added another class that makes use of new (JDK11) features from Files class. The new Scanner(source, charset) that you've used was added in JDK10, so I think that you may be using 11 already. Here's the simplified code:
private static void editFile(File source, File target) {
Charset charset = StandardCharsets.ISO_8859_1;
String fileContent;
try {
fileContent = Files.readString(source.toPath(), charset);
} catch (IOException exception) {
System.err.println("Could not read input file as a single String.");
exception.printStackTrace();
return;
}
String newContent = regex(fileContent);
try {
Files.writeString(target.toPath(), newContent, charset);
} catch (IOException exception) {
System.err.println("Could not write out edited file!");
exception.printStackTrace();
}
}
Feel free to clone the repository or check it on GitHub and use whichever code version you prefer.

Reading InputStream from webservice response

What I am trying to do seems to be a quite simple thing, get an InputStream from a Jersey webservice which is returned from a class RestResponse. But I don't get the stream to my client:
public class RestResponse {
private InputStream responseStream;
public RestResponse(InputStream responseBodyStream) throws IOException{
this.responseStream = responseBodyStream;
//here I can get the stream contents from this.responseStream
}
public InputStream getResponseStream() throws IOException {
//here stream content is empty, if called from outside
//only contains content, if called from constructor
return this.responseStream;
}
}
public class HttpURLConnectionClient{
public RestResponse call(){
try{
....
InputStream in = httpURLConnection.getInputStream();
RestResponse rr = new RestResponse(in);
}finally{
in.close(); <- this should be the suspect
}
}
}
RestResponse rr = httpURLConnectionClient.call()//call to some url
rr.getResponseStream(); //-> stream content is empty
Any ideas, what I am missing? Is is not possible to just pipe the stream through?

Certain types of InputStream can only be read once in Java. Based on your comment above, it appears that you are using the InputStream when you pipe it to System.out. Try commenting out the call to System.out and see if you can access your InputStream. Also make sure that the stream is not being consumed anywhere else in the code before the point where you need it.
Update:
It appears that your actual problem was being caused by closing the InputStream before you got a chance to use it. So the solution is to keep the stream open until you need it, and close it afterwards.
Typically, it is not a good design practice to open a stream and keep it open for a long time, because then the underlying resource won't be available to anyone else who needs it. So you should open the stream, and consume it only when you actually need it.

Can't read previously written JSON data (Unterminated String error) in Java

I'm using the twitter4j package for an information retrieval class and have collected some tweets. However, for the next part of the assignment, I am to use Lucene to index on the tweets. In order to do this, my thought was to save the tweets as JSON Strings to a file and then reread them when needed. However, I'm running into an error.
When the file is written, I can see the entire JSON object just fine. The total object is quite large (2500 characters). However, when reading back from the file, I get a Unterminated string at xxxx error. I am using the TwitterObjectFactory methods to both write and read the string. Here is a sample code:
Writing:
public void onStatus(Status status) {
try{
String jsonString = TwitterObjectFactory.getRawJSON(status);
output.write(jsonString+"\n");
numTweets++;
if(numTweets > 10){
synchronized(lock){
lock.notify();
}
}
}
catch(IOException e){
e.printStackTrace();
}
}
Reading:
Scanner input = new Scanner(file);
while(input.hasNext()){
Status status = TwitterObjectFactory.createStatus(input.nextLine());
System.out.println(status.getUser().getScreenName());
}
This works only some of the time. If I run the program multiple times and get many tweets, the program almost always crashes after 2-3 tweets have been read from the file, always with the same error. If you'd like to replicate the code, you can follow this example. I've added a synchronized block in order to close the stream after 10 tweets, but it's not necessary to replicate the error.
Can someone explain what is happening? My guess is that there's something wrong with the way I'm encoding the JSON into the file. I'm using BufferedWriter wrapping an OutputStreamWriter in order to output in UTF-8 format.
Edit: I do close the stream. Here's the bottom snippet of the code:
twitterStream.addListener(listener);
twitterStream.sample("en");
try{
synchronized(lock){
lock.wait();
}
}
catch(InterruptedException e){
e.printStackTrace();
}
twitterStream.clearListeners();
twitterStream.cleanUp();
twitterStream.shutdown();
output.close();

You probably need to flush your output, before you notify the reader. Otherwise parts of your String will stay in the buffer.
public void onStatus(Status status) {
try{
String jsonString = TwitterObjectFactory.getRawJSON(status);
output.write(jsonString+"\n");
output.flush();
numTweets++;
if(numTweets > 10){
synchronized(lock){
lock.notify();
}
}
}
catch(IOException e){
e.printStackTrace();
}
}

I don't see the code where you properly close the BufferedWriter. If you don't close it manually before the first program ends, then data might remain in the internal buffer and never written to the file.
You can also try to open the file in a text editor and look at the contents. Tools like http://codebeautify.org/jsonviewer or http://jsonlint.com/ allow you to validate/beautify the contents to see errors.
Lastly, try BufferedReader( new InputStreamReader( new FileInputStream(file), "UTF-8" ) ). Maybe non-ASCII characters in the input are confusing Scanner.

javax.imageio.ImageIO Problem creating ImageInputStream

I have a Servlet in Tomcat 5.5 that reads local images sitting on a folder. The image is then sent back to an Applet.
I'm getting this "javax.imageio.IIOException: Can't create an ImageInputStream!" error and not sure whats causing it.
Has anyone had this problem before? Could this be a Thread issue in the ImageIO? I can't reproduce this issue since it occurs about 3 times for every 1000 requests.
EDIT: This is the Servlet code that reads the image. I just use the ImageIO.read(File) in its static form inside the Servlet's doPost method the same way below:
doPost(req,resp){
...
BufferedImage image = ImageIO.read(imageFile);
...
}
Here is the source code for javax.imageio.ImageIO.read(File):
public static BufferedImage read(File input) throws IOException {
if (input == null) {
throw new IllegalArgumentException("input == null!");
}
if (!input.canRead()) {
throw new IIOException("Can't read input file!");
}
ImageInputStream stream = createImageInputStream(input);
if (stream == null) {
throw new IIOException("Can't create an ImageInputStream!");
}
BufferedImage bi = read(stream);
if (bi == null) {
stream.close();
}
return bi;
}

If the sole functional requirement is to read images from local disk and return it unmodified to the HTTP response using a servlet, then you do not need the ImageIO at all. It only adds unnecessary overhead and other problems like you're having now.
Get rid of the ImageIO stuff and just stream the raw image straight from disk to HTTP response, along a set of proper response headers. For example,
String name = request.getParameter("name");
File file = new File("/path/to/images", name);
response.setContentType(getServletContext().getMimeType(file.getName()));
response.setHeader("Content-Length", String.valueOf(file.length()));
response.setHeader("Content-Disposition", "inline; filename=\"" + file.getName() + "\"");
InputStream input = null;
OutputStream output = null;
try {
input = new BufferedInputStream(new FileInputStream(file));
output = new BufferedOutputStream(response.getOutputStream());
byte[] buffer = new byte[8192];
for (int length; (length = input.read(buffer)) > 0;) {
output.write(buffer, 0, length);
}
} finally {
if (output != null) try { output.close(); } catch (IOException logOrIgnore) {}
if (input != null) try { input.close(); } catch (IOException logOrIgnore) {}
}
That's all. You only need ImageIO whenever you would like to manipulate the image in server's memory before returning it, e.g. resizing, transforming or something.
Another, more robust, example of such a servlet can be found here and a more advanced one here.

The source I have (Java5 but I doubt it has changed a lot) states that if there are no ImageInputStream service providers registered, the createImageInputStream method returns null and thus you get that exception.
From the JavaDoc on IIORegistry.getDefaultInstance() which is used by ImageIO:
Each ThreadGroup will receive its own instance; this allows different Applets in the same browser (for example) to each have their own registry.
Thus it might actually be a threading problem in that you get a plain new instance of IIORegistry.
Edit: digging deeper into the source I found the following:
Most likely you'd get a FileImageInputStream, since you pass in a file. However, if an exception occurs the service provider returns null. Thus there might be a FileNotFoundException or any other IOException being thrown which causes the stream not to be created.
Unfortunately, there's no logging in the code, thus you'd have to debug somehow. It's probably due to missing file permissions, a corrupted/incomplete file or the file missing.
Here's the Java5 source for FileImageInputStreamSpi#createInputStreamInstance()
public ImageInputStream createInputStreamInstance(Object input,
boolean useCache,
File cacheDir) {
if (input instanceof File) {
try {
return new FileImageInputStream((File)input);
} catch (Exception e) {
return null;
}
} else {
throw new IllegalArgumentException();
}
}

Where is your use of close() methods within the exception handling? Streams have to be closed when there are exceptions, too, as well as normal termination of the block of the code.
The symptom sounds like you run out of heap space or sometime.
It is not the coding errors that others pointed out, since the problem is intermittent.

How / when to delete a file in java?

The problem is, user clicks a button in JSP, which will export the displayed data. So what i am doing is, creating a temp. file and writing the contents in it [ resultSet >> xml >> csv ], and then writing the contents to ServletResponse. After closing the respons output stream, i try to delete the file, but every time it returns false.
code;
public static void writeFileContentToResponse ( HttpServletResponse response , String fileName ) throws IOException{
ServletOutputStream responseoutputStream = response.getOutputStream();
File file = new File(fileName);
if (file.exists()) {
file.deleteOnExit();
DataInputStream dis = new DataInputStream(new FileInputStream(
file));
response.setContentType("text/csv");
int size = (int) file.length();
response.setContentLength(size);
response.setHeader("Content-Disposition",
"attachment; filename=\"" + file.getName() + "\"");
response.setHeader("Pragma", "public");
response.setHeader("Cache-control", "must-revalidate");
if (size > Integer.MAX_VALUE) {
}
byte[] bytes = new byte[size];
dis.read(bytes);
FileCopyUtils.copy(bytes, responseoutputStream );
}
responseoutputStream.flush();
responseoutputStream.close();
file.delete();
}
i have used 'file.deleteOnExit();' and file.delete(); but none of them is working.

file.deleteOnExit() isn't going to produce the result you want here - it's purpose is to delete the file when the JVM exits - if this is called from a servlet, that means to delete the file when the server shuts down.
As for why file.delete() isn't working - all I see in this code is reading from the file and writing to the servlet's output stream - is it possible when you wrote the data to the file that you left the file's input stream open? Files won't be deleted if they're currently in use.
Also, even though your method throws IOException you still need to clean up things if there's an exception while accessing the file - put the file operations in a try block, and put the stream.close() into a finally block.

Don't create that file.
Write your data directly from your resultset to your CSV responseoutputStream.
That saves time, memory, diskspace and headache.
If you realy need it, try using File.createTempFile() method.
These files will be deleted when your VM stops normaly if they haven't been deleted before.

I'm assuming you have some sort of concurrency issue going on here. Consider making this method non-static, and use a unique name for your temp file (like append the current time, or use a guid for a filename). Chances are that you're opening the file, then someone else opens it, so the first delete fails.

as I see it, you are not closing the DataInputStream dis - this results to the false status, when you do want to delete file. Also, you should handle the streams in try-catch-finally block and close them within finally. The code is a bit rough, but it is safe:
DataInputStream dis = null;
try
{
dis = new DataInputStream(new FileInputStream(
file));
... // your other code
}
catch(FileNotFoundException P_ex)
{
// catch only Exceptions you want, react to them
}
finally
{
if(dis != null)
{
try
{
dis.close();
}
catch (IOException P_ex)
{
// handle exception, again react only to exceptions that must be reacted on
}
}
}

How are you creating the file. You probably need to use createTempFile.
You should be able to delete a temporary file just fine (No need for deleteOnExit). Are you sure the file isn't in use, when you are trying to delete it? You should have one file per user request (That is another reason you should avoid temp files and store everything in memory).

you can try piped input and piped output stream. those buffers need two threads one to feed the pipe (exporter) and the other (servlet) to consume data from the pipe and write it to the response output stream

You really don't want to create a temporary file for a request. Keep the resulting CSV in memory if at all possible.
You may need to tie the writing of the file in directly with the output. So parse a row of the result set, write it out to response stream, parse the next row and so on. That way you only keep one row in memory at a time. Problem there is that the response could time out.
If you want a shortcut method, take a look at Display tag library. It makes it very easy to show a bunch of results in a table and then add pre-built export options to said table. CSV is one of those options.

You don't need a temporary file. The byte buffer which you're creating there based on the file size may also cause OutOfMemoryError. It's all plain inefficient.
Just write the data of the ResultSet immediately to the HTTP response while iterating over the rows. Basically: writer.write(resultSet.getString("columnname")). This way you don't need to write it to a temporary file or to gobble everything in Java's memory.
Further, most JDBC drivers will by default cache everything in Java's memory before giving anything to ResultSet#next(). This is also inefficient. You'd like to let it give the data immediately row-by-row by setting the Statement#setFetchSize(). How to do it properly depends on the JDBC driver used. In case of for example MySQL, you can read it up in its JDBC driver documentation.
Here's a kickoff example, assuming that you're using MySQL:
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
response.setContentType("text/csv");
response.setCharacterEncoding("UTF-8");
Connection connection = null;
Statement statement = null;
ResultSet resultSet = null;
PrintWriter writer = response.getWriter();
try {
connection = database.getConnection();
statement = connection.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY);
statement.setFetchSize(Integer.MIN_VALUE);
resultSet = statement.executeQuery("SELECT col1, col2, col3 FROM tbl");
while (resultSet.next()) {
writer.append(resultSet.getString("col1")).append(',');
writer.append(resultSet.getString("col2")).append(',');
writer.append(resultSet.getString("col3")).println();
// Note: don't forget to escape quotes/commas as per RFC4130.
}
} catch (SQLException e) {
throw new ServletException("Retrieving CSV rows from DB failed", e);
} finally {
if (resultSet != null) try { resultSet.close; } catch (SQLException logOrIgnore) {}
if (statement != null) try { statement.close; } catch (SQLException logOrIgnore) {}
if (connection != null) try { connection.close; } catch (SQLException logOrIgnore) {}
}
}
That's it. This way effectlvely only one database row is been kept in the memory all the time.