Develop Reference

make sure slf4j captures stack traces in the log files

Let's say I have simple code:
public static void Main(){
throw new NullPointerException("this is an npe");
}
How with slf4j do you make sure that that exception is logged to the log file. Is there a default setting in spring that you can set in application properties, or in the logback file, that makes sure this gets captured, even though it is not in the try catch?
Please note, I am NOT looking for a solution to a webcontroller, or how to use log.error(). I know how to do those. I am looking for an overarching setting so that we don't lose exceptions throughout the application. This can apply to OOM, or printing a heap dump etc. I am including spring as a tag just in case the solution is in the applications.properties, as this is a spring project.

Ok as you have said, you can add a controller advice:
#ControllerAdvice
public class YourCustomHandler extends ResponseEntityExceptionHandler {
#Autowired
MessageSource messageSource;
#Override
protected ResponseEntity<Object> handleMissingServletRequestParameter(MissingServletRequestParameterException ex, HttpHeaders headers, HttpStatus status, WebRequest request) {
// you can do logs here and pass a custom object that will parse to json object
return buildResponseEntity(apiError);
}
#ExceptionHandler(YourCustomException.class)
protected ResponseEntity<Object> handleTthException(
YourCustomException ex, Locale locale) {
return buildResponseEntity(apiError);
}
private ResponseEntity<Object> buildResponseEntity(ApiError apiError) {
return new ResponseEntity<>(apiError, apiError.getStatus());
}
}

Based off of this: https://sysgears.com/articles/how-to-redirect-stdout-and-stderr-writing-to-a-log4j-appender/, I have found something I believe works.
At beginning of psvm:
public static void main(String[] args) {
SpringApplication.run(Main.class, args);
System.setErr(new PrintStream(new LoggingOutputStream(), true));
}
This says to print errors to my LoggingOutputStream.
Code for this class:
public class LoggingOutputStream extends OutputStream {
/**
* Default number of bytes in the buffer.
*/
private static final int DEFAULT_BUFFER_LENGTH = 2048;
/**
* Indicates stream state.
*/
private boolean hasBeenClosed = false;
/**
* Internal buffer where data is stored.
*/
private byte[] buf;
/**
* The number of valid bytes in the buffer.
*/
private int count;
/**
* Remembers the size of the buffer.
*/
private int curBufLength;
/**
* The logger to write to.
*/
private Logger log = LoggerFactory.getLogger(LoggingOutputStream.class);
public LoggingOutputStream(){
curBufLength = DEFAULT_BUFFER_LENGTH;
buf = new byte[curBufLength];
count = 0;
}
/**
*
* Writes the specified byte to this output stream.
*
* #param b the byte to write
* #throws IOException if an I/O error occurs.
*/
public void write(final int b) throws IOException {
if (hasBeenClosed) {
log.error("The stream has been closed.");
throw new IOException("The stream has been closed.");
}
// don't log nulls
if (b == 0) {
return;
}
// would this be writing past the buffer?
if (count == curBufLength) {
// grow the buffer
final int newBufLength = curBufLength +
DEFAULT_BUFFER_LENGTH;
final byte[] newBuf = new byte[newBufLength];
System.arraycopy(buf, 0, newBuf, 0, curBufLength);
buf = newBuf;
curBufLength = newBufLength;
}
buf[count] = (byte) b;
count++;
}
/**
* Flushes this output stream and forces any buffered output
* bytes to be written out.
*/
#Override
public void flush() {
if (count == 0) {
return;
}
final byte[] bytes = new byte[count];
System.arraycopy(buf, 0, bytes, 0, count);
String str = new String(bytes);
log.error(str);
count = 0;
}
/**
* Closes this output stream and releases any system resources
* associated with this stream.
*/
#Override
public void close() {
flush();
hasBeenClosed = true;
}
}
Now, all errors are "intercepted" and routed to my log file.

How do I write objects via a byte buffer?

I'm trying to:
Write an object (or a series of objects of different types/classes) into a file
Read them back
Check the instances and cast them into objects of their same type/class again
I could find these two classes, and this is how I use them. But the data[] array doesn't make much sense to me. Why do you have to put an empty array of data into the deserialize method?
public static byte[] serialize(Object obj) throws IOException {
ByteArrayOutputStream out = new ByteArrayOutputStream();
ObjectOutputStream os = new ObjectOutputStream(out);
os.writeObject(obj);
return out.toByteArray();
}
public static Object deserialize(byte[] data)
throws IOException, ClassNotFoundException {
ByteArrayInputStream in = new ByteArrayInputStream(data);
ObjectInputStream is = new ObjectInputStream(in);
return is.readObject();
}
public static void main(String[] args) {
try {
Thing p = new Thing(2,4);
byte[]data = new byte[10240];
serialize(p);
Object des = deserialize(data);
} catch (IOException | ClassNotFoundException ex) {
Logger.getLogger(Pruebiña.class.getName())
.log(Level.SEVERE, null, ex);
}
}
How can I fix this? Now I'm having the following error, when the program reaches the deserialize line:
java.io.StreamCorruptedException: invalid stream header: 00000000
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:806)
What can I do to fix this, and being able to write and read the objects back? And yes, the class Thing is Serializable.

If you want to write to a File you don't need the byte arrays at all use
FileInputStream and FileOutputStream eg.
public static void serialize(Object obj, File f) throws IOException {
try (ObjectOutputStream out = new ObjectOutputStream(new FileOutputStream(f))) {
out.writeObject(obj);
}
}
public static Object deserialize(File f)
throws IOException, ClassNotFoundException {
try (ObjectInputStream is = new ObjectInputStream(new FileInputStream(f))) {
return is.readObject();
}
}
static class Thing implements Serializable {
int a,b,c;
}
public static void main(String[] args) throws IOException, ClassNotFoundException {
File f = new File("object.dat");
Thing orig = new Thing();
serialize(orig, f);
Thing back = (Thing) deserialize(f);
}

You create the array in serialize, you don't need to create your own array.
Just do this:
byte[] data = serialize(p);
Instead of this:
byte[]data = new byte[10240];
serialize(p);

Count the bytes written to file via BufferedWriter formed by GZIPOutputStream

I have a BufferedWriter as shown below:
BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(
new GZIPOutputStream( hdfs.create(filepath, true ))));
String line = "text";
writer.write(line);
I want to find out the bytes written to the file with out querying file like
hdfs = FileSystem.get( new URI( "hdfs://localhost:8020" ), configuration );
filepath = new Path("path");
hdfs.getFileStatus(filepath).getLen();
as it will add overhead and I don't want that.
Also I cant do this:
line.getBytes().length;
As it give size before compression.

You can use the CountingOutputStream from Apache commons IO library.
Place it between the GZIPOutputStream and the file Outputstream (hdfs.create(..)).
After writing the content to the file you can read the number of written bytes from the CountingOutputStream instance.

If this isn't too late and you are using 1.7+ and you don't wan't to pull in an entire library like Guava or Commons-IO, you can just extend the GZIPOutputStream and obtain the data from the associated Deflater like so:
public class MyGZIPOutputStream extends GZIPOutputStream {
public MyGZIPOutputStream(OutputStream out) throws IOException {
super(out);
}
public long getBytesRead() {
return def.getBytesRead();
}
public long getBytesWritten() {
return def.getBytesWritten();
}
public void setLevel(int level) {
def.setLevel(level);
}
}

You can make you own descendant of OutputStream and count how many time write method was invoked

This is similar to the response by Olaseni, but I moved the counting into the BufferedOutputStream rather than the GZIPOutputStream, and this is more robust, since def.getBytesRead() in Olaseni's answer is not available after the stream has been closed.
With the implementation below, you can supply your own AtomicLong to the constructor so that you can assign the CountingBufferedOutputStream in a try-with-resources block, but still retrieve the count after the block has exited (i.e. after the file is closed).
public static class CountingBufferedOutputStream extends BufferedOutputStream {
private final AtomicLong bytesWritten;
public CountingBufferedOutputStream(OutputStream out) throws IOException {
super(out);
this.bytesWritten = new AtomicLong();
}
public CountingBufferedOutputStream(OutputStream out, int bufSize) throws IOException {
super(out, bufSize);
this.bytesWritten = new AtomicLong();
}
public CountingBufferedOutputStream(OutputStream out, int bufSize, AtomicLong bytesWritten)
throws IOException {
super(out, bufSize);
this.bytesWritten = bytesWritten;
}
#Override
public void write(byte[] b) throws IOException {
super.write(b);
bytesWritten.addAndGet(b.length);
}
#Override
public void write(byte[] b, int off, int len) throws IOException {
super.write(b, off, len);
bytesWritten.addAndGet(len);
}
#Override
public synchronized void write(int b) throws IOException {
super.write(b);
bytesWritten.incrementAndGet();
}
public long getBytesWritten() {
return bytesWritten.get();
}
}

how to convert PrintWriter to String or write to a File?

I am generating dynamic page using JSP, I want to save this dynamically generated complete page in file as archive.
In JSP, everything is written to PrintWriter out = response.getWriter();
At the end of page, before sending response to client I want to save this page, either in file or in buffer as string for later treatment.
How can I save Printwriter content or convert to String?

To get a string from the output of a PrintWriter, you can pass a StringWriter to a PrintWriter via the constructor:
#Test
public void writerTest(){
StringWriter out = new StringWriter();
PrintWriter writer = new PrintWriter(out);
// use writer, e.g.:
writer.print("ABC");
writer.print("DEF");
writer.flush(); // flush is really optional here, as Writer calls the empty StringWriter.flush
String result = out.toString();
assertEquals("ABCDEF", result);
}

Why not use StringWriter instead? I think this should be able to provide what you need.
So for example:
StringWriter strOut = new StringWriter();
...
String output = strOut.toString();
System.out.println(output);

It will depend on: how the PrintWriter is constructed and then used.
If the PrintWriter is constructed 1st and then passed to code that writes to it, you could use the Decorator pattern that allows you to create a sub-class of Writer, that takes the PrintWriter as a delegate, and forwards calls to the delegate, but also maintains a copy of the content that you can then archive.
public class DecoratedWriter extends Writer
{
private final Writer delegate;
private final StringWriter archive = new StringWriter();
//pass in the original PrintWriter here
public DecoratedWriter( Writer delegate )
{
this.delegate = delegate;
}
public String getForArchive()
{
return this.archive.toString();
}
public void write( char[] cbuf, int off, int len ) throws IOException
{
this.delegate.write( cbuf, off, len );
this.archive.write( cbuf, off, len );
}
public void flush() throws IOException
{
this.delegate.flush();
this.archive.flush();
}
public void close() throws IOException
{
this.delegate.close();
this.archive.close();
}
}

You cannot get it with just your PrintWriter object. It flushes the data, and does not hold any content within itself. This isn't the object you should be looking at to get the entire string,

The best way I think is prepare your response in other object like StringBuffer, and fush its content to the response, and after save the content stored in that variable to the file.

This helped me: for obtaining a SOAP-able object as XML string.
JAXBContext jc = JAXBContext.newInstance(o.getClass());
Marshaller m = jc.createMarshaller();
StringWriter writer = new StringWriter();
m.marshal( o, new PrintWriter(writer) );
return writer.toString();

Along similar lines to what cdc is doing - you can extend PrintWriter and then create and pass around an instance of this new class.
Call getArchive() to get a copy of the data that's passed through the writer.
public class ArchiveWriter extends PrintWriter {
private StringBuilder data = new StringBuilder();
public ArchiveWriter(Writer out) {
super(out);
}
public ArchiveWriter(Writer out, boolean autoFlush) {
super(out, autoFlush);
}
public ArchiveWriter(OutputStream out) {
super(out);
}
public ArchiveWriter(OutputStream out, boolean autoFlush) {
super(out, autoFlush);
}
public ArchiveWriter(String fileName) throws FileNotFoundException {
super(fileName);
}
public ArchiveWriter(String fileName, String csn) throws FileNotFoundException, UnsupportedEncodingException {
super(fileName, csn);
}
public ArchiveWriter(File file) throws FileNotFoundException {
super(file);
}
public ArchiveWriter(File file, String csn) throws FileNotFoundException, UnsupportedEncodingException {
super(file, csn);
}
#Override
public void write(char[] cbuf, int off, int len) {
super.write(cbuf, off,len);
data.append(cbuf, off, len);
}
#Override
public void write(String s, int off, int len) {
super.write(s, off,len);
data.append(s, off, len);
}
public String getArchive() {
return data.toString();
}
}

Record size of objects as they're being serialized?

What's the best way to record the size of certain objects as they are being serialized? For example, once objects of type A, B, C are serialized, record the size of their serialized bytes. We can get the size of the entire object graph via getBytes, but we'd like to break it down as to what are the largest contributors to the overall serialized size.
ObjectOutputStream offers writeObjectOverride, but we don't want to rewrite the serialization process. In simplified terms, we need to be aware of when we encounter a certain object prior to serialization, record the total current byte count, and then after it's serialized, take the difference of byte counts. It seems like encompassing writeSerialData would work, but the method is private.
Ideas?
Thanks.
--- UPDATE ---
The answers/suggestions below are insightful. Below is what I have so far. Let me know your thoughts. Thanks.
// extend to get a handle on outputstream
MyObjectOutputStream extends ObjectOutputStream {
private OutputStream out;
public MyObjectOutputStream(out) {
super(out);
this.out = out;
}
public OutputStream getOut() {
return this.out;
}
}
// counter
public static class CounterOutputStream extends FilterOutputStream {
private int bytesWritten = 0;
...
public int getBytesWritten() {
return this.bytesWritten;
}
public void resetCounter() {
bytesWritten = 0;
}
private void update(int len) {
bytesWritten += len;
}
}
// go serialize
ByteArrayOutputStream out = new ByteArrayOutputStream();
ObjectOutputStream oos = new MyObjectOutputStream(new CounterOutputStream(out, 1024));
// record serialized size of this class; do this for every interested class
public class MyInterestingObject {
...
private void writeObject(ObjectOutputStream out) throws IOException {
CounterOutputStream counter = null;
if (out instanceof MyObjectOutputStream) {
counter = (CounterOutputStream)((MyObjectOutputStream)out).getOut();
counter.resetCounter();
}
// continue w/ standard serialization of this object
out.defaultWriteObject();
if (counter != null) {
logger.info(this.getClass() + " bytes written: " + counter.getBytesWritten());
// TODO: store in context or somewhere to be aggregated post-serialization
}
}
}

The simplest solution would be to wrap the OutputStream you're using with an implementation that will count bytes written.
import java.io.IOException;
import java.io.OutputStream;
public class CountingOutputStream extends OutputStream {
private int count;
private OutputStream out;
public CountingOutputStream(OutputStream out) {
this.out = out;
}
public void write(byte[] b) throws IOException {
out.write(b);
count += b.length;
}
public void write(byte[] b, int off, int len) throws IOException {
out.write(b, off, len);
count += len;
}
public void flush() throws IOException {
out.flush();
}
public void close() throws IOException {
out.close();
}
public void write(int b) throws IOException {
out.write(b);
count++;
}
public int getBytesWritten() {
return count;
}
}
Then you would just use that
CountingOutputStream s = new CountingOutputStream(out);
ObjectOutputStream o = new ObjectOutputStream(s);
o.write(new Object());
o.close();
// s.getBytesWritten()

You could implement Externalizable rather than Serializable on any objects you need to capture such data from. You could then implement field-by-field byte counting in the writeExternal method, maybe by handing off to a utility class. Something like
public void writeExternal(ObjectOutput out) throws IOException
{
super.writeExternal(out);
out.writeUTF(this.myString == null ? "" : this.myString);
ByteCounter.getInstance().log("MyClass", "myString", this.myString);
}
Another hackish way would be to stick with Serializable, but to use the readResolve or writeReplace hooks to capture whatever data you need, e.g.
public class Test implements Serializable
{
private String s;
public Test(String s)
{
this.s = s;
}
private Object readResolve()
{
System.err.format("%s,%s,%s,%d\n", "readResolve", "Test", "s", s.length());
return this;
}
private Object writeReplace()
{
System.err.format("%s,%s,%s,%d\n", "writeReplace", "Test", "s", s.length());
return this;
}
public static void main(String[] args) throws Exception
{
File tmp = File.createTempFile("foo", "tmp");
ObjectOutputStream out = new ObjectOutputStream(new FileOutputStream(tmp));
Test test = new Test("hello world");
out.writeObject(test);
out.close();
ObjectInputStream in = new ObjectInputStream(new FileInputStream(tmp));
test = (Test)in.readObject();
in.close();
}
}