Develop Reference

How can I recreate a chained OutputStream with only filename modified

I have got an OutputStream which can be initialized as a chain of OutputStreams. There could be any level of chaining .Only thing guaranteed is that at the end of the chain is a FileOutputStream.
I need to recreate this chained outputStream with a modified Filename in FileOutputStream. This would have been possible if out variable (which stores the underlying chained outputStream) was accessible ; as shown below.
public OutputStream recreateChainedOutputStream(OutputStream os) throws IOException {
if(os instanceof FileOutputStream) {
return new FileOutputStream("somemodified.filename");
} else if (os instanceof FilterOutputStream) {
return recreateChainedOutputStream(os.out);
}
}
Is there any other way of achieving the same?

You can use reflection to access the os.out field of the FilterOutputStream, this has however some drawbacks:
If the other OutputStream is also a kind of RolloverOutputStream, you can have a hard time reconstructing it,
If the other OutputStream has custom settings, like GZip compression parameter, you cannot reliable read this
If there is a
A quick and dirty implementation of recreateChainedOutputStream( might be:
private final static Field out;
{
try {
out = FilterInputStream.class.getField("out");
out.setAccessible(true);
} catch(Exception e) {
throw new RuntimeException(e);
}
}
public OutputStream recreateChainedOutputStream(OutputStream out) throws IOException {
if (out instanceof FilterOutputStream) {
Class<?> c = ou.getClass();
COnstructor<?> con = c.getConstructor(OutputStream.class);
return con.invoke(this.out.get(out));
} else {
// Other output streams...
}
}
While this may be ok in your current application, this is a big no-no in the production world because the large amount of different kind of OutputStreams your application may recieve.
A better way to solve would be a kind of Function<String, OutputStream> that works as a factory to create OutputStreams for the named file. This way the external api keeps its control over the OutputStreams while your api can adress multiple file names. An example of this would be:
public class MyApi {
private final Function<String, OutputStream> fileProvider;
private OutputStream current;
public MyApi (Function<String, OutputStream> fileProvider, String defaultFile) {
this.fileProvider = fileProvider;
selectNewOutputFile(defaultFile);
}
public void selectNewOutputFile(String name) {
OutputStream current = this.current;
this.current = fileProvider.apply(name);
if(current != null) current.close();
}
}
This can then be used in other applications as:
MyApi api = new MyApi(name->new FileOutputStream(name));
For simple FileOutputStreams, or be used as:
MyApi api = new MyApi(name->
new GZIPOutputStream(
new CipherOutputStream(
new CheckedOutputStream(
new FileOutputStream(name),
new CRC32()),
chipper),
1024,
true)
);
For a file stream that stored checksummed using new CRC32(), chipped using chipper, gzip according to a 1024 buffer with sync write mode.

What is the difference between Spark Serialization and Java Serialization?

I'm using Spark + Yarn and I have a service that I want to call on distributed nodes.
When I serialize this service object "by hand" in a Junit test using java serialization, all inner collections of the service are well serialized and deserialized :
#Test
public void testSerialization() {
try (
ConfigurableApplicationContext contextBusiness = new ClassPathXmlApplicationContext("spring-context.xml");
FileOutputStream fileOutputStream = new FileOutputStream("myService.ser");
ObjectOutputStream objectOutputStream = new ObjectOutputStream(fileOutputStream);
) {
final MyService service = (MyService) contextBusiness.getBean("myServiceImpl");
objectOutputStream.writeObject(service);
objectOutputStream.flush();
} catch (final java.io.IOException e) {
logger.error(e.getMessage(), e);
}
}
#Test
public void testDeSerialization() throws ClassNotFoundException {
try (
FileInputStream fileInputStream = new FileInputStream("myService.ser");
ObjectInputStream objectInputStream = new ObjectInputStream(fileInputStream);
) {
final MyService myService = (MyService) objectInputStream.readObject();
// HERE a functionnal test who proves the service has been fully serialized and deserialized .
} catch (final java.io.IOException e) {
logger.error(e.getMessage(), e);
}
}
But when I try to call this service via my Spark launcher, wether I broadcast the service object or not, some inner collection (a HashMap) disappears (is not serialized) like if it was tagged as "transient" (but it's not transient neither static) :
JavaRDD<InputOjbect> listeInputsRDD = sprkCtx.parallelize(listeInputs, 10);
JavaRDD<OutputObject> listeOutputsRDD = listeInputsRDD.map(new Function<InputOjbect, OutputObject>() {
private static final long serialVersionUID = 1L;
public OutputObject call(InputOjbect input) throws TarificationXmlException { // Exception
MyOutput output = service.evaluate(input);
return (new OutputObject(output));
}
});
same result if I broadcast the service :
final Broadcast<MyService> broadcastedService = sprkCtx.broadcast(service);
JavaRDD<InputOjbect> listeInputsRDD = sprkCtx.parallelize(listeInputs, 10);
JavaRDD<OutputObject> listeOutputsRDD = listeInputsRDD.map(new Function<InputOjbect, OutputObject>() {
private static final long serialVersionUID = 1L;
public OutputObject call(InputOjbect input) throws TarificationXmlException { // Exception
MyOutput output = broadcastedService.getValue().evaluate(input);
return (new OutputObject(output));
}
});
If I launch this same Spark code in local mode instead of yarn cluster mode, it works perfectly.
So my question is : What is the difference between Spark Serialization and Java Serialization ? (I'm not using Kryo or any customized serialization).
EDIT : when I try with Kryo serializer (without registering explicitly any class), I have the same problem.

Ok, I've found it out thanks to one of our experimented data analyst.
So, what was this mystery about ?
It was NOT about serialization (java or Kryo)
It was NOT about some pre-treatment or post-treatment Spark would do before/after serialization
It was NOT about the HashMap field which is fully serializable (this one is obvious if u read the first example I give, but not for everyone ;)
So...
The whole problem was about this :
"if I launch this same Spark code in local mode instead of yarn cluster
mode, it works perfectly."
In "yarn cluster" mode the collection was unable to be initialized, cause it was launched on a random node and couldn't access to, the initial reference datas on disk. In local mode, there was a clear exception when the initial datas where not found on disk, but in cluster mode it was fully silent and it looked like the problem was about serialization.
Using "yarn client" mode solved this for us.

Add txt files to a runnable JAR file

I'm trying to make a runnable jar file and I'm having problems with my .txt files.
My program also have images, but fortunately I've figured out how to manage them. I'm using something like this with them and it works just fine both Eclipse and the jar:
logoLabel.setIcon(new ImageIcon(getClass().getResource("/logo.png")));
My problem is when I've something like this in one of my classes:
try {
employeeList = (TreeSet<Employee>) ListManager.readFile("list/employeeList.txt");
} catch (ClassNotFoundException i) {
i.printStackTrace();
} catch (IOException i) {
i.printStackTrace();
}
And this in the class ListManager that I use to read my lists serialized in the .txt files:
public static Object readFile(String file) throws IOException, ClassNotFoundException {
ObjectInputStream is = new ObjectInputStream(new FileInputStream(file));
Object o = is.readObject();
is.close();
return o;
}
I also have a similar method to write in the files.
I've tried several combinations that I've found here:
How to include text files with Executable Jar
Creating Runnable Jar with external files included
Including a text file inside a jar file and reading it
I've also tried with slash, without slash, using openStream, not using openStream... But or I get a NullPointerException or it doesn't compile at all...
Maybe is something silly or maybe is a concept error that I've of how URL class works, I'm new to programming...
Thank you very much in advance for your advice!
EDIT:
It's me again... The answer Raniz gave was just what I needed and it worked perfect, but now my problem is with the method that I use to write in the files...
public static void writeFile(Object o, String file) throws IOException {
ObjectOutputStream os = new ObjectOutputStream(new FileOutputStream(file));
os.writeObject(o);
os.close();
}
try {
ListManager.writeFile(employeeList.getEmployeeList(), "lists/employeeList.txt");
} catch (IOException i) {
i.printStackTrace();
}
Could you help me please? I don't know what I should use to replace FileOutputStream, because I think there is the problem again, am I right?
Thank you very much!

The problem is that you're trying to access a file inside of a JAR archive as a file in the file system (because that's what FileInputStream is for) and that won't work.
You can convert readFile to use an URL instead and let URL handle opening the stream for you:
public static Object readFile(URL url) throws IOException, ClassNotFoundException {
ObjectInputStream is = new ObjectInputStream(url.openStream());
Object o = is.readObject();
is.close();
return o;
}
You should also put your code in a try-statement since it currently doesn't close the streams if an IOException occurs:
public static Object readFile(URL url) throws IOException, ClassNotFoundException {
try(ObjectInputStream is = new ObjectInputStream(url.openStream())) {
Object o = is.readObject();
return o;
}
}
try {
employeeList = (TreeSet<Employee>) ListManager.readFile(getClass().getResource("/list/employeeList.txt"));
} catch (ClassNotFoundException i) {
i.printStackTrace();
} catch (IOException i) {
i.printStackTrace();
}
I also have a similar method to write in the files.
That won't work if the files are inside the JAR so you should probably consider having your files outside your JAR.

Yes, if you want to read resources from inside a jar file, you shouldn't use FileInputStream. Perhaps you should add a readResource method:
public static Object readResource(Class clazz, String resource)
throws IOException, ClassNotFoundException {
try (ObjectInputStream is =
new ObjectInputStream(clazz.getResourceAsStream(resource))) {
return is.readObject();
}
}
(I'd also suggest updating your readFile method to use a try-with-resources block - currently if there's an exception you won't close the stream...)
Note that when you say "I also have a similar method to write in the files" - you won't be able to easily write to a resource in the jar file.

GroovyClassLoading Mechanism

I'm really newbie to groovy scripting but following some tutorial I tried to dynamically load some groovy class within my java code using parseClass() method of GroovyClassLoader.
I wrote some snippet and it worked fine for me. The problem is that I don't clearly understand what groovy engine is doing beyond my view and how those scripts are compiled?
Does a new class gets creted and loaded into jvm? Or does my application uses some cached sources?
Here is the class I'm trying to parse:
private static class MyScript {
#Override
public String toString()
{
StringBuilder builder = new StringBuilder();
builder.append("public class SomeClass\n");
builder.append("{\n");
builder.append("Some code...").append("\n");
builder.append("}\n");
return builder.toString();
}
The I load it with build() as below:
private Class MyGroovyBuilder {
private Script script = new Script();
public String build() throws TemplateCompilationException
//
String groovyText = script.toString();
//
CompilerConfiguration config = new CompilerConfiguration();
//
byte[] bytes;
try
{
bytes = groovyText.getBytes(config.getSourceEncoding());
}
catch (UnsupportedEncodingException e)
{
throw new TemplateCompilationException(e, groovyText);
}
//
InputStream in = new ByteArrayInputStream(bytes);
GroovyCodeSource gcs = new GroovyCodeSource(in, "SomeName", "/groovy/shell");
GroovyClassLoader loader = new
GroovyClassLoader(Thread.currentThread().getContextClassLoader(), config);
Class<?> scriptClass;
try
{
scriptClass = loader.parseClass(gcs, false);
}
catch (CompilationFailedException e)
{
throw new GroovyCompilationException(e, "SomeName", groovyText);
}
catch (ClassFormatError e)
{
throw new GroovyCompilationException(e, "SomeName", groovyText);
}
return scriptClass.getName();
}
Any clarification is greatelly appreciated.
BR.

After loading class it appears in your class loader, and can be accessed like any other class.
There is a simple tutorial [here], that show how to load class from string.
In simplest case, you can load class, and hold it's Class object, using it to create objects dynamically. For field access or method invokation you can rely on Groovy dynamic nature.
There is no "cached source" or smth like that behind the scene and you can forget, from where your class is loaded. You can also cache classes, that are already compiled, and save them somewhere, as described [here]. It will drastically improve performance, if you need to load same class often.
But it will be better, to dig down in topic, because dynamic class loading is advanced Java/Groovy technique, it's whole infrastructure of chained classloaders, so it's better to refer documentation about them.
Links below may be helpful.
http://javarevisited.blogspot.ru/2012/12/how-classloader-works-in-java.html
How to use URLClassLoader to load a *.class file?

How to reload previously loaded classes that changed during runtime?

I'm writing an application that needs to reload a previously loaded class during runtime. The reason is that the class is auto-generated during runtime, and the new implementation changes the way the app works. I generate only one object of the said class, and I stripped it from all dependencies but an interface that defines constant values. There's no problem reseting the values of any or all of the members when reloading. I know exactly when it changes and I can control it. The only problem I have is the reload itself.
From what I read, I should use a ClassLoader. I tried to do so, but I can't make it work.
I tried the following:
Getting the current ClassLoader (myClassObject.getClass().getClassLoader()) and using it to reload the class - Doesn't work. It probably keeps loading the old implementation.
Generating my own (AKA copy-paste from SO with modifications) - Doesn't work because the ClassLoader I generate is different than the one that generated the class (Exception: myClass cannot be casted to myClass).
Creating a constructor that sets the ClassLoader of the superclass doesn't seem to have any effect.
Using my new ClassLoader to generate the class that has myClassObject as a member solved the ClassLoader mismatch for myClassObject, but created a new mismatch one level up. I used getClassLoader() everytime and I see they don't match.
I tried adding -Djava.system.class.loader=com.test.Reoader com.test.myMainClass to make it my default reloader, but I get an error from the compiler.
Google keeps pointing me back to the same stuff I read already.
EDIT: I tried creating an interface and reload the class implementing it. That didn't solve it either.
I know I should override the default ClassLoader, but nothing I do seems to succeed at that.
My ClassLoader:
public class Reloader extends ClassLoader {
public Reloader(){
super(Reloader.class.getClassLoader());
}
#Override
public Class<?> loadClass(String s) {
return findClass(s);
}
#Override
public Class<?> findClass(String s) {
try {
byte[] bytes = loadClassData(s);
return defineClass(s, bytes, 0, bytes.length);
} catch (IOException ioe) {
try {
return super.loadClass(s);
} catch (ClassNotFoundException ignore) {}
ioe.printStackTrace(java.lang.System.out);
return null;
}
}
private byte[] loadClassData(String className) throws IOException {
File f = new File("out\\production\\ManoCPU\\" + className.replaceAll("\\.", "/") + ".class");
int size = (int) f.length();
byte buff[] = new byte[size];
FileInputStream fis = new FileInputStream(f);
DataInputStream dis = new DataInputStream(fis);
dis.readFully(buff);
dis.close();
return buff;
}
}
Thanks very much to anyone that can help.

You can only load a class once (per instance of a classloader). That means you have to throw away the classloader you have loaded your class with and instantiate a new one for your updated version of the class.
When dealing with multiple class loaders also have in mind that if you load the same class with several classloaders they are NOT recognized as being the same class.