Spark application unable to read custom log4j.properties - java

I have been at this for several days now, my objective is simple.
I am setting up a SparkConf() object inside a Java Application and I need to specify a custom path to the log4j.properties file. The application is meant to run on a Spark Worker which has the custom log4j.properties file required.
It seems like my Spark configuration is unable to find this and is using the default file.
I have added the log4j.properties file in several places inside the worker pod like :/app/spark/conf/log4j.properties. But it doesn't seem to work.
Here's how I'm trying to set the custom path:
SparkConf sc = new SparkConf().setMaster(master)
.set("spark.driver.extraJavaOptions", "-Dlog4j.configuration=/app/spark/conf/log4j.properties")
.set("spark.executor.extraJavaOptions", "-Dlog4j.configuration=/app/spark/conf/log4j.properties")
The last two statements are currently having no effect on the Spark Configuration. Any idea what's wrong with this? Is something missing on my end?
Help...

Related

FileNotFoundException in SpringBoot when running in Docker Container

My Springboot Application is running fine in IDE but when I create fat jar file and run on docker it gives the error. I am connecting my application with firebase so i want to include the serviceAccountKey.json file from the resource folder. The application runs fine in my ide, but while deploying it over the docker container it gives the error of file not found. Though when include the file and print it path it doesn't give any sort of error . But when i give the file path to fileInputStream it produces the error. I have tried multiple ways but nothing seems to work. I'm including the file using classLoader.getResource("filename.json").
I tried to skip the inclusion of file and do it by saving files content in a string and then sending it to stream but this method crashes the server whenever i query firebase.
this is the code where error is occurring. Notice that I'm printing the file path and it gets printed in the output before showing error. I have also tried file.getAbsoluteFile instead of path but doesn't work. Probably I'm doing it wrong or probably i have to mention the path in some other place as well which i don't know about. If anyone has done this before then please help me on this.
File path is getting printed but FileStream can't get it
An alternative to having a credential in a json file that is packaged with the application, would be that you set an environment variable upon starting the application, and instead load the key via application.yml. This way, you don't need to package the secrets into the application jar file.
1. create config class
#Configuration
#ConfigurationProperties(value = "my-custom-config")
public class MyConfig {
private String serviceAccountKey;
// getters/setters etc. if not using lombok
}
2. use above config
in other spring beans by injecting the config class, and retrieve the secret with getter, e.g. config.getServiceAccountKey();
3. Add config to your application.yml file
# application.yml
my-custom-config:
serviceAccountKey: ${ENV_VARIABLE_NAME} # <---- this way you can bind an env variable to your config.
4. Define env variable in container on startup
On instantiating the docker container, provide env option and define an environment variable.
docker run -env ENV_VARIABLE_NAME=<value> ...

Apache Commons Daemon specify classpath

I try to deploy a program as windows service with apache commons daemon. i have everything set up fine beside the classpath.
Classpath=C:\test\test-service\lib\*
specifies where the service itself lies, but the service needs some configuration files to run and these are under
Classpath=C:\test\test-service\conf\*
now i struggle to get it to work so that the program uses C:\test\test-service\* as classpath and not just one of the both specified above. sadly C:\test\test-service\* does not work and throws a ClassDefNotFoundException for the starter class. that error is solved by C:\test\test-service\lib\* but then i get the error that my config files can't be read. therefore i thought, why not also add the config path to the classpath like this:
Classpath=C:\test\test-service\lib\*;C:\test\test-service\conf\*
but this still throws the FileNotFoundException. does anyone has a solution to this?
So I was able to solve this by myself. Following works:
Classpath= C:\test\test-service\conf\;C:\test\test-service\lib\*
it takes all files in conf folder by default as well as all files from lib as the * states

Heroku Unable To Find XML Config File

I uploaded a Spring application to Heroku but the application crashed with the following error:
java.io.FileNotFoundException: class path resource [com/myname/myapp/config/dao-context.xml
The file is definitely there, and it is in GIT, and the app runs successfully locally.
Any ideas what is happening here?
I suspect that when you are running locally, it is picking up the file on the classpath as a regular file on the filesystem (i.e. not inside of a JAR).
On Heroku, it is probably inside of a JAR file, which means it is not a regular file, and must be read as an input stream, which might look like this:
ClassLoader cl = this.getClass().getClassLoader();
InputStream inputStream = cl.getResourceAsStream("com/myname/myapp/config/dao-context.xml");
You can probably reproduce the problem locally by running the same command that's in your Procfile.
If this is not the case, then make sure the file exists on Heroku by running this command:
$ heroku run ls com/myname/myapp/config/dao-context.xml
For future visitors to this question, I overcame the problem by converting my DAO XML config file to the Java Config method, therefore Spring no longer required that XML file. This didn't directly solve the issue of being unable to find the XML file, but the added benefit is that I am now using the more modern and less verbose Java Config method.

Java spring parametric .xml config file

I have a Spring Java 1.8 project exported in a .jar. When I want to run this Java .jar application, I'd like to give a file into the argument which .xml config file should the program use in the relative folder. I tried so many ways, but none of them works.
For example:
sampleJavaPorgram.jar config1.xml
(config1.xml and sampleJavaProgram.jar is in the same directory)
shoud load config1.xml like that:
ApplicationContext context = new ClassPathXmlApplicationContext("config1.xml");
Maybe use Command-Line Argument, to get name of this XML file and then pass it to AppContext.
Or you might use System properties

Spring Boot: Unable to read resources in lib/*.jar using a class in lib/*.jar

Iam using spring boot org.springframework.boot.loader.JarLauncher to run my self executable spring boot app jar.
Now, I have my self executable jar packed like this (For brevity Iam just adding the files only that are needed to show the problem):
Main.jar
---META-INF
---lib
---a.jar
---com
---comp
---Abc.class
---folder1
---1.txt
---2.txt
---b.jar
and so on.
In my Abc.class Iam trying to load the resource 1.txt which is working fine when I run it in eclipse; but the story starts when I run using command line as self executable jar. I was not able to read that resource and ends up with null pointer error.
After reading few threads, I learnt that my IDEs does some magic on Class Loaders and so it was working fine, which will not be the case when I run it as executable jar
This is how I was Loading the files and the all the possible options I have tried to no luck.
Option 1:
Abc.class.getResourceAsStream("\folder1\1.txt")
Option 2:
Abc.class.getClassLoader().getResourceAsStream("folder1\1.txt")
Option 3: Reading thread, I have tried considered the current thread class loader context as below too, But to no luck again
Thread.currentThread().getContextClassLoader().getResourceAsStream("folder1\1.txt")
Can any one advise. How should I write my code to be able to read my resource that is in the jar and by the class that is in the same jar ?
PS: Spring boot is not adding class-path entry to MANIFEST.MF and if I have to do something around that, how do I do that ? In-fact I tried -cp when running my jar setting it to current directory, but that have not worked either
In Spring, the best way to access a Resource is via Resource APIs. For a Classpath resource what you should use a ClassPathResource and it would look something like this:
Resource resource = new ClassPathResource("/my/resource.json", this.getClass());
After obtaining a Resource you can either get a File reference via getFile() or get an InputStream straight off by calling getInputStream().
Spring provides quite a few different implementations of the Resource interface, take a look at the list of known implementations in the docs
Use Spring's class ClassPathResource to load file from classpath.
For example you can read file into String like this from classpath:
String fileContent = FileUtils.readFileToString(
new ClassPathResource("folder1" + File.separator + "1.txt").getFile());
It doesn't matter what kind of build tool or project structure you are using as long as that file is on classpath.

Categories

Resources