Posting this self-answered so when I break it again in a few months an answer will actually come up in google.
Simple Java project where Flyway API is used to migrate database schema. A resource directory of sql scripts following the V1__init.sql, V2__updateCustomerTable.sql convention is used to check against the schema_version metadata table and migrate if needed.
Everything works fine until the jar is proguarded. The sql scripts are definitely packed into the jar file, but they are not found:
2017-04-10 17:17:13,612 [main] DEBUG (?:?) - Validating migrations ...
2017-04-10 17:17:13,884 [main] DEBUG (?:?) - Scanning for classpath resources at 'classpath:db/migration/postgres' (Prefix: '', Suffix: '.sql')
2017-04-10 17:17:13,885 [main] DEBUG (?:?) - Determining location urls for classpath:db/migration/postgres using ClassLoader sun.misc.Launcher$AppClassLoader#8b819f ...
2017-04-10 17:17:13,885 [main] WARN (?:?) - Unable to resolve location classpath:db/migration/postgres
2017-04-10 17:17:13,899 [main] DEBUG (?:?) - Scanning for classpath resources at 'classpath:db/migration/postgres' (Prefix: 'V', Suffix: '.sql')
2017-04-10 17:17:13,900 [main] DEBUG (?:?) - Scanning for classpath resources at 'classpath:db/migration/postgres' (Prefix: 'R', Suffix: '.sql')
An un-proguarded jar finds them with the following log messages (note that it looks in the jar file now):
2017-04-10 17:07:16,612 [main] DEBUG (?:?) - Validating migrations ...
2017-04-10 17:07:16,613 [main] DEBUG (?:?) - Scanning for classpath resources at 'classpath:db/migration/postgres' (Prefix: 'V', Suffix: '.sql')
2017-04-10 17:07:16,614 [main] DEBUG (?:?) - Scanning URL: jar:file:/C:/Dev/Sanbox/myjar.jar!/db/migration/postgres
2017-04-10 17:07:16,614 [main] DEBUG (?:?) - JBoss VFS v2 available: false
2017-04-10 17:07:16,615 [main] DEBUG (?:?) - Filtering out resource: db/migration/postgres/ (filename: )
2017-04-10 17:07:16,615 [main] DEBUG (?:?) - Found resource: db/migration/postgres/V1__init.sql
2017-04-10 17:07:16,616 [main] DEBUG (?:?) - Found resource: db/migration/postgres/V2__updateCustomerTable.sql
The culprit was that you need to tell proguard to leave your db/migrations directory alone (or wherever you put your migration scripts) in your config file, else flyway will not be able to find your scripts even if they seem to be in the right place in the jar. This is the line I needed in my proguard config:
-keepdirectories db/migration/**
If you change where you put your sql scripts down the road, this config will also need to be updated.
Related
I have a dataflow running in NiFi 1.12.0, the relevant properties from this installation is here:
nifi.sensitive.props.key=
nifi.sensitive.props.key.protected=
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
nifi.sensitive.props.provider=BC
nifi.sensitive.props.additional.keys=
I am facing migration issue in nifi when I upgrade base version to 1.16.3 from 1.12.0. which having following properties.
nifi.sensitive.props.key=testPassword
nifi.sensitive.props.key.protected=
nifi.sensitive.props.algorithm=NIFI_ARGON2_AES_GCM_256
nifi.sensitive.props.provider=BC
nifi.sensitive.props.additional.keys=
I am getting following exception while I execute command for migrating flow file.
nifi#nifi-service-0:/opt/nifi/nifi-toolkit-current/bin$ ./encrypt-config.sh -n $NIFI_HOME/conf/nifi.properties -f /opt/nifi/data/old_flow.xml.gz -s testPassword -x
[main] WARN org.apache.nifi.properties.ConfigEncryptionTool - The source nifi.properties and destination nifi.properties are identical [/opt/nifi/nifi-current/conf/nifi.properties] so the original will be overwritten
[main] WARN org.apache.nifi.properties.ConfigEncryptionTool - The source flow.xml.gz and destination flow.xml.gz are identical [/opt/nifi/data/old_flow.xml.gz] so the original will be overwritten
[main] WARN org.apache.nifi.properties.AbstractBootstrapPropertiesLoader - System Property [nifi.properties.file.path] not found: Using Relative Path [conf/nifi.properties]
[main] INFO org.apache.nifi.properties.NiFiPropertiesLoader - Loading Application Properties [/opt/nifi/nifi-current/conf/nifi.properties]
[main] INFO org.apache.nifi.properties.NiFiPropertiesLoader - Loading Application Properties [/opt/nifi/nifi-current/conf/nifi.properties]
[main] INFO org.apache.nifi.properties.ConfigEncryptionTool - Loaded NiFiProperties instance with 138 properties
[main] INFO org.apache.nifi.properties.NiFiPropertiesLoader - Loading Application Properties [/opt/nifi/nifi-current/conf/nifi.properties]
[main] INFO org.apache.nifi.properties.ConfigEncryptionTool - Migrating flow.xml file at /opt/nifi/data/old_flow.xml.gz. This could take a while if the flow XML is very large.
[main] ERROR org.apache.nifi.properties.ConfigEncryptionTool - Encountered an error: Decryption Failed with Algorithm [AES/GCM/NoPadding]
Encountered an error migrating flow content
usage: org.apache.nifi.properties.ConfigEncryptionTool [-h] [-v] [-n <file>] [-o <file>] [-l <file>] [-i <file>] [-a <file>] [-u <file>] [-f <file>] [-g <file>]
[-b <file>] [-S <protectionScheme>] [-k <keyhex>] [-e <keyhex>] [-H <protectionScheme>] [-p <password>] [-w <password>] [-r] [-m] [-x] [-s
<password|keyhex>] [-A <algorithm>] [-P <algorithm>] [-c]
This tool reads from a nifi.properties and/or login-identity-providers.xml file with plain sensitive configuration values, prompts the user for a root key, and
encrypts each value. It will replace the plain value with the protected value in the same file (or write to a new file if specified). It can also be used to
migrate already-encrypted values in those files or in flow.xml.gz to be encrypted with a new key.
Please help to solve this issue
Solved by myself.
Issue can be resolved by following steps
Before migration if you don't have nifi.sensitive.props.key set, set it using following command ${NIFI_TOOLKIT_PAT}/bin/encrypt-config.sh -f /opt/nifi/nifi-current/data/flow.xml.gz -p ${NIFI_HOME}/conf/nifi.properties -s <NEW_KEY_TO_SET> -x
Once key is set upgrade nifi. Since in newer version algorithm is changed set it using command ${NIFI_HOME}/bin/nifi.sh set-sensitive-properties-algorithm <NEW_ALGORITHM>
Once algorithm set, encrypt again using command ${NIFI_TOOLKIT_PAT}/bin/encrypt-config.sh -f /opt/nifi/nifi-current/data/flow.xml.gz -p ${NIFI_HOME}/conf/nifi.properties -s <NEW_KEY_TO_SET> -x
Now you will get all compatible files with respect your latest version
I have a spring boot rest application on azure app service. Anytime I deploy an application with azure DevOps pipeline or with azure-webapp-maven-plugin directly, the deployment works successfully as I can see all the uploaded files in site/wwwroot/webapps/ROOT from the Kudu Debug Console. When I try to hit any endpoint or even the application URL, it returns a 404(Not Found error).
I pulled up the logs and this is what I find.
[DEBUG] 2020-05-24 10:45:31,644 org.springframework.core.env.PropertySourcesPropertyResolver logKeyFound - - Found key 'spring.liveBeansView.mbeanDomain' in PropertySource 'servletContextInitParams' with value of type String
[INFO] 2020-05-24 10:45:31,660 com.company.product.core.MainApplication logStarted - - Started MainApplication in 37.424 seconds (JVM running for 65.143)
[DEBUG] 2020-05-24 10:45:32,347 org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext doClose - - Closing org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext#60d5a711, started on Sun May 24 10:44:58 GMT 2020
[DEBUG] 2020-05-24 10:45:32,378 org.springframework.core.env.PropertySourcesPropertyResolver logKeyFound - - Found key 'spring.liveBeansView.mbeanDomain' in PropertySource 'servletContextInitParams' with value of type String
[DEBUG] 2020-05-24 10:45:32,456 org.springframework.context.support.DefaultLifecycleProcessor stop - - Stopping beans in phase 2147483647
[DEBUG] 2020-05-24 10:45:32,456 org.springframework.context.support.DefaultLifecycleProcessor lambda$doStop$2 - - Bean 'documentationPluginsBootstrapper' completed its stop procedure
[DEBUG] 2020-05-24 10:45:32,456 org.springframework.jmx.export.annotation.AnnotationMBeanExporter destroy - - Unregistering JMX-exposed beans on shutdown
[DEBUG] 2020-05-24 10:45:32,456 org.springframework.jmx.export.annotation.AnnotationMBeanExporter unregisterBeans - - Unregistering JMX-exposed beans
[INFO] 2020-05-24 10:45:32,456 org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor shutdown - - Shutting down ExecutorService 'applicationTaskExecutor'
The application shuts down right after it's initialized. It looks to me as if the app service is sending a shutdown command to the spring boot application.
Can anybody help with this? I'm pressed on time. Thanks in advance.
Usually we deploy spring boot app to azure web app by using jar file. Here are the steps:
1.Go to your webapp in the portal->Configuration->check the Java container. We will use Java SE instead of Tomcat.
2.Go to your webapp in the portal->Advanced Tools->Debug console->Cmd. Put your jar file under wwwroot folder. Need to name it as app.jar.
3.The result
Could one please help me figure out the reason why my application written in Kotlinfails to run after it was built with Intellij Idea and deployed to Ubuntu 19.10 virtual machine ?
I use the following components:
Ktor 1.3.0
Netty for Ktor
Launch logs:
15:22:08.488 [main] DEBUG io.netty.util.internal.logging.InternalLoggerFactory - Using SLF4J as the default logging framework
15:22:08.530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - -Dio.netty.noUnsafe: false
15:22:08.530 [main] DEBUG io.netty.util.internal.PlatformDependent0 - Java version: 11
15:22:08.568 [main] DEBUG io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
15:22:08.569 [main] DEBUG io.netty.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
15:22:08.569 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
15:22:08.577 [main] DEBUG io.netty.util.internal.PlatformDependent0 - direct buffer constructor: unavailable
java.lang.UnsupportedOperationException: Reflective setAccessible(true) disabled
at io.netty.util.internal.ReflectionUtil.trySetAccessible(ReflectionUtil.java:31)
at io.netty.util.internal.PlatformDependent0$4.run(PlatformDependent0.java:225)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.PlatformDependent0.<clinit>(PlatformDependent0.java:219)
at io.netty.util.internal.PlatformDependent.isAndroid(PlatformDependent.java:273)
at io.netty.util.internal.PlatformDependent.<clinit>(PlatformDependent.java:92)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:225)
at io.netty.channel.epoll.Native.<clinit>(Native.java:57)
at io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:39)
at io.ktor.server.netty.EventLoopGroupProxy$Companion.create(NettyApplicationEngine.kt:189)
at io.ktor.server.netty.NettyApplicationEngine.<init>(NettyApplicationEngine.kt:74)
at io.ktor.server.netty.Netty.create(Embedded.kt:14)
at io.ktor.server.netty.Netty.create(Embedded.kt:12)
at io.ktor.server.engine.EmbeddedServerKt.embeddedServer(EmbeddedServer.kt:79)
at io.ktor.server.engine.EmbeddedServerKt.embeddedServer$default(EmbeddedServer.kt:77)
at app.common.ApplicationKt.main(Application.kt:96)
15:22:08.581 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: available, true
15:22:08.582 [main] DEBUG io.netty.util.internal.PlatformDependent0 - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable
java.lang.IllegalAccessException: class io.netty.util.internal.PlatformDependent0$6 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module #6d3af739
at java.base/jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:361)
at java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:591)
at java.base/java.lang.reflect.Method.invoke(Method.java:558)
at io.netty.util.internal.PlatformDependent0$6.run(PlatformDependent0.java:335)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.PlatformDependent0.<clinit>(PlatformDependent0.java:326)
at io.netty.util.internal.PlatformDependent.isAndroid(PlatformDependent.java:273)
at io.netty.util.internal.PlatformDependent.<clinit>(PlatformDependent.java:92)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:225)
at io.netty.channel.epoll.Native.<clinit>(Native.java:57)
at io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:39)
at io.ktor.server.netty.EventLoopGroupProxy$Companion.create(NettyApplicationEngine.kt:189)
at io.ktor.server.netty.NettyApplicationEngine.<init>(NettyApplicationEngine.kt:74)
at io.ktor.server.netty.Netty.create(Embedded.kt:14)
at io.ktor.server.netty.Netty.create(Embedded.kt:12)
at io.ktor.server.engine.EmbeddedServerKt.embeddedServer(EmbeddedServer.kt:79)
at io.ktor.server.engine.EmbeddedServerKt.embeddedServer$default(EmbeddedServer.kt:77)
at app.common.ApplicationKt.main(Application.kt:96)
15:22:08.586 [main] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.DirectByteBuffer.<init>(long, int): unavailable
15:22:08.587 [main] DEBUG io.netty.util.internal.PlatformDependent - sun.misc.Unsafe: available
15:22:08.710 [main] DEBUG io.netty.util.internal.PlatformDependent - maxDirectMemory: 249364480 bytes (maybe)
15:22:08.711 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
15:22:08.711 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
15:22:08.713 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.maxDirectMemory: -1 bytes
15:22:08.713 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.uninitializedArrayAllocationThreshold: -1
15:22:08.714 [main] DEBUG io.netty.util.internal.CleanerJava9 - java.nio.ByteBuffer.cleaner(): available
15:22:08.714 [main] DEBUG io.netty.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
15:22:08.716 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - -Dio.netty.native.workdir: /tmp (io.netty.tmpdir)
15:22:08.717 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - -Dio.netty.native.deleteLibAfterLoading: true
15:22:08.717 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - -Dio.netty.native.tryPatchShadedId: true
15:22:08.718 [main] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_transport_native_epoll_x86_64', trying other loading mechanism.
I'm reading jdbc.properties file using properties placeholder as below.
<context:property-placeholder location="classpath*:jdbc.properties" />
In the properties file I'm storing jdbc configuration along with password. When I run the springframework in debug mode. It's printing the properties value which is password. Is there any solution to fix this? Here is my log statement.
2014-05-20 18:37:32,076 [main] DEBUG o.s.c.e.PropertySourcesPropertyResolver - Could not find key 'database.password' in any property source. Returning [null]
2014-05-20 18:37:32,076 [main] DEBUG o.s.c.e.PropertySourcesPropertyResolver - Searching for key 'database.password' in [localProperties]
2014-05-20 18:37:32,076 [main] DEBUG o.s.c.e.PropertySourcesPropertyResolver - Found key 'database.password' in [localProperties] with type [String] and value 'pass'
2014-05-20 18:37:32,076 [main] TRACE o.s.util.PropertyPlaceholderHelper - Resolved placeholder 'database.password'
This an old post and I'm not sure if you just want to remove them from the log selectively but for a simple answer I use the following in application.properties.
logging.level.root=error
logging.level.com.springcloudexample=trace
The second line is the application root namespace.
I've been trying to configure a Java application (specifically my integration tests) with absolutely no success. I originally had a single XMLConfiguration which I could load and use without issue. My directory setup looks like:
src
test
java
mypackage
resources
Before, I had a file myconfig.xml in the resources directory. I was loading it it like so:
private static XMLConfiguration configuration = null;
public static void configure() {
try {
configuration = new XMLConfiguration("myconfig.xml");
}
catch (ConfigurationException e) {
System.err.println(e);
}
}
Now I'm trying to make it so that users can configure the application themselves by writing a configuration file in /etc or /home or whatever, so I made a new file in the same location called config-test.xml:
<?xml version="1.0" encoding="ISO-8859-1"?>
<configuration>
<!-- Attempt to load configuration provided on the system -->
<xml fileName="/etc/myconf/myconfig.xml" config-optional="true" />
<!-- Load default configuration from the classpath -->
<xml fileName="myconfig.xml" />
</configuration>
The file /etc/myconf/myconfig.xml does not exist, but that's the whole point of making it optional. I've tried a few different approaches to loading my configuration, including how the documentation does it: http://commons.apache.org/configuration/userguide/howto_configurationbuilder.html. What I have currently is the following:
private static CombinedConfiguration configuration = null;
public static void configure() {
try {
DefaultConfigurationBuilder builder =
new DefaultConfigurationBuilder("config-test.xml");
configuration = builder.getConfiguration(true);
System.err.println("Yay");
}
catch (ConfigurationException e) {
System.err.println("Herp");
}
catch (Exception e) {
System.err.println("Derp");
}
}
When I run my integration tests, I see no Yay, I see no Herp and I see no Derp. If I place a println before I try to load things, it does print so configure is being called. The output I get from commons configuration amounts to:
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running TestSuite
13:36:57.752 [main] DEBUG o.a.c.c.ConfigurationUtils - ConfigurationUtils.locate(): base is null, name is config-test.xml
13:36:57.753 [main] DEBUG o.a.c.c.DefaultFileSystem - Could not locate file config-test.xml at null: no protocol: config-test.xml
13:36:57.756 [main] DEBUG o.a.c.c.ConfigurationUtils - Loading configuration from the context classpath (config-test.xml)
13:36:57.766 [main] DEBUG o.a.c.c.DefaultConfigurationBuilder - Creating configuration null with name null
13:36:57.795 [main] DEBUG o.a.c.c.ConfigurationUtils - ConfigurationUtils.locate(): base is null, name is config-test.xml
13:36:57.795 [main] DEBUG o.a.c.c.DefaultFileSystem - Could not locate file config-test.xml at null: no protocol: config-test.xml
13:36:57.796 [main] DEBUG o.a.c.c.ConfigurationUtils - Loading configuration from the context classpath (config-test.xml)
13:36:57.800 [main] DEBUG o.a.c.c.DefaultConfigurationBuilder - Creating configuration null with name null
Tests run: 56, Failures: 2, Errors: 0, Skipped: 16, Time elapsed: 2.589 sec <<< FAILURE!
Results :
Failed tests: initialize(mypackage.ImportTestIT): org/apache/commons/beanutils/PropertyUtils
initialize(mypackage.TransferTestIT): org/apache/commons/beanutils/PropertyUtils
Tests run: 56, Failures: 2, Errors: 0, Skipped: 16
Those initialize methods are just calling configure and then trying to get data from the configuration that is loaded. I've been working with this for a couple hours now and could really use another pair of eyes. Any ideas? Is there anything I could have changed that I'm not mentioning that would affect this? Thanks.
Edit #1 (1:58pm): After making sure beanutils were on the classpath, I now get a brutal explosion of stack trace that starts with
Running TestSuite
14:00:16.088 [main] DEBUG o.a.c.c.ConfigurationUtils - ConfigurationUtils.locate(): base is null, name is config-test.xml
14:00:16.089 [main] DEBUG o.a.c.c.DefaultFileSystem - Could not locate file config-test.xml at null: no protocol: config-test.xml
14:00:16.093 [main] DEBUG o.a.c.c.ConfigurationUtils - Loading configuration from the context classpath (config-test.xml)
14:00:16.103 [main] DEBUG o.a.c.c.DefaultConfigurationBuilder - Creating configuration null with name null
14:00:16.178 [main] DEBUG o.a.c.c.ConfigurationUtils - ConfigurationUtils.locate(): base is file:///home/pgarrity/projects/myproject/target/test-classes/config-test.xml, name is /etc/myconf/myproject/myconfig.xml
14:00:16.179 [main] DEBUG o.a.c.c.DefaultFileSystem - Could not locate file /etc/myconf/myproject/myconfig.xml at file:///home/pgarrity/projects/myproject/target/test-classes/config-test.xml: /etc/myconf/myproject/myconfig.xml (No such file or directory)
14:00:16.181 [main] DEBUG o.a.c.c.DefaultConfigurationBuilder - Load failed for optional configuration xml: Cannot locate configuration source /etc/myconf/myproject/myconfig.xml
14:00:16.185 [main] WARN o.a.c.c.DefaultConfigurationBuilder - Internal error
org.apache.commons.configuration.ConfigurationException: Cannot locate configuration source /etc/myconf/myproject/myconfig.xml
at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:259) ~[commons-configuration-1.8.jar:1.8]
at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:238) ~[commons-configuration-1.8.jar:1.8]
And so on and so on... but I do see 'Yay' now. Now what I don't get is that it doesn't seem to be looking for my non-optional configuration source, or assuming it's in a location I never told it to look.
Edit #2:
I looked through my output a bit more and buried in the complaints from commons I found that it seemed to load the correct file eventually. I think I may have it working? Maybe my path to the properties change when I load it like this... I've got some stuff to try. Anyway, the problem that I asked about has been fixed. Thanks for the help.
You seem to be missing commons-beanutils on your classpath. Make sure that jar is there.
Pfft. A multi-page document talks about how to load a configuration file. But did they forget to implement detection of file URLs in absolute Windows paths such as c:\Users\USER\FOO\bar.xml? Commons 1.9 will show this scrupulous message,
DEBUG DefaultFileSystem - Could not locate file C:\Users\USER\FOO\bar.xml at null: unknown protocol: c
https://github.com/apache/commons-configuration/blob/CONFIGURATION_1_9/src/main/java/org/apache/commons/configuration/DefaultFileSystem.java#L281