I followed this page to install jetty on a Amazon EC2 instance of mine (Ubuntu 14.04.2 LTS). Jetty is running. However, I can't make it use logback for logging. Below are relevant information. What did I do wrong here?
service check
$ service jetty check
Checking arguments to Jetty:
START_INI = /opt/jetty/start.ini
START_D = /opt/jetty/start.d
JETTY_HOME = /opt/jetty
JETTY_BASE = /opt/jetty
JETTY_CONF = /opt/jetty/etc/jetty.conf
JETTY_PID = /tmp/jetty.pid
JETTY_START = /opt/jetty/start.jar
JETTY_LOGS = /opt/jetty/logs
JETTY_STATE = /opt/jetty/jetty.state
CLASSPATH =
JAVA = /usr/bin/java
JAVA_OPTIONS = -Djetty.logs=/opt/jetty/logs -Djetty.home=/opt/jetty -Djetty.base=/opt/jetty -Djava.io.tmpdir=/tmp
JETTY_ARGS = jetty.state=/opt/jetty/jetty.state jetty-started.xml
RUN_CMD = /usr/bin/java -Djetty.logs=/opt/jetty/logs -Djetty.home=/opt/jetty -Djetty.base=/opt/jetty -Djava.io.tmpdir=/tmp -jar /opt/jetty/start.jar jetty.state=/opt/jetty/jetty.state jetty-started.xml
/etc/default/jetty
$ cat /etc/default/jetty
# Defaults for jetty see /etc/init.d/jetty for more
# change to 0 to allow Jetty to start
NO_START=0
# change to 'no' or uncomment to use the default setting in /etc/default/rcS
VERBOSE=yes
# Run Jetty as this user ID (default: jetty)
# Set this to an empty string to prevent Jetty from starting automatically
JETTY_USER=jetty
# Listen to connections from this network host
# Use 0.0.0.0 as host to accept all connections.
# Uncomment to restrict access to localhost
JETTY_HOST=0.0.0.0
# The network port used by Jetty
JETTY_PORT=8080
# Timeout in seconds for the shutdown of all webapps
JETTY_SHUTDOWN=30
# Additional arguments to pass to Jetty
#JETTY_ARGS=
# Extra options to pass to the JVM
#JAVA_OPTIONS="-Xmx256m -Djava.awt.headless=true -Djava.library.path=/usr/lib"
# Home of Java installation.
JAVA=/usr/bin/java
JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
JETTY_HOME=/opt/jetty
# The first existing directory is used for JAVA_HOME (if JAVA_HOME is not
# defined in /etc/default/jetty). Should contain a list of space separated directories.
#JDK_DIRS="/usr/lib/jvm/default-java /usr/lib/jvm/java-6-sun"
# Java compiler to use for translating JavaServer Pages (JSPs). You can use all
# compilers that are accepted by Ant's build.compiler property.
#JSP_COMPILER=jikes
# Jetty uses a directory to store temporary files like unpacked webapps
#JETTY_TMP=/var/cache/jetty
# Jetty uses a config file to setup its boot classpath
#JETTY_START_CONFIG=/etc/jetty/start.config
# Default for number of days to keep old log files in /var/log/jetty/
#LOGFILE_DAYS=14
/opt/jetty/etc/jetty.conf
$ cat /opt/jetty/etc/jetty.conf
# ========================================================
# jetty.conf Configuration for jetty.sh script
# --------------------------------------------------------
# This file is used by the jetty.sh script to provide
# extra configuration arguments for the start.jar command
# created by that script.
#
# Each line in this file becomes an arguement to start.jar
# in addition to those found in the start.ini file
# =======================================================
#jetty-logging.xml
jetty-started.xml
/opt/jetty/modules/logging.mod
$ cat /opt/jetty/modules/logging.mod
#
# Jetty std err/out logging
#
[name]
logging
[files]
logs/
[lib]
lib/logging/*.jar
resources/
[ini-template]
## Logging Configuration
# Configure jetty logging for default internal behavior STDERR output
# -Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.StdErrLog
# Configure jetty logging for slf4j
-Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.Slf4jLog
# Configure jetty logging for java.util.logging
# -Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.JavaUtilLog
# STDERR / STDOUT Logging
# Number of days to retain logs
# jetty.log.retain=90
# Directory for logging output
# Either a path relative to ${jetty.base} or an absolute path
# jetty.logs=logs
/opt/jetty/resources/logback.xml
$ cat /opt/jetty/resources/logback.xml
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<!-- Insert the current time formatted as "yyyyMMdd'T'HHmmss" under
the key "bySecond" into the logger context. This value will be
available to all subsequent configuration elements. -->
<timestamp key="bySecond" datePattern="yyyyMMdd'T'HHmmss"/>
<!-- console appender -->
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{100} - %msg%n</pattern>
</encoder>
</appender>
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>/opt/jetty/logs/server.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!-- daily rollover -->
<fileNamePattern>/opt/jetty/logs/server.%d{yyyy-MM-dd}.log</fileNamePattern>
<!-- keep 30 days' worth of history -->
<maxHistory>30</maxHistory>
</rollingPolicy>
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{100} - %msg%n</pattern>
</encoder>
</appender>
<logger name="org.eclipse.jetty" level="INFO"/>
<logger name="org.springframework" level="INFO"/>
<logger name="org.apache" level="INFO"/>
<root level="DEBUG">
<appender-ref ref="CONSOLE" />
<appender-ref ref="FILE" />
</root>
</configuration>
/opt/jetty/start.ini
$ cat /opt/jetty/start.ini
#===========================================================
# Jetty Startup
#
# Starting Jetty from this {jetty.home} is not recommended.
#
# A proper {jetty.base} directory should be configured, instead
# of making changes to this {jetty.home} directory.
#
# See documentation about {jetty.base} at
# http://www.eclipse.org/jetty/documentation/current/startup.html
#
# A demo-base directory has been provided as an example of
# this sort of setup.
#
# $ cd demo-base
# $ java -jar ../start.jar
#
#===========================================================
# To disable the warning message, comment the following line
--module=home-base-warning
# ---------------------------------------
# Module: server
--module=server
##
## Server Threading Configuration
##
# minimum number of threads
threads.min=10
# maximum number of threads
threads.max=200
# thread idle timeout in milliseconds
threads.timeout=60000
# buffer size for output
jetty.output.buffer.size=32768
# request header buffer size
jetty.request.header.size=8192
# response header buffer size
jetty.response.header.size=8192
# should jetty send the server version header?
jetty.send.server.version=true
# should jetty send the date header?
jetty.send.date.header=false
# What host to listen on (leave commented to listen on all interfaces)
#jetty.host=0.0.0.0
# Dump the state of the Jetty server, components, and webapps after startup
jetty.dump.start=false
# Dump the state of the Jetty server, before stop
jetty.dump.stop=false
# Enable delayed dispatch optimisation
jetty.delayDispatchUntilContent=false
# ---------------------------------------
# Module: deploy
--module=deploy
## DeployManager configuration
# Monitored Directory name (relative to jetty.base)
# jetty.deploy.monitoredDirName=webapps
# ---------------------------------------
# Module: websocket
--module=websocket
# ---------------------------------------
# Module: ext
--module=ext
# ---------------------------------------
# Module: resources
--module=resources
# ---------------------------------------
# Module: jsp
--module=jsp
# JSP Configuration
# Select JSP implementation, choices are
# glassfish : The reference implementation
# default in jetty <= 9.1
# apache : The apache version
# default jetty >= 9.2
jsp-impl=apache
# To use a non-jdk compiler for JSP compilation when using glassfish uncomment next line
# -Dorg.apache.jasper.compiler.disablejsr199=true
# ---------------------------------------
# Module: jstl
--module=jstl
# JSTL Configuration
# The glassfish jsp-impl includes JSTL by default and this module
# is not required to activate it.
# The apache jsp-impl does not include JSTL by default and this module
# is required to put JSTL on the container classpath
#
# Initialize module logging
#
--module=logging
# ---------------------------------------
# Module: http
--module=http
### HTTP Connector Configuration
-Djava.net.preferIPv4Stack=true
## HTTP port to listen on
jetty.port=8080
## HTTP idle timeout in milliseconds
http.timeout=30000
## HTTP Socket.soLingerTime in seconds. (-1 to disable)
# http.soLingerTime=-1
## Parameters to control the number and priority of acceptors and selectors
# http.selectors=1
# http.acceptors=1
# http.selectorPriorityDelta=0
# http.acceptorPriorityDelta=0
First (and most important) don't change, edit, modify, add, remove, delete, mangle, rename, touch, etc anything in ${jetty.home}
That is your jetty-distribution directory, its not meant to be changed, unpack it, then leave it alone.
It even tells you this when you startup!
2015-05-06 06:34:58.838:WARN:oejs.HomeBaseWarning:main: This instance of Jetty is not
running from a separate {jetty.base} directory, this is not recommended.
See documentation at http://www.eclipse.org/jetty/documentation/current/startup.html
2015-05-06 06:34:58.959:INFO:oejs.Server:main: jetty-9.2.10.v20150310
To fix this:
Start by creating a new directory somewhere (anywhere) this will become your ${jetty.base}
The ${jetty.base} directory is where you configure your instance of a running Jetty.
The documentation has instructions on 3 different configurations for logback in a ${jetty.base}
Just Logback
Capturing all logging events on the server side from log4j, commons-logging, slf4j, jetty log, and java.util.logging and routing them to Logback
Forcing centralized logging of all logging, even those produced in webapps, to a single logback configuration
If you follow those directions, you'll wind up with:
a ${jetty.base} that has a start.ini for you
the libraries you'll need in ${jetty.base}/lib/logging
an overridden ${jetty.base}/modules/logging.mod
the logback configuration in ${jetty.base}/resources/logback.xml
the jetty logging configuration in ${jetty.base}/resources/jetty-logging.properties
From here you setup the rest of the modules you'll need in ${jetty.base}/start.ini along with your webapps and other bits and bobs you'll need (never once editing ${jetty.home})
Finally, you'll edit your /etc/default/jetty to add a JETTY_BASE that points to this directory.
To test if this ${jetty.base} configuration makes sense, you'll use the command line.
CD into your new ${jetty.base} and run
[mybase]$ java -jar /path/to/jetty-dist/start.jar
That will start up that configuration as a normal process (not a service), from which you'll be able to see if the configuration is behaving the way you need it to.
I solved the problem by following the official example: http://eclipse.org/jetty/documentation/current/example-logging-logback-centralized.html. The actual steps are quite similar to what Joakim mentioned in his answer.
These are basically the commands I used:
[base]$ mkdir modules
[base]$ cd modules
[modules]$ curl -O https://raw.githubusercontent.com/jetty-project/logging-modules/master/capture-all/logging.mod
[modules]$ curl -O https://raw.githubusercontent.com/jetty-project/logging-modules/master/centralized/webapp-logging.mod
[modules]$ cd ..
[base]$ java -jar /opt/jetty/start.jar --add-to-start=logging,webapp-logging
Run the following command and Jetty do all the things:
[my-base]$ java -jar ../start.jar --add-to-start=logging-logback
More configuration: https://www.eclipse.org/jetty/documentation/jetty-9/index.html#configuring-logging
Related
I have trying to install elastic search engine in my windows .. at first I download zip file, extract it and then write the command ..
after I getting some errors but I solved them this is the last error I get and can't solve :
-XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m,
-Xms4020m, -Xmx4020m, -XX:MaxDirectMemorySize=2107637760, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=zip, --module-path=C:\elasticsearch-8.4.3\elasticsearch-8.4.3\lib, --add-modules=jdk.net, -Djdk.module.main=org.elasticsearch.server]
this is my jvm.options file
################################################################
##
## JVM configuration
##
################################################################
##
## WARNING: DO NOT EDIT THIS FILE. If you want to override the
## JVM options in this file, or set any additional options, you
## should create one or more files in the jvm.options.d
## directory containing your adjustments.
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/8.4/jvm-options.html
## for more information.
##
################################################################
################################################################
## IMPORTANT: JVM heap size
################################################################
##
## The heap size is automatically configured by Elasticsearch
## based on the available memory in your system and the roles
## each node is configured to fulfill. If specifying heap is
## required, it should be done through a file in jvm.options.d,
## which should be named with .options suffix, and the min and
## max should be set to the same value. For example, to set the
## heap to 4 GB, create a new file in the jvm.options.d
## directory containing these lines:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/8.4/heap-size.html
## for more information
##
################################################################
################################################################
## Expert settings
################################################################
##
## All settings below here are considered expert settings. Do
## not adjust them unless you understand what you are doing. Do
## not edit them in this file; instead, create a new file in the
## jvm.options.d directory containing your adjustments.
##
################################################################
-XX:+UseG1GC
## JVM temporary directory
-Djava.io.tmpdir=${ES_TMPDIR}
## heap dumps
# generate a heap dump when an allocation from the Java heap fails; heap dumps
# are created in the working directory of the JVM unless an alternative path is
# specified
-XX:+HeapDumpOnOutOfMemoryError
# exit right after heap dump on out of memory error
-XX:+ExitOnOutOfMemoryError
# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=data
# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=logs/hs_err_pid%p.log
## GC logging
-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m
this is my elasticsearch.yml file :
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# --------------------------------- Readiness ----------------------------------
#
# Enable an unauthenticated TCP readiness endpoint on localhost
#
#readiness.port: 9399
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false
#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 18-10-2022 14:38:08
#
# --------------------------------------------------------------------------------
# Enable security features
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
ingest.geoip.downloader.enabled: false
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["SAM-SAM"]
# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0
# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0
#----------------------- END SECURITY AUTO CONFIGURATION -------------------------
and I have memory, I have 50GB free on C disk and 345GB in D
where is the problem? and how can I solve it?
I'm trying to deploy a Spark job on Kubernetes, using kubectl apply -f <config_file.yml> (after building Docker image based on Dockerfile). The pod is successfuly created on K8s, then quickly stops with a Failed status. Nothing in the logs help understanding where the error comes from. Other jobs have been successfully deployed on the K8s cluster using the same Dockerfile and config file.
The spark job is supposed to read data from a kafka topic, parse it and outout it in console.
Any idea what might be causing the job to fail?
Dockerfile, built using docker build --rm -f "Dockerfile" xxxxxxxx:80/apache/myapp-test . && docker push xxxxxxxx:80/apache/myapp-test :
FROM xxxxxxxx:80/apache/spark:v2.4.4-gcs-prometheus
#USER root
ADD myapp.jar /jars
RUN adduser --no-create-home --system spark
RUN chown -R spark /prometheus /opt/spark
USER spark
config_file.yml :
apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
name: myapp
namespace: spark
labels:
app: myapp-test
release: spark-2.4.4
spec:
type: Java
mode: cluster
image: "xxxxxxxx:80/apache/myapp-test"
imagePullPolicy: Always
mainClass: spark.jobs.app.streaming.Main
mainApplicationFile: "local:///jars/myapp.jar"
sparkVersion: "2.4.4"
restartPolicy:
type: OnFailure
onFailureRetries: 5
onFailureRetryInterval: 30
onSubmissionFailureRetries: 0
onSubmissionFailureRetryInterval: 0
driver:
cores: 1
memory: "1G"
labels:
version: 2.4.4
monitoring:
exposeDriverMetrics: true
exposeExecutorMetrics: true
prometheus:
jmxExporterJar: "/prometheus/jmx_prometheus_javaagent-0.11.0.jar"
port: 8090
imagePullSecrets:
- xxx
Logs :
++ id -u
+ myuid=100
++ id -g
+ mygid=65533
+ set +e
++ getent passwd 100
+ uidentry='spark:x:100:65533:Linux User,,,:/home/spark:/sbin/nologin'
+ set -e
+ '[' -z 'spark:x:100:65533:Linux User,,,:/home/spark:/sbin/nologin' ']'
+ SPARK_K8S_CMD=driver
+ case "$SPARK_K8S_CMD" in
+ shift 1
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ grep SPARK_JAVA_OPT_
+ + sed sort -t_ 's/[^=]*=\(.*\)/\1/g'-k4
-n
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
+ '[' -n '' ']'
+ '[' -n '' ']'
+ PYSPARK_ARGS=
+ '[' -n '' ']'
+ R_ARGS=
+ '[' -n '' ']'
+ '[' '' == 2 ']'
+ '[' '' == 3 ']'
+ case "$SPARK_K8S_CMD" in
+ CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$#")
+ exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=192.168.225.14 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class spark.jobs.app.streaming.Main spark-internal
20/04/20 09:27:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
log4j:WARN No appenders could be found for logger (org.apache.spark.deploy.SparkSubmit$$anon$2).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Pod events as shown with kubectl describe pod :
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 15m default-scheduler Successfully assigned spark/myapp-driver to xxxxxxxx.preprod.local
Warning FailedMount 15m kubelet, xxxxxxxx.preprod.local MountVolume.SetUp failed for volume "spark-conf-volume" : configmap "myapp-1587388343593-driver-conf-map" not found
Warning DNSConfigForming 15m (x4 over 15m) kubelet, xxxxxxxx.preprod.local Search Line limits were exceeded, some search paths have been omitted, the applied search line is: spark.svc.cluster.local svc.cluster.local cluster.local preprod.local
Normal Pulling 15m kubelet, xxxxxxxx.preprod.local Pulling image "xxxxxxxx:80/apache/myapp-test"
Normal Pulled 15m kubelet, xxxxxxxx.preprod.local Successfully pulled image "xxxxxxxx:80/apache/myapp-test"
Normal Created 15m kubelet, xxxxxxxx.preprod.local Created container spark-kubernetes-driver
Normal Started 15m kubelet, xxxxxxxx.preprod.local Started container spark-kubernetes-driver
You have to review conf/spark-env.(sh|cmd)
Start by configuring the logging
Spark uses log4j for logging. You can configure it by adding a
log4j.properties file in the conf directory. One way to start is to
copy the existing log4j.properties.template located there.
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Set everything to be logged to the console
log4j.rootCategory=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN
# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark_project.jetty=WARN
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR
# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR
I am trying to get centralised logging working with log4j and rsyslog.
What I have so far
Solr running inside tomcat6 on RHEL6, using the following log4j and sl4j libs
# lsof -u tomcat | grep log4j
java 14503 tomcat mem REG 253,0 9711 10208 /usr/share/java/tomcat6/slf4j-log4j12-1.6.6.jar
java 14503 tomcat mem REG 253,0 481535 10209 /usr/share/java/tomcat6/log4j-1.2.16.jar
java 14503 tomcat mem REG 253,0 378088 1065276 /usr/share/java/log4j-1.2.14.jar
java 14503 tomcat 20r REG 253,0 378088 1065276 /usr/share/java/log4j-1.2.14.jar
java 14503 tomcat 21r REG 253,0 481535 10209 /usr/share/java/tomcat6/log4j-1.2.16.jar
java 14503 tomcat 35r REG 253,0 9711 10208 /usr/share/java/tomcat6/slf4j-log4j12-1.6.6.jar
#
Solr is using the following log4j.properties file (via -Dlog4j.configuration=file:///opt/solr/lib/log4j.properties)
# Logging level
log4j.rootLogger=INFO, file, CONSOLE, SYSLOG
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=%-4r [%t] %-5p %c %x \u2013 %m%n
#- size rotation with log cleanup.
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.MaxFileSize=4MB
log4j.appender.file.MaxBackupIndex=9
#- File to log to and log format
log4j.appender.file.File=/var/log/tomcat6/solr.log
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%-5p - %d{yyyy-MM-dd HH:mm:ss.SSS}; %C; %m\n
log4j.logger.org.apache.zookeeper=WARN
log4j.logger.org.apache.hadoop=WARN
# set to INFO to enable infostream log messages
log4j.logger.org.apache.solr.update.LoggingInfoStream=OFF
#- Local syslog server
log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
log4j.appender.SYSLOG.syslogHost=localhost
log4j.appender.SYSLOG.facility=LOCAL1
log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout
log4j.appender.SYSLOG.layout.ConversionPattern=${sysloghostname} %-4r [%t] java %-5p %c %x %m%n
log4j.appender.SYSLOG.Header=true
On the same server I have rsyslog running and accepting log messages from log4j.
# rpmquery -a | grep syslog
rsyslog-5.8.10-7.el6_4.x86_64
#
rsyslog config
# #### MODULES ####
$MaxMessageSize 32k
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
$ModLoad imfile # provides file monitoring support
#
$ModLoad imudp.so
$UDPServerRun 514
$WorkDirectory /var/lib/rsyslog # where to place spool files
# #### GLOBAL DIRECTIVES ####
# # Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
$ActionQueueType LinkedList # run asynchronously
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionResumeRetryCount -1 # infinite retries if host is down
$ActionSendStreamDriverMode 0 # require TLS for the connection
$ActionSendStreamDriverAuthMode anon # chain and server are verified
#local1.*;*.* ##(o)XXXXXXXX:5544
local1.* /var/log/remote.log
# # The authpriv file has restricted access.
authpriv.* /var/log/secure
# # Log all the mail messages in one place.
mail.* -/var/log/maillog
# # Log cron stuff
cron.* /var/log/cron
# # Everybody gets emergency messages
*.emerg *
# # Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# # Save boot messages also to boot.log
local7.* /var/log/boot.log
I am catching local1 messages from Solr's logj4 and redirecting them to /var/log/remote.log
Everything works as expected. Sample INFO message
Oct 31 13:57:08 hostname.here 3431839 [http-8080-10] java INFO org.apache.solr.core.SolrCore [collection1] webapp=/solr path=/select params={indent=true&q=*:*&wt=json&rows=1} hits=42917 status=0 QTime=1
And stack traces are on the same line as the ERROR message
Oct 31 12:27:17 hostname.here 157666248 [http-8080-7] java ERROR org.apache.solr.core.SolrCore org.apache.solr.common.SolrException: undefined field *#012#011at org.apache.solr.schema.IndexSchema.getDynamicFieldType(IndexSchema.java:1223)#012... Cut for brevity....#011at java.lang.Thread.run(Thread.java:724)#012
Note #012 as line ending and #011 tab.
Using this setup I can ship the logs to remote rsyslog server over TCP and pipe them into fluentd/elaticsearch/kibana etc... everything works as expected.
The problem
I am now trying to get another webapp running inside the same tomcat container to log as above, everything works as expected apart from stack traces, each line of a stack trace ends up on a separate line (separate syslog message)
Oct 31 12:54:47 hostname.here 4909 [main] java ERROR org.hibernate.tool.hbm2ddl.SchemaUpdate could not get database metadata
Oct 31 12:54:47 hostname.here org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Communications link failure
Oct 31 12:54:47 hostname.here
Oct 31 12:54:47 hostname.here The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.)
The webapp ships with it's own log4j libs and log4j.xml config. Libs are of the same version as those used by solr.
log4j.xml file for this app
<appender name="SYSLOG" class="org.apache.log4j.net.SyslogAppender">
<param name="SyslogHost" value="localhost" />
<param name="Facility" value="LOCAL1" />
<param name="Header" value="false" />
<property name="facilityPrinting" value="false"/>
<param name="Threshold" value="DEBUG" />
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern"
value="%-4r [%t] java %-5p %c %x %m%n"/>
</layout>
</appender>
I would like to see stack traces from the new application to appear on the same line just like with Solr.
Does anyone know if this is a log4j config issue?
Many thanks.
I've been working on something similar lately (in fact, I have this question open presently you may be able to help with).
This is probably not a very good answer, but it's more information than I can fit in a comment so here you go (I hope some of it is new to you). The rsyslog imfile rsyslog documentation has this section:
ReadMode [mode]
This mode should defined when having multiline messages. The value can range from 0-2 and determines the multiline detection method.
0 (default) - line based (Each line is a new message)
1 - paragraph (There is a blank line between log messages)
2 - indented (New log messages start at the beginning of a line. If a line starts with a space it is part of the log message before it)
The imudp rsyslog docs have no such configuration option. My guess is that the UDP input module doesn't support multiline logging. Thus, each line of the stack trace is sent out as a separate log entry.
Do you have any configuration files in /etc/rsyslog.d? There may be more information in there.
I have a java program and I want to log in /var/log/messages file on fedora machine. I am usin log4j SyslogAppender but its not working.
my log4j properties file contains
# Set root category priority to INFO and its only appender to CONSOLE.
log4j.rootCategory=INFO, CONSOLE, SYSLOG
#log4j.rootCategory=INFO, CONSOLE, LOGFILE
# Set the enterprise logger priority to DEBUG
log4j.logger.com.locaid=INFO, CONSOLE, LOGFILE, SYSLOG
# CONSOLE is set to be a ConsoleAppender using a PatternLayout.
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
# LOGFILE is set to be a File appender using a PatternLayout.
log4j.appender.LOGFILE=org.apache.log4j.FileAppender
log4j.appender.LOGFILE.File=/home/dev/app.log
log4j.appender.LOGFILE.Append=true
log4j.appender.LOGFILE.layout=org.apache.log4j.PatternLayout
log4j.appender.LOGFILE.layout.ConversionPattern=[%d{dd/MM/y HH:mm:ss}][%t][%1p] %c - %m%n
log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
log4j.appender.SYSLOG.syslogHost=localhost
log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout
log4j.appender.SYSLOG.layout.conversionPattern=%d{ISO8601} %-5p [%t] %c{2} %x - %m%n
log4j.appender.SYSLOG.Facility=LOCAL1
log4j.appender.SYSLOG.Threshold=debug
log4j.appender.SYSLOG.FacilityPrinting=true
in /etc/rsyslog.conf i have
local1.* /var/log/app.log
in /etc/sysconfig/rsyslog have
SYSLOGD_OPTIONS="-r -m 0 -c 4"
On restarting rsyslog service app.log file is created but no logs are being appended. I have also tried with default USER facility its not working although logger -p LOCAL1.info cmd is working and appending log to app.log. Need help.
I can't see anything wrong at the log4j end, but my /etc/default/rsyslog (on Ubuntu) says
# Options for rsyslogd
# -m 0 disables 'MARK' messages (deprecated, only used in compat mode < 3)
# -r enables logging from remote machines (deprecated, only used in compat mode < 3)
# -x disables DNS lookups on messages received with -r
# -c compatibility mode
# See rsyslogd(8) for more details
which suggests that -r and -m 0 won't work in combination with -c 4. Instead of trying to set remote access here, you should edit your /etc/rsyslogd.conf and add (or uncomment)
$ModLoad imudp
$UDPServerRun 514
#This Configuration File is used for Logger Module which is used for either using Log4J or SysLog4J
log4j.rootLogger = DEBUG,LOGFILE
#------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------
#[Log4j]
#------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------
# Log4j implements Rolling File Appender Configurations
log4j.appender.LOGFILE=org.apache.log4j.RollingFileAppender
# The log4j layout
log4j.appender.LOGFILE.layout=org.apache.log4j.PatternLayout
# The Log4j configuration file location
log4j.appender.LOGFILE.File= C:/Documents and Settings/bgh28706/Desktop/log_output.cfg
# The Log4j conversion layout to apply for
log4j.appender.LOGFILE.layout.ConversionPattern=%d{ISO8601} %t %p %c %M %m %n
# The Log4j maximum file size to keep for before renaming to the backup file
log4j.appender.LOGFILE.MaxFileSize=5KB
# The maximum number of log4j files to be kept in the system
log4j.appender.LOGFILE.MaxBackupIndex=2
#Additivity set to False makes the output not to be produced in any other appender
log4j.additivity.LOGFILE.file=false
#------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------
#[Syslog]
#------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------
# The syslog appender to be configured for the syslog configuration to affect
log4j.appender.SYSLOGFILE=org.apache.log4j.net.SyslogAppender
# The hostname to log the syslogger information
log4j.appender.SYSLOGFILE.SyslogHost=localhost
# The facility name in the logger where the log file shall be logged
log4j.appender.SYSLOGFILE.facility=local5
# The log filename layout of the syslogger appender
log4j.appender.SYSLOGFILE.layout=org.apache.log4j.PatternLayout
# The syslogger configuration pattern
log4j.appender.SYSLOGFILE.layout.ConversionPattern=%d{ISO8601} %t %p %c %M %m %n
#Additivity set to False makes the output not to be produced in any other appender
log4j.additivity.SYSLOGFILE.file=false
See if this can help you out as it is running fine for mine!
Regards
Anand Bhat
Problem here is
SYSLOGD_OPTIONS="-r -m 0 -c 4"
Should be
SYSLOGD_OPTIONS="-r -m 0"
These options do not work together
I have an application called "Update.jar" that I'm trying to use with the java service wrapper (JSW), but when I start the service (either from SERVICES.MSC or StartUpdate-NT.bat) the application doesn't run, even though the service is showing as started in SERVICES.MSC. There should be an icon displayed in the system tray through out the length of the runtime.
I've successfully launched the app:
by executing the .jar
by running Update.bat in [wrapper]/bin/ directory
by executing from the command line
Below is my wrapper.conf file:
#encoding=UTF-8
# Configuration files must begin with a line specifying the encoding
# of the the file.
#********************************************************************
# Wrapper License Properties (Ignored by Community Edition)
#********************************************************************
# Professional and Standard Editions of the Wrapper require a valid
# License Key to start. Licenses can be purchased or a trial license
# requested on the following pages:
# http://wrapper.tanukisoftware.com/purchase
# http://wrapper.tanukisoftware.com/trial
# Include file problems can be debugged by removing the first '#'
# from the following line:
#include.debug
# The Wrapper will look for either of the following optional files for a
# valid License Key. License Key properties can optionally be included
# directly in this configuration file.
#include ../conf/wrapper-license.conf
#include ../conf/wrapper-license-%WRAPPER_HOST_NAME%.conf
# The following property will output information about which License Key(s)
# are being found, and can aid in resolving any licensing problems.
#wrapper.license.debug=TRUE
#********************************************************************
# Wrapper Localization
#********************************************************************
# Specify the locale which the Wrapper should use. By default the system
# locale is used.
#wrapper.lang=en_US # en_US or ja_JP
# Specify the location of the Wrapper's language resources. If these are
# missing, the Wrapper will default to the en_US locale.
wrapper.lang.folder=../lang
#********************************************************************
# Wrapper Java Properties
#********************************************************************
# Java Application
# Locate the java binary on the system PATH:
wrapper.java.command=java
# Specify a specific java binary:
#set.JAVA_HOME=/java/path
#wrapper.java.command=%JAVA_HOME%/bin/java
# Tell the Wrapper to log the full generated Java command line.
#wrapper.java.command.loglevel=INFO
# Java Main class. This class must implement the WrapperListener interface
# or guarantee that the WrapperManager class is initialized. Helper
# classes are provided to do this for you. See the Integration section
# of the documentation for details.
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp update.Tray
# Java Classpath (include wrapper.jar) Add class path elements as
# needed starting from 1
wrapper.java.classpath.1=../lib/wrapper.jar
wrapper.java.classpath.2=../lib/Update.jar
# Java Library Path (location of Wrapper.DLL or libwrapper.so)
wrapper.java.library.path.1=../lib
# Java Bits. On applicable platforms, tells the JVM to run in 32 or 64-bit mode.
wrapper.java.additional.auto_bits=TRUE
# Java Additional Parameters
wrapper.java.additional.1=
# Initial Java Heap Size (in MB)
#wrapper.java.initmemory=3
# Maximum Java Heap Size (in MB)
#wrapper.java.maxmemory=64
# Application parameters. Add parameters as needed starting from 1
#wrapper.app.parameter.1=update.Tray
#********************************************************************
# Wrapper Logging Properties
#********************************************************************
# Enables Debug output from the Wrapper.
# wrapper.debug=TRUE
# Format of output for the console. (See docs for formats)
wrapper.console.format=PM
# Log Level for console output. (See docs for log levels)
wrapper.console.loglevel=INFO
# Log file to use for wrapper output logging.
wrapper.logfile=../logs/wrapper.log
# Format of output for the log file. (See docs for formats)
wrapper.logfile.format=LPTM
# Log Level for log file output. (See docs for log levels)
wrapper.logfile.loglevel=INFO
# Maximum size that the log file will be allowed to grow to before
# the log is rolled. Size is specified in bytes. The default value
# of 0, disables log rolling. May abbreviate with the 'k' (kb) or
# 'm' (mb) suffix. For example: 10m = 10 megabytes.
wrapper.logfile.maxsize=0
# Maximum number of rolled log files which will be allowed before old
# files are deleted. The default value of 0 implies no limit.
wrapper.logfile.maxfiles=0
# Log Level for sys/event log output. (See docs for log levels)
wrapper.syslog.loglevel=NONE
#********************************************************************
# Wrapper General Properties
#********************************************************************
# Allow for the use of non-contiguous numbered properties
wrapper.ignore_sequence_gaps=TRUE
# Title to use when running as a console
wrapper.console.title=Test Wrapper Sample Application
#********************************************************************
# Wrapper JVM Checks
#********************************************************************
# Detect DeadLocked Threads in the JVM. (Requires Standard Edition)
wrapper.check.deadlock=TRUE
wrapper.check.deadlock.interval=10
wrapper.check.deadlock.action=RESTART
wrapper.check.deadlock.output=FULL
# Out Of Memory detection.
# (Ignore output from dumping the configuration to the console. This is only needed by the TestWrapper sample application.)
wrapper.filter.trigger.999=wrapper.filter.trigger.*java.lang.OutOfMemoryError
wrapper.filter.allow_wildcards.999=TRUE
wrapper.filter.action.999=NONE
# (Simple match)
wrapper.filter.trigger.1000=java.lang.OutOfMemoryError
# (Only match text in stack traces if -XX:+PrintClassHistogram is being used.)
#wrapper.filter.trigger.1000=Exception in thread "*" java.lang.OutOfMemoryError
#wrapper.filter.allow_wildcards.1000=TRUE
wrapper.filter.action.1000=RESTART
wrapper.filter.message.1000=The JVM has run out of memory.
#********************************************************************
# Wrapper Email Notifications. (Requires Professional Edition)
#********************************************************************
# Common Event Email settings.
#wrapper.event.default.email.debug=TRUE
#wrapper.event.default.email.smtp.host=<SMTP_Host>
#wrapper.event.default.email.smtp.port=25
#wrapper.event.default.email.subject=[%WRAPPER_HOSTNAME%:%WRAPPER_NAME%:%WRAPPER_EVENT_NAME%] Event Notification
#wrapper.event.default.email.sender=<Sender email>
#wrapper.event.default.email.recipient=<Recipient email>
# Configure the log attached to event emails.
#wrapper.event.default.email.attach_log=TRUE
#wrapper.event.default.email.maillog.lines=50
#wrapper.event.default.email.maillog.format=LPTM
#wrapper.event.default.email.maillog.loglevel=INFO
# Enable specific event emails.
#wrapper.event.wrapper_start.email=TRUE
#wrapper.event.jvm_prelaunch.email=TRUE
#wrapper.event.jvm_start.email=TRUE
#wrapper.event.jvm_started.email=TRUE
#wrapper.event.jvm_deadlock.email=TRUE
#wrapper.event.jvm_stop.email=TRUE
#wrapper.event.jvm_stopped.email=TRUE
#wrapper.event.jvm_restart.email=TRUE
#wrapper.event.jvm_failed_invocation.email=TRUE
#wrapper.event.jvm_max_failed_invocations.email=TRUE
#wrapper.event.jvm_kill.email=TRUE
#wrapper.event.jvm_killed.email=TRUE
#wrapper.event.jvm_unexpected_exit.email=TRUE
#wrapper.event.wrapper_stop.email=TRUE
# Specify custom mail content
wrapper.event.jvm_restart.email.body=The JVM was restarted.\n\nPlease check on its status.\n
#********************************************************************
# Wrapper Windows NT/2000/XP Service Properties
#********************************************************************
# WARNING - Do not modify any of these properties when an application
# using this configuration file has been installed as a service.
# Please uninstall the service before modifying this section. The
# service can then be reinstalled.
# Name of the service
wrapper.name=Auto-update
# Display name of the service
wrapper.displayname=Auto-update
# Description of the service
wrapper.description=Auto-update
# Service dependencies. Add dependencies as needed starting from 1
wrapper.ntservice.dependency.1=
# Mode in which the service is installed. AUTO_START, DELAY_START or DEMAND_START
wrapper.ntservice.starttype=AUTO_START
# Allow the service to interact with the desktop.
wrapper.ntservice.interactive=true
Wrapper.log contents:
STATUS | wrapper | 2011/08/10 10:31:56 | Auto-update service installed.
STATUS | wrapper | 2011/08/10 10:32:07 | Starting the Auto-update service...
STATUS | wrapper | 2011/08/10 10:32:07 | --> Wrapper Started as Service
STATUS | wrapper | 2011/08/10 10:32:07 | Java Service Wrapper Community Edition 32-bit 3.5.10
STATUS | wrapper | 2011/08/10 10:32:07 | Copyright (C) 1999-2011 Tanuki Software, Ltd. All Rights Reserved.
STATUS | wrapper | 2011/08/10 10:32:07 | http://wrapper.tanukisoftware.com
STATUS | wrapper | 2011/08/10 10:32:07 |
STATUS | wrapper | 2011/08/10 10:32:08 | Launching a JVM...
INFO | jvm 1 | 2011/08/10 10:32:08 | WrapperManager: Initializing...
STATUS | wrapper | 2011/08/10 10:32:11 | Auto-update started.
Could someone please point me at the right direction?
can you set the loglevel to debug by setting
wrapper.debug=true
and rerun your application as service and post. From the log file you posted, your application seems to run... what happens after starting? does it shut down?
What OS are you running?
Please note that starting with Windows Vista, all Services run in an isolated desktop (session 0), since that, you wouldn't be able to see the tray icon in your user desktop...
Small correction (unrelated to your problem), also please change in your conf file:
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp update.Tray
to
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp
wrapper.app.parameter.1=update.Tray
cheers,