Develop Reference

What does the source tag mean in HttpEventCollectorLogbackAppender?

I'm using HttpEventCollectorLogbackAppender for writing my java application logs to the splunk server. I've been trying this for very long and still haven't been able to get my logs into splunk.
Can someone please explain what does the source tag refers to in the HttpEventLogbackAppender?
Below is the HttpEventLogbackAppender in my logback.xml file:
<appender name="splunk-httpeventcollector-appender"
class="com.splunk.logging.appenders.logback.HttpEventCollectorLogbackAppender">
<url>${SPLUNK_HOST_URL}</url>
<host>${CFG_DC}_${APP_ENV}_${CONTAINER_ID}</host>
<token>${SPLUNK_TOKEN}</token>
<source></source> // what does this refer to?
<index>${SPLUNK_INDEX}</index>
<disableCertificateValidation>true</disableCertificateValidation>
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>%d{ISO8601} [%thread] loglevel=%-5level %logger{36} - remotehost=%mdc{req.remoteHost} forwardedfor=%mdc{req.xForwardedFor} requestmethod=%mdc{req.method} requesturi=%mdc{req.requestURI}</Pattern>
</layout>
<batch_size_count>500</batch_size_count>
<send_mode>parallel</send_mode>
</appender>

From Splunk Documentaion , I found the following : Hope it will help you
Link - http://docs.splunk.com/Documentation/Splunk/7.1.2/Data/Aboutdefaultfields
source - The source of an event is the name of the file, stream, or other input from which the event originates.
For data monitored from files and directories, the value of source is the full path, such as /archive/server1/var/log/messages.0 or /var/log/.
The value of source for network-based data sources is the protocol and port, such as UDP:514.
This topic focuses on three key default fields:
host
source
sourcetype
Defining host, source, and sourcetype
The host, source, and sourcetype fields are defined as follows:
host - An event host value is typically the hostname, IP address, or fully qualified domain name of the network host from which the event originated. The host value lets you locate data originating from a specific device. For more information on hosts, see About hosts.
sourcetype - The source type of an event is the format of the data input from which it originates, such as access_combined or cisco_syslog. The source type determines how your data is to be formatted. For more information on source types, see Why source types matter.
Source vs sourcetype
Source and source type are both default fields, but they are entirely different otherwise, and can be easily confused.
The source is the name of the file, stream, or other input from which a particular event originates.
The sourcetype determines how Splunk software processes the incoming data stream into individual events according to the nature of the data.
Events with the same source type can come from different sources, for example, if you monitor source=/var/log/messages and receive direct syslog input from udp:514. If you search sourcetype=linux_syslog, events from both of those sources are returned.
Git Hub -
Logback configuration looks like:
```xml
<!-- Splunk HTTP Appender -->
<appender name="splunkHttpAppender" class="com.splunk.logging.HttpEventCollectorLogbackAppender">
<url>${lsplunk.http.url}</url>
<token>${splunk.http.token}</token>
<source>${splunk.source}</source>
<host>${splunk.httpevent.listener.host}</host>
<messageFormat>${splunk.event.message.format}</messageFormat>
<disableCertificateValidation>${splunk.cert.disable-validation}</disableCertificateValidation>
<layout class="ch.qos.logback.classic.PatternLayout">
<pattern>%date{ISO8601} [%thread] %level: %msg%n</pattern>
</layout>
</appender>
<logger name="com.example.app" additivity="false" level="INFO">
<appender-ref ref="splunkHttpAppender"/>
</logger>
<root level="INFO">
<appender-ref ref="splunkHttpAppender"/>
</root>

Or another way you can do it sent your application logs to s3 (AWS)and from their you can configue the path in inputs.conf splunk path and mention indexer in outputs.conf

Service Bus on Windows Server (1.1) using Java and AMQP 1.0 Timeout issue

I wrote a test program based on below example to connect to service bus(1.1) installed on windows server using amqp 1.0 java api.
https://msdn.microsoft.com/en-us/library/dn574799.aspx?f=255&MSPPError=-2147217396
Everything is fine till session creation. I am getting "Peer did not create remote endpoint for link, target" exception when I tried to create MessageProducer. I used different versions of qpid-amqp-1-0(0.20 to 0.32) but the error is same.
I put some debug statements in source code of qpid-amqp and I observed SendingLinkEndPoint Target became null after few secs.
End point target: Target{address=testnamespace/testqueue}
End point target: Target{address=testnamespace/testqueue}
End point target: Target{address=testnamespace/testqueue}
End point target: Target{address=testnamespace/testqueue}
Attach{name=testnamespace/testqueue}
End point target: null
javax.jms.JMSException: Peer did not create remote endpoint for link, target: testnamespace/testqueue
RECV: com.microsoft:timeout\xa1\xbcThe operation did not complete within the allocated time 00:00:15.0675072 for object connection
I made sure the user got domain suffix as per below post
Connecting to Service Bus on Windows Server (1.1) using Java and AMQP 1.0
I enabled qpid jms debug in log4j.properties but I dont see any debug statements displayed on the console. Not sure what else I need to do to see what other user oberved(in above post).
log4j.rootLogger=TRACE, stdout
log4j.logger.org.apache.qpid.jms=DEBUG
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d [%-15.15t] - %-5p%-30.30c{1} - %m%n
the test code looks like below
String connectionString = "amqps://" + encode(userName) + ":" + encode(password) + "#" + fqdn;
Hashtable<String, String> env = new Hashtable<String, String>();
env.put(Context.INITIAL_CONTEXT_FACTORY, "org.apache.qpid.amqp_1_0.jms.jndi.PropertiesFileInitialContextFactory");
env.put(Context.PROVIDER_URL, "blah.txt");
env.put("connectionfactory.ServiceBusConnectionFactory", connectionString);
Context context = null;
ConnectionFactory connectionFactory = null;
Connection connection = connectionFactory.createConnection();
System.setProperty("javax.net.ssl.trustStore","C:\\Program Files (x86)\\Java\\jre1.8.0_111\\lib\\security\\cacerts");
System.setProperty("javax.net.ssl.trustStorePassword",pwd);
Session session = null;
MessageProducer producer = null;
try
{
System.out.println("Creating session\n");
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
System.out.println("Creating Queue Impl\n");
QueueImpl queueImpl = QueueImpl.createQueue("testnamespace/testqueue");
System.out.println("Creating producer\n");
producer = session.createProducer(queueImpl);
}
catch (Exception e)`enter code here`
{
System.out.println("Exception creating session/producer\n");
return;
}
All the required ports are opened and enabled debug/trace at windows service level in event viewer but I am unable to identify whats the problem.
Any help would be greatly appreciated.
Thank you,
S.

Hiveserver2 not responding

I have a hiveserver(hiveserver2) running on port 10000.
If I run command:
netstat -nl | grep 10000
I get:
tcp 0 0 0.0.0.0:10000 0.0.0.0:* LISTEN
so the server is up and running.
My hive-site.xml settings:
<property>
<name>hive.server2.thrift.port</name>
<value>10000</value>
</property>
My code:
public class ThriftAgent {
private static final String HOST = "localhost";
private static final int PORT = 10000;
public static void main(String[] args) throws Exception {
TSocket transport = new TSocket(HOST, PORT);
transport.open();
TBinaryProtocol protocol = new TBinaryProtocol(transport);
Client client = new ThriftHive.Client(protocol);
client.execute("show tables");
final List<String> results = client.fetchAll();
for (String result : results) {
System.out.println(result);
}
transport.close();
}
}
I have tried different URL combos but it freezes at client.execute() and does not go any further than that. It does not throw any exceptions either.
I have also tried to disable authentication but that did not help either as per thread
Requests hang when using Hiveserver2 Thrift Java client
If I connect through JDBC to same host it works.
Also if I start HiveServer (not hiveserver2) it works so something is fishy with hiveserver2.

Well, with no error output to guide us, it could be several things.
It's been a while since I set up a Hive2 Server, but, you may want to define the IP address(or host) in the hive-site.xml using the hive.server2.thrift.bind.host property.
If you set the property above to 'localhost' you'll need to make certain the /etc/hosts file is set to resolve you properly, and if it's on another machine and you use the name, same thing. I would recommend testing with an IP address, and then move to name.
The comment requesting more information is a good one though, there is not much to go on here. What version of Hive2 are you using? What Hadoop distro? There will be differences affecting your solution depending on the answers.

Check if you have IPv6 disabled or in your environment settings make sure you add this to your java options:
-Djava.net.preferIPv4Stack=true
For the first one you should set this parameters on your /etc/sysctl.conf file:
#disable ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
After a reboot check if it's disabled with:
cat /proc/sys/net/ipv6/conf/all/disable_ipv6
For the second one you can add this in you hadoop-env.sh
export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true
or in your .bashrc file:
alias java="java -Djava.net.preferIPv4Stack=true"

Log4j2 SecureTcpSocketServer Setup

I'm trying to set up a log4j2 SecureTcpSocketServer that will receive logs from a remote application (logging with log4j2). I've already prototyped this using the TcpSocketServer, which can be run from the command line by providing a port number and configuration file. The SecureTcpSocketServer however requires additional information to construct, such as a 'LogEventBridge' object and a 'SslConfiguration' object. I don't see how I could provide these objects to the command line so I decided to write a Java program to initialize and run the server. Here is my code:
public class log4j2SocketServer {
public static void main(String[] args) {
KeyStoreConfiguration keyStoreConf = null;
TrustStoreConfiguration trustStoreConf = null;
SecureTcpSocketServer<ObjectInputStream> myServer = null;
try {
keyStoreConf = new KeyStoreConfiguration("keystore.jks", "password", null, null);
} catch (StoreConfigurationException e) {
e.printStackTrace();
}
try {
trustStoreConf = new TrustStoreConfiguration("keystore.jks", "password", null, null);
} catch (StoreConfigurationException e) {
e.printStackTrace();
}
SslConfiguration sslConf = SslConfiguration.createSSLConfiguration("SSL", keyStoreConf, trustStoreConf);
try {
myServer = new SecureTcpSocketServer<ObjectInputStream>(5514,new ObjectInputStreamLogEventBridge(),sslConf);
} catch (IOException e) {
e.printStackTrace();
}
while(true){}
}
}
Using this code, the server seems to start up and listen on port 5514 (I used netstat to verify). On the client side, my log4j2 application seems to connect, but I don't see any of the logs coming through. Here is my client code:
Xml Configuration file:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn" name="remoteLogging" packages="">
<Appenders>
<Socket name="socket" host="xx.xx.xx.xx" port="5514">
<SerializedLayout />
<SSL>
<KeyStore location="keystore.jks" password="password"/>
<TrustStore location="keystore.jks" password="password"/>
</SSL>
</Socket>
</Appenders>
<Loggers>
<Root level="error">
<AppenderRef ref="socket"/>
</Root>
</Loggers>
</Configuration>
Here is my Java code:
public class remoteLogging {
static final Logger log = LogManager.getLogger(remoteLogging.class.getName());
public static void main(String[] args) {
log.debug("Hello this is a debug message");
log.info("Hello this is an info message");
log.warn("Hello this is a warn message");
log.error("Hello this is a error message");
}
}
My primary concern is the way I wrote my server code. When I started writing the code, I figured all I'd need to do is instantiate a SecureTcpSocketServer. I did this, and then (obviously) the program ran, and then...ended, and garbage collected my server. So then I decided to put a infinite loop in right afterwards, to keep the server alive. I seriously doubt this is the proper way to do this, but I'm fairly new to coding and I'm not sure what the best approach is. My question is, is this implementation okay? I mean there's obviously something going wrong, since I'm not receiving logs...Is this loop causing problems? Or is there something else wrong that I haven't considered?
One more thing: I would expect my client application to run, connect to the server and send its logs, and then terminate. Right now, it seems to connect, but no logs seem to be sent, and then it just hangs until I manually terminate it. This makes me think the server-side infinite loop is interfering somehow?

Looking at the source for SecureTcpSocketServer and its superclass TcpSocketServer, I think just constructing an instance is not enough to start the server. You need to call its run() method to get it to start accepting connections.
In your log4j2SocketServer class, replace while(true){} with myServer.run();. This method will block until active is set to false or until the serverSocket is closed.

where are gwt-log log messages?

I'm trying to use gwt-log on my gwt application but I can't get any log message from this library( I don't know where should I found them) I have added these configuration to my gwt.xml file
<inherits name="com.allen_sauer.gwt.log.gwt-log-DEBUG" />
<extend-property name="log_level" values="DEBUG"/>
<set-property name="log_level" value="DEBUG"/>
<set-property name="log_DivLogger" value="DISABLED" />
and I have modified my entry point class as follows:
public void onModuleLoad() {
Log.setUncaughtExceptionHandler();
DeferredCommand.addCommand(new Command() {
public void execute() {
onModuleLoad2();
}
});
}
private void onModuleLoad2() {
if (!Log.isLoggingEnabled()) {
Window.alert("Logging is disabled. No log messages will appears.");
}
Log.trace("This is a 'TRACE' test message");
Log.debug("This is a 'DEBUG' test message");
Log.info("This is a 'INFO' test message");
Log.warn("This is a 'WARN' test message");
Log.error("This is a 'ERROR' test message");
Log.fatal("This is a 'FATAL' test message");
}
and I can get these messages printed on console but the problem is that I can't get any other message printed either in server side or in client side,So is there something I'm missing?
Thanks

You should be able to see them in the development console (Jetty container wrap), in hosted mode. You can find a log of all HTTP traffic + gwt logs for a specific "host".
I'm not exactly sure how you can forward them to write to an external destination for production mode, though you should be able.

Make sure you setup the required servlet in web.xml.
As a side note, if you are using GWT 2.1 or higher, you can start using the new Logging framework, which emulates Java's built-in logging framework.