Develop Reference

P2P Hazelcast Session Replication for Tomcat6 Web Clustering - Session Synchronization Fails

I believe I am missing some configuration either on load balancing side or in tomcat, it would great if someone could help and provide the solution.
What am i trying to do?
Setup two tomcat instances with P2P Hazelcast session replication for tomcat6 by referring to this guide
I have placed the hazelcast-all-.jar, hazelcast-tomcat-sessionmanager-.jar and hazelcast.xml in the folder $CATALINA_HOME/lib/
Updated the listener and the context configs as shown below
Updated Tomcat Failover and the jvmRoute Parameter as shown below
So far, i was able to bring up the tomcat instances which connect to each others hazelcast instances, I see that in the tomcat logs.
The users connects to the load balancer and are routed to the balancer members - no issues here.
But when I take one of the tomcat instance down to test failover and to make sure users who were connected to tom instance were able to continue to cat instance with the same session and without having to sign in again. But I believe the load balancer seems to be creating a new session and forces user to sign in again to establish a session with cat instance. I suspect it this but not sure.
Or are the sessions not being replicated and synced in both hazelcast instances? - This is happening
Do we have a client where I could monitor hazelcast sessions? - Now I do
I did refer to this issue which refers setting up hazelcast.sessionId instead of JSESSIONID but does not provide any specifics about how to.
Versions -
Apache Haus - 2.2.32 (for load balancing)
Hazelcast - 3.8.6
Java 8
Tomcat - 6.0.48
Session objects that need to be clustered are Serializable.
stickysession=JSESSIONID
Load Balancer Configuration
<Proxy balancer://mycluster>
BalancerMember http://IPAddress1:8080/app/ route=tom
BalancerMember http://IPAddress2:8080/app/ route=cat
ProxySet lbmethod=byrequests stickysession=JSESSIONID|jsessionid
</Proxy>
ProxyPass /app/ balancer://mycluster/
ProxyPassReverse /app/ balancer://mycluster/
tom instance - server.xml
<Listener className="com.hazelcast.session.P2PLifecycleListener"/>
<Engine name="Catalina" defaultHost="localhost" jvmRoute="tom">
tom instance - context.xml
<Manager className="com.hazelcast.session.HazelcastSessionManager" sticky=true/>
cat instance - server.xml
<Listener className="com.hazelcast.session.P2PLifecycleListener"/>
<Engine name="Catalina" defaultHost="localhost" jvmRoute="cat">
cat instance - context.xml
<Manager className="com.hazelcast.session.HazelcastSessionManager" sticky=true/>
Update:
I believe the below is the same issue I have session sync failure

Although I don't have much experience of using Apache HTTP server as a load balancer, looking at this page, I think you need to update your load balancer configuration as follows:
<Proxy balancer://mycluster>
BalancerMember http://IPAddress1:8080/app/ route=tom
BalancerMember http://IPAddress2:8080/app/ route=cat
ProxySet lbmethod=byrequests stickysession=hazelcast.sessionId
</Proxy>
ProxyPass /app/ balancer://mycluster/
ProxyPassReverse /app/ balancer://mycluster/
Do we have a client where I could monitor hazelcast sessions?
You can use Hazelcast Management Center to monitor the cluster and browse the session entries that are stored in the Hazelcast IMap. All sessions are stored in a Hazelcast IMap keyed by their IDs. Please note that you need to enable Management Center on the members of your cluster before being able to see any data on Management Center.
You can also have a look into the Docker code samples for container based session replication to see a working configuration of session replication. Although there's none that uses Apache as a load balancer, there's one that uses Nginx which might be of help to you.

I got it working with the below configuration. Figured it needed both jsessionid and hazelcast.sessionid, the session replication works great now.
<Proxy balancer://mycluster>
BalancerMember http://IPAddress1:8080/app/ route=tom
BalancerMember http://IPAddress2:8080/app/ route=cat
ProxySet lbmethod=byrequests stickysession=JSESSIONID|jsessionid|hazelcast.sessionId
</Proxy>
ProxyPass /app/ balancer://mycluster/
ProxyPassReverse /app/ balancer://mycluster/

Tunelling websockets calls through Apache on TLS to non TLS Tomcat server

We are using websockets in one of our projects. Our setup has an Apache Web Server and a different server with Tomcat instance. Apache is on TLS but the tomcat instance does not have TLS.
We are trying to tunnel the websockets through the apache (wss) to tomcat instance (ws).
Is this possible ? The initial handshake is successful and we get a 101 Response status. After that when we try to send data through the web socket, it does not reach the tomcat instance.
Any help would be greatly appreciated.
Below is the section of configuration used for websockets from the httpd.config file.
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule ssl_module modules/mod_ssl.so
ProxyPass wss://apache/ws/connect ws://tomcatinstance/wsapp/connect
ProxyPassReverse wss://apache/ws/connect ws://tomcatinstance/wsapp/connect

Apache 2.4.6 Https Page slowness issue

I configured Apache 2.4 server, which used as a proxy server to my tomcat. There are two(maybe more) applications deployed on my Tomcat & both are accessible only using apache web server IP and port(443-https).
But whenever I tried to access Login.htm page, it took almost 10-15 sec to load a single page. When I tried to log in using username & password, here welcome page took 1 min 20 sec to load.
However, in my local environment (local IP and port(8080-http)) it took 1 sec for login page & 3-4 sec for welcome page. Note - there is no apache web server configured on my local environment.
I tried a lot of performance tuning methods on my apache web server, but none of them working for me. I am not sure what is missing here. Any help appreciated.
Thanks in advance.
Below is my apache server configuration.
Listen 443 https
SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300
SSLRandomSeed startup file:/dev/urandom 512
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
<VirtualHost <APP_SERVER_IP_ADDR>:443>
ServerName <APP_SERVER_IP_ADDR>
ServerAlias <APP_SERVER_IP_ADDR>
ErrorLog /var/log/httpd/ssl_error_log
TransferLog /var/log/httpd/ssl_access_log
LogLevel info
SSLEngine on
SSLProtocol all -SSLv2
SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SEED:!IDEA
SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
<Files ~ "\.(cgi|shtml|phtml|php3?)$">
SSLOptions +StdEnvVars
</Files>
<Directory "/var/www/cgi-bin">
SSLOptions +StdEnvVars
</Directory>
BrowserMatch "MSIE [2-5]" \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0
CustomLog logs/ssl_request_log \
"%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
ProxyRequests Off
SSLProxyEngine On
ProxyPreserveHost On
RewriteEngine On
HostnameLookups off
<Proxy <APP_SERVER_IP_ADDR>:443>
Order deny,allow
Allow from all
</Proxy>
ProxyPass /application1 http://<WEB_SERVER_IP_ADDR>:9660/application1
ProxyPassReverse /application1 http://<WEB_SERVER_IP_ADDR>:9660/application1
ProxyPass /application2 http://<WEB_SERVER_IP_ADDR>:9660/application2
ProxyPassReverse /application2 http://<WEB_SERVER_IP_ADDR>:9660/application2
<Location /proxy/>
ProxyPassReverse /
Order deny,allow
Allow from all
</Location>
Header edit Location ^http://<APP_SERVER_IP_ADDR>/ https://<APP_SERVER_IP_ADDR>/
Timeout 10
ProxyTimeout 10
ProxyBadHeader Ignore
</VirtualHost>

After digging a lot into apache API, I finally found that there is a big problem in my application.
We are trying to implement the 3-Tier architecture in our production system. As we are using apache web server (as a proxy server) from Web-Server to App Server, there are lot of Js, CSS and images files transferred from my application.
When user requesting Login.htm (or accessing any other page) from WebServer, this request will be responded by my app server. As there are lot of Js, CSS and images files are transferred from App to Web, and finally Web to browser, the signle request consumes around 15 seconds.
To overcome this situation, I moved all Js, CSS and images related files to web server. So all HTML related stuff now loading from Web server and not from app server. For this, I just added below lines in my ssl.conf
DocumentRoot "/var/www/html"
ProxyPass /application1/resources !
The path present in DocumentRoot is context path of my Web server, where I created 'application1' directory and added 'resources' directory where all js, css and image files are present.
Now my application take 4-7 seconds to load Login.htm page.

Kerberos how to get principal or client name?

I have a user configured in AD with delegated kerberos ticket:
klist
Ticket cache: FILE:/tmp/krb5cc_527
Default principal: user1#EXAMPLE
Valid starting Expires Service principal 11/27/15 16:28:27 11/28/15 02:28:27 krbtgt/EXAMPLE.com#EXAMPLE.COM
How can I get this 'user1'? On this client domain account (client side jsp? Or server side?) I want to extract this value after button click and pass it back (with backurl) to another java app.
EDIT:
My Apache configuration:
<Location /kerb >
AuthType Kerberos
AuthName "auth-realm"
KrbMethodNegotiate off
KrbMethodK5Passwd off
KrbServiceName HTTP
Krb5Keytab /etc/krb5.keytab
require valid-user
</Location>
ProxyPreserveHost On
ProxyPass /kerb ajp://120.201.131.169:8019/myApp
ProxyPassReverse /kerb ajp://120.201.131.169:8019/myApp
But I received
[Sun Dec 13 18:17:32 2015] [debug] src/mod_auth_kerb.c(1944): [client
126.185.3.202] kerb_authenticate_user entered with user (NULL) and auth_type Kerberos

It depends. In my case I have an Apache server configured with mod_kerb and forwarding http requests to Tomcat by means of AJP.
In such scenario, Tomcat AJP conector is configured with tomcatAuthentication=false and I can get user authenticated from JSP and Servlet using request.getRemoteUser().
Obviously, the user string comes with domain info after # so you have to consider it.
If you are interested in my solution, I can elaborate my answer.
Edit
I edit my answer to give more info about configuring Tomcat to use Kerberos.
Configure NTP
First, it is quite common to have NTP clients configured in every system AD server, Apache server and Tomcat server. If there is no date and time synchronization, it is quite common to get clock skew too great or postdating problems.
Create an AD principal for the server
You need to create a principal into AD to use for server principal authentication. It is necessary to get a keytab file for this principal. I am sorry, I can't tell you how to do this.
Install and configure Kerberos on Apache server
Once you have your server principal and keytab file, it is time to configure Apache server. Install kerberos into that system and configure /etc/krb5.conf. A sample of this file is:
HERE.YOUR.KERB.DOMAIN = {
kdc = your.dns.kerb.domain
admin_server = your.dns.kerb.domain
}
Check with:
kinit -k -t keytab.file HTTP/principal.dns.name#HERE.YOUR.KERB.DOMAIN
klist
that your server is right configured.
Install and configure mod_auth_kerb
Install mod_auth_kerb apache module and configure its use in every location, directory, virtual host or whatever you need, see below. This configuration is very dependent on your kerberos server, you will have to play with some parameters as KDC verification, negotiation, be or not authoritative, ...
The most important parameter is Krb5Keytab, but you can check this page to understand all parameters. Here you are with a sample location:
<Location /sample/>
AuthType Kerberos
AuthName "auth-realm"
KrbMethodNegotiate on
KrbMethodK5Passwd off
Krb5Keytab /your/path/to/keytab.file
require valid-user
</Location>
When you try to access this location http://your.apache.server/sample Apache will try to check user credentials by means of kerberos.
AJP Configuration
For AJP configuration (I already answer this before, but I cut and paste here and adapt for this question) the procedure is the following:
Install Apache module for AJP, usually it is called something like libapache2-mod-jk. (In debian/ubuntu you can run sudo apt-get install libapache2-mod-jk).
Then you will have a new module calledjk or similar. You have to enable it (In debian/ubuntu you can run sudo a2enmod jk).
Default configuration will serve mostly, open it a see where does JkWorkersFile point. This file is needed to configure the workers that manage communication with tomcat apps.
Create workers file (if it does not exists). A workers file is more or less as following.
Sample workers file:
ps=/
worker.list=worker1,worker2,...
# worker1 definition
worker.worker1.port=8009
worker.worker1.host=host or ip
worker.worker1.type=ajp13
# worker2 definition
....
Every worker can point to different tomcat server. Port must be the same that configured into $CATALINA_HOME/conf/server.xml. In this file there is a connector for AJP protocol:
<Connector port="8009" protocol="AJP/1.3"
redirectPort="8443" tomcatAuthentication="false"/>
Every worker has to point to this port.
Finally, you can configure your location (or whatever) using JkMount workerName to indicate Apache that this url has to be forwarded to the proper worker:
<Location /sample/>
JkMount worker1
AuthType Kerberos
AuthName "auth-realm"
KrbMethodNegotiate on
KrbMethodK5Passwd off
Krb5Keytab /your/path/to/keytab.file
require valid-user
</Location>
There are plenty of samples an documentation. Here you are with Tomcat official docs: https://tomcat.apache.org/connectors-doc/webserver_howto/apache.html
Web app authentication
You don't need to configure anything about security constraint in web.xml, with this configuration, Apache will authenticate users instead Tomcat, and Tomcat will receive user's principal name into HTTP request.
Tomcat (and any other servlet container) will encapsulate the user's principal into request.getRemoteUser().
Hope it helps.

I got it! I don't know what exactly was wrong but now It works.
I can simple get principal from HTTP Header with the following configuration (I moved this configuration from /conf.d to main /conf/httpd.conf file). What is imported. On RHEL httpd server apache user should have rights to read /etc/krb5.keytab. In my case:
ps -ef | grep httpd
apache 27537 27535 0 16:18 ? 00:00:00 /usr/sbin/httpd
<VirtualHost myhost.domain.com:80>
ServerName myhost.domain.com
<Location /myApp >
# SSLRequireSSL
AuthType Kerberos
KrbMethodNegotiate On
KrbMethodK5Passwd Off
KrbServiceName HTTP/nmyhost.domain.com#EXAMPLE.COM
KrbAuthRealms EXAMPLE.COM
Krb5KeyTab /etc/krb5.keytab
require valid-user
RewriteEngine On
RewriteCond %{LA-U:REMOTE_USER} (.+)
RewriteRule . - [E=RU:%1]
Header add X-Remote-User "%{RU}e" env=RU
</Location>
ProxyRequests Off
ProxyPreserveHost On
ProxyPass /myApp ajp://126.101.100.169:8029/myApp
ProxyPassReverse /myApp ajp://126.101.100.169:8029/myApp
</VirtualHost>

Try checking your keytab file. I had similar issue and when i ran
cat httpd.keytab
it showed plain text, which isnt normal, keytab file should contain ASCII chars, its a binary file. Had to regenerate it with Domain Admin account "$user.name" and then it worked.

Kerberos SSO with Apache and Tomcat under JDK5

I'm new with this authentication through kerberos protocol so I tried to read a lot of howto on it but seems like I can't find any specifics with my constraints. Here is what I have :
An Active Directory Server on which users authenticate to log into their workstations
Each end user uses IE 7 to connect to my intranet application
An Apache server with load balancing
Some Tomcats servers acting as workers for the Apache server.
on each tomcat, I have 2 jakarta servlet running, users connect only on one servlet (further i will call it the servlet as if there is only one)
my tomcats need to run under jdk5. not jdk6 or jdk4. it's jdk5 period.
Now I want one to automatically get logged on my servlet. Basically I just need my servlet to retrieve the client's principal then I can manage the rest.
Based on what I understood, my client has a ticket, he ask the KDC for a special ticket for accessing the apache server, then he tries to connect to the Apache server. Based on his keytab, the apache server then decode the auth data and grant/refuse the access to specified resource.
Am I right? please guide me through this, I've been reading pages for 4 days and still no clue on which solution is the more appropriate. I tried mod_auth_kerberos for Apache but instead of grabbing the user's ticket he ask it like a basic auth. Apparently spgneo
Thanks

Ok I got this working :
Install Kerberos 5 + apache 2 + mod_auth_kerb.
On your AD, generate a keytab with only the principal you will use for Apache, I use HTTP/apache.mydom.com#MYDOM.COM
Put this keytab file on your apache server and make it readable only
by your Apache user.
Then edit your apache conf with these directive for your secure
location
apache.conf:
[…]
ServerName apache.mydom.com:80
[…]
LoadModule auth_kerb_module modules/mod_auth_kerb.so
[…]
<LocationMatch /secure)>
[… some other stuff …]
Order allow,deny
Allow from all
AuthType Kerberos
AuthName "Authentification requise"
KrbAuthRealms MYDOM.COM
#this allows user to be saved in the request
KrbSaveCredentials on
#this one force Negotiate AuthType instead of basic fallback
KrbMethodNegotiate on
#this trim the realm from username saved in the request (request.getRemoteUser() will give you "user" instead of "user#MYDOM.COM"
KrbLocalUserMapping on
KrbAuthoritative on
KrbVerifyKDC on
Krb5Keytab /install/binaries/httpd/apache.keytab
KrbServiceName HTTP
require valid-user
</LocationMatch>
And the one thing I almost failed to find on the web, you have to modify your tomcat server config (tomcat/conf/server.xml) :
<Connector [... AJP connector configuration ...] request.tomcatAuthentication="false"/>
This is really important because without it you tomcat won't retrieve any info from tomcat auth.
Don't forget too, DNS is really really really really important for a Kerberos install. If you have any issue try checking your DNS for all of your servers.