Develop Reference

Apache 2.4.6 Https Page slowness issue

I configured Apache 2.4 server, which used as a proxy server to my tomcat. There are two(maybe more) applications deployed on my Tomcat & both are accessible only using apache web server IP and port(443-https).
But whenever I tried to access Login.htm page, it took almost 10-15 sec to load a single page. When I tried to log in using username & password, here welcome page took 1 min 20 sec to load.
However, in my local environment (local IP and port(8080-http)) it took 1 sec for login page & 3-4 sec for welcome page. Note - there is no apache web server configured on my local environment.
I tried a lot of performance tuning methods on my apache web server, but none of them working for me. I am not sure what is missing here. Any help appreciated.
Thanks in advance.
Below is my apache server configuration.
Listen 443 https
SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300
SSLRandomSeed startup file:/dev/urandom 512
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
<VirtualHost <APP_SERVER_IP_ADDR>:443>
ServerName <APP_SERVER_IP_ADDR>
ServerAlias <APP_SERVER_IP_ADDR>
ErrorLog /var/log/httpd/ssl_error_log
TransferLog /var/log/httpd/ssl_access_log
LogLevel info
SSLEngine on
SSLProtocol all -SSLv2
SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SEED:!IDEA
SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
<Files ~ "\.(cgi|shtml|phtml|php3?)$">
SSLOptions +StdEnvVars
</Files>
<Directory "/var/www/cgi-bin">
SSLOptions +StdEnvVars
</Directory>
BrowserMatch "MSIE [2-5]" \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0
CustomLog logs/ssl_request_log \
"%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
ProxyRequests Off
SSLProxyEngine On
ProxyPreserveHost On
RewriteEngine On
HostnameLookups off
<Proxy <APP_SERVER_IP_ADDR>:443>
Order deny,allow
Allow from all
</Proxy>
ProxyPass /application1 http://<WEB_SERVER_IP_ADDR>:9660/application1
ProxyPassReverse /application1 http://<WEB_SERVER_IP_ADDR>:9660/application1
ProxyPass /application2 http://<WEB_SERVER_IP_ADDR>:9660/application2
ProxyPassReverse /application2 http://<WEB_SERVER_IP_ADDR>:9660/application2
<Location /proxy/>
ProxyPassReverse /
Order deny,allow
Allow from all
</Location>
Header edit Location ^http://<APP_SERVER_IP_ADDR>/ https://<APP_SERVER_IP_ADDR>/
Timeout 10
ProxyTimeout 10
ProxyBadHeader Ignore
</VirtualHost>

After digging a lot into apache API, I finally found that there is a big problem in my application.
We are trying to implement the 3-Tier architecture in our production system. As we are using apache web server (as a proxy server) from Web-Server to App Server, there are lot of Js, CSS and images files transferred from my application.
When user requesting Login.htm (or accessing any other page) from WebServer, this request will be responded by my app server. As there are lot of Js, CSS and images files are transferred from App to Web, and finally Web to browser, the signle request consumes around 15 seconds.
To overcome this situation, I moved all Js, CSS and images related files to web server. So all HTML related stuff now loading from Web server and not from app server. For this, I just added below lines in my ssl.conf
DocumentRoot "/var/www/html"
ProxyPass /application1/resources !
The path present in DocumentRoot is context path of my Web server, where I created 'application1' directory and added 'resources' directory where all js, css and image files are present.
Now my application take 4-7 seconds to load Login.htm page.

Kerberos how to get principal or client name?

I have a user configured in AD with delegated kerberos ticket:
klist
Ticket cache: FILE:/tmp/krb5cc_527
Default principal: user1#EXAMPLE
Valid starting Expires Service principal 11/27/15 16:28:27 11/28/15 02:28:27 krbtgt/EXAMPLE.com#EXAMPLE.COM
How can I get this 'user1'? On this client domain account (client side jsp? Or server side?) I want to extract this value after button click and pass it back (with backurl) to another java app.
EDIT:
My Apache configuration:
<Location /kerb >
AuthType Kerberos
AuthName "auth-realm"
KrbMethodNegotiate off
KrbMethodK5Passwd off
KrbServiceName HTTP
Krb5Keytab /etc/krb5.keytab
require valid-user
</Location>
ProxyPreserveHost On
ProxyPass /kerb ajp://120.201.131.169:8019/myApp
ProxyPassReverse /kerb ajp://120.201.131.169:8019/myApp
But I received
[Sun Dec 13 18:17:32 2015] [debug] src/mod_auth_kerb.c(1944): [client
126.185.3.202] kerb_authenticate_user entered with user (NULL) and auth_type Kerberos

It depends. In my case I have an Apache server configured with mod_kerb and forwarding http requests to Tomcat by means of AJP.
In such scenario, Tomcat AJP conector is configured with tomcatAuthentication=false and I can get user authenticated from JSP and Servlet using request.getRemoteUser().
Obviously, the user string comes with domain info after # so you have to consider it.
If you are interested in my solution, I can elaborate my answer.
Edit
I edit my answer to give more info about configuring Tomcat to use Kerberos.
Configure NTP
First, it is quite common to have NTP clients configured in every system AD server, Apache server and Tomcat server. If there is no date and time synchronization, it is quite common to get clock skew too great or postdating problems.
Create an AD principal for the server
You need to create a principal into AD to use for server principal authentication. It is necessary to get a keytab file for this principal. I am sorry, I can't tell you how to do this.
Install and configure Kerberos on Apache server
Once you have your server principal and keytab file, it is time to configure Apache server. Install kerberos into that system and configure /etc/krb5.conf. A sample of this file is:
HERE.YOUR.KERB.DOMAIN = {
kdc = your.dns.kerb.domain
admin_server = your.dns.kerb.domain
}
Check with:
kinit -k -t keytab.file HTTP/principal.dns.name#HERE.YOUR.KERB.DOMAIN
klist
that your server is right configured.
Install and configure mod_auth_kerb
Install mod_auth_kerb apache module and configure its use in every location, directory, virtual host or whatever you need, see below. This configuration is very dependent on your kerberos server, you will have to play with some parameters as KDC verification, negotiation, be or not authoritative, ...
The most important parameter is Krb5Keytab, but you can check this page to understand all parameters. Here you are with a sample location:
<Location /sample/>
AuthType Kerberos
AuthName "auth-realm"
KrbMethodNegotiate on
KrbMethodK5Passwd off
Krb5Keytab /your/path/to/keytab.file
require valid-user
</Location>
When you try to access this location http://your.apache.server/sample Apache will try to check user credentials by means of kerberos.
AJP Configuration
For AJP configuration (I already answer this before, but I cut and paste here and adapt for this question) the procedure is the following:
Install Apache module for AJP, usually it is called something like libapache2-mod-jk. (In debian/ubuntu you can run sudo apt-get install libapache2-mod-jk).
Then you will have a new module calledjk or similar. You have to enable it (In debian/ubuntu you can run sudo a2enmod jk).
Default configuration will serve mostly, open it a see where does JkWorkersFile point. This file is needed to configure the workers that manage communication with tomcat apps.
Create workers file (if it does not exists). A workers file is more or less as following.
Sample workers file:
ps=/
worker.list=worker1,worker2,...
# worker1 definition
worker.worker1.port=8009
worker.worker1.host=host or ip
worker.worker1.type=ajp13
# worker2 definition
....
Every worker can point to different tomcat server. Port must be the same that configured into $CATALINA_HOME/conf/server.xml. In this file there is a connector for AJP protocol:
<Connector port="8009" protocol="AJP/1.3"
redirectPort="8443" tomcatAuthentication="false"/>
Every worker has to point to this port.
Finally, you can configure your location (or whatever) using JkMount workerName to indicate Apache that this url has to be forwarded to the proper worker:
<Location /sample/>
JkMount worker1
AuthType Kerberos
AuthName "auth-realm"
KrbMethodNegotiate on
KrbMethodK5Passwd off
Krb5Keytab /your/path/to/keytab.file
require valid-user
</Location>
There are plenty of samples an documentation. Here you are with Tomcat official docs: https://tomcat.apache.org/connectors-doc/webserver_howto/apache.html
Web app authentication
You don't need to configure anything about security constraint in web.xml, with this configuration, Apache will authenticate users instead Tomcat, and Tomcat will receive user's principal name into HTTP request.
Tomcat (and any other servlet container) will encapsulate the user's principal into request.getRemoteUser().
Hope it helps.

I got it! I don't know what exactly was wrong but now It works.
I can simple get principal from HTTP Header with the following configuration (I moved this configuration from /conf.d to main /conf/httpd.conf file). What is imported. On RHEL httpd server apache user should have rights to read /etc/krb5.keytab. In my case:
ps -ef | grep httpd
apache 27537 27535 0 16:18 ? 00:00:00 /usr/sbin/httpd
<VirtualHost myhost.domain.com:80>
ServerName myhost.domain.com
<Location /myApp >
# SSLRequireSSL
AuthType Kerberos
KrbMethodNegotiate On
KrbMethodK5Passwd Off
KrbServiceName HTTP/nmyhost.domain.com#EXAMPLE.COM
KrbAuthRealms EXAMPLE.COM
Krb5KeyTab /etc/krb5.keytab
require valid-user
RewriteEngine On
RewriteCond %{LA-U:REMOTE_USER} (.+)
RewriteRule . - [E=RU:%1]
Header add X-Remote-User "%{RU}e" env=RU
</Location>
ProxyRequests Off
ProxyPreserveHost On
ProxyPass /myApp ajp://126.101.100.169:8029/myApp
ProxyPassReverse /myApp ajp://126.101.100.169:8029/myApp
</VirtualHost>

Try checking your keytab file. I had similar issue and when i ran
cat httpd.keytab
it showed plain text, which isnt normal, keytab file should contain ASCII chars, its a binary file. Had to regenerate it with Domain Admin account "$user.name" and then it worked.

Multi-tenancy on tomcat

I've set of JSON APIs that are exposed and implemented on Tomcat.
I would like to implement Multi-tenancy for these APIs on Tomcat with the following URL approach:
companyname1.domain.com/api/getUsers...
companyname2.domain.com/api/getUsers...
companyname3.domain.com/api/getUsers...
Let me know if there is a best practice for implementing it using context or other mechanism. I don't want to create a separate Tomcat instance for each and every company.
In addition is there any way to create it dynamically once company is registered.
Thank you in advance,
Moshe

Create several virtual hosts in your Tomcats server.xml, that listen to your domains. These should point to different webapps directories, hosting your particular application in the ROOT directory:
<Host name="localhost" appBase="domain1-webapps" autoDeploy="true" unpackWARs="true"></Host>
<Host name="companyname1.domain.com" appBase="domain1-webapps" autoDeploy="true" unpackWARs="true"></Host>
<Host name="companyname2.domain.com" appBase="domain2-webapps" autoDeploy="true" unpackWARs="true"></Host>
...

This can be done using multiple reverse proxies that provide different parameters to the Tomcat webapp. The simplest setup (with Apache HTTP and mod_proxy_ajp) would probably be to preserve the original request's host and resolve that inside the web app.
<VirtualHost *:80>
ServerName companyname1.domain.com
ProxyPass / ajp://localhost:8009/
ProxyPassReverse / ajp://localhost:8009/
ProxyPreserveHost On
</VirtualHost>
<VirtualHost *:80>
ServerName companyname2.domain.com
ProxyPass /api ajp://localhost:8009/
ProxyPassReverse / ajp://localhost:8009/
ProxyPreserveHost On
</VirtualHost>
<VirtualHost *:80>
ServerName companyname3.domain.com
ProxyPass / ajp://localhost:8009/
ProxyPassReverse / ajp://localhost:8009/
ProxyPreserveHost On
</VirtualHost>

I can not comment because of my reputation is below than 50.
if URI is same like
companyname1.domain.com/api/getUsers
companyname2.domain.com/api/getUsers
companyname3.domain.com/api/getUsers
then Apache web server found domain companyname1.domain.com and from httpd file, it send call to tomcat application server through AJP connector. But problem is that how can it found which war file to run.
So application name is required in URL
companyname1.domain.com/abc/api/getUsers
companyname2.domain.com/xyz/api/getUsers
companyname3.domain.com/fgf/api/getUsers

We can use server context setting. As per tomcat server specification.
The Host element represents a virtual host, which is an association of a network name for a server (such as "www.mycompany.com")
with the particular server on which Tomcat is running. For clients to be able to connect to a Tomcat server using its network name,
this name must be registered in the Domain Name Service (DNS) server that manages the Internet domain you belong to - contact your Network Administrator
for more information.
In many cases, System Administrators wish to associate more than one network name (such as www.mycompany.com and company.com)
with the same virtual host and applications. This can be accomplished using the Host Name Aliases feature discussed below.
One or more Host elements are nested inside an Engine element.
Inside the Host element, you can nest Context elements for the web applications associated with this virtual host.
Exactly one of the Hosts associated with each Engine MUST have a name matching the defaultHost attribute of that Engine.
Clients normally use host names to identify the server they wish to connect to. This host name is also included in the HTTP request headers.
Tomcat extracts the host name from the HTTP headers and looks for a Host with a matching name.
If no match is found, the request is routed to the default host.
The name of the default host does not have to match a DNS name (although it can) since any request where the
DNS name does not match the name of a Host element will be routed to the default host.
For more info. go through this link.
:https://tomcat.apache.org/tomcat-8.0-doc/config/host.html

apache2.4.6 wstunnel proxying to tomcat 7 error unexpected 200 repsonse

I have everything being proxy correctly just that my websockets keep giving me an unexpected 200 response error. I am using native websockets on chrome and firefox.
my conf file looks like this
<VirtualHost example.com:80>
ServerName example.com
ProxyRequests Off
ProxyPass /websocket/ ws://localhost:8080/XXXXX/websocket/ retry=0
ProxyPassReverse /websocket/ ws://localhost:8080/XXXXX/websocket/ retry=0
ProxyPass / http://localhost:8080/XXXXX/
ProxyPassReverse / http://localhost:8080/XXXXX/
ProxyPassReverseCookiePath /XXXXX /
</VirtualHost>
Here is the exact error, so there is no confusion.
Error during WebSocket handshake: Unexpected response code: 200

I had a similar Problem with modproxy and wstunnel module in apache2 version 2.4.10 on Ubuntu 15.04. I used tcpflow to figure out what happened and saw that my HTTP Proxy worked correctly and sent everything to my nodejs server, but my websocket Proxy directly returned the website at /.
Luckily, i had a working Debian server with nearly the same configuration, and I found out the only difference was the following:
On the non-working server:
<Directory />
Options FollowSymLinks
AllowOverride None
</Directory>
On the working server:
<Directory />
Options FollowSymLinks
AllowOverride None
Require all denied
</Directory>
I don't know why this worked in my case, but I hope adding the Directory directive to your VirtualHost will solve your Problem too.

Sending redirect in Tomcat web application behind a Apache 2 proxy (mod_proxy)

I have a web application on tomcat http://localhost:8080/WebApp/
The I have configrued Apache 2 (mod_proy) so that the web application is directly accessible by localhost with out port and name: e.g http://localhost
<VirtualHost localhost:80>
ProxyPreserveHost On
ProxyPass / http://localhost:8080/WebApp/
ProxyPassReverse / http://localhost:8080/WebApp/
</VirtualHost>
The index.html is shown correctly on http://localhost.
But if a servlet redirects:
#WebServlet(description = "...", urlPatterns = { "/login" })
public class LoginServlet extends HttpServlet
{
#Override
protected void doGet(HttpServletRequest request,
HttpServletResponse response) throws IOException
{
response.sendRedirect("a.html");
}
}
and I use the URL http://localhost/login - I am redirected to http://localhost/WebApp/a.html
How do I get the correct redirect to http://localhost/a.html?

Thanks to Stuart and his link to this blog I found a solution:
Reverse Proxying Tomcat Web Applications Behind Apache
Solution: ProxyPreserveHost must be turned off!
Reason: If it is switched on, the response headers returned by the proxy backend will contain “localhost” or the real domain without the port number (or 80). So the ProxyPassReverse pattern does not match (because of the different port and if another domain name is used, also the domain name will not match).
Config:
<VirtualHost localhost:80>
ProxyPreserveHost Off
ProxyPass / http://localhost:8080/WebApp/
ProxyPassReverse / http://localhost:8080/WebApp/
</VirtualHost>
But this works only via http, not via ajp (I don’t know why).
If you still want to use ajp you could use the following workaround - Let Apache do another redirect after the wrong redirect:
<VirtualHost localhost:80>
ProxyPass /WebApp !
ProxyPass / ajp://localhost:8009/WebApp/
ProxyPassReverse / ajp://localhost:8009/WebApp/
RedirectMatch 301 ^/WebApp/(.*)$ /$1
RedirectMatch 301 ^/WebApp$ /
</VirtualHost>
The ProxyPass /WebApp ! directive is needed to exclude the path from further processing in mod_proxy (because proxy directives are evaluated before redirect directives)
Then the RedirectMatch directives redirect everything stating with /WebApp/... respectively /WebApp to the URL without /WebApp at the beginning.
The only drawback is that you must not have any sub folder named WebApp in your web application

I also had this problem and spent some time on it. I believe that if you change your apache httpd configuration to the following your redirect will work:
<VirtualHost localhost:80>
ProxyPreserveHost On
ProxyPass / http://localhost:8080/WebApp/
ProxyPassReverse / http://localhost/WebApp/
ProxyPassReverseCookiePath /WebApp /
</VirtualHost>
This is because the tomcat response headers will contain the proxy headers (i.e. the Location header is http://localhost/WebApp rather than http://localhost:8080/WebApp) because ProxyPreserveHost is switched On.
As a footnote: This also works with you want to change your webapps context. Say you wanted to change the publicly visible context to context you can use the following:
<VirtualHost localhost:80>
ProxyPreserveHost On
ProxyPass /context/ http://localhost:8080/WebApp/
ProxyPassReverse /context/ http://localhost/WebApp/
ProxyPassReverseCookiePath /WebApp /context
</VirtualHost>
For reference, I found this blog post extremely helpful: Reverse Proxying Tomcat Web Applications Behind Apache

you have use to AJP Connector to connect apache2 & tomcat , it will be the perfect solutions for this.
if you need how to configure this, tell me i will explain this detail

Use forwarding instead of redirection
I think your problem is the use of sendRedirect. Calling sendRedirect is actually suppose to show the browser that the URL has been redirected. If you want to hide that you need to use forwarding.In your servlet try this instead of sendRedirect.
String servletPath = request.getServletPath();
if(servletPath.equals("/app1")){
ServletContext ctx = request.getServletContext().getContext("/app1");
RequestDispatcher dispatcher=ctx.getServletContext().getRequestDispatcher( "/app1/app1.html" ); // or wherever you actually keep app1.html
dispatcher.forward( request, response );
}
Inside your context.xml set crossContext = "true" so you can forward requests to other web applications.
<Context crossContext="true" ....../>

I had the same problem while tried to redirect the apache2(running on port 80) request to tomcat(application server running on port 8080).
This is the configuration which is working perfectly.
Go to /etc/apache2/sites-available/000-default.conf and add the following config:
<VirtualHost *:80>
# The ServerName directive sets the request scheme, hostname and port that
# the server uses to identify itself. This is used when creating
# redirection URLs. In the context of virtual hosts, the ServerName
# specifies what hostname must appear in the request's Host: header to
# match this virtual host. For the default virtual host (this file) this
# value is not decisive as it is used as a last resort host regardless.
# However, you must set it for any further virtual host explicitly.
#ServerName www.example.com
# for redirecting the websocket requests
ProxyPass /ws ws://localhost:7681/
#ProxyPass /ws ws://localhost:7681/
ProxyPassReverse /ws ws://localhost:7681/
ServerAdmin webmaster#localhost
DocumentRoot /var/www/html
# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
# error, crit, alert, emerg.
# It is also possible to configure the loglevel for particular
# modules, e.g.
#LogLevel info ssl:warn
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
# For most configuration files from conf-available/, which are
# enabled or disabled at a global level, it is possible to
# include a line for only one particular virtual host. For example the
# following line enables the CGI configuration for this host only
# after it has been globally disabled with "a2disconf".
#Include conf-available/serve-cgi-bin.conf
# for redirecting the http request
ProxyPass /applicationContextUrl ' http://localhost:8080/applicationContextUrl
ProxyPassReverse /applicationContextUrl http://localhost:8080/applicationContextUrl
ProxyPassReverseCookiePath /applicationContextUrl /
ProxyPassReverseCookieDomain localhost applicationContextUrl
ProxyRequests off
ProxyTimeout 15
ErrorLog ${APACHE_LOG_DIR}/nirad_error.log
LogLevel debug
CustomLog ${APACHE_LOG_DIR}/nirad_access.log combined
<Proxy *>
AddDefaultCharset off
Order deny,allow
Allow from all
#Require all denied
Require all granted
Require local
</Proxy>
</VirtualHost>
Done.
Now goto terminal and hit the following command.
sudo a2enmod proxy_http (for http redirection).
sudo a2enmod proxy_wstunnel (for websocket redirection)
and sudo service apache2 restart
run your application server on port 8080