Develop Reference

selenium-hub exit status 137

I used Docker for Selenium Grid Hub and Nodes to collect data, including
selenium/hub: 3.141.59-iron
selenium/node-fire_fox 3.141.59-iron
but one day after running, selenium-hub hung the following exception:
INFO exited: selenium-hub (exit status 137; not expected)
selenium/node-fire_fox exception is:
INFO [SelfRegisteringRemote$1.run] - Couldn't register this node: The hub is down or not responding: Failed to connect to selenium-hub/172.24.0.2:4444
What should I do?

This is not a problem related to Selenium, but an issue with Docker.
The following link has a summary of root causes for the error exit status 137; not expected:
https://success.docker.com/article/what-causes-a-container-to-exit-with-code-137
In short, for some reason, Docker is killing the process, in your case, selenium-hub.
For Mac and Windows, you probably need to increase the memory dedicated to Docker in Docker preferences.
This answer here on SO has a screenshot showing how to do it.

if you are using docker I dont understand why are you leaving you hub up and running .
checkout their hub page : https://github.com/SeleniumHQ/docker-selenium/tree/master/Hub
hub launch : docker run -d -p 4444:4444 --name selenium-hub selenium/hub
nodes launch :
$ docker run -d --link selenium-hub:hub -v /dev/shm:/dev/shm selenium/node-chrome
$ docker run -d --link selenium-hub:hub -v /dev/shm:/dev/shm selenium/node-firefox
I would write a pipeline and define my hubimage and node images which I bring up at the beginning of the test session and remove the images at the end of the session.
try{
hubimage
nodeimage
runtests
} catch(anything){
throw anything;
} finally {
sh "docker rmi \$(docker inspect --format='{{range .RepoTags}} {{.}} {{end}}' ${hubimage});"
sh "docker rmi \$(docker inspect --format='{{range .RepoTags}} {{.}} {{end}}' ${nodeimage});"
}
see more informations about pipelines : https://jenkins.io/doc/book/pipeline/

PIVOTAL GPDB- External table gphdfs protocol command ended with error. sh: java: command not found

We have small array of Greenplum database.
When trying to read External table in it.
Getting error
proddb=# select count(*) from ext_table;
ERROR: external table gphdfs protocol command ended with error. sh: java: command not found (seg0 slice1 sdw:
40000 pid=8675)
DETAIL:
Command: 'gphdfs://path/to/hdfs External table revenuereport_stg0, file gphdfs://Path/to/hdfs
We tried :
Checked Java env on greenplum master host.
Also checked , Setting up - the parameters for GPDB
[gpadmin#admin ~]$ gpconfig -c gp_hadoop_home -v "'/usr/lib/gphd'"
[gpadmin#admin ~]$ gpconfig -c gp_hadoop_target_version -v "'gphd-2.0'"
But it is failing with this error
[gpadmin#mdw ~]$ gpconfig -c gp_hadoop_home -v "'/usr/lib/gphd'"
20170123:02:02:04:017762 gpconfig:mdw:gpadmin-[ERROR]:-failed updating the postgresql.conf files on host: sdw
20170123:02:02:04:017762 gpconfig:mdw:gpadmin-[ERROR]:-failed updating the postgresql.conf files on host: mdw
20170123:02:02:09:017762 gpconfig:mdw:gpadmin-[ERROR]:-finished with errors
Therefore ,Test for HDFS Access from greenplum host is not working.
Checked if HDFS is accessible from any of the segment servers
[gpadmin#sdw1 ~]$hdfs dfs -ls hdfs://hdm2:8020/
Any help on it would bemuch appreciated !

It looks like a path issue to me .Please set right JAVA_HOME in hadoop-env.sh file
Also ,Please have a look into the following articles for better understanding on configuring gphdfs with gpdb .
https://discuss.pivotal.io/hc/en-us/articles/202635496-How-to-access-HDFS-data-via-GPDB-external-table-with-gphdfs-protocol
https://discuss.pivotal.io/hc/en-us/articles/203083906-Understanding-GPHDFS-Configurations
https://discuss.pivotal.io/hc/en-us/articles/221492507-One-time-HDFS-Protocol-Installation-for-GPHDFS-access-to-HDP-2-x-cluster
Thanks
Pratheesh Nair

export JAVA_HOME=/usr/local/jdk18
export HADOOP_HOME=/opt/apps/hadoop
export GP_JAVA_OPT='-Xmx1000m -XX:+DisplayVMOutputToStderr'
export PATH=$JAVA_HOME/bin:$PATH
export KRB5CCNAME=$GP_SEG_DATADIR/gpdb-gphdfs.krb5cc
JAVA=$JAVA_HOME/bin/java
java_home 和 hadoop_home 要给具体数值，置于最前面，写成从环境变量获取JAVA_HOME=$JAVA_HOME，GP处理时获取会为空值。

Error registering PhantomJS node in Selenium Grid

I have the following problem
I successfully launched Selenium Grid hub with:
java -jar selenium-server-standalone-2.53.0.jar -role hub
After that I tried to launch PhantomJS node with:
phantomjs --webdriver=8090 --webdriver-selenium-grid-hub=http://localhost:4444
but got Error:
[INFO - 2016-03-25T13:56:28.397Z] GhostDriver - Main - running on
port 8090 [INFO - 2016-03-25T13:56:28.397Z] GhostDriver - Main -
registering to Selenium HUB 'http://localhost:4444' version: using
'127.0.0.1:8090' with
org.openqa.grid.selenium.proxy.DefaultRemoteProxy as remote proxy.
[ERROR - 2016-03-25T13:56:28.400Z] GhostDriver - main.fail -
{"line":97,"sourceURL":"phantomjs://platform/hub_register.js","stack":"register#phantomjs://platform/hub_register.js:97:79\nglobal
code#phantomjs://code/main.js:78:37"}
phantomjs://platform/console++.js:263 in error
Standard node works ok, but PhantomJS failes. What I'm doing wrong?

For anyone still looking, there are linux and windows x64 builds available here with the workaround of v2.1.1 (windows build is on a pull request - the hex editor workaround wont work for windows cos the .exe's are compressed)
https://github.com/mrorgues/PhantomJSCustomEdition

workaround***
d/l source from phantomjs http://phantomjs.org/build.html and edit hub_register.js in the src/ghostdriver
comment out lines as shown below:
//var ghostdriver = ghostdriver || {};
return {
capabilities: [{
browserName: "phantomjs",
version: version,
//platform: ghostdriver.system.os.name + '-' + ghostdriver.system.os.version + '-' + ghostdriver.system.os.architecture,
maxInstances: 1,
seleniumProtocol: "WebDriver"
compile and everything works!
shortcut:
https://github.com/darwin10/phantomjs
Original Source : https://github.com/detro/ghostdriver/issues/394

Try replacing localhost with local IP
phantomjs --webdriver=8090 --webdriver-selenium-grid-hub=http://localhost:4444
with
phantomjs --webdriver=8090 --webdriver-selenium-grid-hub=http://127.0.0.1:4444

You can override part of binary file with the dd command.
I'm going to replace platform: with //atform::
$ cp /usr/local/bin/phantomjs .
$ ./phantomjs --version
2.1.1
$ xxd -c 16 phantomjs | grep -C 2 'platform:'
01c6aaa0: 7273 696f 6e3a 2076 6572 7369 6f6e 2c0a rsion: version,.
01c6aab0: 2020 2020 2020 2020 2020 2020 2020 2020
01c6aac0: 706c 6174 666f 726d 3a20 6768 6f73 7464 platform: ghostd
01c6aad0: 7269 7665 722e 7379 7374 656d 2e6f 732e river.system.os.
01c6aae0: 6e61 6d65 202b 2027 2d27 202b 2067 686f name + '-' + gho
$ printf '//' | dd of=phantomjs conv=notrunc bs=1 seek=$((0x01c6aac0+0))
2+0 records in
2+0 records out
2 bytes transferred in 0.000117 secs (17085 bytes/sec)
$ xxd phantomjs | grep 'atform:'
01c6aac0: 2f2f 6174 666f 726d 3a20 6768 6f73 7464 //atform: ghostd
$ ./phantomjs --version
2.1.1
$ mv phantomjs /usr/local/bin/
If you cannot locate the string, try to increase octets per line (xxd -c 256).

I have spent 2 months trying to fix this problem, trying everything possible, my main issue was that many solutions won't connect to the grid, and ones that do are in docker containers what can only connect to a local grid hosted on the same Docker host.
So... I finally built my own solution which works:
https://github.com/madhavajay/selenium-node-phantomjs
Docker image is madhavajay/selenium-node-phantomjs
It uses PhantomJS 2.1.1 Custom Build with fixed Ghostdriver and then my own tweaks to the latest fork of Ghostdriver to allow a custom --remoteHost param all wrapped up in a Docker Container with configurable Environment Variables.
I hope this saves someone the lost weeks I suffered.

I had the same issue. Instead of downloading the source code and compiling, as suggested by Ojen G., I used a fix posted by neuro-sys to make the change in the executable. My OS is Ubuntu 16.04.
Download replacestrings.c program.
wget https://gist.githubusercontent.com/neuro-sys/3bf00b6cf28a93e07e44/raw/52f715fd49fbd271a9450b61d5cd3cf29907a5a0/replacestring.c
Compile program
gcc replacestring.c -o replacestring
Replace Java source code string in binary, creating fixed binary.
./replacestring "$(strings /usr/lib/phantomjs/phantomjs | grep "platform: ghostdriver")" " " < /usr/lib/phantomjs/phantomjs > f_phantomjs
Make binary executable; and put as replacement
chmod +x f_phantomjs
sudo chown root:root f_phantomjs
sudo mv f_phantomjs /usr/lib/phantomjs
cd /usr/lib/phantomjs
sudo echo "Fixed error in phantomjs; old file in e_phantomjs" > README.txt
sudo mv phantomjs e_phantomjs
sudo ln f_phantomjs phantomjs
After these steps, it runs fine (Selenium hub already running):
phantomjs --webdriver=8080 --webdriver-selenium-grid-hub=http://localhost:4444
[INFO - 2016-05-07T16:33:35.534Z] GhostDriver - Main - running on port 8080
[INFO - 2016-05-07T16:33:35.534Z] GhostDriver - Main - registering to Selenium HUB 'http://localhost:4444' version: using '127.0.0.1:8080' with org.openqa.grid.selenium.proxy.DefaultRemoteProxy as remote proxy.
[INFO - 2016-05-07T16:33:35.604Z] HUB Register - register - Registered with grid hub: http://localhost:4444/ (ok)

Following this gist:
https://gist.github.com/neuro-sys/41e368839a9b20dafb34
Editing the binary to put a // in front of the offending line worked for me, but not the string replace thing.
-> Follow the part about using hdex
Search for platform:
and replace with //atform:
by replacing the letters pl with // (2F2F in hex)

Munin jmx configuration

I am trying to enable JMX monitoring on Munin
I have followed the guide at:
https://github.com/munin-monitoring/contrib/tree/master/plugins/java/jmx
It tells me:
1) Files from "plugin" folder must be copied to /usr/share/munin/plugins (or another - where your munin plugins located)
2) Make sure that jmx_ executable : chmod a+x /usr/share/munin/plugins/jmx_
3) Copy configuration files that you want to use, from "examples" folder, into /usr/share/munin/plugins folder
4) create links from the /etc/munin/plugins folder to the /usr/share/munin/plugins/jmx_
The name of the link must follow wildcard pattern:
jmx_<configname>,
where configname is the name of the configuration (config filename without extension), for example:
ln -s /usr/share/munin/plugins/jmx_ /etc/munin/plugins/jmx_process_memory
I have done exatly this but whern i run ./jmx_process_memory, I just get:
Error: Could not find or load main class org.munin.plugin.jmx.memory
The actual config file is called java_process_memory.conf, so i have also tried naming the symlink jmx_java_process_memory, but get the same error.
I have had success by naming the symlink jmx_Threads as described here:
http://blog.johannes-beck.name/?p=160
I can see that org.munin.plugin.jmx.Threads is the name of a class within munin-jmx-plugins.jar, and the other classes seem to work also. But this is not what the Munin guide tells me to do, so is the documentation wrong? What is the purpose of the config files, they must be there for a reason? There are example config files for Tomcat, which is where my real interest lies, so I need to understand this. Without being able the get it working as per the guide though im a bit stuck!
Can anyone put me right on this?
Cheers
NFV

I was stuck with somehow the same issue.
What i did to get something working a little bit better but still not perfectly.
I'm on RHEL :
[root#bus|in plugins]# cat /etc/munin/plugin-conf.d/munin-node
[diskstats]
user munin
[iostat_ios]
user munin
[jmx_*]
env.ip 192.168.1.101
env.port 5054 <- being the port configured for your jmx
then
[root#bus|in plugins]# ls -l /etc/munin/plugins/jmx_MultigraphAll
lrwxrwxrwx 1 root root 29 14 mars 15:36 /etc/munin/plugins/jmx_MultigraphAll -> /usr/share/munin/plugins/jmx_
and I modified the /usr/share/munin/plugins/jmx_ with the following :
#!/bin/sh
# -*- sh -*-
: << =cut
=head1 NAME
jmx_ - Wildcard plugin to monitor Java application servers via JMX
=head1 APPLICABLE SYSTEMS
Tested with Tomcat 4.1/5.0/5.5/6.0 on Sun JVM 5/6 and OpenJDK.
Any JVM that supports JMX should in theory do.
Needs nc in path for autoconf.
=head1 CONFIGURATION
[jmx_*]
env.ip 127.0.0.1
env.port 5400
env.category jvm
env.username monitorRole
env.password SomethingSecret
env.JRE_HOME /usr/lib/jvm/java-6-sun/jre
env.JAVA_OPTS -Xmx128m
Needed configuration on the Tomcat side: add
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=5400 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false
to CATALINA_OPTS in your startup scripts.
Replace authenticate=false with
-Dcom.sun.management.jmxremote.password.file=/etc/tomcat/jmxremote.password \
-Dcom.sun.management.jmxremote.access.file=/etc/tomcat/jmxremote.access
...if you want authentication.
jmxremote.password:
monitorRole SomethingSecret
jmxremote.access:
monitorRole readonly
You may need higher access levels for some counters, notably ThreadsDeadlocked.
=head1 BUGS
No encryption supported in the JMX connection.
The plugins available reflect the most interesting aspects of a
JVM runtime. This should be extended to cover things specific to
Tomcat, JBoss, Glassfish and so on. Patches welcome.
=head1 AUTHORS
=encoding UTF-8
Mo Amini, Diyar Amin and Younes Hajji, Høgskolen i Oslo/Oslo
University College.
Shell script wrapper and integration by Erik Inge Bolsø, Redpill
Linpro AS.
Previous work on JMX plugin by Aleksey Studnev. Support for
authentication added by Ingvar Hagelund, Redpill Linpro AS.
=head1 LICENSE
GPLv2
=head1 MAGIC MARKERS
#%# family=auto
#%# capabilities=autoconf suggest
=cut
MUNIN_JAR="/usr/share/java/munin-jmx-plugins.jar"
if [ "x$JRE_HOME" != "x" ] ; then
JRE=$JRE_HOME/bin/java
export JRE_HOME=$JRE_HOME
fi
JAVA_BIN=${JRE:-/opt/jdk/jre/bin/java}
ip=${ip:-192.168.1.101}
port=${port:-5054}
if [ "x$1" = "xsuggest" ] ; then
echo MultigraphAll
exit 0
fi
if [ "x$1" = "xautoconf" ] ; then
NC=`which nc 2>/dev/null`
if [ "x$NC" = "x" ] ; then
echo "no (nc not found)"
exit 0
fi
$NC -n -z $ip $port >/dev/null 2>&1
CONNECT=$?
$JAVA_BIN -? >/dev/null 2>&1
JAVA=$?
if [ $JAVA -ne 0 ] ; then
echo "no (java runtime not found at $JAVA_BIN)"
exit 0
fi
if [ ! -e $MUNIN_JAR ] ; then
echo "no (munin jmx classes not found at $MUNIN_JAR)"
exit 0
fi
if [ $CONNECT -eq 0 ] ; then
echo "yes"
exit 0
else
echo "no (connection to $ip:$port failed)"
exit 0
fi
fi
if [ "x$1" = "xconfig" ] ; then
param=config
else
param=Tomcat
fi
scriptname=${0##*/}
jmxfunc=${scriptname##*_}
prefix=${scriptname%_*}
if [ "x$jmxfunc" = "x" ] ; then
echo "error, plugin must be symlinked in order to run"
exit 1
fi
ip=$ip port=$port $JAVA_BIN -cp $MUNIN_JAR $JAVA_OPTS org.munin.plugin.jmx.$jmxfunc $param $prefix
And you have to add the right permissions and owner:group on what you define as the JRE, for example here :
[root#bus|in plugins]# ls -ld /opt/jdk
drwxrwxr-x 8 nobody nobody 4096 8 oct. 15:03 /opt/jdk
Now I can run (and I can see it's using nobody:nobody as user:group, maybe something to play with in the conf) :
[root#bus|in plugins]# munin-run jmx_MultigraphAll -d
# Processing plugin configuration from /etc/munin/plugin-conf.d/df
# Processing plugin configuration from /etc/munin/plugin-conf.d/fw_
# Processing plugin configuration from /etc/munin/plugin-conf.d/hddtemp_smartctl
# Processing plugin configuration from /etc/munin/plugin-conf.d/munin-node
# Processing plugin configuration from /etc/munin/plugin-conf.d/postfix
# Processing plugin configuration from /etc/munin/plugin-conf.d/sendmail
# Setting /rgid/ruid/ to /99/99/
# Setting /egid/euid/ to /99 99/99/
# Setting up environment
# Environment ip = 192.168.1.101
# Environment port = 5054
# About to run '/etc/munin/plugins/jmx_MultigraphAll'
multigraph jmx_memory
Max.value 2162032640
Committed.value 1584332800
Init.value 1613168640
Used.value 473134248
multigraph jmx_MemoryAllocatedHeap
Max.value 1037959168
Committed.value 1037959168
Init.value 1073741824
Used.value 275414584
multigraph jmx_MemoryAllocatedNonHeap
Max.value 1124073472
Committed.value 546373632
Init.value 539426816
Used.value 197986088
[...]
multigraph jmx_ProcessorsAvailable
ProcessorsAvailable.value 1
Now I'm trying to get it to work for different JVMs on the same host, because this is for a single one.
I hope that can help you.
edit :
actually I did the modifications to use with several java processes having their own jmx ports.
What you have to add them there :
[root#bus|in plugins]# cat /etc/munin/plugin-conf.d/munin-node
[diskstats]
user munin
[iostat_ios]
user munin
[admin_jmx_*]
env.ip 192.168.1.101
env.port 5054
[managed_jmx_*]
env.ip 192.168.1.101
env.port 5055
[jboss_jmx_*]
env.ip 192.168.1.101
env.port 1616
and then create the links :
[root#bus|in plugins]# ls -l /etc/munin/plugins/*_jmx_*
lrwxrwxrwx 1 root root 29 14 mars 15:36 /etc/munin/plugins/admin_jmx_MultigraphAll -> /usr/share/munin/plugins/jmx_
lrwxrwxrwx 1 root root 29 14 mars 16:51 /etc/munin/plugins/jboss_jmx_MultigraphAll -> /usr/share/munin/plugins/jmx_
lrwxrwxrwx 1 root root 29 14 mars 16:03 /etc/munin/plugins/managed_jmx_MultigraphAll -> /usr/share/munin/plugins/jmx_
and I commented out the ip and port from the /usr/share/munin/plugins/jmx_ file, but I'm not sure it plays a role.

Solr Configuration

I tried to installed Solr using:
java -jar start.jar
However I downloaded the source code and didn't compile it (Didn't pay attention). And the error was:
http://localhost:8983/solr/admin/
HTTP ERROR: 404
Problem accessing /solr/admin/. Reason:
NOT_FOUND
Then I downloaded the compiled version of solr but when trying to run the example configuration I'm getting exception:
java.net.BindException: Address already in use
Is there a way to revert solr configuration and start from scratch? Looks like the configuration got messed up. I don't see anything related to it in the manual.
Here is the error:
2011-07-10 22:41:27.631:WARN::failed SocketConnector#0.0.0.0:8983: java.net.BindException: Address already in use
2011-07-10 22:41:27.632:WARN::failed Server#c4e21db: java.net.BindException: Address already in use
2011-07-10 22:41:27.632:WARN::EXCEPTION
java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:383)
at java.net.ServerSocket.bind(ServerSocket.java:328)
at java.net.ServerSocket.<init>(ServerSocket.java:194)
at java.net.ServerSocket.<init>(ServerSocket.java:150)
at org.mortbay.jetty.bio.SocketConnector.newServerSocket(SocketConnector.java:80)
at org.mortbay.jetty.bio.SocketConnector.open(SocketConnector.java:73)
at org.mortbay.jetty.AbstractConnector.doStart(AbstractConnector.java:283)
at org.mortbay.jetty.bio.SocketConnector.doStart(SocketConnector.java:147)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.Server.doStart(Server.java:235)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.xml.XmlConfiguration.main(XmlConfiguration.java:985)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.mortbay.start.Main.invokeMain(Main.java:194)
at org.mortbay.start.Main.start(Main.java:534)
at org.mortbay.start.Main.start(Main.java:441)
at org.mortbay.start.Main.main(Main.java:119)
Jul 10, 2011 10:41:27 PM org.apache.solr.core.SolrCore registerSearcher
INFO: [] Registered new searcher Searcher#5b6b9e62 main

This means you already have an application running on that particular port.
Run:
$ lsof -i :8983
This gives you a list of any application running on that port. In my case, Solr is already running, and I get back:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 10289 patricia 111u IPv6 399410 0t0 TCP *:8983 (LISTEN)
Kill this process by filling in your PID:
$ kill 10289
And then try running Solr again.

The java.net.BindException means that you are attempting to restart solr while an earlier instance continues to run, or less probably that you have something else running on port 8983. You should find that process, kill it, and then start solr again.

Its bound to a some other application. In case if its a important app you can change jetty default port using following:
java -Djetty.port=8181 -jar start.jar

ps aux | grep solr
OR
ps aux | grep start.jar
AND the get then process id and kill it:
kill -9 #PID#
example : kill -9 4989
UPDATE:
after killing the process if you want to reinstall solr you soul first uninstall it, the following is one of the solutions to uninstall it:
sudo service solr stop
sudo rm -r /var/solr
sudo rm -r /opt/solr-5.3.1
sudo rm -r /opt/solr
sudo rm /etc/init.d/solr
sudo deluser --remove-home solr
sudo deluser --group solr
An now you can reinstall it with no problem.

If sudo lsof -i:8983 won't help finding application running on the same port, the common mistake is Tomcat misconfiguration (if you're using it).
For example by default Tomcat listens on port 8005 for SHUTDOWN command and if you set another Connector to listen on the same port, you'll get port conflict.
So please double check in server.xml if these ports are different:
<Server port="8005" shutdown="SHUTDOWN">
<Connector port="8983" protocol="HTTP/1.1"

Maybe some crazy idea is to use docker to read a complete extended step by step and repeatable installation:
Here dockerhub to select the specific version tu run in docker Docker Hub Solr
And here github to read the docker recipe Solr docker Recipe