Develop Reference

ONOS Service Start FrameworkEvent Error and GUI not ready yet

I have installed ONOS 2.3.0 on an Ubuntu Server 18.04.4 virtual machine running on Hyper-V following this steps (taken from here and here):
Firstly, I have installed Java 11 (openjdk-11-jdk and openjdk-11-jre), maven and curl;
then I have downloaded ONOS 2.3.0 from here and extracted it with tar xzf onos-2.3.0.tar.gz;
lastly, I exported the required environment variable export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64.
When I try to launch it using the command ./onos-service start (tested both from a normal user and sudo), it gives me the following errors:
21:54:57.869 ERROR [onos-core-net] FrameworkEvent ERROR - org.onosproject.onos-core-net
org.osgi.framework.ServiceException: Service factory returned null. (Component: org.onosproject.store.cfg.DistributedComponentConfigStore (6))
at org.apache.felix.framework.ServiceRegistrationImpl.getFactoryUnchecked(ServiceRegistrationImpl.java:380)
at org.apache.felix.framework.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:247) org.apache.felix.framework.EventDispatcher.fireEventImmediately(EventDispatcher.java:834)
[...]
at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1373)
at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308) at java.base/java.lang.Thread.run(Thread.java:834)
[...]
21:54:57.881 WARN [NettyMessagingService] Failed to bind TCP server to port 0.0.0.0:9876 due to {}
java.net.BindException: Address already in use
at java.base/sun.nio.ch.Net.bind0(Native Method)
[...]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Thread.java:834)
21:54:57.899 ERROR [onos-core-primitives] bundle org.onosproject.onos-core-primitives:2.3.0 (192)[org.onosproject.store.atomix.impl.AtomixManager(115)] : The activate method has thrown an exception
java.util.concurrent.CompletionException: java.net.BindException: Address already in use
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
[...]
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.BindException: Address already in use
at java.base/sun.nio.ch.Net.bind0(Native Method)
at java.base/sun.nio.ch.Net.bind(Net.java:455)
at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227)
at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:132)
at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:563)
... 12 more
Connecting to karaf instance with ssh -p 8101 karaf#localhost confirm that ONOS is working (at least partially), the web interface login loads, but after login it hangs saying that ONOS GUI not ready yet... please stand by....
Does anyone has an idea about how to solve this problem?
Thanks in advance.
UPDATE 19-03-2020: I have prepared another virtual machine following exactly the same steps on another PC using VirtualBox and lower virtual resources assigned, and it works. Honestly i don't understand why it fails on the Hyper-V configuration.
UPDATE 20-03-2020: I have reinstalled Ubuntu configuring the network directly from the installer, and prerequisites and dependecies of ONOS offline (downloaded on another machine via sudo apt install --download-only <package-name>) and it worked. I think the problem was related to something in the network configuration that didn't let him recognize its own process on port 9876 (see the WARN above).
Hope this can be helpful for others.

I had this problem. ONOS is locked to the IP at first install. I grepped for my IP in the /onos folder and was able to reset the binding by deleting the following files that contained the IP. They were rebuilt at next ONOS run.
grep -rl 192.168. --exclude=*.log ~/onos
rm ~/onos/apache-karaf-4.2.9/data/db/partitions/data/partitions/1/raft-partition-1.conf
rm ~/onos/apache-karaf-4.2.9/data/db/partitions/data/partitions/1/raft-partition-1.meta
rm ~/onos/apache-karaf-4.2.9/data/db/partitions/data/partitions/1/.raft-partition-1.lock
rm ~/onos/apache-karaf-4.2.9/data/db/partitions/system/partitions/1/.system-partition-1.lock
rm ~/onos/apache-karaf-4.2.9/data/db/partitions/system/partitions/1/system-partition-1.conf
rm ~/onos/apache-karaf-4.2.9/data/db/partitions/system/partitions/1/system-partition-1.meta

I have faced this issue after changing the IP address of the controller (Host machine).
The quick way to solve it is to set the IP controller as it was (Static)
then reboot your machine
after putting the URL (YourIP:8181/onos/ui/index.html)
Karaf will ask you for login in credentials, use (username:karaf/password:karaf)
then on ONOS's login page, use onos/rocks as credentials.
Good luck..

Kubernetes "fatal alert: protocol_version" error when creating a Deployment

We're running a kubernetes cluster on the Google Cloud Platform, which creates a Deployment with 8 hazelcast-based replicas. We've had this running fine for over a month, but recently, we started receiving the below error message whenever we try to start our deployment (non-relevant stack frames omitted):
2016-07-15 12:58:02,117 [My-hazelcast.my-deployment-368708980-8v7ig # my-deployment-368708980-8v7ig] ERROR - [10.68.5.3]:5701 [MyProject] [3.6.2] Error executing: GET at: https://kubernetes.default.svc/api/v1/namespaces/default/endpoints/my-service. Cause: Received fatal alert: protocol_version
io.fabric8.kubernetes.client.KubernetesClientException: Error executing: GET at: https://kubernetes.default.svc/api/v1/namespaces/default/endpoints/my-service. Cause: Received fatal alert: protocol_version
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestException(OperationSupport.java:272) ~[kubernetes-client-1.3.66.jar:na]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:205) ~[kubernetes-client-1.3.66.jar:na]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:196) ~[kubernetes-client-1.3.66.jar:na]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:483) ~[kubernetes-client-1.3.66.jar:na]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:108) ~[kubernetes-client-1.3.66.jar:na]
at com.noctarius.hazelcast.kubernetes.ServiceEndpointResolver.resolve(ServiceEndpointResolver.java:62) ~[hazelcast-kubernetes-discovery-0.9.2.jar:na]
at com.noctarius.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy.discoverNodes(HazelcastKubernetesDiscoveryStrategy.java:74) ~[hazelcast-kubernetes-discovery-0.9.2.jar:na]
at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.discoverNodes(DefaultDiscoveryService.java:74) ~[hazelcast-all-3.6.2.jar:3.6.2]
....
Caused by: javax.net.ssl.SSLException: Received fatal alert: protocol_version
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[na:1.7.0_95]
at sun.security.ssl.Alerts.getSSLException(Alerts.java:154) ~[na:1.7.0_95]
at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1991) ~[na:1.7.0_95]
...
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:203) ~[kubernetes-client-1.3.66.jar:na]
... 18 common frames omitted
When I google this error, I get a lot of hits about TLS protocol version mismatch. Apparently, Java 8 assumes a different TLS protocol version (TLS 1.2) than Java 7 and 6(TLS 1.0). However, all of our containers run the same docker image (based off of the hazelcast/hazelcast:3.6.2 image), which is based off of Java 7, so there should be no protocol version mismatch (and this layer of our image has not changed).
We've tried to revert all of our recent changes in an attempt to resolve this error, to no avail. And frankly, nobody on our team has changed anything receltly related to SSL or the Hazelcast Kubernetes discovery mechanism. We recently updated our google cloud SDK components (gcloud components update) at the urging of the Cloud SDK tools ("Updates are available for some Cloud SDK components."). We're now running Google Clouds SDK version 117.0.0, but I don't see any breaking changes related to SSL or TLS in the release notes.
Why would we suddenly start seeing this "fatal alert: protocol_version" error message in our kubernetes pods, and how can I resolve it?

The initial google searches indicating this was a TLS version error (version 1.0 vs 1.2 incompatibility) turned out to be useful. This answer to a question about a similar SSLException protocol_version error is what pointed me in the right direction.
I got a test container to run, and using kubectl exec my-test-pod -i -t -- /bin/bash -il to launch an interactive bash shell into the container, I determined that the Hazelcast discovery service could NOT connect using TLS 1.0, but could using TLS 1.2:
/opt/hazelcast# curl -k --tlsv1.0 https://kubernetes.default.svc/api/v1/namespaces/default/endpoints/my-service
curl: (35) error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version
/opt/hazelcast# curl -k --tlsv1.2 https://kubernetes.default.svc/api/v1/namespaces/default/endpoints/my-service
Unauthorized # <-- Unauthorized is expected, as I didn't specify a user/passwd.
I am still not sure what exactly changed, possibly a layer of a public Docker container we use, possibly something within Google cloud service (Java 7 is End of Life, after all), and the fine folks at Hazelcast suggested perhaps the REST API had been updated. But evidently something changed that was causing the discovery service expect clients to TLS version 1.2.
The solution was to download the Hazelcast Docker image we were using, and tweak it to use Java 8 instead of Java 7, and then rebuild the image in our own development sandbox:
$ pwd
/home/jdoe/devel/hazelcast-docker-3.6.2/hazelcast-oss
$ head -n3 Dockerfile
FROM java:8
ENV HZ_VERSION 3.6.2
ENV HZ_HOME /opt/hazelcast/
Voila! Our Deployment is running again.

Apache Connection Refused when running Docker-client Java API

I am trying to install the Docker-client Remote API library ( https://github.com/spotify/docker-client ) to do some image searches and inspect image data (all in public repositories). I have the boot2docker VM downloaded, installed and running. Commands such as "Docker pull ubuntu" work fine but I would like to do this via a Java program now. I used the Eclipse IDE Egit plugin to import the github project and created a Maven/Java project from the existing Master branch. The source code is completely imported and no errors reported. I then tried writing a simple test:
final DockerClient docker = DefaultDockerClient.fromEnv().build();
//docker.pull("busybox");
List<ImageSearchResult> results = docker.searchImages("ubuntu");
for (ImageSearchResult res : results) {
System.out.println(res.getName());
}
However, when running the code in Eclipse I get the following error:
Exception in thread "main" com.spotify.docker.client.DockerException: java.util.concurrent.ExecutionException: javax.ws.rs.ProcessingException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:2375 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: connect
at com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:1109)
at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:1028)
at com.spotify.docker.client.DefaultDockerClient.searchImages(DefaultDockerClient.java:653)
at com.spotify.docker.client.main.Test.main(Test.java:28)
I tried setting up an apache server on that port but then I get:
Exception in thread "main" com.spotify.docker.client.DockerRequestException: Request error: GET http://localhost:2375/v1.12/images/search?term=ubuntu: 404
at com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:1100)
at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:1028)
at com.spotify.docker.client.DefaultDockerClient.searchImages(DefaultDockerClient.java:653)
at com.spotify.docker.client.main.Test.main(Test.java:28)
Can anyone tell me what I am supposed to do here to make my search/pull call work? This is my first try with Docker and I've searched through the documentation and googled the problem but can't find anyone with a similar problem.
Thank you!
EDIT: I am running docker in Windows 7 via the pre-built VM Boot2Docker. Maybe the Docker daemon running inside that is not accessible from programs outside of the VM such as Eclipse?
EDIT: solved it by upgrading to v1.6 instead of v1.5 which makes the daemon available in the Windows host. Current problem is that all my API calls are returning "The server failed to respond with a valid HTTP response"

I encountered a similar issue and I managed to solve this issue by using the following way, to build up the DockerClient:
final DockerClient docker = DefaultDockerClient.builder()
.uri(URI.create("unix:///var/run/docker.sock"))
.build();
I had been getting the same exception but adding the above URI part helped me to solve the issue.
A better explanation for a issue similar to the above and how to solve it has been provided in the following issue tracker.
https://github.com/spotify/docker-maven-plugin/issues/61

The Java program does essentially a docker search: that can only work in an environment where the docker engine is present.
Either in the boot2docker VM.
Or in a full Linux host.

I did encounter the same problem on Mac with eclipse and Docker version 1.10.3, I did search for a solution before I settled for a workaround - Using docker CLI docker-manager to create a new virtualbox and get the DOCKER_HOST and DOCKER_CERT_PATH values of that virtualbox and create a new builder.
In my case: I have created a virtual box default2 using docker CLI command docker-machine create -d virtualbox default2
Docker CLI
$ docker-machine env
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.99.103:2376"
export DOCKER_CERT_PATH="/Users/XXXX/.docker/machine/machines/default2"
export DOCKER_MACHINE_NAME="default2"
Docker-client JAVA
DockerCertificates defaultCertificates = new DockerCertificates(Paths.get("/Users/XXXX/.docker/machine/machines/default2"));
DockerClient docker = DefaultDockerClient.builder()
.uri("https://192.168.99.103:2376")
.dockerCertificates(defaultCertificates)
.build();

Ganymed SSH for java connect issue

We are using the Ganymed-SSH library and facing this error while doing SSH to another machine.
[root#XXXX test]# java -classpath .:ganymed-ssh2-build210.jar Basic
ERROR:java.io.IOException: There was a problem while connecting to 10.X.X.X:22
java.io.IOException: There was a problem while connecting to 10.X.X.X:22
at ch.ethz.ssh2.Connection.connect(Connection.java:699)
at ch.ethz.ssh2.Connection.connect(Connection.java:490)
at Basic.main(Basic.java:27)
Caused by: java.io.IOException: Key exchange was not finished, connection is closed.
at ch.ethz.ssh2.transport.KexManager.getOrWaitForConnectionInfo(KexManager.java:91)
at ch.ethz.ssh2.transport.TransportManager.getConnectionInfo(TransportManager.java:229)
at ch.ethz.ssh2.Connection.connect(Connection.java:655)
... 2 more
Caused by: java.io.IOException: Cannot read full block, EOF reached.
at ch.ethz.ssh2.crypto.cipher.CipherInputStream.getBlock(CipherInputStream.java:81)
at ch.ethz.ssh2.crypto.cipher.CipherInputStream.read(CipherInputStream.java:108)
at ch.ethz.ssh2.transport.TransportConnection.receiveMessage(TransportConnection.java:231)
at ch.ethz.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:669)
at ch.ethz.ssh2.transport.TransportManager$1.run(TransportManager.java:468)
at java.lang.Thread.run(Thread.java:636)
Can anyone explain what could be the issue here? Where should we start to debug from?
SSH access from normal shell works correctly.

Probably there is some problem with your public key

ssh failed after 141742-01/02 patch on solaris 10 !
Enabled aes192/aes256 support in ssh/sshd does not work on S10u3 or older released
A workaround is to disable the use of aes192/aes256 ciphers for ssh and sshd. Change the two config files /etc/ssh/ssh_config and /etc/ssh/sshd_config and add the following line:
Ciphers aes128-ctr,aes128-cbc,arcfour,3des-cbc,blowfish-cbc
You’ll have to restart sshd to pickup the change (“svcadm restart ssh”).
Source:
http://blog.mydream.com.hk/howto/matching-cipher-is-not-supported-aes256-cbc

Jenkins Build fails with SVNException

I am trying to run a jenkins build. The below is my configuration.
Jenkins Version : 1.445
Deployment : as a WAR
Java : 1.7 (Server 64 bit)
Tomcat Ver : 7.0.23
It gives the following error:
Started by user jenkins-admin
Updating https://myrepo.mycompany.com/svn/repos/myrepo
ERROR: Failed to update https://myrepo.mycompany.com/svn/repos/myrepo
org.tmatesoft.svn.core.SVNException: svn: OPTIONS /svn/repos/myrepo failed
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:298)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:283)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:271)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:533)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:98)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1011)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
at org.tmatesoft.svn.core.wc.SVNBasicClient.createRepository(SVNBasicClient.java:342)
at org.tmatesoft.svn.core.wc.SVNBasicClient.createRepository(SVNBasicClient.java:330)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.update(SVNUpdateClient.java:535)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:401)
at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:135)
at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:136)
at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:787)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:768)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:752)
at hudson.FilePath.act(FilePath.java:783)
at hudson.FilePath.act(FilePath.java:765)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:742)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:684)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1195)
at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:571)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:460)
at hudson.model.Run.run(Run.java:1404)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:481)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:238)
Caused by: org.tmatesoft.svn.core.SVNErrorMessage: svn: OPTIONS /svn/repos/myrepo failed
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:200)
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:146)
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:89)
... 27 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: OPTIONS request failed on '/svn/repos/myrepo'
svn: unknown host
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
The problem is it does NOT fail all the time. However, it fails 70-80% of the time.
The credentials and the URL for svn repo are correct.
UPDATE: I have tried every possibility now. The problem persists.
I tried with JDK 6 (both 32 and 64 bit). The problem does NOT go.
I am working on RHEL 5.6. Can anybody please help?

I noticed this line:
Caused by: org.tmatesoft.svn.core.SVNException: svn: OPTIONS request failed on '/svn/repos/myrepo'
svn: unknown host
Whenever I have a problem with Jenkins, I log into the Jenkins system as the user running Jenkins, go to the working directory inside Jenkins, and from a command line, try to do whatever Jenkins is attempting to do (like update my working directory). That usually will help you point out a particular error.
Try running an update a few times from the Jenkin's job's working directory and see if you get the same error. If you do, it could point out a network issue (unknown host).
If you can run updates from the working directory on your Jenkins server in your job's working directory, then the problem may lie with SVNKit itself. What version of Subversion and SVNKit are you using?
Try deleting the working directory and see if a clean checkout helps.

I noticed this error after upgrading my "Jenkins Subversion Plugin" from v1.42 to v1.44.
To Fix:
Please check the version of "Jenkins Subversion plugin" if it v1.44, downgrade to lower version.
Seems like we have some issues with the new Subversion Plugin.

The node running the build can not resolve the host name. Possibly a fully qualified name is required: rather than mysvnbox something like mysvnbox.mydomain.com; Or possibly the node is blocked by a firewall. The fact that it works sometimes would seem to indicate at least one node has visibility or can use an abbreviated domain specification. Good luck.

I have faced the same issue in Jenkins. It was kept on failing during checking out the code. I tried to increase Jenkins work space but it was prompted for the crowd authentication. It was failed to update and I tried clear the authentication date from subversion and restarted my build server (You can log out with your credentials and log-in). This will work.