I currently work with Netbeans on Windows machine to develop topologies. When I deploy in local mode:
LocalCluster cluster = new LocalCluster();
cluster.submitTopology("word-count", conf, builder.createTopology());
everything works just fine, but when I try to :
StormSubmitter.submitTopology("word", conf, builder.createTopology());
it obviously tries to deploy the topology in a cluster mode and fails since I dont have storm nimbus running on my local computer. I do have storm deployed on one Digital Ocean droplet, but my current (and not convenient) solution is to copy the JAR file and use the storm jar... command to deploy.
My question is: is there a way to tell Netbeans what is my nimbus IP address, so it can deploy it remotely? (and save me the time)Thank you in advance!
Check this link
Now I can develope topologies in Netbeans, test them locally, and eventually deploy them to my Nimbus on the cluster. This solution works great for me!!!
Add to conf file:
conf.put(Config.NIMBUS_HOST, "123.456.789.101); //YOUR NIMBUS'S IP
conf.put(Config.NIMBUS_THRIFT_PORT,6627); //int is expected here
Also, add the following :
System.setProperty("storm.jar", <path-to-jar>); //link to exact file location (w/ dependencies)
to avoid the following error:[main] INFO backtype.storm.StormSubmitter - Jar not uploaded to master yet. Submitting jar...
Exception in thread "main" java.lang.RuntimeException: Must submit topologies using the 'storm' client script so that StormSubmitter knows which jar to upload.
Cheers!
Yeah, definitely you can tell your topology about your nimbus IP. Following is the example code to submit topology on remote cluster.
Map storm_conf = Utils.readStormConfig();
storm_conf.put("nimbus.host", "<Nimbus Machine IP>");
Client client = NimbusClient.getConfiguredClient(storm_conf)
.getClient();
String inputJar = "C:\\workspace\\TestStormRunner\\target\\TestStormRunner-0.0.1-SNAPSHOT-jar-with-dependencies.jar";
NimbusClient nimbus = new NimbusClient(storm_conf, "<Nimbus Machine IP>",
<Nimbus Machine Port>);
// upload topology jar to Cluster using StormSubmitter
String uploadedJarLocation = StormSubmitter.submitJar(storm_conf,
inputJar);
String jsonConf = JSONValue.toJSONString(storm_conf);
nimbus.getClient().submitTopology("testtopology",
<uploadedJarLocation>, jsonConf, builder.createTopology());
Here is the working example : Submitting a topology to Remote Storm Cluster
You can pass those information using the conf map parameters .. you can pass a key, value pair as per your requirements
for a list of accepted parameters check this page ..
Related
Currently I'm facing a problem where I don't know what to do to resolve it. I'm developing a simple application that transfer files through different FTP and SFTP servers to be processed. Well, at the beginnig those servers weren't ready, so I used Apache Mina and RebexTinySftpServer to set a FTP server and a SFTP server on my computer to test my development.
With those applications I completed and tested my application locally, so it was time to test it using the servers that will be used in production, but something is wrong with the FTP server and Spring Integration is not detecting the files that are put in the folder to be transferred to the SFTP server.
I have two folders on each server: one for Input files and the another one for Output files. So when Spring Integration detects that there's a new file in the Input folder on the SFTP Server, it transfers the file to the Input folder on FTP Server to be processed for another service. That part works fine.
After that service processes the file, it generates an output file and stores it in the Output folder on the FTP Server to be transferred to the Output folder on the SFTP server, but for some reason Spring Integration is not detecting those files and doesn't transfer none of them. This part is what I don't know how to solve because none Exception is being thrown, so I don't know what part of my code I have to modify.
Here is my code where I define the DefaultFtpSessionFactory:
public DefaultFtpSessionFactory ftpSessionFactory() {
DefaultFtpSessionFactory session = new DefaultFtpSessionFactory();
session.setUsername(username);
session.setPassword(password);
session.setPort(port);
session.setHost(host);
session.setFileType(FTP.ASCII_FILE_TYPE);
session.setClientMode(FTPClient.PASSIVE_LOCAL_DATA_CONNECTION_MODE);
return session;
}
And here is the code where I define the FTP Inbound Adapter:
#Bean
FtpInboundChannelAdapterSpec salidaAS400InboundAdapter() {
return Ftp.inboundAdapter(as400Session.ftpSessionFactory())
.preserveTimestamp(true)
.remoteDirectory(as400Session.getPathSalida())
.deleteRemoteFiles(true)
.localDirectory(new File("tmp/as400/salida"))
.temporaryFileSuffix(".tmp");
}
I should mention that the FTP Server is running on an AS/400 system, so maybe that has something to do with the situation I'm facing.
I found the solution of my problem. I'm posting this in case something similar happens to someone else.
My project is using spring-integration-ftp 5.5.16 that has commons-net 3.8.0 as a dependency. For some reason, that version of commons-net wasn't retrieving the files inside the directory of the AS400, so I excluded that dependency from spring-integration-ftp and added commons-net 3.9.0 in my project. Now everything works fine.
So apologies if what I ask is trivial but I am experimenting with Memcached and Jmeter. I have a Memcached server setup (as far as I can tell) and am able to make telnet requests to it via telnet IP PORT and additionally set and get using commands set and get appropriately.
Now point me to a different application if perhaps this is the wrong choice; but my understanding was that Jmeter should allow me to pound the server with equivalent Set and Get requests.
Unfortunately the experimental platform is a remote linux PC running Rockylinux which is similar to CentOS / RedHat to my understanding (I didn't set this part up); and as a result I do not have a GUI to launch while on the Linux PC. I have however opened Jmeter up on my local PC on windows and understand I should be able to send the test file over and run it.
I followed these instructions to try to setup a TCP sampler and set the "text to send" field as below; after doing the additional step in the link regarding the precompiler.
set tutorialspoint 0 900 9${CR}${LF}
memcached${CR}${LF}
quit${CR}${LF}
Running the above as a headless jmeter session doesn't generate any errors called [./jmeter -n -t "Sample.jmx" -l testresults.jtl"] but when I connect via telnet I'm also not seeing the value for the key "tutorialspoint" get updated. When manually doing the get and set I am seeing updates. Any ideas what I could be doing wrong? Checking the log indicates ResponseCode 200 OK as expected. Is there a good method to debug something in a Headless setup?
Thanks for your time.
I believe the easiest way is using Memcached Java Client library
Download spymemcached-2.12.3.jar and drop it to "lib" folder of your JMeter installation (or any other location in JMeter Classpath)
Restart JMeter to pick the .jar up
Add JSR223 Sampler to your test plan and use the following code snippets:
def client = new net.spy.memcached.MemcachedClient(new InetSocketAddress('your-memcached-host', your-memcached-port)) - for connecting to the server
client.set('tutorialspoint', 900, 'memcached').done to write memcached to the tutorialspoint key for 15 minutes
client.get('tutorialspoint') - to read the value of tutorialspoint key
client.shutdown() - to disconnect
More information on Groovy scripting in JMeter: Apache Groovy - Why and How You Should Use It
Demo:
I want to transfer file from FTP server to HDFS. I tried this method: FTP to HDFS, the demo code as follows:
Configuration conf = new Configuration();
FTPFileSystem ftpfs = new FTPFileSystem();
ftpfs.setConf(conf);
ftpfs.initialize(new URI(ftpConnectStr), conf);
Path homeDirectory = ftpfs.getHomeDirectory();
System.out.println(homeDirectory.toString());
FileStatus[] fileStatuses = ftpfs.listStatus(new Path("/"));
for(FileStatus fileStatus : fileStatuses){
System.out.println(fileStatuses.length);
System.out.println(fileStatus.toString());
}
boolean test = ftpfs.mkdirs(new Path("test"));
System.out.println(test);
The ftpfs.listStatus(new Path("/")) doesn't work, it shows nothing, but the FTP server has two directories and ftpfs.mkdirs(new Path("test")) work fine, the program running result as follows:
and the FTP server dirctory as follows：
I searched in google，but find a little information. I don't know why. If you can help me, I will be very grateful，thanks
As you have found out, the problem is because Hadoop (or rather the underlying Apache Common Net FtpClient) defaults to the FTP active mode, which hardly works nowadays due to ubiquitous NATs and firewalls.
Since Hadoop 2.9, you can set up FTP passive mode by setting fs.ftp.data.connection.mode configuration option to enable the FTP passive mode:
fs.ftp.data.connection.mode=PASSIVE_LOCAL_DATA_CONNECTION_MODE
See https://issues.apache.org/jira/browse/HADOOP-13953
Finally,I found the problem where it is; In FTP Server,the data tansfer mode is set to passive.
Then I debuged the source code of FTPFileSystem,and i found that it didn't set FTP passive mode;
so,I modify the related code of FTPFileSystem to this:
Rerun the program：
and it works fine:
I have a flink project that is connecting to nifi to pull data. The setup to pull get the datastream works just fine when running locally.
.url("http://1.2.3.4:8080/nifi")
.portName("MyPortName")
.requestBatchCount(5)
.buildConfig();
But when I add the .jar to the remote cluster and run the job it throws this:
java.net.UnknownHostException
at sun.nio.ch.Net.translateException(Net.java:177)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:127)
at org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:712)
at org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:685)
at org.apache.nifi.remote.client.socket.EndpointConnectionPool.getEndpointConnection(EndpointConnectionPool.java:301)
at org.apache.nifi.remote.client.socket.SocketClient.createTransaction(SocketClient.java:129)
at org.apache.flink.streaming.connectors.nifi.NiFiSource.run(NiFiSource.java:90)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:78)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:56)
at org.apache.flink.streaming.runtime.tasks.StoppableSourceStreamTask.run(StoppableSourceStreamTask.java:39)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:272)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
at java.lang.Thread.run(Thread.java:745)
The only reason I can find for an UnknownHostException is that it is because the IP of the host name can't be resolved, but I am giving the IP already. There was an issue earlier with it being unable to connect to nifi because I have to set what IP is allowed to access the nifi instance. So I added the AWS server as allowed and it fixed that, but obviously I have this now.
Any help is greatly appreciated!
I figured the problem. I had my nifi cluster and my flink cluster in different regions. Moved the flink cluster to the same region and used either the public or private url for the cluster and it works fine.
I am a beginner storm user. I am trying out drpc server in remote mode. I got drpc server started and configured the drpc server location in yaml file. BUT, I am not understanding how the drpc client code should look like:
https://github.com/nathanmarz/storm-starter/blob/master/src/jvm/storm/starter/BasicDRPCTopology.java
Here is what I did:
Launched DRPC server(s) (storm drpc command)
Configure the locations of the DRPC servers (edited the yaml file. Added the local host name)
Submit DRPC topologies to Storm cluster - did this, looks like the topology is up and running.
But how do I get a client to call/execute on this topology? Do I need something like this?https://github.com/mykidong/storm-finagle-drpc-client/blob/master/src/main/java/storm/finagle/drpc/StormDrpcClient.java ?? I tried but I keep getting this error:
storm/starter/DRPCClient.java:[68,18] error: execute(String,String) in DRPCClient cannot implement execute(String,String) in Iface
[ERROR] overridden method does not throw TException
What am I missing here? thanks
Here is Storm DRPC Document
Maybe useful to understand DRPC Call :)
Just Like following code :
DRPCClient client = new DRPCClient("drpc-host", 3772);
String result = client.execute("reach", "http://twitter.com");
Create a client connection to DRPC-Server-Host: drpc-host at 3772 port .
DRPCClient called "reach" function using argument "http://twitter.com"
and return a string named result