I want to copy output of job from EMR cluster to Amazon S3 pro-grammatically.
How to use S3DistCp in java code to do the same.
hadoop ToolRunner can run this.. since S3DistCP extends Tool
Below is the usage example:
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.util.ToolRunner;
import com.amazon.external.elasticmapreduce.s3distcp.S3DistCp
public class CustomS3DistCP{
private static final Log log = LogFactory.getLog(CustomS3DistCP.class);
public static void main(String[] args) throws Exception {
log.info("Running with args: " + args);
System.exit(ToolRunner.run(new S3DistCp(), args));
}
you have to have s3distcp jar in your classpath
You can call this program from a shell script.
Hope that helps!
Related
I am building a fat jar for a Vertx-Web application. I would like to serve some static files. I packaged the jar file, with webroot folder. See below screenshot for my jar structure:
I was able to load the webroot/static/test.html file by doing:
routingContext.response().sendFile("webroot/static/test.html");
However, I am not able to get the static handler to work. Below is my full code:
package com.jdescript;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import io.vertx.core.AbstractVerticle;
import io.vertx.core.http.HttpServer;
import io.vertx.ext.web.Router;
import io.vertx.ext.web.handler.StaticHandler;
public class WebVerticle extends AbstractVerticle {
private HttpServer httpServer;
#Override
public void start() throws IOException {
httpServer = vertx.createHttpServer();
Router router = Router.router(vertx);
router.route("/static/*").handler(StaticHandler.create());
router.route("/test").handler(routingContext -> {
routingContext.response().sendFile("webroot/static/test.html");
});
httpServer.requestHandler(router::accept).listen(9999);
}
}
In the above example, http://localhost:9999/static/test.html will say "Not Found", while http://localhost:9999/test will render test.html.
Any help will be appreciated.
Was answered by the Vert.x group at https://groups.google.com/forum/?fromgroups#!topic/vertx/yKInZuYcqDE. My webroot should look like "webroot/test.html", instead of "webroot/static/test.html".
I get the following error when I try to run
java -cp 'angus-sdk-java-0.0.2-jar-with-dependencies.jar:.' FaceDetect
I am following a tutorial for face detection in http://angus-doc.readthedocs.io/en/latest/getting-started/java.html . Below is my java code,
import java.io.IOException;
import org.json.simple.JSONObject;
import ai.angus.sdk.Configuration;
import ai.angus.sdk.Job;
import ai.angus.sdk.ProcessException;
import ai.angus.sdk.Root;
import ai.angus.sdk.Service;
import ai.angus.sdk.impl.ConfigurationImpl;
import ai.angus.sdk.impl.File;
public class FaceDetect {
public static void main(String[] args) throws IOException, ProcessException {
Configuration conf = new ConfigurationImpl();
Root root = conf.connect();
Service service = root.getServices().getService("age_and_gender_estimation", 1);
JSONObject params = new JSONObject();
params.put("image", new File("Downloads/IMG_1060.jpg"));
Job job = service.process(params);
System.out.println(job.getResult().toJSONString());
}
}
I don't understand the problem with it. I have tried all the answers in the stack overflow but nothing is working for me.
remove the single qoutes around the classpath:
java -cp angus-sdk-java-0.0.2-jar-with-dependencies.jar:. FaceDetect
Trying to run RecognizeUsingWebSocketsExample provided with IBM Watson SpeechToText Java SDK, but it's failing to create a valid RecognizeOptions object for the sample .wav file provided with the distribution:
Exception in thread "main" java.lang.IllegalArgumentException: When using PCM the audio rate should be specified.
at com.ibm.watson.developer_cloud.util.Validator.isTrue(Validator.java:38)
at com.ibm.watson.developer_cloud.speech_to_text.v1.RecognizeOptions$Builder.contentType(RecognizeOptions.java:95)
at com.ibm.watson.developer_cloud.speech_to_text.v1.RecognizeUsingWebSocketsExample.main(RecognizeUsingWebSocketsExample.java:30)
It appears that the contentType(HttpMediaType.AUDIO_WAV) is being misinterpreted as RAW. Here's the actual (unmodified from distro) code:
package com.ibm.watson.developer_cloud.speech_to_text.v1;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import com.ibm.watson.developer_cloud.http.HttpMediaType;
import com.ibm.watson.developer_cloud.speech_to_text.v1.model.SpeechResults;
import com.ibm.watson.developer_cloud.speech_to_text.v1.websocket.BaseRecognizeCallback;
/**
* Recognize using WebSockets a sample wav file and print the transcript into the console output.
*/
public class RecognizeUsingWebSocketsExample {
private static CountDownLatch lock = new CountDownLatch(1);
public static void main(String[] args) throws FileNotFoundException, InterruptedException {
SpeechToText service = new SpeechToText();
service.setUsernameAndPassword("<username>", "<password>");
FileInputStream audio = new FileInputStream("src/test/resources/speech_to_text/sample1.wav");
RecognizeOptions options = new RecognizeOptions.Builder()
.continuous(true)
.interimResults(true)
.contentType(HttpMediaType.AUDIO_WAV)
.build();
service.recognizeUsingWebSocket(audio, options, new BaseRecognizeCallback() {
#Override
public void onTranscription(SpeechResults speechResults) {
System.out.println(speechResults);
if (speechResults.isFinal())
lock.countDown();
}
});
lock.await(1, TimeUnit.MINUTES);
}
}
I'm using 3.0.0-RC2 snapshot. No problems running examples which do not use RecognizeOptions, like SpeechToTextExample. Thx.
-rg
Sorry, false alarm. I recreated the example project from scratch and it compiled and ran without a hitch. Must have been some weirdness with my Eclipse setup.
I have a Spring Shell-based application and a couple of scripts. Is there an easy way to run the scripts in a JUnit test such that a test fails, if some exception/error occurs during the execution of the script?
The purpose of the tests is to make sure that all correct scripts run without errors.
Update 1:
Here's a little helper class for running scripts in JUnit:
import org.apache.commons.io.FileUtils;
import org.springframework.shell.Bootstrap;
import org.springframework.shell.core.CommandResult;
import org.springframework.shell.core.JLineShellComponent;
import java.io.File;
import java.io.IOException;
import java.util.List;
import static org.fest.assertions.api.Assertions.*;
public class ScriptRunner {
public void runScript(final File file) throws IOException
{
final Bootstrap bootstrap = new Bootstrap();
final JLineShellComponent shell = bootstrap.getJLineShellComponent();
final List<String> lines = FileUtils.readLines(file);
for (final String line : lines) {
execVerify(line, shell);
}
}
private void execVerify(final String command, final JLineShellComponent shell) {
final CommandResult result = shell.executeCommand(command);
assertThat(result.isSuccess()).isTrue();
}
}
You can create an instance of Bootstrap, get the shell out of it and then executeCommand() (including the shell command) on it.
You may be interested in what is done in Spring XD for this: https://github.com/spring-projects/spring-xd/blob/master/spring-xd-shell/src/test/java/org/springframework/xd/shell/AbstractShellIntegrationTest.java (although there are a lot of XD specific details)
I am performing JUnit Testing and receiving log4j:WARN No appenders could be found for logger (class .. ) error when I run test class(s).
Log4j properties file is present inside my folder root directory.
This code seems to work but why doesn't log4j picked up automatically.
package com.folio3.automation;
import org.apache.log4j.BasicConfigurator;
import org.apache.log4j.Logger;
import org.junit.Test;
import junit.framework.Assert;
public class TestClass {
static {
BasicConfigurator.configure();
}
private static final Logger LOG = Logger.getLogger(TestClass.class);
#Test
public void test1(){
LOG.info("test 1 called ");
Assert.assertEquals(true, false);
}
}
Do I have to call BasicConfigurator.configure(); in every class or Base class ?
Is there any way to achieve?
Try adding your application root directory to your VM start command and your problem will disappear.