Develop Reference

Error writing to S3 using TextIO.write() (Google Dataflow)

I have a Google Dataflow code similar to the below
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.io.TextIO;
import org.apache.beam.sdk.io.fs.ResourceId;
import org.apache.beam.sdk.options.*;
import org.apache.beam.sdk.transforms.Create;
public class S3Test {
public static void main(String[] args) {
PipelineOptions options = PipelineOptionsFactory.fromArgs(args).withValidation().create();
Pipeline pipeline = Pipeline.create(options);
ResourceId outputDir = FileSystems.matchNewResource("s3://my-bucket/temp/", true);
pipeline
.apply("New",
Create.of("Hello World!!")
)
.apply(
"Write to S3",
TextIO.write()
.to("s3://my-bucket/test.txt")
.withoutSharding()
.withTempDirectory(outputDir)
);
pipeline.run();
}
}
This works to some extent, in that it first creates a temporary file on my S3 bucket. However, when it tries to rename it to my final desired/input filename, it throws the following error.
Receiver class org.apache.beam.sdk.io.aws.s3.S3FileSystem does not
define or inherit an implementation of the resolved method 'abstract
void rename(java.util.List, java.util.List,
org.apache.beam.sdk.io.fs.MoveOptions[])' of abstract class
org.apache.beam.sdk.io.FileSystem.
I guess I have to override or implement a rename() function, but not sure where I'd start on that.
Has anyone tried to write successfully to S3 using TextIO.write()?
EDIT:
Below is the full message in the Run window:
Dec 24, 2022 4:56:34 PM
org.apache.beam.sdk.io.WriteFiles$WriteShardsIntoTempFilesFn
processElement INFO: Opening writer
8a0dbf70-a0fc-48dd-92f9-8ff4000a8076 for window
org.apache.beam.sdk.transforms.windowing.GlobalWindow#2dddc1b9 pane
PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0,
onTimeIndex=0} destination null Dec 24, 2022 4:56:37 PM
org.apache.beam.sdk.io.FileBasedSink$Writer close INFO: Successfully
wrote temporary file
s3://my-bucket/temp/.temp-beam-13da7e3e-15a9-4285-a90c-30c7c6dcfe99/9e53ec6a8a0dbf70-a0fc-48dd-92f9-8ff4000a8076
Dec 24, 2022 4:56:37 PM
org.apache.beam.sdk.io.WriteFiles$FinalizeTempFileBundles$FinalizeFn
process INFO: Finalizing 1 file results Dec 24, 2022 4:56:37 PM
org.apache.beam.sdk.io.FileBasedSink$WriteOperation
createMissingEmptyShards INFO: Finalizing for destination null num
shards 1. Dec 24, 2022 4:56:37 PM
org.apache.beam.sdk.io.FileBasedSink$WriteOperation moveToOutputFiles
INFO: Will copy temporary file
FileResult{tempFilename=s3://my-bucket/temp/.temp-beam-13da7e3e-15a9-4285-a90c-30c7c6dcfe99/9e53ec6a8a0dbf70-a0fc-48dd-92f9-8ff4000a8076,
shard=0,
window=org.apache.beam.sdk.transforms.windowing.GlobalWindow#2dddc1b9,
paneInfo=PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0,
onTimeIndex=0}} to final location s3://my-bucket/test.txt Exception in
thread "main" org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.AbstractMethodError: Receiver class
org.apache.beam.sdk.io.aws.s3.S3FileSystem does not define or inherit
an implementation of the resolved method 'abstract void
rename(java.util.List, java.util.List,
org.apache.beam.sdk.io.fs.MoveOptions[])' of abstract class
org.apache.beam.sdk.io.FileSystem. at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:374)
at
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:342)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:218)
at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:323) at
org.apache.beam.sdk.Pipeline.run(Pipeline.java:309) at
com.example.S3Test.main(S3Test.java:255) Caused by:
java.lang.AbstractMethodError: Receiver class
org.apache.beam.sdk.io.aws.s3.S3FileSystem does not define or inherit
an implementation of the resolved method 'abstract void
rename(java.util.List, java.util.List,
org.apache.beam.sdk.io.fs.MoveOptions[])' of abstract class
org.apache.beam.sdk.io.FileSystem. at
org.apache.beam.sdk.io.FileSystems.renameInternal(FileSystems.java:323)
at org.apache.beam.sdk.io.FileSystems.rename(FileSystems.java:308)
at
org.apache.beam.sdk.io.FileBasedSink$WriteOperation.moveToOutputFiles(FileBasedSink.java:802)
at
org.apache.beam.sdk.io.WriteFiles$FinalizeTempFileBundles$FinalizeFn.process(WriteFiles.java:1077)

Try updating your pipeline... by default withNumShards() is set to zero, so it will write to a single file
pipeline
.apply("New",
Create.of("Hello World!!")
)
.apply(
"Write to S3",
TextIO.write()
.to("s3://my-bucket/test.txt")
.withNumShards(1)
.withTempDirectory(outputDir)
);

Trouble Connecting to MongoDB with Java Async Client

I've used MongoDB with Python for a while now, and I've recently tried it with Java and i keep getting the same error.
Here is my code (with username and password replaced):
public class MongoClientConnection {
public static void main(String[] args) {
MongoClient mongoClient = MongoClients.create("mongodb://<username>:<password>#cluster0.bog1c.mongodb.net/db?w=majority");
MongoDatabase database = mongoClient.getDatabase("db");
MongoCollection<Document> accounts = database.getCollection("accounts");
Document doc = new Document("name", new String[]{"John", "Smith"})
.append("age", 35)
.append("favorite_color", "blue");
accounts.insertOne(doc, new SingleResultCallback<Void>() {
#Override
public void onResult(final Void result, final Throwable t) {
System.out.println("Inserted!");
}
});
}
}
This was taken directly from this tutorial.
Here is the output:
Jan 15, 2022 3:17:53 PM com.mongodb.diagnostics.logging.JULLogger log
INFO: Cluster created with settings {hosts=[cluster0.bog1c.mongodb.net:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
Jan 15, 2022 3:17:53 PM com.mongodb.diagnostics.logging.JULLogger log
INFO: No server chosen by PrimaryServerSelector from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, all=[ServerDescription{address=cluster0.bog1c.mongodb.net:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
Process finished with exit code 0
I am able to connect to the database using Mongosh. The string given on the cloud.mongodb.com site begins with mongodb+srv://, however it gives me the following error:
Exception in thread "main" java.lang.IllegalArgumentException: uri needs to start with mongodb://
That is why I am using only mongodb://.
I am running the code on Intellij IDEA in Windows.
Any ideas for what I am doing wrong?

How to fix "INFO:Detected dialect: OSS" in eclipse

I am kinda new to programming and I tried to execute the below code but chrome page is getting loaded but no execution happens and I am facing the below error.
Starting ChromeDriver 72.0.3626.69 (3c16f8a135abc0d4da2dff33804db79b849a7c38) on port 27651
Only local connections are allowed.
Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
Apr 19, 2019 12:23:41 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: OSS
My config
windows 10, chrome driver V72 and selenium-3.141.59
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
public class MyClass {
public static void main(String[] args) {
System.setProperty("webdriver.chrome.driver","C:\\Users\\ADMIN\\Desktop\\chromedriver.exe");
WebDriver driver = new ChromeDriver();
String baseUrl = "http://demo.guru99.com/test/newtours/";
String expectedTitle = "Welcome: Mercury Tours";
String actualTitle = "";
actualTitle = driver.getTitle();
if (actualTitle.contentEquals(expectedTitle)){
System.out.println("Test Passed!");
} else {
System.out.println("Test Failed");
}
driver.close();
}
}

The logs you have posted are not errors. It is the log from the chrome driver.
As it is stated in the log, Detected dialect: OSS log level is INFO. It doesn't prevent your test from running.
The code you have posted should surely print in the console Test Failed.
In your code, you have not gone to the URL you have saved in the baseUrl variable. Before you get the title of the page, you should get the page. Otherwise, it won't give you the desired pages title.
Do the following and your code will load the page and will print Test Passed!
driver.get(baseUrl);
actualTitle = driver.getTitle();

sending a cmdarray for exec to process -- hello world

Am I not sending an array of commands, "hello world ", to exec() correctly?
correct output, hello world:
thufir#mordor:~$
thufir#mordor:~$ java -jar NetBeansProjects/HelloExec/dist/HelloExec.jar
Apr 05, 2016 7:11:23 AM net.bounceme.mordor.telnet.Main run
INFO: starting..
Apr 05, 2016 7:11:23 AM net.bounceme.mordor.telnet.Telnet <init>
INFO: connecting..
Apr 05, 2016 7:11:23 AM net.bounceme.mordor.telnet.Telnet connect
INFO: connect..
Apr 05, 2016 7:11:23 AM net.bounceme.mordor.telnet.Telnet connect
INFO: [echo hello 1, echo hello 2, echo hello 3]
Apr 05, 2016 7:11:24 AM net.bounceme.mordor.telnet.Telnet connect
INFO: did process..
Apr 05, 2016 7:11:24 AM net.bounceme.mordor.telnet.Telnet connect
INFO: trying..
hello 1
Apr 05, 2016 7:11:24 AM net.bounceme.mordor.telnet.Telnet read
SEVERE: exiting.. 0
thufir#mordor:~$
code:
package net.bounceme.mordor.telnet;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.ArrayList;
import java.util.List;
import java.util.Properties;
import java.util.logging.Level;
import java.util.logging.Logger;
import static java.lang.System.out;
public class Telnet {
private final static Logger LOG = Logger.getLogger(Telnet.class.getName());
public Telnet() {
LOG.info("connecting..");
}
private List<String> getCommand() {
List<String> commands = new ArrayList<>();
commands.add("echo hello 1");
commands.add("echo hello 2");
commands.add("echo hello 3");
return commands;
}
private void read(Process process) throws IOException, InterruptedException {
BufferedReader input = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line = null;
while ((line = input.readLine()) != null) {
out.println(line);
}
int exitVal = process.waitFor();
LOG.log(Level.SEVERE, "exiting.. {0}", exitVal);
}
private void connect() {
LOG.info("connect..");
Runtime runtime = Runtime.getRuntime();
List<String> cmds = getCommand();
String[] cmdArray = cmds.toArray(new String[cmds.size()]);
String command = cmdArray[0];
LOG.info(cmds.toString());
Process process = null;
try {
process = runtime.exec(command);
} catch (IOException ex) {
LOG.severe("process wrong");
Logger.getLogger(Telnet.class.getName()).log(Level.SEVERE, null, ex);
}
LOG.info("did process..");
LOG.info("trying..");
try {
read(process);
} catch (IOException | InterruptedException ex) {
LOG.severe("read wrong");
Logger.getLogger(Telnet.class.getName()).log(Level.SEVERE, null, ex);
}
}
public void tryConnect() {
connect();
}
}
crash:
thufir#mordor:~$
thufir#mordor:~$ java -jar NetBeansProjects/HelloExec/dist/HelloExec.jar
Apr 05, 2016 7:12:32 AM net.bounceme.mordor.telnet.Main run
INFO: starting..
Apr 05, 2016 7:12:32 AM net.bounceme.mordor.telnet.Telnet <init>
INFO: connecting..
Apr 05, 2016 7:12:32 AM net.bounceme.mordor.telnet.Telnet connect
INFO: connect..
Apr 05, 2016 7:12:32 AM net.bounceme.mordor.telnet.Telnet connect
INFO: [echo hello 1, echo hello 2, echo hello 3]
Apr 05, 2016 7:12:32 AM net.bounceme.mordor.telnet.Telnet connect
SEVERE: process wrong
Apr 05, 2016 7:12:32 AM net.bounceme.mordor.telnet.Telnet connect
SEVERE: null
java.io.IOException: Cannot run program "echo hello 1": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at java.lang.Runtime.exec(Runtime.java:620)
at java.lang.Runtime.exec(Runtime.java:485)
at net.bounceme.mordor.telnet.Telnet.connect(Telnet.java:48)
at net.bounceme.mordor.telnet.Telnet.tryConnect(Telnet.java:66)
at net.bounceme.mordor.telnet.Main.run(Main.java:17)
at net.bounceme.mordor.telnet.Main.main(Main.java:11)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:248)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 6 more
Apr 05, 2016 7:12:32 AM net.bounceme.mordor.telnet.Telnet connect
INFO: did process..
Apr 05, 2016 7:12:32 AM net.bounceme.mordor.telnet.Telnet connect
INFO: trying..
Exception in thread "main" java.lang.NullPointerException
at net.bounceme.mordor.telnet.Telnet.read(Telnet.java:30)
at net.bounceme.mordor.telnet.Telnet.connect(Telnet.java:57)
at net.bounceme.mordor.telnet.Telnet.tryConnect(Telnet.java:66)
at net.bounceme.mordor.telnet.Main.run(Main.java:17)
at net.bounceme.mordor.telnet.Main.main(Main.java:11)
thufir#mordor:~$
when trying to use cmdArray:
process = runtime.exec(cmdArray);
The reasoning:
...it's because when exec is given a single string, it parses it first
(in a way that we don't like). In contrast, when exec is given a
string array, it simply passes it on to the operating system without
parsing it.
https://stackoverflow.com/a/19383655/262852

SSL with Grizzly and Jersey

I'm trying to get grizzly to use SSL encryption and still work fine with Jersey. I've looked all over the Internet, and I find all kinds of different attempts at SSL with Grizzly and Jersey. Seems like there are different ways of doing it depending on which version you are using, and how you decided to implement it. I haven't been able to get any examples to work with my code yet.
Here's how I start up my server:
static HttpServer startSecureServer() throws IOException{
ResourceConfig rc=new PackagesResourceConfig("server.grizzlyresources");
SSLContextConfigurator sslCon=new SSLContextConfigurator();
sslCon.setKeyStoreFile(ConfigLoader.getKeystoreLocation()); // contains server keypair
sslCon.setKeyStorePass(ConfigLoader.getKeystorePassword());
System.out.println("Starting server on port "+ConfigLoader.getHttpsServerPort());
HttpServer secure=GrizzlyServerFactory.createHttpServer(BASE_URI_SECURED, rc);
secure.stop();
HashSet<NetworkListener> lists=new HashSet<NetworkListener>(secure.getListeners());
for (NetworkListener listener : lists){
listener.setSecure(true);
SSLEngineConfigurator ssle=new SSLEngineConfigurator(sslCon);
listener.setSSLEngineConfig(ssle);
secure.addListener(listener);
System.out.println(listener);
}
secure.start();
return secure;
}
private static URI getBaseURISecured(){
return UriBuilder.fromUri("https://0.0.0.0/").port(ConfigLoader.getHttpsServerPort()).build();
}
private static final URI BASE_URI_SECURED = getBaseURISecured();
ConfigLoader loads in information from a config file. When I run this code, it starts up the server, it finds the resources in the server.grizzlyresources package, and it works great! Except for one thing. The server isn't secured. I can telnet into it and send an HTTP request in plain text for one of my resources, and it will return it. So the code works for starting up the server, but the whole SSL part of it is just being bypassed. Any ideas how to fix this or why it might be doing this?
Here's the output to the console when I run it:
Starting server on port 9999
Jan 13, 2014 9:51:08 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
server.grizzlyresources
Jan 13, 2014 9:51:08 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class server.grizzlyresources.SessionResource
class server.grizzlyresources.LoginResource
Jan 13, 2014 9:51:08 AM com.sun.jersey.api.core.ScanningResourceConfig init
INFO: No provider classes found.
Jan 13, 2014 9:51:08 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.12 02/15/2012 04:51 PM'
Jan 13, 2014 9:51:09 AM org.glassfish.grizzly.http.server.NetworkListener start
INFO: Started listener bound to [0.0.0.0:9999]
Jan 13, 2014 9:51:09 AM org.glassfish.grizzly.http.server.HttpServer start
INFO: [HttpServer] Started.
Jan 13, 2014 9:51:09 AM org.glassfish.grizzly.http.server.NetworkListener stop
INFO: Stopped listener bound to [0.0.0.0:9999]
NetworkListener{name='grizzly', host='0.0.0.0', port=9999, secure=true}
Jan 13, 2014 9:51:09 AM org.glassfish.grizzly.http.server.NetworkListener start
INFO: Started listener bound to [0.0.0.0:9999]
Jan 13, 2014 9:51:09 AM org.glassfish.grizzly.http.server.HttpServer start
INFO: [HttpServer] Started.
I'm using Grizzly 2.2.1, and Jersey 1.12.
Thanks a bunch!

Sorry to take so long to post this up here. Alexey's answer led me to the working solution, which is a lot like Wolfgang Fahl's code. Here's what I ended up with:
static HttpServer startSecureServer() throws IOException
{
System.out.println("Starting server on port " + ConfigLoader.getHttpsServerPort());
ResourceConfig rc = new PackagesResourceConfig("com.kinpoint.server.grizzlyresources");
SSLContextConfigurator sslCon = new SSLContextConfigurator();
sslCon.setKeyStoreFile(ConfigLoader.getKeystoreLocation()); // contains server keypair
sslCon.setKeyStorePass(ConfigLoader.getKeystorePassword());
HttpHandler hand = ContainerFactory.createContainer(HttpHandler.class, rc);
HttpServer secure = GrizzlyServerFactory.createHttpServer(BASE_URI_SECURED, hand, true,
new SSLEngineConfigurator(sslCon, false, false, false));
return secure;
}
The second parameter in the SSLEngineConfigurator tells it not to use client mode. That was what was messing me up. Thanks for the help.

IMO you can use different Factory method to initialize secured Grizzly HttpServer:
HttpServer secure = GrizzlyServerFactory.createHttpServer(BASE_URI_SECURED,
ContainerFactory.createContainer(HttpHandler.class, rc),
true,
new SSLEngineConfigurator(sslCon));
If you initialize the server like this, you don't need to stop and reconfigure it again.
Hope this will help.

I have a nice and tested example using Grizzly 2.3.3 in:
https://github.com/danielnuriyev/scriptedstuff/tree/master/src/com/scriptedstuff/server

The following code works with Grizzly 2.3.7 and I am using Jersey 1.18 - this includes code for SSL Client Authentication - if you don't have the keystores this feature will simply be ignored.
/**
* create a Server based on an url and possibly a ResourceConfig
*
* #param url
* #param rc
* #param secure
* - true if SSL should be used
* #param contextPath
* #return
* #throws Exception
*/
public HttpServer createHttpServer(String url, ResourceConfig rc,
boolean secure, String contextPath) throws Exception {
// HttpServer result = GrizzlyServerFactory.createHttpServer(url, rc);
// http://grepcode.com/file/repo1.maven.org/maven2/com.sun.jersey/jersey-grizzly2/1.6/com/sun/jersey/api/container/grizzly2/GrizzlyServerFactory.java#GrizzlyServerFactory.createHttpServer%28java.net.URI%2Ccom.sun.jersey.api.container.grizzly2.ResourceConfig%29
HttpServer result = new HttpServer();
final NetworkListener listener = new NetworkListener("grizzly",
settings.getHost(), settings.getPort());
result.addListener(listener);
// do we need SSL?
if (secure) {
listener.setSecure(secure);
SSLEngineConfigurator sslEngineConfigurator = createSSLConfig(true);
listener.setSSLEngineConfig(sslEngineConfigurator);
}
// Map the path to the processor.
final ServerConfiguration config = result.getServerConfiguration();
final HttpHandler handler = ContainerFactory.createContainer(
HttpHandler.class, rc);
config.addHttpHandler(handler, contextPath);
return result;
}
/**
* create SSL Configuration
*
* #param isServer
* true if this is for the server
* #return
* #throws Exception
*/
private SSLEngineConfigurator createSSLConfig(boolean isServer)
throws Exception {
final SSLContextConfigurator sslContextConfigurator = new SSLContextConfigurator();
// override system properties
final File cacerts = getStoreFile("server truststore",
"truststore_server.jks");
if (cacerts != null) {
sslContextConfigurator.setTrustStoreFile(cacerts.getAbsolutePath());
sslContextConfigurator.setTrustStorePass(TRUSTSTORE_PASSWORD);
}
// override system properties
final File keystore = getStoreFile("server keystore", "keystore_server.jks");
if (keystore != null) {
sslContextConfigurator.setKeyStoreFile(keystore.getAbsolutePath());
sslContextConfigurator.setKeyStorePass(TRUSTSTORE_PASSWORD);
}
//
boolean clientMode = false;
// force client Authentication ...
boolean needClientAuth = settings.isNeedClientAuth();
boolean wantClientAuth = settings.isWantClientAuth();
SSLEngineConfigurator result = new SSLEngineConfigurator(
sslContextConfigurator.createSSLContext(), clientMode, needClientAuth,
wantClientAuth);
return result;
}