Develop Reference

Fetching specific fields from an S3 document

I am using AWS Java SDK in my application to talk to one of my S3 buckets which holds objects in JSON format.
A document may look like this:
{
"a" : dataA,
"b" : dataB,
"c" : dataC,
"d" : dataD,
"e" : dataE
}
Now, for a certain document lets say document1 I need to fetch the values corresponding to field a and b instead of fetching the entire document.
This sounds like something that wouldn't be possible because S3 buckets can have any type of documents in them and not just JSONs.
Is this something that is achievable though?

That's actually doable. You could do selects like you've described, but only for particular formats: JSON, CSV, Parquet.
Imagine having a data.json file in so67315601 bucket in eu-central-1:
{
"a": "dataA",
"b": "dataB",
"c": "dataC",
"d": "dataD",
"e": "dataE"
}
First, learn how to select the fields via the S3 Console. Use "Object Actions" → "Query with S3 Select":
AWS Java SDK 1.x
Here is the code to do the select with AWS Java SDK 1.x:
#ExtendWith(S3.class)
class SelectTest {
#AWSClient(endpoint = Endpoint.class)
private AmazonS3 client;
#Test
void test() throws IOException {
// LINES: Each line in the input data contains a single JSON object
// DOCUMENT: A single JSON object can span multiple lines in the input
final JSONInput input = new JSONInput();
input.setType(JSONType.DOCUMENT);
// Configure input format and compression
final InputSerialization inputSerialization = new InputSerialization();
inputSerialization.setJson(input);
inputSerialization.setCompressionType(CompressionType.NONE);
// Configure output format
final OutputSerialization outputSerialization = new OutputSerialization();
outputSerialization.setJson(new JSONOutput());
// Build the request
final SelectObjectContentRequest request = new SelectObjectContentRequest();
request.setBucketName("so67315601");
request.setKey("data.json");
request.setExpression("SELECT s.a, s.b FROM s3object s LIMIT 5");
request.setExpressionType(ExpressionType.SQL);
request.setInputSerialization(inputSerialization);
request.setOutputSerialization(outputSerialization);
// Run the query
final SelectObjectContentResult result = client.selectObjectContent(request);
// Parse the results
final InputStream stream = result.getPayload().getRecordsInputStream();
IOUtils.copy(stream, System.out);
}
}
The output is:
{"a":"dataA","b":"dataB"}
AWS Java SDK 2.x
The code for the AWS Java SDK 2.x is more cunning. Refer to this ticket for more information.
#ExtendWith(S3.class)
class SelectTest {
#AWSClient(endpoint = Endpoint.class)
private S3AsyncClient client;
#Test
void test() throws Exception {
final InputSerialization inputSerialization = InputSerialization
.builder()
.json(JSONInput.builder().type(JSONType.DOCUMENT).build())
.compressionType(CompressionType.NONE)
.build();
final OutputSerialization outputSerialization = OutputSerialization.builder()
.json(JSONOutput.builder().build())
.build();
final SelectObjectContentRequest select = SelectObjectContentRequest.builder()
.bucket("so67315601")
.key("data.json")
.expression("SELECT s.a, s.b FROM s3object s LIMIT 5")
.expressionType(ExpressionType.SQL)
.inputSerialization(inputSerialization)
.outputSerialization(outputSerialization)
.build();
final TestHandler handler = new TestHandler();
client.selectObjectContent(select, handler).get();
RecordsEvent response = (RecordsEvent) handler.receivedEvents.stream()
.filter(e -> e.sdkEventType() == SelectObjectContentEventStream.EventType.RECORDS)
.findFirst()
.orElse(null);
System.out.println(response.payload().asUtf8String());
}
private static class TestHandler implements SelectObjectContentResponseHandler {
private SelectObjectContentResponse response;
private List<SelectObjectContentEventStream> receivedEvents = new ArrayList<>();
private Throwable exception;
#Override
public void responseReceived(SelectObjectContentResponse response) {
this.response = response;
}
#Override
public void onEventStream(SdkPublisher<SelectObjectContentEventStream> publisher) {
publisher.subscribe(receivedEvents::add);
}
#Override
public void exceptionOccurred(Throwable throwable) {
exception = throwable;
}
#Override
public void complete() {
}
}
}
As you see, it's possible to make S3 selects programmatically!
You might be wondering what are those #AWSClient and #ExtendWith( S3.class )?
This is a small library to inject AWS clients in your tests, named aws-junit5. It would greatly simplify your tests. I am the author. The usage is really simple — try it in your next project!

Velocity template metadata

Does Apache Velocity include a mechanism for adding metadata to a template?
I'm trying to add some extra information to my templates (e.g., type and descriptive name), and then read those to programmatically group templates by type, and list the templates on the UI using their descriptive name.
I've tried to use literal #[[...]]# blocks (and parse them), and #set directives, but both a have issues. They are hacky (require some parsing of the template) and far from elegant.

Hmmm, I'm not aware of anything built-in to do this. To avoid processing a whole template on a first pass though, one trick is to conditionally throw an exception (MetadataFinished below) during that pass, but not normal execution.
Clearly this would still need to compile the whole template up front, though this should come in useful at execution time.
E.g.
import org.apache.commons.io.output.NullWriter;
public class Metadata {
private Map<String, Template> byKey = new LinkedHashMap<>();
private Template currentTemplate;
/** Callback from .vm */
public void set(String key) throws MetadataFinished {
// Only do this in addTemplate()
if (currentTemplate != null) {
byKey.put(key, currentTemplate);
throw new MetadataFinished();
}
}
public void addTemplate(Template template) {
currentTemplate = template;
try {
Context context = new VelocityContext();
context.put("metadata", this);
template.merge(context, new NullWriter());
} catch (MetadataFinished ex) {
// Ignored
} finally {
currentTemplate = null;
}
}
public void execute(String key) {
Template template = byKey.get(key);
Context context = new VelocityContext();
PrintWriter pw = new PrintWriter(System.out);
template.merge(context, pw);
pw.flush();
}
// Extends Error to avoid Velocity adding a wrapping MethodInvocationException
private static class MetadataFinished extends Error {
}
public static void main(String[] args) {
Metadata metadata = new Metadata();
VelocityEngine engine = new VelocityEngine();
engine.setProperty("file.resource.loader.path", "/temp");
engine.init();
String[] fileNames = { "one.vm", "two.vm" };
for (String fileName : fileNames) {
Template template = engine.getTemplate(fileName);
metadata.addTemplate(template);
}
metadata.execute("vm1");
metadata.execute("vm2");
}
}
Then in one.vm:
$!metadata.set("vm1")##
-----------
This is VM1
-----------
The ## there is a bit ugly - it's just to stop a blank line being output. If readability is important, this can be made a bit neater with a macro though:
#metadata("vm2")
-----------
This is VM2
-----------
That macro could be defined in the global VM_global_library.vm:
#macro( metadata $key )
$!metadata.set($key)#end
Just for reference, the output is as expected:
-----------
This is VM1
-----------
-----------
This is VM2
-----------

Unable to get data from java web service

Someone please help me i keep trying but not able to find out why i am unable to get the results.
I have created this java springboot web service where when I run the java application, a web browser page will open and when I type in the URL e.g localhost:8080/runbatchfileparam/test.bat the program will check if the test.bat file exist first. If it does, the web page will show a JSON result {“Result”: true} and the command in the batch file will be executed. If it does not exist, the web page will show {“Result”: false}.
I want to create an ASP.NET Web Service that will use the function created in the java web service. When I run the ASP.NET Web Application, a web browser page will open. User will type in URL something like this: localhost:12345/api/callbatchfile/test.bat. The java web service should be running and I should get either {“Result”: true} or {“Result”: false} when I run the C# ASP.NET Web Application too.
However I only get an empty {} without anything inside the brackets. Why is that so?
Here are my code in ASP.NET
TestController.cs
private TestClient testClient = new TestClient();
public async Task<IHttpActionResult> GET(string fileName)
{
try
{
var result = await testClient.runbatchfile(fileName);
var resultDTO = JsonConvert.DeserializeObject<TestVariable>(result);
return Json(resultDTO);
}
catch (Exception e)
{
var result = "Server is not running";
return Ok(new { ErrorMessage = result });
}
}
TestVariable.cs
public class TestVariable
{
public static int fileName { get; set; }
}
TestClient.cs
private static HttpClient client;
private static string BASE_URL = "http://localhost:8080/";
static TestClient()
{
client = new HttpClient();
client.BaseAddress = new Uri(BASE_URL);
client.DefaultRequestHeaders.Accept.Add(
new MediaTypeWithQualityHeaderValue("application/json"));
}
public async Task<string> runbatchfile(string fileName)
{
var endpoint = string.Format("runbatchfile/{0}", fileName);
var response = await client.GetAsync(endpoint);
return await response.Content.ReadAsStringAsync();
}
WebApiConfig.cs
config.Routes.MapHttpRoute(
name: "TestBatchClient",
routeTemplate: "api/runbatchfile/{fileName}",
defaults: new { action = "GET", controller = "Test" }
);
Someone please do help me. Thank you so much.
EDIT
Java web service
Application.java
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
BatchFileController.java
private static final String template = "Sum, %s!";
#RequestMapping("/runbatchfile/{param:.+}")
public ResultFormat runbatchFile(#PathVariable("param") String fileName) {
RunBatchFile rbf = new RunBatchFile();
return rbf.runBatch(fileName);
}
ResultFormat
private boolean result;
public ResultFormat(boolean result) {
this.result = result;
}
public boolean getResult() {
return result;
}
RunBatchFile.java
public ResultFormat runBatch(String fileName) {
String var = fileName;
String filePath = ("C:/Users/attsuap1/Desktop/" + var);
try {
Process p = Runtime.getRuntime().exec(filePath);
int exitVal = p.waitFor();
return new ResultFormat(exitVal == 0);
} catch (Exception e) {
e.printStackTrace();
return new ResultFormat(false);
}
}

I am not sure if this helps.. but I suspect that the AsyncTask is not really executing...
var result = await testClient.testCallBatchProject(fileName);
I would try something like below:
await testClient.testCallBatchProject(fileName).Delay(1000);
Can you try and check if the same happens for a synchronous call? .. if it does, we can zero down on the above.

string escaping with non-nullable named parameters

I am trying to use the google cloud endpoints Java from android as such:
client:
Core.Builder coreBuilder = new Core.Builder(
AndroidHttp.newCompatibleTransport(), new GsonFactory(), null);
coreBuilder.setApplicationName("myapp");
if (MainActivity.ENDPOINTS_URL != null) {
coreBuilder.setRootUrl(MainActivity.ENDPOINTS_URL);
coreBuilder.setGoogleClientRequestInitializer(new GoogleClientRequestInitializer() {
public void initialize(AbstractGoogleClientRequest<?> request)
throws IOException {
request.setDisableGZipContent(true);
}
});
}
Core core = coreBuilder.build();
myList = core.asdf("x=&+x", myObject);
server:
#ApiMethod(name = "asdf")
public List<String> asdf(#Named("param1") String param1, MyObject myObject) {
if (param1.equals("x=&+x")) {
//should go here, but never does
}
...
While it mostly works, somehow the param1 string does not get correctly transmitted, meaning that "x=&+x" arrives at the server as "x=&%2Bx". Is this a known bug? Or do arguments have to be manually encoded somehow? Or is this somehow particular to my environment?
Appengine SDK V1.8.8 for java, google api 1.17.0-rc, using the dev environment.
Cheers,
Andres

Is it possible to create an URL pointing to an in-memory object?

I'm trying to extend my library for integrating Swing and JPA by making JPA config as automatic (and portable) as can be done, and it means programmatically adding <class> elements. (I know it can be done via Hibernate's AnnotationConfiguration or EclipseLInk's ServerSession, but - portability). I'd also like to avoid using Spring just for this single purpose.
I can create a persistence.xml on the fly, and fill it with <class> elements from specified packages (via the Reflections library). The problem starts when I try to feed this persistence.xml to a JPA provider. The only way I can think of is setting up a URLClassLoader, but I can't think of a way what wouldn't make me write the file to the disk somewhere first, for sole ability to obtain a valid URL. Setting up a socket for serving the file via an URL(localhost:xxxx) seems... I don't know, evil?
Does anyone have an idea how I could solve this problem? I know it sounds like a lot of work to avoid using one library, but I'd just like to know if it can be done.
EDIT (a try at being more clear):
Dynamically generated XML is kept in a String object. I don't know how to make it available to a persistence provider. Also, I want to avoid writing the file to disk.
For purpose of my problem, a persistence provider is just a class which scans the classpath for META-INF/persistence.xml. Some implementations can be made to accept dynamic creation of XML, but there is no common interface (especially for a crucial part of the file, the <class> tags).
My idea is to set up a custom ClassLoader - if you have any other I'd be grateful, I'm not set on this one.
The only easily extendable/configurable one I could find was a URLClassLoader. It works on URL objects, and I don't know if I can create one without actually writing XML to disk first.
That's how I'm setting things up, but it's working by writing the persistenceXmlFile = new File("META-INF/persistence.xml") to disk:
Thread.currentThread().setContextClassLoader(
new URLResourceClassLoader(
new URL[] { persistenceXmlFile.toURI().toURL() },
Thread.currentThread().getContextClassLoader()
)
);
URLResourceClassLoader is URLCLassLoader's subclass, which allows for looking up resources as well as classes, by overriding public Enumeration<URL> findResources(String name).

Maybe a bit late (after 4 years), but for others that are looking for a similar solution, you may be able to use the URL factory I created:
public class InMemoryURLFactory {
public static void main(String... args) throws Exception {
URL url = InMemoryURLFactory.getInstance().build("/this/is/a/test.txt", "This is a test!");
byte[] data = IOUtils.toByteArray(url.openConnection().getInputStream());
// Prints out: This is a test!
System.out.println(new String(data));
}
private final Map<URL, byte[]> contents = new WeakHashMap<>();
private final URLStreamHandler handler = new InMemoryStreamHandler();
private static InMemoryURLFactory instance = null;
public static synchronized InMemoryURLFactory getInstance() {
if(instance == null)
instance = new InMemoryURLFactory();
return instance;
}
private InMemoryURLFactory() {
}
public URL build(String path, String data) {
try {
return build(path, data.getBytes("UTF-8"));
} catch (UnsupportedEncodingException ex) {
throw new RuntimeException(ex);
}
}
public URL build(String path, byte[] data) {
try {
URL url = new URL("memory", "", -1, path, handler);
contents.put(url, data);
return url;
} catch (MalformedURLException ex) {
throw new RuntimeException(ex);
}
}
private class InMemoryStreamHandler extends URLStreamHandler {
#Override
protected URLConnection openConnection(URL u) throws IOException {
if(!u.getProtocol().equals("memory")) {
throw new IOException("Cannot handle protocol: " + u.getProtocol());
}
return new URLConnection(u) {
private byte[] data = null;
#Override
public void connect() throws IOException {
initDataIfNeeded();
checkDataAvailability();
// Protected field from superclass
connected = true;
}
#Override
public long getContentLengthLong() {
initDataIfNeeded();
if(data == null)
return 0;
return data.length;
}
#Override
public InputStream getInputStream() throws IOException {
initDataIfNeeded();
checkDataAvailability();
return new ByteArrayInputStream(data);
}
private void initDataIfNeeded() {
if(data == null)
data = contents.get(u);
}
private void checkDataAvailability() throws IOException {
if(data == null)
throw new IOException("In-memory data cannot be found for: " + u.getPath());
}
};
}
}
}

We can use the Jimfs google library for that.
First, we need to add the maven dependency to our project:
<dependency>
<groupId>com.google.jimfs</groupId>
<artifactId>jimfs</artifactId>
<version>1.2</version>
</dependency>
After that, we need to configure our filesystem behavior, and write our String content to the in-memory file, like this:
public static final String INPUT =
"\n"
+ "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"
+ "<note>\n"
+ " <to>Tove</to>\n"
+ " <from>Jani</from>\n"
+ " <heading>Reminder</heading>\n"
+ " <body>Don't forget me this weekend!</body>\n"
+ "</note>";
#Test
void usingJIMFS() throws IOException {
try (var fs = Jimfs.newFileSystem(Configuration.unix())) {
var path = fs.getPath(UUID.randomUUID().toString());
Files.writeString(path, INPUT);
var url = path.toUri().toURL();
assertThat(url.getProtocol()).isEqualTo("jimfs");
assertThat(Resources.asCharSource(url, UTF_8).read()).isEqualTo(INPUT);
}
}
We can find more examples in the official repository.
If we look inside the jimfs source code we will find the implementation is similar to #NSV answer.