I am using AWS Java SDK in my application to talk to one of my S3 buckets which holds objects in JSON format.
A document may look like this:
{
"a" : dataA,
"b" : dataB,
"c" : dataC,
"d" : dataD,
"e" : dataE
}
Now, for a certain document lets say document1 I need to fetch the values corresponding to field a and b instead of fetching the entire document.
This sounds like something that wouldn't be possible because S3 buckets can have any type of documents in them and not just JSONs.
Is this something that is achievable though?
That's actually doable. You could do selects like you've described, but only for particular formats: JSON, CSV, Parquet.
Imagine having a data.json file in so67315601 bucket in eu-central-1:
{
"a": "dataA",
"b": "dataB",
"c": "dataC",
"d": "dataD",
"e": "dataE"
}
First, learn how to select the fields via the S3 Console. Use "Object Actions" → "Query with S3 Select":
AWS Java SDK 1.x
Here is the code to do the select with AWS Java SDK 1.x:
#ExtendWith(S3.class)
class SelectTest {
#AWSClient(endpoint = Endpoint.class)
private AmazonS3 client;
#Test
void test() throws IOException {
// LINES: Each line in the input data contains a single JSON object
// DOCUMENT: A single JSON object can span multiple lines in the input
final JSONInput input = new JSONInput();
input.setType(JSONType.DOCUMENT);
// Configure input format and compression
final InputSerialization inputSerialization = new InputSerialization();
inputSerialization.setJson(input);
inputSerialization.setCompressionType(CompressionType.NONE);
// Configure output format
final OutputSerialization outputSerialization = new OutputSerialization();
outputSerialization.setJson(new JSONOutput());
// Build the request
final SelectObjectContentRequest request = new SelectObjectContentRequest();
request.setBucketName("so67315601");
request.setKey("data.json");
request.setExpression("SELECT s.a, s.b FROM s3object s LIMIT 5");
request.setExpressionType(ExpressionType.SQL);
request.setInputSerialization(inputSerialization);
request.setOutputSerialization(outputSerialization);
// Run the query
final SelectObjectContentResult result = client.selectObjectContent(request);
// Parse the results
final InputStream stream = result.getPayload().getRecordsInputStream();
IOUtils.copy(stream, System.out);
}
}
The output is:
{"a":"dataA","b":"dataB"}
AWS Java SDK 2.x
The code for the AWS Java SDK 2.x is more cunning. Refer to this ticket for more information.
#ExtendWith(S3.class)
class SelectTest {
#AWSClient(endpoint = Endpoint.class)
private S3AsyncClient client;
#Test
void test() throws Exception {
final InputSerialization inputSerialization = InputSerialization
.builder()
.json(JSONInput.builder().type(JSONType.DOCUMENT).build())
.compressionType(CompressionType.NONE)
.build();
final OutputSerialization outputSerialization = OutputSerialization.builder()
.json(JSONOutput.builder().build())
.build();
final SelectObjectContentRequest select = SelectObjectContentRequest.builder()
.bucket("so67315601")
.key("data.json")
.expression("SELECT s.a, s.b FROM s3object s LIMIT 5")
.expressionType(ExpressionType.SQL)
.inputSerialization(inputSerialization)
.outputSerialization(outputSerialization)
.build();
final TestHandler handler = new TestHandler();
client.selectObjectContent(select, handler).get();
RecordsEvent response = (RecordsEvent) handler.receivedEvents.stream()
.filter(e -> e.sdkEventType() == SelectObjectContentEventStream.EventType.RECORDS)
.findFirst()
.orElse(null);
System.out.println(response.payload().asUtf8String());
}
private static class TestHandler implements SelectObjectContentResponseHandler {
private SelectObjectContentResponse response;
private List<SelectObjectContentEventStream> receivedEvents = new ArrayList<>();
private Throwable exception;
#Override
public void responseReceived(SelectObjectContentResponse response) {
this.response = response;
}
#Override
public void onEventStream(SdkPublisher<SelectObjectContentEventStream> publisher) {
publisher.subscribe(receivedEvents::add);
}
#Override
public void exceptionOccurred(Throwable throwable) {
exception = throwable;
}
#Override
public void complete() {
}
}
}
As you see, it's possible to make S3 selects programmatically!
You might be wondering what are those #AWSClient and #ExtendWith( S3.class )?
This is a small library to inject AWS clients in your tests, named aws-junit5. It would greatly simplify your tests. I am the author. The usage is really simple — try it in your next project!
Related
I am coding a server application that will receive DFT_P03 messages with an added ZPM segment (which i have created a class for as per the HAPI documentation). Currently i am able to access this field as a generic segment when doing the following :
#Override
public Message processMessage(Message t, Map map) throws ReceivingApplicationException, HL7Exception
{
String encodedMessage = new DefaultHapiContext().getPipeParser().encode(t);
logEntryService.logDebug(LogEntry.CONNECTIVITY, "Received message:\n" + encodedMessage + "\n\n");
try
{
InboundMessage inboundMessage = new InboundMessage();
inboundMessage.setMessageTime(new Date());
inboundMessage.setMessageType("Usage");
DFT_P03 usageMessage = (DFT_P03) t;
Segment ZPMSegment = (Segment)usageMessage.get("ZPM");
inboundMessage.setMessage(usageMessage.toString());
Facility facility = facilityService.findByCode(usageMessage.getMSH().getReceivingFacility().getNamespaceID().getValue());
inboundMessage.setTargetFacility(facility);
String controlID = usageMessage.getMSH().getMessageControlID().encode();
controlID = controlID.substring(controlID.indexOf("^") + 1, controlID.length());
inboundMessage.setControlId(controlID);
Message response;
try
{
inboundMessageService.save(inboundMessage);
response = t.generateACK();
logEntryService.logDebug(LogEntry.CONNECTIVITY, "Message ACKed");
}
catch (Exception ex)
{
response = t.generateACK(AcknowledgmentCode.AE, new HL7Exception(ex));
logEntryService.logDebug(LogEntry.CONNECTIVITY, "Message NACKed");
}
return response;
}
catch (IOException e)
{
logEntryService.logDebug(LogEntry.CONNECTIVITY, "Message rejected");
throw new HL7Exception(e);
}
}
I have created a DFT_P03_Custom class as following :
public class DFT_P03_Custom extends DFT_P03
{
public DFT_P03_Custom() throws HL7Exception
{
this(new DefaultModelClassFactory());
}
public DFT_P03_Custom(ModelClassFactory factory) throws HL7Exception
{
super(factory);
String[] segmentNames = getNames();
int indexOfPid = Arrays.asList(segmentNames).indexOf("FT1");
int index = indexOfPid + 1;
Class<ZPM> type = ZPM.class;
boolean required = true;
boolean repeating = false;
this.add(type, required, repeating, index);
}
public ZPM getZPM()
{
return getTyped("ZPM", ZPM.class);
}
}
When trying to typecast the message to a DFT_P03_Custom instance i get a ClassCastException. As per their documentation, i did create the CustomModelClassFactory class but using this i just get tons of validation errors on the controlId field.
I am already using an identical logic to send custom MFN_M01 messages with an added ZFX segment and that works flawlessly. I understand there is some automatic typecasting being done by HAPI when it receives a DFT_P03 message and that is likely what i need to somehow override for it to be able to give me a DFT_P03_Custom instance instead.
If you have some insight on how i can achieve this without having to use a generic segment instance please help!
Thank you!
I finally figured this out. The only way i got this to work was to generate a conformance profile XML file (using an example message from our application as a base) with the messaging workbench on the HAPI site and use the maven plugin to generate the message and segment classes. Only with these classes am i able to correctly parse a message to my custom class. One thing to note is that it DOES NOT work if i try to use the MSH, PID, PV1 or FT1 classes provided by HAPI and use my Z-segment class. It only works if all the segments are the classes generated by the conformance plugin. This combined with a CustomModelClassFactory class (as shown on the HAPI website) and the proper package structure finally allowed me to access my Z-segment.
I have an app. I have a big button that allows the user to sync all their data at once to the cloud. A re-sync feature that allows them to send all their data again. (300+ entries)
I am using RXjava2 and retrofit2. I have my unit test working with a single call. However I need to make N network calls.
What I want to avoid is having the observable call the next item in a queue. I am at the point where I need to implement my runnable. I have seen a bit about Maps but I have not seen anyone use it as a queue. Also I want to avoid having one item fail and it report back as ALL items fail, like the Zip feature would do. Should I just do the nasty manager class that keeps track of a queue? Or is there a cleaner way to send several hundred items?
NOTE: SOLUTION CANNOT DEPEND ON JAVA8 / LAMBDAS. That has proved to be way more work than is justified.
Note all items are the same object.
#Test
public void test_Upload() {
TestSubscriber<Record> testSubscriber = new TestSubscriber<>();
ClientSecureDataToolKit clientSecureDataToolKit = ClientSecureDataToolKit.getClientSecureDataKit();
clientSecureDataToolKit.putUserDataToSDK(mPayloadSecureDataToolKit).subscribe(testSubscriber);
testSubscriber.awaitTerminalEvent();
testSubscriber.assertNoErrors();
testSubscriber.assertValueCount(1);
testSubscriber.assertCompleted();
}
My helper to gather and send all my items
public class SecureDataToolKitHelper {
private final static String TAG = "SecureDataToolKitHelper";
private final static SimpleDateFormat timeStampSimpleDateFormat =
new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
public static void uploadAll(Context context, RuntimeExceptionDao<EventModel, UUID> eventDao) {
List<EventModel> eventModels = eventDao.queryForAll();
QueryBuilder<EventModel, UUID> eventsQuery = eventDao.queryBuilder();
String[] columns = {...};
eventsQuery.selectColumns(columns);
try {
List<EventModel> models;
models = eventsQuery.orderBy("timeStamp", false).query();
if (models == null || models.size() == 0) {
return;
}
ArrayList<PayloadSecureDataToolKit> toSendList = new ArrayList<>();
for (EventModel eventModel : models) {
try {
PayloadSecureDataToolKit payloadSecureDataToolKit = new PayloadSecureDataToolKit();
if (eventModel != null) {
// map my items ... not shown
toSendList.add(payloadSecureDataToolKit);
}
} catch (Exception e) {
Log.e(TAG, "Error adding payload! " + e + " ..... Skipping entry");
}
}
doAllNetworkCalls(toSendList);
} catch (SQLException e) {
e.printStackTrace();
}
}
my Retrofit stuff
public class ClientSecureDataToolKit {
private static ClientSecureDataToolKit mClientSecureDataToolKit;
private static Retrofit mRetrofit;
private ClientSecureDataToolKit(){
mRetrofit = new Retrofit.Builder()
.baseUrl(Utilities.getSecureDataToolkitURL())
.addCallAdapterFactory(RxJavaCallAdapterFactory.create())
.addConverterFactory(GsonConverterFactory.create())
.build();
}
public static ClientSecureDataToolKit getClientSecureDataKit(){
if(mClientSecureDataToolKit == null){
mClientSecureDataToolKit = new ClientSecureDataToolKit();
}
return mClientSecureDataToolKit;
}
public Observable<Record> putUserDataToSDK(PayloadSecureDataToolKit payloadSecureDataToolKit){
InterfaceSecureDataToolKit interfaceSecureDataToolKit = mRetrofit.create(InterfaceSecureDataToolKit.class);
Observable<Record> observable = interfaceSecureDataToolKit.putRecord(NetworkUtils.SECURE_DATA_TOOL_KIT_AUTH, payloadSecureDataToolKit);
return observable;
}
}
public interface InterfaceSecureDataToolKit {
#Headers({
"Content-Type: application/json"
})
#POST("/api/create")
Observable<Record> putRecord(#Query("api_token") String api_token, #Body PayloadSecureDataToolKit payloadSecureDataToolKit);
}
Update. I have been trying to apply this answer to not much luck. I am running out of steam for tonight. I am trying to implement this as a unit test, like I did for the original call for one item.. It looks like something is not right with use of lambda maybe..
public class RxJavaBatchTest {
Context context;
final static List<EventModel> models = new ArrayList<>();
#Before
public void before() throws Exception {
context = new MockContext();
EventModel eventModel = new EventModel();
//manually set all my eventmodel data here.. not shown
eventModel.setSampleId("SAMPLE0");
models.add(eventModel);
eventModel.setSampleId("SAMPLE1");
models.add(eventModel);
eventModel.setSampleId("SAMPLE3");
models.add(eventModel);
}
#Test
public void testSetupData() {
Assert.assertEquals(3, models.size());
}
#Test
public void testBatchSDK_Upload() {
Callable<List<EventModel> > callable = new Callable<List<EventModel> >() {
#Override
public List<EventModel> call() throws Exception {
return models;
}
};
Observable.fromCallable(callable)
.flatMapIterable(models -> models)
.flatMap(eventModel -> {
PayloadSecureDataToolKit payloadSecureDataToolKit = new PayloadSecureDataToolKit(eventModel);
return doNetworkCall(payloadSecureDataToolKit) // I assume this is just my normal network call.. I am getting incompatibility errors when I apply a testsubscriber...
.subscribeOn(Schedulers.io());
}, true, 1);
}
private Observable<Record> doNetworkCall(PayloadSecureDataToolKit payloadSecureDataToolKit) {
ClientSecureDataToolKit clientSecureDataToolKit = ClientSecureDataToolKit.getClientSecureDataKit();
Observable observable = clientSecureDataToolKit.putUserDataToSDK(payloadSecureDataToolKit);//.subscribe((Observer<? super Record>) testSubscriber);
return observable;
}
Result is..
An exception has occurred in the compiler (1.8.0_112-release). Please file a bug against the Java compiler via the Java bug reporting page (http://bugreport.java.com) after checking the Bug Database (http://bugs.java.com) for duplicates. Include your program and the following diagnostic in your report. Thank you.
com.sun.tools.javac.code.Symbol$CompletionFailure: class file for java.lang.invoke.MethodType not found
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':app:compile<MyBuildFlavorhere>UnitTestJavaWithJavac'.
> Compilation failed; see the compiler error output for details.
Edit. No longer trying Lambdas. Even after setting up the path on my mac, javahome to point to 1.8, etc. I could not get it to work. If this was a newer project I would push harder. However as this is an inherited android application written by web developers trying android, it is just not a great option. Nor is it worth the time sink to get it working. Already into the days of this assignment instead of the half day it should have taken.
I could not find a good non lambda flatmap example. I tried it myself and it was getting messy.
If I understand you correctly, you want to make your calls in parallel?
So rx-y way of doing this would be something like:
Observable.fromCallable(() -> eventsQuery.orderBy("timeStamp", false).query())
.flatMapIterable(models -> models)
.flatMap(model -> {
// map your model
//avoid throwing exceptions in a chain, just return Observable.error(e) if you really need to
//try to wrap your methods that throw exceptions in an Observable via Observable.fromCallable()
return doNetworkCall(someParameter)
.subscribeOn(Schedulers.io());
}, true /*because you don't want to terminate a stream if error occurs*/, maxConcurrent /* specify number of concurrent calls, typically available processors + 1 */)
.subscribe(result -> {/* handle result */}, error -> {/* handle error */});
In your ClientSecureDataToolKit move this part into constructor
InterfaceSecureDataToolKit interfaceSecureDataToolKit = mRetrofit.create(InterfaceSecureDataToolKit.class);
With this I have added the console.log functionality to the javax.script.ScriptEngine.
public class Console {
public void log(String text){
System.out.println("console: " + text);
}
}
private static ScriptEngine getJavaScriptEngine(){
ScriptEngine engine = new ScriptEngineManager().getEngineByName("JavaScript");
Console console = new Console();
engine.put("console", console);
return engine;
}
As console and alert etc. are not part of the implementation. After a lot of searching I only found here and eleswhere only the same statement but wondering if there is not a library which does this right ?
I had a similar solution, but found that it would not format objects / arrays. Nor that it would handle multiple arguments (e.g. console.log('this is the answer:', 42)).
To work around this, I had to polyfill console. It doesn't support all of its functions, but it will do for the purpose of logging. To format objects and arrays as JSON, it appeared that the JSON object wasn't available too - so had to polyfill that object too.
There is probably a prettier solution to this, but this setup got me going:
Polyfill the JSON object, take the script mentioned here: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON#Polyfill. Rhino doesn't know about window, so replace:
if (!window.JSON) {
window.JSON = {
// ... rest of code ...
};
}
by:
// if (!window.JSON) {
// window.
JSON = {
// ... rest of code ...
};
// }
Save it under src/main/resources/scripts/json.js.
Then, for the console object, put the following content in src/main/resources/scripts/console.js:
console = {
_format: function(values) {
var msg = [];
for (var i=0;i<values.length;i++)
msg.push(JSON.stringify(values[i]));
return msg.join(', ');
},
log: function() {
log.fine(console._format(arguments));
},
info: function() {
log.info(console._format(arguments));
},
warn: function() {
log.warning(console._format(arguments));
}
};
Note that this console implementation uses a log global variable. In my case, this is a JUL Logger instance, but with a little bit of creativity it can be changed to use another logging framework (e.g. SLF4j).
Finally, to put this all together in Java code, it can be done like this:
public class ConsoleTest {
public static void main(String... args) throws ScriptException, IOException {
ScriptEngineManager factory = new ScriptEngineManager();
ScriptEngine engine = factory.getEngineByName("JavaScript");
engine.put("log", Logger.getLogger("script"));
run(engine, "/scripts/json.js");
run(engine, "/scripts/console.js");
engine.eval("console.log('string value');");
engine.eval("console.warn(['array','value']);");
engine.eval("console.info({a:1,b:'two'});");
}
private void run(ScriptEngine engine, String resourceName) throws ScriptException, IOException {
InputStream in = getClass().getResourceAsStream(resourceName);
Reader reader = new InputStreamReader(in, Charset.forName("UTF-8"));
engine.eval(reader);
reader.close();
}
}
Where can I find Jira issue type values that we pass to IssueBuilder class constructor?
For ex: If i want to create a issue type of bug using jira rest api , We pass value '1L' to Issue Builder class constructor.
IssueInputBuilder issueBuilder = new IssueInputBuilder("Key", 1l);
Similarly what are the values of other jira issue types ?.. Anybody know the values we need to pass ?
If you are using later Jira REST Java Client API (e.g. 4.0), the interface has been changed. You must use following code to browsing all issue types:
private static final String JIRA_SERVER = "http://jiralab";
public static void main(String[] args) {
try {
JiraRestClientFactory factory = new AsynchronousJiraRestClientFactory();
URI uri = new URI(JIRA_SERVER);
JiraRestClient client = factory.createWithBasicHttpAuthentication(uri, "admin", "admin");
listAllIssueTypes(client);
}
catch (Exception ex) {
}
}
private static void listAllIssueTypes(JiraRestClient client) throws Exception {
Promise<Iterable<IssueType>> promise = client.getMetadataClient().getIssueTypes();
Iterable<IssueType> issueTypes = promise.claim();
for (IssueType it : issueTypes) {
System.out.println("Type ID = " + it.getId() + ", Name = " + it.getName());
}
}
If you want to get a list of all available issuetypes, you can use the REST API (/rest/api/2/issuetype). To try that on your JIRA instance, I like to recommend the Atlassian REST API Browser.
Or just look here: Finding the Id for Issue Types
In Java you can get a list of all issuetype object using getAllIssueTypeObjects().
I am trying to use the google cloud endpoints Java from android as such:
client:
Core.Builder coreBuilder = new Core.Builder(
AndroidHttp.newCompatibleTransport(), new GsonFactory(), null);
coreBuilder.setApplicationName("myapp");
if (MainActivity.ENDPOINTS_URL != null) {
coreBuilder.setRootUrl(MainActivity.ENDPOINTS_URL);
coreBuilder.setGoogleClientRequestInitializer(new GoogleClientRequestInitializer() {
public void initialize(AbstractGoogleClientRequest<?> request)
throws IOException {
request.setDisableGZipContent(true);
}
});
}
Core core = coreBuilder.build();
myList = core.asdf("x=&+x", myObject);
server:
#ApiMethod(name = "asdf")
public List<String> asdf(#Named("param1") String param1, MyObject myObject) {
if (param1.equals("x=&+x")) {
//should go here, but never does
}
...
While it mostly works, somehow the param1 string does not get correctly transmitted, meaning that "x=&+x" arrives at the server as "x=&%2Bx". Is this a known bug? Or do arguments have to be manually encoded somehow? Or is this somehow particular to my environment?
Appengine SDK V1.8.8 for java, google api 1.17.0-rc, using the dev environment.
Cheers,
Andres