In Scala, I'm using the AWS Java SDK to upload a simple JSON string (containing a few rows of data from a database). The upload appears to work fine but the content of the uploaded file once it's in the S3 bucket is just "[value: string]" instead of the actual data. Before I upload I print the data out and confirm the data type is String. How do I fix this? I'm assuming it's one small tweak, but not sure what.
(I can't write the data to a file before upload, it has to be from the string, though I'd be willing to massage that into an InputStream or whatever, if need be - not sure how to do that though).
import com.amazonaws.services.s3.AmazonS3Client
import com.amazonaws.auth.BasicAWSCredentials
import com.amazonaws.ClientConfiguration
val query = "[My SQL query]";
val results = spark.sql(query);
val JSONdata = results.toJSON;
var JSONstring:String = JSONdata.toString;
try {
println(JSONstring); // Displays my valid JSON data
println(JSONstring.getClass); // Displays "class java.lang.String"
val result = amazonS3Client.putObject(bucketName, objectKey, JSONstring);
println("Result: " + result.getETag());
} catch {
case e: Exception => println("Error: " + e.getMessage());
}
Thanks
My suspicions is that instead of writing the actual string it tries to write the string object. Maybe it would work if you'd call the putObject method with a parameter of type byte array.
I think some of the answers here might help you
I'm trying to do a loader for my program and the saver into json works, but the reader doesn't. I'm basically converting my list of objects, let's say ArrayList ProjectArr into json which is working, now I'm gonna close terminal and load the info into my initially blank ProjectArr list so I can add the saved data loading feature. here is my reader :
public static ArrayList<Project> MapReaderforProject() {
ObjectMapper objectMapperforProject = new ObjectMapper();
String jsonfile = "C:/Users/elmm/Desktop/DS Project/DsProject/target/test1.json";
File file = new File("C:/Users/elmm/Desktop/DS Project/DsProject/target/test1.json");
try {
ProjectArr = objectMapperforProject.readValue(jsonfile,new TypeReference<ArrayList<Project>>(){});
}
catch (JsonMappingException e) {
} catch (JsonProcessingException e) {
}
System.out.println(ProjectArr.toString());
return ProjectArr;
}
But when I call this even though the JSON file is working and has the info, it's not reading anything and just prints [], empty list basically meaning it's not reading the data from JSON back into my list. Can someone please guide me?
There is some context information missing to give you more help. For example, what is ObjectMapper? Does it come from a library (and from which one) or have you written it yourself?
But what I have recognized so far: In line 3 you create a String which contains the file name. In line 4 you create a File object with the same file name. You could use the String there which you created the line above.
In line 6 you call the readValue method on the ObjectMapper instance and pass the String created in line 3, which contains the file name. Does this method really want a String there with the file name? If this is the ObjectMapper from the Jackson library, you need to parse the JSON code as the first parameter to the readValue call, not the file name.
So it looks like after line 4 you need code to read the content of the file which you should pass to the readValue method.
Check out this short tutorial: https://www.baeldung.com/jackson-object-mapper-tutorial
hth
McPringle
So I'm in the process of developing a Java IRC bot as a bit of a side project for a friend of mine, and while development is going well, I'm a little unsure as how to save the current state of certain variables in between sessions. It doesn't have a GUI, so I didn't think that it would be too complex, but my searching efforts have been futile thus far.
Thanks.
It will depend on the sort of variables you want to keep, but all methods will require you to write some sort data to a file.
If you only need to keep a handful of variables, you could consider implementing a .config file that could be a simple delimited text file.
If it's an entire object that you want to keep track of, say, a player in an irc game, one option you have is to parse the object into JSON, and save it to a textfile, for reading later. You can use Gson for this
example for a 'player' object:
public String savePlayer(String playerName){
Gson gsonPretty = new GsonBuilder().setPrettyPrinting().create();
String playerFile = System.getProperty("user.dir")+"\\players\\"+playerName;
String jsonplayers = gsonPretty.toJson(players.get(playerName));
try{
FileWriter writer = new FileWriter(playerFile+".json");
writer.write(jsonplayers);
writer.close();
return "Player file saved successfully!";
}catch(Exception e){
e.printStackTrace();
}
return "Something went wrong";
}
you can then create a load method that either has the file name hard coded, or a string input to determine which file to load, and use something like:
playerFromJson = gson.fromJson(jsonString, Player.class);
to use that object in the code
My idea is to upload an int array from a Java servlet running on an AWS ec2 microinstance. As I understand it I would have to convert my int array to an java object file first and then upload the file into my bucket, but is there a way to do this "on the fly" without first creating a local file?
If I had to create a local file first, which pathname would it have?
It will like this:
public void arrayToS3(String bucket, String pathInS3, JSONArray array) {
ObjectMetadata metadata = new ObjectMetadata();
byte[] dataInMemory = array.toString().getBytes();
s3client.putObject(bucket, pathInS3, new ByteArrayInputStream(dataInMemory), metadata);
}
Just convert anything into IntputStream. For example, the method arrayToS3 convert the JSONArray to String, and convert String to byte[]. Finally, wrapping byte[] to InputStream.
Everything is in memory. If your data is not very large, it is a simple way to do that. If you data is bigger than the memory set by JVM, out-of-memory will hit you.
I'm creating an installer and there are some resource files (.xmls, .zip files, a .jar file, etc) that must be read during installation, but I'd like to pack them into a custom file (i.e., a .dat file) so that when distributed, users don't get to mess around with them too much. The problem is that the installer must be written in Java and I've never done this sort of thing before in any programming language. Is it even possible? If so, how can I pack it in a way that can be read by my Java app afterwards and how can I make my Java app read it?
There are a lot of questions you'll need to answer for yourself about the requirements of this filetype. Does it need to be compressed? Encrypted? Does it need to support random access reading, or is stream-reading good enough?
I could be wrong, but I don't think that's what you're asking in this question. If I'm reading you correctly, I think you're asking "how do I read & write arbitrary file data?"
So that's the question I'll answer. Update your question if that's not quite what you're looking for.
Custom filetypes can easily be implemented using the DataInputStream and DataOutputStream classes. These will let you read & write primitives (boolean, char, byte, int, long, float, double) to the stream. There are also some convenience methods for reading & writing UTF-8 encoded Strings, byte-arrays, and a few other goodies.
Let's get started.
For the sake of argument, let's pretend that all my data elements are byte arrays. And each of them has a name. So my filetype can be modeled logically as a Map<String, byte[]>. I'd implement my custom filetype reader/writer class like this:
public class MyFileTypeCodec {
public static void writeToFile(File f, Map<String, byte[]> map)
throws IOException {
// Create an output stream
DataOutputStream stream = new DataOutputStream(
new BufferedOutputStream(new FileOutputStream(f))
);
// Delegate writing to the stream to a separate method
writeToStream(stream, map);
// Always be sure to flush & close the stream.
stream.flush();
stream.close();
}
public static Map<String, byte[]> readFromFile(File f)
throws IOException {
// Create an input stream
DataInputStream stream = new DataInputStream(
new BufferedInputStream(new FileInputStream(f))
);
// Delegate reading from the stream to a separate method
Map<String, byte[]> map = readFromStream(stream);
// Always be sure to close the stream.
stream.close();
return map;
}
public static void writeToStream(DataOutputStream stream, Map<String, byte[]> map)
throws IOException {
// First, write the number of entries in the map.
stream.writeInt(map.size());
// Next, iterate through all the entries in the map
for (Map.Entry<String, byte[]> entry : map.entrySet()) {
// Write the name of this piece of data.
stream.writeUTF(entry.getKey());
// Write the data represented by this name, making sure to
// prefix the data with an integer representing its length.
byte[] data = entry.getValue();
stream.writeInt(data.length);
stream.write(data);
}
}
public static Map<String, byte[]> readFromStream(DataInputStream stream)
throws IOException {
// Create the data structure to contain the data from my custom file
Map<String, byte[]> map = new HashMap<String, byte[]>();
// Read the number of entries in this file
int entryCount = stream.readInt();
// Iterate through all the entries in the file, and add them to the map
for (int i = 0; i < entryCount; i++) {
// Read the name of this entry
String name = stream.readUTF();
// Read the data associated with this name, remembering that the
// data has an integer prefix representing the array length.
int dataLength = stream.readInt();
byte[] data = new byte[dataLength];
stream.read(data, 0, dataLength);
// Add this entry to the map
map.put(name, data);
}
return map;
}
}
The basic idea is that you can write any data to an output stream (and read it back again) if you can represent that data as some combination of primitives. Arrays (or other collections) can be prefixed with their length, like I've done here. Or you can avoid writing the length prefix if you put a TERMINUS sentinel at the end (kind of like null-terminated strings).
I always use this kind of setup when I implement a custom filetype codec, with file IO methods delegating down into stream IO methods. Usually, I discover later that the object I'm reading & writing from this stream could be just as easily written into some larger & more complex file.
So I might have a SuperFancyCodec for reading/writing the data for my whole system, and it calls down into my TinySpecialPurposeCodec. As long as the stream reading & writing methods are public, then I can assemble new filetypes using a component-oriented methodology.
The extension usually have very little to do with how the file is interpreted.
If you'd like to have just config.dat instead of config.xml you just rename the file. (You'd typically give an xml-parser an InputStream or a Reader as input, which may read any file, regardless of extension)
If the problem you're describing is about combining multiple files, (.zip, .jar, etc) into a single .dat file, you could for instance zip them together, and name the zip file with a .dat extension. Java has good support for zip-files and can handle the zip file regardless of filename / extension.
Related link: Reading the Contents of a ZIP File
When creating/reading files in Java (or anything else), the file extension is not strictly tyed to the actual structure of the file's data. If I wanted, I could make an XML files file.gweebz. OS's and applications would not know what to do with it, but once opened, it would be clear that it is XML.
That being said, it is often good to follow the conventions already established and usually .dat files are files in a binary format. You can use .dat for what you want, but be warned that some users may have OS bindings for the file type and clicking on your file may cause different-than-expected behavior on their systems.
As for how to do it in Java. Grabbing a file handle in Java is easy...
File myFile = new File("/dir/file.gweebz");
It is as simple as that and you can name it whatever you want. You will need other classes to write and read from the file or to do compression, but I will assume you know how to do that. If not, this site will have the answer.