In Google App Engine, I tried reading a .txt file from a URL. Because the maximum allowed size is 1MB and the file is slightly larger, I'm using an alternative method described here.
So, what I'm trying to do is this:
FetchOptions fo = FetchOptions.Builder.allowTruncate().doNotFollowRedirects();
HTTPRequest request = new HTTPRequest(url,HTTPMethod.GET,FetchOptions.Builder.allowTruncate());
URLFetchService service = URLFetchServiceFactory.getURLFetchService();
HTTPResponse response = service.fetch(request);
My question is now, how can I read this response line by line? I'm trying to process each line which should be possible somehow as the source file is a simple text file.
I can get a byte[] with
byte[] content = response.getContent();
but I'm struggling with the further processing of it.
Or, can I do something completely different to achieve the same thing ?
I'm trying to read it line by line because I don't need all the lines. Processing would be much easier than to put everything in one large string.
You can try:
ByteArrayInputStream bais = new ByteArrayInputStream(content);
BufferedReader reader = new BufferedReader(new InputStreamReader(bais, "utf-8"));
String line = null;
while ((line = reader.readLine()) != null) {
...
}
Alternatively, you can use IOUtils and call IOUtils.lineIterator(reader) (where reader is the InputStreamReader)
Related
I am familiar with the basics of making a GET request in Java using the HttpURLConnection class. In a normal situation where the return type would be a JSON, I'd do something like this:
URL obj = new URL(myURLString);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("GET");
int responseCode = con.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String inpLine;
StringBuffer resp = new StringBuffer();
while ((inpLine = in.readLine()) != null) {
resp.append(inputLine);
}
in.close();
System.out.println(resp.toString());
} else { System.out.println("Request failed");
However, the current endpoint I'm trying to work with sends back a zip file containing various types of files. The 'Content-Type' in that case would be 'application/octet-stream'. Hitting that endpoint with the above code results in a mess of symbols (not sure what it's called) written to the console. Even in Postman, I see the same, and it only works when I use the 'Send & Download' option when making the request, which prompts me to save the response and I get the zip file.
Any help on how to hit the API and download the returned zip file through Java? Thank you in advance.
Edit: What I intend to do with the zip file is save it locally and then other parts of my program will work with the contents.
I guess you can try this API:
try (ZipInputStream zis = new ZipInputStream(con.getInputStream())) {
ZipEntry entry; // kinda self-explained (a file inside zip)
while ((entry = zis.getNextEntry()) != null) {
// do whatever you need :)
// this is just a dummy stuff
System.out.format("File: %s Size: %d Last Modified %s %n",
entry.getName(), entry.getSize(),
LocalDate.ofEpochDay(entry.getTime() / MILLS_IN_DAY));
}
}
In any case, you get a stream, so you can do all the stuff that java IO API allows you to do.
For instance, to save the file you can do something like:
// I am skipping here exception handling, closing stream, etc.
byte[] zipContent = zis.readAllBytes();
new FileOutputStream("some.zip").write(zipContent);
To do something with files inside zip you can do something like:
// again, skipping exceptions, closing, etc.
// also, you'd probably do this in while loop as in the first example
// so let's say we get ZipEntry
ZipEntry entry = zis.getNextEntry();
// crate OutputStream to extract the entry from zip file
final OutputStream os = new FileOutputStream("c:/someDir/" + entry.getName());
byte[] buffer = new byte[1024];
int length;
//read the entry from zip file and extract it to disk
while( (length = zis.read(buffer)) > 0) {
os.write(buffer, 0, length);
}
// at this point you should get your file
I know, this is kinda low level API, you need to deal with streams, bytes, etc., perhaps there are some libs that allows to do this with a single line of code or something :)
I have decrypted data in bytearrayoutputstream. I want to read the data in each line(not sure if that is possible).Could any one guide how I can do that.
The main requirement is to read a encrypted file , decrypt and read the data without writing into the disk. I have already covered encrypt and decrypt part but unable to read the data without writing into disk.Some suggested to use bytearrayoutputStream so stuck now.
ByteArrayOutputStream byteArrayOutputStream=new ByteArrayOutputStream(inputBytes.length);
byteArrayOutputStream.write(outputBytes);
if i simply print the variable it give me all the data at once as below.
SQlServer,"connection string","user name","password"
Oracle,"connection string","user name","password"
I am trying to read the data line wise so i can match the servername and fetch the user name and other details.
To read a byte[] you can use
ByteArrayInputStream in = new ByteArrayInputStream(outputBytes);
and to read this as lines of text you can use
BufferedReader br = new BufferedReader(new InputStreamReader(in));
for (String line; (line = br.readLine()) != null; ) {
// do something with the line
}
I currently am accessing a streaming h264 file and want to save it off for the ability to slice frames. However, I'm having issues saving/opening the .flv file
When pointing to the URL in the address bar - I am told it's an x-flv file.
I then attempt to do the following to save a chunk of the stream.
URL url = new URL("http://foo.bar.com/foo/bar");
HttpURLConnection conn = (HttpURLConnection) url
.openConnection(proxy);
conn.setRequestMethod("GET");
File f = new File("C:\\tmpArea\\tmp.flv");
FileWriter fr = new FileWriter(f);
bw = new BufferedWriter(fr);
BufferedReader br = new BufferedReader(new InputStreamReader(
(conn.getInputStream())));
String output = "";
int i = 0;
while (((output = br.readLine()) != null) && i < 100000) {
bw.write(output);
i++;
}
Upon doing this I've attempted to open the file in VLC Media Player and am told:
No suitable decoder module: VLC does not support the audio or video format "undf".
Unfortunately there is no way for you to fix this.
I then thought, well maybe it's not really an FLV file and that's on me. So I used a run of the mill hex-editor. Opening up the file in the HexEditor gives me the following information:
FLV
onMetaData
duration
width
height
videodatarate
framerate
videocodecid
audiodatarate
audiosamplerate
audiosamplesize
stereo
audiocodecid
encoder
Lavf52.10.6.0
filesize....
Is there a different way I should be trying to save off this data? Is there a conversion/codec issue I'm not seeing?
You are using a Reader and Writer, which are intended to read bytes and convert them to characters, to read a binary file that consists of bytes. The conversion from bytes to characters will corrupt the data. You should be using InputStream and OutputStream instead.
I am using HttpClient to send a request a server which is supposed to return xml data. This data is returned as chunked data. I am then trying to write the received xml data to a file. The code I use is shown below:
HttpEntity entity = response.getEntity();
InputStream instream = entity.getContent();
try {
// do something useful
InputStreamReader isr = new InputStreamReader(instream);
FileWriter pw;
pw = new FileWriter(filename, append);
OutputStreamWriter outWriter = new OutputStreamWriter(new FileOutputStream(filename, append), "UTF-8");
BufferedReader rd = new BufferedReader(isr);
String line = "";
while ((line = rd.readLine()) != null) {
// pw.write(line);
outWriter.write(line);
}
isr.close();
pw.close();
} finally {
instream.close();
}
This results in data that looks as follows to be printed to the file:
This code works for non chunked data. How do I properly handle chunked data responses using HttpClient. Any help is greatly appreciated.
I don't think that your problem is the chunking of data.
XML data is plain text data - chunking it means that it is split into several parts that are transfered after another. Therefore each chunk should contain visible plain text xml data which is obviously not the case as shown in the data picture.
May be the content is encoded compressed via gzip or it is not plain text XML but binary encoded XML (e.g. like WBXML).
What concrete type you have you can see from the sent server response headers, especially the used mime type it contains.
I cannot read and write extended characters (French accented characters, for example) to a text file using the standard InputStreamReader methods shown in the Android API examples. When I read back the file using:
InputStreamReader tmp = new InputStreamReader(in);
BufferedReader reader = new BufferedReader(tmp);
String str;
while ((str = reader.readLine()) != null) {
...
the string read is truncated at the extended characters instead of at the end-of-line. The second half of the string then comes on the next line. I'm assuming that I need to persist my data as UTF-8 but I cannot find any examples of that, and I'm new to Java.
Can anyone provide me with an example or a link to relevant documentation?
Very simple and straightforward. :)
String filePath = "/sdcard/utf8_file.txt";
String UTF8 = "utf8";
int BUFFER_SIZE = 8192;
BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream(filePath), UTF8),BUFFER_SIZE);
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(filePath), UTF8),BUFFER_SIZE);
When you instantiate the InputStreamReader, use the constructor that takes a character set.
InputStreamReader tmp = new InputStreamReader(in, "UTF-8");
And do a similar thing with OutputStreamWriter
I like to have a
public static final Charset UTF8 = Charset.forName("UTF-8");
in some utility class in my code, so that I can call (see more in the Doc)
InputStreamReader tmp = new InputStreamReader(in, MyUtils.UTF8);
and not have to handle UnsupportedEncodingException every single time.
this should just work on Android, even without explicitly specifying UTF-8, because the default charset is UTF-8. if you can reproduce this problem, please raise a bug with a reproduceable test case here:
http://code.google.com/p/android/issues/entry
if you face any such kind of problem try doing this. You have to Encode and Decode your data into Base64. This worked for me. I can share the code if you need it.
Check the encoding of your file by right clicking it in the Project Explorer and selecting properties. If it's not the right encoding you'll need to re-enter your special characters after you change it, or at least that was my experience.