Develop Reference

Downloading corrupted files with OkHttp

The method I wrote to download files always produce corrupted files.
public static String okDownloadToFileSync(final String link, final String fileName, final boolean temp, DownloadStatusManager statusManager, ErrorDisplayerInterface errorDisplayerInterface) {
Request request = new Request.Builder()
.url(link)
.build();
OkHttpClient client = Api.getInstance().getOkHttpClient();
OutputStream output = null;
InputStream input = null;
try {
Response response = client.newCall(request).execute();
//Add the file length to the statusManager
final int contentLength = Integer.parseInt(response.header("Content-Length"));
if (statusManager != null) {
statusManager.add(Hash.md5(link), contentLength);
}
//Get content type to know extension
final String contentType = response.header("Content-Type");
final String ext = contentTypeMap.get(contentType);
Log.i(TAG, link + "\n --> contentType = " + contentType + "\n --> ext = " + ext);
if (ext == null) {
Log.e(TAG, "-----------\next is null, seems like there is a problem with that url : \n " + link + "\n----------");
return null;
} else if (ext.equals("json")) {
Log.e(TAG, "-----------\ndownloadable file seems to be a json, seems like there is a problem with that url : \n " + link + "\n----------");
return null;
}
//Check if file already exists
if (!temp && fileName != null) {
File test = new File(M360Application.getContext().getFilesDir(), fileName + "." + ext);
if (test.exists()) {
Log.i(TAG, "File exists ! : " + test.getPath());
test.delete();
//return test.getAbsolutePath();
}
}
// expect HTTP 200 OK, so we don't mistakenly save error report instead of the file
if (!response.isSuccessful()) {
errorDisplayerInterface.popWarn(null, "Error while downloading " + link, "connection.getResponseCode() != HttpURLConnection.HTTP_OK");
return null;
}
input = response.body().byteStream();
File file;
if (temp) {
file = File.createTempFile(UUID.randomUUID().toString(), ext, M360Application.getContext().getCacheDir());
} else {
file = new File(M360Application.getContext().getFilesDir(), fileName + "." + ext);
}
output = new FileOutputStream(file);
output.write(response.body().bytes());
// byte data[] = new byte[4096];
// long total = 0;
// int count;
// while ((count = input.read(data)) != -1) {
// output.write(data, 0, count);
// total++;
//
// if (statusManager != null) {
// statusManager.update(Hash.md5(link), contentLength - total);
// }
// }
return file.getAbsolutePath();
} catch (IOException e) {
e.printStackTrace();
errorDisplayerInterface.popError(null, e);
} finally {
if (statusManager != null) {
statusManager.finish(Hash.md5(link));
}
try {
if (output != null)
output.close();
if (input != null)
input.close();
} catch (IOException ignored) {
ignored.printStackTrace();
}
}
return null;
}
I access these file via adb, transfer them to my sccard, and there I see that they seem to have the proper size, but has no type according to for instance Linux file command.
Would you know what is missing and how to fix it?
Thank you.
Edit
Simpler version of the code ( but the bug is the same )
public static String okioDownloadToFileSync(final String link, final String fileName) throws IOException {
Request request = new Request.Builder()
.url(link)
.build();
OkHttpClient client = Api.getInstance().getOkHttpClient();
Response response = client.newCall(request).execute();
final int contentLength = Integer.parseInt(response.header("Content-Length"));
//Get content type to know extension
final String contentType = response.header("Content-Type");
final String ext = contentTypeMap.get(contentType);
// expect HTTP 200 OK, so we don't mistakenly save error report instead of the file
if (!response.isSuccessful()) {
return null;
}
File file = new File(M360Application.getContext().getFilesDir(), fileName + "." + ext);
BufferedSink sink = Okio.buffer(Okio.sink(file));
sink.writeAll(response.body().source());
sink.close();
Log.i(TAG, "file.length : " + file.length() + " | contentLength : " + contentLength);
return file.getAbsolutePath();
}
The log : file.length : 2485394 | contentLength : 1399242
Solution
The problem was that I was getting the OkHttpClient from my API singleton, which was used by retrofit and had multiples interceptors. Those interceptors were polluting the response.
So I OkHttpClient client = Api.getInstance().getOkHttpClient(); became OkHttpClient client = new OkHttpClient.Builder().build(); and everything is now ok !
Thanks a lot. I'm dividing the method into smaller pieces right now.

Instead of
output.write(response.body().bytes());
try something like this
byte[] buff = new byte[1024 * 4];
while (true) {
int byteCount = input.read(buff);
if (byteCount == -1) {
break;
}
output.write(buff, 0, byteCount);
}

Uploading multipart file from android to WCF

I am trying to upload .JPG file from android to WCF webservice.
To upload the file from Android i tried 2 ways:
1- Retrofit:
#Multipart
#POST("/UploadFile/{fileName}")
void UploadFile(#Path("fileName") String fileName, #Part("image") TypedFile image, Callback<String> callBack);
2- Android Asynchronous Http Client
here there was 2 options for uploading it:
a- Adding InputStream to the RequestParams:
RequestParams params = new RequestParams();
try {
InputStream stream = new FileInputStream(fileImage);
params.put("image", stream, fileImage.getName() );
client.post(Constants.SERVICE_URL + "/UploadFile/" + fileImage.getName()
, params, getResponseHandler());
} catch (Exception e) {
Utils.LogError("ERROR: " + e.getLocalizedMessage());
}
b- Ading File object to the RequestParams:
RequestParams params = new RequestParams();
try {
//InputStream stream = new FileInputStream(fileImage);
params.put("image", fileImage);
client.post(Constants.SERVICE_URL + "/UploadFile/" + fileImage.getName()
, params, getResponseHandler());
} catch (Exception e) {
Utils.LogError("ERROR: " + e.getLocalizedMessage());
}
All those, sent successfully to server, The received file before parsing looks something like this:
--b1b13fd2-4212-45bb-bb5c-fd4dc074fd1b
Content-Disposition: form-data; name="image"; filename="71d9d7fc-cfa8-40b6-b7aa-5c287cf31c72.jpg"
Content-Type: image/jpeg
Content-Length: 2906
Content-Transfer-Encoding: binary
���� JFIF �� C .................very long string of this stuff
Þq�Ã�9�A?� �0pi1�zq�<�#��:��PV���]|�e�K�mv �ǜ.1�q���&��8��u�m�?�ӵ/���0=8�x�:t�8��>�ׁ���1�POM�k����eea1��ǧq�}8�6��q� � �� .;p1K�g�Onz�Q�oås�a�׌p1�?>3#���z��0=��m$�H ǧ��Ӄ�v?��x��<q��.8܃��� ��2}1�� c���ϧ q�oA�Rt>��t�=�?����2y�q�큊A����:��q�#���_�~�Q�w��Pu��Ƿ�q�#q��{cۦ���}0:b�|�=#��9�BEV���?O��װ�g���׎z<N� ��� v�=�?������=�<}x�#'�d�8��׌e����,�\�4wVV���f�pB���㢁�L{��%$�v裶G8x��b�?���� �]�=:�ӕ����
--b1b13fd2-4212-45bb-bb5c-fd4dc074fd1b--
So I used mulipart parser in order to take out the bytes of the file, the write them to file on the server to finish the uploading.
Here is the code of the multipartparser i used:
public class MultipartParser
{
public MultipartParser(string contents)
{
this.Parse(contents);
}
private void Parse(string contents)
{
Encoding encoding = Encoding.UTF8;
this.Success = false;
// Read the stream into a byte array
byte[] data = encoding.GetBytes(contents);
// Copy to a string for header parsing
string content = contents;
// The first line should contain the delimiter
int delimiterEndIndex = content.IndexOf("\r\n");
if (delimiterEndIndex > -1)
{
string delimiter = content.Substring(0, content.IndexOf("\r\n"));
// Look for Content-Type
Regex re = new Regex(#"(?<=Content\-Type:)(.*?)(?=\r\n)");
Match contentTypeMatch = re.Match(content);
// Look for filename
re = new Regex(#"(?<=filename\=\"")(.*?)(?=\"")");
Match filenameMatch = re.Match(content);
#region added
re = new Regex(#"(?<=Content\-Transfer\-Encoding:)(.*?)(?=\r\n\r\n)");
Match contentTransferEncodingMatch = re.Match(content);
#endregion
// Did we find the required values?
if (contentTypeMatch.Success && filenameMatch.Success && contentTransferEncodingMatch.Success)
{
// Set properties
this.ContentType = contentTypeMatch.Value.Trim();
this.Filename = filenameMatch.Value.Trim();
this.ContentEncoding = contentTransferEncodingMatch.Value.Trim();
// Get the start & end indexes of the file contents
//int startIndex = contentTypeMatch.Index + contentTypeMatch.Length + "\r\n\r\n".Length;
int startIndex = contentTransferEncodingMatch.Index + contentTransferEncodingMatch.Length + "\r\n\r\n".Length;
byte[] delimiterBytes = encoding.GetBytes("\r\n" + delimiter);
string finalDelimeterStr = "\r\n"+delimiter + "--";
byte[] endDilimeterBytes = encoding.GetBytes(finalDelimeterStr);
//byte[] fileBytes = Array.Copy()
//int endIndex = IndexOf(data, endDilimeterBytes, startIndex);
int endIndex = SimpleBoyerMooreSearch(data, endDilimeterBytes);
int contentLength = endIndex - startIndex;
// Extract the file contents from the byte array
byte[] fileData = new byte[contentLength];
Buffer.BlockCopy(data, startIndex, fileData, 0, contentLength);
this.FileContents = fileData;
this.Success = true;
}
}
}
public int SimpleBoyerMooreSearch(byte[] haystack, byte[] needle)
{
int[] lookup = new int[256];
for (int i = 0; i < lookup.Length; i++) { lookup[i] = needle.Length; }
for (int i = 0; i < needle.Length; i++)
{
lookup[needle[i]] = needle.Length - i - 1;
}
int index = needle.Length - 1;
byte lastByte = needle.Last();
while (index < haystack.Length)
{
var checkByte = haystack[index];
if (haystack[index] == lastByte)
{
bool found = true;
for (int j = needle.Length - 2; j >= 0; j--)
{
if (haystack[index - needle.Length + j + 1] != needle[j])
{
found = false;
break;
}
}
if (found)
return index - needle.Length + 1;
else
index++;
}
else
{
index += lookup[checkByte];
}
}
return -1;
}
public static byte[] ToByteArray(Stream stream)
{
byte[] buffer = new byte[32768];
using (MemoryStream ms = new MemoryStream())
{
while (true)
{
int read = stream.Read(buffer, 0, buffer.Length);
if (read <= 0)
return ms.ToArray();
ms.Write(buffer, 0, read);
}
}
}
public bool Success
{
get;
private set;
}
public string ContentType
{
get;
private set;
}
public string ContentEncoding
{
get;
private set;
}
public string Filename
{
get;
private set;
}
public byte[] FileContents
{
get;
private set;
}
}
The parser is taking out the bytes, and parse the received multipart file.
The result file is not showing and it shows error reading file or something.
What i noticed after comparing the files it that the original and received file are different, here is the comparison in Notepad++:
some letters are exists in the original and not exists in the received!
here is the WCF Function declaration and code:
IService.cs:
[OperationContract]
[WebInvoke(Method = "POST", UriTemplate = "/UploadFile/{fileName}"
, ResponseFormat = WebMessageFormat.Json)]
string UploadFile(string fileName ,Stream image);
Service.cs:
public string UploadFile(string fileName, Stream image)
{
string dirPath = System.Web.Hosting.HostingEnvironment.MapPath("~/Logs/");
//string path = dirPath+"log.txt";
// Read the stream into a byte array
byte[] data = MultipartParser.ToByteArray(image);
// Copy to a string
string content = Encoding.UTF8.GetString(data);
File.WriteAllText(dirPath + fileName + ".txt", content); // for checking the result file
MultipartParser parser = new MultipartParser(content);
if (parser != null )
{
if (parser.Success)
{
if (parser.FileContents == null)
return "fail: Null Content";
byte[] bitmap = parser.FileContents;
File.WriteAllBytes(dirPath + fileName +"contents",bitmap);
try
{
using (Image outImage = Image.FromStream(new MemoryStream(bitmap)))
{
outImage.Save(fileName, ImageFormat.Jpeg);
}
return "success";
}
catch (Exception e)
{ // I get this exception all the time
return "Fail: e " + e.Message;
}
}
return "fail not success";
}
return "fail";
}
I tried every possible solution came to my mind, still could not get whats wrong!!! is the problem in the encoding while sending or the parser!??
Please what can be the problem!? i am struggling with this for 3 days!
Thank you ALL :)

The were 2 problems:
The parser
Decoding the bytes to string was not good idea.
I modified the parser to this one and it will take care of the rest:
public class MultipartParser
{
public MultipartParser(Stream stream)
{
this.Parse(stream);
}
private void Parse(Stream stream)
{
this.Success = false;
if(!stream.CanRead)
return;
// Read the stream into a byte array
byte[] data = MultipartParser.ToByteArray(stream);
if (data.Length < 1)
return;
// finding the delimiter (the string in the beginning and end of the file
int delimeterIndex = MultipartParser.SimpleBoyerMooreSearch(data, Encoding.UTF8.GetBytes("\r\n")); // here we got delimeter index
if (delimeterIndex == -1) return;
byte[] delimeterBytes = new byte[delimeterIndex];
Array.Copy(data, delimeterBytes, delimeterIndex);
// removing the very first couple of lines, till we get the beginning of the JPG file
byte[] newLineBytes = Encoding.UTF8.GetBytes("\r\n\r\n");
int startIndex = 0;
startIndex = MultipartParser.SimpleBoyerMooreSearch(data, newLineBytes);
if (startIndex == -1)
return;
int startIndexWith2Lines = startIndex + 4; // 4 is the bytes of "\r\n\r\n"
int newLength = data.Length - startIndexWith2Lines;
byte[] newByteArray = new byte[newLength];
Array.Copy(data, startIndex + 4, newByteArray, 0, newLength - 1);
// check for the end of the stream, is ther same delimeter
int isThereDelimeterInTheEnd = MultipartParser.SimpleBoyerMooreSearch(newByteArray, delimeterBytes);
if (isThereDelimeterInTheEnd == -1) return; // the file corrupted so
int endIndex = isThereDelimeterInTheEnd - delimeterBytes.Length;
byte[] lastArray = new byte[endIndex];
Array.Copy(newByteArray, 0, lastArray, 0, endIndex);
this.FileContents = lastArray;
this.Success = true;
}
static byte[] GetBytes(string str)
{
byte[] bytes = new byte[str.Length * sizeof(char)];
System.Buffer.BlockCopy(str.ToCharArray(), 0, bytes, 0, bytes.Length);
return bytes;
}
static string GetString(byte[] bytes)
{
char[] chars = new char[bytes.Length / sizeof(char)];
System.Buffer.BlockCopy(bytes, 0, chars, 0, bytes.Length);
return new string(chars);
}
public static int SimpleBoyerMooreSearch(byte[] haystack, byte[] needle)
{
int[] lookup = new int[256];
for (int i = 0; i < lookup.Length; i++) { lookup[i] = needle.Length; }
for (int i = 0; i < needle.Length; i++)
{
lookup[needle[i]] = needle.Length - i - 1;
}
int index = needle.Length - 1;
byte lastByte = needle.Last();
while (index < haystack.Length)
{
var checkByte = haystack[index];
if (haystack[index] == lastByte)
{
bool found = true;
for (int j = needle.Length - 2; j >= 0; j--)
{
if (haystack[index - needle.Length + j + 1] != needle[j])
{
found = false;
break;
}
}
if (found)
return index - needle.Length + 1;
else
index++;
}
else
{
index += lookup[checkByte];
}
}
return -1;
}
private int IndexOf(byte[] searchWithin, byte[] serachFor, int startIndex)
{
int index = 0;
int startPos = Array.IndexOf(searchWithin, serachFor[0], startIndex);
if (startPos != -1)
{
while ((startPos + index) < searchWithin.Length)
{
if (searchWithin[startPos + index] == serachFor[index])
{
index++;
if (index == serachFor.Length)
{
return startPos;
}
}
else
{
startPos = Array.IndexOf<byte>(searchWithin, serachFor[0], startPos + index);
if (startPos == -1)
{
return -1;
}
index = 0;
}
}
}
return -1;
}
public static byte[] ToByteArray(Stream stream)
{
byte[] buffer = new byte[32768];
using (MemoryStream ms = new MemoryStream())
{
while (true)
{
int read = stream.Read(buffer, 0, buffer.Length);
if (read <= 0)
return ms.ToArray();
ms.Write(buffer, 0, read);
}
}
}
public bool Success
{
get;
private set;
}
public byte[] FileContents
{
get;
private set;
}
}
So you can use this parser for this kind of multipart files encoding:
--b1b13fd2-4212-45bb-bb5c-fd4dc074fd1b
Content-Disposition: form-data; name="image"; filename="71d9d7fc-cfa8-40b6-b7aa-5c287cf31c72.jpg"
Content-Type: image/jpeg
Content-Length: 2906
Content-Transfer-Encoding: binary
���� JFIF �� C .................very long string of this stuff
Þq�Ã�9�A?� �0pi1�zq�<�#��:��PV���]|�e�K�mv �ǜ.1�q���&��8��u�m�?�ӵ/��Ƿ�q�#q��{cۦ���}0:b�|�=#��9�BEV���?O��װ�g���׎z<N� ��� v�=�?������=�<}x�#'�d�8��׌e����,�\�4wVV���f�pB���㢁�L{��%$�v裶G8x��b�?���� �]�=:�ӕ����
--b1b13fd2-4212-45bb-bb5c-fd4dc074fd1b--
Hope it helps somebody else.

You could try to encode the jpeg to base64 before sending it. As far as I know, this is a proper solution. Decoding it on the server, should be no problem. (Sry, I wanted to write a comment - but I'm not allowed to it)

Find file with the most update information

I have a list of log files, and I need to find which one has a latest edition of a specific line, and all or none could have this line.
The lines in the files look like this:
2013/01/06 16:01:00:283 INFO ag.doLog: xxxx xxxx xxxx xxxx
And I need a line lets say
xx/xx/xx xx:xx:xx:xxx INFO ag.doLog: the line i need
I know how to get an array of files, and if I scan backwards I could find the latest latest line in each file (if it exists).
Biggest problem is that the file could be big (2k lines?) and I want to find the line in a relative fast way (a few seconds), so I am open for suggestion.
Personal ideas:
If a file has the line at X time, then any file that has not found the line before X time should not be scan anymore. This will require to search all files at the same time, which i dont know how.
Atm the code breaks, and I suppose if lack of memory.
Code:
if(files.length>0) { //in case no log files exist
System.out.println("files.length: " + files.length);
for(int i = 0; i < files.length; i++) { ///for each log file look for string
System.out.println("Reading file: " + i + " " + files[i].getName());
RandomAccessFile raf = new RandomAccessFile(files[i].getAbsoluteFile(), "r"); //open log file
long lastSegment = raf.length(); //Finds how long is the files
lastSegment = raf.length()-5; //Sets a point to start looking
String leido = "";
byte array[] = new byte[1024];
/*
* Going back until we find line or file is empty.
*/
while(!leido.contains(lineToSearch)||lastSegment>0) {
System.out.println("leido: " + leido);
raf.seek(lastSegment); //move the to that point
raf.read(array); //Reads 1024 bytes and saves in array
leido = new String(array); //Saves what is read as a string
lastSegment = lastSegment-15; //move the point a little further back
}
if(lastSegment<0) {
raf.seek(leido.indexOf(lineToSearch) - 23); //to make sure we get the date (23 characters long) NOTE: it wont be negative.
raf.read(array); //Reads 1024 bytes and saves in array
leido = new String(array); //make the array into a string
Date date = new SimpleDateFormat("MMMM d, yyyy", Locale.ENGLISH).parse(leido.substring(0, leido.indexOf(" INFO "))); //get only the date part
System.out.println(date);
//if date is bigger than the other save file name
}
}
}

I find the code difficult to verify. One could split the task in a backwards reader, which reads lines from file end to start. And use that for parsing dates line wise.
Mind, I am not going for nice code, but something like this:
public class BackwardsReader implements Closeable {
private static final int BUFFER_SIZE = 4096;
private String charset;
private RandomAccessFile raf;
private long position;
private int readIndex;
private byte[] buffer = new byte[BUFFER_SIZE];
/**
* #param file a text file.
* #param charset with bytes '\r' and '\n' (no wide chars).
*/
public BackwardsReader(File file, String charset) throws IOException {
this.charset = charset;
raf = new RandomAccessFile(file, "r");
position = raf.length();
}
public String readLine() throws IOException {
if (position + readIndex == 0) {
raf.close();
raf = null;
return null;
}
String line = "";
for (;;) { // Loop adding blocks without newline '\n'.
// Search line start:
boolean lineStartFound = false;
int lineStartIndex = readIndex;
while (lineStartIndex > 0) {
if (buffer[lineStartIndex - 1] == (byte)'\n') {
lineStartFound = true;
break;
}
--lineStartIndex;
}
String line2;
try {
line2 = new String(buffer, lineStartIndex, readIndex - lineStartIndex,
charset).replaceFirst("\r?\n?", "");
readIndex = lineStartIndex;
} catch (UnsupportedEncodingException ex) {
Logger.getLogger(BackwardsReader.class.getName())
.log(Level.SEVERE, null, ex);
return null;
}
line = line2 + line;
if (lineStartFound) {
--readIndex;
break;
}
// Read a prior block:
int toRead = BUFFER_SIZE;
if (position - toRead < 0) {
toRead = (int) position;
}
if (toRead == 0) {
break;
}
position -= toRead;
raf.seek(position);
raf.readFully(buffer, 0, toRead);
readIndex = toRead;
if (buffer[readIndex - 1] == (byte)'\r') {
--readIndex;
}
}
return line;
}
#Override
public void close() throws IOException {
if (raf != null) {
raf.close();
}
}
}
And a usage example:
public static void main(String[] args) {
try {
File file = new File(args[0]);
BackwardsReader reader = new BackwardsReader(file, "UTF-8");
int lineCount = 0;
for (;;) {
String line = reader.readLine();
if (line == null) {
break;
}
++lineCount;
System.out.println(line);
}
reader.close();
System.out.println("Lines: " + lineCount);
} catch (IOException ex) {
Logger.getLogger(App.class.getName()).log(Level.SEVERE, null, ex);
}
}

a problem about urlfetch over 1M in app engine

to fetch() over 1M in app engine,i use the range header and then combine those pieces.and my codes:
int startpos=0;
int endpos;
int seg=1;
int len=1;
while(len>0){
endpos=startpos+seg;
httpConn = (HttpURLConnection) u.openConnection();
httpConn.setRequestMethod("GET");
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14");
con.setRequestProperty("Range", "bytes=" + startpos + "-" + endpos);
con.connect();
InputStream in=con.getInputStream();
len=con.getContentLength();
byte[] b=new byte[len];
in.read(b, 0, len);
startpos+=len;
}
but when it goes to the "InputStream in=con.getInputStream();",its debug is " URL Fetch Response too large problems"
so i don't know what the wrong with these codes.
and there are other ways to fetch() over 1M?

Not all HTTP servers support range requests, especially when it comes to frameworks serving dynamic content - they'll simply ignore the Range header and send you the whole response.
The recent release of 1.4.0 increased the URLFetch response limit to 32MB, though, so you no longer need to do this.

I had the same problem and hacked up a little class to simulate an input stream on Appengine using the HTTP range parameter. It allows you to read files bigger then the limit in a line-oriented fashion. I am attaching it below, although you may need to adapt it for your purposes:
package com.theodorebook.AEStreamer;
import java.io.InputStream;
import java.net.URL;
import java.net.URLConnection;
import java.util.Arrays;
import java.util.logging.Logger;
/**
* A class to simulate a stream in appengine, which insists on downloading
* an entire URL before letting you do anything with it. This enables one
* to read files larger than the size limits.
*
* #author Theodore Book (theodorebook at gmail dot com)
*
*/
public class AEStreamer {
private static final int BITE_SIZE = 0x10000; //How big a chunk to grab at a time
private static final byte TERMINATOR = '\n'; //String terminator
private int mCurrentPosition = 0; //The current position in the file
private int mOffset = -1; //The offset of the current block
private long mValidBytes = 0; //The number of valid bytes in the chunk
private byte[] mChunk = new byte[BITE_SIZE];
private boolean mComplete = false;
private String mURL;
private static final Logger log = Logger.getLogger(AEStreamer.class.getName());
public AEStreamer(String url) {
mURL = url;
}
/**
* Returns the next line from the source, or null on empty
* #return
*/
public String readLine() {
String line = "";
//See if we have something to read
if (mCurrentPosition >= mOffset + mValidBytes) {
if (mComplete)
return null;
readChunk();
}
if (mValidBytes == 0)
return null;
//Read until we reach a terminator
int endPtr = mCurrentPosition - mOffset;
while (mChunk[endPtr] != TERMINATOR) {
endPtr++;
//If we reach the end of the block
if (endPtr == mValidBytes) {
line += new String(Arrays.copyOfRange(mChunk, mCurrentPosition - mOffset, endPtr));
mCurrentPosition += (endPtr - mCurrentPosition + mOffset);
if (mComplete) {
return line;
} else {
readChunk();
endPtr = mCurrentPosition - mOffset;
}
}
}
line += new String(Arrays.copyOfRange(mChunk, mCurrentPosition - mOffset, endPtr));
mCurrentPosition += (endPtr - mCurrentPosition + mOffset);
mCurrentPosition++;
return line;
}
/**
* Reads the next chunk from the server
*/
private void readChunk() {
if (mOffset < 0)
mOffset = 0;
else
mOffset += BITE_SIZE;
try {
URL url = new URL(mURL);
URLConnection request = url.openConnection();
request.setRequestProperty("Range", "bytes=" + (mOffset + 1) + "-" + (mOffset + BITE_SIZE));
InputStream inStream = request.getInputStream();
mValidBytes = inStream.read(mChunk);
inStream.close();
} catch (Exception e) {
log.severe("Unable to read " + mURL + ": " + e.getLocalizedMessage());
mComplete = true;
mValidBytes = 0;
return;
}
if (mValidBytes < BITE_SIZE)
mComplete = true;
//log.info("Read " + mValidBytes + " bytes");
}
}

Windows shortcut (.lnk) parser in Java?

I'm currently using Win32ShellFolderManager2 and ShellFolder.getLinkLocation to resolve windows shortcuts in Java. Unfortunately, if the Java program is running as a service under Vista, getLinkLocation, this does not work. Specifically, I get an exception stating "Could not get shell folder ID list".
Searching the web does turn up mentions of this error message, but always in connection with JFileChooser. I'm not using JFileChooser, I just need to resolve a .lnk file to its destination.
Does anyone know of a 3rd-party parser for .lnk files written in Java I could use?
I've since found unofficial documentation for the .lnk format here, but I'd rather not have to do the work if anyone has done it before, since the format is rather scary.

Added comments (some explanation as well as credit to each contributor so far),additional check on the file magic, a quick test to see if a given file might be a valid link (without reading all of the bytes), a fix to throw a ParseException with appropriate message instead of ArrayIndexOutOfBoundsException if the file is too small, did some general clean-up.
Source here (if you have any changes, push them right to the GitHub repo/project.
package org.stackoverflowusers.file;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.text.ParseException;
/**
* Represents a Windows shortcut (typically visible to Java only as a '.lnk' file).
*
* Retrieved 2011-09-23 from http://stackoverflow.com/questions/309495/windows-shortcut-lnk-parser-in-java/672775#672775
* Originally called LnkParser
*
* Written by: (the stack overflow users, obviously!)
* Apache Commons VFS dependency removed by crysxd (why were we using that!?) https://github.com/crysxd
* Headerified, refactored and commented by Code Bling http://stackoverflow.com/users/675721/code-bling
* Network file support added by Stefan Cordes http://stackoverflow.com/users/81330/stefan-cordes
* Adapted by Sam Brightman http://stackoverflow.com/users/2492/sam-brightman
* Based on information in 'The Windows Shortcut File Format' by Jesse Hager <jessehager#iname.com>
* And somewhat based on code from the book 'Swing Hacks: Tips and Tools for Killer GUIs'
* by Joshua Marinacci and Chris Adamson
* ISBN: 0-596-00907-0
* http://www.oreilly.com/catalog/swinghks/
*/
public class WindowsShortcut
{
private boolean isDirectory;
private boolean isLocal;
private String real_file;
/**
* Provides a quick test to see if this could be a valid link !
* If you try to instantiate a new WindowShortcut and the link is not valid,
* Exceptions may be thrown and Exceptions are extremely slow to generate,
* therefore any code needing to loop through several files should first check this.
*
* #param file the potential link
* #return true if may be a link, false otherwise
* #throws IOException if an IOException is thrown while reading from the file
*/
public static boolean isPotentialValidLink(File file) throws IOException {
final int minimum_length = 0x64;
InputStream fis = new FileInputStream(file);
boolean isPotentiallyValid = false;
try {
isPotentiallyValid = file.isFile()
&& file.getName().toLowerCase().endsWith(".lnk")
&& fis.available() >= minimum_length
&& isMagicPresent(getBytes(fis, 32));
} finally {
fis.close();
}
return isPotentiallyValid;
}
public WindowsShortcut(File file) throws IOException, ParseException {
InputStream in = new FileInputStream(file);
try {
parseLink(getBytes(in));
} finally {
in.close();
}
}
/**
* #return the name of the filesystem object pointed to by this shortcut
*/
public String getRealFilename() {
return real_file;
}
/**
* Tests if the shortcut points to a local resource.
* #return true if the 'local' bit is set in this shortcut, false otherwise
*/
public boolean isLocal() {
return isLocal;
}
/**
* Tests if the shortcut points to a directory.
* #return true if the 'directory' bit is set in this shortcut, false otherwise
*/
public boolean isDirectory() {
return isDirectory;
}
/**
* Gets all the bytes from an InputStream
* #param in the InputStream from which to read bytes
* #return array of all the bytes contained in 'in'
* #throws IOException if an IOException is encountered while reading the data from the InputStream
*/
private static byte[] getBytes(InputStream in) throws IOException {
return getBytes(in, null);
}
/**
* Gets up to max bytes from an InputStream
* #param in the InputStream from which to read bytes
* #param max maximum number of bytes to read
* #return array of all the bytes contained in 'in'
* #throws IOException if an IOException is encountered while reading the data from the InputStream
*/
private static byte[] getBytes(InputStream in, Integer max) throws IOException {
// read the entire file into a byte buffer
ByteArrayOutputStream bout = new ByteArrayOutputStream();
byte[] buff = new byte[256];
while (max == null || max > 0) {
int n = in.read(buff);
if (n == -1) {
break;
}
bout.write(buff, 0, n);
if (max != null)
max -= n;
}
in.close();
return bout.toByteArray();
}
private static boolean isMagicPresent(byte[] link) {
final int magic = 0x0000004C;
final int magic_offset = 0x00;
return link.length >= 32 && bytesToDword(link, magic_offset) == magic;
}
/**
* Gobbles up link data by parsing it and storing info in member fields
* #param link all the bytes from the .lnk file
*/
private void parseLink(byte[] link) throws ParseException {
try {
if (!isMagicPresent(link))
throw new ParseException("Invalid shortcut; magic is missing", 0);
// get the flags byte
byte flags = link[0x14];
// get the file attributes byte
final int file_atts_offset = 0x18;
byte file_atts = link[file_atts_offset];
byte is_dir_mask = (byte)0x10;
if ((file_atts & is_dir_mask) > 0) {
isDirectory = true;
} else {
isDirectory = false;
}
// if the shell settings are present, skip them
final int shell_offset = 0x4c;
final byte has_shell_mask = (byte)0x01;
int shell_len = 0;
if ((flags & has_shell_mask) > 0) {
// the plus 2 accounts for the length marker itself
shell_len = bytesToWord(link, shell_offset) + 2;
}
// get to the file settings
int file_start = 0x4c + shell_len;
final int file_location_info_flag_offset_offset = 0x08;
int file_location_info_flag = link[file_start + file_location_info_flag_offset_offset];
isLocal = (file_location_info_flag & 2) == 0;
// get the local volume and local system values
//final int localVolumeTable_offset_offset = 0x0C;
final int basename_offset_offset = 0x10;
final int networkVolumeTable_offset_offset = 0x14;
final int finalname_offset_offset = 0x18;
int finalname_offset = link[file_start + finalname_offset_offset] + file_start;
String finalname = getNullDelimitedString(link, finalname_offset);
if (isLocal) {
int basename_offset = link[file_start + basename_offset_offset] + file_start;
String basename = getNullDelimitedString(link, basename_offset);
real_file = basename + finalname;
} else {
int networkVolumeTable_offset = link[file_start + networkVolumeTable_offset_offset] + file_start;
int shareName_offset_offset = 0x08;
int shareName_offset = link[networkVolumeTable_offset + shareName_offset_offset]
+ networkVolumeTable_offset;
String shareName = getNullDelimitedString(link, shareName_offset);
real_file = shareName + "\\" + finalname;
}
} catch (ArrayIndexOutOfBoundsException e) {
throw new ParseException("Could not be parsed, probably not a valid WindowsShortcut", 0);
}
}
private static String getNullDelimitedString(byte[] bytes, int off) {
int len = 0;
// count bytes until the null character (0)
while (true) {
if (bytes[off + len] == 0) {
break;
}
len++;
}
return new String(bytes, off, len);
}
/*
* convert two bytes into a short note, this is little endian because it's
* for an Intel only OS.
*/
private static int bytesToWord(byte[] bytes, int off) {
return ((bytes[off + 1] & 0xff) << 8) | (bytes[off] & 0xff);
}
private static int bytesToDword(byte[] bytes, int off) {
return (bytesToWord(bytes, off + 2) << 16) | bytesToWord(bytes, off);
}
}

Sam Brightman's solution is for local files only.
I added support for Network files:
Windows shortcut (.lnk) parser in Java?
http://code.google.com/p/8bits/downloads/detail?name=The_Windows_Shortcut_File_Format.pdf
http://www.javafaq.nu/java-example-code-468.html
public class LnkParser {
public LnkParser(File f) throws IOException {
parse(f);
}
private boolean isDirectory;
private boolean isLocal;
public boolean isDirectory() {
return isDirectory;
}
private String real_file;
public String getRealFilename() {
return real_file;
}
private void parse(File f) throws IOException {
// read the entire file into a byte buffer
FileInputStream fin = new FileInputStream(f);
ByteArrayOutputStream bout = new ByteArrayOutputStream();
byte[] buff = new byte[256];
while (true) {
int n = fin.read(buff);
if (n == -1) {
break;
}
bout.write(buff, 0, n);
}
fin.close();
byte[] link = bout.toByteArray();
parseLink(link);
}
private void parseLink(byte[] link) {
// get the flags byte
byte flags = link[0x14];
// get the file attributes byte
final int file_atts_offset = 0x18;
byte file_atts = link[file_atts_offset];
byte is_dir_mask = (byte)0x10;
if ((file_atts & is_dir_mask) > 0) {
isDirectory = true;
} else {
isDirectory = false;
}
// if the shell settings are present, skip them
final int shell_offset = 0x4c;
final byte has_shell_mask = (byte)0x01;
int shell_len = 0;
if ((flags & has_shell_mask) > 0) {
// the plus 2 accounts for the length marker itself
shell_len = bytes2short(link, shell_offset) + 2;
}
// get to the file settings
int file_start = 0x4c + shell_len;
final int file_location_info_flag_offset_offset = 0x08;
int file_location_info_flag = link[file_start + file_location_info_flag_offset_offset];
isLocal = (file_location_info_flag & 2) == 0;
// get the local volume and local system values
//final int localVolumeTable_offset_offset = 0x0C;
final int basename_offset_offset = 0x10;
final int networkVolumeTable_offset_offset = 0x14;
final int finalname_offset_offset = 0x18;
int finalname_offset = link[file_start + finalname_offset_offset] + file_start;
String finalname = getNullDelimitedString(link, finalname_offset);
if (isLocal) {
int basename_offset = link[file_start + basename_offset_offset] + file_start;
String basename = getNullDelimitedString(link, basename_offset);
real_file = basename + finalname;
} else {
int networkVolumeTable_offset = link[file_start + networkVolumeTable_offset_offset] + file_start;
int shareName_offset_offset = 0x08;
int shareName_offset = link[networkVolumeTable_offset + shareName_offset_offset]
+ networkVolumeTable_offset;
String shareName = getNullDelimitedString(link, shareName_offset);
real_file = shareName + "\\" + finalname;
}
}
private static String getNullDelimitedString(byte[] bytes, int off) {
int len = 0;
// count bytes until the null character (0)
while (true) {
if (bytes[off + len] == 0) {
break;
}
len++;
}
return new String(bytes, off, len);
}
/*
* convert two bytes into a short note, this is little endian because it's
* for an Intel only OS.
*/
private static int bytes2short(byte[] bytes, int off) {
return ((bytes[off + 1] & 0xff) << 8) | (bytes[off] & 0xff);
}
/**
* Returns the value of the instance variable 'isLocal'.
*
* #return Returns the isLocal.
*/
public boolean isLocal() {
return isLocal;
}
}

I can recommend this repository on GitHub:
https://github.com/BlackOverlord666/mslinks
There I've found a simple solution to create shortcuts:
ShellLink.createLink("path/to/existing/file.txt", "path/to/the/future/shortcut.lnk");
If you want to read shortcuts:
File shortcut = ...;
String pathToExistingFile = new ShellLink(shortcut).resolveTarget();
If you want to change the icon of the shortcut, use:
ShellLink sl = ...;
sl.setIconLocation("/path/to/icon/file");
You can edit most properties of the shortcutlink such as working directory, tooltip text, icon, command line arguments, hotkeys, create links to LAN shared files and directories and much more...
Hope this helps you :)
Kind regards
Josua Frank

The code plan9assembler linked to appears to work with minor modification. I think it's just the "& 0xff" to prevent sign extension when bytes are upcast to ints in the bytes2short function that need changing. I've added the functionality described in http://www.i2s-lab.com/Papers/The_Windows_Shortcut_File_Format.pdf to concatenate the "final part of the pathname" even though in practice this doesn't seem to be used in my examples. I've not added any error checking to the header or dealt with network shares. Here's what I'm using now:
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.text.DecimalFormat;
import java.text.NumberFormat;
public class LnkParser {
public LnkParser(File f) throws Exception {
parse(f);
}
private boolean is_dir;
public boolean isDirectory() {
return is_dir;
}
private String real_file;
public String getRealFilename() {
return real_file;
}
private void parse(File f) throws Exception {
// read the entire file into a byte buffer
FileInputStream fin = new FileInputStream(f);
ByteArrayOutputStream bout = new ByteArrayOutputStream();
byte[] buff = new byte[256];
while (true) {
int n = fin.read(buff);
if (n == -1) {
break;
}
bout.write(buff, 0, n);
}
fin.close();
byte[] link = bout.toByteArray();
// get the flags byte
byte flags = link[0x14];
// get the file attributes byte
final int file_atts_offset = 0x18;
byte file_atts = link[file_atts_offset];
byte is_dir_mask = (byte) 0x10;
if ((file_atts & is_dir_mask) > 0) {
is_dir = true;
} else {
is_dir = false;
}
// if the shell settings are present, skip them
final int shell_offset = 0x4c;
final byte has_shell_mask = (byte) 0x01;
int shell_len = 0;
if ((flags & has_shell_mask) > 0) {
// the plus 2 accounts for the length marker itself
shell_len = bytes2short(link, shell_offset) + 2;
}
// get to the file settings
int file_start = 0x4c + shell_len;
// get the local volume and local system values
final int basename_offset_offset = 0x10;
final int finalname_offset_offset = 0x18;
int basename_offset = link[file_start + basename_offset_offset]
+ file_start;
int finalname_offset = link[file_start + finalname_offset_offset]
+ file_start;
String basename = getNullDelimitedString(link, basename_offset);
String finalname = getNullDelimitedString(link, finalname_offset);
real_file = basename + finalname;
}
private static String getNullDelimitedString(byte[] bytes, int off) {
int len = 0;
// count bytes until the null character (0)
while (true) {
if (bytes[off + len] == 0) {
break;
}
len++;
}
return new String(bytes, off, len);
}
/*
* convert two bytes into a short note, this is little endian because it's
* for an Intel only OS.
*/
private static int bytes2short(byte[] bytes, int off) {
return ((bytes[off + 1] & 0xff) << 8) | (bytes[off] & 0xff);
}
}

I've also worked( now have no time for that) on '.lnk' in Java. My code is here
It's little messy( some testing trash) but local and network parsing works good. Creating links is implemented too. Please test and send me patches.
Parsing example:
Shortcut scut = Shortcut.loadShortcut(new File("C:\\t.lnk"));
System.out.println(scut.toString());
Creating new link:
Shortcut scut = new Shortcut(new File("C:\\temp"));
OutputStream os = new FileOutputStream("C:\\t.lnk");
os.write(scut.getBytes());
os.flush();
os.close();

The solution of #Code Bling does not work for me for Files in the User directory.
For Example "C:/Users/Username/Filename.txt".
The reason for that is: in The_Windows_Shortcut_File_Format.pdf
that was mentioned by #Stefan Cordes on page 6 it says that only the first 2 bits are important for volumes info.
All other bits might be filled with random garbage when the first bit of volumes info is "0".
So if it comes to:
isLocal = (file_location_info_flag & 2) == 0;
then file_location_info_flag might be "3".
This file is still local but this line of code assigns false to isLocal.
So i suggest the following adjustment to #Code Bling's code:
isLocal = (file_location_info_flag & 1) == 1;

This short code is really usefull...
But two fixes are needed:
the isPotentialValidLink improved not to load file if name doesn't end with ".lnk"
public static boolean isPotentialValidLink(final File file) {
final int minimum_length = 0x64;
boolean isPotentiallyValid = false;
if (file.getName().toLowerCase().endsWith(".lnk"))
try (final InputStream fis = new FileInputStream(file)) {
isPotentiallyValid = file.isFile() && fis.available() >= minimum_length && isMagicPresent(getBytes(fis, 32));
} catch (Exception e) {
// forget it
}
return isPotentiallyValid;
}
the offset has to be computed with 32bits not only a byte...
final int finalname_offset = bytesToDword(link,file_start + finalname_offset_offset) + file_start;
final int basename_offset = bytesToDword(link,file_start + basename_offset_offset) + file_start;

I found other non-professional technique. getting my job done.
File file=new File("C:\\ProgramData\\Microsoft\\Windows\\Start Menu\\Programs\\TeamViewer.lnk");//shortcut link
FileInputStream stream=new FileInputStream(file);
DataInputStream st=new DataInputStream(stream);
byte[] bytes=new byte[(int)stream.getChannel().size()];
stream.read(bytes);
String data=new String(bytes);
int i1=data.indexOf("C:\\Program Files");
int i2=data.indexOf(".exe",i1);
System.out.println(data.substring(i1, i2+4));

The given code works well, but has a bug. A java byte is a signed value from -128 to 127. We want an unsigned value from 0 to 255 to get the correct results. Just change the bytes2short function as follows:
static int bytes2short(byte[] bytes, int off) {
int low = (bytes[off]<0 ? bytes[off]+256 : bytes[off]);
int high = (bytes[off+1]<0 ? bytes[off+1]+256 : bytes[off+1])<<8;
return 0 | low | high;
}