Develop Reference

Java Resolve Response URLConnection Header Value

I send http head request with URLConnection and got header value Content-Disposition Unreadable value like bellow.
Content-Disposition: attachment; filename="৩টি ধাপে সহজেই আতà§à¦¬à¦¬à¦¿à¦¶à§à¦¬à¦¾à¦¸à§€ হয়ে উঠà§à¦¨ | Motivational Video in Bangla.mp4"
How to resolve this text ৩টি ধাপে সহজেই আতà§à¦¬à¦¬à¦¿à¦¶à§à¦¬à¦¾à¦¸à§€ হয়ে উঠà§à¦¨ to ৩টি ধাপে সহজেই আত্ববিশ্বাসী হয়ে উঠুন

Your issue is that the response comes in a.. non-typical, Bengali encoding. I couldn't find the exact one, but seems to be something close to "Windows-1252".
Running the below code gives me the following output, having issues with some composite characters:
public static void main(String[] args) throws UnsupportedEncodingException {
var source = "৩টি ধাপে সহজেই আতà§à¦¬à¦¬à¦¿à¦¶à§à¦¬à¦¾à¦¸à§€ হয়ে উঠà§à¦¨";
var bytes = source.getBytes("Windows-1252");
System.out.println("Expected: " + "৩টি ধাপে সহজেই আত্ববিশ্বাসী হয়ে উঠুন");
System.out.println("Actual : " + new String(bytes, StandardCharsets.UTF_8));
}
Expected: ৩টি ধাপে সহজেই আত্ববিশ্বাসী হয়ে উঠুন
Actual : ৩টি ধাপে সহজেই আত�ববিশ�বাসী হয়ে উ� �ন
The solution may be to find the right decoder for this encoding of Bengali text so you can convert it to Unicode.
Best of luck!

Too much data on Node.js socket?

I'm currently developing a system that gets data from a battery pack of an electric vehicle, stores it in a database and display it on a screen.
So I have a Java - Application that reads the data from a hardware interface, interprets the values and sends it via Socket to a Node.js-Server. (Java App and Webserver are running on the same computer, so Url = localhost)
JAVA APP:
s = new Socket();
s.connect(new InetSocketAddress(URL, PORT));
out = new PrintWriter( s.getOutputStream(), true);
for (DataEntry e : entries){
out.printf(e.toJson());
}
NODE:
sock.on('data', function(data) {
try{
var data = JSON.parse(data);
db.serialize(function(){
db.run("INSERT INTO DataEntry(value, MessageField, time) values(" + data.value + "," + data.messageFieldID + ", STRFTIME('%Y-%m-%d %H:%M:%f'))");
});
} catch(e){}
});
I get about 20 Messages per second from the hardware interface which are converted into 100 Json - Strings. So the webserver has to process one message in 10 ms, which I thought, is manageable.
But here is the problem: If my entries - List (foreach loop) has more than 2 elements, the webserver gets 2 or more of the Json's in one message.
So the first message was divided into 2 parts (ID 41,42) and was processed correctly. But the second message was divided into 5 parts (ID 43-47), and the first 4 of them weren't sent alone, so only the last one was saved correctly.
How can I ensure, that every Json is sent one another?
Isn't there something like a buffer so that the socket.on method is called correctly for every message I send?
I hope somebody of you can help me
Thank you!
Benedikt :)

TCP sockets are just streams and you shouldn't make any assumptions about how much of a "message" is contained in a single packet.
A simple solution to this is to terminate each message with a newline character since JSON cannot contain such a character. From there it's a simple matter of buffering data until you see a newline character. Then call JSON.parse() on your buffer. For example:
var buf = '';
sock.on('data', function(data) {
buf += data;
var p;
// Use a while loop as it may be possible to have multiple
// messages buffered depending on chunk contents
while (~(p = buf.indexOf('\n'))) {
try {
var msg = JSON.parse(buf.slice(0, p));
} catch (ex) {
console.log('Bad JSON message: ' + ex);
}
buf = buf.slice(p + 1);
}
});
You will also need to change printf() to println() on the Java-side so that a newline character will be appended to each message.

Gujarati text in Java String

I have Gujarati Bible and trying to insert each verse in MySQL database using parser written in Java. When I assign Gujarati text to Java String variable it shows junks in debug.
E.g. This is my Gujarati text
હે યહોવા તું મારો દેવ છે;
I assign it to Java String variable as shown below
verse._verseText = "હે યહોવા તું મારો દેવ છે;";
What i see in debug window is all junk characters. Any help is appreciated. If need more information let me know and I will provide as and when asked.
UPDATE
Pasting my parser code here
private Boolean Insert(String _text)
{
BibleVerse verse = new BibleVerse();
String[] data = _text.split("\\|");
try
{
if (data[0].equals(bookName) || bookName.equals("All"))
{
verse._Version = "Gujarati";
verse._book = data[0];
verse._chapter = Integer.parseInt(data[1]);
verse._verse = Integer.parseInt(data[2]);
verse._verseText = new String(data[3].getBytes(), "UTF-8");
_bibleDatabase.Insert(verse);
pcs.firePropertyChange("logupdate", null, data[0] + " " + data[1] + "," + data[2] + " - INSERTED.");
}
else
{
pcs.firePropertyChange("logupdate", null, data[0] + " " + data[1] + "," + data[2] + " - SKIPPED.");
}
return true;
}
catch(Exception e)
{
pcs.firePropertyChange("logupdate", null, "ERROR : " + e.getMessage());
return false;
}
}
Here is the sample line from the text file
Isaiah|25|1|હે યહોવા તું મારો દેવ છે; હું તને મોટો માનીશ, હું તારા નામની સ્તુતિ કરીશ; કેમકે તેં અદભુત કાર્યો કર્યાં છે, તેં વિશ્વાસુપણે તથા સત્યતાથી પુરાતન સંકલ્પો પાર પાડ્યા છે.
UPDATE
Here is the code where I open & read file.
try
{
FileReader _file = new FileReader(this._filename);
_bufferedReader = new BufferedReader(_file);
SwingWorker parseWorker = new SwingWorker()
{
#Override
protected Object doInBackground() throws Exception
{
String line;
String[] data;
int lineno=0;
BibleVerse verse = new BibleVerse();
while ((line = _bufferedReader.readLine()) != null)
{
++lineno;
pcs.firePropertyChange("pgbupdate", null, lineno);
Insert(line);
}
_bufferedReader.close();
return null;
}
#Override
protected void done()
{
pcs.firePropertyChange("logupdate", null, "Parsing complete.");
}
};
parseWorker.execute();
}
catch (Exception e)
{
pcs.firePropertyChange("logupdate", null, "ERROR : " + e.getMessage());
}

The problem is this:
FileReader _file = new FileReader(this._filename);
This reads the file using the platform's default charset. If your data file is not encoded in that charset, you will get incorrect characters.
On Windows, the default charset is almost always UTF-16LE. On most other systems, it's UTF-8.
The easiest solution is to find out the actual encoding of your data file, so you can specify it explicitly in the code. The encoding of a file can be determined with the file command on Unix and Linux systems. In Windows, you may need to examine it with a binary editor, or install something like Cygwin, which has a file command of its own.
Once you know what it is, you should pass it explicitly to the construction of your Reader:
// Replace "UTF-8" with the actual encoding of your data file (if it's not UTF-8).
Reader _file = new InputStreamReader(new FileInputStream(this._filename), "UTF-8");
Once you've done that, there is no reason for any other part of your code to concern itself with bytes. You should replace this:
verse._verseText = new String(data[3].getBytes(), "UTF-8");
with this:
verse._verseText = data[3];

how to inject chinese characters using javascript?
not quite the same problem, but I think the same solution may work in this case.
If the script is inline (in the HTML file), then it's using the
encoding of the HTML file and you won't have an issue.
If the script is loaded from another file:
Your text editor must save the file in an appropriate encoding such as
utf-8 (it's probably doing this already if you're able to save it,
close it, and reopen it with the characters still displaying
correctly) Your web server must serve the file with the right http
header specifying that it's utf-8 (or whatever the enocding happens to
be, as determined by your text editor settings). Here's an example for
how to do this with php: Set http header to utf-8 php If you can't
have your webserver do this, try to set the charset attribute on your
script tag (e.g. > I tried to see what the spec said should happen
in the case of mismatching charsets defined by the tag and the http
headers, but couldn't find anything concrete, so just test and see if
it helps. If that doesn't work, place your script inline

It looks like if you want to store Gujarati text in Java string, you need to use unicode characters. See this: http://jrgraphix.net/r/Unicode/0A80-0AFF
So for example the first Gujarati character:
char example = '0A80';
String result = Character.toString((char)example);

JSONObject contains escape characters

I'm building a simulator to post JSON data to a service I'm running.
The JSON should look like this:
{"sensor":
{"id":"SENSOR1","name":"SENSOR","type":"Temperature","value":100.12,"lastDateValue":"\/Date(1382459367723)\/"}
}
I tried this with the "Advanced REST Client" in Chrome and this works fine. The date get's parsed properly by the ServiceStack webservice.
So, the point is to write a sensor simulator that posts data like this to the web service.
I created this in Java, so I could run it on my raspberry pi.
This is the code:
public static void main(String[] args) {
String url = "http://localhost:63003/api/sensors";
String sensorname = "Simulated sensor";
int currentTemp = 10;
String dateString = "\\" + "/Date(" + System.currentTimeMillis() + ")\\" + "/";
System.out.println(dateString);
System.out.println("I'm going to post some data to: " + url);
//Creating the JSON Object
JSONObject data = new JSONObject();
data.put("id", sensorname);
data.put("name", sensorname);
data.put("type", "Temperature");
data.put("value", currentTemp);
data.put("lastDateValue", dateString);
JSONObject sensor = new JSONObject().put("sensor", data);
//Print out the data to be sent
StringWriter out = new StringWriter();
sensor.write(out);
String jsonText = out.toString();
System.out.print(jsonText);
//Sending the object
HttpClient c = new DefaultHttpClient();
HttpPost p = new HttpPost(url);
p.setEntity(new StringEntity(sensor.toString(), ContentType.create("application/json")));
try {
HttpResponse r = c.execute(p);
} catch (Exception e) {
e.printStackTrace();
}
}
The output of this program is as follows:
\/Date(1382459367723)\/
I'm going to post some data to: http://localhost:63003/api/sensors
{"sensor":{"lastDateValue":"\\/Date(1382459367723)\\/","id":"Simulated sensor","name":"Simulated sensor","value":10,"type":"Temperature"}}
The issue here is that the JSONObject string still contains these escape characters. But when I print the string in the beginning it does not contain the escape characters. Is there any way to get rid of these? My service can't parse these..
This is a sample of what I send with the rest client in chrome:
{"sensor":{"id":"I too, am a sensor!","name":"Willy","type":"Temperature","value":100.12,"lastDateValue":"\/Date(1382459367723)\/"}}

JSONObject is correctly encoding the string. This page describes how string literals are to be escaped in JavaScript (and, by extension, JSON). The following note is important to understanding what happens in your example:
For characters not listed in Table 2.1, a preceding backslash is ignored, but this usage is deprecated and should be avoided.
Your example ("\/Date(1382459367723)\/") uses a preceding backslash before a /. Because / is not in table 2.1, the \ should simply be ignored. If your service doesn't ignore the \, then it either has a bug, or is not a JSON parser (perhaps it uses a data format which is similar to, but not quite, JSON).
Since you need to generate non-conforming JSON, you won't be able to use standard tools to do so. Your two options are to write your own not-quite-JSON encoder, or to avoid characters which must be escaped, such as \ and ".

#pburka is correct. If you want to send it in \/Date(1382459367723)\/ format, try escaping the blackslash twice as below
String dateString = "\\\\" + "/Date(" + System.currentTimeMillis() + ")\\\\" + "/";
In the first pass, dateString will make it as \\/Date(1382459367723)\\/ and finally JSONObject will add extra backslashes internally to it's buffer i.e \\\/Date(1382459367723)\\\/ so that the blackslashes before / will be ignored according to JSON parsing rules and you would get the desired result i.e \/Date(1382459367723)\/

How to handle multipart/alternative mail with JavaMail?

I wrote an application which gets all emails from an inbox, filters the emails which contain a specific string and then puts those emails in an ArrayList.
After the emails are put in the List, I am doing some stuff with the subject and content of said emails. This works all fine for e-mails without an attachment. But when I started to use e-mails with attachments it all didn't work as expected anymore.
This is my code:
public void getInhoud(Message msg) throws IOException {
try {
cont = msg.getContent();
} catch (MessagingException ex) {
Logger.getLogger(ReadMailNew.class.getName()).log(Level.SEVERE, null, ex);
}
if (cont instanceof String) {
String body = (String) cont;
} else if (cont instanceof Multipart) {
try {
Multipart mp = (Multipart) msg.getContent();
int mp_count = mp.getCount();
for (int b = 0; b < 1; b++) {
dumpPart(mp.getBodyPart(b));
}
} catch (Exception ex) {
System.out.println("Exception arise at get Content");
ex.printStackTrace();
}
}
}
public void dumpPart(Part p) throws Exception {
email = null;
String contentType = p.getContentType();
System.out.println("dumpPart" + contentType);
InputStream is = p.getInputStream();
if (!(is instanceof BufferedInputStream)) {
is = new BufferedInputStream(is);
}
int c;
final StringWriter sw = new StringWriter();
while ((c = is.read()) != -1) {
sw.write(c);
}
if (!sw.toString().contains("<div>")) {
mpMessage = sw.toString();
getReferentie(mpMessage);
}
}
The content from the e-mail is stored in a String.
This code works all fine when I try to read mails without attachment. But if I use an e-mail with attachment the String also contains HTML code and even the attachment coding. Eventually I want to store the attachment and the content of an e-mail, but my first priority is to get just the text without any HTML or attachment coding.
Now I tried an different approach to handle the different parts:
public void getInhoud(Message msg) throws IOException {
try {
Object contt = msg.getContent();
if (contt instanceof Multipart) {
System.out.println("Met attachment");
handleMultipart((Multipart) contt);
} else {
handlePart(msg);
System.out.println("Zonder attachment");
}
} catch (MessagingException ex) {
ex.printStackTrace();
}
}
public static void handleMultipart(Multipart multipart)
throws MessagingException, IOException {
for (int i = 0, n = multipart.getCount(); i < n; i++) {
handlePart(multipart.getBodyPart(i));
System.out.println("Count "+n);
}
}
public static void handlePart(Part part)
throws MessagingException, IOException {
String disposition = part.getDisposition();
String contentType = part.getContentType();
if (disposition == null) { // When just body
System.out.println("Null: " + contentType);
// Check if plain
if ((contentType.length() >= 10)
&& (contentType.toLowerCase().substring(
0, 10).equals("text/plain"))) {
part.writeTo(System.out);
} else if ((contentType.length() >= 9)
&& (contentType.toLowerCase().substring(
0, 9).equals("text/html"))) {
part.writeTo(System.out);
} else if ((contentType.length() >= 9)
&& (contentType.toLowerCase().substring(
0, 9).equals("text/html"))) {
System.out.println("Ook html gevonden");
part.writeTo(System.out);
}else{
System.out.println("Other body: " + contentType);
part.writeTo(System.out);
}
} else if (disposition.equalsIgnoreCase(Part.ATTACHMENT)) {
System.out.println("Attachment: " + part.getFileName()
+ " : " + contentType);
} else if (disposition.equalsIgnoreCase(Part.INLINE)) {
System.out.println("Inline: "
+ part.getFileName()
+ " : " + contentType);
} else {
System.out.println("Other: " + disposition);
}
}
This is what is returned from the System.out.printlns
Null: multipart/alternative; boundary=047d7b6220720b499504ce3786d7
Other body: multipart/alternative; boundary=047d7b6220720b499504ce3786d7
Content-Type: multipart/alternative; boundary="047d7b6220720b499504ce3786d7"
--047d7b6220720b499504ce3786d7
Content-Type: text/plain; charset="ISO-8859-1"
'Text of the message here in normal text'
--047d7b6220720b499504ce3786d7
Content-Type: text/html; charset="ISO-8859-1"
Content-Transfer-Encoding: quoted-printable
'HTML code of the message'
This approach returns the normal text of the e-mail but also the HTML coding of the mail. I really don't understand why this happens, I've googled it but it seems like there is no one else with this problem.
Any help is appreciated,
Thanks!

I found reading e-mail with the JavaMail library much more difficult than expected. I don't blame the JavaMail API, rather I blame my poor understanding of RFC-5322 -- the official definition of Internet e-mail.
As a thought experiment: Consider how complicated an e-mail message can become in the real world. It is possible to "infinitely" embed messages within messages. Each message itself may have multiple attachments (binary or human-readable text). Now imagine how complicated this structure becomes in the JavaMail API after parsing.
A few tips that may help when traversing e-mail with JavaMail:
Message and BodyPart both implement Part.
MimeMessage and MimeBodyPart both implement MimePart.
Where possible, treat everything as a Part or MimePart. This will allow generic traversal methods to be built more easily.
These Part methods will help to traverse:
String getContentType(): Starts with the MIME type. You may be tempted to treat this as a MIME type (with some hacking/cutting/matching), but don't. Better to only use this method inside the debugger for inspection.
Oddly, MIME type cannot be extracted directly. Instead use boolean isMimeType(String) to match. Read docs carefully to learn about powerful wildcards, such as "multipart/*".
Object getContent(): Might be instanceof:
Multipart -- container for more Parts
Cast to Multipart, then iterate as zero-based index with int getCount() and BodyPart getBodyPart(int)
Note: BodyPart implements Part
In my experience, Microsoft Exchange servers regularly provide two copies of the body text: plain text and HTML.
To match plain text, try: Part.isMimeType("text/plain")
To match HTML, try: Part.isMimeType("text/html")
Message (implements Part) -- embedded or attached e-mail
String (just the body text -- plain text or HTML)
See note above about Microsoft Exchange servers.
InputStream (probably a BASE64-encoded attachment)
String getDisposition(): Value may be null
if Part.ATTACHMENT.equalsIgnoreCase(getDisposition()), then call getInputStream() to get raw bytes of the attachment.
Finally, I found the official Javadocs exclude everything in the com.sun.mail package (and possibly more). If you need these, read the code directly, or generate the unfiltered Javadocs by downloading the source and running mvn javadoc:javadoc in the mail project module of the project.

Did you find these JavaMail FAQ entries?
How do I read a message with an attachment and save the attachment?
How do I tell if a message has attachments?
How do I find the main message body in a message that has attachments?

Following up on Kevin's helpful advice, analyzing your email content Java object types with respect to their canonical names (or simple names) can be helpful too. For example, looking at one inbox I've got right now, of 486 messages 399 are Strings, and 87 are MimeMultipart. This suggests that - for my typical email - a strategy that uses instanceof to first peel off Strings is best.
Of the Strings, 394 are text/plain, and 5 are text/html. This will not be the case for most; it's reflective of my email feeds into this particular inbox.
But wait - there's more!!! :-) The HTML sneaks in there nevertheless: of the 87 Multipart's, 70 are multipart/alternative. No guarantees, but most (if not all of these) are TEXT + HTML.
Of the other 17 multipart, incidentally, 15 are multipart/mixed, and 2 are multipart/signed.
My use case with this inbox (and one other) is primarily to aggregate and analyze known mailing list content. I can't ignore any of the messages, but an analysis of this sort helps me make my processing more efficient.