Develop Reference

How do I write a character at codepoint 80 to a file in Windows-1252?

I am trying to write bytes to a file in the windows-1252 charset. The example below, writing the raw bytes of a float to a file, is similar to what I'm doing in my actual program.
In the example given, I am writing the raw hex of 1.0f to test.txt. As the raw hex of 1.0f is 3f 80 00 00 I expect to get ?€(NUL)(NUL), as from what I can see in the Windows 1252 Wikipedia article, 0x3f should correspond to '?', 0x80 should correspond to '€', and 0x00 is 'NUL'. Everything goes fine until I actually try to write to the file; at that point, I get a java.nio.charset.UnmappableCharacterException on the console, and after the program stops on that exception the file only has a single '?' in it. The full console output is below the code down below.
It looks like Java considers the codepoint 0x80 unmappable in the windows-1252 codepage. However, this doesn't seem right – all the codepoints should map to actual characters in that codepage. The problem is definitely with the codepoint 0x80, as if I try with 0.5f (3f 00 00 00) it is happy to write ?(NUL)(NUL)(NUL) into the file, and does not throw the exception. Experimenting with other codepages doesn't seem to work either; looking at key encodings supported by the Java language here, only the UTF series will not give me an exception, but due to their encoding they don't give me codepoint 0x80 in the actual file.
I'm going to try just using bytes instead so I don't have to worry about string encoding, but is anyone able to tell me why my code below gives me the exception it does?
Code:
import java.io.IOException;
import java.io.Writer;
import java.nio.charset.Charset;
import java.nio.file.Files;
import java.nio.file.Paths;
public class CharsetTest {
public static void main(String[] args) {
float max = 1.0f;
System.out.println("Checking " + max);
String stringFloatFormatHex = String.format("%08x", Float.floatToRawIntBits(max));
System.out.println(stringFloatFormatHex);
byte[] bytesForFile = javax.xml.bind.DatatypeConverter.parseHexBinary(stringFloatFormatHex);
String stringForFile = new String(bytesForFile);
System.out.println(stringForFile);
String charset = "windows-1252";
try {
Writer output = Files.newBufferedWriter(Paths.get("test.txt"), Charset.forName(charset));
output.write(stringForFile);
output.close();
} catch (IOException e) {
System.err.println(e.getMessage());
e.printStackTrace();
}
}
}
Console output:
Checking 1.0
3f800000
?�
Input length = 1
java.nio.charset.UnmappableCharacterException: Input length = 1
at java.nio.charset.CoderResult.throwException(CoderResult.java:282)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:285)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:129)
at java.io.BufferedWriter.close(BufferedWriter.java:265)
at CharsetTest.main(CharsetTest.java:21)

Edit: The problem is on the instruction String stringForFile = new String(bytesForFile);, below the DatatypeConverter. As I was constructing a string without providing a charset, it uses my default charset, which is UTF-8, which doesn't have a symbol for codepoint 80. However, it only throws an exception when it writes to a file. This doesn't happen in the code below because my refactor (keeping in mind Johannes Kuhn's suggestion in the comments) doesn't use the String(byte[]) constructor without specifying a charset.
Johannes Kuhn's suggestion about the String(byte[]) constructor gave me some good clues. I've ended up with the following code, which looks like it works fine: even printing the € symbol to the console as well as writing it to test.txt. That suggests that codepoint 80 can be translated using the windows-1252 codepage.
If I were to guess at this point why this code works but the other didn't, I'd still be confused, but I would guess it was something around the conversion in javax.xml.bind.DatatypeConverter.parseHexBinary(stringFloatFormatHex);. That looks to be the main difference, although I'm not sure why it would matter.
Anyway, the code below works (and I don't even have to turn it into a string; I can write the bytes to a file with FileOutputStream fos = new FileOutputStream("test.txt"); fos.write(bytes); fos.close();), so I'm happy with this one.
Code:
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.Writer;
import java.nio.ByteBuffer;
import java.nio.charset.Charset;
import java.nio.file.Files;
import java.nio.file.Paths;
public class BytesCharsetTest {
public static void main(String[] args) {
float max = 1.0f;
System.out.println("Checking " + max);
int convInt = Float.floatToRawIntBits(max);
byte[] bytes = ByteBuffer.allocate(4).putInt(convInt).array();
String charset = "windows-1252";
try {
String stringForFile = new String(bytes, Charset.forName(charset));
System.out.println(stringForFile);
Writer output = Files.newBufferedWriter(Paths.get("test.txt"), Charset.forName(charset));
output.write(stringForFile);
output.close();
} catch (IOException e) {
System.err.println(e.getMessage());
e.printStackTrace();
}
}
}
Console output:
Checking 1.0
?€
Process finished with exit code 0

Using Google Translate Java Library, Languages with special chars return question marks

I have setup a Java program that I made for my apprenticeship project that takes in a JSON file of English strings and outputs a different language JSON file that is defined in the console. Some languages like french and Italian will output with the correct translations whereas Russian or Japanese will output with question marks as seen in the images bellow.
I had searched around at saw that I needed to get the bytes of my string and then encode that to UTF-8 I did do this but was still getting question marks so I started to use he standard charsets built into Java and tried different ways of encoding/decoding the string I tried this:
and this gave me a different output of this : Ð?Ñ?ивеÑ?
package com.bis.propertyfiletranslator;
import java.io.IOException;
import java.nio.charset.Charset;
import java.nio.charset.StandardCharsets;
import java.security.GeneralSecurityException;
import java.util.List;
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.googleapis.json.GoogleJsonResponseException;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.translate.Translate;
import com.google.api.services.translate.model.TranslationsListResponse;
import com.google.api.services.translate.model.TranslationsResource;
public class Translator {
public static Translate.Translations.List list;
private static final Charset UTF_8 = Charset.forName("UTF-8");
private static final Charset ISO = Charset.forName("ISO-8859-1");
public static void translateJSONMapThroughGoogle(String input, String output, String API, String language,
List<String> subLists) throws IOException, GeneralSecurityException {
Translate t = new Translate.Builder(GoogleNetHttpTransport.newTrustedTransport(),
JacksonFactory.getDefaultInstance(), null).setApplicationName("PhoenUX-Google-Translate").build();
try {
list = t.new Translations().list(subLists, language).setFormat("text");
list.setKey(API);
} catch (GoogleJsonResponseException e) {
if (e.getDetails().getMessage().equals("Invalid Value")) {
System.err.println(
"\n Language not currently supported, check the accepted language codes and try again.\n\n Language Requested: "
+ language);
} else {
System.out.println(e.getDetails().getMessage());
}
}
for (TranslationsResource translationsResource : response.getTranslations()) {
for (String key : JSONFunctions.jsonHashMap.keySet()) {
JSONFunctions.jsonHashMap.remove(key);
String value = translationsResource.getTranslatedText();
String encoded = new String(value.getBytes(StandardCharsets.UTF_8), StandardCharsets.ISO_8859_1);
JSONFunctions.jsonHashMap.put(key, encoded);
System.out.println(encoded);
break;
}
}
JSONFunctions.outputTranslationsBackToJson(output);
}
}
So this is using the google cloud library, I added a sysout so I could see the results of what I had tried, so this code should be all you need to replicate it.
I expect the output of "Hello" to be "Привет"(russian) actual output is ???? or Ð?Ñ?ивеÑ? dependent on the encoding I use.

String encoded = new String(...) is dead wrong. Just
put(key, value):
Note that System.out.println will always have problems as the OS encoding might be some Windows ANSI encoding. Then it is likely non Unicode-capable - and String contains Unicode.

Cannot read binary file created in Java using Python

I have created a binary file using Java and memory mapping. It contains a list of integers from 1 to 10 million:
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
public class MemoryMapWriter {
public static void main(String[] args) throws FileNotFoundException, IOException, InterruptedException {
File f = new File("file.bin");
f.delete();
FileChannel fc = new RandomAccessFile(f, "rw").getChannel();
long bufferSize=64*1000;
MappedByteBuffer mem =fc.map(FileChannel.MapMode.READ_WRITE, 0, bufferSize);
int start = 0;
long counter=1;
long HUNDREDK=100000;
long startT = System.currentTimeMillis();
long noOfMessage = HUNDREDK * 10 * 10;
for(;;)
{
if(!mem.hasRemaining())
{
start+=mem.position();
mem =fc.map(FileChannel.MapMode.READ_WRITE, start, bufferSize);
}
mem.putLong(counter);
counter++;
if(counter > noOfMessage )
break;
}
long endT = System.currentTimeMillis();
long tot = endT - startT;
System.out.println(String.format("No Of Message %s , Time(ms) %s ",noOfMessage, tot)) ;
}
then I have tried to read it using Python and memory mapping:
import pandas as pd
import numpy as np
import os
import shutil
import re
import mmap
a=np.memmap("file.bin",mode='r',dtype='int64')
print(a[0:9])
but printing first ten element, this is the result:
[ 72057594037927936, 144115188075855872, 216172782113783808,
288230376151711744, 360287970189639680, 432345564227567616,
504403158265495552, 576460752303423488, 648518346341351424,
720575940379279360]
What is wrong with my code?

You have a byte-order problem. 72057594037927936 in binary is 0x0100000000000000, 144115188075855872 is 0x0200000000000000, etc.
Java is writing longs to the buffer in big-endian order (most significant byte first) and Python is interpreting the resulting byte stream in little-endian order (least significant byte first).
One simple fix is to change the Java buffer's ByteOrder attribute:
mem.order(ByteOrder.LITTLE_ENDIAN);
Or tell Python to use big-endian order. Python doesn't seem to have an analogous option for its memmap functions, so this will probably require using struct.unpack_from to specify the byte order.

Google MD5 cracker (in java)

i find below code for Cracking Md5 hashes : (from : aboulton.blogspot.com.tr)
package md5crack;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;
import java.net.URLConnection;
import java.util.logging.Level;
import java.util.logging.Logger;
/**
*
* #author Adam Boulton - Using Google to crack MD5
*/
public class MD5Cracker {
/**
* #param args the command line arguments
*/
public static void main(String[] args) {
// TODO code application logic here
if(args[0] == null || args[0].isEmpty())
{
System.out.println("-= Google MD5 Cracker =-");
System.out.println("-= Adam Boulton - 2010 =- ");
System.out.println("Usage: MD5crack <hash>");
}
String hash = args[0];
String url = String.format("https://www.google.com/search?q=%s", hash);
try {
URL oracle = new URL(url);
URLConnection conn = oracle.openConnection();
//keep Google happy, otherwise connection refused.
conn.setRequestProperty("user-agent", "Mozilla/5.0 Windows NT6.1 WOW64 AppleWebKit/535.7 KHTML, like Gecko Chrome/16.0.912.63 Safari/535.7");
BufferedReader in = new BufferedReader(
new InputStreamReader(
conn.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null) {
String[] words = inputLine.split("\\s+");
for (String word : words) {
String wordHash = DigestUtils.md5Hex(word);
if (wordHash.equals(hash)) {
System.out.println("[*] Found: " + word);
System.exit(0);
}
}
}
System.out.println("[!] No results.");
in.close();
} catch (IOException ex) {
Logger.getLogger(MD5Cracker.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
but in this line i have an error :
String wordHash = DigestUtils.md5Hex(word);
Error :
DigestUtils cannot be resolved
How can i fix that ?
and this is good method or Class for crack and finding Decoded MD5 hash ?
How we can use this with Optimizing on android ?
Thanks.

Your program doesn´t know what DigestUtils is standing for.
You need to either import it, or download codec from the appache common jar and reference it as external jar into your project and import it afterwards

... and this is good method or Class for crack and finding Decoded MD5 hash?
Is it going to be effective? In general, probably not.
There are 340,282,366,920,938,463,463,374,607,431,768,211,456 possible different MD5 hashes. If you have an MD5 hash for an arbitrary string or document, then the chances of finding that hash in Google search results is vanishingly small.
Indeed, it is pretty obvious that Google's search engines wouldn't be able to index more than a miniscule fraction of the possible MD5 hashes. According to this article, the "big 4" internet companies were estimated to have ~1,200 Petabytes of storage in 2013. That's a factor of ~1020 too small just to store all possible MD5 hashes.
However, if you can identify a use-case where mappings between relevant strings and MD5 hashes are systematically published in web pages, then this approach might work ... for those strings. One such use-case would be MD5 hashes for email addresses, if someone has published the mappings in pages that Google indexes.

Calculating Amazon MWS Signature using Java / Filemaker

We've been trying to calculate the Amazon Signature for the past few days using the Java code provided on Amazon's site.
I'm not a seasoned Java developer, but we've managed to use the basis of their code found at http://docs.developer.amazonservices.com/en_US/dev_guide/DG_ClientLibraries.html#DG_OwnClientLibrary__Signatures to generate a signature but we keep getting a "signature doesn't match" error that we're running into a wall debugging. Here is our current Java code with account specific information omitted:
import java.io.UnsupportedEncodingException;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.URLEncoder;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.security.SignatureException;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.Map.Entry;
import java.util.TreeMap;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import org.apache.commons.codec.binary.Base64;
import java.sql.*;
import java.util.Date;
import java.sql.Timestamp;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.TimeZone;
import java.util.Date;
public class Main {
private static final String CHARACTER_ENCODING = "UTF-8";
final static String ALGORITHM = "HmacSHA256";
public static void main(String[] args) throws Exception {
// Change this secret key to yours
String secretKey = "XXXXXXXXXXXXXXXXXXXXXX"; // This secret key does have a forward slash in it (3rd to last character), would that adversely affect anything?
// Use the endpoint for your marketplace
String serviceUrl = "https://mws.amazonservices.com/Orders/2011-01-01";
// Create set of parameters needed and store in a map
HashMap<String, String> parameters = new HashMap<String,String>();
// Add required parameters. Change these as needed.
parameters.put("AWSAccessKeyId", urlEncode("XXXXXXXXXXX"));
parameters.put("Action", urlEncode("ListOrders"));
parameters.put("MarketplaceId.Id.1", urlEncode("ATVPDKIKX0DER"));
parameters.put("Merchant", urlEncode("ALZYDHGPLQNLD"));
parameters.put("OrderStatus.Status.1", urlEncode("PartiallyShipped"));
parameters.put("OrderStatus.Status.2", urlEncode("Unshipped"));
parameters.put("SignatureMethod", urlEncode(ALGORITHM));
parameters.put("SignatureVersion", urlEncode("2"));
parameters.put("Timestamp", urlEncode("2014-01-29T22:11:00Z"));
parameters.put("Version", urlEncode("2011-01-01"));
//parameters.put("SubmittedFromDate", urlEncode("2014-01-28T15:05:00Z"));
// Format the parameters as they will appear in final format
// (without the signature parameter)
String formattedParameters = calculateStringToSignV2(parameters, serviceUrl);
//System.out.println(formattedParameters);
String signature = sign(formattedParameters, secretKey);
System.out.println(urlEncode(signature));
// Add signature to the parameters and display final results
parameters.put("Signature", urlEncode(signature));
System.out.println(calculateStringToSignV2(parameters, serviceUrl));
// TEST AREA
// Signiture sig = new Signiture();
// String HMAC = sig.calculateRFC2104HMAC(parameters, secretKey);
// TEST AREA
}
/* If Signature Version is 2, string to sign is based on following:
*
* 1. The HTTP Request Method followed by an ASCII newline (%0A)
*
* 2. The HTTP Host header in the form of lowercase host,
* followed by an ASCII newline.
*
* 3. The URL encoded HTTP absolute path component of the URI
* (up to but not including the query string parameters);
* if this is empty use a forward '/'. This parameter is followed
* by an ASCII newline.
*
* 4. The concatenation of all query string components (names and
* values) as UTF-8 characters which are URL encoded as per RFC
* 3986 (hex characters MUST be uppercase), sorted using
* lexicographic byte ordering. Parameter names are separated from
* their values by the '=' character (ASCII character 61), even if
* the value is empty. Pairs of parameter and values are separated
* by the '&' character (ASCII code 38).
*
*/
private static String calculateStringToSignV2(Map<String, String> parameters, String serviceUrl)
throws SignatureException, URISyntaxException {
// Sort the parameters alphabetically by storing
// in TreeMap structure
Map<String, String> sorted = new TreeMap<String, String>();
sorted.putAll(parameters);
// Set endpoint value
URI endpoint = new URI(serviceUrl.toLowerCase());
// Create flattened (String) representation
StringBuilder data = new StringBuilder();
/*data.append("POST\n");
data.append(endpoint.getHost());
data.append("\n/");
data.append("\n");*/
Iterator<Entry<String, String>> pairs = sorted.entrySet().iterator();
while (pairs.hasNext()) {
Map.Entry<String, String> pair = pairs.next();
if (pair.getValue() != null) {
data.append( pair.getKey() + "=" + pair.getValue());
}
else {
data.append( pair.getKey() + "=");
}
// Delimit parameters with ampersand (&)
if (pairs.hasNext()) {
data.append( "&");
}
}
return data.toString();
}
/*
* Sign the text with the given secret key and convert to base64
*/
private static String sign(String data, String secretKey)
throws NoSuchAlgorithmException, InvalidKeyException,
IllegalStateException, UnsupportedEncodingException {
Mac mac = Mac.getInstance(ALGORITHM);
//System.out.println(mac);//
mac.init(new SecretKeySpec(secretKey.getBytes(CHARACTER_ENCODING), ALGORITHM));
//System.out.println(mac);//
byte[] signature = mac.doFinal(data.getBytes(CHARACTER_ENCODING));
//System.out.println(signature);//
String signatureBase64 = new String(Base64.encodeBase64(signature), CHARACTER_ENCODING);
System.out.println(signatureBase64);
return new String(signatureBase64);
}
private static String urlEncode(String rawValue) {
String value = (rawValue == null) ? "" : rawValue;
String encoded = null;
try {
encoded = URLEncoder.encode(value, CHARACTER_ENCODING)
.replace("+", "%20")
.replace("*", "%2A")
.replace("%7E","~");
} catch (UnsupportedEncodingException e) {
System.err.println("Unknown encoding: " + CHARACTER_ENCODING);
e.printStackTrace();
}
return encoded;
}
}
Right now we're testing this by hand because I'm trying to submit requests/manage the data through a Filemaker Database, so I may be adding the signature back to the URL query incorrectly. I'm assuming that 1) all of the parameters need to be listed in the query alphabetically and 2) the signature is the only exception that gets appended, last.
For example:
AWSAccessKeyId=AKIAITPQPJO62G4LAE7Q&Action=ListOrders&MarketplaceId.Id.1=XXXXXXXXXXX&Merchant=XXXXXXXXXXX&OrderStatus.Status.1=PartiallyShipped&OrderStatus.Status.2=Unshipped&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2014-01-29T22%3A11%3A00Z&Version=2011-01-01&Signature=tNufJeONZlscTlHs%2FLAWBs7zwsfpIaQcUK%2B5XIPJpcQ%3D
But the response I keep getting is this:
<?xml version="1.0"?>
<ErrorResponse xmlns="https://mws.amazonservices.com/Orders/2011-01-01">
<Error>
<Type>Sender</Type>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.</Message>
</Error>
<RequestID>7532e668-c660-4db6-b129-f5fe5d3fad63</RequestID>
</ErrorResponse>
Any insight would be greatly appreciated. Debugging is fun but I need to get past this already!