Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Closed 7 years ago.
Improve this question
I have a byte array, and I want to compute the MD5 hash in java and C# separately. However, their generate very different result.
below is my C# code:
byte[] input = { 90, 12, 200, 139, 85, 104, 9, 202, 0, 0, 0, 0, 28, 251, 54, 201, 233, 153, 79, 1 };
MD5 md5 = MD5.Create();
byte[] result = md5.ComputeHash(input);
It generate md5 hash: 85,126,37,15,86,254,54,94,243,185,219,84,21,17,192,153,.
and below is the java code:
byte[] input = {90,12,-56,-117,85,104,9,-54,0,0,0,0,28,-5,54,-47,-23,-103,79,1};
byte[] md5 = MessageDigest.getInstance("MD5").digest(input);
and it results in:
-56,-74,-89,-76,9,35,-83,-89,-73,-39,17,83,24,18,-91,-62,
As you can see, the results are quite different. I know c# use unsigned byte, java uses signed byte. There is no way make me believe the results are identical.
Thanks in advance.
Your C# and Java inputs aren't the same.
Let's try to convert the C# input to signed bytes:
byte[] input = { 90, 12, 200, 139, 85, 104, 9, 202, 0, 0, 0, 0, 28, 251, 54, 201, 233, 153, 79, 1 };
sbyte[] signedInput = input.Select(i => unchecked((sbyte)i)).ToArray();
Console.WriteLine(string.Join(", ", signedInput));
This outputs:
90, 12, -56, -117, 85, 104, 9, -54, 0, 0, 0, 0, 28, -5, 54, -55, -23, -103, 79, 1
There's a different byte here, in bold. The Java version contains -47 at this offset.
And just to be sure, we can do a simple check using the Java version's input:
var javaInput = new[] { 90, 12, -56, -117, 85, 104, 9, -54, 0, 0, 0, 0, 28, -5, 54, -47, -23, -103, 79, 1 };
var javaInputUnsigned = javaInput.Select(i => unchecked((byte)i)).ToArray();
var hash = MD5.Create().ComputeHash(javaInputUnsigned).Select(i => unchecked((sbyte)i)).ToArray();
Console.WriteLine(string.Join(", ", hash));
This yields the same result as in the Java version:
-56, -74, -89, -76, 9, 35, -83, -89, -73, -39, 17, 83, 24, 18, -91, -62
Related
I am trying two different ways to decode Phoenix Session Cookie.
First one is Elixir's interaction shell, and the second one is with Java.
Please see the following examples;
IEx
iex(1)> set_cookie = "SFMyNTY.g3QAAAABbQAAAAtfY3NyZl90b2tlbm0AAAAYZFRuNUtQMkJ5YWtKT1JnWUtCeXhmNmdP.l0T3G-i8I5dMwz7lEZnQAeK_WeqEZTxcDeyNY2poz_M"
"SFMyNTY.g3QAAAABbQAAAAtfY3NyZl90b2tlbm0AAAAYZFRuNUtQMkJ5YWtKT1JnWUtCeXhmNmdP.l0T3G-i8I5dMwz7lEZnQAeK_WeqEZTxcDeyNY2poz_M"
iex(2)> [_, payload, _] = String.split(set_cookie, ".", parts: 3)
["SFMyNTY",
"g3QAAAABbQAAAAtfY3NyZl90b2tlbm0AAAAYZFRuNUtQMkJ5YWtKT1JnWUtCeXhmNmdP",
"l0T3G-i8I5dMwz7lEZnQAeK_WeqEZTxcDeyNY2poz_M"]
iex(3)> {:ok, encoded_term } = Base.url_decode64(payload, padding: false)
{:ok,
<<131, 116, 0, 0, 0, 1, 109, 0, 0, 0, 11, 95, 99, 115, 114, 102, 95, 116, 111,
107, 101, 110, 109, 0, 0, 0, 24, 100, 84, 110, 53, 75, 80, 50, 66, 121, 97,
107, 74, 79, 82, 103, 89, 75, 66, 121, 120, 102, ...>>}
iex(4)> :erlang.binary_to_term(encoded_term)
%{"_csrf_token" => "dTn5KP2ByakJORgYKByxf6gO"}
Java
public static String decodePhoenixSessionCookie(String sessionCookie) {
String payload = sessionCookie.split("\\.")[1];
byte[] encoded_term = Base64.getUrlDecoder().decode(payload.getBytes());
return new String(encoded_term);
}
Java Output
�tm_csrf_tokenmdTn5KP2ByakJORgYKByxf6gO
What I wonder is; with the Java way, I can fully achieve field name and it's value, but some gibberish values come with them.
Do you know what's the reason for this?
Do I have a chance to get clean output like Elixir way in Java way?
I'm implementing a pipeline to read RabbitMq queue.
I'm having problems when I read it at unbound stream
it is saying that channel is already closed and ack is not sent to rabbitmq and message still on the queue:
WARNING: Failed to finalize Finalization{expiryTime=2020-11-21T19:33:14.909Z, callback=org.apache.beam.sdk.io.Read$UnboundedSourceAsSDFWrapperFn$$Lambda$378/0x00000001007ee440#4ae82af9} for completed bundle CommittedImmutableListBundle{PCollection=Read RabbitMQ queue/Read(RabbitMQSource)/ParDo(UnboundedSourceAsSDFWrapper)/ParMultiDo(UnboundedSourceAsSDFWrapper)/ProcessKeyedElements/SplittableParDoViaKeyedWorkItems.GBKIntoKeyedWorkItems.out [PCollection], key=org.apache.beam.repackaged.direct_java.runners.local.StructuralKey$CoderStructuralKey#3607f949, elements=[ValueInGlobalWindow{value=ComposedKeyedWorkItem{key=[-55, 41, -123, 97, 13, 104, 92, 61, 92, 122, -19, 112, -90, 16, 7, -97, 89, 107, -80, 12, 9, 120, 10, -97, 72, 114, -62, -105, 101, -34, 96, 48, 30, -96, 8, -19, 23, -115, -9, 87, 1, -58, -127, 70, -59, -24, -40, -111, -63, -119, 51, -108, 126, 64, -4, -120, -41, 9, 56, -63, -18, -18, -1, 17, -82, 90, -32, 110, 67, -12, -97, 10, -107, -110, 13, -74, -47, -113, 122, 27, 52, 46, -111, -118, -8, 118, -3, 20, 71, -109, 65, -87, -94, 107, 114, 116, -110, -126, -79, -123, -67, 18, -33, 70, -100, 9, -81, -65, -2, 98, 33, -122, -46, 23, -103, -70, 79, -23, 74, 9, 5, -9, 65, -33, -52, 5, 9, 101], elements=[], timers=[TimerData{timerId=1:1605986594072, timerFamilyId=, namespace=Window(org.apache.beam.sdk.transforms.windowing.GlobalWindow#4958d651), timestamp=2020-11-21T19:23:14.072Z, outputTimestamp=2020-11-21T19:23:14.072Z, domain=PROCESSING_TIME}]}, pane=PaneInfo.NO_FIRING}], minimumTimestamp=-290308-12-21T19:59:05.225Z, synchronizedProcessingOutputWatermark=2020-11-21T19:23:14.757Z}
com.rabbitmq.client.AlreadyClosedException: channel is already closed due to clean channel shutdown; protocol method: #method<channel.close>(reply-code=200, reply-text=OK, class-id=0, method-id=0)
at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:258)
at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:427)
at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:421)
at com.rabbitmq.client.impl.recovery.RecoveryAwareChannelN.basicAck(RecoveryAwareChannelN.java:93)
at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.basicAck(AutorecoveringChannel.java:428)
at org.apache.beam.sdk.io.rabbitmq.RabbitMqIO$RabbitMQCheckpointMark.finalizeCheckpoint(RabbitMqIO.java:433)
at org.apache.beam.runners.direct.EvaluationContext.handleResult(EvaluationContext.java:195)
at org.apache.beam.runners.direct.QuiescenceDriver$TimerIterableCompletionCallback.handleResult(QuiescenceDriver.java:287)
at org.apache.beam.runners.direct.DirectTransformExecutor.finishBundle(DirectTransformExecutor.java:189)
at org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:126)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
BUT
if I include withMaxNumRecords
I receive the message and ack is sent to rabbitmq queue
but it works as bound data
CODE
my code is like below:
Pipeline p = Pipeline.create(options);
PCollection<RabbitMqMessage> messages = p.apply("Read RabbitMQ queue",
RabbitMqIO.read()
.withUri("amqp://guest:guest#localhost:5672")
.withQueue("queue")
//.withMaxNumRecords(1) // TRANFORM BOUND
);
PCollection<TableRow> rows = messages.apply("Transform Json to TableRow",
ParDo.of(new DoFn<RabbitMqMessage, TableRow>() {
#ProcessElement
public void processElement(ProcessContext c) {
ObjectMapper objectMapper = new ObjectMapper();
String jsonInString = new String(c.element().getBody());
LOG.info(jsonInString);
}
}));
rows.apply(
"Write to BigQuery",
BigQueryIO.writeTableRows()
.to("livelo-analytics-dev:cart_idle.cart_idle_process")
.withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER)
.withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
);
Someone could help with this?
I sent an email to apache dev thread and get an awesome answer from
Boyuan Zhang
that worked as a workaround for me
As a workaround, you can add --experiments=use_deprecated_read when launching your pipeline to bypass the sdf unbounded source wrapper here.
--experiments=use_deprecated_read
Put it as an argument on the command line and worked fine for me
I saw problem in this piece of code:
byte[] buf = new byte[6];
buf = "abcdef".getBytes();
System.out.println(buf.length);
Array was made for 6 bytes. If I get bytes from string with length 6 I will get much more bytes. So how will all these bytes get into this array? But this is working. Moreover, buf.length shows length of that array as it is array of chars not those bytes.
Afterwards, I realized that in
byte[] buf = new byte[6];
6 does not mean much, i.e. I can put there 0 or 1 or 2 or so on and code will work (with buf.length showing length of given string not array - what I see as second problem or discrepancy).
This question is different than Why does Java's String.getBytes() uses “ISO-8859-1” because it have one aspect more, at least: variables assignment oversight (getBytes() returns new array), i.e. it don't fully address my question.
That is not how variable assignments work
Thinking that assigning a 6 byte array to a variable will limit the length of any other arrays assigned to the same variable show a fundamental lack of comprehension on what variable are and how they work.
Really think about why you think assigning a variable to a fixed length array would limit the length of being assigned to another length array?
Strings are Unicode in Java
Strings in Java are Unicode and internally represented as UTF-16 which means they are 2 or 4 bytes per character in memory.
When they are converted to a byte array the number of bytes that represents the string is determined by what encoding is used when converting to the byte[].
Always specify an appropriate character encoding when converting Strings to arrays to get what you expect.
But even then UTF-8 would not guarantee single bytes per character, and ASCII would be not be able to represent non ASCII Unicode characters.
Character encoding is tricky
The ubiquitous internet encoding standard is UTF-8 it will correct in 99.9999999% of all cases, in those cases it isn't converting UTF-8 to the correct encoding is trivial because UTF-8 is so well supported in every toolchain.
Learn to make everything final and you will a lot easier time and less confusion.
import com.google.common.base.Charsets;
import javax.annotation.Nonnull;
import java.util.Arrays;
public class Scratch
{
public static void main(final String[] args)
{
printWithEncodings("Hello World!");
printWithEncodings("こんにちは世界!");
}
private static void printWithEncodings(#Nonnull final String s)
{
System.out.println("s = " + s);
final byte[] defaultEncoding = s.getBytes(); // never do this, you do not know what you will get!
// for ASCII characters the first three will all be the same single byte representations
final byte[] iso88591Encoding = s.getBytes(Charsets.ISO_8859_1);
final byte[] asciiEncoding = s.getBytes(Charsets.US_ASCII);
final byte[] utf8Encoding = s.getBytes(Charsets.UTF_8);
final byte[] utf16Encoding = s.getBytes(Charsets.UTF_16);
System.out.println("Arrays.toString(defaultEncoding) = " + Arrays.toString(defaultEncoding));
System.out.println("Arrays.toString(iso88591) = " + Arrays.toString(iso88591Encoding));
System.out.println("Arrays.toString(asciiEncoding) = " + Arrays.toString(asciiEncoding));
System.out.println("Arrays.toString(utf8Encoding) = " + Arrays.toString(utf8Encoding));
System.out.println("Arrays.toString(utf16Encoding) = " + Arrays.toString(utf16Encoding));
}
}
results in
s = Hello World!
Arrays.toString(defaultEncoding) = [72, 101, 108, 108, 111, 32, 87, 111, 114, 108, 100, 33]
Arrays.toString(iso88591) = [72, 101, 108, 108, 111, 32, 87, 111, 114, 108, 100, 33]
Arrays.toString(asciiEncoding) = [72, 101, 108, 108, 111, 32, 87, 111, 114, 108, 100, 33]
Arrays.toString(utf8Encoding) = [72, 101, 108, 108, 111, 32, 87, 111, 114, 108, 100, 33]
Arrays.toString(utf16Encoding) = [-2, -1, 0, 72, 0, 101, 0, 108, 0, 108, 0, 111, 0, 32, 0, 87, 0, 111, 0, 114, 0, 108, 0, 100, 0, 33]
s = こんにちは世界!
Arrays.toString(defaultEncoding) = [-29, -127, -109, -29, -126, -109, -29, -127, -85, -29, -127, -95, -29, -127, -81, -28, -72, -106, -25, -107, -116, 33]
Arrays.toString(iso88591) = [63, 63, 63, 63, 63, 63, 63, 33]
Arrays.toString(asciiEncoding) = [63, 63, 63, 63, 63, 63, 63, 33]
Arrays.toString(utf8Encoding) = [-29, -127, -109, -29, -126, -109, -29, -127, -85, -29, -127, -95, -29, -127, -81, -28, -72, -106, -25, -107, -116, 33]
Arrays.toString(utf16Encoding) = [-2, -1, 48, 83, 48, -109, 48, 107, 48, 97, 48, 111, 78, 22, 117, 76, 0, 33]
Always specify the Charset encoding!
.bytes(Charset) is always the correct way to convert a String to bytes. Use whatever encoding you need.
Internally supported encodings for JDK7
new byte[6]; has no effect whatsoever as the array reference buf is getting updated with reference of the array returned by "abcdef".getBytes();.
That's because String.getBytes() returns an entirely different array object which is then assigned to buf. You could have just as easily done this:
byte[] buf = "abcdef".getBytes();
System.out.println(buf.length);
While I was testing the send and receive methods which I created for my project I ran into a strange problem.
When I send a certain message using a correlationId that is based on a UUID object, the receiving party gets a slightly modified version of this correlationId (which cannot be deserialised).
On the sending side I do this:
MessageProperties properties = new MessageProperties();
properties.setCorrelationId(MessageSerializer.serialize(UUID.randomUUID().toString()));
On my last test the UUID generated was: "d4170243-9e7e-4c42-9168-f9da4debc5bb"
This generates the following correlationId (when serialized):
[-84, -19, 0, 5, 116, 0, 36, 100, 52, 49, 55, 48, 50, 52, 51, 45, 57, 101, 55, 101, 45, 52, 99, 52, 50, 45, 57, 49, 54, 56, 45, 102, 57, 100, 97, 52, 100, 101, 98, 99, 53, 98, 98]
When I receive the message on the other side this id is slightly changed:
[-17, -65, -67, -17, -65, -67, 0, 5, 116, 0, 36, 100, 52, 49, 55, 48, 50, 52, 51, 45, 57, 101, 55, 101, 45, 52, 99, 52, 50, 45, 57, 49, 54, 56, 45, 102, 57, 100, 97, 52, 100, 101, 98, 99, 53, 98, 98]
When using the RabbitMQ management plugin I noticed that the id already changed upon arrival at the queue.
Tracing my code on the sending side brings me to the send option of the RabbitTemplate class.
RabbitTemplate template = new RabbitTemplate(connection);
template.setExchange("amq.direct");
template.setRoutingKey("some.route");
template.send(message);
But I can't figure out what's causing this problem. I guess it's just me using the correlationId option the wrong way. Could someone help me out?
Appreciate it.
Explanation is the following:
You serialize the UUID string to a byte array
Your serialization prepends non ascii character to this array ([-17, -65, -67, -17, -65, -67, 0, 5, 116, 0, 36,...])
The reference documentation states that the correlation id is a shortstr. RabbitMQ client converts this byte array to a string using
new String(yourArray , "UTF-8").
The non ascii character "corrupt" the conversion from byte[] to string
You can get the same result with the following code:
new String(MessageSerializer.serialize(UUID.randomUUID().toString()) , "UTF-8").getByte("UTF-8")
Which will return:
[-17, -65, -67, -17, -65, -67, 0, 5, 116, 0, 36, 100, 52, 49, 55, 48, 50, 52, 51, 45, 57, 101, 55, 101, 45, 52, 99, 52, 50, 45, 57, 49, 54, 56, 45, 102, 57, 100, 97, 52, 100, 101, 98, 99, 53, 98, 98]
I am trying to encrypt the password using a salt and storing it into the MySQL database.
I referred to this stackOverflow question
My code is similar to this:
private byte[] encrypt(String passwordToSave, byte[] salt)
throws UnsupportedEncodingException
{
int seedBytes = 20;
int hashBytes = 20;
int iterations = 1000;
if(null == salt)
{
SecureRandom rng = new SecureRandom();
salt = rng.generateSeed(seedBytes);
}
PKCS5S2ParametersGenerator kdf = new PKCS5S2ParametersGenerator();
kdf.init(passwordToSave.getBytes("UTF-8"), salt, iterations);
byte[] hash =
((KeyParameter) kdf.generateDerivedMacParameters(8*hashBytes)).getKey();
return hash;
}
I just altered the function little to make use of it for both the purposes.
Encrypt the password while creating the user account and store it with the salt.
Encrypt the user password with the stored salt from database when he is trying to logging in and check it with the stored value of password.
The issue with this is,
I am not getting back what I stored.
I used a lot of different things,
I used Base64 for encoding and stored into DB and decoded using the same while getting it back.
I tried to use VARBINARY and BLOB to save the byte[] data but no luck.
Then I used VARCHAR and just stored the byte[] by creating a new String from it using "UTF-8" encoding type.
I am new to cryptography so if I am wrong, please point it out.
Thanks in advance. :)
EDIT:
The output when I ran the encrypt twice:
Salt : [34, 17, -80, -59, 93, -90, 37, -25, -11, -43, 44, 1, 10, 7, -66, -108, 97, 36, 95, -116]
First Attempt: [-76, -3, 114, -69, 78, 21, -59, 23, 127, -15, 114, -106, -52, 23, 34, 91, 123, 6, 76, -115]
Second Attempt: [-76, -3, 114, -69, 78, 21, -59, 23, 127, -15, 114, -106, -52, 23, 34, 91, 123, 6, 76, -115]
Salt : [34, 17, -80, -59, 93, -90, 37, -25, -11, -43, 44, 1, 10, 7, -66, -108, 97, 36, 95, -116]