I am trying to convert a Java function in C#. Here is the original code:
class SecureRandomString {
private static SecureRandom random = new SecureRandom();
private static Base64.Encoder encoder = Base64.getUrlEncoder().withoutPadding();
public static String generate(String seed) {
byte[] buffer;
if (seed == null) {
buffer = new byte[20];
random.nextBytes(buffer);
}
else {
buffer = seed.getBytes();
}
return encoder.encodeToString(buffer);
}
}
And here is what I did in C#:
public class Program
{
private static readonly Random random = new Random();
public static string Generate(string seed = null)
{
byte[] buffer;
if (seed == null)
{
buffer = new byte[20];
random.NextBytes(buffer);
}
else
{
buffer = Encoding.UTF8.GetBytes(seed);
}
return System.Web.HttpUtility.UrlPathEncode(RemovePadding(Convert.ToBase64String(buffer)));
}
private static string RemovePadding(string s) => s.TrimEnd('=');
}
I wrote some testcases:
Assert(Generate("a"), "YQ");
Assert(Generate("ab"), "YWI");
Assert(Generate("abc"), "YWJj");
Assert(Generate("abcd"), "YWJjZA");
Assert(Generate("abcd?"), "YWJjZD8");
Assert(Generate("test wewqe_%we()21-3012"), "dGVzdCB3ZXdxZV8ld2UoKTIxLTMwMTI");
Assert(Generate("test wewqe_%we()21-3012_"), "dGVzdCB3ZXdxZV8ld2UoKTIxLTMwMTJf");
Assert(Generate("test wewqe_%we()21-3012/"), "dGVzdCB3ZXdxZV8ld2UoKTIxLTMwMTIv");
Assert(Generate("test wewqe_%we()21-3012!"), "dGVzdCB3ZXdxZV8ld2UoKTIxLTMwMTIh");
Assert(Generate("test wewqe_%we()21-3012a?"), "dGVzdCB3ZXdxZV8ld2UoKTIxLTMwMTJhPw");`
And everything works fine, until I try the following one:
Assert(Generate("test wewqe_%we()21-3012?"), "dGVzdCB3ZXdxZV8ld2UoKTIxLTMwMTI_");
My code output dGVzdCB3ZXdxZV8ld2UoKTIxLTMwMTI/ instead of the expected dGVzdCB3ZXdxZV8ld2UoKTIxLTMwMTI_. Why?
I think that the culprit is the encoder. The original code configure its encoder like this Base64.getUrlEncoder().withoutPadding(). The withoutPadding() is basically a TrimEnd("=") but I am not sure how to code the getUrlEncoder().
I looked into this handy conversion table URL Encoding using C# without finding nothing for my case.
I tried HttpUtility.UrlEncode but the output is not right.
What did I missed?
According to Oracle documentation, here is what getUrlEncoder() does:
Returns a Base64.Encoder that encodes using the URL and Filename safe type base64 encoding scheme.
Alright what is "URL and Filename safe". Once more the documenation is helping:
Uses the "URL and Filename safe Base64 Alphabet" as specified in Table 2 of RFC 4648 for encoding and decoding. The encoder does not add any line feed (line separator) character. The decoder rejects data that contains characters outside the base64 alphabet.
We can now look online for the RFC 4648. Here is the Table 2:
Table 2: The "URL and Filename safe" Base 64 Alphabet
Value Encoding Value Encoding Value Encoding Value Encoding
0 A 17 R 34 i 51 z
1 B 18 S 35 j 52 0
2 C 19 T 36 k 53 1
3 D 20 U 37 l 54 2
4 E 21 V 38 m 55 3
5 F 22 W 39 n 56 4
6 G 23 X 40 o 57 5
7 H 24 Y 41 p 58 6
8 I 25 Z 42 q 59 7
9 J 26 a 43 r 60 8
10 K 27 b 44 s 61 9
11 L 28 c 45 t 62 - (minus)
12 M 29 d 46 u 63 _
13 N 30 e 47 v (underline)
14 O 31 f 48 w
15 P 32 g 49 x
16 Q 33 h 50 y (pad) =
It is an encoding table. For example given 0 should output A, given 42 should ouput q, etc.
Let's check the decoding table, the Table 1:
Table 1: The Base 64 Alphabet
Value Encoding Value Encoding Value Encoding Value Encoding
0 A 17 R 34 i 51 z
1 B 18 S 35 j 52 0
2 C 19 T 36 k 53 1
3 D 20 U 37 l 54 2
4 E 21 V 38 m 55 3
5 F 22 W 39 n 56 4
6 G 23 X 40 o 57 5
7 H 24 Y 41 p 58 6
8 I 25 Z 42 q 59 7
9 J 26 a 43 r 60 8
10 K 27 b 44 s 61 9
11 L 28 c 45 t 62 +
12 M 29 d 46 u 63 /
13 N 30 e 47 v
14 O 31 f 48 w (pad) =
15 P 32 g 49 x
16 Q 33 h 50 y
Note that both table are strictly equals minus two things:
'+' is encoded to '-'
'/' is encoded to '_'
You should be able to fix your problem with:
private static string Encode(string s) => s.Replace("+", "-").Replace("/", "_");
Related
According to this answer Java can hold up to 2^31 - 1 characters. I was trying to do benchmarking and stuffs, so I tried to create a large amount of string and write it to a file like this:
import java.io.*
fun main() {
val out = File("ouput.txt").apply { createNewFile() }.printWriter()
sequence {
var x = 0
while (true) {
yield("${++x} ${++x} ${++x} ${++x} ${++x}")
}
}.take(5000).forEach { out.println(it) }
out.close()
}
And then the output.txt file contains like this:
1 2 3 4 5
6 7 8 9 10
11 12 13 14 15
16 17 18 19 20
21 22 23 24 25
// ... 5000 lines
And then I copied all the contents of the file into a string for some benchmarking of some functions, so this is how it looks:
import kotlin.system.*
fun main() {
val inputs = """
1 2 3 4 5
6 7 8 9 10
11 12 13 14 15
16 17 18 19 20
21 22 23 24 25
// ... 5000 lines
24986 24987 24988 24989 24990
24991 24992 24993 24994 24995
24996 24997 24998 24999 25000
""".trimIndent()
measureNanoTime {
inputs.reader().forEachLine { line ->
val (a, b, c, d, e) = line.split(" ").map(String::toInt)
}
}.div(5000).let(::println)
}
The total character count of the file/string is 138894
String can hold up to 2147483647
But the Kotlin code does not compile (last file) It throws compilation error:
e: org.jetbrains.kotlin.codegen.CompilationException: Back-end (JVM) Internal error: wrong bytecode generated
// more lines
The root cause java.lang.IllegalArgumentException was thrown at: org.jetbrains.org.objectweb.asm.ByteVector.putUTF8(ByteVector.java:246)
at org.jetbrains.kotlin.codegen.TransformationMethodVisitor.visitEnd(TransformationMethodVisitor.kt:92)
at org.jetbrains.kotlin.codegen.FunctionCodegen.endVisit(FunctionCodegen.java:971)
... 43 more
Caused by: java.lang.IllegalArgumentException: UTF8 string too large
Here is the total log of exception with stacktrace: https://gist.github.com/Animeshz/1a18a7e99b0c0b913027b7fb36940c07
There is limit in java class file, length of string constant must fit in 16 bits ie. 65535 bytes (not characters) is max length of string in source code.
The class File Format
This question already has answers here:
How to convert byte array to hex format in Java
(4 answers)
Closed 2 years ago.
So i have a piece of code, which converted byte[] from a binary file into int[] which then went on to be converted to String.
Example: 02 09 1A would translate to "2926" using String.valueOf(byte);
But this is where the fun begins: the old Code already split up the array before it became String so it didn't matter. Now i have code which needs to somehow figure out if there is a 2 9 or 29... How can i get the String to
- stay in Hex instead of "switching" to decimal and
- keep the zeros in the string
so i can always grab the next two chars, figure out which info they display and go on?
Input: 05 06 1D 11 07 08 01 32 21 28 2F 20 2E 21 34 22 25 33 01 02 09 0A FF 0B 0C 2!(/ .!4"%3
ÿ
Current Output: 562917781503340473246335234375112910-11112
Expected Output: 05061D110708013221282F202E21342225330102090AFF0B0C
(Didn't put my old code in here since it's based on int which is crap)
If I understand correctly, you have a byte[] as input, and you want to output a string that is all those bytes, in hex form, joined together:
private static String byteArrayToString(byte[] byteArray) {
StringBuilder builder = new StringBuilder();
for (byte b : byteArray) {
int i = Byte.toUnsignedInt(b);
if (i < 16) {
builder.append("0");
}
builder.append(Integer.toHexString(i).toUpperCase());
}
return builder.toString();
}
Note the use of Byte.toUnsignedInt. This converts things like -1 the byte to the int 0xff. This is required because Java's bytes are signed.
Also note where I pad with 0. I only do this if the byte is a single digit in hex. i < 16 is the condition to check this.
Another way:
char[] hexVals = {'0', '1','2','3','4','5','6','7','8','9','A', 'B','C','D','E','F'};
byte[] bytes = {5, 6, (byte)0xAB, (byte)0xde}; // sample input
char[] output = new char[bytes.length*2];
// split each byte's hex digits into two chars
for(int i=0; i < bytes.length; i++) {
output[2*i] = hexVals[(bytes[i] >> 4) & 0xf]; // HO nibble
output[2*i+1] = hexVals[bytes[i] & 0xf]; // LO nibble
}
System.out.println(new String(output)); // 0506ABDE
I'm trying to convert a hex message to ascii
and I don't understand why I get this error
String final = (msg1 + " " + msg2 + " " + msg3);
byte[] s = DatatypeConverter.parseHexBinary(final);
String final = new String(s);
and this is the error I get :(the data is correct )
java.lang.IllegalArgumentException: hexBinary needs to be even-length: 4C 4B 4C 41 36 4C 31 45 32 48 41 37 32 32 31 36 31 FF
parseHexBinary is documented to take the input as defined by xsd:hexBinary.
This means the input string must contain only hexadecimal characters and always pairs. Your input stream contains space characters, that must be removed prior to parsing.
input = input.replaceAll(" +", "");
I have the following code. I can do the reading all well and convert data into hex. The issue is when I send back String replyMessage = "7E81";, the device receives it as "3745". What is wrong? Is it due to my encoding, or do I need to do some conversion before I send back?
w = new BufferedWriter(new OutputStreamWriter(receivedSocketConn1.getOutputStream(),"ISO-8859-15")); //
r = new BufferedReader(new InputStreamReader(receivedSocketConn1.getInputStream(),"ISO-8859-15"));
int nextChar=0;
while ((nextChar=r.read()) != -1) {
StringBuilder sb = new StringBuilder();
sb.append(Integer.toHexString(nextChar));
if (sb.length() < 2) {
sb.insert(0, '0'); // pad with leading zero if needed
}
String hexChar = sb.toString();
System.out.println("\n\n hex value is "+Integer.toHexString(nextChar).toUpperCase()+" "+"Int value is:"+nextChar);
message = message+hexChar;
String messageID=message.substring(2,6);
System.out.println("messageId is :"+messageID);
if(messageID.equals("0100")){
String replyMessage = "7E81";
w.write(replyMessage+"\r\n");
w.flush();
}
}
Based on comments in chat:
the documentation say
Start Byte (1 Byte) 7e
Message ID (2 Byte) 01 00
Message Body Nature (2 Byte) 00 19
Phone no. of Device (6 Byte) 09 40 27 84 94 70
Message Serial number (2 Byte) 00 01
Message Body (N Byte) 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 09 40 27 84 94 70 00
Check Code (1Byte) 19
End Byte (1Byte) 7E
so the start and termination is 7E
for outgoing
Start Byte (1 Byte) 7e
Message ID (2 Byte) 81 00
Message Body Nature (2 Byte) 00 13
Phone no. of Device (6 Byte) 09 40 27 84 94 70
Message Serial number (2 Byte) 00 01
Message Body (N Byte) 00 01 00 32 30 31 31 31 31 30 38 31 31 33 33 32 31 39 36
Check Code (1Byte) 9A
End Byte (1Byte) 7e
This means the protocol in question is a binary protocol, not a textual protocol that sends hex strings, like you thought. As such, your use of OutputStreamWriter, InputStreamReader, StringBuilder, toHexString(), etc are all completely wrong for this protocol.
Each message received and sent begins with a fixed 13-byte header, followed by a variable-length body (the header specifies the body length), and terminated by a fixed 2-byte footer.
With that in mind, try something more like this instead:
final protected static char[] hexArray = "0123456789ABCDEF".toCharArray();
public static String bytesToHex(byte[] bytes) {
char[] hexChars = new char[bytes.length * 2];
for ( int j = 0; j < bytes.length; j++ ) {
int v = bytes[j] & 0xFF;
hexChars[j * 2] = hexArray[v >>> 4];
hexChars[j * 2 + 1] = hexArray[v & 0x0F];
}
return new String(hexChars);
}
...
w = new DataOutputStream(new BufferedOutputStream(receivedSocketConn1.getOutputStream()));
r = new DataInputStream(new BufferedInputStream(receivedSocketConn1.getInputStream()));
...
if (r.readByte() != 0x7E) // start byte
{
// ah oh, something went wrong!!
receivedSocketConn1.close();
return;
}
int messageID = r.readUnsignedShort(); // message ID
int bodyLen = r.readUnsignedShort(); // message body nature (body length)
byte[] phoneNum = new byte[6];
r.readFully(phoneNum); // device phone number
int serialNum = r.readUnsignedShort(); // message serial number
byte[] messageBody = new byte[bodyLen]; // message body
r.readFully(messageBody);
byte checkCode = r.readByte(); // check code
if (r.readByte() != 0x7E) // end byte
{
// ah oh, something went wrong!!
receivedSocketConn1.close();
return;
}
// TODO: validate checkCode if needed...
System.out.println("messageId is : 0x" + Integer.toHexString(messageID));
System.out.println("phoneNum is : " + bytesToHex(phoneNum));
System.out.println("serialNum is : 0x" + Integer.toHexString(serialNum));
System.out.println("messageBody is : " + bytesToHex(messageBody));
// process message data as needed...
switch (messageID)
{
case 0x100:
{
// ...
byte[] replyBody = new byte[19];
replyBody[0] = 0x00;
replyBody[1] = 0x01;
replyBody[2] = 0x00;
replyBody[3] = 0x32;
// and so on...
checkCode = 0x9A; // calculate as needed...
w.writeByte(0x7e); // start byte
w.writeShort(0x8100); // message ID
w.writeShort(replyBody.length); // message body nature (body length)
w.write(phoneNum); // device phone number
w.writeShort(0x0001); // message serial number
w.write(replyBody); // message body
w.writeByte(checkCode); // check code
w.writeByte(0x7e); // end byte
break;
}
// other message IDs as needed...
}
w.flush();
I'm creating a Java Client program that sends a command to server and server sends back an acknowledgement and a response string.
The response is sent back in this manner
client -> server : cmd_string
server -> client : ack_msg(06)
server -> client : response_msg
Client code
public static void readStream(InputStream in) {
byte[] messageByte = new byte[20];// assuming mug size -need to
// know eact msg size ?
boolean end = false;
String dataString = "";
int bytesRead = 0;
try {
DataInputStream in1 = new DataInputStream(in);
// while ctr==2 todo 2 streams
int ctr = 0;
while (ctr < 2) {//counter 2 if ACK if NAK ctr=1 todo
bytesRead = in1.read(messageByte);
if (bytesRead > -1) {
ctr++;
}
dataString += new String(messageByte, 0, bytesRead);
System.out.println("\ninput byte arr "+ctr);
for (byte b : messageByte) {
char c=(char)b;
System.out.print(" "+b);
}
}
System.out.println("MESSAGE: " + dataString + "\n bytesread " + bytesRead + " msg length "
+ dataString.length() + "\n");
char[] chars = dataString.toCharArray();
ArrayList<String> hex=new ArrayList<>();
// int[] msg ;
for (int i = 0; i < chars.length; i++) {
int val = (int) chars[i];
System.out.print(" " + val);
hex.add(String.format("%04x", val));
}
System.out.println("\n"+hex);
} catch (Exception e) {
e.printStackTrace();
}
// ===
}
Output
client Socket created ..
response:
input byte arr 1
6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
input byte arr 2
2 -77 67 79 -77 48 -77 3 -116 0 0 0 0 0 0 0 0 0 0 0
MESSAGE: ##³CO³0³##
(where # is some not supported special character )
bytesread 9 msg length 10
dec: 6 2 179 67 79 179 48 179 3 338
hex: [0006, 0002, 00b3, 0043, 004f, 00b3, 0030, 00b3, 0003, 0152]
bytes: 2 -77 67 79 -77 48 -77 3 -116 0 0 0 0 0 0 0 0 0 0 0 (bytes recieved in 2nd packet)
connection closed
Problem: I'm reading the last value incorrect, I have verified using wireshark the server has sent back the response as 06 02 b3 43
4f b3 30 b3 03 8c
Some how I'm reading the last value in correctly. Is there some issue with the reading stream?
EDIT
Rest of the response is read correctly but the last character should be 8c But is read as 0152Hex
Response from server : 06 02 b3 43 4f b3 30 b3 03 8c
Read by program : [0006, 0002, 00b3, 0043, 004f, 00b3, 0030, 00b3, 0003, 0152]
issue with reading the last character
EDIT 2
Response is received as 2 packets/streams
packet 1 byte arr : 6 (ACK)
packet 2 byte arr: 2 -77 67 79 -77 48 -77 3 -116 (response)
complete response read by client
dec: 6 2 179 67 79 179 48 179 3 338
hex: [0006, 0002, 00b3, 0043, 004f, 00b3, 0030, 00b3, 0003, 0152]
Thanks
The problem in this question was a matter of signed variables versus unsigned variables. When you have a number in computer memory, it is represented by a bunch of bits, each of them 0 or 1. Bytes are generally 8 bits, shorts are 16 etc. In an unsigned number, 8 bits will get you from positive 0 to 255, but not to negative numbers.
This is where signed numbers come in. In a signed number, the first bit tells you whether the following bits represent a negative or positive value. So now you can use 8 bits to represent -128 to +127. (Notice that the positive range is halved, from 255 to 127, because you "sacrifice" half of your range to the negative numbers).
So now what happens if you convert signed to unsigned? Depending on how you do it, things can go wrong. In the problem above, the code char c=(char)b; was converting a signed byte to an unsigned char. The proper way to do this is to "make your byte unsigned" before converting it to a char. You can do that like this: char c=(char)(b&0xFF); more info on casting a byte here.
Essentially, you can just remember that except for char, all java numbers are signed, and all you need to do is paste the &0xFF to make it work for a byte, 0xFFFF to make it work for a short, etc.
The details about why this works are as follows. Calling & means a bitwise and, and 0xFF is hexadecimal for 255. 255 is above the limit of a signed byte (127), so the number b&0xFF gets upgraded to a short by java. However, the short signed bit is on bit 16, while the byte signed bit is on bit 8. So now the byte signed bit becomes a normal 'data' bit in the short, and so the sign of your byte is essentially discarded.
If you do a normal cast, java recognizes that doing direct bitconversion like above would mean that you lose the sign, and java expects you don't like that (at least, that is my assumption), so it preserves the sign for you. So if this isn't what you want, you explicitly tell java what to do.