Develop Reference

Represent long in least amount of characters

I need to represent both very large and small numbers in the shortest string possible. The numbers are unsigned. I have tried just straight Base64 encode, but for some smaller numbers, the encoded string is longer than just storing the number as a string. What would be the best way to most optimally store a very large or short number in the shortest string possible with it being URL safe?

I have tried just straight Base64 encode, but for some smaller numbers, the encoded string is longer than just storing the number as a string
Base64 encoding of binary byte data will make it longer, by about a third. It is not supposed to make it shorter, but to allow safe transport of binary data in formats that are not binary safe.
However, base 64 is more compact than decimal representation of a number (or of byte data), even if it is less compact than base 256 (the raw byte data). Encoding your numbers in base 64 directly will make them more compact than decimal. This will do it:
private static final String base64Chars =
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_";
static String encodeNumber(long x) {
char[] buf = new char[11];
int p = buf.length;
do {
buf[--p] = base64Chars.charAt((int)(x % 64));
x /= 64;
} while (x != 0);
return new String(buf, p, buf.length - p);
}
static long decodeNumber(String s) {
long x = 0;
for (char c : s.toCharArray()) {
int charValue = base64Chars.indexOf(c);
if (charValue == -1) throw new NumberFormatException(s);
x *= 64;
x += charValue;
}
return x;
}
Using this encoding scheme, Long.MAX_VALUE will be the string H__________, which is 11 characters long, compared to its decimal representation 9223372036854775807 which is 19 characters long. Numbers up to about 16 million will fit in a mere 4 characters. That's about as short as you'll get it. (Technically there are two other characters which do not need to be encoded in URLs: . and ~. You can incorporate those to get base 66, which would be a smidgin shorter for some numbers, although that seems a bit pedantic.)

To extend on Stephen C's answer, here is a piece of code to convert to base 62 (but you can increase this by adding more characters to the digits String (just pick what characters are valid for you):
public static String toString(long n) {
String digits = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz";
int base = digits.length();
String s = "";
while (n > 0) {
long d = n % base;
s = digits.charAt(d) + s;
n = n / base;
}
return s;
}
This will never result in the string representation being longer than the digit one.

Assuming that you don't do any compression, and that you restrict yourself to URL safe characters, then the following procedure will give you the most compact encoding possible.
Make a list of all URL safe characters
Count them. Suppose you have N.
Represent your number in base N, representing 0 by the first character, 1 by the 2nd and so on.
So, what about compression ...
If you assume that the numbers you are representing are uniformly distributed across their range, then there is no real opportunity for compression.
Otherwise, there is potential for compression. If you can reduce the size of the common numbers then you can typically achieve a saving by compression. This is how Huffman encoding works.
But the downside is that compression at this level is not perfect across the range of numbers. It reduces the size of some numbers, but it inevitably increases the size of others.
So what does this mean for your use-case?
I think it means that you are looking at the problem the wrong way. You should not be aiming for a minimal encoded size for every number. You should be aiming to minimize the size on average ... averaged over the actual distribution of your numbers.

Better algorithm for complementing integer value excluding the leading zero binary bits

I will explain first what I mean by "complementing integer value excluding the leading zero binary bits" (from now on, I will call it Non Leading Zero Bits complement or NLZ-Complement for brevity).
For example, there is integer number 92. the binary number is 1011100. If we perform normal bitwise-NOT or Complement, the result is: -93 (signed integer) or 11111111111111111111111110100011 (binary). That's because the leading zero bits are being complemented too.
So, for NLZ-Complement, the leading zero bits are not complemented, then the result of NLZ-complementing of 92 or 1011100 is: 35 or 100011 (binary). The operation is performed by XORing the input value with sequence of 1 bits as much as the non-leading zero value. The illustration:
92: 1011100
1111111 (xor)
--------
0100011 => 35
I had made the java algorithm like this:
public static int nonLeadingZeroComplement(int n) {
if (n == 0) {
return ~n;
}
if (n == 1) {
return 0;
}
//This line is to find how much the non-leading zero (NLZ) bits count.
//This operation is same like: ceil(log2(n))
int binaryBitsCount = Integer.SIZE - Integer.numberOfLeadingZeros(n - 1);
//We use the NLZ bits count to generate sequence of 1 bits as much as the NLZ bits count as complementer
//by using shift left trick that equivalent to: 2 raised to power of binaryBitsCount.
//1L is one value with Long literal that used here because there is possibility binaryBitsCount is 32
//(if the input is -1 for example), thus it will produce 2^32 result whom value can't be contained in
//java signed int type.
int oneBitsSequence = (int)((1L << binaryBitsCount) - 1);
//XORing the input value with the sequence of 1 bits
return n ^ oneBitsSequence;
}
I need an advice how to optimize above algorithm, especially the line for generating sequence of 1 bits complementer (oneBitsSequence), or if anyone can suggest better algorithm?
UPDATE: I also would like to know the known term of this non-leading zero complement?

You can get the highest one bit through the Integer.highestOneBit(i) method, shift this one step left, and then subtract 1. This gets you the correct length of 1s:
private static int nonLeadingZeroComplement(int i) {
int ones = (Integer.highestOneBit(i) << 1) - 1;
return i ^ ones;
}
For example,
System.out.println(nonLeadingZeroComplement(92));
prints
35

obviously #keppil has provided shortest solution. Another solution could be like.
private static int integerComplement(int n){
String binaryString = Integer.toBinaryString(n);
String temp = "";
for(char c: binaryString.toCharArray()){
if(c == '1'){
temp += "0";
}
else{
temp += "1";
}
}
int base = 2;
int complement = Integer.parseInt(temp, base);
return complement;
}
For example,
System.out.println(nonLeadingZeroComplement(92));
Prints answer as 35

Java negative BigInteger toString

It seems I have a two's complement issue with Java's BigInteger.
I have a 64-bit integer where only the msb and the second msb are set to 1, the rest is 0.
In decimal this comes up to: -4611686018427387904
The Java side of my application receives this decimal number as a string, and converts it to BigInteger like so:
BigInteger bi = new BigInteger("-4611686018427387904", 10);
Then, it needs to display this number both in binary and hex forms.
I tried to use:
String bin = bi.toString(2);
String hex = bi.toString(16);
but I'm getting:
-100000000000000000000000000000000000000000000000000000000000000
-4000000000000000
whereas I expect to get:
1100000000000000000000000000000000000000000000000000000000000000
c000000000000000
Any tips?

Number always fits in 64 bits:
If your number always fits in 64 bits you can put it in a long and then print the bits / hex digits.
long l = bi.longValue();
String bin = Long.toBinaryString(l);
String hex = Long.toHexString(l);
System.out.println(bin);
System.out.println(hex);
Number may not always fit in 64 bits:
If the number does not always fit in 64 bits, you'll have to solve it "manually". To convert a number to it's two's complement representation you do the following:
If number is positive, do nothing
If number is negative:
Convert it to its absolute value
Complement the bits
Add 1
For a BigInteger the conversion looks as follows:
if (bi.compareTo(BigInteger.ZERO) < 0)
bi = bi.abs().not().add(BigInteger.ONE);
If you print it using bi.toString(2) you'll still get the sign character, instead of a leading 1. This can be solved by simply appending .replace('-', '1') to the string.

There is a BigInteger.toByteArray() method, that returns two's complement representation of BigInteger as a byte[]. All you need is to print that array in hex or binary form:
byte[] bs = bi.toByteArray();
for (byte b: bs) {
System.out.print(String.format("%02X", 0xff & b));
}

The binary number 1100000000000000000000000000000000000000000000000000000000000000 is definitely a positive number, right. It's equal to 2^63 + 2^62.
I don't see why you'd expect a negative number to become positive when you convert to base 2 or base 16.
You are confusing the base n representation with the internal representation of numbers.

If the number is 64 bits or less, then the simple way to solve this is to convert to a long and then use Long.toHexString().

what you mean?
Do you want to get Two's complement？
if you mean that, maybe i can give you an example
import java.util.*;
public class TestBina{
static void printBinaryInt(int i){
System.out.println("int:"+i+",binary:");
System.out.print(" ");
for(int j=31;j>=0;j--)
if(((1<<j)&i)!=0)
System.out.print("1");
else
System.out.print("0");
System.out.println();
}
public static void main(String [] args){
Random rand = new Random();
int i = rand.nextInt();
int j = rand.nextInt();
printBinaryInt(i);
printBinaryInt(j);
printBinaryInt(10);
printBinaryInt(-10);
}
}

Could someone explain to me what the following Java code is doing?

byte s[] = getByteArray()
for(.....)
Integer.toHexString((0x000000ff & s[i]) | 0xffffff00).substring(6);
I understand that you are trying to convert the byte into hex string. What I don't understand is how that is done. For instance if s[i] was 00000001 (decimal 1) than could you please explain:
Why 0x000000ff & 00000001 ? Why not directly use 00000001?
Why result from #1 | 0xffffff00?
Finally why substring(6) is applied?
Thanks.

It's basically because bytes are signed in Java. If you promote a byte to an int, it will sign extend, meaning that the byte 0xf2 will become 0xfffffff2. Sign extension is a method to keep the value the same when widening it, by copying the most significant (sign) bit into all the higher-order bits. Both those values above are -14 in two's complement notation. If instead you had widened 0xf2 to 0x000000f2, it would be 242, probably not what you want.
So the & operation is to strip off any of those extended bits, leaving only the least significant 8 bits. However, since you're going to be forcing those bits to 1 in the next step anyway, this step seems a bit of a waste.
The | operation following that will force all those upper bits to be 1 so that you're guaranteed to get an 8-character string from ffffff00 through ffffffff inclusive (since toHexString doesn't give you leading zeroes, it would translate 7 into "7" rather than the "07" that you want).
The substring(6) is then applied so that you only get the last two of those eight hex digits.
It seems a very convoluted way of ensuring you get a two-character hex string to me when you can just use String.format ("%02x", s[i]). However, it's possible that this particular snippet of code may predate Java 5 when String.format was introduced.
If you run the following program:
public class testprog {
public static void compare (String s1, String s2) {
if (!s1.equals(s2))
System.out.println ("Different: " + s1 + " " + s2);
}
public static void main(String args[]) {
byte b = -128;
while (b < 127) {
compare (
Integer.toHexString((0x000000ff & b) | 0xffffff00).substring(6),
String.format("%02x", b, args));
b++;
}
compare (
Integer.toHexString((0x000000ff & b) | 0xffffff00).substring(6),
String.format("%02x", b, args));
System.out.println ("Done");
}
}
you'll see that the two expressions are identical - it just spits out Done since the two expressions produce the same result in all cases.

How to convert Java long's as Strings while keeping natural order

I'm currently looking at a simple programming problem that might be fun to optimize - at least for anybody who believes that programming is art :) So here is it:
How to best represent long's as Strings while keeping their natural order?
Additionally, the String representation should match ^[A-Za-z0-9]+$. (I'm not too strict here, but avoid using control characters or anything that might cause headaches with encodings, is illegal in XML, has line breaks, or similar characters that will certainly cause problems)
Here's a JUnit test case:
#Test
public void longConversion() {
final long[] longs = { Long.MIN_VALUE, Long.MAX_VALUE, -5664572164553633853L,
-8089688774612278460L, 7275969614015446693L, 6698053890185294393L,
734107703014507538L, -350843201400906614L, -4760869192643699168L,
-2113787362183747885L, -5933876587372268970L, -7214749093842310327L, };
// keep it reproducible
//Collections.shuffle(Arrays.asList(longs));
final String[] strings = new String[longs.length];
for (int i = 0; i < longs.length; i++) {
strings[i] = Converter.convertLong(longs[i]);
}
// Note: Comparator is not an option
Arrays.sort(longs);
Arrays.sort(strings);
final Pattern allowed = Pattern.compile("^[A-Za-z0-9]+$");
for (int i = 0; i < longs.length; i++) {
assertTrue("string: " + strings[i], allowed.matcher(strings[i]).matches());
assertEquals("string: " + strings[i], longs[i], Converter.parseLong(strings[i]));
}
}
and here are the methods I'm looking for
public static class Converter {
public static String convertLong(final long value) {
// TODO
}
public static long parseLong(final String value) {
// TODO
}
}
I already have some ideas on how to approach this problem. Still, I though I might get some nice (creative) suggestions from the community.
Additionally, it would be nice if this conversion would be
as short as possible
easy to implement in other languages
EDIT: I'm quite glad to see that two very reputable programmers ran into the same problem as I did: using '-' for negative numbers can't work as the '-' doesn't reverse the order of sorting:
-0001
-0002
0000
0001
0002

Ok, take two:
class Converter {
public static String convertLong(final long value) {
return String.format("%016x", value - Long.MIN_VALUE);
}
public static long parseLong(final String value) {
String first = value.substring(0, 8);
String second = value.substring(8);
long temp = (Long.parseLong(first, 16) << 32) | Long.parseLong(second, 16);
return temp + Long.MIN_VALUE;
}
}
This one takes a little explanation. Firstly, let me demonstrate that it is reversible and the resultant conversions should demonstrate the ordering:
for (long aLong : longs) {
String out = Converter.convertLong(aLong);
System.out.printf("%20d %16s %20d\n", aLong, out, Converter.parseLong(out));
}
Output:
-9223372036854775808 0000000000000000 -9223372036854775808
9223372036854775807 ffffffffffffffff 9223372036854775807
-5664572164553633853 316365a0e7370fc3 -5664572164553633853
-8089688774612278460 0fbba6eba5c52344 -8089688774612278460
7275969614015446693 e4f96fd06fed3ea5 7275969614015446693
6698053890185294393 dcf444867aeaf239 6698053890185294393
734107703014507538 8a301311010ec412 734107703014507538
-350843201400906614 7b218df798a35c8a -350843201400906614
-4760869192643699168 3dedfeb1865f1e20 -4760869192643699168
-2113787362183747885 62aa5197ea53e6d3 -2113787362183747885
-5933876587372268970 2da6a2aeccab3256 -5933876587372268970
-7214749093842310327 1be00fecadf52b49 -7214749093842310327
As you can see Long.MIN_VALUE and Long.MAX_VALUE (the first two rows) are correct and the other values basically fall in line.
What is this doing?
Assuming signed byte values you have:
-128 => 0x80
-1 => 0xFF
0 => 0x00
1 => 0x01
127 => 0x7F
Now if you add 0x80 to those values you get:
-128 => 0x00
-1 => 0x7F
0 => 0x80
1 => 0x81
127 => 0xFF
which is the correct order (with overflow).
Basically the above is doing that with 64 bit signed longs instead of 8 bit signed bytes.
The conversion back is a little more roundabout. You might think you can use:
return Long.parseLong(value, 16);
but you can't. Pass in 16 f's to that function (-1) and it will throw an exception. It seems to be treating that as an unsigned hex value, which long cannot accommodate. So instead I split it in half and parse each piece, combining them together, left-shifting the first half by 32 bits.

EDIT: Okay, so just adding the negative sign for negative numbers doesn't work... but you could convert the value into an effectively "unsigned" long such that Long.MIN_VALUE maps to "0000000000000000", and Long.MAX_VALUE maps to "FFFFFFFFFFFFFFFF". Harder to read, but will get the right results.
Basically you just need to add 2^63 to the value before turning it into hex - but that may be a slight pain to do in Java due to it not having unsigned longs... it may be easiest to do using BigInteger:
private static final BigInteger OFFSET = BigInteger.valueOf(Long.MIN_VALUE)
.negate();
public static String convertLong(long value) {
BigInteger afterOffset = BigInteger.valueOf(value).add(OFFSET);
return String.format("%016x", afterOffset);
}
public static long parseLong(String text) {
BigInteger beforeOffset = new BigInteger(text, 16);
return beforeOffset.subtract(OFFSET).longValue();
}
That wouldn't be terribly efficient, admittedly, but it works with all your test cases.

If you don't need a printable String, you can encode the long in four chars after you've shifted the value by Long.MIN_VALUE (-0x80000000) to emulate an unsigned long:
public static String convertLong(long value) {
value += Long.MIN_VALUE;
return "" +
(char)(value>>48) + (char)(value>>32) +
(char)(value>>16) + (char)value;
}
public static long parseLong(String value) {
return (
(((long)value.charAt(0))<<48) +
(((long)value.charAt(1))<<32) +
(((long)value.charAt(2))<<16) +
(long)value.charAt(3)) + Long.MIN_VALUE;
}
Usage of surrogate pairs is not a problem, since the natural order of a string is defined by the UTF-16 values in its chars and not by the UCS-2 codepoint values.

There's a technique in RFC2550 -- an April 1st joke RFC about the Y10K problem with 4-digit dates -- that could be applied to this purpose. Essentially, each time the integer's string representation grows to require another digit, another letter or other (printable) character is prepended to retain desired sort-order. The negative rules are more arcane, yielding strings that are harder to read at a glance... but still easy enough to apply in code.
Nicely, for positive numbers, they're still readable.
See:
http://www.faqs.org/rfcs/rfc2550.html