Develop Reference

Unit test using HBaseTestingUtility

I am trying to debug the java code using HBaseTestingUtility library. I already have table created. I need to:
- Insert a value with a key in "myTable"
- Get the value from "myTable" with the key
- Verify the returned value is equal to the value I created
Here is the code that I filled out:
package HbaseUniteTest;
import jdk.nashorn.api.scripting.ScriptUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.*;
import org.apache.hadoop.hbase.client.*;
import org.apache.hadoop.hbase.io.compress.Compression;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.junit.Assert;
import static org.junit.Assert.assertEquals;
public class TestCreateTableClass
{
private final static String tableName = "myTable";
private static ScriptUtils HTableUtil;
public static void main( String[] args ) throws Exception {
//Start the "mini cluster"
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster();
//Get the configuration
//Configuration conf = ...
Configuration conf = testingUtility.getConfiguration();
//Instantiate a connection
Connection connection = ConnectionFactory.createConnection(conf);
//Define table "myTable"
HTableDescriptor table = new HTableDescriptor(TableName.valueOf(tableName));
table.addFamily(new HColumnDescriptor("cf1").setCompressionType(Compression.Algorithm.NONE));
//Create table "myTable"
connection.getAdmin().createTable(table);
//Get the first (and only) table name
String first_table = connection.getAdmin().listTableNames()[0].getNameAsString();
//Verify the returned Table name is equal to the table name we provided
assertEquals(tableName,first_table);
//Insert a value with a key in "myTable"
byte[] key = Bytes.toBytes("some-key");
Put put = new Put(key);
put.add(Bytes.toBytes("colfam1"), Bytes.toBytes("qual1.1"), System.currentTimeMillis(), Bytes.toBytes("val1.1"));
put.add(Bytes.toBytes("colfam1"), Bytes.toBytes("qual1.2"), System.currentTimeMillis(), Bytes.toBytes("val1.2"));
put.add(Bytes.toBytes("colfam2"), Bytes.toBytes("qual2.1"), System.currentTimeMillis(), Bytes.toBytes("val2.1"));
Result converted = HTableUtil.convert(put);
table.put(put);
Result readFromTable = table.get(new Get(key));
Assert.assertArrayEquals(readFromTable.raw(), converted.raw());
//Get the value from "myTable" with the key
//Verify the returned value is equal to the value you created
//Stop the mini cluster
testingUtility.shutdownMiniCluster();
System.out.println("END OF TEST");
}
public static void setHTableUtil(ScriptUtils HTableUtil) {
TestCreateTableClass.HTableUtil = HTableUtil;
}
}
However, I got the following error:
1. The error at this line of code with the function put.add()
put.add(Bytes.toBytes("colfam1"), Bytes.toBytes("qual1.1"), System.currentTimeMillis(), Bytes.toBytes("val1.1"));
The 2nd error on this line of code:
Result converted = HTableUtil.convert(put);
Java cannot find symbol for these 3 methods put(), get(), raw()
table.put(put);
Result readFromTable = table.get(new Get(key));
Assert.assertArrayEquals(readFromTable.raw(), converted.raw());
I also notice some warnings regarding the class HTableDescriptor, HColumnDescriptor have been deprecated. I checked on internet and they advice to use for example "TableDescriptorBuilder" instead but I am not sure how to use it. (Ref: https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java)

1. The error at this line of code with the function put.add().
I think you can use addColumn() like this for adding column.
put.addColumn(Bytes.toBytes("colfam1"), Bytes.toBytes("qual1.1"), System.currentTimeMillis(), Bytes.toBytes("val1.1"));
put.addColumn(Bytes.toBytes("colfam1"), Bytes.toBytes("qual1.2"), System.currentTimeMillis(), Bytes.toBytes("val1.2"));
put.addColumn(Bytes.toBytes("colfam2"), Bytes.toBytes("qual2.1"), System.currentTimeMillis(), Bytes.toBytes("val2.1"));
2. The 2nd error on this line of code:
I'm not familiar with 'ScriptUtils', But I think It works.
Result converted = (Result) HTableUtil.convert(put, Result.class);
3. Java cannot find symbol for these 3 methods put(), get(), raw()
It because you keep using 'HTableDescriptor' to put(), get(), or raw(). 'HTableDescriptor' is used to create table like DDL. You need to use Table class to manipulate using put(), get(), or raw().
Table createdTable = connection.getTable(TableName.valueOf(tableName));
createdTable.put(put);
Result readFromTable = createdTable.get(new Get(key));
Also, I believe class 'Result' doesn't provide raw(). So, you can compare both Results using Result.compareResults() like this.
Result.compareResults(readFromTable, converted);
4. How to use 'TableDescriptorBuilder'
Like I said above, 'Descriptor' is the class for defining your table, column family, column, and so on. So, you need to use it when you make/create them.
//Define table "myTable"
TableDescriptorBuilder table = TableDescriptorBuilder.newBuilder(TableName.valueOf(tableName));
table.setColumnFamily(ColumnFamilyDescriptorBuilder.newBuilder(Bytes.toBytes("cf1")).setCompressionType(Compression.Algorithm.NONE).build());
//Create table "myTable"
connection.getAdmin().createTable(table.build());

How to encode a string to base64 encoding algorithm using PowerBuilder

Requirement:
Given a String
we need to generate Base64 encoded string using the above given string.
How can we implement it using powerBuilder.
For reference, The Java implementation of the above case is as follows:
import org.apache.tomcat.util.codec.binary.Base64;
import java.io.UnsupportedEncodingException;
public String getClientEncoded() throws UnsupportedEncodingException {
String givenString= "Input_String";
String bytesEncoded = Base64.encodeBase64String(valueToHash.getBytes("UTF-8"));
System.out.println("encoded value is " + bytesEncoded);
return bytesEncoded ;
}
============================================================
As per Matt's reply, Used this below code from the 1st link:
String ls_valueToBeEncoded
blob lblob
ls_valueToBeEncoded = "realhowto"
lblob = Blob(ls_valueToBeEncoded)
ULong lul_len, lul_buflen
Boolean lb_rtn
lul_len = Len(ablob_data)
lul_buflen = lul_len * 2
ls_encoded = Space(lul_buflen)
lb_rtn = CryptBinaryToString(ablob_data, &
lul_len, CRYPT_STRING_BASE64, &
ref ls_encoded, lul_buflen) // Used ref ls_encoded to get the string. Otherwise, junk characters gets stored in ls_encoded.`
=======================================
Used the below code in Global External Function:
`FUNCTION boolean CryptBinaryToString ( &
Blob pbBinary, &
ulong cbBinary, &
ulong dwFlags, &
Ref string pszString, &
Ref ulong pcchString ) &
LIBRARY "crypt32.dll" ALIAS FOR "CryptBinaryToStringA;Ansi"`
=========================================
According to the 1st link suggested by Matt, The string "realhowto" should be converted to "cmVhbGhvd3Rv."
But when I tried the above code, I got "cgBlAGEAbABoAG8AdwB0AG8A"
Any advise will be appreciated.

Check out this link
Make sure you look at the comments as well.
Another option is here.
Real's How To is a very good reference for many PowerBuilder tips.

My encryption examples also have the API functions used by Real's example.
http://www.topwizprogramming.com/freecode_bcrypt.html

Checked with the 1st link given by Matt.
The solution worked.
Only thing I was missing earlier was while converting the original string to blob, we need to convert via the following:
String ls_original_string
Blob lblob_test
lblob_test = Blob(ls_original_string, EncodingAnsi!)
The blob lblob_test can be passed as an argument to the function CryptBinaryToString(...)

Generating a segwit address and private key with bitcoinj (paper wallet)

It is possible to generate a valid legacy bitcoin key pair with the following code which is using bitcoinj master branch:
import org.bitcoinj.core.Address;
import org.bitcoinj.core.DumpedPrivateKey;
import org.bitcoinj.core.ECKey;
import org.bitcoinj.core.NetworkParameters;
public class GeneratePrivateKeyBulk {
public static void main(String[] args) {
ECKey key = new ECKey();
Address pubAddress = new Address(NetworkParameters.prodNet(), key.getPubKeyHash());
DumpedPrivateKey privKey = key.getPrivateKeyEncoded(NetworkParameters.prodNet());
System.out.println("Public address: " + pubAddress.toBase58() + "; Private key: " + privKey.toBase58());
}
}
This creates a usable legacy base58 encoded public address and private key e.g. like 1ERzRYYdbibaQt2kuNfgH8spuoqQxYkwQb, L3AuZ2vNt11ac2xSi6AYwzXyftqSVPcSuHNdTsSuRfknXvoRtWzF correspondingly.
The question is how can I do the same operation to obtain a segwit key pair?
I looked at the bitcoinj docs but could not find any API for generating addresses directly as segwit.
By looking at the tests and the segwit pull request I found that the following code (appended to the code above) would produce a segwit address (i.e. one that starts with 3, e.g. 31uLnxKteEYa2u1vgWyVPkTpVfUGduCV82)
Script script = ScriptBuilder.createP2SHOutputScript(1, Collections.singletonList(key));
Address segwitAddress = Address.fromP2SHScript(NetworkParameters.prodNet(), script);
System.out.println("Segwit address: " + segwitAddress.toBase58());
My understanding is that the code above is supposed to be used in a multisig scenario, therefore I am not sure if this is the correct way to derive a segwit address from a single private key. Is this the correct/reliable/safe code for generating a paper segwit wallet?
Also, is there a way to add BIP38 password protection to the private key using bitcoinj? The class BIP38PrivateKey only has methods for decrypting a BIP38 key from an existing base58 representation, but no methods for BIP38 password encryption.

Accessing AWS parameter store values with custom KMS key

I am trying to read AWS parameters from the parameter store using java, i have created the parameters using a custom encryption key. I dont see a sample code in the internet where its using a custom KMS key , the below is the code i currently have which is working (here we are usingthe default KMS key).
AWSSimpleSystemsManagement client= AWSSimpleSystemsManagementClientBuilder.defaultClient();
GetParametersRequest request= new GetParametersRequest();
request.withNames("test.username","test.password")
.setWithDecryption(true);
This will give the results with default KMS key
Does anyone know how to handle this if we have a custom KMS key

just in case, if somebody looking for this (with Default encryption Key)
protected Parameter getParameterFromSSMByName(String parameterKey)
{
AWSCredentialsProvider credentials = InstanceProfileCredentialsProvider.getInstance();
AWSSimpleSystemsManagement simpleSystemsManagementClient = (AWSSimpleSystemsManagement)((AWSSimpleSystemsManagementClientBuilder)((AWSSimpleSystemsManagementClientBuilder)AWSSimpleSystemsManagementClientBuilder.standard().withCredentials(credentials)).withRegion("us-east-1")).build();
GetParameterRequest parameterRequest = new GetParameterRequest();
parameterRequest.withName(parameterKey).setWithDecryption(Boolean.valueOf(true));
GetParameterResult parameterResult = simpleSystemsManagementClient.getParameter(parameterRequest);
return parameterResult.getParameter();
}

Here is #Extreme's answer as a class with imports and a bit of cleanup:
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.auth.InstanceProfileCredentialsProvider;
import com.amazonaws.services.simplesystemsmanagement.AWSSimpleSystemsManagement;
import com.amazonaws.services.simplesystemsmanagement.AWSSimpleSystemsManagementClientBuilder;
import com.amazonaws.services.simplesystemsmanagement.model.GetParameterRequest;
import com.amazonaws.services.simplesystemsmanagement.model.GetParameterResult;
public class AWSSsmHelper
{
private AWSCredentialsProvider credentials = InstanceProfileCredentialsProvider.getInstance();
private AWSSimpleSystemsManagement simpleSystemsManagementClient =
AWSSimpleSystemsManagementClientBuilder.standard().withCredentials(credentials)).withRegion("us-east-1")).build();
public String getParameterFromSSMByName(String parameterKey) {
GetParameterRequest parameterRequest = new GetParameterRequest();
parameterRequest.withName(parameterKey).setWithDecryption(Boolean.valueOf(true));
GetParameterResult parameterResult = simpleSystemsManagementClient.getParameter(parameterRequest);
return parameterResult.getParameter().getValue();
}
}

For GetParameters API, there's no difference between use default KMS key or custom KMS key. It always works like your code. Just make sure the permission for the credential includes the custom key.
The difference only at PutParameter API, when using a default KMS key, you don't need to specify it, when using a custom KMS key, you set its KeyId to the custom key. The KeyId can be one of following examples:
Key ARN Example arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012
Alias ARN Example - arn:aws:kms:us-east-1:123456789012:alias/MyAliasName
Globally Unique Key ID Example - 12345678-1234-1234-1234-123456789012
Alias Name Example - alias/MyAliasName

How to stock and use a shiro's salt from database

I use shiro in application for the authenticate. I use hashed password with a salt and I store them in my database like this :
private User createUserWithHashedPassword(String inName, String inFirstName, String inLastName, String inPassword){
ByteSource salt = randomNumberGenerator.nextBytes(32);
byte[] byteTabSalt = salt.getBytes();
String strSalt = byteArrayToHexString(byteTabSalt);
String hashedPasswordBase64 = new Sha256Hash(inPassword, salt, 1024).toBase64();
return new User(inName,inFirstName,inLastName,hashedPasswordBase64,strSalt);
}
I store the salt with a String in my database. Now in my realm I want to get back my datas from the database, I use a transactionnal service for this. But my salt is a Strong so I want it to turn back as ByteSource type with the static method :
ByteSource byteSourceSalt = Util.bytes(salt); //where the salt is a String
But when I create my SaltedAuthenticationInfo it doesn't auth.
I think my problem is from my convert method :
private String byteArrayToHexString(byte[] bArray){
StringBuffer buffer = new StringBuffer();
for(byte b : bArray) {
buffer.append(Integer.toHexString(b));
buffer.append(" ");
}
return buffer.toString().toUpperCase();
}
Thanks for your help.

As mentioned in the excellent answer https://stackoverflow.com/a/20206115/603901, Shiro's DefaultPasswordService already generates unique salts for each password.
However, there is no need to implement a custom PasswordService to add a private salt (sometimes called "pepper") to the per-user salts. Private salt can be configured in shiro.ini:
[main]
hashService = org.apache.shiro.crypto.hash.DefaultHashService
hashService.hashIterations = 500000
hashService.hashAlgorithmName = SHA-256
hashService.generatePublicSalt = true
# privateSalt needs to be base64-encoded in shiro.ini but not in the Java code
hashService.privateSalt = myVERYSECRETBase64EncodedSalt
passwordMatcher = org.apache.shiro.authc.credential.PasswordMatcher
passwordService = org.apache.shiro.authc.credential.DefaultPasswordService
passwordService.hashService = $hashService
passwordMatcher.passwordService = $passwordService
Java code for generating a matching password hash:
DefaultHashService hashService = new DefaultHashService();
hashService.setHashIterations(HASH_ITERATIONS); // 500000
hashService.setHashAlgorithmName(Sha256Hash.ALGORITHM_NAME);
hashService.setPrivateSalt(new SimpleByteSource(PRIVATE_SALT)); // Same salt as in shiro.ini, but NOT base64-encoded.
hashService.setGeneratePublicSalt(true);
DefaultPasswordService passwordService = new DefaultPasswordService();
passwordService.setHashService(hashService);
String encryptedPassword = passwordService.encryptPassword("PasswordForThisUser");
The resulting hash looks like this:
$shiro1$SHA-256$500000$An4HRyqMJlZ58utACtyGDQ==$nKbIY9Nd9vC89G4SjdnDfka49mZiesjWgDsO/4Ly4Qs=
The private salt is not stored in the database, which makes it harder to crack the passwords if an adversary gains access to a database dump.
This example was created using shiro-1.2.2
Thanks to https://github.com/Multifarious/shiro-jdbi-realm/blob/master/src/test/resources/shiro.ini for help with the syntax for shiro.ini

Have you looked at PasswordMatcher / PasswordService?
This already has all of the encoding/decoding/compare logic built-in. To use it:
Storing password in database:
PasswordService service = new DefaultPasswordService(); // or use injection or shiro.ini to populate this
private User createUserWithHashedPassword(String inName, String inFirstName, String inLastName, String inPassword){
String hashedPasswordBase64 = service.encryptPassword(inPassword);
return new User(inName,inFirstName,inLastName,hashedPasswordBase64,strSalt);
}
Then you can simply use PasswordMatcher as the matcher in your realm.
realm.setCredentialsMatcher(new PasswordMatcher());
or in shiro.ini:
matcher = org.apache.shiro.authc.credential.PasswordMatcher
realm.credentialsMatcher = $matcher

The DefaultPasswordService implementation automatically adds a random salt to each encryptPassword call. That "public" salt will be stored within the "hashedPasswordBase64" that you receive from "encryptPassword".
Because the "public" salt is individually generated for each hashed password one cannot "simply" generate a rainbow table and brute-force all your hashed passwords at once. For each hashed password the attacker would have to generate an own, unique rainbow table because of the unique "public" salt. So far you do not need to put an extra salt into the database.
To make your stored hashed passwords even more secure you can furthermore add a "private" salt that should be stored anywhere else - as long as not in the database. By using a "private" salt you could protect the hashed passwords against a brute-force rainbow-table attack, because the attacker does not know the "private" salt and cannot gain the "private" salt from the database entries.
This is a very basic example how to create a PasswordService that utilizes a "private" salt provided as a constant string and that works as CredentialsMatcher:
public class MyPrivateSaltingPasswortService extends DefaultPasswordService
{
public MyPrivateSaltingPasswortService()
{
super();
HashService service = getHashService();
if (service instanceof DefaultHashService)
{
((DefaultHashService) service).setPrivateSalt(
new SimpleByteSource("MySuperSecretPrivateSalt"));
}
}
}
you then could use your own implementation in shiro.ini:
[main]
saltedService = com.mycompany.MyPrivateSaltingPasswortService
matcher = org.apache.shiro.authc.credential.PasswordMatcher
matcher.passwordService = $saltedService
realm.credentialsMatcher = $matcher
This example was created using shiro-1.2.2

I change my type for the save of my salt. Now I'm using a byte[] instead of a String.
ByteSource salt = randomNumberGenerator.nextBytes(32);
byte[] byteTabSalt = salt.getBytes();
And I stock the byteTabSalt in my database.