Develop Reference

My java Base64 results are not the same as php

I have previously posted a question related to this work link but I am posting a new one because the question is not completely resolved
I am working on converting the completed code into java with php.
It is a function that reads encrypted files, decrypts them by 16 bytes, makes them into a single string, and encodes them with base 64.
php is already on the server and running, and I have to use java to produce the same result.
If you decrypt the read file,
<?xml version="1.0" encoding="utf-8" ?>
<FileInfo>
...
<TextData> (text data) </TextData>
</FileInfo>
(image data)
It is in the format, and the text data in shows php and java exactly the same.
I am trying to encode the image data part into base64, but the result is different from php.
This is the part of the php code I have that decrypts and processes the read file
$fileContentArray = array(16);
$transContentArray = array(16);
$fileRead = fread($fp,16);
for($i = 0 ; $i < strlen($fileRead); $i++){
$fileContentArray[$i] = ord($fileRead[$i]);
}
$seed->SeedDecrypt($fileContentArray,$pdwRoundKey,$transContentArray);
$transStr = call_user_func_array("pack",array_merge(array("C16"),$transContentArray));
$mergeStr .= $transStr;
}
$dataExplode = explode("<TextData>",trim($mergeStr) );
$dataExplode1 = explode("</FileInfo>",trim($dataExplode[1]) );
$dataExplode2 = explode("</TextData>",$dataExplode1[0]);
$textData = iconv("EUC-KR","utf-8",$dataExplode2[0]);
$imageData = base64_encode(trim($dataExplode1[1]));
And this is the same part of the java code I wrote
byte[] fileContentArray=new byte[n];
for(int i=0;i<fileContentArray.length;i++){
fileContentArray[i]=mergeArr[nReadCur+i];
}
seed.SeedDecrypt(fileContentArray, pdwRoundKey, outbuf);
System.arraycopy(outbuf, 0, resultArr, nReadCur, outbuf.length);
nReadCur=nReadCur+fileContentArray.length;
p=p+fileContentArray.length;
if(p>=nFileSize){
fis.close();
break;
}
}//while
mergeStr=new String(resultArr,"MS949");
String[] dataExplode=mergeStr.trim().split("<TextData>");
String[] dataExplode1=dataExplode[1].trim().split("</FileInfo>");
String[] dataExplode2=dataExplode1[0].trim().split("</TextData>");
String textData = "";
String imageData = "";
textData=dataExplode2[0];
imageData=dataExplode1[1];
Encoder encoder=Base64.getEncoder();
Decoder decoder=Base64.getDecoder();
byte[] encArr=encoder.encode(imageData.trim().getBytes("MS949"));
imageData=new String(encArr,"MS949");
As a result of encoding image data into base64
php: R0lGODlhAwNLBPcAAAAAAAAAMwAAZgAAmQAAzAAA/wArAAArMwArZgArmQArzAAr/wBVAABVMwBVZgBVmQBVzABV/wCAAACAMwCAZgCAmQCAzACA/ ... VzpYirO0le55zF0=
java: R0lGODlhAwNLBD8AAAAAAAAAMwAAZgAAPwAAPwAAPwArAAArMwArZgArPwArPwArPwBVAABVMwBVZgBVPwBVPwBVPwA/AAA/MwA/ZgA/PwA/PwA/PwA/ ... DAQEAOz9GPz8/IXY=
As you can see, the result values are output differently.
Is there anything I'm missing? What should I do to make the result of java the same as php?
Also, MergeStr, who has read the file,
java:
GIF89aK? 3 f ? ? ? + +3 +f +? +? +? U U3 Uf U? U? U? ? ?3 ?f ?? ?? ?? ? ?3 챖 첌 ぬ ? ? ?3 ?f ??
...
J뇽杞H?*]苛⒢쬝쥻쒳뎁諾X...
A?h?~?0a?2$ #삁?d?Dd??e ...
...
WC ;홃?뿿!v
php:
GIF89aK? 3 f ? ?  + +3 +f +? +? + U U3 Uf U? U? U 3 f ? ?  ? ? 챖 첌 ぬ ? ? ? ? ? 螂 ?  3 f ? ? ...
A??~?a?$ #삁?d?Dd?e...
...
WC ;홃??v余퍙W:X뒽킉??
Like this, there is a new line text that I didn't put in, and there's a slight difference in result. Is this a simple difference in encoding? And does this affect the base64 conversion?
I tried encoding with UTF-8 and failed again,
and I used this code to load all bytes of the file at once
FileInputStream fis = new FileInputStream(tpf);
fis.read(mergeArr);

Java uses a little bit different encoding for base64 than PHP. You may use these helper functions to make them compatible to the Java version.
function base64url_encode($data)
{
return rtrim(strtr(base64_encode($data), '+/', '-_'), '=');
}
function base64url_decode($data, $strict = false)
{
return base64_decode(strtr($data, '-_', '+/'), $strict);
}

How to encode a string to base64 encoding algorithm using PowerBuilder

Requirement:
Given a String
we need to generate Base64 encoded string using the above given string.
How can we implement it using powerBuilder.
For reference, The Java implementation of the above case is as follows:
import org.apache.tomcat.util.codec.binary.Base64;
import java.io.UnsupportedEncodingException;
public String getClientEncoded() throws UnsupportedEncodingException {
String givenString= "Input_String";
String bytesEncoded = Base64.encodeBase64String(valueToHash.getBytes("UTF-8"));
System.out.println("encoded value is " + bytesEncoded);
return bytesEncoded ;
}
============================================================
As per Matt's reply, Used this below code from the 1st link:
String ls_valueToBeEncoded
blob lblob
ls_valueToBeEncoded = "realhowto"
lblob = Blob(ls_valueToBeEncoded)
ULong lul_len, lul_buflen
Boolean lb_rtn
lul_len = Len(ablob_data)
lul_buflen = lul_len * 2
ls_encoded = Space(lul_buflen)
lb_rtn = CryptBinaryToString(ablob_data, &
lul_len, CRYPT_STRING_BASE64, &
ref ls_encoded, lul_buflen) // Used ref ls_encoded to get the string. Otherwise, junk characters gets stored in ls_encoded.`
=======================================
Used the below code in Global External Function:
`FUNCTION boolean CryptBinaryToString ( &
Blob pbBinary, &
ulong cbBinary, &
ulong dwFlags, &
Ref string pszString, &
Ref ulong pcchString ) &
LIBRARY "crypt32.dll" ALIAS FOR "CryptBinaryToStringA;Ansi"`
=========================================
According to the 1st link suggested by Matt, The string "realhowto" should be converted to "cmVhbGhvd3Rv."
But when I tried the above code, I got "cgBlAGEAbABoAG8AdwB0AG8A"
Any advise will be appreciated.

Check out this link
Make sure you look at the comments as well.
Another option is here.
Real's How To is a very good reference for many PowerBuilder tips.

My encryption examples also have the API functions used by Real's example.
http://www.topwizprogramming.com/freecode_bcrypt.html

Checked with the 1st link given by Matt.
The solution worked.
Only thing I was missing earlier was while converting the original string to blob, we need to convert via the following:
String ls_original_string
Blob lblob_test
lblob_test = Blob(ls_original_string, EncodingAnsi!)
The blob lblob_test can be passed as an argument to the function CryptBinaryToString(...)

Converting CSV file to LIBSVM compatible data file using python

I am doing a project using libsvm and I am preparing my data to use the lib. How can I convert CSV file to LIBSVM compatible data?
CSV File:
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/datasets/data/iris.csv
In the frequencies questions:
How to convert other data formats to LIBSVM format?
It depends on your data format. A simple way is to use libsvmwrite in the libsvm matlab/octave interface. Take a CSV (comma-separated values) file in UCI machine learning repository as an example. We download SPECTF.train. Labels are in the first column. The following steps produce a file in the libsvm format.
matlab> SPECTF = csvread('SPECTF.train'); % read a csv file
matlab> labels = SPECTF(:, 1); % labels from the 1st column
matlab> features = SPECTF(:, 2:end);
matlab> features_sparse = sparse(features); % features must be in a sparse matrix
matlab> libsvmwrite('SPECTFlibsvm.train', labels, features_sparse);
The tranformed data are stored in SPECTFlibsvm.train.
Alternatively, you can use convert.c to convert CSV format to libsvm format.
but I don't wanna use matlab, I use python.
I found this solution as well using JAVA
Can anyone recommend a way to tackle this problem ?

You can use csv2libsvm.py to convert csv to libsvm data
python csv2libsvm.py iris.csv libsvm.data 4 True
where 4 means target index, and True means csv has a header.
Finally, you can get libsvm.data as
0 1:5.1 2:3.5 3:1.4 4:0.2
0 1:4.9 2:3.0 3:1.4 4:0.2
0 1:4.7 2:3.2 3:1.3 4:0.2
0 1:4.6 2:3.1 3:1.5 4:0.2
...
from iris.csv
150,4,setosa,versicolor,virginica
5.1,3.5,1.4,0.2,0
4.9,3.0,1.4,0.2,0
4.7,3.2,1.3,0.2,0
4.6,3.1,1.5,0.2,0
...

csv2libsvm.py does not work with Python3, and also it does not support label targets (string targets), I have slightly modified it. Now It should work with Python3 as well as wıth the label targets.
I am very new to Python, so my code may do not follow the best practices, but I hope it is good enough to help someone.
#!/usr/bin/env python
"""
Convert CSV file to libsvm format. Works only with numeric variables.
Put -1 as label index (argv[3]) if there are no labels in your file.
Expecting no headers. If present, headers can be skipped with argv[4] == 1.
"""
import sys
import csv
import operator
from collections import defaultdict
def construct_line(label, line, labels_dict):
new_line = []
if label.isnumeric():
if float(label) == 0.0:
label = "0"
else:
if label in labels_dict:
new_line.append(labels_dict.get(label))
else:
label_id = str(len(labels_dict))
labels_dict[label] = label_id
new_line.append(label_id)
for i, item in enumerate(line):
if item == '' or float(item) == 0.0:
continue
elif item=='NaN':
item="0.0"
new_item = "%s:%s" % (i + 1, item)
new_line.append(new_item)
new_line = " ".join(new_line)
new_line += "\n"
return new_line
# ---
input_file = sys.argv[1]
try:
output_file = sys.argv[2]
except IndexError:
output_file = input_file+".out"
try:
label_index = int( sys.argv[3] )
except IndexError:
label_index = 0
try:
skip_headers = sys.argv[4]
except IndexError:
skip_headers = 0
i = open(input_file, 'rt')
o = open(output_file, 'wb')
reader = csv.reader(i)
if skip_headers:
headers = reader.__next__()
labels_dict = {}
for line in reader:
if label_index == -1:
label = '1'
else:
label = line.pop(label_index)
new_line = construct_line(label, line, labels_dict)
o.write(new_line.encode('utf-8'))

Rewriting Java BigInteger function in python

I'm trying to rewrite some javacode in a python script. One part of that is to deduce a simple number from a sha256 hash.
in java this function is called:
public static Long getId(byte[] publicKey) {
byte[] publicKeyHash = Crypto.sha256().digest(publicKey);
BigInteger bigInteger = new BigInteger(1, new byte[] {publicKeyHash[7], publicKeyHash[6], publicKeyHash[5],
publicKeyHash[4], publicKeyHash[3], publicKeyHash[2], publicKeyHash[1], publicKeyHash[0]});
return bigInteger.longValue();
}
The publicKey is binairy so I can't print it here, but the publicKeyHash I use for testing is: d9d5c57971eefb085e3abaf7a5a4a6cdb8185f30105583cdb09ad8f61886ec65
To my understandin the third line of this Java code converts d9d5c579 a number.
The number that belongs to the hash above is 4273301882745002507
Now I'm looking for a piece / line of python code to generate that same number from that hash.
def getId(publicKey):
publicKeyHash = binascii.hexlify(publicKey)
p = publicKeyHash
return(struct.unpack("Q",struct.pack("cccccccc",p[7],p[6],p[5],p[4],p[3],p[2],p[1],p[0]))[0])
Was a first attempt however this clearly doesn't work, it does return a number but not the correct one.
Is there anyone here familiar with both languages and able to help my translate this function?

How about (untested):
import hashlib
publicKeyHash = hashlib.sha256.digest(publicKey)
bigInt = 0
for byte in publicKeyHash[:7]:
bigInt <<= 8
bigInt |= byte
This worked:
from hashlib import sha256
import json
import struct
import binascii
def getId(publicKey):
publicKeyHash = sha256(publicKey)
p = publicKeyHash.digest()
b = bytearray(p[:8])
b.reverse()
bigInt = 0
for byte in b:
bigInt <<= 8
bigInt |= byte
#print bigInt
return(bigInt)

deserialize java object from a blob

first of all, i apologize, i'm about to ask a set of dumb questions. i don't know java AT ALL and i don't know if we are allowed to ask questions like these.
if not - delete my topic.
there's a table in oracle that stores a blob. it's binary and i'm able to decode it, the output looks like this
¬í sr /com.epam.insure.credentialing.forms.StorageBeanÀÓ ¯w/§ L variablest Ljava/util/Map;xpsr java.util.HashMapÚÁÃ`Ñ F
loadFactorI thresholdxp?# w t $_hasCompletedt t
$_wf_progresssr java.lang.Integerâ ¤÷‡8 I valuexr java.lang.Number†¬•”à‹ xp t $_wf_statussq ~ t $_form_instance_idsr java.lang.Long;‹äÌ#ß J valuexq ~ ‹©t $_isVisitedt truet 1sq ~ sq ~ ?# `w € _t confidential readable infot 1t confidential readable infot $_errorssr java.util.ArrayListxÒ™Ça I sizexp w
xt regionIdsq ~ ët
confidential readable infot t t $_subbean_errorssq ~ w
xt regiont SOUTHWESTt idt t codet t reqTypeNamet
confidential readable infot t confidential readable infot tint t $_hasCompletedt falset comRequiredt t
lineImpactq ~ t prChiropractorsq ~ t fromTypeReqt not zipt 342t changeToTypeReq6t confidential readable infot t
prPodiatristsq ~ t
$_isValidatedt truet $_hasErrorsq ~ -t EVPapprovalsq ~ sq ~ ?# w Approvedq ~ Ct
NEGOTIATORq ~ Et
Negotiatort datet
03/31/2006q ~ It confidential readable infot q ~ \xt updateRequiredt noq ~ t truet approverssr .forms.StorageBeanList«WtúœG xq ~ w
q ~ Rsq ~ sq ~ ?# w t commentst t decisiont Approvedq ~ Ct RVPq ~ Et RVPt datet
04/04/2006q ~ It t commentst t decisiont Approvedq ~ Ct COOq ~ Et COOt datet
04/14/2006q ~ It ~ †xsq ~ sq ~ ?# w t commentsq ~ Pt decisiont Approvedq ~ Ct CEOq ~ Et CEOt d
so here are my questions
for some reason, when i try to insert the decoded blob value (what i posted above) into a table (i was going to move it to MS Access and parse it there. this would be a horrible solution but i'm desperate) - the only thing that inserts is "’" without the quotes. also, i can't select all and copy it from the DBMS output window, again, the only thing that pastes is "’" without the quotes. it seems like this text is not really there. does anyone have an idea on how to insert it into a table?
if i was to do it the right way and use java, where do i start? excuse this dumbness but i don't even know how to run java code. i found a few sample codes on the net but i don't know where to paste it :)
i did google it and saw that i have to create a .java file in a text editor and then compile it, is that true for my case? i thought maybe that's some different java code, i thought maybe in my case i'd have to run it from oracle because that's where the tables are.
i also have the table structure, i attached a piece of it. this blob stores a table.
anyhow, i'm sure it's obvious by now that i'm clueless. if anyone can point me somewhere i'd really appreciate it.
thank you

Here is an example of oracle 11g java stored function that deserializes java object from blob. As a free bonus added an example of oracle java stored procedure to update blob with serialized java object.
If object's class isn't java built-in (as in my case), you would also need to publish it's source (with all dependencies) in oracle database.
CREATE OR REPLACE JAVA SOURCE NAMED "ServiceParamsBLOBHandler" AS
import java.io.*;
import java.util.*;
public class ServiceParamsBLOBHandler {
private static Object deserialize(InputStream stream) throws Exception {
ObjectInputStream ois = new ObjectInputStream(stream);
try {
return ois.readObject();
} finally {
ois.close();
}
}
private static byte[] serialize(Object object) throws IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(object);
oos.close();
return baos.toByteArray();
}
//#SuppressWarnings("unchecked")
private static List<Map<String, String>> getParams(oracle.sql.BLOB blob) throws Exception {
return (List<Map<String, String>>) deserialize(blob.getBinaryStream());
}
public static oracle.sql.BLOB updatedParamField(oracle.sql.BLOB blob, String paramName, String fieldName, String value)
throws Exception {
List<Map<String, String>> params = getParams(blob);
Map<String, String> param = getParam(params, paramName);
param.put(fieldName, value);
oracle.sql.BLOB res = oracle.sql.BLOB.createTemporary(blob.getOracleConnection(), true, oracle.sql.BLOB.DURATION_CALL);
res.setBytes(1, serialize(params));
return res;
}
public static void updateParamField(oracle.sql.BLOB[] blobs, String paramName, String fieldName, String value)
throws Exception {
oracle.sql.BLOB blob = blobs[0];
List<Map<String, String>> params = getParams(blob);
Map<String, String> param = getParam(params, paramName);
param.put(fieldName, value);
blob.truncate(0);
blob.setBytes(1, serialize(params));
}
private static Map<String, String> getParam(List<Map<String, String>> params, String name) {
for (Map<String, String> param : params) {
if (name.equals(param.get("name"))) {
return param;
}
}
return null;
}
public static String getParamField(oracle.sql.BLOB blob, String paramName, String fieldName) throws Exception {
Map<String, String> param = getParam(getParams(blob), paramName);
return param == null ? null : param.get(fieldName);
}
}
/
alter java source "ServiceParamsBLOBHandler" compile
--select * from SYS.USER_ERRORS
/
CREATE OR REPLACE function getServiceParamField(b IN BLOB, paramName IN VARCHAR2, fieldName IN VARCHAR2) RETURN VARCHAR2
as LANGUAGE JAVA NAME 'ServiceParamsBLOBHandler.getParamField(oracle.sql.BLOB, java.lang.String, java.lang.String) return String';
/
CREATE OR REPLACE function updatedServiceParamField(b IN BLOB, paramName IN VARCHAR2, fieldName IN VARCHAR2, value IN VARCHAR2) RETURN BLOB
as LANGUAGE JAVA NAME 'ServiceParamsBLOBHandler.updatedParamField(oracle.sql.BLOB, java.lang.String, java.lang.String, java.lang.String) return oracle.sql.BLOB';
/
CREATE OR REPLACE PROCEDURE updateServiceParamField(b IN OUT BLOB, paramName IN VARCHAR2, fieldName IN VARCHAR2, value IN VARCHAR2)
AS LANGUAGE JAVA NAME 'ServiceParamsBLOBHandler.updateParamField(oracle.sql.BLOB[], java.lang.String, java.lang.String, java.lang.String)';
/
-- oracle blob read usage example:
select getServiceParamField(byte_value, 'account', 'format') from entity_property where name='params';
-- oracle blob update with java stored function usage example:
update entity_property set byte_value=updatedServiceParamField(byte_value, 'account', 'format', '15')
where name='params' and entity_id = 123
-- oracle blob update with java stored procedure usage example:
BEGIN
FOR c IN (select byte_value from entity_property where name='params' and entity_id = 123 for update) LOOP
updateServiceParamField(c.byte_value, 'account', 'format', '13');
END LOOP;
END;
/
Update
Concrete snippets for the case in question.
1) Full object load
private static String getVariable(oracle.sql.BLOB blob, String name) throws Exception {
ObjectInputStream ois = new ObjectInputStream(blob.getBinaryStream());
try {
//noinspection unchecked
return ((HashMap<String, String>) ((StorageBean) ois.readObject()).variables).get(name);
} finally {
ois.close();
}
}
2) Partial field load
private static String getVariable(oracle.sql.BLOB blob, String name) throws Exception {
ObjectInputStream ois = new ObjectInputStream(blob.getBinaryStream());
try {
ois.skipBytes(variablesOffset);
//noinspection unchecked
return ((HashMap<String, String>) ois.readObject()).get(name);
} finally {
ois.close();
}
}

i will learn to do this in java at some point but since this is a rush - I decided to use SQL to extract fields from the blob. i'm putting this here in case someone else is ever as desperate to do this.
it's a very ugly and slow solution but so far i'm able to get some fields. i will update once i'm done, to say whether i was able to get everything or not.
here's the code i'm using (this is just for 1 field but it will give you an idea)
DECLARE
CURSOR c_dts IS
SELECT Form_ID
FROM NR_DTS_FORMTABLE
WHERE 1 = 1
--AND ROWNUM BETWEEN 501 AND 4500
AND form_ID > 204815
--AND ROWNUM < 5000
AND ROWNUM < 3
--AND form_id IN (SELECT form_id FROM NR_DTS_BLOB)
AND Form_Type_ID = 102;
DTS c_dts%ROWTYPE;
BEGIN
OPEN c_dts;
LOOP
FETCH c_dts INTO DTS;
EXIT WHEN c_dts%NOTFOUND;
DECLARE
v_hold_blob BLOB;
v_len NUMBER;
v_raw_chunk RAW(10000);
v_chr_string VARCHAR2(32767);
-- v_chr_string CLOB;
v_position NUMBER;
c_chunk_len NUMBER := 1;
Form_ID NUMBER;
BEGIN
SELECT form_content
INTO v_hold_blob
FROM NR_DTS_FORMTABLE
WHERE Form_ID = DTS.Form_ID;
v_len := DBMS_LOB.getlength(v_hold_blob);
v_position := 1;
WHILE (v_position <= LEAST(v_len, 32767)) LOOP
v_raw_chunk := DBMS_LOB.SUBSTR(v_hold_blob, c_chunk_len, v_position);
v_chr_string := v_chr_string || CHR(hex_to_decimal(RAWTOHEX(v_raw_chunk)));
v_position := v_position + c_chunk_len;
END LOOP;
--insert into table
INSERT INTO NR_DTS_BLOBFIELDS_VARCHAR(formid
,regionId)
SELECT DTS.Form_ID
,SUBSTR(v_chr_string
,INSTR(v_chr_string, 'regionIdt') + LENGTH('regionIdt') + 2
,INSTR((SUBSTR(v_chr_string, INSTR(v_chr_string, 'regionIdt') + LENGTH('regionIdt') + 2))
,CHR(116) || CHR(0)))
regionId
FROM DUAL;
END;
-- DBMS_OUTPUT.put_line(DTS.Form_ID);
END LOOP;
CLOSE c_dts;
END;