first of all, i apologize, i'm about to ask a set of dumb questions. i don't know java AT ALL and i don't know if we are allowed to ask questions like these.
if not - delete my topic.
there's a table in oracle that stores a blob. it's binary and i'm able to decode it, the output looks like this
¬í sr /com.epam.insure.credentialing.forms.StorageBeanÀÓ ¯w/§ L variablest Ljava/util/Map;xpsr java.util.HashMapÚÁÃ`Ñ F
loadFactorI thresholdxp?# w t $_hasCompletedt t
$_wf_progresssr java.lang.Integerâ ¤÷‡8 I valuexr java.lang.Number†¬•”à‹ xp t $_wf_statussq ~ t $_form_instance_idsr java.lang.Long;‹äÌ#ß J valuexq ~ ‹©t $_isVisitedt truet 1sq ~ sq ~ ?# `w € _t confidential readable infot 1t confidential readable infot $_errorssr java.util.ArrayListxÒ™Ça I sizexp w
xt regionIdsq ~ ët
confidential readable infot t t $_subbean_errorssq ~ w
xt regiont SOUTHWESTt idt t codet t reqTypeNamet
confidential readable infot t confidential readable infot tint t $_hasCompletedt falset comRequiredt t
lineImpactq ~ t prChiropractorsq ~ t fromTypeReqt not zipt 342t changeToTypeReq6t confidential readable infot t
prPodiatristsq ~ t
$_isValidatedt truet $_hasErrorsq ~ -t EVPapprovalsq ~ sq ~ ?# w Approvedq ~ Ct
NEGOTIATORq ~ Et
Negotiatort datet
03/31/2006q ~ It confidential readable infot q ~ \xt updateRequiredt noq ~ t truet approverssr .forms.StorageBeanList«WtúœG xq ~ w
q ~ Rsq ~ sq ~ ?# w t commentst t decisiont Approvedq ~ Ct RVPq ~ Et RVPt datet
04/04/2006q ~ It t commentst t decisiont Approvedq ~ Ct COOq ~ Et COOt datet
04/14/2006q ~ It ~ †xsq ~ sq ~ ?# w t commentsq ~ Pt decisiont Approvedq ~ Ct CEOq ~ Et CEOt d
so here are my questions
for some reason, when i try to insert the decoded blob value (what i posted above) into a table (i was going to move it to MS Access and parse it there. this would be a horrible solution but i'm desperate) - the only thing that inserts is "’" without the quotes. also, i can't select all and copy it from the DBMS output window, again, the only thing that pastes is "’" without the quotes. it seems like this text is not really there. does anyone have an idea on how to insert it into a table?
if i was to do it the right way and use java, where do i start? excuse this dumbness but i don't even know how to run java code. i found a few sample codes on the net but i don't know where to paste it :)
i did google it and saw that i have to create a .java file in a text editor and then compile it, is that true for my case? i thought maybe that's some different java code, i thought maybe in my case i'd have to run it from oracle because that's where the tables are.
i also have the table structure, i attached a piece of it. this blob stores a table.
anyhow, i'm sure it's obvious by now that i'm clueless. if anyone can point me somewhere i'd really appreciate it.
thank you
Here is an example of oracle 11g java stored function that deserializes java object from blob. As a free bonus added an example of oracle java stored procedure to update blob with serialized java object.
If object's class isn't java built-in (as in my case), you would also need to publish it's source (with all dependencies) in oracle database.
CREATE OR REPLACE JAVA SOURCE NAMED "ServiceParamsBLOBHandler" AS
import java.io.*;
import java.util.*;
public class ServiceParamsBLOBHandler {
private static Object deserialize(InputStream stream) throws Exception {
ObjectInputStream ois = new ObjectInputStream(stream);
try {
return ois.readObject();
} finally {
ois.close();
}
}
private static byte[] serialize(Object object) throws IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(object);
oos.close();
return baos.toByteArray();
}
//#SuppressWarnings("unchecked")
private static List<Map<String, String>> getParams(oracle.sql.BLOB blob) throws Exception {
return (List<Map<String, String>>) deserialize(blob.getBinaryStream());
}
public static oracle.sql.BLOB updatedParamField(oracle.sql.BLOB blob, String paramName, String fieldName, String value)
throws Exception {
List<Map<String, String>> params = getParams(blob);
Map<String, String> param = getParam(params, paramName);
param.put(fieldName, value);
oracle.sql.BLOB res = oracle.sql.BLOB.createTemporary(blob.getOracleConnection(), true, oracle.sql.BLOB.DURATION_CALL);
res.setBytes(1, serialize(params));
return res;
}
public static void updateParamField(oracle.sql.BLOB[] blobs, String paramName, String fieldName, String value)
throws Exception {
oracle.sql.BLOB blob = blobs[0];
List<Map<String, String>> params = getParams(blob);
Map<String, String> param = getParam(params, paramName);
param.put(fieldName, value);
blob.truncate(0);
blob.setBytes(1, serialize(params));
}
private static Map<String, String> getParam(List<Map<String, String>> params, String name) {
for (Map<String, String> param : params) {
if (name.equals(param.get("name"))) {
return param;
}
}
return null;
}
public static String getParamField(oracle.sql.BLOB blob, String paramName, String fieldName) throws Exception {
Map<String, String> param = getParam(getParams(blob), paramName);
return param == null ? null : param.get(fieldName);
}
}
/
alter java source "ServiceParamsBLOBHandler" compile
--select * from SYS.USER_ERRORS
/
CREATE OR REPLACE function getServiceParamField(b IN BLOB, paramName IN VARCHAR2, fieldName IN VARCHAR2) RETURN VARCHAR2
as LANGUAGE JAVA NAME 'ServiceParamsBLOBHandler.getParamField(oracle.sql.BLOB, java.lang.String, java.lang.String) return String';
/
CREATE OR REPLACE function updatedServiceParamField(b IN BLOB, paramName IN VARCHAR2, fieldName IN VARCHAR2, value IN VARCHAR2) RETURN BLOB
as LANGUAGE JAVA NAME 'ServiceParamsBLOBHandler.updatedParamField(oracle.sql.BLOB, java.lang.String, java.lang.String, java.lang.String) return oracle.sql.BLOB';
/
CREATE OR REPLACE PROCEDURE updateServiceParamField(b IN OUT BLOB, paramName IN VARCHAR2, fieldName IN VARCHAR2, value IN VARCHAR2)
AS LANGUAGE JAVA NAME 'ServiceParamsBLOBHandler.updateParamField(oracle.sql.BLOB[], java.lang.String, java.lang.String, java.lang.String)';
/
-- oracle blob read usage example:
select getServiceParamField(byte_value, 'account', 'format') from entity_property where name='params';
-- oracle blob update with java stored function usage example:
update entity_property set byte_value=updatedServiceParamField(byte_value, 'account', 'format', '15')
where name='params' and entity_id = 123
-- oracle blob update with java stored procedure usage example:
BEGIN
FOR c IN (select byte_value from entity_property where name='params' and entity_id = 123 for update) LOOP
updateServiceParamField(c.byte_value, 'account', 'format', '13');
END LOOP;
END;
/
Update
Concrete snippets for the case in question.
1) Full object load
private static String getVariable(oracle.sql.BLOB blob, String name) throws Exception {
ObjectInputStream ois = new ObjectInputStream(blob.getBinaryStream());
try {
//noinspection unchecked
return ((HashMap<String, String>) ((StorageBean) ois.readObject()).variables).get(name);
} finally {
ois.close();
}
}
2) Partial field load
private static String getVariable(oracle.sql.BLOB blob, String name) throws Exception {
ObjectInputStream ois = new ObjectInputStream(blob.getBinaryStream());
try {
ois.skipBytes(variablesOffset);
//noinspection unchecked
return ((HashMap<String, String>) ois.readObject()).get(name);
} finally {
ois.close();
}
}
i will learn to do this in java at some point but since this is a rush - I decided to use SQL to extract fields from the blob. i'm putting this here in case someone else is ever as desperate to do this.
it's a very ugly and slow solution but so far i'm able to get some fields. i will update once i'm done, to say whether i was able to get everything or not.
here's the code i'm using (this is just for 1 field but it will give you an idea)
DECLARE
CURSOR c_dts IS
SELECT Form_ID
FROM NR_DTS_FORMTABLE
WHERE 1 = 1
--AND ROWNUM BETWEEN 501 AND 4500
AND form_ID > 204815
--AND ROWNUM < 5000
AND ROWNUM < 3
--AND form_id IN (SELECT form_id FROM NR_DTS_BLOB)
AND Form_Type_ID = 102;
DTS c_dts%ROWTYPE;
BEGIN
OPEN c_dts;
LOOP
FETCH c_dts INTO DTS;
EXIT WHEN c_dts%NOTFOUND;
DECLARE
v_hold_blob BLOB;
v_len NUMBER;
v_raw_chunk RAW(10000);
v_chr_string VARCHAR2(32767);
-- v_chr_string CLOB;
v_position NUMBER;
c_chunk_len NUMBER := 1;
Form_ID NUMBER;
BEGIN
SELECT form_content
INTO v_hold_blob
FROM NR_DTS_FORMTABLE
WHERE Form_ID = DTS.Form_ID;
v_len := DBMS_LOB.getlength(v_hold_blob);
v_position := 1;
WHILE (v_position <= LEAST(v_len, 32767)) LOOP
v_raw_chunk := DBMS_LOB.SUBSTR(v_hold_blob, c_chunk_len, v_position);
v_chr_string := v_chr_string || CHR(hex_to_decimal(RAWTOHEX(v_raw_chunk)));
v_position := v_position + c_chunk_len;
END LOOP;
--insert into table
INSERT INTO NR_DTS_BLOBFIELDS_VARCHAR(formid
,regionId)
SELECT DTS.Form_ID
,SUBSTR(v_chr_string
,INSTR(v_chr_string, 'regionIdt') + LENGTH('regionIdt') + 2
,INSTR((SUBSTR(v_chr_string, INSTR(v_chr_string, 'regionIdt') + LENGTH('regionIdt') + 2))
,CHR(116) || CHR(0)))
regionId
FROM DUAL;
END;
-- DBMS_OUTPUT.put_line(DTS.Form_ID);
END LOOP;
CLOSE c_dts;
END;
Related
So, I have a 16 node cluster where every node has Spark and Cassandra installed while I am using the Spark-Cassandra Connector 3.0.0. I am trying to join a dataset with a cassandra table on the partition key, while also trying to use .repartitionByCassandraReplica.
However it seems I just get an empty rdd with 0 partitions(line 5 below)! Any ideas why?
Encoder<ExperimentForm> ExpEncoder = Encoders.bean(ExperimentForm.class);
//FYI experimentlist is a List<String>
Dataset<ExperimentForm> dfexplistoriginal = sp.createDataset(experimentlist, Encoders.STRING()).toDF("experimentid").as(ExpEncoder);
JavaRDD<ExperimentForm> predf = CassandraJavaUtil.javaFunctions(dfexplistoriginal.toJavaRDD()).repartitionByCassandraReplica("mdb","experiment",experimentlist.size(),CassandraJavaUtil.someColumns("experimentid"),CassandraJavaUtil.mapToRow(ExperimentForm.class));
System.out.println(predf.collect()); //Here it gives an empty dataset with 0 partitions
Dataset<ExperimentForm> newdfexplist = sp.createDataset(predf.rdd(), ExpEncoder);
Dataset<Row> readydfexplist = newdfexplist.as(Encoders.STRING()).toDF("experimentid");
Dataset<Row> metlistinitial = sp.read().format("org.apache.spark.sql.cassandra")
.options(new HashMap<String, String>() {
{
put("keyspace", "mdb");
put("table", "experiment");
}
})
.load().select(col("experimentid"), col("description"), col("intensity")).join(readydfexplist, "experimentid");
In case needed this is the experiment table in Cassandra:
CREATE TABLE experiment(
experimentid varchar,
description text,
rt float,
intensity float,
mz float,
identifier text,
chemical_formula text,
filename text,
PRIMARY KEY ((experimentid),description, rt, intensity, mz, identifier, chemical_formula, filename));
and this is the ExperimentForm class:
public class ExperimentForm {
private String experimentid;
public String getExperimentid() {
return experimentid;
}
public void setExperimentid(String experimentid) {
this.experimentid = experimentid;
}
}
Let me know if you need any additional information.
The answer is basically the same as here Spark-Cassandra: repartitionByCassandraReplica or converting dataset to JavaRDD and back do not maintain number of partitions?
Just had to do the repartitionByCassandraReplica and JoinWithCassandraTable on RDD and then convert back to dataset.
Working with Cassandra and Spark 2.12 (3.2.0) with java.
Cassandra connector 3.1.0
My purpose fetch from s3 do preprocess and insert parallel into Cassandra.
I encounter with problem that I did preprocess on each s3 file that include list of items to insert to Cassandra which look like: JavaRDD<List<SearchEntity>>
How should I pass it to cassandra (as in code example) ? I see it supports single object mapToRow.
maybe I miss something ?
Using the following code
public static void main(String[] args) throws Exception {
SparkConf conf = new SparkConf()
.setAppName("Example Spark App")
.setMaster("local[1]")
.set("spark.cassandra.connection.host", "127.0.0.1");
JavaSparkContext sparkContext = new JavaSparkContext(conf);
sparkContext.hadoopConfiguration().set("fs.s3a.access.key", "XXXX");
sparkContext.hadoopConfiguration().set("fs.s3a.secret.key", "YYYY");
sparkContext.hadoopConfiguration().set("fs.s3a.endpoint", "XXXXX");
sparkContext.hadoopConfiguration().set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem");
sparkContext.hadoopConfiguration().set("mapreduce.input.fileinputformat.input.dir.recursive", "true");
JavaPairRDD<String, PortableDataStream> javaPairRDD = sparkContext.binaryFiles("s3a://root/folder/");
File ROOT = createTempFolder().getAbsoluteFile();
JavaRDD<List<SearchEntity>> listJavaRDD = javaPairRDD.map(rdd -> {
System.out.println("Working on TAR: " + rdd._1);
DataInputStream stream = rdd._2.open();
// some preprocess
List<SearchEntity> toCassandraList = new WorkerTest(ROOT, stream).run();
return toCassandraList;
});
// here I want to take List<SearchEntity> toCassandraList and save them
// but I don't see how as it support only single object ..
CassandraJavaUtil.javaFunctions(listJavaRDD)
.writerBuilder("demoV2", "simple_search",
CassandraJavaUtil.mapToRow(List<SearchEntity> list objects ...)) // here is problem
.saveToCassandra();
System.out.println("Finish run s3ToCassandra:");
sparkContext.stop();
}
The schema was configured before manually only for tests purposes.
CREATE TABLE simple_search (
engine text,
term text,
time bigint,
rank bigint,
url text,
domain text,
pagenum bigint,
descr text,
display_url text,
title text,
type text,
PRIMARY KEY ((engine, term), time , url, domain, pagenum)
) WITH CLUSTERING ORDER BY
(time DESC, url DESC, domain DESC , pagenum DESC);
Both Java and Scala solutions are welcomed
To write the data, you need to work on the SearchEntity, not on the list of the SearchEntity. To do that, you need to use flatMap instead of ordinary map:
JavaRDD<SearchEntity> entriesRDD = javaPairRDD.flatMap(rdd -> {
System.out.println("Working on TAR: " + rdd._1);
DataInputStream stream = rdd._2.open();
// some preprocess
List<SearchEntity> toCassandraList = new WorkerTest(ROOT, stream).run();
return toCassandraList;
});
and then you can just write the as per documentation:
javaFunctions(rdd).writerBuilder("demoV2", "simple_search",
mapToRow(SearchEntity.class)).saveToCassandra();
P.S. But be beware, that if your tar is too big, it may cause memory errors on the workers, when you're creating the List<SearchEntity>. Depending on the file format inside tar file, it could be more optimal to unpack data first, and then read them using Spark.
I want to get value of the column which data type is CLOB. In Java side I am getting attributes of the struct which are coming from the our procedure which called "get_val". attr[4] is empty althought procedure doesn't return empty. It works fine for some CLOB attrubites but sometimes returns empty property.
I am sure that it comes empty because when I try to send length of attr[4]. It shows "0". But from procedure it shows 38 symbols.
What can be reason for this ? How I can fix it ?
public ValueEntity getAttributeValue(String token, Long objectId, Integer attrId, Date bankDate) throws SQLException, IOException{
try{
createContext(token,true);
ocstmt.close();
ocstmt = (OracleCallableStatement) oracleConnection.prepareCall("{ ? = call package.get_val(?,?,?) }");
ocstmt.registerOutParameter(1, OracleTypes.STRUCT, "T_VAL");
setLong(2, objectId);
setInt(3, attrId);
setDate(4, bankDate);
ocstmt.setQueryTimeout(200);
ocstmt.execute();
return DBUtil.ComplexTypeMapping.getValueEntity(oracleConnection, (Struct)ocstmt.getObject(1));
}finally{
releaseAllResources(false);
}
}
public static ValueEntity getValueEntity(OracleConnection connection, Struct struct) throws SQLException, IOException{
if(struct == null) return null;
Object[] attrs = struct.getAttributes();
ValueEntity data = new ValueEntity();
data.setType(bigdecimalAsInteger((BigDecimal)attrs[0]));
data.setNum(bigdecimalAsDouble((BigDecimal)attrs[1]));
data.setStr(attrs[2] != null?String.valueOf(attrs[2]):null);
data.setDate((Date)attrs[3]);
//Clob clobAttr= (Clob)attrs[4];
//long clobLength= clobAttr.length();
//data.setClob( String.valueOf(clobLength));
data.setClob(getClobAsString((Clob)attrs[4]));
data.setNumArr(getDoubleArray((Array)attrs[5]));
data.setStrArr(getStringArray((Array)attrs[6]));
data.setDateArr(getDateArray((Array)attrs[7]));
return data;
}
Finally I have found that problem was related with IBM websphere version.It was 10.0.0.3 but now we are using 10.0.0.9 version.
This question already has an answer here:
Attempt to index local 'v' (a nil value)
(1 answer)
Closed 5 years ago.
There has been a lot of post about this kind of error and most people say it is related to table and array indexing problems. But I am not using tables at all, I am just trying to call a library function I made and I get this error. Here is the lua script called from java:
String script = new String (
"function event_touch ( )"
+ " pb.open_form ('view_monster');"
+ " print ('I ran the lua script');"
+ "end");
PBLUAengine.run_script(script, "event_touch");
This gives me the following error when trapping the exception:
"function event_touch ( ) pb.open_form ('view_monster'); print ('I ran the lua script');end:1 attempt to index ? (a nil value)"
The run_script() function calls the script like this( I am using luaj):
public static LuaValue run_script ( String script )
{
try
{
LuaValue chunk = s_globals.load( script );
return chunk.call();
}
catch ( Exception e)
{
Gdx.app.error ("PBLUAengine.run_script()", e.getMessage() );
}
return null;
}
The library method goes like this and the same piece of code works when called from java:
static class open_form extends OneArgFunction
{
public LuaValue call (LuaValue formname)
{
String tmpstr = (String ) CoerceLuaToJava.coerce(formname, java.lang.String.class );
try
{
PBscreen_game.hide_subscreen(PBscreen_game.MENU_SUBS);
PBscreen_game.show_subscreen ( PBscreen_game.FORM_SUBS);
PBsubscreen_form.open_form ( new PBform_regular ( tmpstr ) );
}
catch (Exception e)
{
Gdx.app.error("PBLUAlibrary.open_form", e.getMessage());
}
return valueOf ( 0 );
}
}
It basically convert the lua parameter to string, create a new from and pass in parameter the string.
The declaration of the library functions goes like this:
public LuaValue call( LuaValue modname, LuaValue env )
{
LuaValue library = tableOf();
library.set( "open_form", new open_form() );
library.set( "open_system_form", new open_system_form() );
env.set( "pb", library );
return library;
}
Which could be the only "table" I can see in the whole system. This is generally used link the right class with the right function name.
Anybody have an idea?
most people say it is related to table and array indexing problems
It's related to table and array indexing. If you try to index an object and that object is nil, you'll get that error:
I am not using tables at all [..] Here is the lua script:
pb.open_form
pb is being indexed. It's probably nil.
I seems that I solved the problem by adding a require line to include the library. So the new script is:
String script = new String (
"require 'com.lariennalibrary.pixelboard.library.PBLUAlibrary'"
+ "function event_touch ( )"
+ " pb.open_form ('view_monster');"
+ " print ('I ran the next button lua script');"
+ "end");
It ask to include my library class which will add all the "pb.*" functions. I probably deleted the line by error, or managed to make it work somehow without it. Since this library will be required by all script, I might append it by default before each script I try to run.
Thanks Again
I am trying to convert a 128 bit binary to a uniqueidentifier in sql that is the same as in .net and java.
I know java uses big endians, so I would like to make that the base.
I can get the correct endianess in .net, but am really struggling with it in SQL Server.
Java:
byte[] bytesOfMessage = "google.com".getBytes("UTF-8");
MessageDigest md = MessageDigest.getInstance("MD5");
byte[] md5 = md.digest(bytesOfMessage);
ByteBuffer bb = ByteBuffer.wrap(md5);
LongBuffer ig = bb.asLongBuffer();
return new UUID(ig.get(0), ig.get(1));
returns 1d5920f4-b44b-27a8-02bd-77c4f0536f5a
.Net
System.Security.Cryptography.MD5 c = System.Security.Cryptography.MD5.Create();
byte[] b = c.ComputeHash(Encoding.UTF8.GetBytes("google.com"));
int z = System.Net.IPAddress.HostToNetworkOrder(BitConverter.ToInt32(b, 0));
short y = System.Net.IPAddress.HostToNetworkOrder(BitConverter.ToInt16(b, 4));
short x = System.Net.IPAddress.HostToNetworkOrder(BitConverter.ToInt16(b, 6));
Guid g = new Guid(z, y, x, b.Skip(8).ToArray());
return g;
returns 1d5920f4-b44b-27a8-02bd-77c4f0536f5a
SQL
DECLARE #s VARCHAR(MAX) = 'google.com' --'goolge.com'
DECLARE #md5 BINARY(16) = HASHBYTES
(
'MD5',
#s
)
DECLARE #a BINARY(4) =
CONVERT
(
BINARY(4),
REVERSE
(
CONVERT
(
BINARY(4),
LEFT(#md5, 4)
)
)
)
DECLARE #b BINARY(2) =
CONVERT
(
BINARY(2),
REVERSE
(
CONVERT
(
BINARY(2),
RIGHT(#md5, 12)
)
)
)
DECLARE #c BINARY(2) =
CONVERT
(
BINARY(2),
REVERSE
(
CONVERT
(
BINARY(2),
RIGHT(#md5, 10)
)
)
)
DECLARE #d BINARY(8) =
CONVERT
(
BINARY(8),
RIGHT(#md5, 8)
)
SELECT
CONVERT
(
UNIQUEIDENTIFIER,
#a + #b + #c + #d
)
returns D86B5A7F-7A25-4895-A6D0-63BA3A706627
I am able to get all three to produce the same value when converting to an int64, but the GUID is baffling me.
Original Issue
Original Answer
If you correct the spelling of google in your SQL example (it's goolge in your post), you get the right result.
SQL Server does not support UTF-8 encoding. See Description of storing UTF-8 data in SQL Server. Use suggestion from Michael Harmon to add your .NET function to SQLServer to do the conversion. See How to encode... for instructions on how to add your .NET function to SQLServer.
Alternatively, don't specify UTF-8 in your Java and .NET code. I believe SQL Server will use same 256 bit encoding for varchar as does Java and .NET. (But not totally sure of this.)