I know that Scalding's default serialization uses Kryo. So for this example, lets say I have a pipe of student objects.
case class Student(name:String, id:String)
val pipe: Pipe[Student] = //....
Then I write that pipe to a TextDelimited file using Kryo.
pipe.write(args("output"))
Now that I have these "output" files, how do I easily read them back into Student objects? My general idea is something like the following but it does not work.
Tsv("Kryo output files").read.map {bytes =>
val instantiator = (new ScalaKryoInstantiator).setRegistrationRequired(true)
val kryo = instantiator.newKryo
kryo.register(classOf[Person])
val deserialized = kryo.readObject(new Input(bytes), classOf[Person])
deserialized
}
How do I deserialize my Kryo written text files into the objects they were originally in?
Related
I am currently facing issue during checkmarx scan. It is highlighting that we are deserializing of Untrusted data in the last line mentioned below. How to rectify this issue ?
Scan Issue : Deserialization of Untrusted Data
Note: We do not have any xsd
String message = request.getParameter("param_name"); // Input xml string
XStream parser = new XStream(new StaxDriver());
MyMessage messageObj = (MyMessage) parser.fromXML(message); // This line is flagged by CHECKMARX SCAN
I will assume that you intended to say that you're getting results for Deserialization of Untrusted Data.
The reason you're getting that message is that XStream will happily attempt to create an instance of just about any object specified in the XML by default. The technique is to allow only the types you intend to be deserialized. One would presume you've ensured those types are safe.
I ran this code derived from your example and verified that the two lines I added were detected as sanitization.
String message = request.getParameter("param_name");
XStream parser = new XStream(new StaxDriver());
parser.addPermission(NoTypePermission.NONE);
parser.allowTypes(new Class[] {MyMessage.class, String.class});
MyMessage messageObj = (MyMessage) parser.fromXML(message);
I added the String.class type since I'd presume some of your properties on MyMessage are String. String itself, like most primitives, is generally safe for deserialization. While the string itself is safe, you'll want to make sure how you use it is safe. (e.g. if you are deserializing a string and passing it to the OS as part of a shell exec, that could be a different vulnerability.)
I stored the java object in hbase (i.e) Let's say I have an object 'User' with 3 parameters like firstname, middlename and lastname. I used the following code for serialization in java
Object object = (object) user;
byte[] byteData = SerializationUtils.serialize((Serializable) object);
and stored in hbase like 'storing complete object (in byte[] format of above) in the Value portion of the KeyValue pair'
It is stored in hbase like (Example)
column=container:container, timestamp=1480016194005, value=\xAC\xED\x00\x05sr\x00&com.test.container\x07\x89\x83\xFA\x7F\xD0F\xA5\x02\x00\x08I\x00\x07classIdJ\x00\x14dateTimeInLongFormatZ\x00\x04rootZ\x00\x09undefinedL\x00\x03keyt\x00\x12Ljava/lang/String;L\x00\x04modeq\x00~\x00\x01L\x00\x04nameq\x00~\x00\x01L\x00\x06userIdq\x00~\x00\x01xp\x00\x00\x00\x02\x00\x00\x01X\x967\xBA\xF0\x00\x00t\x00\x1Econtainer_393_5639181044834024t\x00\x06expandt\x00\x02ert\x00\x08testadmin
when I try to retrieve the data, I used the following deserialization in java and converted back to object of readable format
object = SerializationUtils.deserialize(bytes);
I would like to retrieve the data stored in java format via happybase using python and I achieved it and received the data as available in hbase like
It is stored in hbase like (Example)
column=container:container, timestamp=1480016194005, value=\xAC\xED\x00\x05sr\x00&com.test.container\x07\x89\x83\xFA\x7F\xD0F\xA5\x02\x00\x08I\x00\x07classIdJ\x00\x14dateTimeInLongFormatZ\x00\x04rootZ\x00\x09undefinedL\x00\x03keyt\x00\x12Ljava/lang/String;L\x00\x04modeq\x00~\x00\x01L\x00\x04nameq\x00~\x00\x01L\x00\x06userIdq\x00~\x00\x01xp\x00\x00\x00\x02\x00\x00\x01X\x967\xBA\xF0\x00\x00t\x00\x1Econtainer_393_5639181044834024t\x00\x06expandt\x00\x02ert\x00\x08testadmin
Is there a way to deserialize the java object via python
Thanks Much
Hari
There is a Python library for that:
https://pypi.python.org/pypi/javaobj-py3/
Usage seems pretty easy with:
import javaobj
jobj = self.read_file("obj5.ser")
pobj = javaobj.loads(jobj)
print(pobj)
If a mail is send to my inbox, I recieve a message, and I'm inserting the contents into DB.
I have a org.springframework.integration.core.Message something like follows:
public void receive(Message<?> message)
{
//I am inserting message contents into DB
}
Now in the event of failure, I wanted to have fail safe recovery mechanism, what I am thinking is to serialize the Message object into a file and later deserialize and update to DB.
Question
1. In this situation how to serialize the Message object?
2. Other than serialization any other mechanism that can be used?
EDIT
I have not done Serialization before, I heard like the class should implements Serializable in order to use ObjectOutputStream, in this case I don't want to create a subclass of Message, So how to serialize Message to File?
Sure, there are many serialization mechanisms apart from the jvm one.
XML
JSON
BSON
MessagePack
protobufs
...
Some of them are text-based, some are binary. All have drawbacks and pluses. Text-based ones are human-readable, binary ones are faster and take up less space.
There are java libraries that handle all the above formats: JAXB (XML), Jackson (JSON), etc.
In this situation how to serialize the Message object? Other than serialization any other mechanism that can be used?
Extract all the data you need from the Message and save it. You can do this in any manner you choose.
You can deserialize it by populating a new Message with the data you saved.
I don't know if I probably understood it al right.. but assuming Message is not much more than lots of strings and some integers you can just use directly an ObjectOutputStream and write it to a file (binary) and then readin later. Why not?
Message e = new Message();
ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream("whatever");
oos.writeObject(message);
// read in
ObjectInputStream ois = new ObjectInputStream(new FileInputStream("whatever");
Message e = (Message) ois.readObject();
Just started using soapUI and I like it a lot.
In a particular case using REST, I'm receiving serialized object.
I would like :
to retrieve the serialized byte array and transform it into a Java object
re-transform the java object into an XML response (using JAXB)
so it can be human readable.
Is this feasible?
Be sure to consider using XML serialization (e.g. XStream) instead of binary one to avoid version compatibility problems before using the next solution:
Import your Java class to SoapUI groovy script (as described there) or re-define your Java class in Groovy code with Serializable interface implemented:
class Person implements Serializable { String name; int age }
Use ObjectInputStream and classLoader to load deserialize objects into object:
// use your byte array variable instead of yourByteArray
input = new ByteArrayInputStream(yourByteArray)
// use your object variable instead of yourObject
yourObject = null
input.withObjectInputStream(getClass().classLoader){ ois -> yourObject = ois.readObject() }
Use ObjectOutputStream to serialize updated objects and save them to an XML response:
output = new ByteArrayOutputStream()
output.withObjectOutputStream { oos -> oos << yourObject }
//save serialized data as byte array
output.toByteArray()
I have a java library, I would like to save an instance of a java object to a text file. I have tried to use all java libraries for serialization and deserialization to xml: http://karussell.wordpress.com/2009/09/03/xml-serializers-for-java/ but deserializing does not work. I have posted a question here: http://stackoverflow.com/questions/6139702/deserializing-xml-file-from-xstream but It seems that I could not get a solution for that.
I also have tried to serialize to json but deserialize from json does not work.
So, I would like to know apart of serializing to xml and json, is there any other way to do serialization and deserialization from a java object (cannot modify to add tags: #XmlRootElement(name="customer")) to text file?
Thanks in advance!
The easiest way is probably to use the native Java serialization. It will generate a binary representation of the object, but you can encode the generated byte array with Base64 to transform it to text:
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(myObject);
oos.flush();
byte[] binary = baos.toByteArray();
String text = Base64.encodeBase64String(binary); // Base64 is in the apache commons codec library
// save text to a file
Note that the object, and every object it references (recursively) must implement java.io.Serializable for this to work.
You can use Gson to convert java object to Json and vice versa
Here is example from the Gson user guide.
Or may be apache digester can help.