Develop Reference

Using LARGE JSON Data Source with Apache Flink and Python (2 questions)

Flink's Batch Python API, Version 1.2 supports
read_text(path)
read_csv(path, type)
from_elements(*args)
And some specific data types: (int, float, bool, string) and byte arrays.
Python is pretty strong with dictionaries, and reading a json file through json.load(file) creates a dict type variable, but as far as i searched, dictionary manipulation isn't supported by flink atm.
Question num0: So, besides parsing strings and trying to mess with regular expressions, is there any other way to handle large json files with flink's python api?
I was thinking to preprocess my json file using lists and in combination with from_elements(*args) to achieve something.
Handling Objects seems to be a problem with Python+Flink as well.
Question num1: Is switching to Java my best bet?
I would like to have an easy way to handle json data, as well as using objects derived from these data.

Parsing Serialized Java objects in Python

The string at the bottom of this post is the serialization of a java.util.GregorianCalendar object in Java. I am hoping to parse it in Python.
I figured I could approach this problem with a combination of regexps and key=val splitting, i.e. something along the lines of:
text_inside_brackets = re.search(r"\[(.*)\]", text).group(1)
and
import parse
for x in [parse('{key} = {value}', x) for x in text_inside_brackets.split('=')]:
my_dict[x['key']] = x['value']
My question is: What would be a more principled / robust approach to do this? Are there any Python parsers for serialized Java objects that I could use for this problem? (do such things exist?). What other alternatives do I have?
My hope is to ultimately parse this in JSON or nested Python dictionaries, so that I can manipulate it it any way I want.
Note: I would prefer to avoid a solution relies on Py4J mostly because it requires setting up a server and a client, and I am hoping to do this within a single
Python script.
java.util.GregorianCalendar[time=1413172803113,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="America/New_York",offset=-18000000,dstSavings=3600000,useDaylight=true,transitions=235,lastRule=java.util.SimpleTimeZone[id=America/New_York,offset=-18000000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2014,MONTH=9,WEEK_OF_YEAR=42,WEEK_OF_MONTH=3,DAY_OF_MONTH=13,DAY_OF_YEAR=286,DAY_OF_WEEK=2,DAY_OF_WEEK_IN_MONTH=2,AM_PM=0,HOUR=0,HOUR_OF_DAY=0,MINUTE=0,SECOND=3,MILLISECOND=113,ZONE_OFFSET=-18000000,DST_OFFSET=3600000]

The serialized form of a GregorianCalendar object contains quite a lot of redundancy. In fact, there are only two fields that matter, if you want to reconstitute it:
the time
the timezone
There is code for extracting this in How to convert Gregorian string to Gregorian Calendar?
If you want a more principled and robust approach, I echo mbatchkarov's suggestion to use JSON.

Mapping java variables data to C Structures and write a c compatible file

I have a java class file with three arrayLists, one with type String, one with type Integer and other is ArrayList with type (ArrayList(String)). I have to write these these arraylists to a structure in C with character arrays, integers and short and output a file in a specefic format extension. The file has to be readable again by the same application. What is the best way to trasnfer the data from java to c structure and then output the c structure in a file. Thank you

There is no "C compatible file" format. If you have C structs written to disk file directly, then those are in an ad-hoc binary format. Exact format depends on things like packing and padding of the struct, byte order, word size of the CPU (like, 32 or 64 bit), etc.
So, start by defining the format, then forget it is produced by C.
Once you have the format defined, you can write a program to parse it in Java. If it is short with fixed length records, I'd probably create a class, which internally has just a private byte[] array, and then methods to manipulate it, save it and load it.

I suggest you write/read the data to a ByteBuffer using native byte ordering. The rest is up to you are to how you do it.
A library which might help is Javolution's Struct library which helps you map C structs onto ByteBuffers. This can help with C's various padding rules i.e.the exact layout might not be obvious.

Is there something like C's struct in Java?

struct is necessary when you try to parse some file format like ELF, etc...
Is there something like C's struct in Java?
Or can Java be used to parse ELF/binary format directly in the first place?

If you need a struct for grouping different data of the same type, Java has a class, and a class is better in logically grouping data than a struct because it includes operations on the data as well.
If you want to format ELF, then you may have to look at the "The ELF Parser" section in http://www.icsa.inf.ed.ac.uk/research/groups/hase/manuals/design/javahase.html. See also LibElf and GElf - A Library to Manipulate ELf Files (an old article)

Unfortunatly there is no decent support to read binary structured data in java.
This example reads image header into a byte array and assembles the required information.

There's ByteBuffer.
Edit
This is just to answer how you might parse the ELF format, which seemed to be what the OP was actually asking for.
For example (I assume this is the same format, apologies if it's a completely different ELF format, either way, it shows the same process):
http://jumdbrowser.googlecode.com/svn-history/r3/trunk/UmdBrowser/src/jpcsp/format/Elf32.java

Edit: answer to the first question
Java classes

Porting XML parsing from Java to Objective C

I am trying to port code written in Java to Objective C (for iPhone), but I'm kind of confused about a few lines of my code (mentioned below). How should I port this efficiently?
Namespace nmgrhistory=Namespace.getNamespace("history", "http://www.mywebsite.com/History.xsd");
pEventEl.addContent(new Element("History",nmgrhistory));
Namespace nmgrState=Namespace.getNamespace("state", "http://www.mywebsite.com/State.xsd");
pEventEl.addContent(new Element("State",nmgrState));
Iterator<Element> eld=(Iterator<Element>) pEventEl.getChild(
pEventEl.getName() == "event"? "./history:history/state:state" : "./state:state",pEventEl.getNamespace());
I'm not very sure about the replacements for the classes Namespace, Iterator and Element.
Anybody having idea or having done this before, please enlighten me.

Ok... So although these are not the actual replacements ... But basically what u need for parsing XML in Objecive - C is "NSXMLParser"
so u can say that NSXMLParser is the replacement for Namespace
And for "Iterator" NSXMLParserDelegate has a method named:-
– parser:didStartElement:namespaceURI:qualifiedName:attributes:
OR
– parser:foundCharacters:

I don't know java, but the url's your are pointing at are .xsd files which are xml definition files. XML parsing on iOS is somewhat limited out of the box: NSXMLParser.
I strongly recommend one of the bazillion open source XML parsers. They're much more user friendly.

Well thanks to all for making the efforts to answer, but I got a nice library TouchXML that solves the purpose.