Develop Reference

Parsing files over 2.15 GB in Java using Kaitai Struct

I'm parsing large PCAP files in Java using Kaitai-Struct. Whenever the file size exceeds Integer.MAX_VALUE bytes I face an IllegalArgumentException caused by the size limit of the underlying ByteBuffer.
I haven't found references to this issue elsewhere, which leads me to believe that this is not a library limitation but a mistake in the way I'm using it.
Since the problem is caused by trying to map the whole file into the ByteBuffer I'd think that the solution would be mapping only the first region of the file, and as the data is being consumed map again skipping the data already parsed.
As this is done within the Kaitai Struct Runtime library it would mean to write my own class extending fom KatiaiStream and overwrite the auto-generated fromFile(...) method, and this doesn't really seem the right approach.
The auto-generated method to parse from file for the PCAP class is.
public static Pcap fromFile(String fileName) throws IOException {
return new Pcap(new ByteBufferKaitaiStream(fileName));
}
And the ByteBufferKaitaiStream provided by the Kaitai Struct Runtime library is backed by a ByteBuffer.
private final FileChannel fc;
private final ByteBuffer bb;
public ByteBufferKaitaiStream(String fileName) throws IOException {
fc = FileChannel.open(Paths.get(fileName), StandardOpenOption.READ);
bb = fc.map(FileChannel.MapMode.READ_ONLY, 0, fc.size());
}
Which in turn is limitted by the ByteBuffer max size.
Am I missing some obvious workaround? Is it really a limitation of the implementation of Katiati Struct in Java?

There are two separate issues here:
Running Pcap.fromFile() for large files is generally not a very efficient method, as you'll eventually get all files parsed into memory array at once. A example on how to avoid that is given in kaitai_struct/issues/255. The basic idea is that you'd want to have control over how you read every packet, and then dispose of every packet after you've parsed / accounted it somehow.
2GB limit on Java's mmaped files. To mitigate that, you can use alternative RandomAccessFile-based KaitaiStream implementation: RandomAccessFileKaitaiStream — it might be slower, but it should avoid that 2GB problem.

This library provides a ByteBuffer implementation which uses long offset. I haven't tried this approach but looks promising. See section Mapping Files Bigger than 2 GB
http://www.kdgregory.com/index.php?page=java.byteBuffer
public int getInt(long index)
{
return buffer(index).getInt();
}
private ByteBuffer buffer(long index)
{
ByteBuffer buf = _buffers[(int)(index / _segmentSize)];
buf.position((int)(index % _segmentSize));
return buf;
}
public MappedFileBuffer(File file, int segmentSize, boolean readWrite)
throws IOException
{
if (segmentSize > MAX_SEGMENT_SIZE)
throw new IllegalArgumentException(
"segment size too large (max " + MAX_SEGMENT_SIZE + "): " + segmentSize);
_segmentSize = segmentSize;
_fileSize = file.length();
RandomAccessFile mappedFile = null;
try
{
String mode = readWrite ? "rw" : "r";
MapMode mapMode = readWrite ? MapMode.READ_WRITE : MapMode.READ_ONLY;
mappedFile = new RandomAccessFile(file, mode);
FileChannel channel = mappedFile.getChannel();
_buffers = new MappedByteBuffer[(int)(_fileSize / segmentSize) + 1];
int bufIdx = 0;
for (long offset = 0 ; offset < _fileSize ; offset += segmentSize)
{
long remainingFileSize = _fileSize - offset;
long thisSegmentSize = Math.min(2L * segmentSize, remainingFileSize);
_buffers[bufIdx++] = channel.map(mapMode, offset, thisSegmentSize);
}
}
finally
{
// close quietly
if (mappedFile != null)
{
try
{
mappedFile.close();
}
catch (IOException ignored) { /* */ }
}
}
}

Calculate memory taken by a process(job) using OSHI lib

In my application I get the estimated memory taken by the process from another application. But I am looking to get the exact memory required by the processor to run.
As I am searching online on how to get the correct memory required by the process, I found oshi lib does that. But I didn't find the the way to implement the solution. Can anyone please help me?
OSHI lib: https://github.com/oshi/oshi
FYI: We use OSHI lib to get the systemInfo, hardware, os, centralProcessor and global memory. Below is the code snippet.
oshi.SystemInfo systemInfo = new oshi.SystemInfo();
this.hal = systemInfo.getHardware();
this.os = systemInfo.getOperatingSystem();
this.centralProcessor = this.hal.getProcessor();
this.globalMemory = this.hal.getMemory();

perhaps retrieve the memory usage from the OSProcess class:
OSProcess process = new SystemInfo().getHardware().getOperatingSystem().getProcess(myPid);
process.getVirtualSize();
process.getResidentSetSize();

public static void memoryUtilizationPerProcess(int pid) {
/**
* Resident Size : how much memory is allocated to that process and is in RAM
*/
OSProcess process;
SystemInfo si = new SystemInfo();
OperatingSystem os = si.getOperatingSystem();
process = os.getProcess(pid);
oshi.hardware.GlobalMemory globalMemory = si.getHardware().getMemory();
long usedRamProcess = process.getResidentSetSize();
long totalRam = globalMemory.getTotal();
double res1 = (double) ((usedRamProcess*100)/totalRam);
System.out.println("\nMemory Usage :");
System.out.println("Memory(Ram Used/Total Mem)="+res1+"%");
System.out.println("Resident Size: "+humanReadableByteCountBin(usedRamProcess));
System.out.println("Total Size: "+humanReadableByteCountBin(totalRam));
}
public static String humanReadableByteCountBin(long bytes) {
long absB = bytes == Long.MIN_VALUE ? Long.MAX_VALUE : Math.abs(bytes);
if (absB < 1024) {
return bytes + " B";
}
long value = absB;
CharacterIterator ci = new StringCharacterIterator("KMGTPE");
for (int i = 40; i >= 0 && absB > 0xfffccccccccccccL >> i; i -= 10) {
value >>= 10;
ci.next();
}
value *= Long.signum(bytes);
return String.format("%.1f %ciB", value / 1024.0, ci.current());
}

JnetPcap: reading from offline file very slow

I'm building a sort of custom version of wireshark with jnetpcap v1.4r1425. I just want to open offline pcap files and display them in my tableview, which works great except for the speed.
The files I open are around 100mb with 700k packages.
public ObservableList<Frame> readOfflineFiles1(int numFrames) {
ObservableList<Frame> frameData = FXCollections.observableArrayList();
if (numFrames == 0){
numFrames = Pcap.LOOP_INFINITE;
}
final StringBuilder errbuf = new StringBuilder();
final Pcap pcap = Pcap.openOffline(FileAddress, errbuf);
if (pcap == null) {
System.err.println(errbuf); // Error is stored in errbuf if any
return null;
}
JPacketHandler<StringBuilder> packetHandler = new JPacketHandler<StringBuilder>() {
public void nextPacket(JPacket packet, StringBuilder errbuf) {
if (packet.hasHeader(ip)){
sourceIpRaw = ip.source();
destinationIpRaw = ip.destination();
sourceIp = org.jnetpcap.packet.format.FormatUtils.ip(sourceIpRaw);
destinationIp = org.jnetpcap.packet.format.FormatUtils.ip(destinationIpRaw);
}
if (packet.hasHeader(tcp)){
protocol = tcp.getName();
length = tcp.size();
int payloadOffset = tcp.getOffset() + tcp.size();
int payloadLength = tcp.getPayloadLength();
buffer.peer(packet, payloadOffset, payloadLength); // No copies, by native reference
info = buffer.toHexdump();
} else if (packet.hasHeader(udp)){
protocol = udp.getName();
length = udp.size();
int payloadOffset = udp.getOffset() + udp.size();
int payloadLength = udp.getPayloadLength();
buffer.peer(packet, payloadOffset, payloadLength); // No copies, by native reference
info = buffer.toHexdump();
}
if (packet.hasHeader(payload)){
infoRaw = payload.getPayload();
length = payload.size();
}
frameData.add(new Frame(packet.getCaptureHeader().timestampInMillis(), sourceIp, destinationIp, protocol, length, info ));
//System.out.print(i+"\n");
//i=i+1;
}
};
pcap.loop(numFrames, packetHandler , errbuf);
pcap.close();
return frameData;
}
This code is very fast for the first maybe 400k packages, but after that it slows down a lot. It needs around 1 minute for the first 400k packages and around 10 minutes for the rest. What is the issue here?
It's not that the list is getting too timeconsuming to work with is it? the listmethod add is O(1), isnt it?
I asked about this on the official jnetpcap forums too but it's not very active.
edit:
turn out it slows down massively because of the heap usage. Is there a way to reduce this?

As the profiler showed you, you're running low on memory and it starts to slow down.
Either give more memory with -Xmx or don't load all the packets into memory at once.

How to edit a microsoft window spacing from a java application?

i'm trying to implement steganography's word shifting coding protocol on a microsoft word report using java application. Basicly, it uses an existing report and edit it's spacing to put some secret data. If it's wider, then its 1 bit data. And if it's narrower, then it's 0 bit data. So i wonder what kind of library should i have to start constructing this java app or if java doesn't support this kind of comunication with ms-word what kind language of programming should i use, thank you for your time.

I would recommend using C# and the Microsoft.Office.Interop.Word. You can use the free Visual Studio Community version (https://www.visualstudio.com/products/visual-studio-community-vs), create a console application and add a reference for the interop namespace (in project explorer, right click on references, add reference: COM->Microsoft Word 16.0 Object Library).
Simple example:
namespace WordShiftingExample
{
class Program
{
private static int[] getSpaces(string text)
{
System.Collections.ArrayList list = new System.Collections.ArrayList();
int index = 0;
while (index != text.LastIndexOf(" "))
{
index = text.IndexOf(" ", index + 1);
list.Add(index);
}
return list.ToArray(typeof(int)) as int[];
}
static void Main(string[] args)
{
try
{
Microsoft.Office.Interop.Word.Application winword = new Microsoft.Office.Interop.Word.Application();
winword.ShowAnimation = false;
winword.Visible = false;
object missing = System.Reflection.Missing.Value;
Microsoft.Office.Interop.Word.Document document = winword.Documents.Add(ref missing, ref missing, ref missing, ref missing);
float zero = 0.1F;
float one = 0.15F;
document.Content.Text = "This is a test document.";
//set word-spacing for first two spaces
int[] spaces = getSpaces(document.Content.Text);
document.Range(spaces[0], spaces[0]+1).Font.Spacing=zero;
document.Range(spaces[1], spaces[1]+1).Font.Spacing = one;
//read word-spacing for first two spaces
System.Diagnostics.Debug.WriteLine(document.Range(spaces[0], spaces[0]+1).Font.Spacing); // prints 0.1
System.Diagnostics.Debug.WriteLine(document.Range(spaces[1], spaces[1]+1).Font.Spacing); // prints 0.15
//Save the document
object filename = System.Environment.GetEnvironmentVariable("USERPROFILE")+"\\temp1.docx";
document.SaveAs2(ref filename);
document.Close(ref missing, ref missing, ref missing);
document = null;
winword.Quit(ref missing, ref missing, ref missing);
winword = null;
}
catch (Exception ex)
{
System.Diagnostics.Debug.WriteLine(ex.StackTrace);
}
}
}
}

Get device configuration of android mobile

Is there a function or command to get the basic device configurations of android device? Like the RAM size,OS version,number of processor cores, etc..

All of the device information you can get below the ways. Hope for help.
Get CPU Core of device
/**
* Gets the number of cores available in this device, across all processors.
* Requires: Ability to peruse the filesystem at "/sys/devices/system/cpu"
* #return The number of cores, or 1 if failed to get result
*/
private int getNumCores() {
//Private Class to display only CPU devices in the directory listing
class CpuFilter implements FileFilter {
#Override
public boolean accept(File pathname) {
//Check if filename is "cpu", followed by a single digit number
if(Pattern.matches("cpu[0-9]+", pathname.getName())) {
return true;
}
return false;
}
}
try {
//Get directory containing CPU info
File dir = new File("/sys/devices/system/cpu/");
//Filter to only list the devices we care about
File[] files = dir.listFiles(new CpuFilter());
//Return the number of cores (virtual CPU devices)
return files.length;
} catch(Exception e) {
//Default to return 1 core
return 1;
}
}
Get Total RAM of device
public static String getTotalRAM() {
RandomAccessFile reader = null;
String load = null;
try {
reader = new RandomAccessFile("/proc/meminfo", "r");
load = reader.readLine();
} catch (IOException ex) {
ex.printStackTrace();
} finally {
// Streams.close(reader);
}
return load;
}
Other information of device
String _OSVERSION = System.getProperty("os.version");
String _RELEASE = android.os.Build.VERSION.RELEASE;
String _DEVICE = android.os.Build.DEVICE;
String _MODEL = android.os.Build.MODEL;
String _PRODUCT = android.os.Build.PRODUCT;
String _BRAND = android.os.Build.BRAND;
String _DISPLAY = android.os.Build.DISPLAY;
String _CPU_ABI = android.os.Build.CPU_ABI;
String _CPU_ABI2 = android.os.Build.CPU_ABI2;
String _UNKNOWN = android.os.Build.UNKNOWN;
String _HARDWARE = android.os.Build.HARDWARE;
String _ID = android.os.Build.ID;
String _MANUFACTURER = android.os.Build.MANUFACTURER;
String _SERIAL = android.os.Build.SERIAL;
String _USER = android.os.Build.USER;
String _HOST = android.os.Build.HOST;

You can obtain most of the information you want from the /proc/cpuinfo file. Here is a tutorial on how to load and parse that file: How to get Cpu Information on android
Information about the RAM can be obtained from the /proc/meminfo file

Check out the android.os.Build class.
android.os.Build.HARDWARE
android.os.Build.DEVICE
android.os.Build.MODEL
android.os.Build.PRODUCT
android.os.Build.BOARD
android.os.Build.DISPLAY
etc...
To get the OS version number, you can try
System.getProperty("os.version");

You could look at android.os.Build to determine the phone make and model. From there have a lookup to look up the phone specs based on make and model.