Develop Reference

How to properly parse a tcp packet in java?

I currently have a simple TCP server, which calls a function and passes it the binary TCP payload in the form of a byte array on every new incoming packet, what is the proper way to parse it?
I tried doing it by just slicing it into different byte arrays and processing them separately, but my packets include a variable-length datatype, which I am unable to split from the other data. It really seems like it's not what I should do and I think there is a better way to do it
readVarInt() is a function that parses the variable-length datatype and returns the result and the length of the unparsed datatype in an int[] array
PacketHeader is a class that that has size, id and body fields, id and size are already parsed and the body field contains everything else after the size and id bytes
int[] parsedProtocolVersion = Main.readVarInt(packetHeader.body);
System.out.println(parsedProtocolVersion[0] + ' ' + parsedProtocolVersion[1]);
int[] parsedServerAddressSize = Main.readVarInt( packetHeader.body.subList(parsedProtocolVersion[1], packetHeader.body.size()));
System.out.println("parsed 1 " + parsedServerAddressSize[1] + " " + parsedServerAddressSize[0]);
String parsedServerAddress = String.valueOf(packetHeader.body.subList(parsedServerAddressSize[1], parsedServerAddressSize[0]));
System.out.println("parsed 2");
int parsedServerPort = Integer.parseUnsignedInt((packetHeader.body.subList(parsedServerAddressSize[1]+parsedProtocolVersion[1], parsedServerAddressSize[1]+parsedProtocolVersion[1]+2).toString()));
System.out.println("parsed 3");
int[] parsedNextState = Main.readVarInt(packetHeader.body.subList(parsedServerAddressSize[1]+parsedProtocolVersion[1]+2, packetHeader.body.size()));
System.out.println("parsed 4");
return String.format("%d %s %d %d",parsedProtocolVersion[0], parsedServerAddress, parsedServerPort, parsedNextState[0]);

Reading a specific data type from a line in a text file

I'm trying to work out the average number of items from a list of insurance policies, how do I get the value of only the items(int) from a text file?
Here is an example of data in the text file:
20-Jul-2017 EQ123B 3 40000 30 A 5389 l a
20-Jul-2017 ED423A 2 40000 30 A 5389 k d
31-Jul-2017 ZD123V 4 40000 30 A 5389 s c
Each line represents data for a different insurance policy, with the third column being the amount of items to be insured. I had planned to get the average number of items per policy by getting the total amount of items in the file and dividing that by the number of policies.
Here is my code so far:
try{
int numOfPolicies = 0;
try (Scanner file = new Scanner(new FileReader("policy.txt"))) {
//loop through the file counting each line. Each line represents a policy
while(file.hasNextLine()){
numOfPolicies++;
file.nextLine();
}
}
System.out.println("Total Number of Policies: " + numOfPolicies);
}
catch(FileNotFoundException e){
System.out.println("File not found");
}
As you can see I have already got the number of policies in the file. How do I read only the number of items from each line, and store this in a variable?

If your lines follow this format (whitespace used only as separator) :
20-Jul-2017 EQ123B 3 40000 30 A 5389 l a
To retrieve 3, you could capture the number between the second and the third whitespace.
You could use the String.split() method with the \\s regex and a limit of 4 as you don't care token after the number of policies :
String[] split = file.nextLine().split("\\s+", 4);
You will get the following token :
20-Jul-2017
EQ123B
3
40000 30 A 5389 l a
You could get so the third token :
String number = split[2];

Processing: Write HTML file from arduino input

I'm trying to make a plant monitor using an Arduino and processing.
Processing writes an html file based on the sensor input by the Arduino.
WinSCP is monitoring the file created for changes and directly uploads trough FTP when the file has changed.
The Arduino is sending the following to processing via serial:
45
0
31
40
x
Using the following code in processing I write an html page with this data:
import processing.serial.*;
Serial myPort;
String dataReading = "";
int lol = 0;
String string0 = "<h1>Jurze Plants <img src=\"https://html-online.com/editor/tinymce/plugins/emoticons/img/smiley-laughing.gif\" alt=\"laughing\" /></h1>";
String string1 = "Moisture Level: ";
String string2 = " %<br> Motorstate: ";
String string3 = "<br> Temperature: ";
String string4 = " °C<br> Humidity: ";
String string5 = "%<br>";
void setup() {
size(500, 500);
myPort = new Serial(this, "COM4", 9600);
myPort.bufferUntil('x');
}
void draw() {
}
String [] dataOutput = {};
void serialEvent(Serial myPort) {
dataReading = myPort.readString();
if (dataReading!=null) {
dataOutput = split(dataReading, '\n');
String [] tempfile = {string0,string1,dataOutput[1],string2,dataOutput[2],string3,dataOutput[3],string4,dataOutput[4],string5 };
println("saving to html file...");
saveStrings("data/index.html",tempfile);
}
}
The html code I get the first time is :
<h1>Jurze Plants <img src="https://html-online.com/editor/tinymce/plugins/emoticons/img/smiley-laughing.gif" alt="laughing" /></h1>
Moisture Level: 46 %<br>
Motorstate: 0 <br>
Temperature:31.00 °C <br>
Humidity: 35.00% <br>
Though, after it gets the data from the Arduino for the second time it looks like this:
<h1>Jurze Plants <img src="https://html-online.com/editor/tinymce/plugins/emoticons/img/smiley-laughing.gif" alt="laughing" /></h1>
Moisture Level: %<br>
Motorstate: 46 <br>
Temperature:0 °C <br>
Humidity: 31.00% <br>
I guess there is something wrong with the array?
Any help would be highly appreciated! :D

Time to debug your code! (We can't really do this for you, since we don't have your Arduino.)
Step 1: In your serialEvent() function, use the println() function to print out the value of dataReading. Is the value what you expect?
Step 2: Print out the value of dataOutput. Is that what you expect? Print out each index. Are they all what you expect? Check for extra spaces and control characters.
Step 3: Are the indexes what you expect them to be? I see you're starting with index 1 instead of index 0. Is that what you meant to do?
The point is, you have to print out the values of every variable to make sure they're what you expect. When you find the variable with the wrong value, you can trace back through your code to figure out exactly what's happening.

How to find if HL7 Segment has ended or not if Carriage return is not present

I am working on a tool which will construct a HL7 message in following Way :
Message will start with : 0B
Segment will end with : OD
And Message will end with : 1C0D
So, here i have reached so far, i am able to add OB and add 1C0D in the end of the HL7 Message. I am also able to add OD before at the end of the segment. I am accomplishing with the of code where i will check if Character before Segment name is 0D or not.
But the issue is if text in the message is somewhat like this ...PID| my code will add 0D before PID| which is not correct it should check if its the start of the segment or not.
Please help if someone has worked on similar requirement.
Link to my code is :
Arraylist Sublist IndexOutOfBounds Exception

I had some time to look at this problem. As far as I could understand, you have some piece of code that generates the HL7v2 segments for you and then you want to create a message with the following delimiters:
Segment delimiter: 0x0D (or 13 in ASCII), which is the Carriage Return. It's the segment separator, as per HL7v2 standard;
Message start delimiter: 0x0B (ASCII 11 - Vertical Tab);
Message finish delimiter: 0x1C0D. My guess is that this value is supposed to be the concatenation of 0x1C (ASCII 28 - File Separator) and 0x0D (ASCII 13 - Carriage Return).
With #1 you get HL7v2 messages standard-compliant. With #2 and #3 you are able to clearly define delimiters for the message so that it can be processed and parsed later by some custom processor.
So I took a shot writing some simple code and here's the result:
public class App
{
public static void main( String[] args ) throws Exception
{
String msg = "MSH|^~\\&|HIS|RIH|EKG|EKG|199904140038||ADT^A01||P|2.5" +
"PID|0001|00009874|00001122|A00977|SMITH^JOHN^M|MOM|19581119|F|NOTREAL^LINDA^M|C|564 SPRING ST^^NEEDHAM^MA^02494^US" +
"AL1||SEV|001^POLLEN";
String[] segments = msg.split("(?=PID|AL1)");
System.out.println("Initial message:");
for (String s : segments)
System.out.println(s);
byte hexStartMessage = 0x0B;
byte hexFinishMessage1 = 0x1C;
byte hexFinishMessage2 = 0x0D;
byte hexFinishSegment = 0x0D;
String finalMessage = Byte.toString(hexStartMessage) +
intersperse(segments, hexFinishSegment) +
Byte.toString(hexFinishMessage1) +
Byte.toString(hexFinishMessage2);
System.out.println("\nFinal message:\n" + finalMessage);
}
public static String intersperse(String[] segments, byte delimiter) throws UnsupportedEncodingException {
// uncomment this line if you wish to show the delimiter in the output
//System.out.printf("Byte Delimiter: %s", String.format("%04x", (int)delimiter));
StringBuilder sb = new StringBuilder();
String defaultDelimiter = "";
for (String segment : segments) {
sb.append(defaultDelimiter).append(segment);
defaultDelimiter = Byte.toString(delimiter);
}
return sb.toString();
}
}
I picked up a simple HL7v2 message and I splitted it in segments, according to the segments (name) used in the message, with the help of a regex with a lookahead strategy. This means that, for your messages you'll need to know the segments that are going to be used (you can get that from the standard).
I then interspersed the segment delimiter between each segment (at its end) and added the message start and end delimiters. In this case, for the message end delimiters, I used the 0x1C and 0x0D values separated, but if you need to use a single value then you only need to change the final appends.
Here's the output:
Initial message:
MSH|^~\&|HIS|RIH|EKG|EKG|199904140038||ADT^A01||P|2.5
PID|0001|00009874|00001122|A00977|SMITH^JOHN^M|MOM|19581119|F|NOTREAL^LINDA^M|C|564 SPRING ST^^NEEDHAM^MA^02494^US
AL1||SEV|001^POLLEN
Final message:
11MSH|^~\&|HIS|RIH|EKG|EKG|199904140038||ADT^A01||P|2.5
PID|0001|00009874|00001122|A00977|SMITH^JOHN^M|MOM|19581119|F|NOTREAL^LINDA^M|C|564 SPRING ST^^NEEDHAM^MA^02494^US
AL1||SEV|001^POLLEN2813
As you see, the final message begins with value 11 (0x0B) and ends with 28 (0x1C) and 13 (0x0D). The 13 (0x0D) at the end of each segment is not shown because Java's System.out.println() recognizes it as being the '\r' character and starts a new line because I'm running in Mac OS X. If you try to intersperse the segments with any other character (ex: 0x25 = '%') you'll notice that the final message is printed in a single line:
11MSH|^~\&|HIS|RIH|EKG|EKG|199904140038||ADT^A01||P|2.5%PID|0001|00009874|00001122|A00977|SMITH^JOHN^M|MOM|19581119|F|NOTREAL^LINDA^M|C|564 SPRING ST^^NEEDHAM^MA^02494^US%AL1||SEV|001^POLLEN2813
If I run in Ubuntu, you get to see the message in one line with the segment delimiter:
11MSH|^~\&|HIS|RIH|EKG|EKG|199904140038||ADT^A01||P|2.513PID|0001|00009874|00001122|A00977|SMITH^JOHN^M|MOM|19581119|F|NOTREAL^LINDA^M|C|564 SPRING ST^^NEEDHAM^MA^02494^US13AL1||SEV|001^POLLEN2813

SMPP Submit Long Message and message split

We are using SMPP cloud-hopper library to SMS long long messages to SMS gateway Innovativetxt.com, but it seems like when we split following the long message TO 140 bytes each part. The number of characters in each message gets to 134 character.
However industry standard is kind of 153 character shall be for each part of GSM Encoded long message. Is it something wrong we are doing by having only 134 character when we split via 140 byte? If we trying to submit greater than 140 bytes message, the gateway provider rejects it with message oversized message body.
Shall be split the message to 153 character each to sbumit to SMSC, instead spiting the messages via 140 bytes each.
What is the best way to split long message? By message size i.e 140 bytes or message characters count?
Anyone faced same issues via cloudhopper or other Java-based Library what we shall do.

It's a common confusion. You are doing everything right. Message lengths may be 160 chars (7-bit GSM 03.38), 140 chars (8-bit Latin), 70 chars (16-bit UCS-2). Notice: 160 * 7 == 140 * 8 == 70 * 16.
When you split a long message additional info like total parts number and part index is stored in the message body, so-called User Data Header (UDH). This header also takes place. So, with UDH you left with 153 GSM chars (7-bit), 134 chars/bytes (8-bit) payload or 67 2bytes-unicode chars (16-bit)
See also http://www.nowsms.com/long-sms-text-messages-and-the-160-character-limit
The UDH is 6 bytes long for Contatenated message 8-bit as in your case.
UDH structure
0x05: Length of UDH (5 bytes to follow)
0x00: Concatenated message Information Element (8-bit reference number)
0x03: Length of Information Element data (3 bytes to follow)
0xXX: Reference number for this concatenated message
0xYY: Number of fragments in the concatenated message
0xZZ: Fragment number/index within the concatenated message
Total message length, bits: 160*7 = 140*8 = 1120
UDH length, bits: 6*8 = 48
Left payload, bits: 1120-48 = 1072
For GSM 03.38 you get 1072/7 = 153 GSM (7-bit) chars + 1 filling unused bit.
For Latin you get 1072/8 = 134 (8-bit) chars.
For UCS-2 you get 1072/16 = 67 (16-bit) chars.
As you can see 153 GSM chars equals to 134 bytes minus 1 bit. Probably these 134 chars is what Java reports you. But once you split your long text message you end up with a binary message containing both text and UDH. And you should treat the message as binary. I suggest you to make binary dumps out of the resulting parts and investigate them.

Hello See sample method for sending both short or long SMS
public synchronized String sendSMSMessage(String aMessage,
String aSentFromNumber, String aSendToNumber,
boolean requestDeliveryReceipt) {
byte[] textBytes = CharsetUtil.encode(aMessage,
CharsetUtil.CHARSET_ISO_8859_1);
try {
SubmitSm submitMsg = new SubmitSm();
// add delivery receipt if enabled.
if (requestDeliveryReceipt) {
submitMsg
.setRegisteredDelivery(SmppConstants.REGISTERED_DELIVERY_SMSC_RECEIPT_REQUESTED);
}
submitMsg.setSourceAddress(new Address((byte) 0x03, (byte) 0x00,
aSentFromNumber));
submitMsg.setDestAddress(new Address((byte) 0x01, (byte) 0x01,
aSendToNumber));
if (textBytes != null && textBytes.length > 255) {
submitMsg.addOptionalParameter(new Tlv(SmppConstants.TAG_MESSAGE_PAYLOAD, textBytes, "message_payload"));
}else{
submitMsg.setShortMessage(textBytes);
}
logger.debug("About to send message to " + aSendToNumber
+ ", Msg is :: " + aMessage + ", from :: "
+ aSentFromNumber);
SubmitSmResp submitResp = smppSession.submit(submitMsg, 15000);
logger.debug("Message sent to " + aSendToNumber
+ " with message id " + submitResp.getMessageId());
return submitResp.getMessageId();
} catch (Exception ex) {
logger.error("Exception sending message [Msg, From, To] :: ["
+ aMessage + ", " + aSentFromNumber + ", " + aSendToNumber,
ex);
}
logger.debug("Message **NOT** sent to " + aSendToNumber);
return "Message Not Submitted to " + aSendToNumber;
}