Develop Reference

Why Java GzipOutputStream always write two bytes 03 00 before the trailer if flush() is called first then closed?

I found that when using Java.util.zip.GzipOutputStream, when I called flush() first then close(). The deflator always produce two bytes ('00' and '03' in hex) before writing the trailer. What does these two bytes mean? Why flush() does not flush all bytes in the compressor? I have already set the stream syncFlush to true. (In my use case I need to force flush the stream)
This is the test code I write.
File file = new File("gtest.gz");
OutputStream out = new FileOutputStream(file);
GZIPOutputStream gzipOutputStream = new GZIPOutputStream(out, true);
gzipOutputStream.write("Each gzip file generated by JAVA GzipOutputStream has byte '0003' before Trailer"
.getBytes());
gzipOutputStream.flush();
gzipOutputStream.close();
And I run hexdump -C gtest.gz get
00000000 1f 8b 08 00 00 00 00 00 00 00 72 4d 4c ce 50 48 |..........rML.PH|
00000010 af ca 2c 50 48 cb cc 49 55 48 4f cd 4b 2d 4a 2c |..,PH..IUHO.K-J,|
00000020 49 4d 51 48 aa 54 f0 72 0c 73 54 70 07 ca f9 97 |IMQH.T.r.sTp....|
00000030 96 14 94 96 04 97 14 a5 26 e6 2a 64 24 16 03 25 |........&.*d$..%|
00000040 4b 52 15 d4 0d 0c 0c 8c d5 15 92 52 d3 f2 8b 52 |KR.........R...R|
00000050 15 42 8a 12 81 06 14 01 00 00 00 ff ff 03 00 96 |.B..............|
00000060 ef 3b 5d 50 00 00 00 |.;]P...|
00000067
The last 8 bytes are trailer of Gzip and the two bytes before '03' '00' will always be produced whatever content I change.

Port 0 in DIIOP_IOR.TXT, how do I change it?

I am trying to login remotely into Domino with standalone Java program.
I have ncso.jar (and TrustedCerts.class) in classpath.
The DIIOP_IOR.TXT file is generated by the diiop task. If I copy the file contents directly into my program and try creating the session like this:
String ior = "IOR:....." // 404 bytes
Session session = NotesFactory.createSessionWithIOR(ior, "username", "password");
the result is:
org.omg.CORBA.COMM_FAILURE: java.net.ConnectException: connect: Address is invalid on local machine, or port is not valid on remote machine Host: poseidon.heeros.com Port: 0 vmcid: 0x0 minor code: 1 completed: No
The server name is valid but port 0 seems odd. I tried an online decoder at http://www2.parc.com/istl/projects/ILU/parseIOR/ and here is the result:
object key is <#048525651a-ec68-106c-eee0-007e2d2233b5#00LotusNOI#01#00#01>;
no trustworthy most-specific-type info; unrecognized ORB type;
reachable with IIOP 1.1 at host "poseidon.heeros.com", port 0
...which seems to confirm that the port is incorrect. I have specified the server URL in Internet Sites with an IIOP Site document but there is no field for port there.
Questions:
Where do I set the port that appears in diiop_ior.txt?
Which port should I specify? (I'm guessing 1352)
EDIT
Here is the result of tell diiop show config on server:
Dump of Domino IIOP (DIIOP) Configuration Settings
Full Server Name: CN=Afrodite/O=Heeros
Common Server Name: Afrodite/Heeros
Refresh Interval: 3 minutes
Host Full Name: poseidon.heeros.com
Host Short Name: poseidon
Host Address: 10.163.0.146
Public Host Name/Address: poseidon.heeros.com
TCP Port: 0 Disabled
SSL Port: 63149 Enabled
Initial Net Timeout: 120 seconds
Session Timeout: 60 minutes
Client Session Timeout: 62 minutes
Allow Ambiguous Names: True
Web Name Authentic: False
User Lookup View: ($Users)
Allow Database Browsing: False
Internet Sites: Enabled
Internet Site Name: Heeros
Site Config Loaded from: Domino IIOP and Web Internet Site documents
Site is Default: False
Site Public Host Name/Address: poseidon.heeros.com
Site IOR File: D:\Lotus\Domino\data\domino\html\diiop_ior.txt
Site SSL Key File: D:\Lotus\Domino\data\heeros.kyr
Site Java Key File: D:\Lotus\Domino\data\domino\java\TrustedCerts.class
Site TCP Name/Password Allowed: False
Site TCP Anonymous Allowed: False
Site SSL Name/Password Allowed: True
Site SSL Anonymous Allowed: True
Site Multi-Server Session Authentication: Enabled
Site Multi-Server Session Configuration: LtpaToken
Single Server Cookies: Disabled

It seems that the correct port number is 63148. It must be specified in Server Document at Ports --> Internet Ports --> DIIOP as "TCP/IP port number".
Additionally, in the IIOP Site document, the TCP Authentication must be allowed.

In my experience DIIOP doesn't use SSL/TLS at all. Only the DIIOP_IOR.TXT is downloaded via SSL/TLS. Capture your network traffic with Wireshark or something similar and monitor: port 63148 or port 63149. #lauri-laanti : Could you please test in your environment if the connection is encrypted with wireshark?
Wireshark Ourput: GIOP createSession with Username and Password (Blanked with X)
0000 00 50 56 69 f5 2b 00 50 56 c0 00 02 08 00 45 00 .PVi.+.PV.....E.
0010 00 c0 0d 06 40 00 80 06 bb ca c0 a8 58 01 c0 a8 ....#.......X...
0020 58 15 d2 e0 f6 ac ef b6 47 e8 13 10 53 10 50 18 X.......G...S.P.
0030 01 00 29 bb 00 00 47 49 4f 50 01 00 00 00 00 00 ..)...GIOP......
0040 00 8c 00 00 00 00 00 00 00 05 01 00 00 00 00 00 ................
0050 00 31 04 38 35 32 35 36 35 31 61 2d 65 63 36 38 .1.8525651a-ec68
0060 2d 31 30 36 63 2d 65 65 65 30 2d 30 30 37 65 32 -106c-eee0-007e2
0070 64 32 32 33 33 62 35 00 4c 6f 74 75 73 4e 4f 49 d2233b5.LotusNOI
0080 01 00 01 00 00 00 00 00 00 0e 63 72 65 61 74 65 ..........create
0090 53 65 73 73 69 6f 6e 00 00 00 00 00 00 00 00 00 Session.........
00a0 00 01 00 00 00 00 00 00 00 01 00 00 00 0f 00 00 ................
00b0 00 06 00 61 00 64 00 6d 00 69 00 6e 00 00 00 00 ...a.d.m.i.n....
00c0 00 06 00 XX XX XX XX XX XX XX XX XX XX 00 ...XXXXXXXXXX.
Java Code used:
_diiop_args = new String[]{"-ORBEnableSSLSecurity", "-HTTPEnableSSLSecurity"};
String ior = NotesFactory.getIOR(_diiop_host + ":" + _diiop_port,
_diiop_args, _user_name, _user_pass);
_session = NotesFactory.createSessionWithIOR(ior, _user_name, _user_pass);

Java SSL Streaming - splitted applicationdata

I try to send a byte[] () over a established SSL Connection (handshake etc is done).
The result: The byte[] is spitted into two packets (see debug below):
First packet: just the first byte of the application data (**01**) .
Second packet: the rest (fe db 01 00 ...) 650 Bytes
Is there a way to commit all application data bytes in one packet?
Stream to send 651 Bytes:
**01** fe db 01 00 00 02 83 3c 3f 78 6d 6c 20 76 65 72 73 69 6f 6e 3d 22 31 2e 30 22 20 65 6e 63 6f 64 69 6e 67 3d 22 75 73 2d 61 73 63 69 69 22 20 73 74 61 6e 64 61 6c 6f 6e 65 3d 22 6e 6f 22 3f 3e …
javax.net.debug output
Padded plaintext before ENCRYPTION: len = 32
0000: **01** 06 03 06 46 7F 7F AE D4 E8 30 5D B7 DB 3C 44 ....F.....0]..<D
0010: 02 08 C9 2A A1 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A ...*............
1, WRITE: TLSv1 Application Data, length = 32
[Raw write]: length = 37
0000: 17 03 01 00 20 B3 4E EE CE 5B 69 EC A5 4A 80 7F .... .N..[i..J..
0010: D6 03 35 AF 6A 7B 85 17 B7 46 A2 31 B2 EF 7E D0 ..5.j....F.1....
0020: EA 1B 67 7E ED ..g..
Padded plaintext before ENCRYPTION: len = 672
0000: FE DB 01 00 00 02 83 3C 3F 78 6D 6C 20 76 65 72 .......<?xml ver
0010: 73 69 6F 6E 3D 22 31 2E 30 22 20 65 6E 63 6F 64 sion="1.0" encod
0020: 69 6E 67 3D 22 75 73 2D 61 73 63 69 69 22 20 73 ing="us-ascii" s
0030: 74 61 6E 64 61 6C 6F 6E 65 3D 22 6E 6F 22 3F 3E tandalone="no"?>
[…]

Sun's impl comments:
By default, we counter chosen plaintext issues on CBC mode
ciphersuites in SSLv3/TLS1.0 by sending one byte of application
data in the first record of every payload, and the rest in
subsequent record(s). Note that the issues have been solved in
TLS 1.1 or later.
Experiment with SSLEngine.wrap( largePlainText ) shows that it produces 2 SSL records, the 1st record contains 1 byte of plain text, the 2nd record contains 15846 bytes of plain text.
The receiver API probably handle record-by-record, so it'll return 1 byte for the 1st read.
We can also observe this behavior in other SSL impls, e.g. HTTPS requests from web browsers.
OpenSSL inserts empty records against the attack. If the receiver is Java SSL socket, the input stream cannot return 0 bytes for read(), so the record is skipped. Other receivers may not be prepared for a 0-length record and may break.

The assumption you're making about reading the byte[] exactly as you write them on the other end is a classic TCP mistake. It's not actually specific to SSL/TLS, but could also happen with a TCP connection.
There is no guarantee in TCP (and in SSL/TLS) that the reader's buffer will be filled with the exact same packet length as the packets in the writer's buffer. All TCP guarantees is in-order delivery, so you'll eventually get all your data, but you have to treat it as a stream.
This is why protocols that use TCP rely on indicators and delimiters to tell the other end when to stop reading certain messages.
For example, HTTP 1.1 uses a blank line to indicate when the headers end, and it uses the Content-Length header to tell the recipient what entity length to expect (or chunked transfer encoding). SMTP also uses line returns and . at the end of a message.
If you're designing your own protocol, you need to define a way for the recipient to know when what you define as meaningful units of data are delimited. When you read the data, read such indicators, and fill in your read buffer until you get the amount of bytes you expect or until you find the delimiter that you've defined.

I had the same problem until I saw this page:
http://bugs.java.com/bugdatabase/view_bug.do?bug_id=7157903
So, I run the JVM with -Djsse.enableCBCProtection=false parameter and now the data is not splitted.
Best regards

How to detect Java string inside a .so file?

I'm recompiling a Java program, and I have the following method:
public static native String getMyString(String s);
getMyString is implemented inside mylibrary.so, and looks like follows:
; Scrambler::getMyString(char const*)
EXPORT _ZN9Scrambler18getMyStringEPKc
_ZN9Scrambler18getMyStringEPKc
LDR R1, =(mystring - 0x14FC8)
PUSH {R4-R6,LR}
ADD R1, PC ; "mystring"
MOV R4, R0
BLX strcasecmp
CMP R0, #0
BNE.W loc_150FC
which than ends up in
LDR R5, =(unk_6AE24 - 0x14FD8)
ADD R5, PC
The R5 seems to be the return value here.
The unk_6AE24 looks like follows:
27
68 5D 6D 06 3F 10 6D 26 32 10 56 12 06 3F 32 5D
12 27 26 4B 0D 5D 27 4B 6E 0D 2B 06 32 5D 68 24
3F 32 06 24 70 56 00 63 69 4C 1D 06 5A 7F 0C 34
1E 67 3B 63 32 5F 16 1D 34 5F 1E 32 42 69 18 49
3F 51 49 0C 1F 0C 0E 77 2F 44 69 7F 5A 0C 0B 34
0C 34 1C 24 32 5F 7F 00 00 00 00 1D 30 3B 5F 30
IDA suggests, that the string in question is 27 68 5D 6D or "'h]m", but I'm having my doubts. When I was using c++ ages ago, the strings used to end with 0, which would make for return value of 27 68 5D 6D 06 3F 10 6D 26 32 10 56 12 06 3F 32 5D 12 27 26 4B 0D 5D 27 4B 6E 0D 2B 06 32 5D 68 24 3F 32 06 24 70 56 . Or we could be talking about java string, which may have its own format.
What kind of string will getMyString return?

What kind of string will getMyString return?
It will return a java.lang.String, you can get that from the declaration.
The actual character data will (probably) be one redirection away from that. (a String is probably a pointer to a char array and a size). Keep in mind that in Java, a char is 16 bits.
What are you trying to do, anyway? Much easier to just set a breakpoint right after getMyString returns and look at the return value in a debugger.

Why might RvdProxy.getServices() incorrectly return an empty Array?

The following code snippet attempts to create a Tib DaemonManager connecting to a particular rvd, and then query for that rvd's services.
public static void main(String[] args) throws RuntimeException {
DaemonManager daemonManager = new DaemonManager("http://foo.com:7580");
if(daemonManager.getDaemonType() == DaemonManager.RVD) {
DaemonProxy daemonProxy = daemonManager.getDaemonProxy();
final RvdProxy rvdProxy = (RvdProxy) daemonProxy;
Service[] services = rvdProxy.getServices();
System.out.println(services.length); //prints 0
for (Service service : services) {
System.out.println(service.getNetwork());
}
}
}
This prints zero, even though the web interface for this rvd lists multiple available services. Why might this happen?
The daemon I am connecting to is running v 7.5.1 of the software, and the rvconfig.jar that I am using is from v 7.5.1 as well.
Is there a gotcha when using Tibco's DaemonManager that is causing me to come unstuck?

I used Wireshark to look at the traffic being sent between my RvdProxy and the RVD itself, and it looks like a lot of HTTP GET traffic. For example:
0000 00 1e 0b a4 d1 7c 00 12 d9 7c 8a bf 08 00 45 00 .....|.. .|....E.
0010 03 87 a4 58 40 00 3d 06 66 90 0a 09 14 15 0a 0a ...X#.=. f.......
0020 07 61 1d 9c bf 09 fe 1e d6 82 6e 77 b9 52 80 18 .a...... ..nw.R..
0030 00 1b 05 43 00 00 01 01 08 0a ac 1c 6f 67 07 b2 ...C.... ....og..
0040 86 1f 72 3d 23 30 30 38 30 38 30 3e 3c 66 6f 6e ..r=#008 080><fon
0050 74 20 66 61 63 65 3d 48 65 6c 76 65 74 69 63 61 t face=H elvetica
0060 2c 41 72 69 61 6c 20 73 69 7a 65 3d 32 20 63 6f ,Arial s ize=2 co
0070 6c 6f 72 3d 23 46 46 46 46 46 46 3e 3c 62 3e 43 lor=#FFF FFF><b>C
Sure enough, upon further inspection, it turns out that the requests being made by the proxy are to the web pages served by the Daemon itself - so the proxy API is just a screen-scraper.
But why am I getting no services if they appear on the web pages that are being screen-scraped?
Switch on debugging:
System.setProperty("com.tibco.tibrv.config.debug","classes-to-debug")
And you find that the pattern being used to extract the services from the web-page...
/services,GET~~~~~\
service_detail\\?(\\d+)>\\1</a>.*?size=2>(\\d+\\.\\d+\\.\\d+\\.\\d+)</td>.*?size=2>
(\\d+)</td>.*?size=2>(\\d+)</td>~~~~~\
...fails to match anything! In my case it is because my service networks are not of the form (\\d+\\.\\d+\\.\\d+\\.\\d+) but rather of the form ;(\\d+\\.\\d+\\.\\d+\\.\\d+) - note the leading semicolon. This subtle difference is the source of all of my problems!
This looks like a bug in the rvconfig jar - which needs to be raised with Tibco! :(