I'm working on a Java-based IRC client as a way to learn both Java and more about writing networked applications.
The client I've designed mostly works, except when I post a message. The message goes through alright, but only up to the first space. I've tried everything: I've dumped my text into a StringArray, into a byte array, in a loop. But each time, only the first word of the intended message gets posted.
Here's the part of the code that I believe is relevant, although I'm happy to post the entire code if necessary (it's only a few hundred lines, and I can cut out the unimportant parts):
public void send(String msg) throws UnsupportedEncodingException {
if ( ! msg.startsWith("/")) {
msg = ("PRIVMSG " + chan + " " + msg);
// DEBUG confirm that msg == command+chan+userText
System.out.println(msg);
} else if ( msg.toUpperCase().startsWith("/JOIN ")) {
// System.out.println("\nJoin mode");
chan = msg.substring(6);
msg = (msg.toUpperCase().substring(1) + "\r\n");
} else { // some other command
msg = (msg.toUpperCase().substring(1) + "\r\n");
}
System.out.println(msg);
ostream.print(msg + " \r\n"); // doesn't work
ostream.flush();
}
}
I have also tried this sort of thing:
CRS = msg.split("\\s+");
CharSequence chars = msg;
ostream.printf( "%s,\r\n", msg); // doesn't work
ostream.print( String.join(" ", CRS) + "\r\n" ); // nope
And this:
ostream.append(chars);
ostream.append("\r\n"); // nope
I've also tried all of the above with byte arrays.
This sort of thing, however, does work:
// this, however, works as expected
void pong(String ping) {
String msg = "PONG " + ping;
byte[] bs = null;
bs = (msg.substring(1) + "\r\n").getBytes();
try {
ostream.write(bs);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
I've also tried changing my OutputStream object (the connection to the IRC server) to a PrintStream. Same results.
What about OutputStream am I not comprehending?
The IRC protocol requires you to escape messages that contain spaces with a preceding colon (":"). I think your code actually works, you've just not implemented the IRC protocol correctly.
Try making your PRIVMSG command:
msg = ("PRIVMSG " + chan + " :" + msg);
Only the first word is appearing because the IRC server ignores the trailing content after the first space. A valid message should look like:
PRIVMSG #target :Hello, world
I generate two PDFs from a jasper passing the same parameters to both of them, when I hash these two PDFs their hash is different which I think is the correct result, because eventough they have the same content they were created at different times.
But when I convert these two PDFs to PDF/A their sha1 hash is equal.
Can someone help me with this issue? How is this possible?
EDIT:
private static final String OUTPUT_FORMAT = "fi_pdfa";
public void convert(String exeFullPath, String inputFile,
String outputFile, String fontDirectory) {
String[] execParams = new String[4];
execParams[0] = exeFullPath;
execParams[1] = "\"inputpath_u=" + base64Encode(inputFile) + "\"";
execParams[2] = "\"outputpath_u=" + base64Encode(outputFile) + "\"";
execParams[3] = "\"outputid=" + OUTPUT_FORMAT + "\"";
// execParams[1] = "inputpath_u=\"" + base64Encode(inputFile) + "\"";
// execParams[2] = "outputpath_u=\"" + base64Encode(outputFile) + "\"";
// execParams[3] = "outputid=" + OUTPUT_FORMAT;
// execParams[4] = "fontdirectory=\"" + fontDirectory + "\"";
Runtime runtime = Runtime.getRuntime();
Process process = null;
ExportStatusCode statusCode = null;
ExportShutdownHook shutdownHook = null;
try {
process = runtime.exec(execParams);
// Install a shutdown hook to perform cleanup if we're interrupted.
shutdownHook = new ExportShutdownHook(process);
runtime.addShutdownHook(shutdownHook);
process.waitFor();
InputStream is = process.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
String line = null;
while ((line = reader.readLine()) != null)
System.out.println(line);
reader.close();
runtime.removeShutdownHook(shutdownHook);
shutdownHook.finished();
statusCode = new ExportStatusCode(process.exitValue());
} catch (IOException ex) {
log.error(ex);
if (shutdownHook != null) {
runtime.removeShutdownHook(shutdownHook);
shutdownHook.finished();
}
statusCode = ExportStatusCode.SCCERR_JAVA_IO_ERROR;
} catch (InterruptedException ex) {
log.error(ex);
runtime.removeShutdownHook(shutdownHook);
shutdownHook.finished();
process.destroy();
statusCode = ExportStatusCode.SCCERR_JAVA_INTERRUPTED;
}
}
Based on the description, I don't believe it is an issue with your code so much as with the PDF/A converter being used.
There are some PDF features that require or suggest a timestamp be used (e.g. digital signatures, annotations, PieceInfo dictionaries), and there are some features that require or suggest a unique identifier be used.
The PDF Reference strongly recommends that PDF Producers generate File Identifiers (stored in an array in the trailer dictionary), so even if the rest of the files were created identically, the ID entries in the trailer dictionary should be unique and cause the hash value for the files to be different...which strongly indicates that the ID array is either not being populated by the PDF Converter or that the generation of the identifiers is flawed; This can be verified by opening the two PDFs in a Hex Viewer and scrolling to the bottom of the files.
Disclaimer: Once upon a time, I provided technical support for the OutsideIn products.
i have a performance problem when trying to create a csv file starting from another csv file.
this is how the original file looks:
country,state,co,olt,olu,splitter,ont,cpe,cpe.latitude,cpe.longitude,cpe.customer_class,cpe.phone,cpe.ip,cpe.subscriber_id
COUNTRY-0001,STATE-0001,CO-0001,OLT-0001,OLU0001,SPLITTER-0001,ONT-0001,CPE-0001,28.21487,77.451775,ALL,SIP:+674100002743#IMS.COMCAST.NET,SIP:E28EDADA06B2#IMS.COMCAST.NET,CPE_SUBSCRIBER_ID-QHLHW4
COUNTRY-0001,STATE-0002,CO-0002,OLT-0002,OLU0002,SPLITTER-0002,ONT-0002,CPE-0002,28.294018,77.068924,ALL,SIP:+796107443092#IMS.COMCAST.NET,SIP:58DD999D6466#IMS.COMCAST.NET,CPE_SUBSCRIBER_ID-AH8NJQ
potentially it could be millions of lines like this, i have detected the problem with 1.280.000 lines.
this is the algorithm:
File csvInputFile = new File(csv_path);
int blockSize = 409600;
brCsvInputFile = new BufferedReader(frCsvInputFile, blockSize);
String line = null;
StringBuilder sbIntermediate = new StringBuilder();
skipFirstLine(brCsvInputFile);
while ((line = brCsvInputFile.readLine()) != null) {
createIntermediateStringBuffer(sbIntermediate, line.split(REGEX_COMMA));
}
private static void skipFirstLine(BufferedReader br) throws IOException {
String line = br.readLine();
String[] splitLine = line.split(REGEX_COMMA);
LOGGER.debug("First line detected! ");
createIndex(splitLine);
createIntermediateIndex(splitLine);
}
private static void createIndex(String[] splitLine) {
LOGGER.debug("START method createIndex.");
for (int i = 0; i < splitLine.length; i++)
headerIndex.put(splitLine[i], i);
printMap(headerIndex);
LOGGER.debug("COMPLETED method createIndex.");
}
private static void createIntermediateIndex(String[] splitLine) {
LOGGER.debug("START method createIntermediateIndex.");
com.tekcomms.c2d.xml.model.v2.Metadata_element[] metadata_element = null;
String[] servicePath = newTopology.getElement().getEntity().getService_path().getLevel();
if (newTopology.getElement().getMetadata() != null)
metadata_element = newTopology.getElement().getMetadata().getMetadata_element();
LOGGER.debug(servicePath.toString());
LOGGER.debug(metadata_element.toString());
headerIntermediateIndex.clear();
int indexIntermediateId = 0;
for (int i = 0; i < servicePath.length; i++) {
String level = servicePath[i];
LOGGER.debug("level is: " + level);
headerIntermediateIndex.put(level, indexIntermediateId);
indexIntermediateId++;
// its identificator is going to be located to the next one
headerIntermediateIndex.put(level + "ID", indexIntermediateId);
indexIntermediateId++;
}
// adding cpe.latitude,cpe.longitude,cpe.customer_class, it could be
// better if it would be metadata as well.
String labelLatitude = newTopology.getElement().getEntity().getLatitude();
// indexIntermediateId++;
headerIntermediateIndex.put(labelLatitude, indexIntermediateId);
String labelLongitude = newTopology.getElement().getEntity().getLongitude();
indexIntermediateId++;
headerIntermediateIndex.put(labelLongitude, indexIntermediateId);
String labelCustomerClass = newTopology.getElement().getCustomer_class();
indexIntermediateId++;
headerIntermediateIndex.put(labelCustomerClass, indexIntermediateId);
// adding metadata
// cpe.phone,cpe.ip,cpe.subscriber_id,cpe.vendor,cpe.model,cpe.customer_status,cpe.contact_telephone,cpe.address,
// cpe.city,cpe.state,cpe.zip,cpe.bootfile,cpe.software_version,cpe.hardware_version
// now i need to iterate over each Metadata_element belonging to
// topology.element.metadata
// are there any metadata?
if (metadata_element != null && metadata_element.length != 0)
for (int j = 0; j < metadata_element.length; j++) {
String label = metadata_element[j].getLabel();
label = label.toLowerCase();
LOGGER.debug(" ==label: " + label + " index_pos: " + j);
indexIntermediateId++;
headerIntermediateIndex.put(label, indexIntermediateId);
}
printMap(headerIntermediateIndex);
LOGGER.debug("COMPLETED method createIntermediateIndex.");
}
Reading the entire dataset, 1.280.000 lines take 800 ms! so the problem is in this method
private static void createIntermediateStringBuffer(StringBuilder sbIntermediate, String[] splitLine) throws ClassCastException,
NullPointerException {
LOGGER.debug("START method createIntermediateStringBuffer.");
long start, end;
start = System.currentTimeMillis();
ArrayList<String> hashes = new ArrayList<String>();
com.tekcomms.c2d.xml.model.v2.Metadata_element[] metadata_element = null;
String[] servicePath = newTopology.getElement().getEntity().getService_path().getLevel();
LOGGER.debug(servicePath.toString());
if (newTopology.getElement().getMetadata() != null) {
metadata_element = newTopology.getElement().getMetadata().getMetadata_element();
LOGGER.debug(metadata_element.toString());
}
for (int i = 0; i < servicePath.length; i++) {
String level = servicePath[i];
LOGGER.debug("level is: " + level);
if (splitLine.length > getPositionFromIndex(level)) {
String name = splitLine[getPositionFromIndex(level)];
sbIntermediate.append(name);
hashes.add(name);
sbIntermediate.append(REGEX_COMMA).append(HashUtils.calculateHash(hashes)).append(REGEX_COMMA);
LOGGER.debug(" ==sbIntermediate: " + sbIntermediate.toString());
}
}
// end=System.currentTimeMillis();
// LOGGER.info("COMPLETED adding name hash. " + (end - start) + " ms. " + (end - start) / 1000 + " seg.");
// adding cpe.latitude,cpe.longitude,cpe.customer_class, it should be
// better if it would be metadata as well.
String labelLatitude = newTopology.getElement().getEntity().getLatitude();
if (splitLine.length > getPositionFromIndex(labelLatitude)) {
String lat = splitLine[getPositionFromIndex(labelLatitude)];
sbIntermediate.append(lat).append(REGEX_COMMA);
}
String labelLongitude = newTopology.getElement().getEntity().getLongitude();
if (splitLine.length > getPositionFromIndex(labelLongitude)) {
String lon = splitLine[getPositionFromIndex(labelLongitude)];
sbIntermediate.append(lon).append(REGEX_COMMA);
}
String labelCustomerClass = newTopology.getElement().getCustomer_class();
if (splitLine.length > getPositionFromIndex(labelCustomerClass)) {
String customerClass = splitLine[getPositionFromIndex(labelCustomerClass)];
sbIntermediate.append(customerClass).append(REGEX_COMMA);
}
// end=System.currentTimeMillis();
// LOGGER.info("COMPLETED adding lat,lon,customer. " + (end - start) + " ms. " + (end - start) / 1000 + " seg.");
// watch out metadata are optional, it can appear as a void chain!
if (metadata_element != null && metadata_element.length != 0)
for (int j = 0; j < metadata_element.length; j++) {
String label = metadata_element[j].getLabel();
LOGGER.debug(" ==label: " + label + " index_pos: " + j);
if (splitLine.length > getPositionFromIndex(label)) {
String actualValue = splitLine[getPositionFromIndex(label)];
if (!"".equals(actualValue))
sbIntermediate.append(actualValue).append(REGEX_COMMA);
else
sbIntermediate.append("").append(REGEX_COMMA);
} else
sbIntermediate.append("").append(REGEX_COMMA);
LOGGER.debug(" ==sbIntermediate: " + sbIntermediate.toString());
}//for
sbIntermediate.append("\n");
end = System.currentTimeMillis();
LOGGER.info("COMPLETED method createIntermediateStringBuffer. " + (end - start) + " ms. ");
}
As you can see, this method adds a precalculated line to the StringBuffer, reads every line from input csv file, calculate new data from that lines and finally add the generated line to the StringBuffer, so finally i can create the file with that buffer.
I have run jconsole and i can see that there are no memory leaks, i can see the sawtooths representing the creation of objects and the gc recollecting garbaje. It never traspasses the memory heap threshold.
One thing i have noticed is that the time needed for add a new line to the StringBuffer is completed within a very few ms range, (5,6,10), but is raising with time, to (100-200) ms and i suspect more in a near future, so probably this is the battle horse.
I have tried to analyze the code, i know that there are 3 for loops, but they are very shorts, the first loop iterates over 8 elements only:
for (int i = 0; i < servicePath.length; i++) {
String level = servicePath[i];
LOGGER.debug("level is: " + level);
if (splitLine.length > getPositionFromIndex(level)) {
String name = splitLine[getPositionFromIndex(level)];
sbIntermediate.append(name);
hashes.add(name);
sbIntermediate.append(REGEX_COMMA).append(HashUtils.calculateHash(hashes)).append(REGEX_COMMA);
LOGGER.debug(" ==sbIntermediate: " + sbIntermediate.toString());
}
}
I have meassured the time needed to get the name from the splitline and it is worthless, 0 ms, the same to calculateHash method, 0 ms.
the other loop, are practically the same, iterates over 0 to n, where n is a very tiny int, 3 to 10 for example, so i do not understand why it takes more time to finish the method, the only thing i find is that to add a new line to the buffer is getting slow the process.
I am thinking about a producer consumer multi threaded strategy, a reader thread that reads every line and put them into a circular buffer, another threads take it one by one, process them and add a precalculated line to the StringBuffer, which is thread safe, when the file is fully readed, the reader thread sends a message to to the another threads telling them to stop. Finally i have to save this buffer to a file. What do you think? this is a good idea?
I am thinking about a producer consumer multi threaded strategy, a reader thread that reads every line and put them into a circular buffer, another threads take it one by one, process them and add a precalculated line to the StringBuffer, which is thread safe, when the file is fully readed, the reader thread sends a message to to the another threads telling them to stop. Finally i have to save this buffer to a file. What do you think? this is a good idea?
Maybe, but it's quite a lot of work, I'd try something simpler first.
line.split(REGEX_COMMA)
Your REGEX_COMMA is a string which gets compiled into an regex a million times. It's trivial, but I'd try to use a Pattern instead.
You're producing a lot of garbage with your split. Maybe you should avoid it by manually splitting the input into a reused ArrayList<String> (it's just a few lines).
If all you need is writing the result into a file, it might be better to avoid building one huge String. Maybe a List<String> or even a List<StringBuilder> would be better, maybe writing directly to a buffered stream would do.
You seem to be working with ASCII only. Your encoding is platform dependent which may mean you're using UTF-8, which is possibly slow. Switching to a simpler encoding could help.
Working with byte[] instead of String would most probably help. Bytes are half as big as chars and there's no conversion needed when reading a file. All the operations you do can be done with bytes equally easy.
One thing i have noticed is that the time needed for add a new line to the StringBuffer is completed within a very few ms range, (5,6,10), but is raising with time, to (100-200) ms and i suspect more in a near future, so probably this is the battle horse.
That's resizing, which could be sped up by using the suggested ArrayList<String>, as the amount of data to be copied is much lower. Writing the data out when the buffer gets big would do as well.
I have meassured the time needed to get the name from the splitline and it is worthless, 0 ms, the same to calculateHash method, 0 ms.
Never use currentTimeMillis for this as nanoTime is strictly better. Use a profiler. The problem with a profiler is that it changes what it should measure. As a poor man's profiler, you can compute the sum of all the times spend inside of the suspect method and compare it with the total time.
What's the CPU load and what does GC do when running the program?
I used superCSV library in my project to handle large set of lines. it is relatively fast than manually read the lines. Reference
I'm making a whois in java for android to train about streams and tcp connections.
But I have a problem. I have a php script, I wrote some time ago and I´m trying to make the same in java.
this is my java code :
public String consultawhois(String domain,String tld)
{
String domquest = domain + "." + tld;
String resultado = "";
Socket theSocket;
String hostname = "whois.internic.net";
int port = 43;
try {
theSocket = new Socket(hostname, port, true);
Writer out = new OutputStreamWriter(theSocket.getOutputStream());
out.write(domquest + "\r\n");
out.flush();
DataInputStream theWhoisStream;
theWhoisStream = new DataInputStream(theSocket.getInputStream());
String s;
while ((s = theWhoisStream.readLine()) != null) {
resultado = resultado + s + "\n";
}
}
catch (IOException e) {
}
return resultado;
}
The answer of the server is not correct and I think the problem is that I'm sending a bad query. The query I send is "dominio.com\r\n" and in my php whois code, it works perfectly.
It seems that the DNS query matches multiple records. At least, that is how I interpret the response. In the returned reponse you should see the following line:
To single out one record, look it up with "xxx", where xxx is one of the
of the records displayed above. If the records are the same, look them up
with "=xxx" to receive a full display for each record.
So if you prepend the query with "=" it returns the data of that record only. The following worked for me.
public String consultawhois(String domain,String tld)
{
String domquest = domain + "." + tld;
String resultado = "";
Socket theSocket;
String hostname = "whois.internic.net";
int port = 43;
try {
theSocket = new Socket(hostname, port, true);
Writer out = new OutputStreamWriter(theSocket.getOutputStream());
out.write("="+domquest + "\r\n");
out.flush();
DataInputStream theWhoisStream;
theWhoisStream = new DataInputStream(theSocket.getInputStream());
String s;
while ((s = theWhoisStream.readLine()) != null) {
resultado = resultado + s + "\n";
}
}
catch (IOException e) {
}
return resultado;
}
One thing to consider: Use English for method names, variables, etc. instead of Spanish. It will make your code easier to read internationally. The programming language itself also uses English words. Try to avoid a strange mix of English and your native language.
The lookup for dominio.com results in three matches:
DOMINIO.COM.BR
DOMINIO.COM.ASCPROBIENESTARIDSS.COM
DOMINIO.COM
You should specify wich one you are interested in with the query.
=dominio.com<newline>
This will allways work, even cases where there are no multiple matches.
If I have a servlet running JVM1.4.2, and it is receiving a POST request with form data fields. I use req.getParameterNames() to get, what I would expect, all the query string and form data. However, all I ever get are the querystring parameters.
Literature I am reading from various sources says that getParameterNames() and getParameterValues(String) should be the way to get all query string and posted form data sent by the browser for JDK 1.4. Here is the method I use to extract all the parameters, which I expect would include posted form data :
public Map getParameterMap(HttpServletRequest req) {
Map params= new HashMap();
String name = null;
System.out.println("<< Getting Parameter Map.>>");
Enumeration enumParams = req.getParameterNames();
for (; enumParams.hasMoreElements(); ) {
// Get the name of the request parameter
name = (String)enumParams.nextElement();
// Get the value of the request parameters
// If the request parameter can appear more than once
// in the query string, get all values
String[] values = req.getParameterValues(name);
params.put(name, values);
String sValues = "";
for(int i=0;i<values.length;i++){
if(0<i) {
sValues+=",";
}
sValues +=values[i];
}
System.out.println("Param " + name + ": " + sValues);
}
System.out.println("<< END >>");
return params;
}
This question also agrees with my expectations, but the servlet is not picking up the form data. Obviously I am missing something....
Update: The post data is very straight forward and is not a Multipart form or rich media. Just plain'ol text submitted via an AJAX POST that looks like this in post body
c1=Value%20A&c2=Value%20B&c3=Value%20C
I managed to identify the problem. Because there is so much chatter from JDK 1.5+ and talk of getParameterMaps() method for 1.5, info on how 1.4 handles form post data was scarce and ambiguous. (Please post a comment if you find something that is specific for 1.4).
Pre-1.5 you have to manually get the form data via getInputStream, and then parse it out. I found this method, (posted below), from the java sun site that does a nice job using a Hashtable. I had to make a minor mod for deprecated methods. But seems to work quite robustly, "out of the box", so you should able to just cut-n-paste. I know it's "old tech" but I thought it worthwhile for those who may be in the same situation as me who are stuck on solving (what seems to be) straight forward problems.
public Hashtable parsePostData(int length, ServletInputStream instream) {
String valArray[] = null;
int inputLen, offset;
byte[] postedBytes = null;
boolean dataRemaining=true;
String postedBody;
Hashtable ht = new Hashtable();
//Vector paramOrder = new Vector(10);
StringBuffer sb = new StringBuffer();
if (length <=0) {
return null;
}
postedBytes = new byte[length];
try {
offset = 0;
while(dataRemaining) {
inputLen = instream.read (postedBytes, offset, length - offset);
if (inputLen <= 0) {
throw new IOException ("read error");
}
offset += inputLen;
if((length-offset) ==0) {
dataRemaining=false;
}
}
} catch (IOException e) {
System.out.println("Exception ="+e);
return null;
}
postedBody = new String (postedBytes);
StringTokenizer st = new StringTokenizer(postedBody, "&");
String key=null;
String val=null;
while (st.hasMoreTokens()) {
String pair = (String)st.nextToken();
int pos = pair.indexOf('=');
if (pos == -1) {
throw new IllegalArgumentException();
}
try {
key = URLDecoder.decode(pair.substring(0, pos),"UTF8");
val = java.net.URLDecoder.decode(pair.substring(pos+1,pair.length()),"UTF8");
} catch (Exception e) {
throw new IllegalArgumentException();
}
if (ht.containsKey(key)) {
String oldVals[] = (String []) ht.get(key);
valArray = new String[oldVals.length + 1];
for (int i = 0; i < oldVals.length; i++) {
valArray[i] = oldVals[i];
}
valArray[oldVals.length] = val;
} else {
valArray = new String[1];
valArray[0] = val;
}
ht.put(key, valArray);
String sValues = "";
for(int i=0;i<valArray.length;i++) {
if (0<i) {
sValues+=",";
}
sValues = valArray[i];
}
System.out.println("Form data field " + key + ":" +sValues);
//paramOrder.addElement(key);
}
return ht;
}
That's true. The getParameterNames(), getParameterValues(), and getParameter() methods are the way to access form data unless it's a multipart form, in which case you'll have to use something like Commons Fileupload to parse the multipart request before all the parameters are accessible to you.
Edit: You're probably not encoding the POST data properly in your AJAX call. POST data must carry a Content-Type of application/x-www-form-urlencoded or else multipart/form-data. If you're sending it as something else, it doesn't qualify as a request parameter, and I expect you'd see the behavior you're describing. The solution you've engineered essentially consists of setting up custom parsing of custom content.