when I am reading the code of opentsdb:
try {
System.in.close(); // Release a FD we don't need.
} catch (Exception e) {
log.warn("Failed to close stdin", e);
}
After searching this question on the Internet, I can't find a suitable answer. I don't understand why they write system.in.close(), and I want to know if we don't add this code block, what will happen?
Probably only if you used System.setIn() to override the standard input. One normally does not close the standard input, it's handled by JVM process shutdown.
I have a pipeline with quite a few steps (just above 15). I want to report failures everytime a DoFn fails. I started implementing it through TupleTags with code such as :
try {
... do stuff ...
c.output(successTag, ...);
} catch (Exception e) {
c.output(failureTag, new Failure(...));
}
But since my pipeline contains a lot of steps, this make the pipeline definition code quite hard to read / maintain.
Is there a more global way to achieve it ? Something like raising a custom exception which is handled globally at the pipeline level ?
What you are doing is the correct approach to catch errors and output them differently. You will need this on each step though. You could use a java pattern to reuse it if you prefer. Create a base class for all your ParDos and in processElement add the exception handling code. Then implement your processElement in a separate function (i.e. processElementImpl) which you call in processElement.
I'm trying to get this piece of code to work. It's a basic I/O system that copies one file and pastes it into the same directory with the chosen name. It should be simple but for some reason the program runs, it creates the second file but then it gets stuck. The CPU for Java process sits at around 5% and the file is never completed. It only copies over some of the data and then I'd imagine it's stuck in an infinite loop somewhere.
I've already compared my code with the Byte Streams tutorial on the Oracle website.
EXTRA: I just asked it to output what it was reading and it's stuck on an infinite loop reading the value 255. If that helps. Also, I compiled the code directly off the Oracle website and it does the same thing.
It appears consistent from what I can tell. Can anyone tell me what I'm doing wrong? Thank you.
(P.S: I'm Using Eclipse 4.2.0).
This is what I'm doing to copy the file:
package fileIO;
import java.io.*;
import system.Debug;
public class fileUtil {
public static void copyFileTo(String file2Copy, String file2Paste) {
FileInputStream fin = null;
FileOutputStream fout = null;
try {
fin = new FileInputStream(file2Copy);
fout = new FileOutputStream(file2Paste);
int aByte;
while ((aByte = fin.read()) != -1) {
fout.write(aByte);
}
} catch (FileNotFoundException e) {
Debug.out("Error: File Not Found: " + file2Copy);
} catch (IOException e) {
Debug.out("Error: File IO Exception Copying: " + file2Copy);
} catch (Exception e) {
Debug.out("Error: General Exception Closing Streams:" + file2Copy);
} finally {
try {
fin.close();
fout.close();
} catch (IOException e) {
Debug.out("Error: File IO Exception Closing Streams: " + file2Copy);
} catch (Exception e) {
Debug.out("Error: General Exception Closing Streams:" + file2Copy);
}
}
}
}
In my program main class I run this:
fileUtil.copyFileTo("google.bmp", "google(1).bmp");
Try to do fout.flush() before you close the OutputStream.
Okay, so I found out what was happening. Was a really nooby mistake.
I'll put my pride aside encase anyone else has this problem. It's not an infinite loop, it's just that copying using ByteStreams takes AGES. I was expecting a fast result from small image files but even small image files take a long long time to copy. I let it run for 30 seconds and the program terminated properly and I got my image copy just fine.
Thank god it's solved, I was beginning to worry.
... or do not invent bother re-inventing the wheel: use FileUtils.copyFile from the proven Apache commons-io which does it in one line.
(beware: this comment is not as innocent as it seems: File.rename does not work well on Windows shares - commons-io is always the safe bet to do these things)
Edit
Stackoverflow is not a goog place for "homework" - or you must at least say so. It is not that your problem is not real. It is that your objectives differ: you want to learn something, we want to make it work reliably with minimum maintenance.
...which leads to my second point: when you are in professional life, never program this again. As you discovered, even if you make it work, it may be extremely inefficient, handle errors incorrectly, etc.. This is particularly true with IO which is always more tricky than it seems.
Finally, since I gave you a link to a well trusted library under Apache 2.0 license, maybe you could have had a look at the source code ?
I need to use the Java-based OpenNLP library in my PHP code. For example, I need to use its Sentence Detector component (en-sent.bin) for analysing text variables in my PHP code.
In its documentation, that API can be accessed from a Java code as follows:
InputStream modelIn = new FileInputStream("en-sent.bin");
try {
SentenceModel model = new SentenceModel(modelIn);
}
catch (IOException e) {
e.printStackTrace();
}
finally {
if (modelIn != null) {
try {
modelIn.close();
}
catch (IOException e) {
}
}
}
How can do the same thing in PHP?
In other words, what is the PHP-equivalent to the above Java code?
The solution I'm about to implement is writing a java based backend (because just the OpenNLP tool's output just won't cut it) and using PHP's execute function to run it. If you want it to be even faster, I suggest making it a daemon, making PHP connect to it and send it data to return (it'd probably be a unix socket). Although it's kinda complex, it seems less error prone than a PHP/Java Bridge.
There would have to be a PHP API for accessing OpenNLP. A quick search doesn't show anything. The only other thing I can think of is using a PHP/Java Bridge of some sort, but that's more involved. See http://php-java-bridge.sourceforge.net/pjb/, for example.
I would like to get to query Windows Vista Search service directly ( or indirectly ) from Java.
I know it is possible to query using the search-ms: protocol, but I would like to consume the result within the app.
I have found good information in the Windows Search API but none related to Java.
I would mark as accepted the answer that provides useful and definitive information on how to achieve this.
Thanks in advance.
EDIT
Does anyone have a JACOB sample, before I can mark this as accepted?
:)
You may want to look at one of the Java-COM integration technologies. I have personally worked with JACOB (JAva COm Bridge):
http://danadler.com/jacob/
Which was rather cumbersome (think working exclusively with reflection), but got the job done for me (quick proof of concept, accessing MapPoint from within Java).
The only other such technology I'm aware of is Jawin, but I don't have any personal experience with it:
http://jawinproject.sourceforge.net/
Update 04/26/2009:
Just for the heck of it, I did more research into Microsoft Windows Search, and found an easy way to integrate with it using OLE DB. Here's some code I wrote as a proof of concept:
public static void main(String[] args) {
DispatchPtr connection = null;
DispatchPtr results = null;
try {
Ole32.CoInitialize();
connection = new DispatchPtr("ADODB.Connection");
connection.invoke("Open",
"Provider=Search.CollatorDSO;" +
"Extended Properties='Application=Windows';");
results = (DispatchPtr)connection.invoke("Execute",
"select System.Title, System.Comment, System.ItemName, System.ItemUrl, System.FileExtension, System.ItemDate, System.MimeType " +
"from SystemIndex " +
"where contains('Foo')");
int count = 0;
while(!((Boolean)results.get("EOF")).booleanValue()) {
++ count;
DispatchPtr fields = (DispatchPtr)results.get("Fields");
int numFields = ((Integer)fields.get("Count")).intValue();
for (int i = 0; i < numFields; ++ i) {
DispatchPtr item =
(DispatchPtr)fields.get("Item", new Integer(i));
System.out.println(
item.get("Name") + ": " + item.get("Value"));
}
System.out.println();
results.invoke("MoveNext");
}
System.out.println("\nCount:" + count);
} catch (COMException e) {
e.printStackTrace();
} finally {
try {
results.invoke("Close");
} catch (COMException e) {
e.printStackTrace();
}
try {
connection.invoke("Close");
} catch (COMException e) {
e.printStackTrace();
}
try {
Ole32.CoUninitialize();
} catch (COMException e) {
e.printStackTrace();
}
}
}
To compile this, you'll need to make sure that the JAWIN JAR is in your classpath, and that jawin.dll is in your path (or java.library.path system property). This code simply opens an ADO connection to the local Windows Desktop Search index, queries for documents with the keyword "Foo," and print out a few key properties on the resultant documents.
Let me know if you have any questions, or need me to clarify anything.
Update 04/27/2009:
I tried implementing the same thing in JACOB as well, and will be doing some benchmarks to compare performance differences between the two. I may be doing something wrong in JACOB, but it seems to consistently be using 10x more memory. I'll be working on a jcom and com4j implementation as well, if I have some time, and try to figure out some quirks that I believe are due to the lack of thread safety somewhere. I may even try a JNI based solution. I expect to be done with everything in 6-8 weeks.
Update 04/28/2009:
This is just an update for those who've been following and curious. Turns out there are no threading issues, I just needed to explicitly close my database resources, since the OLE DB connections are presumably pooled at the OS level (I probably should have closed the connections anyway...). I don't think I'll be any further updates to this. Let me know if anyone runs into any problems with this.
Update 05/01/2009:
Added JACOB example per Oscar's request. This goes through the exact same sequence of calls from a COM perspective, just using JACOB. While it's true JACOB has been much more actively worked on in recent times, I also notice that it's quite a memory hog (uses 10x as much memory as the Jawin version)
public static void main(String[] args) {
Dispatch connection = null;
Dispatch results = null;
try {
connection = new Dispatch("ADODB.Connection");
Dispatch.call(connection, "Open",
"Provider=Search.CollatorDSO;Extended Properties='Application=Windows';");
results = Dispatch.call(connection, "Execute",
"select System.Title, System.Comment, System.ItemName, System.ItemUrl, System.FileExtension, System.ItemDate, System.MimeType " +
"from SystemIndex " +
"where contains('Foo')").toDispatch();
int count = 0;
while(!Dispatch.get(results, "EOF").getBoolean()) {
++ count;
Dispatch fields = Dispatch.get(results, "Fields").toDispatch();
int numFields = Dispatch.get(fields, "Count").getInt();
for (int i = 0; i < numFields; ++ i) {
Dispatch item =
Dispatch.call(fields, "Item", new Integer(i)).
toDispatch();
System.out.println(
Dispatch.get(item, "Name") + ": " +
Dispatch.get(item, "Value"));
}
System.out.println();
Dispatch.call(results, "MoveNext");
}
} finally {
try {
Dispatch.call(results, "Close");
} catch (JacobException e) {
e.printStackTrace();
}
try {
Dispatch.call(connection, "Close");
} catch (JacobException e) {
e.printStackTrace();
}
}
}
As few posts here suggest you can bridge between Java and .NET or COM using commercial or free frameworks like JACOB, JNBridge, J-Integra etc..
Actually I had an experience with with one of these third parties (an expensive one :-) ) and I must say I will do my best to avoid repeating this mistake in the future. The reason is that it involves many "voodoo" stuff you can't really debug, it's very complicated to understand what is the problem when things go wrong.
The solution I would suggest you to implement is to create a simple .NET application that makes the actual calls to the windows search API. After doing so, you need to establish a communication channel between this component and your Java code. This can be done in various ways, for example by messaging to a small DB that your application will periodically pull. Or registering this component on the machine IIS (if exists) and expose simple WS API to communicate with it.
I know that it may sound cumbersome but the clear advantages are: a) you communicate with windows search API using the language it understands (.NET or COM) , b) you control all the application paths.
Any reason why you couldn't just use Runtime.exec() to query via search-ms and read the BufferedReader with the result of the command? For example:
public class ExecTest {
public static void main(String[] args) throws IOException {
Process result = Runtime.getRuntime().exec("search-ms:query=microsoft&");
BufferedReader output = new BufferedReader(new InputStreamReader(result.getInputStream()));
StringBuffer outputSB = new StringBuffer(40000);
String s = null;
while ((s = output.readLine()) != null) {
outputSB.append(s + "\n");
System.out.println(s);
}
String result = output.toString();
}
}
There are several libraries out there for calling COM objects from java, some are opensource (but their learning curve is higher) some are closed source and have a quicker learning curve. A closed source example is EZCom. The commercial ones tend to focus on calling java from windows as well, something I've never seen in opensource.
In your case, what I would suggest you do is front the call in your own .NET class (I guess use C# as that is closest to Java without getting into the controversial J#), and focus on making the interoperability with the .NET dll. That way the windows programming gets easier, and the interface between Windows and java is simpler.
If you are looking for how to use a java com library, the MSDN is the wrong place. But the MSDN will help you write what you need from within .NET, and then look at the com library tutorials about invoking the one or two methods you need from your .NET objects.
EDIT:
Given the discussion in the answers about using a Web Service, you could (and probably will have better luck) build a small .NET app that calls an embedded java web server rather than try to make .NET have the embedded web service, and have java be the consumer of the call. For an embedded web server, my research showed Winstone to be good. Not the smallest, but is much more flexible.
The way to get that to work is to launch the .NET app from java, and have the .NET app call the web service on a timer or a loop to see if there is a request, and if there is, process it and send the response.