I have a library that needs to create a schema in MySQL from Java. Currently, I have a dump of the schema that I just pipe into the mysql command. This works okay, but it is not ideal because:
It's brittle: the mysql command needs to be on the path: usually doesn't work on OSX or Windows without additional configuration.
Also brittle because the schema is stored as statements, not descriptively
Java already can access the mysql database, so it seems silly to depend on an external program to do this.
Does anyone know of a better way to do this? Perhaps...
I can read the statements in from the file and execute them directly from Java? Is there a way to do this that doesn't involve parsing semicolons and dividing up the statements manually?
I can store the schema in some other way - either as a config file or directly in Java, not as statements (in the style of rails' db:schema or database.yml) and there is a library that will create the schema from this description?
Here is a snippet of the existing code, which works (when mysql is on the command line):
if( db == null ) throw new Exception ("Need database name!");
String userStr = user == null ? "" : String.format("-u %s ", user);
String hostStr = host == null ? "" : String.format("-h %s ", host);
String pwStr = pw == null ? "" : String.format("-p%s ", pw);
String cmd = String.format("mysql %s %s %s %s", hostStr, userStr, pwStr, db);
System.out.println(cmd + " < schema.sql");
final Process pr = Runtime.getRuntime().exec(cmd);
new Thread() {
public void run() {
try (OutputStream stdin = pr.getOutputStream()) {
Files.copy(f, stdin);
}
catch (IOException e) { e.printStackTrace(); }
}
}.start();
new Thread() {
public void run() {
try (InputStream stdout = pr.getInputStream() ) {
ByteStreams.copy(stdout, System.out);
}
catch (IOException e) { e.printStackTrace(); }
}
}.start();
int exitVal = pr.waitFor();
if( exitVal == 0 )
System.out.println("Create db succeeded!");
else
System.out.println("Exited with error code " + exitVal);
The short answer (as far as i know) is no.
You will have to do some parsing of the file into separate statements.
I have faced the same situation and you can find many questions on this topic here on SO.
some like here will show a parser. others can direct to tools Like this post from apache that can convert the schema to an xml format and then can read it back.
My main intention when writing this answer is to tell that I chose to use the command line in the end.
extra configuration: maybe it is an additional work but you can do it by config or at runtime based on the system you are running inside. you do the effort one time and you are done
depending on external tool: it is not as bad as it seems. you have some benefits too.
1- you don't need to write extra code or introduce additional libraries just for parsing the schema commands.
2- the tool is provided by the vendor. it is probably more debugged and tested than any other code that will do the parsing.
3- it is safer on the long run. any additions or changes in the format of dump that "might" break the parser will most probably be supported with the tool that comes with the database release. you won't need to do any change in your code.
4- the nature of the action where you are going to use the tool (creating schema) does not suggest frequent usage, minimizing the risk of it becoming a performance bottle neck.
I hope you can find the best solution for your needs.
Check out Yank, and more specifically the code examples linked to on that page. It's a light-weight persistence layer build on top of DBUtils, and hides all the nitty-gritty details of handling connections and result sets. You can also easily load a config file like you mentioned. You can also store and load SQL statements from a properties file and/or hard code the SQL statements in your code.
Related
According to this link, if the source has a problem when opening and throws an exception and it is also in try parentheses, JVM closes it. My question is how to inform the user now that this source is closed and we encountered a problem when opening this resource? In other words, how can this exception be handled?
Seems trivial. Usually, java code is running in some sort of 'no user interaction' environment (servers and the like). The right move is to let the exception bubble up - you want the daily job that is halfway through reading through the database to open the related file to then send the logs to long term storage or whatever it is to completely abort and write a note in the log file. Usually for jobs like that, there's some atomary functionality (in this case, perhaps each such file is independent of the others, and its okay to leave the 'broken' one in place for now until a server admin can look at it whilst continuing to process the remainder - in that case, the 'do the backup rotation thing on THIS file' is the atomary functionality): Catch all exceptions and write code that does what you want when the job fails. For example, my servers can send notifications straight to admin phones (via telegram or pushover, or using slack API, and there are many services that automate this for you too), if it's important, you'd write that in your catch block.
For code that is directly 'triggered' by a user, let's say a 'save file' function, then it's not so much 'the resource is now closed' - resources are not long lived (they cannot be - not if you use try-with-resources). They were either never open in the first place (you attempt to save a file to a dir that doesn't exist - the act of trying to make the new OutputStream already failed, it was never open to begin with), or, perhaps it did open, but it was to a USB stick and the user pulled it out halfway through saving. The resource is just closed, effectively, whether in java you .close() it or not - the entire stick is gone!!
The only thing the 'safe close' aspect of try-with-resources did for you is ensure that your Java Process isn't wasting a file handle.
You handle it the same way you handle pretty much any 'unrecoverable' (you can't write software that hypnotises the user into sticking that USB stick back into the machine, obviously - it is not recoverable as a consequence, like most exceptions) problem: You toss up a dialog box that explains the situation.
try (OutputStream out = Files.newOutputStream(saveGameFile)) {
boardState.save(out);
} catch (IOException e) {
// show dialog here
}
Even when using a try-with-resources, the catch clause still works.
private static void printFile() throws MyCustomException {
try(FileInputStream input = new FileInputStream("file.txt")) {
int data = input.read();
while(data != -1){
System.out.print((char) data);
data = input.read();
}
} catch (IOException e) {
throw new MyCustomException("There was an error while opening the resource", e);
}
}
I am trying to access SVN through the process command in Java as part of a larger GUI to see what files are on the SVN. After much research, I have significantly refined my methods, however I still cannot accomplish it. If I run the code in the GUI, it just hangs. To discover what that problem was, I simplified it and ran it as a console program. When I ran it there, it displayed a request for my GNOME keyring. My code enters the password but the console does not seem to accept it. My code follows:
public class SvnJavaTest{
public static void main(String[] args){
try {
String[] commands = {"svn", "ls", "https://svnserver"};
Process beginProcess = Runtime.getRuntime().exec(commands, null, new File("/home/users/ckorb/Desktop"));
BufferedReader br = new BufferedReader(new InputStreamReader(beginProcess.getInputStream()));
BufferedWriter write = new BufferedWriter(new OutputStreamWriter(beginProcess.getOutputStream()));
write.write("password");
write.flush();
String line=br.readLine();
while (line != null){
System.out.println(line);
line =br.readLine();
}
br.close();
write.close();
beginProcess.waitFor();
} catch (IOException e1) {
e1.printStackTrace();
} catch (InterruptedException ie) {
ie.printStackTrace();
}
}
}
I don't get any errors running this and if I type in my password manually into the console and then run it, it works because it remembers my password. I have looked and found that there are some packages that would automatically enter my keyring on login but that isn't really an option. Thank you very much.
The main problem with a solution like this is that you don't really have control over stdin and stdout. A malicious person can wrap the svn command with a shell script that makes a copy of the stdin (thereby capturing all the passwords your program transmits). While shell's flexibility makes it great in so many ways, it is the same flexibility that you are connecting to, and you'd better be comfortable with it (and it's consequences).
That is the real reason why it is better to use a Java API to use the client, there's a much smaller chance of injecting code which captures sensitive data (and better error reporting).
Use the SVN Kit library instead.
Better to use svn java API (there are several, I am not sure which one is better), it's more straightforward solution.
Answering you question - you could provide auth info in the url: https://username:password#svnserver
I am passing Yaml created with PyYaml to SnakeYaml and Snakeyaml does not seem to recognize anything beyond the first line where !! exists and python/object is declared. I already have identical objects setup in Java. Is there an example out there that shows a loadAll into an object array where the object type is asserted or assigned?
Good call... was away from the computer when I originally posted.
Here is the data from PyYaml that I am trying to use SnakeYaml to get into a Java application:
--- !!python/object:dbmethods.Project.Project {dblogin: kirtstrim7900, dbname: 92218kirtstrim_wfrogls,dbpw: 1234567895#froggy, preference1: '', preference2: '', preference3: '', projName: CheckPoint Firewall Audit - imp, projNo: 1295789430544+CheckPoint Firewall Audit - imp, projectowner: kirtcathey#sysrisk.com,result1label: Evidence, result2label: Recommend, result3label: Report, resultlabel: Response,role: owner, workstep1label: Objective, workstep2label: Policy, workstep3label: Guidance,worksteplabel: Procedure}
Not just a single instance of the above, but several objects, so need to use loadAll in SnakeYaml.... unless somebody knows better.
As for the code, this is all I have from SnakeYaml docs:
for (Object data : yaml.loadAll(sb.toString())) {
System.out.println(data.toString());
}
Then, this error is thrown:
Exception in thread "AWT-EventQueue-0" Can't construct a java object for tag:yaml.org,2002:java/object: ......
Caused by: org.yaml.snakeyaml.error.YAMLException: Class not found: ......
As you can see from the small code snippet, EVEN without all this information supplied, anybody who knows the answer about how to cast an object arbitrarily could PROBABLY answer the question.
Thx.
Parsed off the two exclamation points (!!) at the beginning of each entry and now I get:
mapping values are not allowed here
in "", line 1, column 73:
as an error. The whole point of using YAML was to reduce coding related to parsing. If I have to turn around and parse incoming and outgoing code for whatever reason, then YAML sucks!! And will gladly revert back XML or anything else that will allow a python middleware to talk to a java application.
To achieve the same result you may:
configure PyYAML to skip the tag (exactly as you did with the comment "Convert objects to a dictionary of their representation")
configure SnakeYAML to create the object you expect (exactly as you did with "projectData = gson.fromJson(mystr, ProjectData[].class); ")
If you are lost (before you say "it sucks") you may ask a question in the corresponding mailing lists. It may help you to find a proper solution in the future.
Fixed. YAML sucks, so don't use it. All kinds of Google results about how SnakeYAML is derived from PyYaml and what-not, but nobody clearly states exactly what dumps format from PyYaml works with what loadAll routines with SnakeYAML.
Also, performance with YAML is horrid, JSON is far simpler and easier to implement. In Python, where our middleware resides (and most crunching occurs), YAML takes almost twice the time to process than JSON!!
If you are using Python 2.6 or greater, just
import json
json_doc = json.dumps(projects, default=convert_to_builtin_type)
print json_doc
def convert_to_builtin_type(obj):
print 'default(', repr(obj), ')'
# Convert objects to a dictionary of their representation
d = { '__class__':obj.__class__.__name__,
'__module__':obj.__module__,
}
d.update(obj.__dict__)
return d
Then on the Java client (loading) side, use GSon -- this took a lot of head-scratching and searches to figure out because ALL examples on the 'net are virtually useless. Every blogger with 500 ads per page shows you how to convert one single, stupid object and last time I created an app, I used lists, arrays, or anything that held more than one object!!
try {
serverAddress = new URL("http://127.0.0.1:5000/projects/" + ruser.getUserEmail()+"+++++"+ruser.getUserHash());
//set up out communications stuff
connection = null;
//Set up the initial connection
connection = (HttpURLConnection)serverAddress.openConnection();
connection.setRequestMethod("GET");
connection.setDoOutput(true);
connection.setReadTimeout(100000);
connection.connect();
//get the output stream writer and write the output to the server
//not needed in this example
rd = new BufferedReader(new InputStreamReader(connection.getInputStream()));
sb = new StringBuilder();
while ((line = rd.readLine()) != null)
{
sb.append(line + '\n');
}
String mystr = sb.toString();
// Now do the magic.
Gson gson = new Gson();
projectData = gson.fromJson(mystr, ProjectData[].class);
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (ProtocolException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
finally
{
//close the connection, set all objects to null
connection.disconnect();
rd = null;
sb = null;
connection = null;
}
return projectData;
Done! In a nutshell - YAML sucks and use JSON!! Also, the http connection code is mostly snipped off of this site...now I need to figure out https.
I am trying to implement some unit testing for an old framework. I am attempting to mock out the database layer. Unfortunately our framework is a bit old and not quite using best practices so there is no clear separation of concerns. I am bit worried that trying to mock out the database layer might make the JVM load a huge number of classes that won't even be used.
I don't really understand class loaders that well so this might not be a problem. Is there a way to take a peak at all the classes a particular ClassLoader has loaded to prove what is going on under the hood?
You can create your own Classloader and use that to load during the unit test. Have your own custom Classloader print out what it's doing.
Or if you just want to know which classes are loaded, do:
java -verbose:class
Be warned that using
java -verbose
Will produce an enormous amount of output. Log the output to a file and then use grep. If you have the 'tee' filter you could try this:
java -verbose | tee classloader.log
grep class classloader.log
I am not sure. But there is one way I see it could be done. It maybe overrly ridiculous though. You can try aspects and put a pointcut for loadclass.
Also maybe the jvm argument -verbose maybe helpful.
As an alternative way, for a particular Class-loader as you mentioned, you can use this code snippet. Just change value of obj variable if you want.
Object obj = this;
ClassLoader classLoader = obj.getClass().getClassLoader();
File file = new File("classloderClasses.txt");
if (file.exists()) {
file.delete();
}
if (classLoader != null) {
try {
Class clClass = classLoader.getClass();
while (clClass != ClassLoader.class) {
clClass = clClass.getSuperclass();
}
java.lang.reflect.Field classesField = clClass.getDeclaredField("classes");
classesField.setAccessible(true);
Vector classes = (Vector) classesField.get(classLoader);
FileOutputStream fos = new FileOutputStream("classloderClasses.txt", true);
fos.write(("******************** " + classLoader.toString() + " ******************** " + "\n").getBytes());
fos.write(Arrays.toString(classes.toArray()).getBytes());
fos.close();
} catch (Exception exception) {
exception.printStackTrace();
// TODO
}
}
I would like to get to query Windows Vista Search service directly ( or indirectly ) from Java.
I know it is possible to query using the search-ms: protocol, but I would like to consume the result within the app.
I have found good information in the Windows Search API but none related to Java.
I would mark as accepted the answer that provides useful and definitive information on how to achieve this.
Thanks in advance.
EDIT
Does anyone have a JACOB sample, before I can mark this as accepted?
:)
You may want to look at one of the Java-COM integration technologies. I have personally worked with JACOB (JAva COm Bridge):
http://danadler.com/jacob/
Which was rather cumbersome (think working exclusively with reflection), but got the job done for me (quick proof of concept, accessing MapPoint from within Java).
The only other such technology I'm aware of is Jawin, but I don't have any personal experience with it:
http://jawinproject.sourceforge.net/
Update 04/26/2009:
Just for the heck of it, I did more research into Microsoft Windows Search, and found an easy way to integrate with it using OLE DB. Here's some code I wrote as a proof of concept:
public static void main(String[] args) {
DispatchPtr connection = null;
DispatchPtr results = null;
try {
Ole32.CoInitialize();
connection = new DispatchPtr("ADODB.Connection");
connection.invoke("Open",
"Provider=Search.CollatorDSO;" +
"Extended Properties='Application=Windows';");
results = (DispatchPtr)connection.invoke("Execute",
"select System.Title, System.Comment, System.ItemName, System.ItemUrl, System.FileExtension, System.ItemDate, System.MimeType " +
"from SystemIndex " +
"where contains('Foo')");
int count = 0;
while(!((Boolean)results.get("EOF")).booleanValue()) {
++ count;
DispatchPtr fields = (DispatchPtr)results.get("Fields");
int numFields = ((Integer)fields.get("Count")).intValue();
for (int i = 0; i < numFields; ++ i) {
DispatchPtr item =
(DispatchPtr)fields.get("Item", new Integer(i));
System.out.println(
item.get("Name") + ": " + item.get("Value"));
}
System.out.println();
results.invoke("MoveNext");
}
System.out.println("\nCount:" + count);
} catch (COMException e) {
e.printStackTrace();
} finally {
try {
results.invoke("Close");
} catch (COMException e) {
e.printStackTrace();
}
try {
connection.invoke("Close");
} catch (COMException e) {
e.printStackTrace();
}
try {
Ole32.CoUninitialize();
} catch (COMException e) {
e.printStackTrace();
}
}
}
To compile this, you'll need to make sure that the JAWIN JAR is in your classpath, and that jawin.dll is in your path (or java.library.path system property). This code simply opens an ADO connection to the local Windows Desktop Search index, queries for documents with the keyword "Foo," and print out a few key properties on the resultant documents.
Let me know if you have any questions, or need me to clarify anything.
Update 04/27/2009:
I tried implementing the same thing in JACOB as well, and will be doing some benchmarks to compare performance differences between the two. I may be doing something wrong in JACOB, but it seems to consistently be using 10x more memory. I'll be working on a jcom and com4j implementation as well, if I have some time, and try to figure out some quirks that I believe are due to the lack of thread safety somewhere. I may even try a JNI based solution. I expect to be done with everything in 6-8 weeks.
Update 04/28/2009:
This is just an update for those who've been following and curious. Turns out there are no threading issues, I just needed to explicitly close my database resources, since the OLE DB connections are presumably pooled at the OS level (I probably should have closed the connections anyway...). I don't think I'll be any further updates to this. Let me know if anyone runs into any problems with this.
Update 05/01/2009:
Added JACOB example per Oscar's request. This goes through the exact same sequence of calls from a COM perspective, just using JACOB. While it's true JACOB has been much more actively worked on in recent times, I also notice that it's quite a memory hog (uses 10x as much memory as the Jawin version)
public static void main(String[] args) {
Dispatch connection = null;
Dispatch results = null;
try {
connection = new Dispatch("ADODB.Connection");
Dispatch.call(connection, "Open",
"Provider=Search.CollatorDSO;Extended Properties='Application=Windows';");
results = Dispatch.call(connection, "Execute",
"select System.Title, System.Comment, System.ItemName, System.ItemUrl, System.FileExtension, System.ItemDate, System.MimeType " +
"from SystemIndex " +
"where contains('Foo')").toDispatch();
int count = 0;
while(!Dispatch.get(results, "EOF").getBoolean()) {
++ count;
Dispatch fields = Dispatch.get(results, "Fields").toDispatch();
int numFields = Dispatch.get(fields, "Count").getInt();
for (int i = 0; i < numFields; ++ i) {
Dispatch item =
Dispatch.call(fields, "Item", new Integer(i)).
toDispatch();
System.out.println(
Dispatch.get(item, "Name") + ": " +
Dispatch.get(item, "Value"));
}
System.out.println();
Dispatch.call(results, "MoveNext");
}
} finally {
try {
Dispatch.call(results, "Close");
} catch (JacobException e) {
e.printStackTrace();
}
try {
Dispatch.call(connection, "Close");
} catch (JacobException e) {
e.printStackTrace();
}
}
}
As few posts here suggest you can bridge between Java and .NET or COM using commercial or free frameworks like JACOB, JNBridge, J-Integra etc..
Actually I had an experience with with one of these third parties (an expensive one :-) ) and I must say I will do my best to avoid repeating this mistake in the future. The reason is that it involves many "voodoo" stuff you can't really debug, it's very complicated to understand what is the problem when things go wrong.
The solution I would suggest you to implement is to create a simple .NET application that makes the actual calls to the windows search API. After doing so, you need to establish a communication channel between this component and your Java code. This can be done in various ways, for example by messaging to a small DB that your application will periodically pull. Or registering this component on the machine IIS (if exists) and expose simple WS API to communicate with it.
I know that it may sound cumbersome but the clear advantages are: a) you communicate with windows search API using the language it understands (.NET or COM) , b) you control all the application paths.
Any reason why you couldn't just use Runtime.exec() to query via search-ms and read the BufferedReader with the result of the command? For example:
public class ExecTest {
public static void main(String[] args) throws IOException {
Process result = Runtime.getRuntime().exec("search-ms:query=microsoft&");
BufferedReader output = new BufferedReader(new InputStreamReader(result.getInputStream()));
StringBuffer outputSB = new StringBuffer(40000);
String s = null;
while ((s = output.readLine()) != null) {
outputSB.append(s + "\n");
System.out.println(s);
}
String result = output.toString();
}
}
There are several libraries out there for calling COM objects from java, some are opensource (but their learning curve is higher) some are closed source and have a quicker learning curve. A closed source example is EZCom. The commercial ones tend to focus on calling java from windows as well, something I've never seen in opensource.
In your case, what I would suggest you do is front the call in your own .NET class (I guess use C# as that is closest to Java without getting into the controversial J#), and focus on making the interoperability with the .NET dll. That way the windows programming gets easier, and the interface between Windows and java is simpler.
If you are looking for how to use a java com library, the MSDN is the wrong place. But the MSDN will help you write what you need from within .NET, and then look at the com library tutorials about invoking the one or two methods you need from your .NET objects.
EDIT:
Given the discussion in the answers about using a Web Service, you could (and probably will have better luck) build a small .NET app that calls an embedded java web server rather than try to make .NET have the embedded web service, and have java be the consumer of the call. For an embedded web server, my research showed Winstone to be good. Not the smallest, but is much more flexible.
The way to get that to work is to launch the .NET app from java, and have the .NET app call the web service on a timer or a loop to see if there is a request, and if there is, process it and send the response.