I'm toying with an idea for creating a Java application to automate a process that I have to do regularly and before I start any coding I thought I would seek advice as to the best way to approach it.
Basically, the application I use has a large number of images present on the screen at any one time, and what I would like to know is if there is a way to have Java identify if any of these two images are the same. If they are, I would like to automate mouse movement and button clicks.
After a bit of reading, I'm thinking that the PixelGrabber and Robot classes might be the right way to start, but like I said, I'm looking for any information on this that can be offered.
What are your suggestions?
I believe the Robot class and a Pixel Grabber would be sufficient. If you are inclined to program the solution yourself, maybe for educational purpose, by all means please do. If you, however, don't want to reinvent the wheel, you may take a look at this project:
http://sikuli.org/
I, for example, use it to do stuff that would be hard to achieve with Selenium alone. If you still can't achieve your goal after some scripting, Sikuli provides a nice API which you can use from inside your java program.
The Robot class would be sufficient to take images and being able to inspect pixels. But it seems to make more sense, to recreate your desktop with images inside of a java application (a very simple gallery application). Then operations are simplier. An other way of realizing operations I do not see.
Related
I'm new to programming but I really wish to improve, right now learning Java. Basically there is this game, online strategy it's pretty basic with most of the images are static and very few of them change. I wish to change how I view the game from my end and add some functionality to it as well.
For example the website displays an orange rectangle, I wish to display a red rectangle by drawing over it. Or drawing a button when, if clicked, re-directs me to another part of the game. Originally I am using Selenium Chrome WebDriver to navigate the website but it seems you can't paint or display your own images over it. So I was wondering if I should use Selenium headless in the background performing tasks while I just make a Java Application. Make the WebDriver capture the image every second and display it on the Java Application every second, and than edit that image to my liking? Would this be effective way to do it or is there better options?
It is not clear what you are actually trying to achieve here. And whether your approach is likely to be "effective" depends what you are trying to achieve!
However, I can see some issues:
I don't know if it is technically possible to get the Selenium WebDriver to do that ...
Capturing images at a high rate could be problematic from a performance perspective.
Once you have captured the images:
Modifying the images by hand would be tedious and labour intensive. (And probably not what you want to do.)
Modifying static images programatically would be possible, but fiddly to implement and incredibly fragile. The slightest change to the game's screen rendering could result in your "tweaks" writing to the wrong place on the image.
Making changes to the dynamic part of images would be next to impossible ... except in the destructive sense.
You can't affect game play by this approach. Nothing you do to the images will impact on the browser and hence on the server. You can't add functionality this way.
In short, this is a poor idea, and unlikely to achieve anything worthwhile.
Can I think of any good options? No I can't.
The best I can think of would be to:
reverse engineer the HTML / Javascript uploaded to your browser,
figure out what you want to change in that content, and
place an HTTP proxy between your web browser and the server that injects the relevant changes into the content.
However:
It will be a lot of tricky and tedious work to implement.
It will be fragile; i.e. changes to the game / game server are liable to break your "hacks".
It could be a violation of the game owner's copyright and/or the license that grants you the right to use the game. This could result in your game accounts being cancelled, along with the accounts of anyone else who uses your "hacks".
My advice? Just play the game!
I m currently working on my project of remote desktop administration. I m using robot class to capture images and send over network. It works well but bit slower.
Because all the time we need to captuure and send image its too costly. Is it possible to detect only a portion of screen which is changed and send only that portion?
Please any one guide me on this. Thank you!!!
The keyword you're looking for (in order to be able to look this up and figure the solution yourself) is dirty rectangles.
You can look into some code here.
I looked into this awhile back, and the image capture is implemented particularly inefficiently. I don't recall the specific detail, but it was pretty bad the way they did it. I felt, at the time, that the only way to do it better would be to implement it in JNI. Which you could use JNA to shortcut.
I don't know if any platform's screen capture routines will allow only changed sections to be sent, but you could implement a decent image diff; although that could get expensive too. You would really need to measure whats going on to see if it works for you.
I am trying to create a screen like this one
Initially I was trying to port a console program to java. I have found things like jcurses and charva but there is almost no documentation on them and I really cant understand how to use them.
So I figured that I could create a console like screen that resembled the one above.
what library / framwork would I be best using. Should I use swing as it if fully portable?
what would be the best approach being that I need to be able to navigate and alter the 00 in the picture above?
Easily usable would be great but as long as it has good documentation that I can learn it from that would be fine.
(Answering as if you're looking for a hex editor.)
http://jhecomponent.sourceforge.net/
http://www.fifesoft.com/hexeditor/
http://hexedit-lib.sourceforge.net/
Played a bit with the fifesoft.com offering, kinda cool.
If you're not looking for a hex editor, can you be more specific? If you just want a cursor-addressable window you'll probably have to suck it up and figure out something like jcurses or libjcsi :)
If you prefer to create a full fledged GUI with Java you could certainly use Swing. I would prefer SWT as a matter of personal taste, the widget library of Eclipse. You will find a lot of snippets and tutorials (same for Swing). Here is the Widget Library.
Here is a discussion about SWT versus Swing.
You could use the table layout manager to create the layout shown in your screenshot. If you rewrite the application from scretch, you should be able to handle the GUI events and update the widgets according to your application needs. You will easily find articles when you search for swt and table. If you would like to keep the code base and just exchange the GUI, I 'm not sure about the best approach. Maybe, the libs jcurses and charva are the way to go.
Unless you are looking to run the app on a headless VM (where a console/tty is all you've got), I'd use Swing. Create a JTable with a custom TableModel (to provide the data) and custom renderers (to provide the hex formatting) and everything should just work. Consider deriving from the various DefaultXXX implementations to save a lot of work.
I want to automate an external application, but I have several problems:
How can I recognize a button or other field of an external application in Java?
I use the Robot class in Java for making notepad automation where I open notepad, select file menu, and save or exit, etc.
The problem is, it needs X,Y coordinates for the mouse pointer to go to the proper location.
I want to make it more dynamic, i.e. it should recognize the file menu of a running notepad anywhere on the desktop.
How can this be done in Java? Is there any class in Java I can use to do this?
Thanks everyone to give me response, I want to be more specific i want to know how can i make ui automation by using any tool if it is not possible in java or using any api of java.automation tool must be freeware.....i am searching net for that i found AutoIt is like that.But if any one do this type of things please share his/her experiance means is it possible to do that in AutoIt or not possible if not then which tool do that kind of things.
It is easy to integrate Sikuli into a Java-application since it is written in Java. Sikuli uses image recognition to find elements visible on the screen like buttons and such. It is very easy to use and provides an alternative for tasks that are difficult to handle with static positioning, like finding moving windows and such.
Take a look at this: http://sikuli.org/docx/faq/030-java-dev.html
Hope this helps!
You should have a look at Sikuli. It takes as inputs images of the ui elements to select an area in the targeted app. It's a UI Automation Application
That's a bit difficult to install (at least on Debian/Ubuntu, where I tested it), as you'll need a recent version of OpenCV, a particular version of JXGrabKey but the quality of the program worth the trip. Good Luck
Java doesn't have an API to examine the UI of another application; that would be a very big security risk.
Which is why the Robot class can only record events (key presses, mouse movements and clicks) but not which UI element was involved in most cases.
It would be possible to do more if the external application was written in Java because then, you could analyze the objects in memory but for obvious reasons, this isn't possible for C++ or .NET applications.
I can't seem to find anybody who has done or posted something like this; Essentially I want to design my own UI in photoshop and then slice down the images to use it in a Java application. Essentially coding in the PSD file as the GUI. Is this possible? If so, can anybody lead me in the right direction?
I'm not sure what editor to use for this sort of stuff. I am using the Eclipse IDE and I know there is a Visual Editor but, I already have the actual design for every component in a PSD file. All I want to do is to start incorporating this into the application. Thanks.
It depends on how far your design goes. If you simply want to have normal Swing components on top of your image this is easy. Convert your PSD into (for example) PNG, create a custom JPanel subclass that loads the image and overwrite the paintComponent() method to draw the image instead of the normal background. All child components can then be set to be transparent with setOpaque(false). This puts your image into the background and puts the components float on top of it.
If you want to change how individual components look, its a lot more work. You basically need to implement a new Look&Feel for Swing. I wouldn't recommend going that route, unless you really have to, we are talking about weeks of work here, and it requires a lot of testing to really make it work properly on all platforms.
Alternately, there are already tons of custom Look&Feels available, I suggest you take a look at some freely available ones (just google "java look and feel"). Many of them can be customized to some degree (how much depends on the actual implementation, so take a close look at the source/documentation for each of them).
You might want to take a look at NetBeans which has a Swing GUI Builder. You would have to redraw your components there, and then write all the code to process the events. It is sometimes good to start with that, though often times it is less frustrating to lay them out with code by hand as it can difficult to make changes in code and have the builder keep up. There is nothing I know that will let you start from a photoshop image and proceed to building a GUI. Sounds like a good project to make someone rich. :-)