I am working on a research project.The scenario is this.
I am taking the screenshot of my desktop and then I process it using an API to get the position of a certain text on my Desktop.e.g , say I have the browser open on my desktop and I am on stackoverflow.Now I want to search the position of the logo stackoverflow on the screenshot taken.Then I want to simulate a click on it.I am using Java platform.
Now I have 2 questions:
1)Is there any free API(OCR) which I can use to process the screenshot to fetch text position (or can be done by some trick) and gives good results.
Or Any way you can suggest that I can use (instead of taking screenshot and processing it) to get the position of any text on the screen.
2)How can I simulate the click on the screen using the code by a background program running(I mean I have done it in Swing and other language UIs but this time its different as Now I want to click on the screen.
If I understood you right you want to move your mouse and click on the screen. That not that hard you could use the robot class from Java!
For example:
Robot rob = new Robot();
rob.keyPress( KeyEvent.VK_ENTER );
or what ever, there are so much bottons and movements you could with it. A list of all methods you find here.
And your other question I can't answer. I think there is no API that is able to search a text and give you the position. But what I know is that the robot class is able to capture the screen and put it into a BufferedImage. With it you could compare two pictures.
Maybe you could get use of this but I don't know if it is what you search.
Related
I have been working on a libGDX project and was wondering if I could create a login screen with the use of some kind of text inputs. I have been looking all around for text input fields, but the only one I can seem to find are those that pop up, which I don't really want to use.
My project is a desktop game only and uses the screen implementations combinede with a game super class.
Is there any other way to get text input inside the window, so not as a pop up?
As said in the comment, the right way to make a login screen is to use Scene2d.ui and its TextField. Check out the Scene2d.ui wiki page and Skin Composer to get going.
I have 2 exact same images (pattern, color, size ,everything is same) on the screen and in this case how do I click on Image 1?
Whenever I am using the s.click() function, Sikuli tries to click on Image 1 some times and sometimes on Image 2.
I have tried Pattern and exists but they are also not working as expected.
Pattern imagePatternPath = new Pattern("Image.png").exact();
screen.click(imagePatternPath);
I expect to click on Image 1, but the actual output is sometime it clicks on Image 1 and sometimes on Image 2.
If there are two absolutely similar patterns on the screen, there is no way for Sikuli to distinguish them without some help. You have few options to resolve this problem.
location on the screen
If the patterns appear in known areas on the screen, you can limit the search to the expected area and thus, avoid picking the wrong pattern.
search by proximity
If you have any other objects that appear only next to one of the patterns you are attempting to locate but not the other, you can use them as pivots.
Have a look into Sikuli code, especially into the Region class here. It exposes various options to modify the search area around found patterns. For example:
public Region grow(int range) {
return grow(range, range);
}
Try looking at the patch notes next time, they are very helpful. But here is what I would try:
result = findAll("Image.png")
This will return a List which you can iterate through to click the item you wat so in this case you would click the first item.
I am working on a program that will allow the user to record steps of simple tasks and then generate a file to send to people that show these steps. So if you left click on a window it will say "User Left Clicked on Google Chrome" with an appropriate screenshot and highlighted cursor for visbility.
I am using Java Native Hook Found here for the global mouse/key listeners and Java Native Access Found here to get the title of the application that is clicked.
I would like to include something that highlights an area where text is entered. At the moment I am thinking of taking a screenshot when the user clicks a textbox and then storing all the keys that are pressed (for the guide) and taking a second screenshot after the text has been input, and also adding a highlight outline around the text.
I feel like it would be easier to generate the highlighting if I could get the location of the caret but i'm not exactly sure how to do this for global applications.
I want to write an address in the address bar of a browser as well as click on a link using java Robot class. How can I track the different objects in a certain window?
Just giving a look at the API http://docs.oracle.com/javase/7/docs/api/java/awt/Robot.html, anyone who do this should know that via Robot Class there is no "trackComponent(Component specificComponent)" method, you got 2 things that may help you:
1-getPixelColor (more than help, seems useless for you by now, maybe i'm wrong).
2-createScreenCapture.
the second method is maybe the answer for your problem, you could take a picture of the screen and with some image processor (javaCV could help you on this: https://code.google.com/p/javacv/) you could then track the components on the screen you took (for instance: from pixels xxx to pixels yyy is the Address bar of browser), of course you need to read some documentation about javaCV (OpenCV) for get this done, after that just use the method for move cursor and enter keys for fill the components, hope someone give a simpler way to do this, but i think this way you learn a bit of JavaCV a really powerful tool.
Firstly I am no longer a student and doing this for other purposes, so don't hold back on the help ;)
I want to incorporate a simple program into my webpage. I want 4 buttons labelled right arm, left arm, activate voice and walk forward. There will be a box above these buttons showing an image of a robot and as the buttons are pressed by the user I want a different image to be loaded in the box.
So if the right arm button is pressed the image with the robot raising its right arm will need to be displayed.
So basically all I want the buttons to do is to load the image that belongs to each one. I am assuming java is the best choice? I have the open source Java package, would I need any other software when it comes to embedding it into a webpage? My webpage is done on dreamweaver and I am pretty good with html. Would appreciate it if someone could point me to the right direction.....Thanks
Don't use Java for this. Use JavaScript. Something like this:
HTML:
<img src="one.jpg" id="firstImage" />
<img src="two.jpg" id="secondImage" />
<button id="doSomething">Do Something</button>
<button id="doSomethingElse">Do Something Else</button>
CSS:
img {
display: none;
}
JavaScript:
var hideImages = function() {
$('#firstImage').hide();
$('#secondImage').hide();
};
$('#doSomething').click(function() {
hideImages();
$('#firstImage').show();
});
$('#doSomething').click(function() {
hideImages();
$('#secondImage').show();
});
What this essentially does is initially display no images, just buttons. Then as each button is clicked, the corresponding image is displayed. (And all other images are first hidden, since previous button clicks would have displayed previous images.)
This is an overly simple proof of concept, of course. At the very least you'll want to use better variable names :) This just demonstrates the idea of how you'd show/hide images in JavaScript in response to button clicks. (This also assumes the use of jQuery, which is a safe assumption these days. The easiest way to use that is to refer to a CDN link in your page, using a standard HTML script tag.) You can see this code in action here (though the images are broken, of course.)
If there are a lot of buttons and a lot of images, you may be able to re-factor the code to be less repetitive as well. Maybe store the images in an array and have a single button click handler which can associate the sending button with the correct array element, etc. That's up to you.
There are many technologies that could help you: javascript is one of them and it would be much simpler than using Java for such simple thing. You can use Java applet if you really want to use Java for that project.