Sikuli on Java does not recognize images - java

I have been trying to automatize some tasks on my computer and did choose Sikuli from Java to do so (I work with Java everyday and didn't know any automation tool using java, sikuli was the first I found). I use java with maven and eclipse as IDE. I have added Sikuli as a Maven dependency.
<dependency>
<groupId>com.sikulix</groupId>
<artifactId>sikulixapi</artifactId>
<version>2.0.5</version>
</dependency>
I tried to do some simple stuff. I did a few screenshots of parts of my screen using windows' screenshot tool, and wanted sikuli to hover it. It works quite fine for one image, but not at all for the others. It seems that the bigger the image the better it works as I did not have success for any small images. The one working is a screen of a whole window (reduced to ~1/4 of my screen).
I also tried to find a button inside this window, to find the windows logo on bottom left, to find a screen of my package explorer, but none work correctly.
I played with similar() using various values, but it didn't improve the results. In some cases (button inside the window) it did find a result for some low similar value, but it was another button. The weird part is : its finding this other button which is bright blue, while the one i'm looking for is purple.
My pc background never changes, I did some screen.highlight() and its looking at the correct screen (dual screen). It's not an issue with the path to images (already solved this one).
Do you have any idea of what I could try ? I have read about people having different success rate depending on whether they were using Sikuli IDE or another IDE. So maybe I could give sikuli IDE a try.
I can give code samples as soon as I am back home.
The code I'm using to test :
public class CleanTest {
static Screen screen = new Screen();
public static void main(String[] args) throws FindFailed, AWTException, IOException, InterruptedException {
String pathYourSystem = System.getProperty("user.dir") + "\\";
System.out.println(pathYourSystem);
Pattern pLauncher = new Pattern(pathYourSystem+"img\\full_launcher.PNG").similar(0.9d);
Desktop.getDesktop().open(new File("path_to_an_exe_opening_a_launcher"));
screen.wait(pLauncher, 300);
screen.mouseMove();
System.out.println("launcher found");
}
}
It works with the "full launcher" image, but it doesn't find a sub-part of the launcher (a button). I tried to make some code to test if there was some threshold for the similar parameter :
double similarValue = 1d;
Pattern pLauncher = new Pattern(pathYourSystem+"img\\the_button.PNG").similar(similarValue);
Desktop.getDesktop().open(new File("path_to_an_exe_opening_a_launcher"));
while(!screen.has(pLauncher)) {
similarValue-=0.1;
pLauncher = new Pattern(pathYourSystem+"img\\login.PNG").similar(similarValue);
}
System.out.println(similarValue);
screen.mouseMove();
it finds something at around 0.5, but it's a totally different button.
Thank you !
EDIT: if someone has the same issue, try to use sikulix IDE to take the screenshots. It works with the screenshots taken by the IDE.

This is a simple test, that completely stays within the SikuliX features.
import org.sikuli.basics.Debug;
import org.sikuli.script.*;
public class SikulixTest {
public static void main(String[] args) {
System.out.println("SikulixTest");
Screen scr = new Screen();
// craete an image to be searched on the screen
Image img = new Image(scr.userCapture());
// try to find it
Match mImg = scr.exists(img);
if (mImg != null) {
// show sthg. when found
Debug.info("%s", mImg);
mImg.highlight(2);
}
}
}
This is RaiMan from SikuliX

Related

How to add a seekbar to a video played using vlcj in Java Swing?

I’ve trimmed down the code to only the relevant parts and posted it below. The code works fine. The video plays when you run it but it doesn’t have a seekbar.
public class Screen {
//JFrmae
private JFrame frame;
// Panel which I add the canvas to
private JPanel pVid = new JPanel();
// Canvas
Canvas canvas = new Canvas();
// Embedded Media Player
EmbeddedMediaPlayer emp;
/**
* Create the application.
*/
public Screen() {
initialize();
}
/**
* Initialize the contents of the frame.
*/
private void initialize() {
//Frame
frame = new JFrame();
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
frame.setVisible(true);
//Adding the panel to the frame
frame.getContentPane().add(pVid);
//Adding the canvas to the panel
pVid.add(canvas);
//Setting canvas size
canvas.setSize(715, 402);
//Loading the VLC native library
NativeLibrary.addSearchPath(RuntimeUtil.getLibVlcLibraryName(), "lib");
Native.loadLibrary(RuntimeUtil.getLibVlcLibraryName(), LibVlc.class);
//Initializing the media player
MediaPlayerFactory mpf = new MediaPlayerFactory();
//Misc
emp = mpf.newEmbeddedMediaPlayer(new Win32FullScreenStrategy(frame));
emp.setVideoSurface(mpf.newVideoSurface(canvas));
//Video file name and playing
String file = "video.mp4";
emp.prepareMedia(file);
emp.play();
//pack method
frame.pack();
}
/**
* Launch the application.
*/
public static void main(String[] args) {
EventQueue.invokeLater(new Runnable() {
public void run() {
try {
Screen window = new Screen();
window.frame.setVisible(true);
} catch (Exception e) {
e.printStackTrace();
}
}
});
}
}
I’ve looked for an answer online for the last 4 days. Finally I decided to ask here. The official website for vlcj has pictures of a vlcj player they’ve made. There is a seekbar in those pictures. Link to the webpage which has the pics: http://capricasoftware.co.uk/#/projects/vlcj
They have a number of useful tutorials there but they don’t have any instructions for adding the seekbar.
Then I tried downloading their vlcj-player project from their GitHub page. It showed an error because it couldn’t resolve the “com.google.common.collect.ImmutableList” which is supposed to be imported. (At the moment I’m reading about ImmutableList and stuff and see if there’s a way to fix it.) Since I couldn’t figure that out yet, I looked for a class named seekbar or the like in their project. I couldn’t find any.
I also searched elsewhere online for the answer but I just couldn’t find it. I’d really appreciate any help. Thank you.
Edit:
(This edit is in response to the suggestion given to me by #caprica. Read their comment to this question and my reply to that in the comment to understand what I'm talking about here in this edit. I think it'll be useful for others in the future.)
All right, there must have been some problem with my Eclipse or computer. (I’ll type out how I fixed it at the end of this comment.) It’s working now. I’ll type out what I did step by step so that may be it’ll be useful to others in the future to download and install the project.
Download the project.
Import it as a Maven project. (Import > Maven > Existing Maven Project)
Now in Eclipse right click the imported project and select Run As > Maven Install
And that’s it. Now you can just run the project normally. If you don’t know how to run the project, do it like this. Right click the project and select Run As > Java Application and then Select VlcjPlayer – uk.co.caprica.vlcplayer.
Alternatively you can open the class where the main method is and run it. VlcjPlayer class is where the main method is located. The class is in the package uk.co.caprica.vlcplayer.
The problem I faced was that somehow all the necessary files didn’t get downloaded when I ran it as Maven Install. But it worked fine in another computer. Since I knew where the files are downloaded to, I just copied the folder from that PC and put it in the same place in my PC. The folder name is ‘repository’. It’s location is C:\Users\User Name\ .m2. Perhaps Eclipse in this PC has some problem. I’ll reinstall it later to avoid problems in the future.
And this may be useful, the VLC that’s installed in this PC is 64 bit. Not sure if that makes a difference but mentioning it just in case.
Now that the app is working fine I will see the code and see how the seekbar is made. Thanks a lot #caprica for telling me that I should import it as a Maven project. :)
The Basic Controls tutorial shows the essential approach: Add a panel of buttons to the frame and give each button an ActionListener that invokes the relevant media player command. As an example, this notional Rewind button would "skip backwards 10 seconds (-10,000 milliseconds)."
JPanel controlsPane = new JPanel();
JButton rewindButton = new JButton("Rewind");
rewindButton.addActionListener(new ActionListener() {
#Override
public void actionPerformed(ActionEvent e) {
mediaPlayerComponent.getMediaPlayer().skip(-10000);
}
});
controlsPane.add(rewindButton);
frame.add(controlsPane, BorderLayout.SOUTH);
The software design is up to you, but you should at least be aware of
JToolBar, seen here and here.
Action, seen here and cited here.
Timer, seen here as a way to repeat an Action.
All right, guys. I’ve figured out how to do it. I’m not sure how it was done in the official Vlcj project but I’ve figured out my own simple way by learning from the official project.
It just takes a few lines of code. It’s very simple.
These are the steps you have to follow:
Create a JSlider.
To the JSlider, add a mouseMotionListener (‘mouseDragged’ to be exact).
Inside that put in the code which would update the video position based on
the change in the JSlider.
Create a Timer.
Put the code inside it to set the value of the JSlider based on the position
of the video.
And that’s it!
This is the code. It comes inside the initialize() method which you can see in the code I’ve given in the question. (And of course you'll also have to create the JSlider and add it to the panel. I haven't shown the code since it's simple.)
js.addMouseMotionListener(new MouseMotionAdapter() {
#Override
public void mouseDragged(MouseEvent e) {
if (js.getValue() / 100 < 1) {
emp.setPosition((float) js.getValue() / 100);
}
}
});
Timer timer = new Timer(100, new ActionListener() {
#Override
public void actionPerformed(ActionEvent e) {
js.setValue(Math.round(emp.getPosition() * 100));
}
});
timer.start();
Some explanation.
The value you get when you use emp.getPosition always seems to be in decimals. It’s something like 0.1334344 at the start of the video and it’s something like 0.998988 at the end. But the value of JSlider is in int. From 0 to 100. So in the mouseMotionListener added to the JSlider I’ve converted the int value of the JSlider to float by dividing it by 100.
And in the action listener inside the timer I’ve multiplied the value of the video position by 100 and then rounded it off to make it an int value. So that value can be set in the JSlider to make it move in sync with the video.
I’m sure the code is rudimentary and there could be some best practices which I may not have followed. Sorry about that but I’m just getting into java by learning the stuff which I find interesting. Those who are good at java and have used such code in an actual project can comment below with how it can be improved.

Is there any way to record the screenshots of minimized browser windows

I am using Java and selenium to write some tests. I need to have my screen records while the tests are running it makes much easier for me to track if any bugs occurs. The problem is that I need to run more than one tests at the same time and as I have only one monitor I cannot record all of their screen records at the same time so I have to run the test one after each other. I was wondering if there is any way that I can run all my tests and actually minimize their browsers windows but still record what is going on at each minimized chrome window. My question may sound a bit very strange but that makes my testing very faster.
Yes, definitely we can take multiple screenshots. There is no affect whether the browser is in minimize or Maximize condition. Just you have to switch the new opened window & add "Take screenshot" method after each method where you have to take screenshot.
Take screenshot method can work in both mode while browser is either in Minimize or Maximize condition.
For screenshot you can may use the below code:
public void getscreenshot() throws Exception
{
File scrFile = ((TakesScreenshot)driver).getScreenshotAs(OutputType.FILE);
//The below method will save the screen shot in d drive with name "screenshot.png"
FileUtils.copyFile(scrFile, new File("D:\\screenshot.png"));
}
or you can opt for the multi-screen capture and the code for that is below :
public void GoogleAbout() throws Exception {
driver.get(baseUrl); // Enter the URL as per your choice
driver.findElement(By.linkText(Prop.getProperty("AboutLinkText"))).click(); //find the web element
MultiScreenShot multiScreens = new MultiScreenShot("C:\\New\\","GoogleAbout"); // Here we need to create the object which will take two arguement one is the path to save the file and second one is class name
multiScreens.multiScreenShot(driver);
driver.findElement(By.linkText("More about our philosophy")).click();
multiScreens.multiScreenShot(driver);
}
To enable the multi-screenshot you have to download the JAR file and then attached it to your project and then :
import multiScreenShot.MultiScreenShot;

Image Comparison in selenium using sikuli

We are using sikuli for image comparison in our automation, but i'm getting below error when i ran the test.
We are using sikuli because we are doing image comparison for small part of the web page.
Error i'm getting:
FindFailed: can not find P(D:\Automation\test-data\Student.jpg) S: 0.99 on the screen.
Line 1574, in file Region.java
code is:
public static void FindPattern(String BaseImage)
{
try{
Pattern imagepattern = new Pattern(BaseImage);
imagepattern.similar((float)0.99);
Screen screen=new Screen();
screen.find(imagepattern)
}catch(Exception e){
e.printStackTrace();
}
}
Does the same image work if you test it from Sikuli IDE? If it does then there could be couple of things which are causing the failure.
1.Application and script sync issue. You might be looking for the image even before it is available on the page.
2.The screenshot may be having too much of background in it.

How to simulate a real mouse click using java?

I'm attempting to perform a mouse click in Java, to click something in an external program. To do this, I'm using java.awt.robot, and the following code:
Robot bot = new Robot();
int mask = InputEvent.MOUSE_BUTTON1_DOWN;
bot.mouseMove(x, y);
bot.mousePress(mask);
bot.mouseRelease(mask);
Here's the problem. The external program is able to detect that this click is computer-generated and not human-generated, and hence, its rejecting this click.
I have already tried moving the mouse there naturally and that didn't have any effect. So my guess is, that it must be listening to the keyboard state or such, and telling from that, that the click is computer generated.
What do I have to do to set all keyboard / mouse states to act in the same way as a normal mouse click would?
Well I had the same exact requirement, and Robot class is perfectly fine for me. It works on windows 7 and XP (tried java 6 & 7).
public static void click(int x, int y) throws AWTException{
Robot bot = new Robot();
bot.mouseMove(x, y);
bot.mousePress(InputEvent.BUTTON1_DOWN_MASK);
bot.mouseRelease(InputEvent.BUTTON1_DOWN_MASK);
}
May be you could share the name of the program that is rejecting your click?
FYI, in newer versions of Windows, there's a new setting where if a program is running in Adminstrator mode, then another program not in administrator mode, cannot send any clicks or other input events to it. Check your source program to which you are trying to send the click (right click -> properties), and see if the 'run as administrator' checkbox is selected.
it works in Linux. perhaps there are system settings which can be changed in Windows to allow it.
jcomeau#aspire:/tmp$ cat test.java; javac test.java; java test
import java.awt.event.*;
import java.awt.Robot;
public class test {
public static void main(String args[]) {
Robot bot = null;
try {
bot = new Robot();
} catch (Exception failed) {
System.err.println("Failed instantiating Robot: " + failed);
}
int mask = InputEvent.BUTTON1_DOWN_MASK;
bot.mouseMove(100, 100);
bot.mousePress(mask);
bot.mouseRelease(mask);
}
}
I'm assuming InputEvent.MOUSE_BUTTON1_DOWN in your version of Java is the same thing as InputEvent.BUTTON1_DOWN_MASK in mine; I'm using 1.6.
otherwise, that could be your problem.
I can tell it worked because my Chrome browser was open to http://docs.oracle.com/javase/7/docs/api/java/awt/Robot.html when I ran the program, and it changed to Debian.org because that was the link in the bookmarks bar at (100, 100).
[added later after cogitating on it today]
it might be necessary to trick the listening program by simulating a smoother mouse movement. see the answer here: How to move a mouse smoothly throughout the screen by using java?
With all respect the most likely thing is that you are mistaken about why the click is being 'rejected'. Why do you think some program is trying to determine if it's human or not? The Robot class (have used it a lot) should send messages that the operating system has no way to distinguish from a user doing the click.
Some applications may detect click source at low OS level. If you really need that kind of hack, you may just run target app in virtual machine's window, and run cliker in host OS, it can help.
You could create a simple AutoIt Script that does the job for you, compile it as an executable and perform a system call there.
in au3 Script:
; how to use: MouseClick ( "button" [, x, y [, clicks = 1 [, speed = 10]]] )
MouseClick ( "left" , $CmdLine[1], $CmdLine[1] )
Now find aut2exe in your au3 Folder or find 'Compile Script to .exe' in your Start Menu and create an executable.
in your Java class call:
Runtime.getRuntime().exec(
new String[]{
"yourscript.exe",
String.valueOf(mypoint.x),
String.valueOf(mypoint.y)}
);
AutoIt will behave as if it was a human and won't be detected as a machine.
Find AutoIt here: https://www.autoitscript.com/

How to write java program in Sikuli?

I have recorded the GUI desktop application using SIKULI as below:
App.open ("C:\\Program Files\\acd\\bin\\VPNClient.exe")
sleep(1)
type ("mganda1")
sleep(1)
click( ) //click OK
I want to convert this script into Java. So I am trying as below:
package com.arcot.test.vpn;
import org.sikuli.script.*;
public class AuthLogin {
public static void main(String[] args) {
Screen s = new Screen();
App myApp = new App("application-identifier") ;
myApp.open ("C:\\Program Files\\acd\\bin\\VPNClient.exe");
//How to simulate the type, sleep and click functions here?
I am searching for java examples to understand the objects relation and how to use it to simulate the recorded scripts. Please provide if any of you know the links that help me.
Best regards,
Madhu
After your program, proceed in following way:
package com.arcot.test.vpn;
import org.sikuli.script.*;
public class AuthLogin {
public static void main(String[] args) {
Screen s = new Screen();
App myApp = new App("application-identifier") ;
myApp.open ("C:\\Program Files\\acd\\bin\\VPNClient.exe");
Kindly proceed in this way,
-Create one image folder inside your package "img"
-Copy all the respective images in the img folder
-Assign the image names in a folder to a different variables
For doing operations, use follwing command:
s.type("mganda1");
s.sleep(time);
s.click("ok.png");
Regards,
Npesik
Madhu,
I'm not sure why you recorded the script to lunch that app with sikuli. All of the commands yu use don't invoke any images and can all be written without the sikuli ide.
I would make the following changes to your original sikuli/jython script
App.open ("C:\\Program Files\\acd\\bin\\VPNClient.exe")
sleep(1)
//change to
wait(path to image, FOREVER)
//By changing to a wait there is an implicit find as defined by the path to the image
type ("mganda1")
//if there are issues verifying focus invoke type with the img option
sleep(1)
//use wait instead of sleep
click( ) //click OK
//What are you clicking on?
Regarding Java, here's the link to Sikuli javadocs

Categories

Resources