How to send key and mouse events to a Java applet? - java

I'm trying to control some Java game from FireFox window. How can I send key and mouse events to that Java applet?
I'm using Windows XP if that matters.
Edit: I'm not trying to do this with Java even though i have the tag here. A c++ solution would be optimal.

You might try using Robot, but this might not work in FireFox. You can also use methods like abstractbutton.doClick()
If Robot doesn't work, key events you can synthesize by just setting text on a component, and mouse events you can use doClick() and requestFocus()
If none of that works, you might be able to accomplish your goals working with javascript and an html page.

Here is something that will work for keystrokes:
The recommended methods for both these actions are using SendInput
This website is perfect for beginning to understand sendinput
To find windows targets use Spy++, documentation
but I do have other examples below:
Example here is for Notepad using postmessage.
#include "TCHAR.h"
#include "Windows.h"
int _tmain(int argc, _TCHAR* argv[])
{
HWND hwndWindowTarget;
HWND hwndWindowNotepad = FindWindow(NULL, L"Untitled - Notepad");
if (hwndWindowNotepad)
{
// Find the target Edit window within Notepad.
hwndWindowTarget = FindWindowEx(hwndWindowNotepad, NULL, L"Edit", NULL);
if (hwndWindowTarget) {
PostMessage(hwndWindowTarget, WM_CHAR, 'G', 0);
}
}
return 0;
}
You may also like to look at windows hooks, which can send mouse input
Or User32 mouse_event:
[DllImport("User32.Dll")]
private static extern void mouse_event(UInt32 dwFlags, int dx, int dy, UInt32 dwData, int dwExtraInfo);
[DllImport("user32.dll")]
static extern bool SetCursorPos(int X, int Y);
public enum MouseEventFlags
{
LEFTDOWN = 0x00000002,
LEFTUP = 0x00000004,
MIDDLEDOWN = 0x00000020,
MIDDLEUP = 0x00000040,
MOVE = 0x00000001,
ABSOLUTE = 0x00008000,
RIGHTDOWN = 0x00000008,
RIGHTUP = 0x00000010
}
public static void SendLeftClick(int X, int Y)
{
mouse_event((uint)MouseEventFlags.LEFTDOWN, 0, 0, 0, 0);
mouse_event((uint)MouseEventFlags.LEFTUP, 0, 0, 0, 0);
}

Related

Writing a PyCharm extension to display text beside source code

I wrote an Eclipse PyDev plugin, and I'm trying to port it to PyCharm, but I can't find an easy way to display a text view next to the Python source code. Is there any way easier than copying the PsiAwareTextEditorImpl class and all its helpers?
I got the splitter control working so I can display something next to the Python source code. In this prototype branch, I displayed a text editor next to the Python editor. Then I created a text file for each Python file to hold the display text, but that displays the text file's name, and it would be a pain to manage a bunch of temporary files.
I stepped through the standard editor class, PsiAwareTextEditorImpl, and found that it has a ton of helper classes, and eventually calls EditorPainter.paintTextWithEffects(). Here are some of the things it does to paint text:
private void paintTextWithEffects(Graphics2D g, Rectangle clip, int startVisualLine, int endVisualLine) {
final CharSequence text = myDocument.getImmutableCharSequence();
// ...
VisualLinesIterator visLinesIterator = new VisualLinesIterator(myEditor, startVisualLine);
while (!visLinesIterator.atEnd()) {
int visualLine = visLinesIterator.getVisualLine();
if (visualLine > endVisualLine || visualLine >= lineCount) break;
int y = visLinesIterator.getY();
final boolean paintSoftWraps = paintAllSoftWraps ||
myEditor.getCaretModel().getLogicalPosition().line == visLinesIterator.getStartLogicalLine();
final int[] currentLogicalLine = new int[] {-1};
paintLineFragments(g, clip, visLinesIterator, y + myView.getAscent(), new LineFragmentPainter() {
#Override
public void paintBeforeLineStart(Graphics2D g, TextAttributes attributes, int columnEnd, float xEnd, int y) {
// ...
}
#Override
public void paint(Graphics2D g, VisualLineFragmentsIterator.Fragment fragment, int start, int end,
TextAttributes attributes, float xStart, float xEnd, int y) {
// ...
}
#Override
public void paintAfterLineEnd(Graphics2D g, Rectangle clip, IterationState iterationState, int columnStart, float x, int y) {
// ...
}
});
visLinesIterator.advance();
}
ComplexTextFragment.flushDrawingCache(g);
}
That seems like a ton of work if I have to reproduce that, so is there some existing component that I can use to display a block of text that isn't coming from a file? Should I be creating my own instance of DocumentImpl and somehow wiring that to an editor?
Here's what the Eclipse plugin looks like with the Python code on the left, and the text display on the right.
You can create instance of LightVirtualFile, fill with any content and display in editor as file of existing type or your custom type.

OpenCV:Detection colored spots

I would like to ask for a help with library OpenCV. I want to ask you if you know the best way how to detect a colored spot from picture. For example I need to create application which can calculate size of "dirty spot" on tshirt. Let's say that there is a brown tshirt and there is also a dirty spot made by katchup or by something else.
Could you recommend me algorithm or technics how to calculate it? Or some tutorial?
I wouldn't ask you for help but I am running out of time and perhaps you meet with that problem before.
Thank you very much.
In openCV samples codes (under cpp samples section), you can find a .cpp file named "bgfg_segm.cpp". Although that code is for motion tracking but i think that you can use to detect the spots also.
There by pressing the "spacebar key", you can start/stop updation of background. Once you have decided your background, then anything extra will be detected as a spot on it.
Strategy: Bring the cloth infront of webcam and once it is selected as a background, then press "spacebar key" to stop further changes in the background. Then, your program should be able to track any change in the color of your cloth.
The code is below:
#include "opencv2/core/core.hpp"
#include "opencv2/imgproc/imgproc.hpp"
#include "opencv2/video/background_segm.hpp"
#include "opencv2/highgui/highgui.hpp"
#include <stdio.h>
using namespace std;
using namespace cv;
int main()
{
VideoCapture cap;
bool update_bg_model = true;
cap.open(0);
namedWindow("image", CV_WINDOW_NORMAL);
namedWindow("foreground mask", CV_WINDOW_NORMAL);
namedWindow("foreground image", CV_WINDOW_NORMAL);
namedWindow("mean background image", CV_WINDOW_NORMAL);
// Declare "object " of class "BackgroundSubtractorMOG2"
BackgroundSubtractorMOG2 bg_model;//(100, 3, 0.3, 5);
Mat img, fgmask, fgimg;
for(;;)
{
cap >> img;
if( fgimg.empty() )
fgimg.create(img.size(), img.type());
//update the model
bg_model(img, fgmask, update_bg_model ? -1 : 0); // "bg_model" is object of class "BackgroundSubtractorMOG2" as declared above.
fgimg = Scalar::all(0);
img.copyTo(fgimg, fgmask);
Mat bgimg;
bg_model.getBackgroundImage(bgimg);
imshow("image", img);
imshow("foreground mask", fgmask);
imshow("foreground image", fgimg);
if(!bgimg.empty())
imshow("mean background image", bgimg );
char k = (char)waitKey(30);
if( k == 27 ) break;
if( k == ' ' ) // Change the Background updation status by Spacebar key
{
update_bg_model = !update_bg_model; // initially "bool update_bg_model = true"
if(update_bg_model)
printf("Background update is on\n");
else
printf("Background update is off\n");
}
}
return 0;
}

Key Pressing partially working

I've been trying to do some kind of autoclicker and have the following code in java:
import java.awt.event.*;
import java.awt.*;
class keyStroke {
public void Execute() throws AWTException {
int n = 0;
while(n < 100){
Robot r = new Robot();
r.delay(1000);
r.keyPress(KeyEvent.VK_1);
r.keyRelease(KeyEvent.VK_1);
++n;
}
}
}
It works pretty fine clicking the key 1, but, it doesn't work in some games.
It looks to be working only on chatbox and accessing to it (enter key), but aside from that, nothing else works (like using a skill or moving).
Then, I decided to also try in C++, with the following code
#include <iostream>
#include <windows.h>
#include <cstdlib>
using namespace std;
void SendKey (char Vk){
char VkKey = VkKeyScan(Vk);
keybd_event(VkKey, 0, 0, 0);
keybd_event(VkKey, 0, KEYEVENTF_KEYUP, 0);
}
int main(){
while(true){
SendKey('1');
Sleep(1000);
}
}
And the same thing happens.
What am U doing wrong? If the keypress doesn't work for this case I have to find something else?
I know from experience that some game input doesn't use an event based structure. Some games only check once every frame if a key is pressed. This means that your chance of pressing the key at that exact moment are zero.
Scripting utilities such as the logitech keyboard scripting tool face a similar problem and there it helps to have a delay between press and release.
Aside: chat windows usually have to use an input event as typing would be almost impossible if key presses are only registered once per frame.

How do I set the position of the mouse in Java?

I'm doing some Swing GUI work with Java, and I think my question is fairly straightforward; How does one set the position of the mouse?
As others have said, this can be achieved using Robot.mouseMove(x,y). However this solution has a downfall when working in a multi-monitor situation, as the robot works with the coordinate system of the primary screen, unless you specify otherwise.
Here is a solution that allows you to pass any point based global screen coordinates:
public void moveMouse(Point p) {
GraphicsEnvironment ge =
GraphicsEnvironment.getLocalGraphicsEnvironment();
GraphicsDevice[] gs = ge.getScreenDevices();
// Search the devices for the one that draws the specified point.
for (GraphicsDevice device: gs) {
GraphicsConfiguration[] configurations =
device.getConfigurations();
for (GraphicsConfiguration config: configurations) {
Rectangle bounds = config.getBounds();
if(bounds.contains(p)) {
// Set point to screen coordinates.
Point b = bounds.getLocation();
Point s = new Point(p.x - b.x, p.y - b.y);
try {
Robot r = new Robot(device);
r.mouseMove(s.x, s.y);
} catch (AWTException e) {
e.printStackTrace();
}
return;
}
}
}
// Couldn't move to the point, it may be off screen.
return;
}
You need to use Robot
This class is used to generate native system input events for the purposes of test automation, self-running demos, and other applications where control of the mouse and keyboard is needed. The primary purpose of Robot is to facilitate automated testing of Java platform implementations.
Using the class to generate input events differs from posting events to the AWT event queue or AWT components in that the events are generated in the platform's native input queue. For example, Robot.mouseMove will actually move the mouse cursor instead of just generating mouse move events...
Robot.mouseMove(x,y)
Check out the Robot class.
The code itself is the following:
char escCode = 0x1B;
System.out.print(String.format("%c[%d;%df",escCode,row,column));
This code is incomplete by itself, so I recommend placing it in a method and calling it something like 'positionCursor(int row, int column)'.
Here is the code in full (method and code):
void positionCursor(int row, int column) {
char escCode = 0x1B;
System.out.print(String.format("%c[%d;%df",escCode,row,column));
}

Using Xlib via JNA to move a window

I'm using JNA to manipulate application windows on Linux by sending Xlib messages but can't seem to move a window.
My original implementation executed wmctrl on the shell to move the windows and that successfully moved the windows. Unfortunately, there's a noticeable amount of overhead associated with calling shell programs from Java, so now I'm trying to make direct API calls using JNA. I'm using the X11 example available from the JNA website and can successfully do a few tricks, such as enumerating the window IDs and reading window properties, so I know JNA+Xlib is at least partially working.
First I tried moving the windows directly using XMoveWindow() but the window manager was apparently blocking those calls.
I ran across a thread that suggested I needed to send a client message using XSendMessage(), so I've done that below, but apparently XSendMessage() is failing because the window doesn't move and I get a return value of 0. I'm guessing I omitted something obvious, but can't quite figure it out. Any suggestions?
Note that, for the purposes of this example, the main method has a window ID hard-coded. This is the window ID of the window I'm trying to move (obtained using wmctrl -l on the console).
import com.sun.jna.NativeLong;
import com.sun.jna.Pointer;
import com.sun.jna.examples.unix.X11;
import com.sun.jna.examples.unix.X11.Atom;
import com.sun.jna.examples.unix.X11.AtomByReference;
import com.sun.jna.examples.unix.X11.Display;
import com.sun.jna.examples.unix.X11.Window;
import com.sun.jna.examples.unix.X11.WindowByReference;
import com.sun.jna.examples.unix.X11.XEvent;
import com.sun.jna.examples.unix.X11.XTextProperty;
import com.sun.jna.examples.unix.X11.XWindowAttributes;
import com.sun.jna.ptr.IntByReference;
import com.sun.jna.ptr.NativeLongByReference;
import com.sun.jna.ptr.PointerByReference;
private static final int FALSE = 0; /** C-style boolean "false" */
private static final int TRUE = 1; /** C-style boolean "true" */
public static void main(String[] args) {
setWindowPos(new Window(0x01300007), 100, 100, 600, 400); // update the Window constructor with the appropriate ID given by wmctrl -l
}
public static boolean setWindowPos(Window window, int x, int y, int w, int h) {
final X11 x11 = X11.INSTANCE;
Display display = x11.XOpenDisplay(null);
NativeLong mask = new NativeLong(X11.SubstructureRedirectMask | X11.SubstructureNotifyMask | X11.ResizeRedirectMask);
XEvent event = new XEvent();
String msg = "_NET_MOVERESIZE_WINDOW"; //$NON-NLS-1$
long grflags = 0l; // use the default gravity of the window
if (x != -1) grflags |= (1 << 8);
if (y != -1) grflags |= (1 << 9);
if (w != -1) grflags |= (1 << 10);
if (h != -1) grflags |= (1 << 11);
event.xclient.type = X11.ClientMessage;
event.xclient.serial = new NativeLong(0l);
event.xclient.send_event = TRUE;
event.xclient.message_type = x11.XInternAtom(display, msg, false);
event.xclient.window = window;
event.xclient.format = 32;
event.xclient.data.l[0] = new NativeLong(grflags); // gravity flags
event.xclient.data.l[1] = new NativeLong(x);
event.xclient.data.l[2] = new NativeLong(y);
event.xclient.data.l[3] = new NativeLong(w);
event.xclient.data.l[4] = new NativeLong(h);
int status = x11.XSendEvent(display, x11.XDefaultRootWindow(display), FALSE, mask, event);
x11.XFlush(display); // need to XFlush if we're not reading X events
if (status == 0) { // 0 indicates XSendEvent failed
logger.error("setWindowPos: XSendEvent failed (" + msg + ")"); //$NON-NLS-1$
return false;
}
return true;
}
This might be a bit of a late answers but anyway...
What happens when you try to move a window is that the window (called "the client") sends an XConfigureRequest to the window manager. This happens because the window manager tells the X server that he is boss (by setting the substructure override flag on the client's parent).
The only way to bypass this is to set the override redirect flag on your client, do the move, and disable the override redirect flag (so that everything goes back to 'normal').
gl & hf.
Have you looked at XConfigureWindow?
I haven't actually tested this out yet since I just implemented it tonight and I'm developing on Windows, but it's worth a try....
public static interface X11Ext extends Library
{
public static X11Ext INSTANCE = (X11Ext)Native.loadLibrary("X11", X11Ext.class);
public int XConfigureWindow(X11.Display display, X11.Window window, int value_mask, XWindowChanges changes);
/**
* Use value_mask flags:
* CWX
* CWY
* CWWidth
* CWHeight
* CWBorderWidth
* CWSibling
* CWStackMode
*/
public class XWindowChanges extends Structure
{
public int x;
public int y;
public int width;
public int height;
public int border_width;
public X11.Window sibling;
public int stack_mode;
}
}

Categories

Resources