I would like to know how I would kind of like run a online browser game, like tetris, as a Java application so I could control it (like in tetris using the wasd or arrow keys to move blocks) from within the application I program for the game. Not sure if that makes sense but don't know how else to explain it. The purpose is to hook it up to an AI program that plays the game, but yeah I don't know how to allow the program to control the game using the games allowed inputs.
Java.awt.Robot seems to be what you need
It should be able to simulate any keyboard or mouse input, but if you need to do anything too complicated I would suggest something other than Java - it doesn't get too close to the operating system, so something like c++ may be more accurate.
Related
So I'm coding a side project that involves Twitch chat using commands to play games on my PC. The question I have is, I want it to recognize if a window is active then do something. Just for a random example, let's say that MS Paint is open. If another window is active, the chat won't be able to use commands anymore. but when it's active again, they can. Does anyone have any ideas on how to approach this? If I had to write pseudo-code:
if (window.active("MS PAINT"){
// do something
}
Any hints or points in a direction would be appreciated.
I'm new to programming but I really wish to improve, right now learning Java. Basically there is this game, online strategy it's pretty basic with most of the images are static and very few of them change. I wish to change how I view the game from my end and add some functionality to it as well.
For example the website displays an orange rectangle, I wish to display a red rectangle by drawing over it. Or drawing a button when, if clicked, re-directs me to another part of the game. Originally I am using Selenium Chrome WebDriver to navigate the website but it seems you can't paint or display your own images over it. So I was wondering if I should use Selenium headless in the background performing tasks while I just make a Java Application. Make the WebDriver capture the image every second and display it on the Java Application every second, and than edit that image to my liking? Would this be effective way to do it or is there better options?
It is not clear what you are actually trying to achieve here. And whether your approach is likely to be "effective" depends what you are trying to achieve!
However, I can see some issues:
I don't know if it is technically possible to get the Selenium WebDriver to do that ...
Capturing images at a high rate could be problematic from a performance perspective.
Once you have captured the images:
Modifying the images by hand would be tedious and labour intensive. (And probably not what you want to do.)
Modifying static images programatically would be possible, but fiddly to implement and incredibly fragile. The slightest change to the game's screen rendering could result in your "tweaks" writing to the wrong place on the image.
Making changes to the dynamic part of images would be next to impossible ... except in the destructive sense.
You can't affect game play by this approach. Nothing you do to the images will impact on the browser and hence on the server. You can't add functionality this way.
In short, this is a poor idea, and unlikely to achieve anything worthwhile.
Can I think of any good options? No I can't.
The best I can think of would be to:
reverse engineer the HTML / Javascript uploaded to your browser,
figure out what you want to change in that content, and
place an HTTP proxy between your web browser and the server that injects the relevant changes into the content.
However:
It will be a lot of tricky and tedious work to implement.
It will be fragile; i.e. changes to the game / game server are liable to break your "hacks".
It could be a violation of the game owner's copyright and/or the license that grants you the right to use the game. This could result in your game accounts being cancelled, along with the accounts of anyone else who uses your "hacks".
My advice? Just play the game!
I'm writing a Java application to automate character actions in an online game overnight (specifically, it catches fish in Final Fantasy XI). The app makes heavy use of java's Robot class both for emulating user keyboard input and for detecting color changes on certain parts of the screen. It also uses multithreading and a swing GUI.
The application seems to work perfectly when I test it without the game running, just using screenshots to trigger the apps responses into notepad. But for some reason, when I actually launch FFXI and start the program, all of my keyboard and mouse manipulations just stop working altogether. The program is still running, and the Robot class is still able to read pixel colors. But Robot.keyPress, Robot.keyRelease, Robot.mouseMove, Robot.mousePress and Robot.mouseRelease all do nothing. It's the strangest thing-- to test it, I wrote a simple loop that just keeps typing letters, and focused notepad. I'd then start the game, refocus notepad, and it would do nothing. Then I'd exit the game, and it'd start working again immediately.
Has anyone else come across something like this, where specific software will stop certain functions of java from working?
Also, to make this more interesting-- Last year I wrote a very similar program using the same classes and programming techniques to automate healing a party in the game as they fight. Last year, this program worked perfectly. After running into these problems I dug up that old program, ran it without making any changes, and found that it too was having the same problems. The only differences between now and when it was working: I was running Windows Vista and now I'm running Windows 7, and several new Java versions as well as FFXI versions have been released.
What the hell is going on? (if anyone needs to see my source code, email me at mikejturley#gmail.com. I'm trying to keep it to myself.)
FFXI has code to prevent cheating. Quite effectively, it would seem.
If possible, try it in WinXP. I myself have also written a bot for an online game that uses much of the same concepts (i.e. using Java Robot to read pixel colors and simulate key-presses and mouse-clicks).
Under WinXP:
Bot works as intended in all cases.
Under Win7:
Outside of the game, bot works as intended. Ingame, simulated input failed (pixel reads were okay, I think).
I want to be able to record mouse movements, clicks and keyboard input from a user. It would be great if it was a cross platform solution.
I'd like to get back something like this (pseudo code):
mouse moved to 500, 500
mouse double clicked
mouse moved to 800, 300
mouse left clicked
keyboard typed "Hello World"
Does either C++ or Java have any classes that can do this? If I was using C++, I would probably working with the QT framework.
Edit:
I should have said this originally, but I want to record the movements and clicks outside of the applications gui, so on the desktop too.
GLUT does this, but it's tied to OpenGL which might be overkill for your project.
OpenGL is cross-platform.
I don't believe there's a cross-platform toolkit specifically for grabbing input from a window and nothing more, but most toolkits provide this capability. Two good options are:
Use SDL, as it's fairly lightweight and can handle simple input.
Implement the functionality natively per platform, as it should be trivial in X11, Windows, Mac OS X, etc.
On Windows, this is called a Journal Record Hook. You should write the hook part in C or C++, it might be technically possible to do in java, but it's not a good idea, you want your hook procedure to have as few dependencies as possible, and to be a quick as possible. System wide hooks, especially journal add a lot of overhead to keyboard and mouse input, you want to minized your impact as much as possible.
You Install Windows hooks by using SetWindowsHookEx passing WH_JOURNALRECORD to get a Journal Record Hook.
You could also (maybe) get this working by installing both WH_KEYBOARD_LL and WH_MOUSE_LL, but your two hook procedures would be called separately, and you would have to write your own code to put the events in order.
I doubt you will find a cross-platform solution.
It sounds like Qt might allow you to implement event filters that extend beyond the application to the window system. See also Qt - top level widget with keyboard and mouse event transparency?
If you want to trap events across the whole GUI system, not just one app, there's not much likelihood of a cross platform solution. However, the event hooking part could easily be separated from the recording part, so you could make most of the program cross-platform.
For Windows, you need this 17 year old (!) document. (Man, I'm getting old!)
Recently I made an autoclicker for a game, which would allow you to use F1 and F2 to turn it on or off, once I finished the application I thought to myself, "oh great, I forgot Java doesn't let you record outside keystrokes and mouse movements", is there any simple non-native ways to do this?
Thanks.
Nope. You can do application wide keyboard and mouse handling, but you have very few options if your application is not the foreground application. The best you can do is discover the mouse location:
java.awt.MouseInfo.getPointerInfo().getLocation()
This will always work. You are out of luck for keyboard input though. For that you will need to write native code.