A friend of mine asked me to implement a blue and a red pointer to represent the inputs of two separate mice to expedite a mixing desk scenario for real time audio mixing. I'd love to, but as much as I think it is a great idea, I don't have a clue as to where to start looking for a possible solution.
Where should I start researching a viable method of implementing dual mouse inputs?
Look at jinput.
I have had multiple keyboards working with it, I am nearly certain it supports multiple mice too.
dont know about java.. but for C#/c++ you can try the
Microsoft Windows MultiPoint Software Development Kit
i've tried it on windows.. it works with 2 USB mice.
It depends on which operating system you intend to use.
On Windows, you can use:
CPNMouse - a driver+software combination, very flexible and allows to completely hide some of the mice from the operating system.
RawInput - an API provided by Windows XP only. You can use it to distinguish between two mouse inputs, draw the cursors yourself and hide the main cursor. Take a look at the code of SDGT, a C# usage of this API.
For both solutions you would have to build a JNI bridge to your application
If you are using X.Org (X11) server, there is patched version called MPX that should support multiple mice even for legacy applications. It should now be a part of the X.Org trunk, but I'm not very familiar with it. Anyway it has an API so you can use it via JNI bridge.
You can use multiple devices, but at the Java level, all mouse events are coalesced into a single stream. The event does not include which mouse it came from. You did say you wanted to mix audio, right? Well this mix might be interesting, but surely not what you want.
I'd suggest using the Java-supported midi interface and connecting some simple midi controller device with multiple knobs or trackballs. These will come in as midi events, and you can examine the state for the details you need.
Related
I have made a Screen Recorder using Java Swing and Xuggler 5.4. I have developed it in Windows 8 64 bit. It's working excellent for Windows. But at client side on Linux's environment , nothing is working. I have searched thoroughly but not getting any solutions. I have checked this thread , but it didn't work for me.
Then I tried to create simple Transparent window in Linux but it's also not working. I was not able to click through the Resizeable Panel. I have used the same JRE version (1.7) for both. Have I miss understood Java's Cross Platform Support as far as Swing is concerned?
Please Give Me Some Advice...
I have always found logging to be the best debugging tool at your disposal! Many a times, java debuggers take you into APIs where you need not go every time. Logging values of your variables, and generic 'I have reached till this point' statements make life a lot easier.
So, I suppose you have ample logging done in your code. That could give you clues on what's happening on your client's system.
Are the right environment variables set? Are they pointing to the correct Java versions you need.
If there are some specific Screen capturing requirements(plugins / modules / API) your code has, are they available on the Linux m/c?
Like #MadProgrammer said, in the end, Java has to talk with the native graphics APIs to render your screen.
I would try to debug it in this way -
Check whether my main screen loads or no(by disabling the screen capture functions for a while).
if not, dig deeper.
Check whether all necessary components for capturing screen(audio and video) are available.
Check whether the code is being run with appropriate permissions to control the h/w devices you may need.
I'm looking for a language or library to allow me to simulate key strokes at the maximum level possible, without physically pressing the key.
(My specific measure of the level of the keystroke is whether or not it will produce the same output as a physical Key Press when my computer is already running key listeners (such as MouseKeys and StickyKeys)).
I've tried many methods of keystroke emulation;
The java AWT library, Java win32api, python win32com sendKeys, python ctypes Key press, and many more libraries for python and Java, but none of them simulate the key stroke at a close enough level to actual hardware.
(When Windows MouseKeys is active, sending a key stroke of a colon, semi colon or numpad ADD key just produces those characters, where as a physical press performs the MouseKeys click)
I believe such methods must involve sending the strokes straight to an application, rather than passing them just to the OS.
I'm coming to the idea that no library for these high (above OS code) level languages will produce anything adequate. I fear I might have to stoop to some kind of BIOS programming.
Does anybody have any useful information on the matter whatsoever?
How would I go about emulating key presses in lower level languages?
Should I be looking for a my-hardware-specific solution (some kind of Fujitsu hardware API)?
I almost feel it would be easier to program a robot to simply sit by the hardware and press the keys.
Thanks!
Second solution, super convoluted, a ton of virtualization, diabolical, totally untested, but theoretically should work, unless the datasnip program doesn't actually write to the keyboard buffer, but instead simulates keystrokes like you've been trying to. That would suck, and I would find the description of their product to be highly misleading.
You'll need:
http://www.priority1design.com.au/datasnip.html
http://sourceforge.net/projects/com0com/
And some knowledge of writing to com ports... which, looks like there's a good python module here: http://pyserial.sourceforge.net/
First, write a small program that will send characters, hexcode, etc. as necessary to the COM port of your choosing. Second, create a virtual COM port pair using com0com. At this point, connect your program to one of the COM ports created, and datasnip to the other, making sure that both sides of the communication are using identical baud rates, parity and stop bits, and data length parameters.
At this point you should have a keyboard that is identical to a hardware one, as far as the OS can tell.
I used this solution http://oblita.com/interception.html
It fully meets my needs (sending keystrokes to direct input game)
Can you use an MSDN language? They're fairly simple and well documented (google searches are better than the msdn website, for navigation, usually)
I found this bit, and I couldn't tell you if it simulates keystrokes at a level close enough to the hardware, but it does appear to be able to generate keystroke events that other programs can recognize, which is what your ultimate goal is.
http://msdn.microsoft.com/en-us/library/ms171548.aspx
The 'Send' and 'SendWait' Methods seem to be the key ones.
http://msdn.microsoft.com/en-us/library/system.windows.forms.sendkeys.sendwait.aspx
Godspeed.
I'm not on a Windows box to test it against MouseKeys, so no guarantees that it will work, but have you tried AutoHotkey?
I want to automate an external application, but I have several problems:
How can I recognize a button or other field of an external application in Java?
I use the Robot class in Java for making notepad automation where I open notepad, select file menu, and save or exit, etc.
The problem is, it needs X,Y coordinates for the mouse pointer to go to the proper location.
I want to make it more dynamic, i.e. it should recognize the file menu of a running notepad anywhere on the desktop.
How can this be done in Java? Is there any class in Java I can use to do this?
Thanks everyone to give me response, I want to be more specific i want to know how can i make ui automation by using any tool if it is not possible in java or using any api of java.automation tool must be freeware.....i am searching net for that i found AutoIt is like that.But if any one do this type of things please share his/her experiance means is it possible to do that in AutoIt or not possible if not then which tool do that kind of things.
It is easy to integrate Sikuli into a Java-application since it is written in Java. Sikuli uses image recognition to find elements visible on the screen like buttons and such. It is very easy to use and provides an alternative for tasks that are difficult to handle with static positioning, like finding moving windows and such.
Take a look at this: http://sikuli.org/docx/faq/030-java-dev.html
Hope this helps!
You should have a look at Sikuli. It takes as inputs images of the ui elements to select an area in the targeted app. It's a UI Automation Application
That's a bit difficult to install (at least on Debian/Ubuntu, where I tested it), as you'll need a recent version of OpenCV, a particular version of JXGrabKey but the quality of the program worth the trip. Good Luck
Java doesn't have an API to examine the UI of another application; that would be a very big security risk.
Which is why the Robot class can only record events (key presses, mouse movements and clicks) but not which UI element was involved in most cases.
It would be possible to do more if the external application was written in Java because then, you could analyze the objects in memory but for obvious reasons, this isn't possible for C++ or .NET applications.
Some swing code I write in my computer behave different on my colleague's computer, and in my PC, and in my notebook.
I wonder, is there something I can do to my Swing applications behave the same in every computer?
I want to have sure a algorithm I've tested in my computer will work the same way in my clients computers.
E.g.
Problem to focus JTextField works fine in my notebook with Windows XP, but not in my collague's computer with Windows XP, nither in my work computer with Ubuntu.
obs. the specific JTextField problem is not the subject of this question.
Problems with Swing apps on different platforms are common and they are caused by the simple fact that no matter what level of abstraction Java offers it has to play ball at some point with the native components of the underlying operating system. Event though Swing only uses the windows(frame) and draw everything by itself - discrepancies are very very common.
I develop a mutliplatform Swing application - and users on Windows are reporting all sorts of issues that Linux users don't have and vice versa. Sadly there is no silver bullet for such problems - extensive testing and nasty fixes are the only game in town.
And everything come exceptionally buggy and dirty in the area of pluggable look and feels. For example - resizing a JSplitPane with metal or nimbus is super fast(as expected), but if you use GTK+ plaf, everything goes to hell. This is a more serious(performance) problem - visual problems(missing borders, components not fitting properly containers, etc) have no end... Despite all of this Swing continues to be one of the best bet for multiplatform desktop applications.
I wonder, is there something I can do
to my Swing applications behave the
same in every computer?
I'm working on a complex Java Swing app that is shipped on OS X / Windows / Linux so just like Bozhidar answered, the issues are all far too real.
For some components, if you happen to have some UI design/programming skills, you can simply write your own component: I realize it's probably not a helpful answer, but it works.
For example, we wanted a drop-down "find-as-you-type" popup list (like the one that appears when you start a search on Google's main search page) that would look and work the same on Linux/Windows/OS X. After trying countless of "solutions" full of Swing idiosynchrasies that would not work everywhere (like, guess what, focus issues ; ) we decided to simply write our own component "from scractch".
We can intercept mouse and keyboard events on both OS X / Windows / Linux: we can write a component that not only looks but also behaves identically on all three platforms.
In addition to the "find-as-you-type", we also wrote our own tooltip-popup component, a dual progress bar (to progress bar in one to show producer/consumer style progress in a single bar) and a complex component involving several "text fields" which was absurdly complex and broken when we tried to do it using Swing (and broken in different ways on different platforms, like weird focus issues or caret not showing, etc.). So we "went dirty" and rewrote the entire component ourselves.
Result? Working identically on all platforms where Java can give you notifications about mouse and keyboard events...
I realize this may not be what you want to hear: I happen to have worked on both games UI and mobile apps UI back in the days and I have some graphic skills so it's not "hard" for me to write good looking UI components.
Sadly if you want some Java UI component to look and behave identically on all platforms, it's sometimes your only alternative...
I want to have sure a algorithm I've
tested in my computer will work the
same way in my clients computers
An "algorithm" should work fine.
You get problems when you rely on the ordering of events, which may be different on various platforms. One of the most common I know about is holding down a key so that is repeats:
a) on Windows you get keyPressed, keyPressed, keyPressed, .... keyReleased.
b) on Unix you get keyPressed, keyReleased, keyPressed, keyReleased ...
By the way a comment would be nice as to whether my suggestion in your "textfield" posting works or not. As I mentioned, I don't have a Ubuntu platform to test it on, so I'm curious as to the result.
I want to be able to record mouse movements, clicks and keyboard input from a user. It would be great if it was a cross platform solution.
I'd like to get back something like this (pseudo code):
mouse moved to 500, 500
mouse double clicked
mouse moved to 800, 300
mouse left clicked
keyboard typed "Hello World"
Does either C++ or Java have any classes that can do this? If I was using C++, I would probably working with the QT framework.
Edit:
I should have said this originally, but I want to record the movements and clicks outside of the applications gui, so on the desktop too.
GLUT does this, but it's tied to OpenGL which might be overkill for your project.
OpenGL is cross-platform.
I don't believe there's a cross-platform toolkit specifically for grabbing input from a window and nothing more, but most toolkits provide this capability. Two good options are:
Use SDL, as it's fairly lightweight and can handle simple input.
Implement the functionality natively per platform, as it should be trivial in X11, Windows, Mac OS X, etc.
On Windows, this is called a Journal Record Hook. You should write the hook part in C or C++, it might be technically possible to do in java, but it's not a good idea, you want your hook procedure to have as few dependencies as possible, and to be a quick as possible. System wide hooks, especially journal add a lot of overhead to keyboard and mouse input, you want to minized your impact as much as possible.
You Install Windows hooks by using SetWindowsHookEx passing WH_JOURNALRECORD to get a Journal Record Hook.
You could also (maybe) get this working by installing both WH_KEYBOARD_LL and WH_MOUSE_LL, but your two hook procedures would be called separately, and you would have to write your own code to put the events in order.
I doubt you will find a cross-platform solution.
It sounds like Qt might allow you to implement event filters that extend beyond the application to the window system. See also Qt - top level widget with keyboard and mouse event transparency?
If you want to trap events across the whole GUI system, not just one app, there's not much likelihood of a cross platform solution. However, the event hooking part could easily be separated from the recording part, so you could make most of the program cross-platform.
For Windows, you need this 17 year old (!) document. (Man, I'm getting old!)