I am getting live video feedback from Parrot AR.Drone 2.0. I am able to get the incoming video streams from drone(using command-ffplay tcp://192.168.1.1:5555) and successfully output the live video for me. I notice that ffplay will display its own frame along with the live video.
So, is that possible to "direct" or put the frame into our own Java frame in application? How could I achieve that if I wish to implement that function in my own JCheckBox? E.G. If I click JCheckBox, it should automatically get live video streams from drone and display for me in application instead of using ffplay frame?
This question is old but I just found it so I decided to write a possible solution for other users.
There are to ways to solve this one wrapping yourself the FFmpeg CLI for Java or using a wrapped FFmpeg library. The former will take a quite time and effort to make it. While the latter will be more practical to use.
For example, a nice library is JavaCV. You just need to add the maven repository to your project using the pom.xml file:
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>javacv-platform</artifactId>
<version>1.4.3</version>
</dependency>
Then you can create a SimplePlayer class and use the FFmpegFrameGrabber class to decode a frame that is converted into an image and displayed in your Java app.
public class SimplePlayer
{
private static volatile Thread playThread;
private AnimationTimer timer;
private int counter;
public SimplePlayer(String source, GrabberListener grabberListener)
{
if (grabberListener == null) return;
if (source.isEmpty()) return;
counter = 0;
playThread = new Thread(() -> {
try {
FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(source);
grabber.start();
grabberListener.onMediaGrabbed(grabber.getImageWidth(), grabber.getImageHeight());
Java2DFrameConverter converter = new Java2DFrameConverter();
while (!Thread.interrupted()) {
Frame frame = grabber.grab();
if (frame == null) {
break;
}
if (frame.image != null) {
Image image = SwingFXUtils.toFXImage(converter.convert(frame), null);
Platform.runLater(() -> {
grabberListener.onImageProcessed(image);
});
}
grabber.stop();
grabber.release();
Platform.exit();
} catch (Exception exception) {
System.exit(1);
}
});
playThread.start();
}
public void stop()
{
playThread.interrupt();
}
}
You can find the full implementation in this GitHub repository.
You need to decode the almost-but-not-quite H264 video format that the AR.Drone uses. To do that, you need to do two things:
Handle the AR.Drone's custom video framing, which uses headers in their PaVE format. The format is documented in section 7.3 of the AR.Drone Developer Guide, but almost any of the existing AR.Drone libraries have code to handle PaVE headers.
Decode the H264 video frames that remain. Xuggler wraps native libraries with Java, and is probably the best, fastest way to decode H264. An alternative is the h264j library, which is pure Java, but is slower and has some decoding glitches.
For more info, see these related questions:
Xuggler and playing from live stream
https://stackoverflow.com/questions/30307003/use-a-xuggler-videostream-in-javacv
Related
Is there any solution to "Extract a frame from video file in Java using core library without importing external libraries"?
Say for I saw Image, BufferedStrategy, BufferCapabilities in Java AWT libraries.
The Java Media Framework API (JMF) enables audio, video and other time-based media operations, without use of any third party library.
Seeking frames inside a movie with JMF.
xuggler is a good third party library, widely used.
I think that you should use Xuggler from here or you can find it in maven.
In the github repository is a sample under demos with the file:
DecodeAndCaptureFrames.java
According to this answer on another question, you can do that without external libraries by leveraging features of JavaFX.
Quoting original answer below:
You can use the snapshot() of MediaView. First connect a mediaPlayer
to a MediaView component, then use mediaPlayer.seek() to seek the
video position. And then you can use the following code to extract the
image frame:
int width = mediaPlayer.getMedia().getWidth();
int height = mediaPlayer.getMedia().getHeight();
WritableImage wim = new WritableImage(width, height);
MediaView mv = new MediaView();
mv.setFitWidth(width);
mv.setFitHeight(height);
mv.setMediaPlayer(mediaPlayer);
mv.snapshot(null, wim);
try {
ImageIO.write(SwingFXUtils.fromFXImage(wim, null), "png", new File("/test.png"));
} catch (Exception s) {
System.out.println(s);
}
I am using IP Webcam APP for android and it is streaming MJPEG video through the local url :
http://192.168.0.2:8080/video
I was able to show the video using VLC player and this piece of code in C++.
On the OpenCV 2.2 I opened the url using:
VideoCapture cap;
cap.open("http://192.168.0.2:8080/video?dummy=param.mjpg");
It worked in C++, but I want it to work in Java. I was able to run OpenCV2.4.9 using Java when taking pictures from my built in webcam. This is my code for taking the images from a url in Java.
System.loadLibrary("opencv_java249");
VideoCapture capture = new VideoCapture();
capture.open("http://192.168.0.2:8080/video?dummy=param.mjpg");
But the capture.open does not open the streaming and I could not debug it properly. I know that it might be a issue with the ffmpeg, since it works on OpenCV2.2. I also know that my OpenCV2.2 is specific for MS 2010 and might be more complete.
Would it help if I compile the OpenCV2.4.9 from sources? Is there a file that I could add to solve that problem? Is there another way of receiving the video from the IP camera and using on OpenCV?
I took a while to figure it out. I could not receive the stream directly from OpenCVJava.I downloaded
http://www.mediafire.com/download/ayxwnwnqv3mpg39/javacv-0.7-bin.zip http://www.mediafire.com/download/2rkk0rjwxov7ale/javacv-0.7-cppjars.zip
Which I believe to be a java wrapper into OpenCV in C. I took this link from this video.
htttp://www.youtube.com/watch?v=mIYaHCyZICI
After unziping the zip I added the jars into my project and Used this code:
package javaapplication7;
import java.io.IOException;
import com.googlecode.javacv.OpenCVFrameGrabber;
import com.googlecode.javacv.CanvasFrame;
import com.googlecode.javacv.cpp.opencv_core.IplImage;
public class JavaApplication7 {
public static void main(String[] args) throws Exception {
OpenCVFrameGrabber grabber = new OpenCVFrameGrabber("http://192.168.0.2:8080/video?dummy=param.mjpg");
grabber.setFormat("mjpeg");
grabber.start();
for (int k=0; k<20000; k++){
System.out.print(k);
}
IplImage frame = grabber.grab();
CanvasFrame canvasFrame = new CanvasFrame("Camera");
canvasFrame.setCanvasSize(frame.width(), frame.height());
while (canvasFrame.isVisible() && (frame = grabber.grab()) != null) {
canvasFrame.showImage(frame);
}
grabber.stop();
canvasFrame.dispose();
System.exit(0);
}
}
Which I got from:
htttp://stackoverflow.com/questions/14251290/cvcreatefilecapture-error-could-not-create-camera-capture-with-javacv
It takes 15-20 seconds to start catching the streaming. But I was impressed with the delay which is much smaller than VLC. It is 1-2 seconds comparing to 3-4 seconds on VLC. I would like to upvote the guy who I took the answer from but I dont have enough reputation/
I also bumped into the same problem as you but the easiest method i figured out was to use droid cam instead of the Ip webcam app.Check it out here
i am trying to simulate a live view using a canon Camera.
I am interacting with the cam using the CanonSDK, i get an image every a short period in order to simulate a video frame by frame. This works fine, i am using java to do the backend and send the images trough BlazeDS to flex.
The problem is not getting the image, the problem is that when i load a new image using something like:
image.source=my_new_image;
the new image is loaded but it produces a short white blink and it ruins the video...
So i would like to know if the is a way to update an image on flex avoiding the blinking problem, or if i could make a video streaming from java and pick it up with flex...
Thanks in advance!!!
The easy way is to use a technique called double buffering, using two Loaders - one for the image which is visible, and one for the image which is being loaded and is invisible. When the image has completed loading it becomes visible, and the other one becomes invisible and the process repeats.
In terms of efficiency, it would be better to at least use a socket connection to the server for transferring the image bytes, preferably in AMF format since it has little overhead. This is all fairly possible in BlazeDS with some scripting.
For better efficiency you may try using a real-time frame or video encoder on the server, however decoding the video on the client will be challenging. For best performance it will be better to use the built-in video decoder and a streaming server such as Flash Media Server.
UPDATE (example script):
This example loads images over HTTP. A more efficient approach would be to use an AMF socket (mentioned above) to transfer the image, then use Loader.loadBytes() to display it.
private var loaderA:Loader;
private var loaderB:Loader;
private var foregroundLoader:Loader;
private var backgroundLoader:Loader;
public function Main()
{
loaderA = new Loader();
loaderB = new Loader();
foregroundLoader = loaderA;
backgroundLoader = loaderB;
loadNext();
}
private function loadNext():void
{
trace("loading");
backgroundLoader.contentLoaderInfo.addEventListener(Event.COMPLETE, loaderCompleteHandler);
backgroundLoader.load(new URLRequest("http://www.phpjunkyard.com/randim/randim.php?type=1"));
}
private function loaderCompleteHandler(event:Event):void
{
trace("loaded");
var loaderInfo:LoaderInfo = event.target as LoaderInfo;
var loader:Loader = loaderInfo.loader;
loader.contentLoaderInfo.removeEventListener(Event.COMPLETE, loaderCompleteHandler);
if (contains(foregroundLoader))
removeChild(foregroundLoader);
var temp:Loader = foregroundLoader;
foregroundLoader = backgroundLoader;
backgroundLoader = temp;
addChild(foregroundLoader);
loadNext();
}
Can I do it only with the lti-Civil classes or I need FMJ or JMF ?
do I use the actual Stream or set of captured Images from CaptureObserver?
Thanks.
captureStream = system.openCaptureDeviceStream(info.getDeviceID());
captureStream.setObserver(new CaptureObserver(){
public void onError(CaptureStream arg0, CaptureException arg1) {
}
public void onNewImage(CaptureStream streamSource, Image image) {
try
{
imageFrame.setImage(AWTImageConverter.toBufferedImage(image));
bufferedImage = AWTImageConverter.toBufferedImage(image);
}
catch (Throwable t)
{ t.printStackTrace(); }
}
});
captureStream.start();
If I understand your question correctly, the introduction on Lti-civil site have the answer:
LTI-CIVIL is a Java library for capturing images from a video source such as a USB camera. It provides a simple API and does not depend on or use JMF! The FMJ project integrates LTI CIVIL into the JMF architecture by providing a civil: datasource. For more information, visit screenshots and project status sections of this website
As far as I can see your code is not using any JMF specific classes so the above code should work with plain LTI-CIVIL.
EDIT:
Based on your clarification, I think you will need additional support for encoding the video as LTI-CIVIL seems to be focused solely on capturing. Out of FMJ and JMF, FMJ is probably a better choice as JMF's development appears to be ceased. For other alternatives see this section of wikipedia article about JMF
I am looking into an image processing problem for semi-real time detection of certain scenarios. My goal is to have the live video arrive as Motion JPEG frames in my Java code somehow.
I am familiar with the Java Media Framework and, sadly, I think we can consider that an effectively dead API. I am also familiar with Axis boxes and, while I really like their solution, I would appreciate any critical feedback on my specific points of interest.
This is how I define "best" for the purpose of this discussion:
Latency - if I'm controlling the camera using this video stream, I would like to keep my round-trip latency at less than 100 milliseconds if possible. That's measured as the time between my control input to the time when I see the visible change. EDIT some time later: another thing to keep in mind is that camera control is likely to be a combination of manual and automatic (event triggers). We need to see those pictures right away, even if the high quality feed is archived separately.
Cost - free / open source is better than not free.
Adjustable codec parameters - I need to be able to tune the codec for certain situations. Sometimes a high-speed low-resolution stream is actually easier to process.
"Integration" with Java - how much trouble is it to hook this solution to my code? Am I sending packets over a socket? Hitting URLs? Installing Direct3D / JNI combinations?
Windows / Linux / both? - I would prefer an operating system agnostic solution because I have to deliver to several flavors of OS but there may be a solution that is optimal for one but not the other.
NOTE: I am aware of other image / video capture codecs and that is not the focus of this question. I am specifically not interested in streaming APIs (e.g., MPEG4) due to the loss of frame accuracy. However, if there is a solution to my question that delivers another frame-accurate data stream, please chime in.
Follow-up to this question: at this point, I am strongly inclined to buy appliances such as the Axis video encoders rather than trying to capture the video in software or on the PC directly. However, if someone has alternatives, I'd love to hear them.
This JavaCV implementation works fine.
CODE:
import com.googlecode.javacv.OpenCVFrameGrabber;
import com.googlecode.javacv.cpp.opencv_core.IplImage;
import static com.googlecode.javacv.cpp.opencv_highgui.*;
public class CaptureImage {
private static void captureFrame() {
// 0-default camera, 1 - next...so on
final OpenCVFrameGrabber grabber = new OpenCVFrameGrabber(0);
try {
grabber.start();
IplImage img = grabber.grab();
if (img != null) {
cvSaveImage("capture.jpg", img);
}
} catch (Exception e) {
e.printStackTrace();
}
}
public static void main(String[] args) {
captureFrame();
}
}
There is also post on viewing live video from Camera .And configuration for JavaCV :
I think this will meet your requirements.
FMJ can definitely capture video and turn it into MJPEG frames.
Regarding the dead-ness of JMF, are you aware of the FMJ implementation? I don't know whether it qualifies as the "best" solution, but it's probably worth adding to the discussion.
Below is shown a very simple implementation using Marvin Framework. Using Marvin you can add real time video processing easily.
import javax.swing.JFrame;
import marvin.gui.MarvinImagePanel;
import marvin.image.MarvinImage;
import marvin.video.MarvinJavaCVAdapter;
import marvin.video.MarvinVideoInterface;
public class SimpleVideoTest extends JFrame implements Runnable{
private MarvinVideoInterface videoAdapter;
private MarvinImage image;
private MarvinImagePanel videoPanel;
public SimpleVideoTest(){
super("Simple Video Test");
// Create the VideoAdapter and connect to the camera
videoAdapter = new MarvinJavaCVAdapter();
videoAdapter.connect(0);
// Create VideoPanel
videoPanel = new MarvinImagePanel();
add(videoPanel);
// Start the thread for requesting the video frames
new Thread(this).start();
setSize(800,600);
setVisible(true);
}
public static void main(String[] args) {
SimpleVideoTest t = new SimpleVideoTest();
t.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
}
#Override
public void run() {
while(true){
// Request a video frame and set into the VideoPanel
image = videoAdapter.getFrame();
videoPanel.setImage(image);
}
}
}
Another example applying multiple algorithms for real time video processing.
This is my JavaCV implementation with high resolution video output and no noticeable drop in the frame-rate than other solutions (only when my webcam refocuses do I notice a slight drop, only for a moment though).
import java.awt.image.BufferedImage;
import java.io.File;
import javax.swing.JFrame;
import com.googlecode.javacv.CanvasFrame;
import com.googlecode.javacv.OpenCVFrameGrabber;
import com.googlecode.javacv.OpenCVFrameRecorder;
import com.googlecode.javacv.cpp.opencv_core.IplImage;
public class Webcam implements Runnable {
IplImage image;
static CanvasFrame frame = new CanvasFrame("Web Cam");
public static boolean running = false;
public Webcam()
{
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
}
#Override
public void run()
{
try
{
grabber.setImageWidth(800);
grabber.setImageHeight(600);
grabber.start();
while (running)
{
IplImage cvimg = grabber.grab();
BufferedImage image;
if (cvimg != null)
{
// opencv_core.cvFlip(cvimg, cvimg, 1); // mirror
// show image on window
image = cvimg.getBufferedImage();
frame.showImage(image);
}
}
grabber.stop();
frame.dispose();
} catch (Exception e)
{
e.printStackTrace();
}
}
public static void main(String... args)
{
Webcam webcam = new Webcam();
webcam.start();
}
public void start()
{
new Thread(this).start();
running = true;
}
public void stop()
{
running = false;
}
}
Have you ever looked at Processing.org? It's basically a simplified application framework for developing "artsy" applications and physical computing platforms, but it's based on Java and you can dig down to the "real" Java underneath.
The reason it came to mind is that there are several video libraries for Processing which are basically Java components (at least I think they are - the site has all the technical information you might need). There is a tutorial on using the Processing libraries and tools in the Eclipse IDE. There are also numerous examples on video capture and processing.
Even if you can't use the libraries directly, Processing is a great language/environment for working out algorithms. There are several great examples of image and video capture and real-time processing there.