Develop Reference

Error of FFmpeg on Java in "av_image_copy_to_buffer" method during decoding H.264 stream

I'm trying to decode H.264 stream, which is sent over Socket from an Android application to a computer. And I also want to show the decoded stream using JavaFX. I searched for a long time, and decided to use JavaCV / FFmpeg. However I got error from FFmpeg. (I was inspired by this code.)
Questions:
Why does FFmpeg make error?
Is it a correct way to convert AVFrame to javafx.scene.image.Image?
I'm using:
javacv-platform 1.4.4
ffmpeg-platform 4.1-1.4.4
Code:
This is a part of import and class fields, and method which runs once at the first time. (Actually the content of initialize() is wrapped by try~catch.)
import javafx.scene.image.Image;
private avcodec.AVCodec avCodec;
private avcodec.AVCodecContext avCodecContext;
private avutil.AVDictionary avDictionary;
private avutil.AVFrame avFrame;
public void initialize() {
avCodec = avcodec_find_decoder(AV_CODEC_ID_H264);
if (avCodec == null) {
throw new RuntimeException("Can't find decoder");
}
avCodecContext = avcodec_alloc_context3(avCodec);
if (avCodecContext == null) {
throw new RuntimeException("Can't allocate decoder context");
}
int result = avcodec_open2(avCodecContext, avCodec, (AVDictionary) null);
if (result < 0) {
throw new RuntimeException("Can't open decoder");
}
avFrame = av_frame_alloc();
if (avFrame == null) {
throw new RuntimeException("Can't allocate frame");
}
}
And this is a method which is called every time when I receive a packet from Android. byte[] data is the packet data starting with 0x00, 0x00, 0x00, 0x01.
The place where I get error is number_of_written_bytes. It always gets <0.
private void decode(byte[] data) {
AVPacket avPacket = new AVPacket();
av_init_packet(avPacket);
avPacket.pts(AV_NOPTS_VALUE);
avPacket.dts(AV_NOPTS_VALUE);
BytePointer bytePointer = new BytePointer(data);
bytePointer.capacity(data.length);
avPacket.data(bytePointer);
avPacket.size(data.length);
avPacket.pos(-1);
avcodec_send_packet(avCodecContext, avPacket);
int result = avcodec_receive_frame(avCodecContext, avFrame);
if (result >= 0) {
int bufferOutputSize = av_image_get_buffer_size(avFrame.format(), avFrame.width(), avFrame.height(), 16);
Pointer pointer = av_malloc(bufferOutputSize);
BytePointer outputPointer = new BytePointer(pointer);
int number_of_written_bytes = av_image_copy_to_buffer(outputPointer, bufferOutputSize, avFrame.data(), avFrame.linesize(), avFrame.chroma_location(), avFrame.width(), avFrame.height(), 1);
if (number_of_written_bytes < 0) {
//The process always come here.
throw new RuntimeException("Can't copy image to buffer");
}
System.out.println("decode success");
Image image = new Image(new ByteArrayInputStream(outputPointer.asBuffer().array()));
} else {
System.out.println("decode failed");
}
}
Anything is helpful for me. Thanks.

Video Streaming on network using java sockets

I am developing a networking java application. I wanted to stream a video from network (maybe using sockets). I search on the internet but I didnt find any working server and client code to stream video from a server to a client.
Can anyone find streaming server and client or code a simple program so that I can understand how streaming is done using java.
PS. I fount an assignment related to this on internet. But it has error and some methods are also unimplemented. If you can remove errors and complete the methods it will also be helpful..
http://cs.anu.edu.au/student/comp3310/2004/Labs/lab6/lab5.html

See: Any simple (and up to date) Java frameworks for embedding movies within a Swing Application?, just refer to the JavaFX only code sample (you don't need any Swing code).
import javafx.application.Application;
import javafx.scene.*;
import javafx.scene.media.*;
import javafx.stage.Stage;
public class VideoPlayerExample extends Application {
public static void main(String[] args) throws Exception { launch(args); }
#Override public void start(final Stage stage) throws Exception {
final MediaPlayer oracleVid = new MediaPlayer(
new Media("http://download.oracle.com/otndocs/products/javafx/oow2010-2.flv")
);
stage.setScene(new Scene(new Group(new MediaView(oracleVid)), 540, 208));
stage.show();
oracleVid.play();
}
}
So, encode your video to a format understood by JavaFX (e.g. h264 encoded mp4) and place it on a http server and you can load the video data over http from your JavaFX client. Ensure that your client is a certified system configuration for media playback using JavaFX.
That is probably sufficient for what you need.
If you need something a bit more fancy, JavaFX also supports http live streaming, which you can read up on and see if you need (which you probably don't). I don't have instructions on setting up a http live streaming server, nor a link to somewhere on the internet on how to do that (you would have to do your own research on that if you want to go that route).
Also, note, I converted the mjpeg player lab assignment you reference in your question to JavaFX to answer the question: Display RTP MJPEG. It is useful if you want to understand at a low level how such video playback is done. However, I would not recommend using this method for your video playback for a production project - instead just use the built-in JavaFX MediaPlayer.

Here are the basic code: http://xuggle.googlecode.com/svn/trunk/java/xuggle-xuggler/src/com/xuggle/xuggler/demos/DecodeAndPlayAudioAndVideo.java
but I changed it to:
package Pasban;
/**
*
* #modified by Pasban
*/
import javax.sound.sampled.AudioFormat;
import javax.sound.sampled.AudioSystem;
import javax.sound.sampled.DataLine;
import javax.sound.sampled.LineUnavailableException;
import javax.sound.sampled.SourceDataLine;
import com.xuggle.xuggler.Global;
import com.xuggle.xuggler.IAudioSamples;
import com.xuggle.xuggler.IContainer;
import com.xuggle.xuggler.IPacket;
import com.xuggle.xuggler.IPixelFormat;
import com.xuggle.xuggler.IStream;
import com.xuggle.xuggler.IStreamCoder;
import com.xuggle.xuggler.ICodec;
import com.xuggle.xuggler.IVideoPicture;
import com.xuggle.xuggler.IVideoResampler;
import com.xuggle.xuggler.Utils;
import com.xuggle.xuggler.demos.VideoImage;
import java.awt.Dimension;
/**
* Takes a media container, finds the first video stream,
* decodes that stream, and then plays the audio and video.
*
* This code does a VERY coarse job of matching time-stamps, and thus
* the audio and video will float in and out of slight sync. Getting
* time-stamps syncing-up with audio is very system dependent and left
* as an exercise for the reader.
*
* #author aclarke
*
*/
public class DecodeAndPlayAudioAndVideo {
/**
* The audio line we'll output sound to; it'll be the default audio device on your system if available
*/
private static SourceDataLine mLine;
/**
* The window we'll draw the video on.
*
*/
private static VideoImage mScreen = null;
private static long mSystemVideoClockStartTime;
private static long mFirstVideoTimestampInStream;
/**
* Takes a media container (file) as the first argument, opens it,
* plays audio as quickly as it can, and opens up a Swing window and displays
* video frames with <i>roughly</i> the right timing.
*
* #param args Must contain one string which represents a filename
*/
#SuppressWarnings("deprecation")
public static void main(String[] args) {
String filename = "http://techslides.com/demos/sample-videos/small.mp4";
// Let's make sure that we can actually convert video pixel formats.
if (!IVideoResampler.isSupported(IVideoResampler.Feature.FEATURE_COLORSPACECONVERSION)) {
throw new RuntimeException("you must install the GPL version of Xuggler (with IVideoResampler support) for this demo to work");
}
// Create a Xuggler container object
IContainer container = IContainer.make();
// Open up the container
if (container.open("http://techslides.com/demos/sample-videos/small.mp4", IContainer.Type.READ, null) < 0) {
throw new IllegalArgumentException("could not open file: " + filename);
}
// query how many streams the call to open found
int numStreams = container.getNumStreams();
// and iterate through the streams to find the first audio stream
int videoStreamId = -1;
IStreamCoder videoCoder = null;
int audioStreamId = -1;
IStreamCoder audioCoder = null;
for (int i = 0; i < numStreams; i++) {
// Find the stream object
IStream stream = container.getStream(i);
// Get the pre-configured decoder that can decode this stream;
IStreamCoder coder = stream.getStreamCoder();
if (videoStreamId == -1 && coder.getCodecType() == ICodec.Type.CODEC_TYPE_VIDEO) {
videoStreamId = i;
videoCoder = coder;
} else if (audioStreamId == -1 && coder.getCodecType() == ICodec.Type.CODEC_TYPE_AUDIO) {
audioStreamId = i;
audioCoder = coder;
}
}
if (videoStreamId == -1 && audioStreamId == -1) {
throw new RuntimeException("could not find audio or video stream in container: " + filename);
}
/*
* Check if we have a video stream in this file. If so let's open up our decoder so it can
* do work.
*/
IVideoResampler resampler = null;
if (videoCoder != null) {
if (videoCoder.open() < 0) {
throw new RuntimeException("could not open audio decoder for container: " + filename);
}
if (videoCoder.getPixelType() != IPixelFormat.Type.BGR24) {
// if this stream is not in BGR24, we're going to need to
// convert it. The VideoResampler does that for us.
resampler = IVideoResampler.make(videoCoder.getWidth(), videoCoder.getHeight(), IPixelFormat.Type.BGR24,
videoCoder.getWidth(), videoCoder.getHeight(), videoCoder.getPixelType());
openJavaVideo(videoCoder);
if (resampler == null) {
throw new RuntimeException("could not create color space resampler for: " + filename);
}
}
/*
* And once we have that, we draw a window on screen
*/
}
if (audioCoder != null) {
if (audioCoder.open() < 0) {
throw new RuntimeException("could not open audio decoder for container: " + filename);
}
/*
* And once we have that, we ask the Java Sound System to get itself ready.
*/
try {
openJavaSound(audioCoder);
} catch (LineUnavailableException ex) {
throw new RuntimeException("unable to open sound device on your system when playing back container: " + filename);
}
}
/*
* Now, we start walking through the container looking at each packet.
*/
IPacket packet = IPacket.make();
mFirstVideoTimestampInStream = Global.NO_PTS;
mSystemVideoClockStartTime = 0;
while (container.readNextPacket(packet) >= 0) {
/*
* Now we have a packet, let's see if it belongs to our video stream
*/
if (packet.getStreamIndex() == videoStreamId) {
/*
* We allocate a new picture to get the data out of Xuggler
*/
IVideoPicture picture = IVideoPicture.make(videoCoder.getPixelType(),
videoCoder.getWidth(), videoCoder.getHeight());
/*
* Now, we decode the video, checking for any errors.
*
*/
int bytesDecoded = videoCoder.decodeVideo(picture, packet, 0);
if (bytesDecoded < 0) {
throw new RuntimeException("got error decoding audio in: " + filename);
}
/*
* Some decoders will consume data in a packet, but will not be able to construct
* a full video picture yet. Therefore you should always check if you
* got a complete picture from the decoder
*/
if (picture.isComplete()) {
IVideoPicture newPic = picture;
/*
* If the resampler is not null, that means we didn't get the video in BGR24 format and
* need to convert it into BGR24 format.
*/
if (resampler != null) {
// we must resample
newPic = IVideoPicture.make(resampler.getOutputPixelFormat(), picture.getWidth(), picture.getHeight());
if (resampler.resample(newPic, picture) < 0) {
throw new RuntimeException("could not resample video from: " + filename);
}
}
if (newPic.getPixelType() != IPixelFormat.Type.BGR24) {
throw new RuntimeException("could not decode video as BGR 24 bit data in: " + filename);
}
long delay = millisecondsUntilTimeToDisplay(newPic);
// if there is no audio stream; go ahead and hold up the main thread. We'll end
// up caching fewer video pictures in memory that way.
try {
if (delay > 0) {
Thread.sleep(delay);
}
} catch (InterruptedException e) {
return;
}
// And finally, convert the picture to an image and display it
mScreen.setImage(Utils.videoPictureToImage(newPic));
}
} else if (packet.getStreamIndex() == audioStreamId) {
/*
* We allocate a set of samples with the same number of channels as the
* coder tells us is in this buffer.
*
* We also pass in a buffer size (1024 in our example), although Xuggler
* will probably allocate more space than just the 1024 (it's not important why).
*/
IAudioSamples samples = IAudioSamples.make(1024, audioCoder.getChannels());
/*
* A packet can actually contain multiple sets of samples (or frames of samples
* in audio-decoding speak). So, we may need to call decode audio multiple
* times at different offsets in the packet's data. We capture that here.
*/
int offset = 0;
/*
* Keep going until we've processed all data
*/
while (offset < packet.getSize()) {
int bytesDecoded = audioCoder.decodeAudio(samples, packet, offset);
if (bytesDecoded < 0) {
throw new RuntimeException("got error decoding audio in: " + filename);
}
offset += bytesDecoded;
/*
* Some decoder will consume data in a packet, but will not be able to construct
* a full set of samples yet. Therefore you should always check if you
* got a complete set of samples from the decoder
*/
if (samples.isComplete()) {
// note: this call will block if Java's sound buffers fill up, and we're
// okay with that. That's why we have the video "sleeping" occur
// on another thread.
playJavaSound(samples);
}
}
} else {
/*
* This packet isn't part of our video stream, so we just silently drop it.
*/
do {
} while (false);
}
}
/*
* Technically since we're exiting anyway, these will be cleaned up by
* the garbage collector... but because we're nice people and want
* to be invited places for Christmas, we're going to show how to clean up.
*/
if (videoCoder != null) {
videoCoder.close();
videoCoder = null;
}
if (audioCoder != null) {
audioCoder.close();
audioCoder = null;
}
if (container != null) {
container.close();
container = null;
}
closeJavaSound();
closeJavaVideo();
}
private static long millisecondsUntilTimeToDisplay(IVideoPicture picture) {
/**
* We could just display the images as quickly as we decode them, but it turns
* out we can decode a lot faster than you think.
*
* So instead, the following code does a poor-man's version of trying to
* match up the frame-rate requested for each IVideoPicture with the system
* clock time on your computer.
*
* Remember that all Xuggler IAudioSamples and IVideoPicture objects always
* give timestamps in Microseconds, relative to the first decoded item. If
* instead you used the packet timestamps, they can be in different units depending
* on your IContainer, and IStream and things can get hairy quickly.
*/
long millisecondsToSleep = 0;
if (mFirstVideoTimestampInStream == Global.NO_PTS) {
// This is our first time through
mFirstVideoTimestampInStream = picture.getTimeStamp();
// get the starting clock time so we can hold up frames
// until the right time.
mSystemVideoClockStartTime = System.currentTimeMillis();
millisecondsToSleep = 0;
} else {
long systemClockCurrentTime = System.currentTimeMillis();
long millisecondsClockTimeSinceStartofVideo = systemClockCurrentTime - mSystemVideoClockStartTime;
// compute how long for this frame since the first frame in the stream.
// remember that IVideoPicture and IAudioSamples timestamps are always in MICROSECONDS,
// so we divide by 1000 to get milliseconds.
long millisecondsStreamTimeSinceStartOfVideo = (picture.getTimeStamp() - mFirstVideoTimestampInStream) / 1000;
final long millisecondsTolerance = 50; // and we give ourselfs 50 ms of tolerance
millisecondsToSleep = (millisecondsStreamTimeSinceStartOfVideo
- (millisecondsClockTimeSinceStartofVideo + millisecondsTolerance));
}
return millisecondsToSleep;
}
/**
* Opens a Swing window on screen.
*/
/**
* Forces the swing thread to terminate; I'm sure there is a right
* way to do this in swing, but this works too.
*/
private static void closeJavaVideo() {
System.exit(0);
}
private static void openJavaSound(IStreamCoder aAudioCoder) throws LineUnavailableException {
AudioFormat audioFormat = new AudioFormat(aAudioCoder.getSampleRate(),
(int) IAudioSamples.findSampleBitDepth(aAudioCoder.getSampleFormat()),
aAudioCoder.getChannels(),
true, /* xuggler defaults to signed 16 bit samples */
false);
DataLine.Info info = new DataLine.Info(SourceDataLine.class, audioFormat);
mLine = (SourceDataLine) AudioSystem.getLine(info);
/**
* if that succeeded, try opening the line.
*/
mLine.open(audioFormat);
/**
* And if that succeed, start the line.
*/
mLine.start();
}
private static void playJavaSound(IAudioSamples aSamples) {
/**
* We're just going to dump all the samples into the line.
*/
byte[] rawBytes = aSamples.getData().getByteArray(0, aSamples.getSize());
mLine.write(rawBytes, 0, aSamples.getSize());
}
private static void closeJavaSound() {
if (mLine != null) {
/*
* Wait for the line to finish playing
*/
mLine.drain();
/*
* Close the line.
*/
mLine.close();
mLine = null;
}
}
private static void openJavaVideo(IStreamCoder videoCoder) {
mScreen = new VideoImage();
mScreen.setPreferredSize(new Dimension(videoCoder.getWidth(), videoCoder.getHeight()));
mScreen.setLocationRelativeTo(null);
}
}
Things I changed:
private static void openJavaVideo(IStreamCoder videoCoder) {
mScreen = new VideoImage();
mScreen.setPreferredSize(new Dimension(videoCoder.getWidth(), videoCoder.getHeight()));
mScreen.setLocationRelativeTo(null);
}
Moved openJavaVideo method into videoStream detector:
openJavaVideo(videoCoder);
Changed the first part of the main:
public static void main(String[] args) {
String filename = "http://techslides.com/demos/sample-videos/small.mp4";
// Let's make sure that we can actually convert video pixel formats.
if (!IVideoResampler.isSupported(IVideoResampler.Feature.FEATURE_COLORSPACECONVERSION)) {
throw new RuntimeException("you must install the GPL version of Xuggler (with IVideoResampler support) for this demo to work");
}
// Create a Xuggler container object
IContainer container = IContainer.make();
// Open up the container
if (container.open("http://techslides.com/demos/sample-videos/small.mp4", IContainer.Type.READ, null) < 0) {
throw new IllegalArgumentException("could not open file: " + filename);
}
Actually, the important part was:
if (container.open("http://techslides.com/demos/sample-videos/small.mp4", IContainer.Type.READ, null) < 0) {
throw new IllegalArgumentException("could not open file: " + filename);
}

Xuggle is/was one of the best:
Streaming video with Xuggler
Right now I don;t have a complete project, but I believe it had an example with its demo files.
Search google for xuggle video streaming demo or similar keywords with Xuggler. It is easy to use and support most of the known formats as it wraps FFMPEG with itself.

I came to another idea it may worth trying.
Create a JavaFX application.
Add a web browser into it, check webengine
create a template webpage which it contains a html player, or load the page in your server where it accept file id, then create a page that create a player for that file, similar to youtube, then auto play it.
This will be much better idea it you could do it.
There are sample codes for webengine and javaFX. Once you loaded a page, say, youtube or vimeo and played a video there, then sky is the limit :)

HttpServer - HttpExchange - Seekable Stream

I work on a sample java http server and a .Net client (on tablet).
using my http sever, the .Net client must be able to download files.
It's working perfectly, but now I have to be able to resume download after a connection disruption.
Here some code :
Java server : ( It is launched in a seperate thread, hence the run method).
public void run() {
try {
server = com.sun.net.httpserver.HttpServer.create(
new InetSocketAddress(
portNumber), this.maximumConnexion);
server.setExecutor(executor);
server.createContext("/", new ConnectionHandler(this.rootPath));
server.start();
} catch (IOException e1) {
//For debugging
e1.printStackTrace();
}
}
my HttpHandler : (only the part dealing with GET request)
/**
* handleGetMethod : handle GET request. If the file specified in the URI is
* available, send it to the client.
*
* #param httpExchange
* #throws IOException
*/
private void handleGetMethod(HttpExchange httpExchange) throws IOException {
File file = new File(this.rootPath + this.fileRef).getCanonicalFile();
if (!file.isFile()) {
this.handleError(httpExchange, 404);
} else if (!file.getPath().startsWith(this.rootPath.replace('/', '\\'))) { // windows work with anti-slash!
// Suspected path traversal attack.
System.out.println(file.getPath());
this.handleError(httpExchange, 403);
} else {
//Send the document.
httpExchange.sendResponseHeaders(200, file.length());
System.out.println("file length : "+ file.length() + " bytes.");
OutputStream os = httpExchange.getResponseBody();
FileInputStream fs = new FileInputStream(file);
final byte[] buffer = new byte[1024];
int count = 0;
while ((count = fs.read(buffer)) >= 0) {
os.write(buffer, 0, count);
}
os.flush();
fs.close();
os.close();
}
}
And now my .Net Client: (simplified)
try{
Stream response = await httpClient.GetStreamAsync(URI + this.fileToDownload.Text);
FileSavePicker savePicker = new FileSavePicker();
savePicker.SuggestedStartLocation = PickerLocationId.DocumentsLibrary;
// Dropdown of file types the user can save the file as
savePicker.FileTypeChoices.Add("Application/pdf", new List<string>() { ".pdf" });
// Default file name if the user does not type one in or select a file to replace
savePicker.SuggestedFileName = "new doc";
StorageFile file = await savePicker.PickSaveFileAsync();
if (file != null)
{
const int BUFFER_SIZE = 1024*1024;
using (Stream outputFileStream = await file.OpenStreamForWriteAsync())
{
using (response)
{
var buffer = new byte[BUFFER_SIZE];
int bytesRead;
do
{
bytesRead = response.Read(buffer, 0, BUFFER_SIZE);
outputFileStream.Write(buffer, 0, bytesRead);
} while (bytesRead > 0);
}
outputFileStream.Flush();
}
}
}
catch (HttpRequestException hre)
{ //For debugging
this.Display.Text += hre.Message;
this.Display.Text += hre.Source;
}
catch (Exception ex)
{
//For debugging
this.Display.Text += ex.Message;
this.Display.Text += ex.Source;
}
So, to resume the download I would like to use some seek operation within the .Net client part.
But every time I try something like response.Seek(offset, response.Position); , an error occurs informing that the Stream does not support seek operations.
Yes, It does not, but how I can specify (in my server side) to use seekable Stream?
Does the method HttpExchange.setStreams can be useful?
Or, I do not need to modify the stream but to configure my HttpServer instance?
Thanks.

Well use Range, Accept-Range and Content-Range fields works. There is just a little bit of work to do in order to send the correct part of the file and to set the response's headers.
The server may inform client that it support the Range field by setting the Accept-Range field:
responseHeader.set("Accept-Ranges", "bytes");
And then set the Content-range field when partial file are sent :
responseHeader.set("Content-range", "bytes " + this.offSet + "-" + this.range + "/" + this.fileLength);
Finally the return code must be set to 206 (Partial Content).
For more information about Range, Accept-Range and Content-Range fields see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
NB : Opera 12.16 use the field "Range" to resume download but it seems that IE 10 and Firefox 22 do not use this field. May be some seekable streams as I was looking for originally. If anyone have an answer to this, I will be glad to read it =).

Generate waveform image from audio stream of a video

I need to generate a waveform of an audio stream of a video,
currently I'm using xuggler and java to do some little things, but seems like I'm not able to get a byte array of the inputstream audio of my video from IAudioSamples.
Now I'm searching for an easier way to do it since xuggler is really becoming hard to understand, I've searched online and I've found this:
http://codeidol.com/java/swing/Audio/Build-an-Audio-Waveform-Display/
should work on .wav files, but when I try the code on a video or a .mp3 the AudioInputStream returns "cannot find an audio input stream"
can someone tell me a way to get byte[] array of the audiostream of one video so that I can follow the tutorial to create a waveform?
also if you have suggestion or other library that could help me I would be glad

Because mp3 it's an encoded format, you'll need before to decode it to get ray data (bytes) from it.
class Mp3FileXuggler {
private boolean DEBUG = true;
private String _sInputFileName;
private IContainer _inputContainer;
private int _iBitRate;
private IPacket _packet;
private int _iAudioStreamId;
private IStreamCoder _audioCoder;
private int _iSampleBufferSize;
private int _iInputSampleRate;
private static SourceDataLine mLine;
private int DECODED_AUDIO_SECOND_SIZE = 176375; /** bytes */
private int _bytesPerPacket;
private byte[] _residualBuffer;
/**
* Constructor, prepares stream to be readed
* #param input input File
* #throws UnsuportedSampleRateException
*/
public Mp3FileXuggler(String sFileName) throws UnsuportedSampleRateException{
this._sInputFileName = sFileName;
this._inputContainer = IContainer.make();
this._iSampleBufferSize = 18432;
this._residualBuffer = null;
/** Open container **/
if (this._inputContainer.open(this._sInputFileName, IContainer.Type.READ, null) < 0)
throw new IllegalArgumentException("Could not read the file: " + this._sInputFileName);
/** How many streams does the file actually have */
int iNumStreams = this._inputContainer.getNumStreams();
this._iBitRate = this._inputContainer.getBitRate();
if (DEBUG) System.out.println("Bitrate: " + this._iBitRate);
/** Iterate the streams to find the first audio stream */
this._iAudioStreamId = -1;
this._audioCoder = null;
boolean bFound = false;
int i = 0;
while (i < iNumStreams && bFound == false){
/** Find the stream object */
IStream stream = this._inputContainer.getStream(i);
IStreamCoder coder = stream.getStreamCoder();
/** If the stream is audio, stop looking */
if (coder.getCodecType() == ICodec.Type.CODEC_TYPE_AUDIO){
this._iAudioStreamId = i;
this._audioCoder = coder;
this._iInputSampleRate = coder.getSampleRate();
bFound = true;
}
++i;
}
/** If none was found */
if (this._iAudioStreamId == -1)
throw new RuntimeException("Could not find audio stream in container: " + this._sInputFileName);
/** Otherwise, open audiocoder */
if (this._audioCoder.open(null,null) < 0)
throw new RuntimeException("could not open audio decoder for container: " + this._sInputFileName);
this._packet = IPacket.make();
//openJavaSound(this._audioCoder);
/** Dummy read one packet to avoid problems in some audio files */
this._inputContainer.readNextPacket(this._packet);
/** Supported sample rates */
switch(this._iInputSampleRate){
case 22050:
this._bytesPerPacket = 2304;
break;
case 44100:
this._bytesPerPacket = 4608;
break;
}
}
public byte[] getSamples(){
byte[] rawBytes = null;
/** Go to the correct packet */
while (this._inputContainer.readNextPacket(this._packet) >= 0){
//System.out.println(this._packet.getDuration());
/** Once we have a packet, let's see if it belongs to the audio stream */
if (this._packet.getStreamIndex() == this._iAudioStreamId){
IAudioSamples samples = IAudioSamples.make(this._iSampleBufferSize, this._audioCoder.getChannels());
// System.out.println(">> " + samples.toString());
/** Because a packet can contain multiple set of samples (frames of samples). We may need to call
* decode audio multiple times at different offsets in the packet's data */
int iCurrentOffset = 0;
while(iCurrentOffset < this._packet.getSize()){
int iBytesDecoded = this._audioCoder.decodeAudio(samples, this._packet, iCurrentOffset);
iCurrentOffset += iBytesDecoded;
if (samples.isComplete()){
rawBytes = samples.getData().getByteArray(0, samples.getSize());
//playJavaSound(samples);
}
}
return rawBytes;
}
else{
/** Otherwise drop it */
do{}while(false);
}
}
return rawBytes; /** This will return null at this point */
}
}
Use this class in order to get the raw data from a mp3 file, and with them feed your spectrum drawer.

Unable to understand this method (How does it try to match the frame rate ?)

I came across this snippet while going through the tutorial on how to decode a video :
private static long millisecondsUntilTimeToDisplay(IVideoPicture picture)
{
/**
* We could just display the images as quickly as we decode them, but it turns
* out we can decode a lot faster than you think.
*
* So instead, the following code does a poor-man's version of trying to
* match up the frame-rate requested for each IVideoPicture with the system
* clock time on your computer.
*
* Remember that all Xuggler IAudioSamples and IVideoPicture objects always
* give timestamps in Microseconds, relative to the first decoded item. If
* instead you used the packet timestamps, they can be in different units depending
* on your IContainer, and IStream and things can get hairy quickly.
*/
long millisecondsToSleep = 0;
if (mFirstVideoTimestampInStream == Global.NO_PTS)
{
// This is our first time through
mFirstVideoTimestampInStream = picture.getTimeStamp();
// get the starting clock time so we can hold up frames
// until the right time.
mSystemVideoClockStartTime = System.currentTimeMillis();
millisecondsToSleep = 0;
} else {
long systemClockCurrentTime = System.currentTimeMillis();
long millisecondsClockTimeSinceStartofVideo = systemClockCurrentTime - mSystemVideoClockStartTime;
// compute how long for this frame since the first frame in the stream.
// remember that IVideoPicture and IAudioSamples timestamps are always in MICROSECONDS,
// so we divide by 1000 to get milliseconds.
long millisecondsStreamTimeSinceStartOfVideo = (picture.getTimeStamp() - mFirstVideoTimestampInStream)/1000;
final long millisecondsTolerance = 50; // and we give ourselfs 50 ms of tolerance
millisecondsToSleep = (millisecondsStreamTimeSinceStartOfVideo -
(millisecondsClockTimeSinceStartofVideo+millisecondsTolerance));
}
return millisecondsToSleep;
}
I have scratched a lot but don't understand what does this method do ? what are we returning ? And why we are making the thread to sleep after the method returns ( what is the purpose of the method ?)
This is the complete code in the link :
import javax.sound.sampled.AudioFormat;
import javax.sound.sampled.AudioSystem;
import javax.sound.sampled.DataLine;
import javax.sound.sampled.LineUnavailableException;
import javax.sound.sampled.SourceDataLine;
import com.xuggle.xuggler.demos.*;
import com.xuggle.xuggler.Global;
import com.xuggle.xuggler.IAudioSamples;
import com.xuggle.xuggler.IContainer;
import com.xuggle.xuggler.IPacket;
import com.xuggle.xuggler.IPixelFormat;
import com.xuggle.xuggler.IStream;
import com.xuggle.xuggler.IStreamCoder;
import com.xuggle.xuggler.ICodec;
import com.xuggle.xuggler.IVideoPicture;
import com.xuggle.xuggler.IVideoResampler;
import com.xuggle.xuggler.Utils;
public class DecodeAndPlayAudioAndVideo
{
/**
* The audio line we'll output sound to; it'll be the default audio device on your system if available
*/
private static SourceDataLine mLine;
/**
* The window we'll draw the video on.
*
*/
private static VideoImage mScreen = null;
private static long mSystemVideoClockStartTime;
private static long mFirstVideoTimestampInStream;
/**
* Takes a media container (file) as the first argument, opens it,
* plays audio as quickly as it can, and opens up a Swing window and displays
* video frames with <i>roughly</i> the right timing.
*
* #param args Must contain one string which represents a filename
*/
#SuppressWarnings("deprecation")
public static void main(String[] args)
{
if (args.length <= 0)
throw new IllegalArgumentException("must pass in a filename as the first argument");
String filename = args[0];
// Let's make sure that we can actually convert video pixel formats.
if (!IVideoResampler.isSupported(IVideoResampler.Feature.FEATURE_COLORSPACECONVERSION))
throw new RuntimeException("you must install the GPL version of Xuggler (with IVideoResampler support) for this demo to work");
// Create a Xuggler container object
IContainer container = IContainer.make();
// Open up the container
if (container.open(filename, IContainer.Type.READ, null) < 0)
throw new IllegalArgumentException("could not open file: " + filename);
// query how many streams the call to open found
int numStreams = container.getNumStreams();
// and iterate through the streams to find the first audio stream
int videoStreamId = -1;
IStreamCoder videoCoder = null;
int audioStreamId = -1;
IStreamCoder audioCoder = null;
for(int i = 0; i < numStreams; i++)
{
// Find the stream object
IStream stream = container.getStream(i);
// Get the pre-configured decoder that can decode this stream;
IStreamCoder coder = stream.getStreamCoder();
if (videoStreamId == -1 && coder.getCodecType() == ICodec.Type.CODEC_TYPE_VIDEO)
{
videoStreamId = i;
videoCoder = coder;
}
else if (audioStreamId == -1 && coder.getCodecType() == ICodec.Type.CODEC_TYPE_AUDIO)
{
audioStreamId = i;
audioCoder = coder;
}
}
if (videoStreamId == -1 && audioStreamId == -1)
throw new RuntimeException("could not find audio or video stream in container: "+filename);
/*
* Check if we have a video stream in this file. If so let's open up our decoder so it can
* do work.
*/
IVideoResampler resampler = null;
if (videoCoder != null)
{
if(videoCoder.open() < 0)
throw new RuntimeException("could not open audio decoder for container: "+filename);
if (videoCoder.getPixelType() != IPixelFormat.Type.BGR24)
{
// if this stream is not in BGR24, we're going to need to
// convert it. The VideoResampler does that for us.
resampler = IVideoResampler.make(videoCoder.getWidth(), videoCoder.getHeight(), IPixelFormat.Type.BGR24,
videoCoder.getWidth(), videoCoder.getHeight(), videoCoder.getPixelType());
if (resampler == null)
throw new RuntimeException("could not create color space resampler for: " + filename);
}
/*
* And once we have that, we draw a window on screen
*/
openJavaVideo();
}
if (audioCoder != null)
{
if (audioCoder.open() < 0)
throw new RuntimeException("could not open audio decoder for container: "+filename);
/*
* And once we have that, we ask the Java Sound System to get itself ready.
*/
try
{
openJavaSound(audioCoder);
}
catch (LineUnavailableException ex)
{
throw new RuntimeException("unable to open sound device on your system when playing back container: "+filename);
}
}
/*
* Now, we start walking through the container looking at each packet.
*/
IPacket packet = IPacket.make();
mFirstVideoTimestampInStream = Global.NO_PTS;
mSystemVideoClockStartTime = 0;
while(container.readNextPacket(packet) >= 0)
{
/*
* Now we have a packet, let's see if it belongs to our video stream
*/
if (packet.getStreamIndex() == videoStreamId)
{
/*
* We allocate a new picture to get the data out of Xuggler
*/
IVideoPicture picture = IVideoPicture.make(videoCoder.getPixelType(),
videoCoder.getWidth(), videoCoder.getHeight());
/*
* Now, we decode the video, checking for any errors.
*
*/
int bytesDecoded = videoCoder.decodeVideo(picture, packet, 0);
if (bytesDecoded < 0)
throw new RuntimeException("got error decoding audio in: " + filename);
/*
* Some decoders will consume data in a packet, but will not be able to construct
* a full video picture yet. Therefore you should always check if you
* got a complete picture from the decoder
*/
if (picture.isComplete())
{
IVideoPicture newPic = picture;
/*
* If the resampler is not null, that means we didn't get the video in BGR24 format and
* need to convert it into BGR24 format.
*/
if (resampler != null)
{
// we must resample
newPic = IVideoPicture.make(resampler.getOutputPixelFormat(), picture.getWidth(), picture.getHeight());
if (resampler.resample(newPic, picture) < 0)
throw new RuntimeException("could not resample video from: " + filename);
}
if (newPic.getPixelType() != IPixelFormat.Type.BGR24)
throw new RuntimeException("could not decode video as BGR 24 bit data in: " + filename);
long delay = millisecondsUntilTimeToDisplay(newPic);
// if there is no audio stream; go ahead and hold up the main thread. We'll end
// up caching fewer video pictures in memory that way.
try
{
if (delay > 0)
Thread.sleep(delay);
}
catch (InterruptedException e)
{
return;
}
// And finally, convert the picture to an image and display it
mScreen.setImage(Utils.videoPictureToImage(newPic));
}
}
else if (packet.getStreamIndex() == audioStreamId)
{
/*
* We allocate a set of samples with the same number of channels as the
* coder tells us is in this buffer.
*
* We also pass in a buffer size (1024 in our example), although Xuggler
* will probably allocate more space than just the 1024 (it's not important why).
*/
IAudioSamples samples = IAudioSamples.make(1024, audioCoder.getChannels());
/*
* A packet can actually contain multiple sets of samples (or frames of samples
* in audio-decoding speak). So, we may need to call decode audio multiple
* times at different offsets in the packet's data. We capture that here.
*/
int offset = 0;
/*
* Keep going until we've processed all data
*/
while(offset < packet.getSize())
{
int bytesDecoded = audioCoder.decodeAudio(samples, packet, offset);
if (bytesDecoded < 0)
throw new RuntimeException("got error decoding audio in: " + filename);
offset += bytesDecoded;
/*
* Some decoder will consume data in a packet, but will not be able to construct
* a full set of samples yet. Therefore you should always check if you
* got a complete set of samples from the decoder
*/
if (samples.isComplete())
{
// note: this call will block if Java's sound buffers fill up, and we're
// okay with that. That's why we have the video "sleeping" occur
// on another thread.
playJavaSound(samples);
}
}
}
else
{
/*
* This packet isn't part of our video stream, so we just silently drop it.
*/
do {} while(false);
}
}
/*
* Technically since we're exiting anyway, these will be cleaned up by
* the garbage collector... but because we're nice people and want
* to be invited places for Christmas, we're going to show how to clean up.
*/
if (videoCoder != null)
{
videoCoder.close();
videoCoder = null;
}
if (audioCoder != null)
{
audioCoder.close();
audioCoder = null;
}
if (container !=null)
{
container.close();
container = null;
}
closeJavaSound();
closeJavaVideo();
}
What does The following method do ?
private static long millisecondsUntilTimeToDisplay(IVideoPicture picture)
{
/**
* We could just display the images as quickly as we decode them, but it turns
* out we can decode a lot faster than you think.
*
* So instead, the following code does a poor-man's version of trying to
* match up the frame-rate requested for each IVideoPicture with the system
* clock time on your computer.
*
* Remember that all Xuggler IAudioSamples and IVideoPicture objects always
* give timestamps in Microseconds, relative to the first decoded item. If
* instead you used the packet timestamps, they can be in different units depending
* on your IContainer, and IStream and things can get hairy quickly.
*/
long millisecondsToSleep = 0;
if (mFirstVideoTimestampInStream == Global.NO_PTS)
{
// This is our first time through
mFirstVideoTimestampInStream = picture.getTimeStamp();
// get the starting clock time so we can hold up frames
// until the right time.
mSystemVideoClockStartTime = System.currentTimeMillis();
millisecondsToSleep = 0;
} else {
long systemClockCurrentTime = System.currentTimeMillis();
long millisecondsClockTimeSinceStartofVideo = systemClockCurrentTime - mSystemVideoClockStartTime;
// compute how long for this frame since the first frame in the stream.
// remember that IVideoPicture and IAudioSamples timestamps are always in MICROSECONDS,
// so we divide by 1000 to get milliseconds.
long millisecondsStreamTimeSinceStartOfVideo = (picture.getTimeStamp() - mFirstVideoTimestampInStream)/1000;
final long millisecondsTolerance = 50; // and we give ourselfs 50 ms of tolerance
millisecondsToSleep = (millisecondsStreamTimeSinceStartOfVideo -
(millisecondsClockTimeSinceStartofVideo+millisecondsTolerance));
}
return millisecondsToSleep;
}
/**
* Opens a Swing window on screen.
*/
private static void openJavaVideo()
{
mScreen = new VideoImage();
}
/**
* Forces the swing thread to terminate; I'm sure there is a right
* way to do this in swing, but this works too.
*/
private static void closeJavaVideo()
{
System.exit(0);
}
private static void openJavaSound(IStreamCoder aAudioCoder) throws LineUnavailableException
{
AudioFormat audioFormat = new AudioFormat(aAudioCoder.getSampleRate(),
(int)IAudioSamples.findSampleBitDepth(aAudioCoder.getSampleFormat()),
aAudioCoder.getChannels(),
true, /* xuggler defaults to signed 16 bit samples */
false);
DataLine.Info info = new DataLine.Info(SourceDataLine.class, audioFormat);
mLine = (SourceDataLine) AudioSystem.getLine(info);
/**
* if that succeeded, try opening the line.
*/
mLine.open(audioFormat);
/**
* And if that succeed, start the line.
*/
mLine.start();
}
private static void playJavaSound(IAudioSamples aSamples)
{
/**
* We're just going to dump all the samples into the line.
*/
byte[] rawBytes = aSamples.getData().getByteArray(0, aSamples.getSize());
mLine.write(rawBytes, 0, aSamples.getSize());
}
private static void closeJavaSound()
{
if (mLine != null)
{
/*
* Wait for the line to finish playing
*/
mLine.drain();
/*
* Close the line.
*/
mLine.close();
mLine=null;
}
}
}

Rough algorithm in pseudocode:
Is this the first frame?
> Yes, save the frame time and the current time.
> No, do the following:
See how much time has passed since the first frame was displayed in System Time
See the difference in time between the current frame and the first frame
If there is a discrepancy
>Return a number of milliseconds to sleep for, else return 0.
So, what you then get is the overall algorithm of:
Decode frame
Check if we need to delay the frame (the method in question)
Delay
Display frame
In this way, the program will never display frames faster than the variable frame rate declared by the video. The method in question maintains the state of previous frame times and calculates how long to sleep for.
EDIT: The delay is needed because you can decode frames (much!) faster than the video's frame rate. Let's say you have a fairly slow machine running this program, and it takes 10ms to decode a frame. Let's also say that you have video that has a variable frame rate, but is roughly 10 frames per second (or 100ms per frame). Now if you take this step out of our 'overall algorithm':
Decode frame (10ms)
Display frame (1ms)
Decode frame (10ms)
Display frame (1ms)
If this was happening you would find 1 frame displayed every 10ms, meaning that the video will be displayed at 100 frames per second, which is wrong!
EDIT2: I guess what you're asking is why don't we do this?
Decode frame
Frame Delta = Current Frame Time - Previous Frame Time
Delay (for Delta milliseconds)
Display frame
The problem with this is what happens if it takes a long time to decode or display a frame? This would cause the frame rate to be significantly slower than the frame rate in the file.
Instead, this algorithm syncs the first frame to the system time, then does a little bit of extra calculation:
long systemTimeChange = currentSystemTime - firstFrameSystemTime;
long frameTimeChange = currentFrameTime - firstFrameTime;
// Subtract the time elapsed.
long differenceInChanges = frameTimeChange - systemTimeChange;
if(differenceInChanges > 0) {
// It was faster to decode than the frame rate!
Thread.sleep(differenceInChanges);
}

system time actually denotes the time at which the particular frame has been decoded and the frameTimeroughly denotes the frame rate the video has. So the difference goes like this : discrepancy = frameRate - decodeRate + tolerance Tolerance may be useful when decoding video takes larger time than it takes or the media takes longer time to display . Here is what you get from the difference :
Since decoding is too fast compared to the frame rate of video we have to wait some time and won't display that frame right now. And we use systemTimeStamp to sync our frames and hold it till the right time. In the above picture you see how fast the decoding rate is but frame rate is slow compared to the decoding rate.

seems like this:
* So instead, the following code does a poor-man's version of trying to
* match up the frame-rate requested for each IVideoPicture with the system
* clock time on your computer.
the delay is to try to match the framerate.