Related
I am developing a networking java application. I wanted to stream a video from network (maybe using sockets). I search on the internet but I didnt find any working server and client code to stream video from a server to a client.
Can anyone find streaming server and client or code a simple program so that I can understand how streaming is done using java.
PS. I fount an assignment related to this on internet. But it has error and some methods are also unimplemented. If you can remove errors and complete the methods it will also be helpful..
http://cs.anu.edu.au/student/comp3310/2004/Labs/lab6/lab5.html
See: Any simple (and up to date) Java frameworks for embedding movies within a Swing Application?, just refer to the JavaFX only code sample (you don't need any Swing code).
import javafx.application.Application;
import javafx.scene.*;
import javafx.scene.media.*;
import javafx.stage.Stage;
public class VideoPlayerExample extends Application {
public static void main(String[] args) throws Exception { launch(args); }
#Override public void start(final Stage stage) throws Exception {
final MediaPlayer oracleVid = new MediaPlayer(
new Media("http://download.oracle.com/otndocs/products/javafx/oow2010-2.flv")
);
stage.setScene(new Scene(new Group(new MediaView(oracleVid)), 540, 208));
stage.show();
oracleVid.play();
}
}
So, encode your video to a format understood by JavaFX (e.g. h264 encoded mp4) and place it on a http server and you can load the video data over http from your JavaFX client. Ensure that your client is a certified system configuration for media playback using JavaFX.
That is probably sufficient for what you need.
If you need something a bit more fancy, JavaFX also supports http live streaming, which you can read up on and see if you need (which you probably don't). I don't have instructions on setting up a http live streaming server, nor a link to somewhere on the internet on how to do that (you would have to do your own research on that if you want to go that route).
Also, note, I converted the mjpeg player lab assignment you reference in your question to JavaFX to answer the question: Display RTP MJPEG. It is useful if you want to understand at a low level how such video playback is done. However, I would not recommend using this method for your video playback for a production project - instead just use the built-in JavaFX MediaPlayer.
Here are the basic code: http://xuggle.googlecode.com/svn/trunk/java/xuggle-xuggler/src/com/xuggle/xuggler/demos/DecodeAndPlayAudioAndVideo.java
but I changed it to:
package Pasban;
/**
*
* #modified by Pasban
*/
import javax.sound.sampled.AudioFormat;
import javax.sound.sampled.AudioSystem;
import javax.sound.sampled.DataLine;
import javax.sound.sampled.LineUnavailableException;
import javax.sound.sampled.SourceDataLine;
import com.xuggle.xuggler.Global;
import com.xuggle.xuggler.IAudioSamples;
import com.xuggle.xuggler.IContainer;
import com.xuggle.xuggler.IPacket;
import com.xuggle.xuggler.IPixelFormat;
import com.xuggle.xuggler.IStream;
import com.xuggle.xuggler.IStreamCoder;
import com.xuggle.xuggler.ICodec;
import com.xuggle.xuggler.IVideoPicture;
import com.xuggle.xuggler.IVideoResampler;
import com.xuggle.xuggler.Utils;
import com.xuggle.xuggler.demos.VideoImage;
import java.awt.Dimension;
/**
* Takes a media container, finds the first video stream,
* decodes that stream, and then plays the audio and video.
*
* This code does a VERY coarse job of matching time-stamps, and thus
* the audio and video will float in and out of slight sync. Getting
* time-stamps syncing-up with audio is very system dependent and left
* as an exercise for the reader.
*
* #author aclarke
*
*/
public class DecodeAndPlayAudioAndVideo {
/**
* The audio line we'll output sound to; it'll be the default audio device on your system if available
*/
private static SourceDataLine mLine;
/**
* The window we'll draw the video on.
*
*/
private static VideoImage mScreen = null;
private static long mSystemVideoClockStartTime;
private static long mFirstVideoTimestampInStream;
/**
* Takes a media container (file) as the first argument, opens it,
* plays audio as quickly as it can, and opens up a Swing window and displays
* video frames with <i>roughly</i> the right timing.
*
* #param args Must contain one string which represents a filename
*/
#SuppressWarnings("deprecation")
public static void main(String[] args) {
String filename = "http://techslides.com/demos/sample-videos/small.mp4";
// Let's make sure that we can actually convert video pixel formats.
if (!IVideoResampler.isSupported(IVideoResampler.Feature.FEATURE_COLORSPACECONVERSION)) {
throw new RuntimeException("you must install the GPL version of Xuggler (with IVideoResampler support) for this demo to work");
}
// Create a Xuggler container object
IContainer container = IContainer.make();
// Open up the container
if (container.open("http://techslides.com/demos/sample-videos/small.mp4", IContainer.Type.READ, null) < 0) {
throw new IllegalArgumentException("could not open file: " + filename);
}
// query how many streams the call to open found
int numStreams = container.getNumStreams();
// and iterate through the streams to find the first audio stream
int videoStreamId = -1;
IStreamCoder videoCoder = null;
int audioStreamId = -1;
IStreamCoder audioCoder = null;
for (int i = 0; i < numStreams; i++) {
// Find the stream object
IStream stream = container.getStream(i);
// Get the pre-configured decoder that can decode this stream;
IStreamCoder coder = stream.getStreamCoder();
if (videoStreamId == -1 && coder.getCodecType() == ICodec.Type.CODEC_TYPE_VIDEO) {
videoStreamId = i;
videoCoder = coder;
} else if (audioStreamId == -1 && coder.getCodecType() == ICodec.Type.CODEC_TYPE_AUDIO) {
audioStreamId = i;
audioCoder = coder;
}
}
if (videoStreamId == -1 && audioStreamId == -1) {
throw new RuntimeException("could not find audio or video stream in container: " + filename);
}
/*
* Check if we have a video stream in this file. If so let's open up our decoder so it can
* do work.
*/
IVideoResampler resampler = null;
if (videoCoder != null) {
if (videoCoder.open() < 0) {
throw new RuntimeException("could not open audio decoder for container: " + filename);
}
if (videoCoder.getPixelType() != IPixelFormat.Type.BGR24) {
// if this stream is not in BGR24, we're going to need to
// convert it. The VideoResampler does that for us.
resampler = IVideoResampler.make(videoCoder.getWidth(), videoCoder.getHeight(), IPixelFormat.Type.BGR24,
videoCoder.getWidth(), videoCoder.getHeight(), videoCoder.getPixelType());
openJavaVideo(videoCoder);
if (resampler == null) {
throw new RuntimeException("could not create color space resampler for: " + filename);
}
}
/*
* And once we have that, we draw a window on screen
*/
}
if (audioCoder != null) {
if (audioCoder.open() < 0) {
throw new RuntimeException("could not open audio decoder for container: " + filename);
}
/*
* And once we have that, we ask the Java Sound System to get itself ready.
*/
try {
openJavaSound(audioCoder);
} catch (LineUnavailableException ex) {
throw new RuntimeException("unable to open sound device on your system when playing back container: " + filename);
}
}
/*
* Now, we start walking through the container looking at each packet.
*/
IPacket packet = IPacket.make();
mFirstVideoTimestampInStream = Global.NO_PTS;
mSystemVideoClockStartTime = 0;
while (container.readNextPacket(packet) >= 0) {
/*
* Now we have a packet, let's see if it belongs to our video stream
*/
if (packet.getStreamIndex() == videoStreamId) {
/*
* We allocate a new picture to get the data out of Xuggler
*/
IVideoPicture picture = IVideoPicture.make(videoCoder.getPixelType(),
videoCoder.getWidth(), videoCoder.getHeight());
/*
* Now, we decode the video, checking for any errors.
*
*/
int bytesDecoded = videoCoder.decodeVideo(picture, packet, 0);
if (bytesDecoded < 0) {
throw new RuntimeException("got error decoding audio in: " + filename);
}
/*
* Some decoders will consume data in a packet, but will not be able to construct
* a full video picture yet. Therefore you should always check if you
* got a complete picture from the decoder
*/
if (picture.isComplete()) {
IVideoPicture newPic = picture;
/*
* If the resampler is not null, that means we didn't get the video in BGR24 format and
* need to convert it into BGR24 format.
*/
if (resampler != null) {
// we must resample
newPic = IVideoPicture.make(resampler.getOutputPixelFormat(), picture.getWidth(), picture.getHeight());
if (resampler.resample(newPic, picture) < 0) {
throw new RuntimeException("could not resample video from: " + filename);
}
}
if (newPic.getPixelType() != IPixelFormat.Type.BGR24) {
throw new RuntimeException("could not decode video as BGR 24 bit data in: " + filename);
}
long delay = millisecondsUntilTimeToDisplay(newPic);
// if there is no audio stream; go ahead and hold up the main thread. We'll end
// up caching fewer video pictures in memory that way.
try {
if (delay > 0) {
Thread.sleep(delay);
}
} catch (InterruptedException e) {
return;
}
// And finally, convert the picture to an image and display it
mScreen.setImage(Utils.videoPictureToImage(newPic));
}
} else if (packet.getStreamIndex() == audioStreamId) {
/*
* We allocate a set of samples with the same number of channels as the
* coder tells us is in this buffer.
*
* We also pass in a buffer size (1024 in our example), although Xuggler
* will probably allocate more space than just the 1024 (it's not important why).
*/
IAudioSamples samples = IAudioSamples.make(1024, audioCoder.getChannels());
/*
* A packet can actually contain multiple sets of samples (or frames of samples
* in audio-decoding speak). So, we may need to call decode audio multiple
* times at different offsets in the packet's data. We capture that here.
*/
int offset = 0;
/*
* Keep going until we've processed all data
*/
while (offset < packet.getSize()) {
int bytesDecoded = audioCoder.decodeAudio(samples, packet, offset);
if (bytesDecoded < 0) {
throw new RuntimeException("got error decoding audio in: " + filename);
}
offset += bytesDecoded;
/*
* Some decoder will consume data in a packet, but will not be able to construct
* a full set of samples yet. Therefore you should always check if you
* got a complete set of samples from the decoder
*/
if (samples.isComplete()) {
// note: this call will block if Java's sound buffers fill up, and we're
// okay with that. That's why we have the video "sleeping" occur
// on another thread.
playJavaSound(samples);
}
}
} else {
/*
* This packet isn't part of our video stream, so we just silently drop it.
*/
do {
} while (false);
}
}
/*
* Technically since we're exiting anyway, these will be cleaned up by
* the garbage collector... but because we're nice people and want
* to be invited places for Christmas, we're going to show how to clean up.
*/
if (videoCoder != null) {
videoCoder.close();
videoCoder = null;
}
if (audioCoder != null) {
audioCoder.close();
audioCoder = null;
}
if (container != null) {
container.close();
container = null;
}
closeJavaSound();
closeJavaVideo();
}
private static long millisecondsUntilTimeToDisplay(IVideoPicture picture) {
/**
* We could just display the images as quickly as we decode them, but it turns
* out we can decode a lot faster than you think.
*
* So instead, the following code does a poor-man's version of trying to
* match up the frame-rate requested for each IVideoPicture with the system
* clock time on your computer.
*
* Remember that all Xuggler IAudioSamples and IVideoPicture objects always
* give timestamps in Microseconds, relative to the first decoded item. If
* instead you used the packet timestamps, they can be in different units depending
* on your IContainer, and IStream and things can get hairy quickly.
*/
long millisecondsToSleep = 0;
if (mFirstVideoTimestampInStream == Global.NO_PTS) {
// This is our first time through
mFirstVideoTimestampInStream = picture.getTimeStamp();
// get the starting clock time so we can hold up frames
// until the right time.
mSystemVideoClockStartTime = System.currentTimeMillis();
millisecondsToSleep = 0;
} else {
long systemClockCurrentTime = System.currentTimeMillis();
long millisecondsClockTimeSinceStartofVideo = systemClockCurrentTime - mSystemVideoClockStartTime;
// compute how long for this frame since the first frame in the stream.
// remember that IVideoPicture and IAudioSamples timestamps are always in MICROSECONDS,
// so we divide by 1000 to get milliseconds.
long millisecondsStreamTimeSinceStartOfVideo = (picture.getTimeStamp() - mFirstVideoTimestampInStream) / 1000;
final long millisecondsTolerance = 50; // and we give ourselfs 50 ms of tolerance
millisecondsToSleep = (millisecondsStreamTimeSinceStartOfVideo
- (millisecondsClockTimeSinceStartofVideo + millisecondsTolerance));
}
return millisecondsToSleep;
}
/**
* Opens a Swing window on screen.
*/
/**
* Forces the swing thread to terminate; I'm sure there is a right
* way to do this in swing, but this works too.
*/
private static void closeJavaVideo() {
System.exit(0);
}
private static void openJavaSound(IStreamCoder aAudioCoder) throws LineUnavailableException {
AudioFormat audioFormat = new AudioFormat(aAudioCoder.getSampleRate(),
(int) IAudioSamples.findSampleBitDepth(aAudioCoder.getSampleFormat()),
aAudioCoder.getChannels(),
true, /* xuggler defaults to signed 16 bit samples */
false);
DataLine.Info info = new DataLine.Info(SourceDataLine.class, audioFormat);
mLine = (SourceDataLine) AudioSystem.getLine(info);
/**
* if that succeeded, try opening the line.
*/
mLine.open(audioFormat);
/**
* And if that succeed, start the line.
*/
mLine.start();
}
private static void playJavaSound(IAudioSamples aSamples) {
/**
* We're just going to dump all the samples into the line.
*/
byte[] rawBytes = aSamples.getData().getByteArray(0, aSamples.getSize());
mLine.write(rawBytes, 0, aSamples.getSize());
}
private static void closeJavaSound() {
if (mLine != null) {
/*
* Wait for the line to finish playing
*/
mLine.drain();
/*
* Close the line.
*/
mLine.close();
mLine = null;
}
}
private static void openJavaVideo(IStreamCoder videoCoder) {
mScreen = new VideoImage();
mScreen.setPreferredSize(new Dimension(videoCoder.getWidth(), videoCoder.getHeight()));
mScreen.setLocationRelativeTo(null);
}
}
Things I changed:
private static void openJavaVideo(IStreamCoder videoCoder) {
mScreen = new VideoImage();
mScreen.setPreferredSize(new Dimension(videoCoder.getWidth(), videoCoder.getHeight()));
mScreen.setLocationRelativeTo(null);
}
Moved openJavaVideo method into videoStream detector:
openJavaVideo(videoCoder);
Changed the first part of the main:
public static void main(String[] args) {
String filename = "http://techslides.com/demos/sample-videos/small.mp4";
// Let's make sure that we can actually convert video pixel formats.
if (!IVideoResampler.isSupported(IVideoResampler.Feature.FEATURE_COLORSPACECONVERSION)) {
throw new RuntimeException("you must install the GPL version of Xuggler (with IVideoResampler support) for this demo to work");
}
// Create a Xuggler container object
IContainer container = IContainer.make();
// Open up the container
if (container.open("http://techslides.com/demos/sample-videos/small.mp4", IContainer.Type.READ, null) < 0) {
throw new IllegalArgumentException("could not open file: " + filename);
}
Actually, the important part was:
if (container.open("http://techslides.com/demos/sample-videos/small.mp4", IContainer.Type.READ, null) < 0) {
throw new IllegalArgumentException("could not open file: " + filename);
}
Xuggle is/was one of the best:
Streaming video with Xuggler
Right now I don;t have a complete project, but I believe it had an example with its demo files.
Search google for xuggle video streaming demo or similar keywords with Xuggler. It is easy to use and support most of the known formats as it wraps FFMPEG with itself.
I came to another idea it may worth trying.
Create a JavaFX application.
Add a web browser into it, check webengine
create a template webpage which it contains a html player, or load the page in your server where it accept file id, then create a page that create a player for that file, similar to youtube, then auto play it.
This will be much better idea it you could do it.
There are sample codes for webengine and javaFX. Once you loaded a page, say, youtube or vimeo and played a video there, then sky is the limit :)
Will I'm working on project using JnetPcap API,I was able to list to run the ClassicPcapExample successfully
public class ClassicPcapExample {
/**
* Main startup method
*
* #param args
* ignored
*/
public static void main(String[] args) {
List<PcapIf> alldevs = new ArrayList<PcapIf>(); // Will be filled with NICs
StringBuilder errbuf = new StringBuilder(); // For any error msgs
/***************************************************************************
* First get a list of devices on this system
**************************************************************************/
int r = Pcap.findAllDevs(alldevs, errbuf);
if (r == Pcap.NOT_OK || alldevs.isEmpty()) {
System.err.printf("Can't read list of devices, error is %s", errbuf
.toString());
return;
}
System.out.println("Network devices found:");
int i = 0;
for (PcapIf device : alldevs) {
String description =
(device.getDescription() != null) ? device.getDescription()
: "No description available";
System.out.printf("#%d: %s [%s]\n", i++, device.getName(), description);
}
PcapIf device = alldevs.get(0); // We know we have atleast 1 device
System.out
.printf("\nChoosing '%s' on your behalf:\n",
(device.getDescription() != null) ? device.getDescription()
: device.getName());
/***************************************************************************
* Second we open up the selected device
**************************************************************************/
int snaplen = 64 * 1024; // Capture all packets, no trucation
int flags = Pcap.MODE_PROMISCUOUS; // capture all packets
int timeout = 10 * 1000; // 10 seconds in millis
Pcap pcap =
Pcap.openLive(device.getName(), snaplen, flags, timeout, errbuf);
if (pcap == null) {
System.err.printf("Error while opening device for capture: "
+ errbuf.toString());
return;
}
the problem I m having now , is my wireless interface is not listed among the interfaces , so I can sniff HTTP Packets .
Here is the Output of this program :
Network devices found:
#0: \Device\NPF_{273EF1C6-92B4-446F-9D88-553E18695A27} [VMware Virtual Ethernet Adapter]
#1: \Device\NPF_{C69FC3BE-1E6C-415B-9AAC-36D4654C7AD8} [Microsoft]
#2: \Device\NPF_{46AC6814-0644-4B81-BAC9-70FEB2002E07} [VMware Virtual Ethernet Adapter]
#3: \Device\NPF_{037F3BF4-B510-4A1D-90C0-1014FB3974F7} [Microsoft]
#4: \Device\NPF_{CA7D4FF0-B88B-4D0D-BBDC-A1923AF8D4B3} [Realtek PCIe GBE Family Controller]
#5: \Device\NPF_{3E2983E7-11F8-415A-BC81-E1B99CA8B092} [Microsoft]
Choosing 'VMware Virtual Ethernet Adapter' on your behalf:
jnetpcap didn't list your wireless interface as "wireless" like wireshark does.
It's listed as Microsoft instead, so your interface is one of the Microsoft devices in that list.
Let only your wireless access the net, then try the sniffer on each one until you find which is the wireless interface.
I am new to Java, and although I have mastered the syntaxes and constructs, I am having difficult time in getting processed digital audio samples from a mic.
What I am trying to achieve is very simple, while in the long run, I am trying to create a very simple spectrogram, but just to understand/master the audio manipulation process I am trying to start from scratch.
Here is my problem
When a microphone detects a single beep or any sound, I want to capture that binary data, and just simply display it, in its raw format.
Is that really too much to ask from JAVA?
I have read about analog/digital signals, FFT, matlab and I have searched many links in so like these one:
Is there a way of recording audio of streaming broadcast from a webpage?
Working with audio in Java
OpenAL playback captured audio data c++
and the famous introduction from oracle
http://docs.oracle.com/javase/tutorial/sound/capturing.html
and this is actually a good tutorial http://www.developer.com/java/other/article.php/3380031/Spectrum-Analysis-using-Java-Sampling-Frequency-Folding-Frequency-and-the-FFT-Algorithm.htm
But they all fall short of providing a solution to my answer.
I am not asking for a code, although it would it would be awesome, just to read every line and understand about the mechanics involved, but a simple hint would be nice as well.
And here is a simple code, to capture bytes, but only from an existing wav file
import java.io.FileInputStream;
import java.io.IOException;
public class Boo{
public static void main(String[] arguments){
try {
FileInputStream file = new FileInputStream("beep.wav");
boolean eof = false;
int count = 0;
while(!eof){
int input = file.read();
System.out.print(input + "");
if(input == -1)
eof = true;
else
count++;
}
file.close();
System.out.println("\nBytes read: " + count);
}catch (IOException e){
System.out.println("Error - " + e.toString());
}
}
}
After Bounty
-For better clarity-
All I am trying to make it just a simple program to read from mic. and show the binary data of the sound it caputures in a real time.
Think of it like a spectrogram, when sound is captured the graph goes up and down depending on the variety of the signal level, but in this case, there is no need to convert the binary data to audio graph, just only to show any raw data itself. No need to write/read files. Just capture from mic, and show what is read from the mic.
If the above provides to be difficult, as I have searched all over the web, and couldn't find anything helpful, you can just give me guides/directions..
thanks
javax.sound.sampled package should have all you need.
Example:
int duration = 5; // sample for 5 seconds
TargetDataLine line = null;
// find a DataLine that can be read
// (maybe hardcode this if you have multiple microphones)
Info[] mixerInfo = AudioSystem.getMixerInfo();
for (int i = 0; i < mixerInfo.length; i++) {
Mixer mixer = AudioSystem.getMixer(mixerInfo[i]);
Line.Info[] targetLineInfo = mixer.getTargetLineInfo();
if (targetLineInfo.length > 0) {
line = (TargetDataLine) mixer.getLine(targetLineInfo[0]);
break;
}
}
if (line == null)
throw new UnsupportedOperationException("No recording device found");
AudioFormat af = new AudioFormat(11000, 8, 1, true, false);
line.open(af);
line.start();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buf = new byte[(int)af.getSampleRate() * af.getFrameSize()];
long end = System.currentTimeMillis() + 1000 * duration;
int len;
while (System.currentTimeMillis() < end && ((len = line.read(buf, 0, buf.length)) != -1)) {
baos.write(buf, 0, len);
}
line.stop();
line.close();
baos.close();
Afterwards, you can dig the bytes out of your byte array output stream. Or you can of course process them inside the while loop if you prefer.
Here is some code copy-pasted from Sphinx4 microphone support. I hope it will be useful.
And a link to Sphinx homepage: http://cmusphinx.sourceforge.net/sphinx4/.
/**
* <p/> A Microphone captures audio data from the system's underlying audio input systems. Converts these audio data
* into Data objects. When the method <code>startRecording()</code> is called, a new thread will be created and used to
* capture audio, and will stop when <code>stopRecording()</code> is called. Calling <code>getData()</code> returns the
* captured audio data as Data objects. </p> <p/> This Microphone will attempt to obtain an audio device with the format
* specified in the configuration. If such a device with that format cannot be obtained, it will try to obtain a device
* with an audio format that has a higher sample rate than the configured sample rate, while the other parameters of the
* format (i.e., sample size, endianness, sign, and channel) remain the same. If, again, no such device can be obtained,
* it flags an error, and a call <code>startRecording</code> returns false. </p>
*/
public class Microphone extends BaseDataProcessor {
/**
* The property for the sample rate of the data.
*/
#S4Integer(defaultValue = 16000)
public static final String PROP_SAMPLE_RATE = "sampleRate";
/**
* The property that specifies whether or not the microphone will release the audio between utterances. On
* certain systems (Linux for one), closing and reopening the audio does not work too well. The default is false for
* Linux systems, true for others.
*/
#S4Boolean(defaultValue = true)
public final static String PROP_CLOSE_BETWEEN_UTTERANCES = "closeBetweenUtterances";
/**
* The property that specifies the number of milliseconds of audio data to read each time from the underlying
* Java Sound audio device.
*/
#S4Integer(defaultValue = 10)
public final static String PROP_MSEC_PER_READ = "msecPerRead";
/**
* The property for the number of bits per value.
*/
#S4Integer(defaultValue = 16)
public static final String PROP_BITS_PER_SAMPLE = "bitsPerSample";
/**
* The property specifying the number of channels.
*/
#S4Integer(defaultValue = 1)
public static final String PROP_CHANNELS = "channels";
/**
* The property specify the endianness of the data.
*/
#S4Boolean(defaultValue = true)
public static final String PROP_BIG_ENDIAN = "bigEndian";
/**
* The property specify whether the data is signed.
*/
#S4Boolean(defaultValue = true)
public static final String PROP_SIGNED = "signed";
/**
* The property that specifies whether to keep the audio data of an utterance around until the next utterance
* is recorded.
*/
#S4Boolean(defaultValue = false)
public final static String PROP_KEEP_LAST_AUDIO = "keepLastAudio";
/**
* The property that specifies how to convert stereo audio to mono. Currently, the possible values are
* "average", which averages the samples from at each channel, or "selectChannel", which chooses audio only from
* that channel. If you choose "selectChannel", you should also specify which channel to use with the
* "selectChannel" property.
*/
#S4String(defaultValue = "average", range = {"average", "selectChannel"})
public final static String PROP_STEREO_TO_MONO = "stereoToMono";
/**
* The property that specifies the channel to use if the audio is stereo
*/
#S4Integer(defaultValue = 0)
public final static String PROP_SELECT_CHANNEL = "selectChannel";
/**
* The property that specifies the mixer to use. The value can be "default," (which means let the
* AudioSystem decide), "last," (which means select the last Mixer supported by the AudioSystem), which appears to
* be what is often used for USB headsets, or an integer value which represents the index of the Mixer.Info that is
* returned by AudioSystem.getMixerInfo(). To get the list of Mixer.Info objects, run the AudioTool application with
* a command line argument of "-dumpMixers".
*
* #see edu.cmu.sphinx.tools.audio.AudioTool
*/
#S4String(defaultValue = "default")
public final static String PROP_SELECT_MIXER = "selectMixer";
private AudioFormat finalFormat;
private AudioInputStream audioStream;
private TargetDataLine audioLine;
private BlockingQueue<Data> audioList;
private Utterance currentUtterance;
private boolean doConversion;
private final int audioBufferSize = 160000;
private volatile boolean recording;
private volatile boolean utteranceEndReached = true;
private RecordingThread recorder;
// Configuration data
private AudioFormat desiredFormat;
private Logger logger;
private boolean closeBetweenUtterances;
private boolean keepDataReference;
private boolean signed;
private boolean bigEndian;
private int frameSizeInBytes;
private int msecPerRead;
private int selectedChannel;
private String selectedMixerIndex;
private String stereoToMono;
private int sampleRate;
/**
* #param sampleRate sample rate of the data
* #param bitsPerSample number of bits per value.
* #param channels number of channels.
* #param bigEndian the endianness of the data
* #param signed whether the data is signed.
* #param closeBetweenUtterances whether or not the microphone will release the audio between utterances. On
* certain systems (Linux for one), closing and reopening the audio does not work too well. The default is false for
* Linux systems, true for others
* #param msecPerRead the number of milliseconds of audio data to read each time from the underlying
* Java Sound audio device.
* #param keepLastAudio whether to keep the audio data of an utterance around until the next utterance
* is recorded.
* #param stereoToMono how to convert stereo audio to mono. Currently, the possible values are
* "average", which averages the samples from at each channel, or "selectChannel", which chooses audio only from
* that channel. If you choose "selectChannel", you should also specify which channel to use with the
* "selectChannel" property.
* #param selectedChannel the channel to use if the audio is stereo
* #param selectedMixerIndex the mixer to use. The value can be "default," (which means let the
* AudioSystem decide), "last," (which means select the last Mixer supported by the AudioSystem), which appears to
* be what is often used for USB headsets, or an integer value which represents the index of the Mixer.Info that is
* returned by AudioSystem.getMixerInfo(). To get the list of Mixer.Info objects, run the AudioTool application with
* a command line argument of "-dumpMixers".
*/
public Microphone(int sampleRate, int bitsPerSample, int channels,
boolean bigEndian, boolean signed, boolean closeBetweenUtterances, int msecPerRead, boolean keepLastAudio,
String stereoToMono, int selectedChannel, String selectedMixerIndex) {
initLogger();
this.bigEndian = bigEndian;
this.signed = signed;
this.desiredFormat = new AudioFormat
((float) sampleRate, bitsPerSample, channels, signed, bigEndian);
this.closeBetweenUtterances = closeBetweenUtterances;
this.msecPerRead = msecPerRead;
this.keepDataReference = keepLastAudio;
this.stereoToMono = stereoToMono;
this.selectedChannel = selectedChannel;
this.selectedMixerIndex = selectedMixerIndex;
}
public Microphone() {
}
/*
* (non-Javadoc)
*
* #see edu.cmu.sphinx.util.props.Configurable#newProperties(edu.cmu.sphinx.util.props.PropertySheet)
*/
#Override
public void newProperties(PropertySheet ps) throws PropertyException {
super.newProperties(ps);
logger = ps.getLogger();
sampleRate = ps.getInt(PROP_SAMPLE_RATE);
int sampleSizeInBits = ps.getInt(PROP_BITS_PER_SAMPLE);
int channels = ps.getInt(PROP_CHANNELS);
bigEndian = ps.getBoolean(PROP_BIG_ENDIAN);
signed = ps.getBoolean(PROP_SIGNED);
desiredFormat = new AudioFormat
((float) sampleRate, sampleSizeInBits, channels, signed, bigEndian);
closeBetweenUtterances = ps.getBoolean(PROP_CLOSE_BETWEEN_UTTERANCES);
msecPerRead = ps.getInt(PROP_MSEC_PER_READ);
keepDataReference = ps.getBoolean(PROP_KEEP_LAST_AUDIO);
stereoToMono = ps.getString(PROP_STEREO_TO_MONO);
selectedChannel = ps.getInt(PROP_SELECT_CHANNEL);
selectedMixerIndex = ps.getString(PROP_SELECT_MIXER);
}
/**
* Constructs a Microphone with the given InputStream.
*/
#Override
public void initialize() {
super.initialize();
audioList = new LinkedBlockingQueue<Data>();
DataLine.Info info
= new DataLine.Info(TargetDataLine.class, desiredFormat);
/* If we cannot get an audio line that matches the desired
* characteristics, shoot for one that matches almost
* everything we want, but has a higher sample rate.
*/
if (!AudioSystem.isLineSupported(info)) {
logger.info(desiredFormat + " not supported");
AudioFormat nativeFormat
= DataUtil.getNativeAudioFormat(desiredFormat,
getSelectedMixer());
if (nativeFormat == null) {
logger.severe("couldn't find suitable target audio format");
} else {
finalFormat = nativeFormat;
/* convert from native to the desired format if supported */
doConversion = AudioSystem.isConversionSupported
(desiredFormat, nativeFormat);
if (doConversion) {
logger.info
("Converting from " + finalFormat.getSampleRate()
+ "Hz to " + desiredFormat.getSampleRate() + "Hz");
} else {
logger.info
("Using native format: Cannot convert from " +
finalFormat.getSampleRate() + "Hz to " +
desiredFormat.getSampleRate() + "Hz");
}
}
} else {
logger.info("Desired format: " + desiredFormat + " supported.");
finalFormat = desiredFormat;
}
}
/**
* Gets the Mixer to use. Depends upon selectedMixerIndex being defined.
*
* #see #newProperties
*/
private Mixer getSelectedMixer() {
if (selectedMixerIndex.equals("default")) {
return null;
} else {
Mixer.Info[] mixerInfo = AudioSystem.getMixerInfo();
if (selectedMixerIndex.equals("last")) {
return AudioSystem.getMixer(mixerInfo[mixerInfo.length - 1]);
} else {
int index = Integer.parseInt(selectedMixerIndex);
return AudioSystem.getMixer(mixerInfo[index]);
}
}
}
/**
* Creates the audioLine if necessary and returns it.
*/
private TargetDataLine getAudioLine() {
if (audioLine != null) {
return audioLine;
}
/* Obtain and open the line and stream.
*/
try {
/* The finalFormat was decided in the initialize() method
* and is based upon the capabilities of the underlying
* audio system. The final format will have all the
* desired audio characteristics, but may have a sample
* rate that is higher than desired. The idea here is
* that we'll let the processors in the front end (e.g.,
* the FFT) handle some form of downsampling for us.
*/
logger.info("Final format: " + finalFormat);
DataLine.Info info = new DataLine.Info(TargetDataLine.class,
finalFormat);
/* We either get the audio from the AudioSystem (our
* default choice), or use a specific Mixer if the
* selectedMixerIndex property has been set.
*/
Mixer selectedMixer = getSelectedMixer();
if (selectedMixer == null) {
audioLine = (TargetDataLine) AudioSystem.getLine(info);
} else {
audioLine = (TargetDataLine) selectedMixer.getLine(info);
}
/* Add a line listener that just traces
* the line states.
*/
audioLine.addLineListener(new LineListener() {
#Override
public void update(LineEvent event) {
logger.info("line listener " + event);
}
});
} catch (LineUnavailableException e) {
logger.severe("microphone unavailable " + e.getMessage());
}
return audioLine;
}
/**
* Opens the audio capturing device so that it will be ready for capturing audio. Attempts to create a converter if
* the requested audio format is not directly available.
*
* #return true if the audio capturing device is opened successfully; false otherwise
*/
private boolean open() {
TargetDataLine audioLine = getAudioLine();
if (audioLine != null) {
if (!audioLine.isOpen()) {
logger.info("open");
try {
audioLine.open(finalFormat, audioBufferSize);
} catch (LineUnavailableException e) {
logger.severe("Can't open microphone " + e.getMessage());
return false;
}
audioStream = new AudioInputStream(audioLine);
if (doConversion) {
audioStream = AudioSystem.getAudioInputStream
(desiredFormat, audioStream);
assert (audioStream != null);
}
/* Set the frame size depending on the sample rate.
*/
float sec = ((float) msecPerRead) / 1000.f;
frameSizeInBytes =
(audioStream.getFormat().getSampleSizeInBits() / 8) *
(int) (sec * audioStream.getFormat().getSampleRate());
logger.info("Frame size: " + frameSizeInBytes + " bytes");
}
return true;
} else {
logger.severe("Can't find microphone");
return false;
}
}
/**
* Returns the format of the audio recorded by this Microphone. Note that this might be different from the
* configured format.
*
* #return the current AudioFormat
*/
public AudioFormat getAudioFormat() {
return finalFormat;
}
/**
* Returns the current Utterance.
*
* #return the current Utterance
*/
public Utterance getUtterance() {
return currentUtterance;
}
/**
* Returns true if this Microphone is recording.
*
* #return true if this Microphone is recording, false otherwise
*/
public boolean isRecording() {
return recording;
}
/**
* Starts recording audio. This method will return only when a START event is received, meaning that this Microphone
* has started capturing audio.
*
* #return true if the recording started successfully; false otherwise
*/
public synchronized boolean startRecording() {
if (recording) {
return false;
}
if (!open()) {
return false;
}
utteranceEndReached = false;
if (audioLine.isRunning()) {
logger.severe("Whoops: audio line is running");
}
assert (recorder == null);
recorder = new RecordingThread("Microphone");
recorder.start();
recording = true;
return true;
}
/**
* Stops recording audio. This method does not return until recording has been stopped and all data has been read
* from the audio line.
*/
public synchronized void stopRecording() {
if (audioLine != null) {
if (recorder != null) {
recorder.stopRecording();
recorder = null;
}
recording = false;
}
}
/**
* This Thread records audio, and caches them in an audio buffer.
*/
class RecordingThread extends Thread {
private boolean done;
private volatile boolean started;
private long totalSamplesRead;
private final Object lock = new Object();
/**
* Creates the thread with the given name
*
* #param name the name of the thread
*/
public RecordingThread(String name) {
super(name);
}
/**
* Starts the thread, and waits for recorder to be ready
*/
#Override
public void start() {
started = false;
super.start();
waitForStart();
}
/**
* Stops the thread. This method does not return until recording has actually stopped, and all the data has been
* read from the audio line.
*/
public void stopRecording() {
audioLine.stop();
try {
synchronized (lock) {
while (!done) {
lock.wait();
}
}
} catch (InterruptedException e) {
e.printStackTrace();
}
// flush can not be called here because the audio-line might has been set to null already by the mic-thread
// audioLine.flush();
}
/**
* Implements the run() method of the Thread class. Records audio, and cache them in the audio buffer.
*/
#Override
public void run() {
totalSamplesRead = 0;
logger.info("started recording");
if (keepDataReference) {
currentUtterance = new Utterance
("Microphone", audioStream.getFormat());
}
audioList.add(new DataStartSignal(sampleRate));
logger.info("DataStartSignal added");
try {
audioLine.start();
while (!done) {
Data data = readData(currentUtterance);
if (data == null) {
done = true;
break;
}
audioList.add(data);
}
audioLine.flush();
if (closeBetweenUtterances) {
/* Closing the audio stream *should* (we think)
* also close the audio line, but it doesn't
* appear to do this on the Mac. In addition,
* once the audio line is closed, re-opening it
* on the Mac causes some issues. The Java sound
* spec is also kind of ambiguous about whether a
* closed line can be re-opened. So...we'll go
* for the conservative route and never attempt
* to re-open a closed line.
*/
audioStream.close();
audioLine.close();
System.err.println("set to null");
audioLine = null;
}
} catch (IOException ioe) {
logger.warning("IO Exception " + ioe.getMessage());
ioe.printStackTrace();
}
long duration = (long)
(((double) totalSamplesRead /
(double) audioStream.getFormat().getSampleRate()) * 1000.0);
audioList.add(new DataEndSignal(duration));
logger.info("DataEndSignal ended");
logger.info("stopped recording");
synchronized (lock) {
lock.notify();
}
}
/**
* Waits for the recorder to start
*/
private synchronized void waitForStart() {
// note that in theory we coulde use a LineEvent START
// to tell us when the microphone is ready, but we have
// found that some javasound implementations do not always
// issue this event when a line is opened, so this is a
// WORKAROUND.
try {
while (!started) {
wait();
}
} catch (InterruptedException ie) {
logger.warning("wait was interrupted");
}
}
/**
* Reads one frame of audio data, and adds it to the given Utterance.
*
* #param utterance
* #return an Data object containing the audio data
* #throws java.io.IOException
*/
private Data readData(Utterance utterance) throws IOException {
// Read the next chunk of data from the TargetDataLine.
byte[] data = new byte[frameSizeInBytes];
int channels = audioStream.getFormat().getChannels();
long collectTime = System.currentTimeMillis();
long firstSampleNumber = totalSamplesRead / channels;
int numBytesRead = audioStream.read(data, 0, data.length);
// notify the waiters upon start
if (!started) {
synchronized (this) {
started = true;
notifyAll();
}
}
if (logger.isLoggable(Level.FINE)) {
logger.info("Read " + numBytesRead
+ " bytes from audio stream.");
}
if (numBytesRead <= 0) {
return null;
}
int sampleSizeInBytes =
audioStream.getFormat().getSampleSizeInBits() / 8;
totalSamplesRead += (numBytesRead / sampleSizeInBytes);
if (numBytesRead != frameSizeInBytes) {
if (numBytesRead % sampleSizeInBytes != 0) {
throw new Error("Incomplete sample read.");
}
data = Arrays.copyOf(data, numBytesRead);
}
if (keepDataReference) {
utterance.add(data);
}
double[] samples;
if (bigEndian) {
samples = DataUtil.bytesToValues
(data, 0, data.length, sampleSizeInBytes, signed);
} else {
samples = DataUtil.littleEndianBytesToValues
(data, 0, data.length, sampleSizeInBytes, signed);
}
if (channels > 1) {
samples = convertStereoToMono(samples, channels);
}
return (new DoubleData
(samples, (int) audioStream.getFormat().getSampleRate(),
collectTime, firstSampleNumber));
}
}
/**
* Converts stereo audio to mono.
*
* #param samples the audio samples, each double in the array is one sample
* #param channels the number of channels in the stereo audio
*/
private double[] convertStereoToMono(double[] samples, int channels) {
assert (samples.length % channels == 0);
double[] finalSamples = new double[samples.length / channels];
if (stereoToMono.equals("average")) {
for (int i = 0, j = 0; i < samples.length; j++) {
double sum = samples[i++];
for (int c = 1; c < channels; c++) {
sum += samples[i++];
}
finalSamples[j] = sum / channels;
}
} else if (stereoToMono.equals("selectChannel")) {
for (int i = selectedChannel, j = 0; i < samples.length;
i += channels, j++) {
finalSamples[j] = samples[i];
}
} else {
throw new Error("Unsupported stereo to mono conversion: " +
stereoToMono);
}
return finalSamples;
}
/**
* Clears all cached audio data.
*/
public void clear() {
audioList.clear();
}
/**
* Reads and returns the next Data object from this Microphone, return null if there is no more audio data. All
* audio data captured in-between <code>startRecording()</code> and <code>stopRecording()</code> is cached in an
* Utterance object. Calling this method basically returns the next chunk of audio data cached in this Utterance.
*
* #return the next Data or <code>null</code> if none is available
*/
#Override
public Data getData() throws DataProcessingException {
getTimer().start();
Data output = null;
if (!utteranceEndReached) {
try {
output = audioList.take();
} catch (InterruptedException ie) {
throw new DataProcessingException("cannot take Data from audioList", ie);
}
if (output instanceof DataEndSignal) {
utteranceEndReached = true;
}
}
getTimer().stop();
// signalCheck(output);
return output;
}
/**
* Returns true if there is more data in the Microphone.
* This happens either if the a DataEndSignal data was not taken from the buffer,
* or if the buffer in the Microphone is not yet empty.
*
* #return true if there is more data in the Microphone
*/
public boolean hasMoreData() {
return !(utteranceEndReached && audioList.isEmpty());
}
}
Have a look at xuggle. They dropped support for the project but it still has all the documentation and the google-group will sometimes give you a good answer.
As for specifically reading audio off hardware, start with these demos and work your way from there.
The hardest part about this is streaming the data from your mic, which is pretty well explained in the article you linked.
So stream it into xuggle using the oracle docs, then manipulate the audio how you see fit.
I didn't try followings, I just post them for just helping you. you may try those and may help you.
Record streaming audio in java?
http://docs.blackberry.com/en/developers/deliverables/11942/CS_recording_audio_from_a_player_740038_11.jsp
http://ganeshtiwaridotcomdotnp.blogspot.com/2011/12/java-sound-capture-from-microphone.html
http://edenti.deis.unibo.it/utils/Java-tips/Capturing%20Audio%20with%20Java%20Sound%20API.txt
http://docs.oracle.com/javase/tutorial/sound/capturing.html
I came across this snippet while going through the tutorial on how to decode a video :
private static long millisecondsUntilTimeToDisplay(IVideoPicture picture)
{
/**
* We could just display the images as quickly as we decode them, but it turns
* out we can decode a lot faster than you think.
*
* So instead, the following code does a poor-man's version of trying to
* match up the frame-rate requested for each IVideoPicture with the system
* clock time on your computer.
*
* Remember that all Xuggler IAudioSamples and IVideoPicture objects always
* give timestamps in Microseconds, relative to the first decoded item. If
* instead you used the packet timestamps, they can be in different units depending
* on your IContainer, and IStream and things can get hairy quickly.
*/
long millisecondsToSleep = 0;
if (mFirstVideoTimestampInStream == Global.NO_PTS)
{
// This is our first time through
mFirstVideoTimestampInStream = picture.getTimeStamp();
// get the starting clock time so we can hold up frames
// until the right time.
mSystemVideoClockStartTime = System.currentTimeMillis();
millisecondsToSleep = 0;
} else {
long systemClockCurrentTime = System.currentTimeMillis();
long millisecondsClockTimeSinceStartofVideo = systemClockCurrentTime - mSystemVideoClockStartTime;
// compute how long for this frame since the first frame in the stream.
// remember that IVideoPicture and IAudioSamples timestamps are always in MICROSECONDS,
// so we divide by 1000 to get milliseconds.
long millisecondsStreamTimeSinceStartOfVideo = (picture.getTimeStamp() - mFirstVideoTimestampInStream)/1000;
final long millisecondsTolerance = 50; // and we give ourselfs 50 ms of tolerance
millisecondsToSleep = (millisecondsStreamTimeSinceStartOfVideo -
(millisecondsClockTimeSinceStartofVideo+millisecondsTolerance));
}
return millisecondsToSleep;
}
I have scratched a lot but don't understand what does this method do ? what are we returning ? And why we are making the thread to sleep after the method returns ( what is the purpose of the method ?)
This is the complete code in the link :
import javax.sound.sampled.AudioFormat;
import javax.sound.sampled.AudioSystem;
import javax.sound.sampled.DataLine;
import javax.sound.sampled.LineUnavailableException;
import javax.sound.sampled.SourceDataLine;
import com.xuggle.xuggler.demos.*;
import com.xuggle.xuggler.Global;
import com.xuggle.xuggler.IAudioSamples;
import com.xuggle.xuggler.IContainer;
import com.xuggle.xuggler.IPacket;
import com.xuggle.xuggler.IPixelFormat;
import com.xuggle.xuggler.IStream;
import com.xuggle.xuggler.IStreamCoder;
import com.xuggle.xuggler.ICodec;
import com.xuggle.xuggler.IVideoPicture;
import com.xuggle.xuggler.IVideoResampler;
import com.xuggle.xuggler.Utils;
public class DecodeAndPlayAudioAndVideo
{
/**
* The audio line we'll output sound to; it'll be the default audio device on your system if available
*/
private static SourceDataLine mLine;
/**
* The window we'll draw the video on.
*
*/
private static VideoImage mScreen = null;
private static long mSystemVideoClockStartTime;
private static long mFirstVideoTimestampInStream;
/**
* Takes a media container (file) as the first argument, opens it,
* plays audio as quickly as it can, and opens up a Swing window and displays
* video frames with <i>roughly</i> the right timing.
*
* #param args Must contain one string which represents a filename
*/
#SuppressWarnings("deprecation")
public static void main(String[] args)
{
if (args.length <= 0)
throw new IllegalArgumentException("must pass in a filename as the first argument");
String filename = args[0];
// Let's make sure that we can actually convert video pixel formats.
if (!IVideoResampler.isSupported(IVideoResampler.Feature.FEATURE_COLORSPACECONVERSION))
throw new RuntimeException("you must install the GPL version of Xuggler (with IVideoResampler support) for this demo to work");
// Create a Xuggler container object
IContainer container = IContainer.make();
// Open up the container
if (container.open(filename, IContainer.Type.READ, null) < 0)
throw new IllegalArgumentException("could not open file: " + filename);
// query how many streams the call to open found
int numStreams = container.getNumStreams();
// and iterate through the streams to find the first audio stream
int videoStreamId = -1;
IStreamCoder videoCoder = null;
int audioStreamId = -1;
IStreamCoder audioCoder = null;
for(int i = 0; i < numStreams; i++)
{
// Find the stream object
IStream stream = container.getStream(i);
// Get the pre-configured decoder that can decode this stream;
IStreamCoder coder = stream.getStreamCoder();
if (videoStreamId == -1 && coder.getCodecType() == ICodec.Type.CODEC_TYPE_VIDEO)
{
videoStreamId = i;
videoCoder = coder;
}
else if (audioStreamId == -1 && coder.getCodecType() == ICodec.Type.CODEC_TYPE_AUDIO)
{
audioStreamId = i;
audioCoder = coder;
}
}
if (videoStreamId == -1 && audioStreamId == -1)
throw new RuntimeException("could not find audio or video stream in container: "+filename);
/*
* Check if we have a video stream in this file. If so let's open up our decoder so it can
* do work.
*/
IVideoResampler resampler = null;
if (videoCoder != null)
{
if(videoCoder.open() < 0)
throw new RuntimeException("could not open audio decoder for container: "+filename);
if (videoCoder.getPixelType() != IPixelFormat.Type.BGR24)
{
// if this stream is not in BGR24, we're going to need to
// convert it. The VideoResampler does that for us.
resampler = IVideoResampler.make(videoCoder.getWidth(), videoCoder.getHeight(), IPixelFormat.Type.BGR24,
videoCoder.getWidth(), videoCoder.getHeight(), videoCoder.getPixelType());
if (resampler == null)
throw new RuntimeException("could not create color space resampler for: " + filename);
}
/*
* And once we have that, we draw a window on screen
*/
openJavaVideo();
}
if (audioCoder != null)
{
if (audioCoder.open() < 0)
throw new RuntimeException("could not open audio decoder for container: "+filename);
/*
* And once we have that, we ask the Java Sound System to get itself ready.
*/
try
{
openJavaSound(audioCoder);
}
catch (LineUnavailableException ex)
{
throw new RuntimeException("unable to open sound device on your system when playing back container: "+filename);
}
}
/*
* Now, we start walking through the container looking at each packet.
*/
IPacket packet = IPacket.make();
mFirstVideoTimestampInStream = Global.NO_PTS;
mSystemVideoClockStartTime = 0;
while(container.readNextPacket(packet) >= 0)
{
/*
* Now we have a packet, let's see if it belongs to our video stream
*/
if (packet.getStreamIndex() == videoStreamId)
{
/*
* We allocate a new picture to get the data out of Xuggler
*/
IVideoPicture picture = IVideoPicture.make(videoCoder.getPixelType(),
videoCoder.getWidth(), videoCoder.getHeight());
/*
* Now, we decode the video, checking for any errors.
*
*/
int bytesDecoded = videoCoder.decodeVideo(picture, packet, 0);
if (bytesDecoded < 0)
throw new RuntimeException("got error decoding audio in: " + filename);
/*
* Some decoders will consume data in a packet, but will not be able to construct
* a full video picture yet. Therefore you should always check if you
* got a complete picture from the decoder
*/
if (picture.isComplete())
{
IVideoPicture newPic = picture;
/*
* If the resampler is not null, that means we didn't get the video in BGR24 format and
* need to convert it into BGR24 format.
*/
if (resampler != null)
{
// we must resample
newPic = IVideoPicture.make(resampler.getOutputPixelFormat(), picture.getWidth(), picture.getHeight());
if (resampler.resample(newPic, picture) < 0)
throw new RuntimeException("could not resample video from: " + filename);
}
if (newPic.getPixelType() != IPixelFormat.Type.BGR24)
throw new RuntimeException("could not decode video as BGR 24 bit data in: " + filename);
long delay = millisecondsUntilTimeToDisplay(newPic);
// if there is no audio stream; go ahead and hold up the main thread. We'll end
// up caching fewer video pictures in memory that way.
try
{
if (delay > 0)
Thread.sleep(delay);
}
catch (InterruptedException e)
{
return;
}
// And finally, convert the picture to an image and display it
mScreen.setImage(Utils.videoPictureToImage(newPic));
}
}
else if (packet.getStreamIndex() == audioStreamId)
{
/*
* We allocate a set of samples with the same number of channels as the
* coder tells us is in this buffer.
*
* We also pass in a buffer size (1024 in our example), although Xuggler
* will probably allocate more space than just the 1024 (it's not important why).
*/
IAudioSamples samples = IAudioSamples.make(1024, audioCoder.getChannels());
/*
* A packet can actually contain multiple sets of samples (or frames of samples
* in audio-decoding speak). So, we may need to call decode audio multiple
* times at different offsets in the packet's data. We capture that here.
*/
int offset = 0;
/*
* Keep going until we've processed all data
*/
while(offset < packet.getSize())
{
int bytesDecoded = audioCoder.decodeAudio(samples, packet, offset);
if (bytesDecoded < 0)
throw new RuntimeException("got error decoding audio in: " + filename);
offset += bytesDecoded;
/*
* Some decoder will consume data in a packet, but will not be able to construct
* a full set of samples yet. Therefore you should always check if you
* got a complete set of samples from the decoder
*/
if (samples.isComplete())
{
// note: this call will block if Java's sound buffers fill up, and we're
// okay with that. That's why we have the video "sleeping" occur
// on another thread.
playJavaSound(samples);
}
}
}
else
{
/*
* This packet isn't part of our video stream, so we just silently drop it.
*/
do {} while(false);
}
}
/*
* Technically since we're exiting anyway, these will be cleaned up by
* the garbage collector... but because we're nice people and want
* to be invited places for Christmas, we're going to show how to clean up.
*/
if (videoCoder != null)
{
videoCoder.close();
videoCoder = null;
}
if (audioCoder != null)
{
audioCoder.close();
audioCoder = null;
}
if (container !=null)
{
container.close();
container = null;
}
closeJavaSound();
closeJavaVideo();
}
What does The following method do ?
private static long millisecondsUntilTimeToDisplay(IVideoPicture picture)
{
/**
* We could just display the images as quickly as we decode them, but it turns
* out we can decode a lot faster than you think.
*
* So instead, the following code does a poor-man's version of trying to
* match up the frame-rate requested for each IVideoPicture with the system
* clock time on your computer.
*
* Remember that all Xuggler IAudioSamples and IVideoPicture objects always
* give timestamps in Microseconds, relative to the first decoded item. If
* instead you used the packet timestamps, they can be in different units depending
* on your IContainer, and IStream and things can get hairy quickly.
*/
long millisecondsToSleep = 0;
if (mFirstVideoTimestampInStream == Global.NO_PTS)
{
// This is our first time through
mFirstVideoTimestampInStream = picture.getTimeStamp();
// get the starting clock time so we can hold up frames
// until the right time.
mSystemVideoClockStartTime = System.currentTimeMillis();
millisecondsToSleep = 0;
} else {
long systemClockCurrentTime = System.currentTimeMillis();
long millisecondsClockTimeSinceStartofVideo = systemClockCurrentTime - mSystemVideoClockStartTime;
// compute how long for this frame since the first frame in the stream.
// remember that IVideoPicture and IAudioSamples timestamps are always in MICROSECONDS,
// so we divide by 1000 to get milliseconds.
long millisecondsStreamTimeSinceStartOfVideo = (picture.getTimeStamp() - mFirstVideoTimestampInStream)/1000;
final long millisecondsTolerance = 50; // and we give ourselfs 50 ms of tolerance
millisecondsToSleep = (millisecondsStreamTimeSinceStartOfVideo -
(millisecondsClockTimeSinceStartofVideo+millisecondsTolerance));
}
return millisecondsToSleep;
}
/**
* Opens a Swing window on screen.
*/
private static void openJavaVideo()
{
mScreen = new VideoImage();
}
/**
* Forces the swing thread to terminate; I'm sure there is a right
* way to do this in swing, but this works too.
*/
private static void closeJavaVideo()
{
System.exit(0);
}
private static void openJavaSound(IStreamCoder aAudioCoder) throws LineUnavailableException
{
AudioFormat audioFormat = new AudioFormat(aAudioCoder.getSampleRate(),
(int)IAudioSamples.findSampleBitDepth(aAudioCoder.getSampleFormat()),
aAudioCoder.getChannels(),
true, /* xuggler defaults to signed 16 bit samples */
false);
DataLine.Info info = new DataLine.Info(SourceDataLine.class, audioFormat);
mLine = (SourceDataLine) AudioSystem.getLine(info);
/**
* if that succeeded, try opening the line.
*/
mLine.open(audioFormat);
/**
* And if that succeed, start the line.
*/
mLine.start();
}
private static void playJavaSound(IAudioSamples aSamples)
{
/**
* We're just going to dump all the samples into the line.
*/
byte[] rawBytes = aSamples.getData().getByteArray(0, aSamples.getSize());
mLine.write(rawBytes, 0, aSamples.getSize());
}
private static void closeJavaSound()
{
if (mLine != null)
{
/*
* Wait for the line to finish playing
*/
mLine.drain();
/*
* Close the line.
*/
mLine.close();
mLine=null;
}
}
}
Rough algorithm in pseudocode:
Is this the first frame?
> Yes, save the frame time and the current time.
> No, do the following:
See how much time has passed since the first frame was displayed in System Time
See the difference in time between the current frame and the first frame
If there is a discrepancy
>Return a number of milliseconds to sleep for, else return 0.
So, what you then get is the overall algorithm of:
Decode frame
Check if we need to delay the frame (the method in question)
Delay
Display frame
In this way, the program will never display frames faster than the variable frame rate declared by the video. The method in question maintains the state of previous frame times and calculates how long to sleep for.
EDIT: The delay is needed because you can decode frames (much!) faster than the video's frame rate. Let's say you have a fairly slow machine running this program, and it takes 10ms to decode a frame. Let's also say that you have video that has a variable frame rate, but is roughly 10 frames per second (or 100ms per frame). Now if you take this step out of our 'overall algorithm':
Decode frame (10ms)
Display frame (1ms)
Decode frame (10ms)
Display frame (1ms)
If this was happening you would find 1 frame displayed every 10ms, meaning that the video will be displayed at 100 frames per second, which is wrong!
EDIT2: I guess what you're asking is why don't we do this?
Decode frame
Frame Delta = Current Frame Time - Previous Frame Time
Delay (for Delta milliseconds)
Display frame
The problem with this is what happens if it takes a long time to decode or display a frame? This would cause the frame rate to be significantly slower than the frame rate in the file.
Instead, this algorithm syncs the first frame to the system time, then does a little bit of extra calculation:
long systemTimeChange = currentSystemTime - firstFrameSystemTime;
long frameTimeChange = currentFrameTime - firstFrameTime;
// Subtract the time elapsed.
long differenceInChanges = frameTimeChange - systemTimeChange;
if(differenceInChanges > 0) {
// It was faster to decode than the frame rate!
Thread.sleep(differenceInChanges);
}
system time actually denotes the time at which the particular frame has been decoded and the frameTimeroughly denotes the frame rate the video has. So the difference goes like this : discrepancy = frameRate - decodeRate + tolerance Tolerance may be useful when decoding video takes larger time than it takes or the media takes longer time to display . Here is what you get from the difference :
Since decoding is too fast compared to the frame rate of video we have to wait some time and won't display that frame right now. And we use systemTimeStamp to sync our frames and hold it till the right time. In the above picture you see how fast the decoding rate is but frame rate is slow compared to the decoding rate.
seems like this:
* So instead, the following code does a poor-man's version of trying to
* match up the frame-rate requested for each IVideoPicture with the system
* clock time on your computer.
the delay is to try to match the framerate.
I'm working on a simple program to read a continuous stream of data from a plain old serial port. The program is written in Processing. Performing a simple read of data and dumping into to the console works perfectly fine, but whenever I add any other functionality (graphing,db entry) to the program, the port starts to become de-synchronized and all data from the serial port starts to become corrupt.
The incoming data from the serial port is in the following format :
A [TAB] //start flag
Data 1 [TAB]
Data 2 [TAB]
Data 3 [TAB]
Data 4 [TAB]
Data 5 [TAB]
Data 6 [TAB]
Data 7 [TAB]
Data 8 [TAB]
COUNT [TAB] //count of number of messages sent
Z [CR] //end flag followed by carriage return
So as stated, if I run the program below and simply have it output to the console, it runs fine without issue for several hours. If I add the graphing functionality or database connectivity, the serial data starts to come in garbled and serial port handler is never able to decode the message correctly again. I've tried all sorts of workarounds to this issue, thinking it is a timing problem but reducing the speed of the serial port doesn't seem to make a change.
If you see the serial port handler, I provide a large buffer just in case the terminating Z character is chopped off. I check to make sure the A and Z characters are in the correct place and in turn that the created "substring" is the correct length. When the program starts to fail, the substring will continuously fail this check until the program just crashes. Any ideas? I've tried several different ways of reading the serial port and am just beginning to wonder if I am missing something stupid here.
//Serial Port Tester
import processing.serial.*;
import processing.net.*;
import org.gwoptics.graphics.graph2D.Graph2D;
import org.gwoptics.graphics.graph2D.traces.ILine2DEquation;
import org.gwoptics.graphics.graph2D.traces.RollingLine2DTrace;
import de.bezier.data.sql.*;
SQLite db;
RollingLine2DTrace r1,r2,r3,r4;
Graph2D g;
Serial mSerialport; //the serial port
String[] svalues = new String[8]; //string values
int[] values = new int[8]; //int values
int endflag = 90; //Z
byte seperator = 13; //carriage return
class eq1 implements ILine2DEquation {
public double computePoint(double x,int pos) {
//data function for graph/plot
return (values[0] - 32768);
}
}
void connectDB()
{
db = new SQLite( this, "data.sqlite" );
if ( db.connect() )
{
db.query( "SELECT name as \"Name\" FROM SQLITE_MASTER where type=\"table\"" );
while (db.next())
{
println( db.getString("Name") );
}
}
}
void setup () {
size(1200, 1000);
connectDB();
println(Serial.list());
String portName = Serial.list()[3];
mSerialport = new Serial(this, portName, 115200);
mSerialport.clear();
mSerialport.bufferUntil(endflag); //generate serial event when endflag is received
background(0);
smooth();
//graph setup
r1 = new RollingLine2DTrace(new eq1(),250,0.1f);
r1.setTraceColour(255, 0, 0);
g = new Graph2D(this, 1080, 500, false);
g.setYAxisMax(10000);
g.addTrace(r1);
g.position.y = 50;
g.position.x = 100;
g.setYAxisTickSpacing(500);
g.setXAxisMax(10f);
}
void draw () {
background(200);
//g.draw(); enable this and program crashes quickly
}
void serialEvent (Serial mSerialport)
{
byte[] inBuffer = new byte[200];
mSerialport.readBytesUntil(seperator, inBuffer);
String inString = new String(inBuffer);
String subString = "";
int startFlag = inString.indexOf("A");
int endFlag = inString.indexOf("Z");
if (startFlag == 0 && endFlag == 48)
{
subString = inString.substring(startFlag+1,endFlag);
}
else
{
println("ERROR: BAD MESSAGE DISCARDED!");
subString = "";
}
if ( subString.length() == 47)
{
svalues = (splitTokens(subString));
values = int(splitTokens(subString));
println(svalues);
// if (db.connect()) //enable this and program crashes quickly
// {
// if ( svalues[0] != null && svalues[7] != null)
// {
// statement = svalues[7] + ", " + svalues[0] + ", " + svalues[1] + ", " + svalues[2] + ", " + svalues[3] + ", " + svalues[4] + ", " + svalues[5] + ", " + svalues[6];
// db.execute( "INSERT INTO rawdata (messageid,press1,press2,press3,press4,light1,light2,io1) VALUES (" + statement + ");" );
// }
// }
}
}
While I'm not familiar with your specific platform, my first thought from reading your problem description is that you still have a timing problem. At 115,200bps, data is coming in rather quickly-- more than 10 characters every millisecond. As such, if you spend precious time opening a database (slow file IO) or drawing graphics (also potentially slow), you might well not be able to keep up with the data.
As such, it might be a good idea to put the serial port processing on its own thread, interrupt, etc. That might make the multitasking much easier. Again, this is just an educated guess.
Also, you say that your program "crashes" when you enable the other operations. Do you mean that the entire process actually crashes, or that you get corrupted data, or both? Is it possible that you are overrunning your 200 byte inBuffer[]? At 115kbps, it wouldn't take but 20ms to do so.