I know that exoplayer has support for RTSP, but I need C++ code that works on players from lots of OSs, so I need to parse the RTP packet in C++ to NAL units before passing to exoplayer
I found a way to decode RTP packets using live555 and extract its NAL units. According to ExoPlayer's documentation:
Components common to all ExoPlayer implementations are:
A MediaSource that defines the media to be played, loads the media, and from which the loaded media can be read.
A MediaSource is
injected via ExoPlayer.prepare at the start of playback. ...
So I need a custom MediaSource that can extract NAL units from my C++ code.
At the class reference for MediaSource we can see that there are already some MediaSources available. I though maybe SmoothStreaming MediaSource could work but there's no description of what it does exactly and in its constructor I have to provide an Uri or an SsManifest (what).
I can see that there exists a NAL unit utility in this library so maybe things are already half done
So how to build or use an already available MediaSource to read NAL units for ExoPlayer to play?
As an additinal, how would you pass the NAL units from C++ to Java? In the code I found it's simply storing them in a C++ buffer. Should I read this buffer in Java somehow?
UPDATE:
I've been researching how this library works. It all begins with a MediaSource which have objects like Extractor and DataSource. Seems that ExtractorMediaSource is a MediaSource where you can provide your own Extractor and DataSource.
As I understood, DataSource is a class that gets the raw bytes from any possible place, be it a file read or a network packet. Based on the available Extractor classes on the library like Mp4Extractor and Mp3Extractor, an Extractor is something that will interpret the data read from DataSource. The two main methods from the Extractor interface are:
void init(ExtractorOutput output)
int read(ExtractorInput input, PositionHolder seekPosition)
I don't know what are ExtractorInput and ExtractorInput for, but they look important.
So somehow Extractor reads from DataSource, parses it and sends to Renderer in a common format?
I need to know how is this common format so I can parse the NAL units that I read from a custom DataSource.
You cannot play NAL units or raw H.264 stream with ExoPlayer. The picture data must exist within a supported container/format.
It's not clear what your mysterious C++ code is doing, is it an NDK setup? What's its role in Android decoding? Are you saying you're unable to pass [from the C++ function] a data array into some Android function as function parameter? Is it something like this Java to C++ setup? It's not clear what your real problem is...
If you insist on Exoplayer, I can tell you that FLV is the one (on containers list) that might be the best option since it can be built in real-time (re-muxing). You first create an FLV header that holds SPS and PPS data then followed by keyframe (extracted H264 data from the first NAL). You'll have to get familiar with FLV bytes structure, but each frame header is around 13 bytes followed by NAL data, repeat for each frame until end. This woud be realtime transcoding.
As a second option, for Android, you could just use MediaCodec to decode the H264 as extracted from the NAL units. Here is a useful example source. Just use as:
MediaCodec.createDecoderByType("video/avc"); //then later give NAL units
Study also functions of this other source for ideas of how it works to via Android's own decoder.
Other starting points:
Android Decode raw h264 stream with MediaCodec
(NAL units).
Decode H264 raw stream using mediacodec
I did something very similar, you can check my blog post about some aspects on customizing ExoPlayer (https://medium.com/#mahmoud.mohamed.bahaa/diving-into-exoplayer-getting-more-control-over-the-framework-cac436d1472c)
Related
My goal is to edit audio data. I already unpacked data stored in .wav file by using AudioInputStream.read() method and saved it into a byte[] array. So I'm curious if there any libraries in Java that can manipulate raw audio data that I just got? I can't use javax.sound.sampled.Mixer because I don't have any available mixers. By "manipulate raw audio data" I mean adding a reverb effect, normalizing effect, etc.
The best idea is to use appropriate drivers. Or you can reuse algorithms used in Audacity.
I'm trying to record from the microphone to a wav file as per this example. At the same time, I need to be able to test for input level/volume and send an alert if it's too low. I've tried what's described in this link and seems to work ok.
The issue comes when trying to record and read bytes at the same time using one TargetDataLine (bytes read for monitoring are being skipped for recording and vice-versa.
Another thing is that these are long processes (hours probably) so memory usage should be considered.
How should I proceed here? Any way to clone TargetDataLine? Can I buffer a number of bytes while writing them with AudioSystem.write()? Is there any other way to write to a .wav file without filling the system memory?
Thanks!
If you are using a TargetDataLine for capturing audio similar to the example given in the Java Tutorials, then you have access to a byte array called "data". You can loop through this array to test the volume level before outputting it.
To do the volume testing, you will have to convert the bytes to some sort of sensible PCM data. For example, if the format is 16-bit stereo little-endian, you might take two bytes and assemble to either a signed short or a signed, normalized float, and then test.
I apologize for not looking more closely at your examples before posting my "solution".
I'm going to suggest that you extend InputStream, making a customized version that also performs the volume test. Override the 'read' method so that it obtains the byte that it returns from the code you have that tests the volume. You'll have to modify the volume-testing code to work on a per-byte basis and to pass through the required byte.
You should then be able to use this extended InputStream as an argument when you create the AudioInputStream for the output-to-wav stage.
I've used this approach to save audio successfully via two data sources: once from an array that is populated beforehand, once from a streaming audio mix passing through a "mixer" I wrote to combine audio data sources. The latter would be more like what you need to do. I haven't done it from a microphone source, though. But the same approach should work, as far as I can tell.
I've a stream with contains a audio/video stream. The video encoding is H.264 AVC and the audio encoding is G.711 ยต-law (PCMU). (I have no control over the output format.)
When I try to display the video into a VideoView frame, the device says that it cannot play the video. I guess that's because Android doesn't support the aforementioned audio encoding.
Is there a way to somehow display the selected stream in the VideoView?
Can I just use a Java code snippet to 'compute' the codec? I've seen the class android.net.rtp.AudioCodec, which contains a field PCMU, and I can imagine that class would be useful.
Or do I have to insert a library or some native code (FFmpeg) into my application and use that? If yes, how?
How to fix it?
If you want to manipulate a stream before you display it, you are unfortunately getting into tricky territory on an Android device - if you do have any possibility of doing the conversion on the serve side it will likely by much easier (and you should also usually have more 'horsepower' on the server side).
Assuming that you do need to convert on the server side, then a technique you can use is to stream the video from the server to your app, convert it as needed and then 'stream' from a localhost server in your app to a VideoView in your app. Roughly the steps are:
Stream the encrypted file from the server as usual, 'chunk by chunk'
On your Android device, read from the stream and convert each chunk as it is received
Using a localhost http server on your Android device, now 'serve' the converted chunks to the MediaPlayer (the media player should be set up to use a URL pointing at your localhost http server)
An example of this approach, I believe, is LibMedia: sure: http://libeasy.alwaysdata.net (note, this is not a free library, AFAIK).
For the actual conversion you can use ffmpeg as you suggest - there are a number of ffmpeg wrapper for Android projects that you can either use or look at to design your own. Some examples:
http://hiteshsondhi88.github.io/ffmpeg-android-java/
https://github.com/jhotovy/android-ffmpeg
Take a look at the note about libffmpeginvoke in the second link in particular if you are planning to write your won wrapper.
One thing to be aware of is that compressions techniques, in particular video compression, often use an approach where one packet or frame is compressed relative to the frames before it (and sometimes even the frames after it). For example the first frame might be a 'key frame', the next five frames might just contain data to explain how they differ from the key frame, and the the seventh frame might be another key frame and so on. This means you generally want a 'chunk' which has all the required key frames if you are converting the chunk from one format to another.
I don't think you will have this problem converting from PCM to AAC audio, but it useful to be aware of the concept anyway.
Hi,
We are students working on a Java project where we have to:
Capture the video stream from a webcam
Merge this stream with a video file (we take the average of colors of each pixel, so the two streams are superimposed)
Transmit the result of this merge on the network, using RTP or RTSP (theoretically, it will be received by 2 Android tablets).
One thing that's very important is that all these operations should be in real-time (or almost). The RTP stream should be received in the same time that the webcam is capturing the video.
In order to do this, we use the Java Media Framework (JMF) API. The first and second points have been successfully implemented : each second, 30 BufferedImage from the webcam are merged with 30 BufferedImage from the video file. The result is displayed on a classic JFrame (we do that using only Swing and JMF), and it works very good.
We still have to do the third point. Sending a video stream via RTP is not very hard.
But here is the matter: due to points 1 and 2, we do not have a video stream, but a serie of BufferedImages. We know how to get a video file from these BufferedImage. But it's only a video file recorded on the hard drive, and it can't be sent on the network in real-time. So how can we make an on-the-fly stream from these BufferedImage that can be directly sent via RTP?
Thanks in advance for your help.
I'd like to improve my answer.
First there is the question whether you really have to use RTP/RTSP. If you don't have to, you can send your image data just through a DatagramSocket and assume the receiver knows how to decode them. Use something like
DatagramSocket socket = new DatagramSocket(port);
DatagramPacket packet = new DatagramPacket(new byte[1], 1);
packet.setAddress(receiver.getAddress());
packet.setPort(port);
while(running)
{
byte[] data = getMergedImageData();
packet.setData(data);
packet.setLength(data.length);
socket.send(packet);
}
If you have to use RTP you should have a look at the mjsip project. Basically you have to create a valid RTP header (e.g. the first 12 bytes in your databuffer). This is rather simple if you know where each bit belongs to.
Depending on how you encode your images you might have to be wary of additional requirements. For example, when sending Jpeg through RTP you have to remove the complete Jpeg header and create an abbreviated RTP/Jpeg header and place it between the RTP header and the payload. The receiver will have to recreate the Jpeg header from the abbreviated header. For Jpeg, also make sure to add an EOI marker to your image if not already present. I think in this regard ffmpeg can do a lot of work for you if you dare to dive into JNI.
For more information on how the RTP payloads have to be tailored see this.
Cheers
~
I want to split a video file into multiple parts and then rejoin some of them to make a new video file.
I am doing it by looping over the packets using xuggle and then writing some of them (after adjusting its timestamps) to the new file, but when I play the file, there is some disturbance in the transition frames. (It might be because the decoding of frame depends upon its preceding frame which has been discarded as part of the program)
How can I get rid of the disturbance?
Ideally you split on keyframes, since they usually don't depend on preceding frames.
The IPacket class has a isKey function to test for this condition.
I'm not sure what sort of compression format you are working with though. I have tried splitting a mp4 stream with xuggler, and found the results to be quite buggy.