Develop Reference

Cache a sound and create instaces of it - java

I am currently programming a game and now I also want to add sound. My current method works fine but I am not happy with it.
new Sound(new Resource().readAndGetStream("small_click.wav")).play();
This line of code reads the file small_click.wav whenever it is getting executed. But I think it is not very efficient to always read the resource file when it's needed.
So what I want to do now is caching a sound in a variable or something to not have to load the sound from file again. But I also want to create a new object from the sound, so I can play it mutiple times and it overlaps in the speakers.
I can't find a way to do this. I already tried to use Threads but.. this code works without any threads.
If you want to know, here is the code of the Sound class:
public Sound(InputStream audioSrc) {
try {
InputStream bufferedIn = new BufferedInputStream(audioSrc);
AudioInputStream audioStream = AudioSystem.getAudioInputStream(bufferedIn);
clip = AudioSystem.getClip();
clip.open(audioStream);
} catch {
...exception handling...
}
}
public void play() {
clip.setFramePosition(0);
clip.start();
}
And If you want to know what the "new Resource().readAndGetStream()" does:
It basically loads a resource and returns an InputStream of that resource with getResourceAsStream().

With the Sound class that you have, you can easily create a "cache". For example, create an array of type Sound[] soundCache and execute the first part of the code line you gave in your example.
soundCache[0] = new Sound(new Resource().readAndGetStream("small_click.wav"));
You could even consider making constants to correspond to each sound.
final int SMALL_CLICK = 0;
Then, when it is time to play the sound, execute your play function.
soundCache[SMALL_CLICK].play();
Going from here to having concurrent playbacks of a given sound is quite a bit more involved. If continuing to work with Clip as your basis, I don't know of any way to get overlapping playbacks of the same sound resource except by making and managing as many copies of the Clip as you might want to allow to be heard at once.
When faced with this coding challenge, I ended up writing my own library, AudioCue. It is available on Github, and has a very permissive license.
The basic concept is to store the audio data in an array of signed PCM floats, and manage the concurrent playback by having multiple "cursors" that can independently iterate through the PCM data and feed it to a SourceDataLine.
These cursors can be managed in real time. You can change the rate at which they travel through the data and scale the volume level of the data, allowing frequency and volume changes.
I did my best to keep the API as similar to a Clip as practical, so that the class would be easy to use for coders familiar with Java's Clip.
Feel free to examine the code for ideas/examples or make use of this library in your project.

Related

I'm creating an app in which multiple devices can connect to each other in a LAN and play songs from each other's device.
Right now, when device A requests a song from device B, B sends the song file back to A over a socket. Once the song finishes downloading, A starts playing it using MediaPlayer.
Now the problem with this is that even on a high speed LAN, it can take a couple of seconds to finish fetching the song, so there's a noticeable delay between selecting a song and it's playback actually starting.
I want to find a way to play a song while it's still being downloaded. Here are some possible solutions I found, but wasn't able to explore for certain reasons:
In the MediaPlayer documentation, I noticed a constructor that takes a MediaDataSource implementation allowing me to manually return parts of the media file whenever the MediaPlayer requires it so I could hold off returning a byte until it finishes downloading (if it hasn't finished downloading already), effectively acting like a "buffering" action. It seemed like a solution but unfortunately, my app's minSdk is set to 16 and MediaDataSource is for 23 and above only, so I couldn't try it out.
One option was to use MediaPlayer's setDataSource(FileDescriptor fd, long offset, long length) method. This would allow me to tell the MediaPlayer to only play up to a certain byte of the song. That way, I could wait for more parts of the file (or the entire file) to become available, and then use setNextMediaPlayer() and pass in a new MediaPlayer object that prepares the entire song and is made to seek up to the point where the previous media player object will stop playing so that there's a seamless transition.
But there's another problem with this. I need to be able to calculate the millisecond position that would be reached at that last specified byte of the first incomplete media player object. Otherwise I wouldn't know what position to seek the next media player object to in order to get a seamless transition. This calculation seems impossible for lossy formats.
I don't really know if this option will work or not, I'm just making assumptions. I noticed that setDataSource() takes a Uri. If that Uri points to a file on the phone, the media player just loads the entire file. But if the Uri points to an audio file on the internet that needs to be streamed, it figures that out on it's own and it handles all the details of downloading and buffering. So what I want to know is, is it possible to expose the song on device B as a Uri to device A so that media player just treats it as if it's a song on the internet? All this time I was using sockets and manually copying a file from one device to another so I have no idea how this would work. Could anyone explain if this is possible?
There's actually a reason why I haven't been exploring ways to "stream" a song from one device to another. That's because I want to song to be cached on device B so that later if I switch to another song and then back the previously played song from device A, it shouldn't have to stream it again. It should just play the cached copy.
Finally, I came across ExoPlayer. It seems to provide a large amount of customization. It seems like I could make custom implementations of it's classes to handle all the buffering. But the documentation and examples are few and far too complicated for me. I have no idea how to pull it off. Is this solution too complex for what I want to achieve?
Hope someone can point me in the right direction here.

Ended up using an answer from here:
MediaPlayer stutters at start of mp3 playback
This solution sets up a local server that the MediaPlayer can use to stream the file. The server only sends bytes to the MediaPlayer while those bytes are available. Otherwise, it blocks until more bytes are available. Thus the MediaPlayer buffers it as if it were a file from a web server.
I took the same class provided in that answer and just tweaked it to fit my situation.

I have a layout that has about 60 buttons and each one, when pressed, plays a different audio file. I have all my audio files as mp3s in my assets folder and to play them I'm basically using the same code as is used in the Google NDK samples "native-audio" project:
https://github.com/googlesamples/android-ndk
I have 10 identical native functions (just with uniquely named variables) that work like this..
function to play sound:
jboolean Java_com_example_nativeaudio_Fretboard_player7play(JNIEnv* env, jclass clazz, jobject assetManager, jstring filename)
{
SLresult result;
// convert Java string to UTF-8
const char *utf8 = (*env)->GetStringUTFChars(env, filename, NULL);
assert(NULL != utf8);
// use asset manager to open asset by filename
AAssetManager* mgr = AAssetManager_fromJava(env, assetManager);
assert(NULL != mgr);
AAsset* asset = AAssetManager_open(mgr, utf8, AASSET_MODE_UNKNOWN);
// release the Java string and UTF-8
(*env)->ReleaseStringUTFChars(env, filename, utf8);
// the asset might not be found
if (NULL == asset) {
return JNI_FALSE;
}
// open asset as file descriptor
off_t start, length;
int fd = AAsset_openFileDescriptor(asset, &start, &length);
assert(0 <= fd);
AAsset_close(asset);
// configure audio source
SLDataLocator_AndroidFD loc_fd = {SL_DATALOCATOR_ANDROIDFD, fd, start, length};
SLDataFormat_MIME format_mime = {SL_DATAFORMAT_MIME, NULL, SL_CONTAINERTYPE_UNSPECIFIED};
SLDataSource audioSrc = {&loc_fd, &format_mime};
// configure audio sink
SLDataLocator_OutputMix loc_outmix = {SL_DATALOCATOR_OUTPUTMIX, outputMixObject};
SLDataSink audioSnk = {&loc_outmix, NULL};
// create audio player
const SLInterfaceID ids[3] = {SL_IID_SEEK, SL_IID_MUTESOLO, SL_IID_VOLUME};
const SLboolean req[3] = {SL_BOOLEAN_TRUE, SL_BOOLEAN_TRUE, SL_BOOLEAN_TRUE};
result = (*engineEngine)->CreateAudioPlayer(engineEngine, &p7PlayerObject, &audioSrc, &audioSnk,
3, ids, req);
assert(SL_RESULT_SUCCESS == result);
(void)result;
// realize the player
result = (*p7PlayerObject)->Realize(p7PlayerObject, SL_BOOLEAN_FALSE);
assert(SL_RESULT_SUCCESS == result);
(void)result;
// get the play interface
result = (*p7PlayerObject)->GetInterface(p7PlayerObject, SL_IID_PLAY, &p7PlayerPlay);
assert(SL_RESULT_SUCCESS == result);
(void)result;
if (NULL != p7PlayerPlay) {
// play
result = (*p7PlayerPlay)->SetPlayState(p7PlayerPlay, SL_PLAYSTATE_PLAYING);
assert(SL_RESULT_SUCCESS == result);
(void)result;
}
return JNI_TRUE;
}
function to stop that sound:
void Java_com_example_nativeaudio_Fretboard_player7stop(JNIEnv* env, jclass clazz)
{
SLresult result;
// make sure the asset audio player was created
if (NULL != p7PlayerPlay) {
// set the player's state
result = (*p7PlayerPlay)->SetPlayState(p7PlayerPlay, SL_PLAYSTATE_STOPPED);
assert(SL_RESULT_SUCCESS == result);
(void)result;
// destroy file descriptor audio player object, and invalidate all associated interfaces
(*p7PlayerObject)->Destroy(p7PlayerObject);
p7PlayerObject = NULL;
p7PlayerPlay = NULL;
}
}
this is easy to deal with, but I want to minimize latency and avoid having to do (*engineEngine)->CreateAudioPlayer() every time I want to play a different file. Is there any way to just change the audioSrc used by the audio player without having to destroy and recreate it from scratch every time?
As a bonus, where can I read more about this stuff? Seems pretty difficult to find any information on OpenSL ES anywhere.

We're in the same boat, I'm currently familiarizing myself too with the NDK and OpenSL ES. My answer is based on my experience entirely consisting of ~2 days of experimentation so there might be better approaches but the information might help you on your way.
I have 10 identical native functions (just with uniquely named variables) that work like this..
If I understood your case correctly, you don't need to have duplicate functions for this. The only thing which differs in these calls is the button pressed and ultimately the sound to play and this can be passed as parameters through the JNI call. You can store the created player and data in a globally accessible structure so you can retrieve it when you need to stop/replay it, maybe using the buttonId as a key to a map.
[..]but I want to minimize latency and avoid having to do (*engineEngine)->CreateAudioPlayer() every time I want to play a different file. Is there any way to just change the audioSrc used by the audio player without having to destroy and recreate it from scratch every time?
Yes, constantly creating and destroying players is costly and can lead to fragmentation of the heap (as stated in the OpenSL ES 1.0 Specification). First, I thought he DynamicSourceItf would allow you to switch data sources but it seems that this interface is not intended to be used like that, at least on Android 6 this returns 'feature unsupported'.
I doubt that creating a player for each unique sound would be a good solution especially since playing the same sound multiple times on top of each other (as it's common in a game for example) would require an arbitrary amount of additional players for that same sound.
Buffer Queues
BufferQueues are queues of individual buffers which a player will process when playing. When all the buffers have been processed, the player 'stops' (it's official state is still 'playing' though) but will resume as soon as new buffers are being enqueued.
What this allows you to do is to create as many players as overlapping sounds you require. When you want to play a sound, you iterate over these players until you've found one which is not currently processing buffers (BufferQueueItf->GetState(...) provides this information or a callback can be registered so you can tag players as being 'free'). Then, you enqueue as many buffers as your sound needs which will start playing immediately.
The format of a BufferQueue is, as far as I know, locked at creation. So you have to make sure that you either have all your input buffers in the same format or you create different BufferQueue (and players) for each format.
Android Simple BufferQueue
According to the Android NDK documentation, the BufferQueue interface is expected to have significant changes in the future. They have extracted a simplified interface with most of BufferQueue's functionality and called it AndroidSimpleBufferQueue. This interface is not expected to change and thus makes your code more future proof.
The main functionality you loose by using the AndroidSimpleBufferQueue is to be able to use non-PCM source data, so you'd have to decode your files before use. This can be done in OpenSL ES using a AndroidSimpleBufferQueue as a sink. More recent APIs have additional support using the MediaCodec and it's NDK implementation NDKMedia (checkout the native-codec example).
Resources
The NDK documentation does contain some important information which are hard to find anywhere else. Here's the OpenSL ES specific page.
It might be close to 600 pages and hard to digest, but the OpenSL ES 1.0 Specification should be your primary resource of information. I highly recommend reading chapter 4 as it gives a good overview of how things work. Chapter 3 has a bit more information on the specific design. Then, I just jump around using the search function to read up on interfaces and objects as I go.
Understanding OpenSL ES
Once you have understood the basic principles of how OpenSL works, it seems to be quite straightforward. There are media objects (players and recorders, etc) and data sources (inputs) and data sinks (outputs). You essentially connect an input to a media object which routes the processed data to its connected output.
Sources, Sinks and Media Objects are all documented in the specification including their interfaces. With that information, it really is just about picking the building blocks you require and plugging them together.
Update 07/29/16
From my tests, it seems as if both BufferQueue and AndroidSimpleBufferQueue do not support non-PCM data, at least not on the systems I've tested (Nexus 7 # 6.01, NVidia Shield K1 # 6.0.1) so you will need to decode your data before you can use it.
I tried using the NDK versions of the MediaExtractor and MediaCodec but there are several caveats to watch out for:
MediaExtractor does not seem to correctly return the UUID information required for decoding with crypto, at least not for the files I've tested. AMediaExtractor_getPsshInfo returns a nullptr.
The API does not always behave as the comments in the header claim. Checking for EOS (end of stream) in the MediaExtractor for example seems to be most reliable by checking the amount of bytes returned instead of checking the AMediaExtractor_advance function's return value.
I'd recommend staying in Java for the decoding process as these APIs are more mature, definitely more tested and you might get more functionality out of it. Once you have the buffers of raw PCM data, you can pass it to native code which allows you to reduce latency.

I have a small company and I want all the work to be professional, I was thinking if there is a way to scan the contract (signed) and I fill the name of the customer, phone number, address and day's date in some interface then I print the contract signed with all info I entered (every info has specific location in the contract).
is this possible? what would be the best way to do it.
p.s. I'm a beginner java programmer

Well if you are a beginner java programmer I would avoid scanning an image at all costs, it in and of itself isn't too hard however what you want to do with it is; you would need to branch off first. I would suggest searching for things like IO in java and how to create an Image in java, before considering this large feat.If you feel as though you are ready here is some code which could get you on your way:
try {
Robot robo = new Robot();
}catch(Exception e){}
//Do what you would like to prepare the contract
BufferedImage contract;
contract = robo.getScreenCapture(new Rectangle(screenWidth, screenHeight));
ImageIO.write(contract, "format e.g. png", FilePathandName);
I respect your curiosity and goals though ;)

Suppose I have a sound file that I wish to play in this manner:
Start from the 0:00 mark.
When the sound file ends, replay from the 0:30.00 mark.
Essentially, I wish for the song to play from 0:00 to 0:30, then loop from 0:30 to the end.
How do you do this?
Usually, when playing Audio Files, I use AudioClip.
AudioClip audio = Applet.newAudioClip( this.getClass().getResource( "..." ) );
audio.loop();
It's because of bad experiences with loading the audio file when the project is packed into a .jar file. If the method you recommend uses another class, I would still be more than happy to listen. However, if you've encountered the same problem as I have and know how to fix it, please do tell me.

You might use a Clip for this, or if the clip is too large, feed the bytes directly to a SourceDataLine.

I'm developing applications for Android. I need to have some kind of audio mixing effect.
How can I combine 2 or more audio tracks into 1 audio track?
Or adding an audio track into a certain stream of an audio?
Can anyone give me samples where I can head start?

Maybe you could try to port this code to Android?
Mixing of multiple AudioInputStreams to one AudioInputStream.
This class takes a collection of AudioInputStreams and mixes
them together. Being a subclass of AudioInputStream itself,
reading from instances of this class behaves as if the mixdown
result of the input streams is read
.