Develop Reference

Reading MIDI messages in Java: missing note-off event?

private static List<Note> midiEventsToNotes(List<MidiEvent> midiEvents) {
List<Note> output = new ArrayList<>();
Predicate<MidiEvent> noteEvent = me -> me.getMessage().getStatus() >> REST_OF_STATUS_BYTE == NOTE_ON
|| me.getMessage().getStatus() >> REST_OF_STATUS_BYTE == NOTE_OFF;
List<MidiEvent> noteEvents = midiEvents.stream().filter(noteEvent).collect(Collectors.toList());
for (int i = 0; i < noteEvents.size(); i++) {
int eventType = noteEvents.get(i).getMessage().getStatus() >> REST_OF_STATUS_BYTE;
if (eventType == NOTE_ON) {
byte pitch = noteEvents.get(i).getMessage().getMessage()[1];
int startBeat = (int) (noteEvents.get(i).getTick() / ticksPerBeat);
for (MidiEvent ne: noteEvents) {
int pairType = ne.getMessage().getStatus() >> REST_OF_STATUS_BYTE;
byte pitch2 = ne.getMessage().getMessage()[1];
if (pairType == NOTE_OFF && pitch == pitch2) {
int value = (int) (ne.getTick() / ticksPerBeat - startBeat);
int pianoKey = pitch - MIDI_A0_VALUE;
output.add(new Note(startBeat, value, pianoKey));
break;
}
}
}
}
return output;
}
I'm trying to read a MIDI file (convert data in the file to data that is handled by my application's model). Here's the method in question. It takes a list of MidiEvent, which should just be a list of all MidiEvent in the file's sequence. The method should output a list of Note, Note being a class in the model. First the method filters the list down to only note-on and note-off events. Then, for each event, if the event is a note-on, it tries to pair it with the corresponding note-off and instantiate a Note.
I've been testing it with midi files containing only one note. As expected, the debugger tells me that there are two elements in the filtered list noteEvents, but they are somehow both note-on events (they have the same status byte), and obviously the method doesn't work because of that. Is there something wrong with the method, or is it how Java is converting the file to events, or are the midi files just bad?

The MIDI Specification says:
MIDI provides two roughly equivalent means of turning off a note (voice). A note may be turned off either by sending a Note-Off message for the same note number and channel, or by sending a Note-On message for that note and channel with a velocity value of zero. The advantage to using "Note-On at zero velocity" is that it can avoid sending additional status bytes when Running Status is employed.
Due to this efficiency, sending Note-On messages with velocity values of zero is the most commonly used method.

Java: ASCII random line file access with state

Is there a better [pre-existing optional Java 1.6] solution than creating a streaming file reader class that will meet the following criteria?
Given an ASCII file of arbitrary large size where each line is terminated by a \n
For each invocation of some method readLine() read a random line from the file
And for the life of the file handle no call to readLine() should return the same line twice
Update:
All lines must eventually be read
Context: the file's contents are created from Unix shell commands to get a directory listing of all paths contained within a given directory; there are between millions to a billion files (which yields millions to a billion lines in the target file). If there is some way to randomly distribute the paths into a file during creation time that is an acceptable solution as well.

In order to avoid reading in the whole file, which may not be possible in your case, you may want to use a RandomAccessFile instead of a standard java FileInputStream. With RandomAccessFile, you can use the seek(long position) method to skip to an arbitrary place in the file and start reading there. The code would look something like this.
RandomAccessFile raf = new RandomAccessFile("path-to-file","rw");
HashMap<Integer,String> sampledLines = new HashMap<Integer,String>();
for(int i = 0; i < numberOfRandomSamples; i++)
{
//seek to a random point in the file
raf.seek((long)(Math.random()*raf.length()));
//skip from the random location to the beginning of the next line
int nextByte = raf.read();
while(((char)nextByte) != '\n')
{
if(nextByte == -1) raf.seek(0);//wrap around to the beginning of the file if you reach the end
nextByte = raf.read();
}
//read the line into a buffer
StringBuffer lineBuffer = new StringBuffer();
nextByte = raf.read();
while(nextByte != -1 && (((char)nextByte) != '\n'))
lineBuffer.append((char)nextByte);
//ensure uniqueness
String line = lineBuffer.toString();
if(sampledLines.get(line.hashCode()) != null)
i--;
else
sampledLines.put(line.hashCode(),line);
}
Here, sampledLines should hold your randomly selected lines at the end. You may need to check that you haven't randomly skipped to the end of the file as well to avoid an error in that case.
EDIT: I made it wrap to the beginning of the file in case you reach the end. It was a pretty simple check.
EDIT 2: I made it verify uniqueness of lines by using a HashMap.

Pre-process the input file and remember the offset of each new line. Use a BitSet to keep track of used lines. If you want to save some memory, then remember the offset of every 16th line; it is still easy to jump into the file and do a sequential lookup within a block of 16 lines.

Since you can pad the lines, I would do something along those lines, and you should also note that even then, there may exist a limitation with regards to what a List can actually hold.
Using a random number each time you want to read the line and adding it to a Set would also do, however this ensures that the file is completely read:
public class VeryLargeFileReading
implements Iterator<String>, Closeable
{
private static Random RND = new Random();
// List of all indices
final List<Long> indices = new ArrayList<Long>();
final RandomAccessFile fd;
public VeryLargeFileReading(String fileName, long lineSize)
{
fd = new RandomAccessFile(fileName);
long nrLines = fd.length() / lineSize;
for (long i = 0; i < nrLines; i++)
indices.add(i * lineSize);
Collections.shuffle(indices);
}
// Iterator methods
#Override
public boolean hasNext()
{
return !indices.isEmpty();
}
#Override
public void remove()
{
// Nope
throw new IllegalStateException();
}
#Override
public String next()
{
final long offset = indices.remove(0);
fd.seek(offset);
return fd.readLine().trim();
}
#Override
public void close() throws IOException
{
fd.close();
}
}

If the number of files is truly arbitrary it seems like there could be an associated issue with tracking processed files in terms of memory usage (or IO time if tracking in files instead of a list or set). Solutions that keep a growing list of selected lines also run in to timing-related issues.
I'd consider something along the lines of the following:
Create n "bucket" files. n could be determined based on something that takes in to account the number of files and system memory. (If n is large, you could generate a subset of n to keep open file handles down.)
Each file's name is hashed, and goes into an appropriate bucket file, "sharding" the directory based on arbitrary criteria.
Read in the bucket file contents (just filenames) and process as-is (randomness provided by hashing mechanism), or pick rnd(n) and remove as you go, providing a bit more randomosity.
Alternatively, you could pad and use the random access idea, removing indices/offsets from a list as they're picked.

Creating a large number of multiple outputs

I am creating a large number of output files, for example 500. I am getting already being created exception,as shoen below. The program recovers by itself when the number of output files is small. For ex. if its 50 files, though this exception occurs, the program starts running successfully after printing this exception several times.
But, for many files, it eventually fails with an IOException.
I have pasted the error and then the code below:
12/10/29 15:47:27 INFO mapred.JobClient: Task Id : attempt_201210231820_0235_r_000004_3, Status : FAILED
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /home/users/mlakshm/preopa406/data-r-00004 for DFSClient_attempt_201210231820_0235_r_000004_3 on client 10.0.1.100, because this file is already being created by DFSClient_attempt_201210231820_0235_r_000004_2 on 10.0.1.130
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1406)
I have pasted the code :
In the Reduce method, I have the below logic to generate ouputs:
int data_hash = (int)data_str.hashCode();
int data_int1 = 0;
int k = 500;
int check1 = 0;
for (int l = 10; l>0; l++)
{
if((data_hash%l==0)&&(check1 == 0))
{
check1 = 1;
int range = (int) k/10;
String check = "true";
while(range > 0 && check.equals("true"))
{
if(data_hash % range-1 == 0)
{
check = "false";
data_int1 = range*10;
}
}
}
}
mos.getCollector("/home/users/mlakshm/preopa407/cdata"+data_int1, reporter).collect(new Text(t+" "+alsort.get(0)+" "+alsort.get(1)), new Text(intersection));
PLs help!

The problem is that all the reducer are trying to write files with the same naming scheme.
The reason it's doing this because
mos.getCollector("/home/users/mlakshm/preopa407/cdata"+data_int1, reporter).collect(new Text(t+" "+alsort.get(0)+" "+alsort.get(1)), new Text(intersection));
Set's the file name based on a characteristic of the data not the identity of the reducer.
You have a couple of choices :
Rework your map job so so that the key that's emitted matches up with the hash that your calculating in this job. That would make sure that each reducer got a span of values.
Include in the file name a identifier that is unqiue to each mapper. This would leave you with a set of part files for each reducer.
Could you perhaps explain why your using multiple outputs here? I don't think you need to.

Determining Latency in Audio Processing

I've been working on implementing a system for real-time audio capture and analysis within an existing music software project. The goal of this system is to begin capturing audio when the user presses the record button (or after a specified count-in period), determine the notes the user sings or plays, and notate these notes on a musical staff. The gist of my method is to use one thread to capture chunks of audio data and put them into a queue, and another thread to remove the data from the queue and perform the analysis.
This scheme works well, but I am having trouble quantifying the latency between the onset of audio capture and playback of the MIDI backing instruments. Audio capture begins before the MIDI instruments begin playing back, and the user is presumably going to be synchronizing his or her performance with the MIDI instruments. Therefore, I need to ignore audio data captured before the backing MIDI instruments begin playing and only analyze audio data collected after that point.
Playback of the backing tracks is handled by a body of code that has been in place for quite a while and maintained by someone else, so I would like to avoid refactoring the whole program if possible. Audio capture is controlled with a Timer object and a class that extends TimerTask, instances of which are created in a lumbering (~25k lines) class called Notate. Notate also keeps tabs on the objects that handle playback of the backing tracks, by the way. The Timer’s .scheduleAtFixedRate() method is used to control periods of audio capture, and the TimerTask notifies the capture thread to begin by calling .notify() on the queue (ArrayBlockingQueue).
My strategy for calculating the time gap between the initialization of these two processes has been to subtract the timestamp taken just before capture begins (in milliseconds) from the timestamp taken at the moment playback begins, which I'm defining this as when the .start() method is called on the Java Sequencer object that is in charge of the MIDI backing tracks. I then use the result to determine the number of audio samples that I expect to have been captured during this interval (n) and ignore the first n * 2 bytes in the array of captured audio data (n * 2 because I am capturing 16-bit samples, whereas the data is stored as a byte array… 2 bytes per sample).
However, this method is not giving me accurate results. The calculated offset is always less than I expect it to be, such that there remains a non-trivial (and unfortunately varied) amount of “empty” space in the audio data after beginning analysis at the designated position. This causes the program to attempt to analyze audio data collected when the user had not yet begun to play along with the backing MIDI instruments, effectively adding rests - the absence of musical notes - at the begging of the user’s musical passage and ruining the rhythm values calculated for all subsequent notes.
Below is the code for my audio capture thread, which also determines the latency and corresponding position offset for the array of captured audio data. Can anyone offer insight into why my method for determining latency is not working correctly?
public class CaptureThread extends Thread
{
public void run()
{
//number of bytes to capture before putting data in the queue.
//determined via the sample rate, tempo, and # of "beats" in 1 "measure"
int bytesToCapture = (int) ((SAMPLE_RATE * 2.) / (score.getTempo()
/ score.getMetre()[0] / 60.));
//temporary buffer - will be added to ByteArrayOutputStream upon filling.
byte tempBuffer[] = new byte[target.getBufferSize() / 5];
int limit = (int) (bytesToCapture / tempBuffer.length);
ByteArrayOutputStream outputStream = new ByteArrayOutputStream(bytesToCapture);
int bytesRead;
try
{ //Loop until stopCapture is set.
while (!stopCapture)
{ //first, wait for notification from TimerTask
synchronized (thisCapture)
{
thisCapture.wait();
}
if (!processingStarted)
{ //the time at which audio capture begins
startTime = System.currentTimeMillis();
}
//start the TargetDataLine, from which audio data is read
target.start();
//collect 1 captureInterval's worth of data
for (int n = 0; n < limit; n++)
{
bytesRead = target.read(tempBuffer, 0, tempBuffer.length);
if (bytesRead > 0)
{ //Append data to output stream.
outputStream.write(tempBuffer, 0, bytesRead);
}
}
if (!processingStarted)
{
long difference = (midiSynth.getPlaybackStartTime()
+ score.getCountInTime() * 1000 - startTime);
positionOffset = (int) ((difference / 1000.)
* SAMPLE_RATE * 2.);
if (positionOffset % 2 != 0)
{ //1 sample = 2 bytes, so positionOffset must be even
positionOffset += 1;
}
}
if (outputStream.size() > 0)
{ //package data collected in the output stream into a byte array
byte[] capturedAudioData = outputStream.toByteArray();
//add captured data to the queue for processing
processingQueue.add(capturedAudioData);
synchronized (processingQueue)
{
try
{ //notify the analysis thread that data is in the queue
processingQueue.notify();
} catch (Exception e)
{
//handle the error
}
}
outputStream.reset(); //reset the output stream
}
}
} catch (Exception e)
{
//handle error
}
}
}
I am looking into using a Mixer object to synchronize the TargetDataLine which is accepting data from the microphone and the Line that handles playback from the MIDI instruments. Now to find the Line that handles playback... Any ideas?

Google has a good open source app called AudioBufferSize that you are probably familiar with. I modified this app the test one way latency- that is to say, the time between when a user presses a button and the sound is played by the Audio API. Here is the code I added to AudioBufferSize to achieve this. Could you use such an approach to provide the timing delta between the event and when the user perceives it?
final Button latencyButton = (Button) findViewById(R.id.latencyButton);
latencyButton.setOnClickListener(new OnClickListener() {
public void onClick(View v) {
mLatencyStartTime = getCurrentTime();
latencyButton.setEnabled(false);
// Do the latency calculation, play a 440 hz sound for 250 msec
AudioTrack sound = generateTone(440, 250);
sound.setNotificationMarkerPosition(count /2); // Listen for the end of the sample
sound.setPlaybackPositionUpdateListener(new OnPlaybackPositionUpdateListener() {
public void onPeriodicNotification(AudioTrack sound) { }
public void onMarkerReached(AudioTrack sound) {
// The sound has finished playing, so record the time
mLatencyStopTime = getCurrentTime();
diff = mLatencyStopTime - mLatencyStartTime;
// Update the latency result
TextView lat = (TextView)findViewById(R.id.latency);
lat.setText(diff + " ms");
latencyButton.setEnabled(true);
logUI("Latency test result= " + diff + " ms");
}
});
sound.play();
}
});
There is a reference to generateTone which looks likes this:
private AudioTrack generateTone(double freqHz, int durationMs) {
int count = (int)(44100.0 * 2.0 * (durationMs / 1000.0)) & ~1;
short[] samples = new short[count];
for(int i = 0; i < count; i += 2){
short sample = (short)(Math.sin(2 * Math.PI * i / (44100.0 / freqHz)) * 0x7FFF);
samples[i + 0] = sample;
samples[i + 1] = sample;
}
AudioTrack track = new AudioTrack(AudioManager.STREAM_MUSIC, 44100,
AudioFormat.CHANNEL_OUT_STEREO, AudioFormat.ENCODING_PCM_16BIT,
count * (Short.SIZE / 8), AudioTrack.MODE_STATIC);
track.write(samples, 0, count);
return track;
}
Just realized, this question is multi-years old. Sorry, maybe someone will find it useful.

Strange FileInputStream/DataFileInputStream behaviour: seek()ing to odd positions

The good:
so, I have this binary data file (size - exactly 640631 bytes), and I'm trying to make Java read it.
I have two interchangeable classes implemented as layers for reading that data. One of them uses RandomAccessFile, which works great and all.
The bad:
Another one (the one this question is mostly about) tries to use FileInputStream and DataInputStream so that the very same data could be (at least theoretically) be read on MIDP 2.0 (CLDC 1.1) Java configuration (which doesn't have RandomAccessFile).
In that class, I open the data file like this:
FileInputStream res = new FileInputStream(new File(filename));
h = new DataInputStream(res);
...and implement seek()/skip() like this (position is a long that takes note of a current position in a file):
public void seek(long pos) throws java.io.IOException {
if (! this.isOpen()) {
throw new java.io.IOException("No file is open");
}
if (pos < position) {
// Seek to the start, then skip some bytes
this.reset();
this.skip(pos);
} else if (pos > position) {
// skip the remaining bytes until the position
this.skip(pos - position);
}
}
and
public void skip(long bytes) throws java.io.IOException {
if (! this.isOpen()) {
throw new java.io.IOException("No file is open");
}
long skipped = 0, step = 0;
do {
step = h.skipBytes((int)(bytes - skipped));
if (step < 0) {
throw new java.io.IOException("skip() failed");
}
skipped += step;
} while (skipped < bytes);
position += bytes;
}
The ugly:
The problem with the second class (the FileInputStream/DataInputStream one) is that sometimes it decides to reset the file position to some strange place in a file :) This happens both when I run this on J2SE (a computer) and J2ME (a mobile phone). Here's an example of the actual usage of that reader class and a bug that occurs:
// Open the data file
Reader r = new Reader(filename);
// r.position = 0, actual position in a file = 0
// Skip to where the data block that is needed starts
// (determined by some other code)
r.seek(189248);
// r.position = 189248, actual position in a file = 189248
// Do some reading...
r.readID(); r.readName(); r.readSurname();
// r.position = 189332, actual position in a file = 189332
// Skip some bytes (an unneeded record)
r.skip(288);
// r.position = 189620, actual position in a file = 189620
// Do some more reading...
r.readID(); r.readName(); r.readSurname();
// r.position = 189673, actual position in a file = 189673
// Skip some bytes (an unneeded record)
r.skip(37);
// AAAAND HERE WE GO:
// r.position = 189710, actual position in a file = 477
I was able to determine that when asked to skip another 37 bytes, Java positioned the file pointer to byte 477 from the very start or file instead.
"Freshly" (just after opening a file) seeking to a position 189710 (and beyond that) works OK. However, reopening a file every time I need a seek() is just painfully slow, especially on a mobile phone.
What has happened?

I can see nothing wrong with this. Are you positive of the r.position value before the last skip? Unless there's an underlying bug in the JDK streams or if you have multiple multiple threads using the Reader, then the only possibility I can guess at is that something is modifying the position value incorrectly when you read your fields.