I use the FFT function from the Libgdx library for a project in Android, where I process the accelerometer signal for create a signal spectrum.
I need to normalize the output from accelerometer data, i read there isn't a "correct" way to do this but is conventional. Someone use dividing by 1/N in FFT, other by 1/sqrt(N).
I didn't understand if this is conventional for who implements the library, this mean that every library have his normalization factor, or is conventional for the user than I can decide for aesthetic representation.
If it depends on library, which is the normalization factor for FFT in LIBGDX library?
Edit1: I searched already inside documentation but I found nothing. Here is it: http://libgdx-android.com/docs/api/com/badlogic/gdx/audio/analysis/FFT.html
I was about to say "just check the documentation", but it turns out that it's terrible, and doesn't say one way or the other!
Still, you could determine the scale factor empirically. Just run an FFT on all-ones dataset. There will be one non-zero bin in the output. There are three likely values of this bin:
1.0: The scale was 1/N
sqrt(N): The scale was 1/sqrt(N)
N: The scale was 1
You can do the same trick for the inverse FFT, although it's redundant. The forward and inverse scale factors must multiply to 1/N.
There's a specific normalization depending on if you want the spectrum or power spectral density. Oli provided a good test for determining the 1/N, 1/sqrt(N) or no scaling that the library performs.
Here's a document that explains everything in great detail along with a comprehensive comparison of window functions.
http://edoc.mpg.de/395068
Related
This is supposed to be for an android app, so the language in question is obviously Java.
I'm trying to record some audio and get the dominant frequency. This is for a very specific purpose, and the frequencies I need to be detected are pure sounds made by another device. I have the recording part done, so the only thing that I need to do is calculate the frequency from the buffer it generates.
I know I'm supposed to use something called FFT, so I put these into my project: http://introcs.cs.princeton.edu/java/97data/FFT.java, and http://introcs.cs.princeton.edu/java/97data/Complex.java.html
I know there are many questions about this, but none of them give an answer that I can understand. Others have broken links.
Anyone know how to do this, and explain in a relatively simple manner?
Generally a DFT (FFT included) implementation will take N time-domain samples (your recording) and produce N/2 complex values in the frequency domain. The angle of the complex value represents the phase and the absolute value of it represents the amplitude. Usually the values output will be ordered from lowest frequency to highest frequency.
Some implementations may output N complex values, but the extra values are redundant unless your input contains complex values. It should not in your case. This is why many implementations input real values and output N/2 complex values, as this is the most common use of FFT.
So, you will want to calculate the absolute value of the output since the amplitude is what you are interested in. The absolute value of a complex number is the square root of the sum of the square of it's real and the square of it's complex component.
The exact frequencies of each value will depend on the number of samples of input and the interval between the samples. The frequency of value at position i (assuming i goes from 0 to N/2 - 1) will be i * (sampling frequency) / N.
This is assuming your N is even, rather than trying to explain the case of N being odd I'll recommend you keep N even for simplicity. For the case of FFT N will always be a power of two so N will always be even anyway.
If you're looking for a tone over a minimum time T then I'd also recommend processing the input in blocks of T/2 size.
Fourier transforms are a mathematical technique that lets you go back and forth between time and frequency domains for time-dependent signals.
FFT is a computer algorithm for calculating discrete transforms quickly and efficiently.
You'll take a sample of your time signal and apply FFT to it to get the amplitude versus frequency for the sample.
It's not an easy topic if you don't have the mathematical background. It assumes a good knowledge of trigonometry (sines and cosines), functions, and calculus. If you don't have that, it'll be difficult to read and understand any reference you can find.
If you don't have that background, do your best to treat a library FFT function as a black box and use what it gives back.
What I am trying to achieve is distinguishing separate notes in an audio file. For simplicity lets say that a couple of notes are played one after another. The main question is how to determine when the next note is played ?
What I have already done is reading samples from an audio file and doing Fourier Transform on those samples using JTransforms library. Here's what I get:
.
Then I have calculated spectrum based on the data that FFT returned, and here's what I get:
.
As I understand the bigger "columns" on the chart are the harmonics, and the small ones are noise and other non-harmonic overtones, right ?
After that I have tried to do same process with audio file with two notes played one after another, but the result was kind of the same.
As a side question, do any of you know some lightweight and fast libraries for visualizing such data ? Because using JFreeChart for bigger datasets is a real pain for my processor.
To detect successive frequency bursts of different frequencies, and some of their time domain parameters, one can use overlapping short FFT windows (length shorter than the expected burst length) and look for where the relative magnitudes of frequency peaks swap order, or fall above/below thresholds. If you know, a priori, the frequencies involved, you can use the Goertzel filters instead of FFTs, with sliding windows or successive approximation in time for finer time domain granularity.
For pitched notes (such as music), one can do something similar, except using a pitch detection/estimation method (instead of simple FFT magnitudes, which are not reliable) on sufficiently short time domain windows of data.
In particular, I want to generate a tolerance interval, for which I would need to have the values of Zx for x some value on the standard normal.
Does the Java standard library have anything like this, or should I roll my own?
EDIT: Specifically, I'm looking to do something akin to linear regression on a set of images. I have two images, and I want to see what the degree of correlation is between their pixels. I suppose this might fall under computer vision as well.
Simply calculate Pearson correlation coefficient between those two images.
You will have 3 coefficients because of R,G,B channels needs to be analyzed separately.
Or you can calculate 1 coefficient just for intensity levels of images,... or you could calculate correlation between Hue values of images after converting to HSV or HSL color space.
Do whatever your see fits :-)
EDIT: Correlation coefficient may be maximized only after scaling and/or rotating some image. This may be a problem or not - depends on your needs.
You can use the complete statistical power of R using rJava/JRI. This includes correlations between pixels and so on.
Another option is to look around at imageJ, which contains libraries for many image manipulations, mathematics and statistics. It's an application allright, but the library is useable in development as well. It comes with an extensive developers manual. On a sidenote, imageJ can be combined with R as well.
imageJ allows you to use the correct methods for finding image similarity measures, based on fourier transformations or other methods. More info can be found in Digital Image Processing with Java an ImageJ. See also this paper.
Another one is the Commons-Math. This one also contains the basic statistical tools.
See also the answers on this question and this question.
It seems you want to compare to images to see how similar they are. In this case, the first two things to try are SSD (sum of squared differences) and normalized correlation (this is closely related to what 0x69 suggests, Pearson correlation) between the two images.
You can also try normalized correlation over small (corresponding) windows in the two images and add up the results over several (all) small windows in the image.
These two are very simple methods which you can write in a few minutes.
I'm not sure however what this has to do with hypothesis testing or linear regression, you might want to edit to clarify this part of your question.
The Dell Streak has been discovered to have an FM radio which has very crude controls. 'Scanning' is unavailable by default, so my question is does anyone know how, using Java on Android, one might 'listen' to the FM radio as we iterate up through the frequency range detecting white noise (or a good signal) so as to act much like a normal radio's seek function?
I have done some practical work on this specific area, i would recommend (if you have a little time for it) to try just a little experimentation before resorting to fft'ing. The pcm stream can be interpreted very complexely and subtly (as per high quality filtering and resampling) but can also be practically treated for many purposes as the path of a wiggly line.
White noise is unpredictable shaking of the line, which is never-the-less quite continuous in intensity (rms, absolute mean..) Acoustic content is recurrent wiggling and occasional surprises (jumps, leaps) :]
Non-noise like content of a signal may be estimated by performing quick calculations on a running window of the pcm stream.
For example, noise will strongly tend to have a higher value for the absolute integral of its derivative, than non-noise. I think that is the academic way of saying this:
loop(n+1 to n.length)
{ sumd0+= abs(pcm[n]);
sumd1+= abs(pcm[n]-pcm[n-1]);
}
wNoiseRatio = ?0.8; //quite easily discovered, bit tricky to calculate.
if((sumd1/sumd0)<wNoiseRatio)
{ /*not like noise*/ }
Also, the running absolute average over ~16 to ~30 samples of white noise will tend to vary less, over white noise than acoustic signal:
loop(n+24 to n.length-16)
{ runAbsAve1 += abs(pcm[n]) - abs(pcm[n-24]); }
loop(n+24+16 to n.length)
{ runAbsAve2 += abs(pcm[n]) - abs(pcm[n-24]); }
unusualDif= 5; //a factor. tighter values for longer measures.
if(abs(runAbsAve1-runAbsAve2)>(runAbsAve1+runAbsAve2)/(2*unusualDif))
{ /*not like noise*/ }
This concerns how white noise tends to be non-sporadic over large enough span to average out its entropy. Acoustic content is sporadic (localised power) and recurrent (repetitive power).
The simple test reacts to acoustic content with lower frequencies and could be drowned out by high frequency content. There are simple to apply lowpass filters which could help (and no doubt other adaptions).
Also, the root mean square can be divided by the mean absolute sum providing another ratio which should be particular to white noise, though i cant figure what it is right now. The ratio will also differ for the signals derivatives as well.
I think of these as being simple formulaic signatures of noise. I'm sure there are more..
Sorry to not be more specific, it is fuzzy and imprecise advice, but so is performing simple tests on the output of an fft. For better explaination and more ideas perhaps check out statistical and stochastic(?) measurements of entropy and randomness on wikipedia etc.
Use a Fast Fourier Transform.
This is what you can use a Fast Fourier Transform for. It analyzes the signal and determines the strength of the signal at various frequencies. If there's a spike in the FFT curve at all, it should indicate that the signal is not simply white noise.
Here is a library which supports FFT's. Also, here is a blog with source code in case you want to learn about what the FFT does.
If you don't have FFT tools available, just a wild suggestion:
Try to compress a few milliseconds of audio.
A typical feature of noise is that it compresses much less than clear signal.
As far as I know there is no API or even drivers for the FM Radio in the Android SDK and unless Dell releases one you will have to roll your own. It's actually even worse than that. All(?) new chipsets has FM Radio but not all phones has an FM Radio application.
The old Windows Mobile had the same problem.
For white noise detection you need to do FFT and see that it has more or less continious spectrum. But recording from FM might be a problem.
Just high pass filtering it will give a good idea, and has sometimes been used for squelch on fm radios.
Note that this is comparable to what the derivative suggestion was getting at - taking the derivative is a simple form of high pass filter, and taking the absolute value of that a crude way of measuring power.
Do you have a subscription to the IEEE Xplore library? There are countless papers (one picked at random) on this very topic.
A very simplistic method would be to observe the "flatness" of the power spectral density. One could take this by using a Fast Fourier Transform of the signal in the time domain and find the standard deviation of the spectral density. If it is below some threshold, you have your white noise.
The main question here is: what type of signal do you have access to?
I bet you don't have direct access to the analog EM signal directly. So no use of FFT on this signal possible. You can't also try to build a phased-lock loop, which is the way your standard old radio tuner works ("Scanning" in your case).
Your only option is indeed to pick one frequency and listen too it (and try do detect when it's noise with FFT on sound). You might even only have access to the FFTed signal.
Problem here: If you want to detect a potential frequency using white noise you will pick up signals too easily.
Anyway, here is what I would try to do with this strategy:
Double integrate the autocorrelation of the spectral density over a fraction of a second of audio. And this for each frequency.
Then look for a FM frequency where this number is maxed.
Little explanation here:
Spectral density gives you a signal which most used frequencies are maxed.
If a bit of time later if the same frequencies are maxed then you have some supposedly clear audio. You get this by integrating the autocorrelation the spectral density for one audio frequency for a fraction of a second (using some function that grows larger than linear might also work)
You then just have to integrate this for all audio frequencies
Also be careful to normalize the integrals: a loud white noise signal should not get a higher score than a clear but low audio signal.
Several people have mentioned the FFT, which you'll want to do, but to then detect white noise you need to make sure that the magnitude is relatively constant over the range of audio frequencies. You'll want to look at magnitudes only, you can throw away the phases. You can compute an average and standard deviation for the magnitudes in O(N) time. For white noise, you should find the standard deviation to be a relatively small fraction of the average. If I remember my statistics right, it should be about (1/sqrt(N)) of the average.
I have a bunch of sets of data (between 50 to 500 points, each of which can take a positive integral value) and need to determine which distribution best describes them. I have done this manually for several of them, but need to automate this going forward.
Some of the sets are completely modal (every datum has the value of 15), some are strongly modal or bimodal, some are bell-curves (often skewed and with differing degrees of kertosis/pointiness), some are roughly flat, and there are any number of other possible distributions (possion, power-law, etc.). I need a way to determine which distribution best describes the data and (ideally) also provides me with a fitness metric so that I know how confident I am in the analysis.
Existing open-source libraries would be ideal, followed by well documented algorithms that I can implement myself.
Looking for a distribution that fits is unlikely to give you good results in the absence of some a priori knowledge. You may find a distribution that coincidentally is a good fit but is unlikely to be the underlying distribution.
Do you have any metadata available that would hint at what the data means? E.g., "this is open-ended data sampled from a natural population, so it's some sort of normal distribution", vs. "this data is inherently bounded at 0 and discrete, so check for the best-fitting Poisson".
I don't know of any distribution solvers for Java off the top of my head, and I don't know of any that will guess which distribution to use. You could examine some statistical properties (skew/etc.) and make some guesses here--but you're more likely to end up with an accidentally good fit which does not adequately represent the underlying distribution. Real data is noisy and there are just too many degrees of freedom if you don't even know what distribution it is.
This may be above and beyond what you want to do, but it seems the most complete approach (and it allows access to the wealth of statistical knowledge available inside R):
use JRI to communicate with the R statistical language
use R, internally, as indicated in this thread
Look at Apache commons-math.
What you're looking for comes under the general heading of "goodness of fit." You could search on "goodness of fit test."
Donald Knuth describes a couple popular goodness of fit tests in Seminumerical Algorithms: the chi-squared test and the Kolmogorov-Smirnov test. But you've got to have some idea first what distribution you want to test. For example, if you have bell curve data, you might try normal or Cauchy distributions.
If all you really need the distribution for is to model the data you have sampled, you can make your own distribution based on the data you have:
1. Create a histogram of your sample: One method for selecting the bin size is here. There are other methods for selecting bin size, which you may prefer.
2. Derive the sample CDF: Think of the histogram as your PDF, and just compute the integral. It's probably best to scale the height of the bins so that the CDF has the right characteristics ... namely that the value of the CDF at +Infinity is 1.0.
To use the distribution for modeling purposes:
3. Draw X from your distribution: Make a draw Y from U(0,1). Use a reverse lookup on your CDF of the value Y to determine the X such that CDF(X) = Y. Since the CDF is invertible, X is unique.
I've heard of a package called Eureqa that might fill the bill nicely. I've only downloaded it; I haven't tried it myself yet.
You can proceed with a three steps approach, using the SSJ library:
Fit each distribution separately using maximum likelihood estimation (MLE). Using SSJ, this can be done with the static method getInstanceFromMLE(double[] x,
int n) available on each distribution.
For each distribution you have obtained, compute its goodness-of-fit with the real data, for example using Kolmogorov-Smirnov: static void kolmogorovSmirnov (double[] data, ContinuousDistribution dist, double[] sval,double[] pval), note that you don't need to sort the data before calling this function.
Pick the distribution having the highest p-value as your best fit distribution