Does anyone know of a library with a working implementation of backpropagation through time?
Any of Java/Python/C#/VB.NET/F# (preferably the last one) will do!
Assuming you're already using some library for BP, it should be (TM) rather straightforward to implement BPTT using BP as a step in the process.
The Wikipedia entry for BPTT [1] includes relevant pseudo code.
My own starting point, about 18 years ago, was "The Truck Backer-Upper: An Example of Self-Learning in Neural Networks" [2].
[1] http://en.wikipedia.org/wiki/Backpropagation_through_time
[2] http://www-isl.stanford.edu/~widrow/papers/c1989thetruck.pdf
I've used NeuronDotNet only for a limited time though. It allows you to create a feed-forward BackPropagation NN. I especially liked their use of intuitively named classes. Good luck!
This is a .net library.
I'm from a Java background but Encog has a .net implementation as well (and is a seriously good framework for NNets, with good time series support)
Can't help with an F# framework, but what domain are you coding for? If it's finance I'll reassert the "take a look at Encog"
Perhaps pybrain would do? The docstring for its BackpropTrainer class suggests that it does backpropagation through time:
class BackpropTrainer(Trainer):
"""Trainer that trains the parameters of a module according to a
supervised dataset (potentially sequential) by backpropagating the errors
(through time)."""
What about this one ? Just a Google search to help...
I've had good experiences with Weka - In my view one of the best and almost certainly the most comprehensive general purpose machine learning libraries around.
You could certainly do BPTT with Weka - you may find a ready made classifier that does what you need but even if not you can just chain a few normal backpropagation units together as per the very good wikipedia article on BPTT
I made backpropagation algorithm in Java quite time ago. I uploaded it into GitHub, maybe you can find it useful: https://github.com/bernii/NeuralNetwokPerceptronKohonen
Let me now if it was helpful :)
You can use TensorFlow's dynamic_rnn() function (API doc). TensorFlow's tutorial on Recurrent Neural Networks will help.
Also, this great blog post provides a nice introduction to predicting sequences using TensorFlow. Here's another blog post with some code to predict a time series.
Related
What algorithm does Named Entity Recognition (NER) use? I mean how does it match and tag all the entities?
Basically, There are lot of algorithm Named Entity Recognition but the most famous are
Conditional Random Field(CRF) Algorithm is used.
Stanford CoreNLP for Java
Spacy
NLTK
Another most effective one is using Deep Learning like Recurrent Neural Network (LSTM).
http://nlp.town/blog/ner-and-the-road-to-deep-learning/
https://towardsdatascience.com/deep-learning-for-ner-1-public-datasets-and-annotation-
methods-8b1ad5e98caf
there are lot other articles and paper out there
Hope this will help :)
NER can be performed by different algorithms, from simple string matching using grep: http://labs.fc.ul.pt/mer/ to advanced machine learning techniques: https://nlp.stanford.edu/software/CRF-NER.shtml
Basically, It Depends on the approach that you are following.
For Name Entity Recognition There are n no.of approaches available. Most typically Conditional Random Field(CRF) Algorithm is used. Another most effective one is using Deep Learning like Recurrent Neural Network (LSTM). There might be other algorithms as well.
Hope this will help :)
If you want to play around with a nice, pre-built Bidirectional CRF-LSTM (currently one of the better performing models) in Keras, I would recommend the python package Anango as a very quick way to play around with an NER system.
Although this initial question is tagged with Java. It is important to point out that you are going to find all the latest and greatest algorithms for this type of Machine Learning implemented in Python.
Spacy (a Python package) is also a very good way to get started although I found it took a little more time to set something up quickly. Also, I'm not sure how easy it would be to alter their NER algorithm that is pre-built. I believe it uses CNNs.
A good overview of where things are at with NER:
A Survey on Recent Advances in Named Entity Recognition from Deep
Learning models
in my project, I am using the library called LIUM_SpkDiarization-4.7.jar, but I am not quite sure how does it work. Could anyone, please, explain it a bit?
Also, I'm using it with python.
The link to the library is: https://voiceid.googlecode.com/svn-history/r11/trunk/scripts/LIUM_SpkDiarization-4.7.jar
Thanks in advance.
I was not aware of this tool. It looks really cool. Have you checked their wiki? They have some papers on how the system works: http://lium3.univ-lemans.fr/diarization/doku.php
Basically, they compute MFCC Mel Frequency Cepstrum Coefficients (standard technique). This is the fundamental step. It generates a feature space to work with. It is similar to computing FFT over sliding window in time. Ultimately clustering is performed on these time sliced features using Bayesian Information Criterion (BIC) methods. First to segment the time based feature space, then to cluster, and find consistent features for each speaker. HMM, viterbi, EM, and sometimes GMM can be used as well.
I don't know the algorithm well enough to explain it in detail, but this should also help: http://lium3.univ-lemans.fr/diarization/doku.php/overview
I have heard of this algorithm, but is this algorithm good to use it with Bayesian Belief networks? Hugin is based on it and I'm looking for a book / article on this algorithm.
The algorithm is described in this paper. It is quite detailed and should be a good point to start.
I haven't kept track of this research area for a while, but I can point you towards the
CiteSeerX search engine if you don't know it already. (http://citeseerx.ist.psu.edu/)
Searching for papers which cite Shenoy & Shafer's An axiomatic framework for Bayesian and belief function propagation (1990) will give you a list of other researchers who have tried to apply the algorithm.
I am not familiar with the algorithm but another place to check for information would be
a search in google scholar.
Pulcinella is a tool for Propagating Uncertainty through Local Computations based on the general framework af valuation systems proposed by Shenoy and Shafer
Pulcinella is freely available for
educational and strictly
non-commercial use. Pulcinella is
written in Common Lisp. It has been
tested on Allegro CL on Macintosh, and
on Lucid CL, Allegro CL, and CLisp on
a Sun. The code is just "pure" common
lisp, so it should also run on any
other reasonable implementation of
common-lisp (well, you know...). To
get the latest version, click here.
Alternatively, you can get Pulcinella
by anonymous ftp from
ftp://aass.oru.se/pub/saffiotti. The
Pulcinella tar archive includes a few
examples, taken from the User's
Manual. If you fetch this program, you
are expected to send a letter at the
address below, stating that you will
use Pulcinella for research and
non-commercial use only.
Also here is some references.
Even More references:
An Algorithm for Bayesian Belief Network Construction from Data
A Tutorial on Learning With Bayesian Networks
http://en.wikipedia.org/wiki/Bayesian_network#External_links
Is there any open source implementation of LSI in Java? I want to use that library for my project. I have seen jLSI but it implements some other model of LSI. I want a standard model.
Have you considered LDA (Latent Dirichlet allocation)? I haven't really either, but I encountered the same problem with LSI recently (patents). From what I understand LDA is a related/more powerful technique. http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation apparently has some links to open-source implementations.
A google search for java LSI leads to a similar question that recommends SemanticVectors. A package built on top of Lucene that is 'similar' to LSI. I don't know if it's closer than the jLSI implementation.
That thread also mentions that LSI is patented and there aren't a lot of implementations of it. So if you need a standard implementation you may have to use a language other than java.
The S-Space Package has an open source version of LSA, with bindings for the LSI document vectors. (Both approaches operate on the same term-document matrix and are equivalent except in the output.) It's a fairly scalable approach that uses the thin-SVD. I've used it to run LSI on all of Wikipedia with no issue (after removing the infrequent terms with less than 5 occurrences).
As Scott Ray mentioned, the SemanticVectors package also has a good LSI implementation that recently switched to using the same thin-SVD (SVDLIBJ), so you might check that out as if you hadn't before.
a google search for NLP tools provide this slides which i think helps ...
I believe that LSA/LSI was patented in 1989, which means the patent should have just expired. Hopefully we will see some nice open source applications soon.
Have you tried the Semantic Vector package?
http://code.google.com/p/semanticvectors/
I'm planning to develop program in Java which will provide diagnosis. The data set is divided into two parts one for training and the other for testing. My program should learn to classify from the training data (BTW which contain answer for 30 questions each in new column, each record in new line the last column will be diagnosis 0 or 1, in the testing part of data diagnosis column will be empty - data set contain about 1000 records) and then make predictions in testing part of data :/
I've never done anything similar so I'll appreciate any advice or information about solution to similar problem.
I was thinking about Java Machine Learning Library or Java Data Mining Package but I'm not sure if it's right direction... ? and I'm still not sure how to tackle this challenge...
Please advise.
All the best!
I strongly recommend you use Weka for your task
Its a collection of machine learning algorithms with a user friendly front-end which facilitates a lot of different kinds of feature and model selection strategies
You can do a lot of really complicated stuff using this without really having to do any coding or math
The makers have also published a pretty good textbook that explains the practical aspects of data mining
Once you get the hang of it, you could use its API to integrate any of its classifiers into your own java programs
Hi As Gann Bierner said, this is a classification problem. The best classification algorithm for your needs I know of is, Ross Quinlan algorithm. It's conceptually very easy to understand.
For off-the-shelf implementations of the classification algorithms, the best bet is Weka. http://www.cs.waikato.ac.nz/ml/weka/. I have studied Weka but not used, as I discovered it a little too late.
I used a much simpler implementation called JadTi. It works pretty good for smaller data sets such as yours. I have used it quite a bit, so can confidently tell so. JadTi can be found at:
http://www.run.montefiore.ulg.ac.be/~francois/software/jaDTi/
Having said all that, your challenge will be building a usable interface over web. To do so, the dataset will be of limited use. The data set basically works on the premise that you have the training set already, and you feed the new test dataset in one step, and you get the answer(s) immediately.
But my application, probably yours also, was a step by step user discovery, with features to go back and forth on the decision tree nodes.
To build such an application, I created a PMML document from my training set, and built a Java Engine that traverses each node of the tree asking the user to give an input (text/radio/list) and use the values as inputs to the next possible node predicate.
The PMML standard can be found here: http://www.dmg.org/ Here you need the TreeModel only. NetBeans XML Plugin is a good schema-aware editor for PMML authoring. Altova XML can do a better job, but costs $$.
It is also possible to use an RDBMS to store your dataset and create the PMML automagically! I have not tried that.
Good luck with your project, please feel free to let me know if you need further inputs.
There are various algorithms that fall into the category of "machine learning", and which is right for your situation depends on the type of data you're dealing with.
If your data essentially consists of mappings of a set of questions to a set of diagnoses each of which can be yes/no, then I think methods that could potentially work include neural networks and methods for automatically building a decision tree based on the test data.
I'd have a look at some of the standard texts such as Russel & Norvig ("Artificial Intelligence: A Modern Approach") and other introductions to AI/machine learning and see if you can easily adapt the algorithms they mention to your particular data. See also O'Reilly, "Programming Collective Intelligence" for some sample Python code of one or two algorithms that might be adaptable to your case.
If you can read Spanish, the Mexican publishing house Alfaomega have also published various good AI-related introductions in recent years.
This is a classification problem, not really data mining. The general approach is to extract features from each data instance and let the classification algorithm learn a model from the features and the outcome (which for you is 0 or 1). Presumably each of your 30 questions would be its own feature.
There are many classification techniques you can use. Support vector machines is popular as is maximum entropy. I haven't used the Java Machine Learning library, but at a glance I don't see either of these. The OpenNLP project has a maximum entropy implementation. LibSVM has a support vector machine implementation. You'll almost certainly have to modify your data to something that the library can understand.
Good luck!
Update: I agree with the other commenter that Russel and Norvig is a great AI book which discusses some of this. Bishop's "Pattern Recognition and Machine Learning" discusses classification issues in depth if you're interested in the down and dirty details.
Your task is classical for neural networks, which are intended first of all to solve exactly classification tasks. Neural network has rather simple realization in any language, and it is the "mainstream" of "machine learning", closer to AI than anything other.
You just implement (or get existing implementation) standart neural network, for example multilayered network with learning by error back propagation, and give it learning examples in cycle. After some time of such learning you will get it working on real examples.
You can read more about neural networks starting from here:
http://en.wikipedia.org/wiki/Neural_network
http://en.wikipedia.org/wiki/Artificial_neural_network
Also you can get links to many ready implementations here:
http://en.wikipedia.org/wiki/Neural_network_software