I'm planning on building a feature for an e-commerce platform I developed in Java to display related products in much the same way Amazon does. There are a few different metrics for relating products that I want to explore.
Purchase history (purchased at the same time)
Related by family/type (similar product classifications)
Intentionally related (boosting results; "Buy this!")
While I would probably be able to develop my own datamining library, it wouldn't be very portable and I dare say it wouldn't be very good either.
There are several packages out there for doing this sort of thing but I don't feel like I am in a position to evaluate which package or solution would work best for me. Any input anecdotal or from personal experience would be greatly appreciated.
Note: I've tagged this as Neural networking because of a python talk I was at where a neural-like-network was used for datamining, I'm not convinced a neural network is the best choice for this job.
Take a look at Apache Mahout
There are some artificial algorithm techniques used for data mining, such as C4.5 or ID3. These algorithm does classification. Other techniques such as ant clustering, neural networks or genetic algorithms are used for classification purposes in data mining.
As far as algorithms, I don't know much but ID3/C4.5 can be easily programmed.
Hope this helps.
Related
I am a fresh college graduate and at my new job I have been tasked with the job of merging two different applications. The applications are Cesium (a web-based service written in JavaScript, HTML and CSS) and the Ozone Widget Framework ( Written in JAVA). This seems like a fairly complex task to start out with. How would I go about doing this? or learning how to merge such technologies?
Advice/help is greatly appreciated!
The OWF website says it already supports Cesium:
A fully functional map widget that is designed to be extensible by other programs for their particular needs, and currently provides Google Earth, OpenLayers, Cesium and WorldWind renderers
I work on Cesium, but have never used OWF so I can't speak to the details.
There is also a Cesium plugin that provides GWT wrappers for Java developers that may be of interest to you.
Patrick
There seems to be very little documentation in the Ozone Widget Framework. In fact, a search for it has only yielded 6 results in Stack Overflow.
I have experience with both OWF and Cesium. OWF is particularly difficult, for the reasons mentioned here. Unless you're working in a community that does a lot with OWF, you're not going to get a lot of help.
Most likely the approach that will yield you the best results in a sane amount of time will be to create your own OWF widget (which can be just a page, like any other, that is imported through the admin tool or made available in a app/widget store). You could then use OZONE Synapse (ugh) to integrate with other widgets in OWF.
Cesium is a pretty darn good tool. I'm not really a fan of OWF for a number of reasons. If you can dissuade your employer from using it, that may be worthwhile. If it's a direct client requirement, then you'll probably want to see if you can join OWF specific developer communities.
I'm starting a project in which sentiment analysis is going to take center stage. Specifically, we'll be doing sentiment analysis of Twitter, Facebook, YouTube and other social network data.
I know of OpenNLP from Apache. It appears great but I think it's a little heavyweight for what I want to do in addition to it's dependence on Hadoop and the like. I haven't used it before and I may be wrong in my assessment of it.
I've seen elsewhere on this site about Stanford NLP. I can't seem to get a good starting point with this library; a tutorial sort of.
Also, I've read about Sentiment Anaysis APIs like AlchemyAPI on this site, but I want a solution I'm fully in control of. I just want a library I can bundle with my application.
In a nut shell I'm looking for a solution that is lightweight, and that I can set up in my local PC. Also, a pointer to a good starting point for Stanford NLP or OpenNLP will be appreciated very much.
UPDATE:
I've gone through the UIMA documentation and its support for components like OpenNLP components and other third party components, in addition to its inbuilt text processing capabilities makes it an attractive starting point. It's open architecture makes me feel it's ideal for what I want to achieve. Additional recommendation or advice will still be appreciated very much.
You should also take a look at:
SenticNet
CLIPS
All of these are easy to integrate with Python. You also can use NLTK, a great library for doing NLP.
I have read the explanation in http://en.wikipedia.org/wiki/PageRank and i understand that the page rank is calculated by incoming links and out going links.
I have a crawler while crawls a webpage and store in db i need an page-rank algorithm.
I have a db with following values
Title
url
content_html
outgoing_links(external domain)
internal_links(the links with same domain of the url)
can u please explain do i need any other value to compute the page rank and. please explain how to compute it using java
PageRank is, at its heart, a linear algebra eigenvalue problem:
http://www.rose-hulman.edu/~bryan/googleFinalVersionFixed.pdf
If you don't know linear algebra or eigenvalue problems, or aren't willing to read this paper, it's unlikely that you'll be able to tackle this problem. As Einstein said, "Make the problem as simple as possible, but no simpler..."
The paper's title is old; it refers to Google's market cap circa 2004. It's up to $211B this morning.
The technology hasn't stood still in all that time. Google continues to tweak the algorithm in proprietary ways. But this paper explains the heart of it.
You have a few options. If you want to do it all yourself then duffymo's solution is perfect but if you want to use existing libraries I would suggest something similar to Jung for graphs.
I'm not sure if your familiar with graphs but they can be used to store the structure of the links and pagerank is often included in most libraries. Depending on how you want to do it, a good in memory solution is Jung but if you need persistent database storage than loading your data into Neo4J would work(you would have to install gremlin to do the pagerank).
The above are Java solutions but if you want to do it yourself(and like me don't like dry research papers) then I would highly suggest the book programming collective intelligence. They go through(chapter 4? I think) creating a search engine from scratch that includes pagerank and neural networks to monitor clicks. The only problem, based on your requirements above, is the book is written in python but you can easily apply the logic to java. If you know a bit of python already then you can even download the books source code for free and check out the software(but there is no explanation on the math behind the code in the source code).
Hope that helps
Does anyone know of a library with a working implementation of backpropagation through time?
Any of Java/Python/C#/VB.NET/F# (preferably the last one) will do!
Assuming you're already using some library for BP, it should be (TM) rather straightforward to implement BPTT using BP as a step in the process.
The Wikipedia entry for BPTT [1] includes relevant pseudo code.
My own starting point, about 18 years ago, was "The Truck Backer-Upper: An Example of Self-Learning in Neural Networks" [2].
[1] http://en.wikipedia.org/wiki/Backpropagation_through_time
[2] http://www-isl.stanford.edu/~widrow/papers/c1989thetruck.pdf
I've used NeuronDotNet only for a limited time though. It allows you to create a feed-forward BackPropagation NN. I especially liked their use of intuitively named classes. Good luck!
This is a .net library.
I'm from a Java background but Encog has a .net implementation as well (and is a seriously good framework for NNets, with good time series support)
Can't help with an F# framework, but what domain are you coding for? If it's finance I'll reassert the "take a look at Encog"
Perhaps pybrain would do? The docstring for its BackpropTrainer class suggests that it does backpropagation through time:
class BackpropTrainer(Trainer):
"""Trainer that trains the parameters of a module according to a
supervised dataset (potentially sequential) by backpropagating the errors
(through time)."""
What about this one ? Just a Google search to help...
I've had good experiences with Weka - In my view one of the best and almost certainly the most comprehensive general purpose machine learning libraries around.
You could certainly do BPTT with Weka - you may find a ready made classifier that does what you need but even if not you can just chain a few normal backpropagation units together as per the very good wikipedia article on BPTT
I made backpropagation algorithm in Java quite time ago. I uploaded it into GitHub, maybe you can find it useful: https://github.com/bernii/NeuralNetwokPerceptronKohonen
Let me now if it was helpful :)
You can use TensorFlow's dynamic_rnn() function (API doc). TensorFlow's tutorial on Recurrent Neural Networks will help.
Also, this great blog post provides a nice introduction to predicting sequences using TensorFlow. Here's another blog post with some code to predict a time series.
I'm planning to develop program in Java which will provide diagnosis. The data set is divided into two parts one for training and the other for testing. My program should learn to classify from the training data (BTW which contain answer for 30 questions each in new column, each record in new line the last column will be diagnosis 0 or 1, in the testing part of data diagnosis column will be empty - data set contain about 1000 records) and then make predictions in testing part of data :/
I've never done anything similar so I'll appreciate any advice or information about solution to similar problem.
I was thinking about Java Machine Learning Library or Java Data Mining Package but I'm not sure if it's right direction... ? and I'm still not sure how to tackle this challenge...
Please advise.
All the best!
I strongly recommend you use Weka for your task
Its a collection of machine learning algorithms with a user friendly front-end which facilitates a lot of different kinds of feature and model selection strategies
You can do a lot of really complicated stuff using this without really having to do any coding or math
The makers have also published a pretty good textbook that explains the practical aspects of data mining
Once you get the hang of it, you could use its API to integrate any of its classifiers into your own java programs
Hi As Gann Bierner said, this is a classification problem. The best classification algorithm for your needs I know of is, Ross Quinlan algorithm. It's conceptually very easy to understand.
For off-the-shelf implementations of the classification algorithms, the best bet is Weka. http://www.cs.waikato.ac.nz/ml/weka/. I have studied Weka but not used, as I discovered it a little too late.
I used a much simpler implementation called JadTi. It works pretty good for smaller data sets such as yours. I have used it quite a bit, so can confidently tell so. JadTi can be found at:
http://www.run.montefiore.ulg.ac.be/~francois/software/jaDTi/
Having said all that, your challenge will be building a usable interface over web. To do so, the dataset will be of limited use. The data set basically works on the premise that you have the training set already, and you feed the new test dataset in one step, and you get the answer(s) immediately.
But my application, probably yours also, was a step by step user discovery, with features to go back and forth on the decision tree nodes.
To build such an application, I created a PMML document from my training set, and built a Java Engine that traverses each node of the tree asking the user to give an input (text/radio/list) and use the values as inputs to the next possible node predicate.
The PMML standard can be found here: http://www.dmg.org/ Here you need the TreeModel only. NetBeans XML Plugin is a good schema-aware editor for PMML authoring. Altova XML can do a better job, but costs $$.
It is also possible to use an RDBMS to store your dataset and create the PMML automagically! I have not tried that.
Good luck with your project, please feel free to let me know if you need further inputs.
There are various algorithms that fall into the category of "machine learning", and which is right for your situation depends on the type of data you're dealing with.
If your data essentially consists of mappings of a set of questions to a set of diagnoses each of which can be yes/no, then I think methods that could potentially work include neural networks and methods for automatically building a decision tree based on the test data.
I'd have a look at some of the standard texts such as Russel & Norvig ("Artificial Intelligence: A Modern Approach") and other introductions to AI/machine learning and see if you can easily adapt the algorithms they mention to your particular data. See also O'Reilly, "Programming Collective Intelligence" for some sample Python code of one or two algorithms that might be adaptable to your case.
If you can read Spanish, the Mexican publishing house Alfaomega have also published various good AI-related introductions in recent years.
This is a classification problem, not really data mining. The general approach is to extract features from each data instance and let the classification algorithm learn a model from the features and the outcome (which for you is 0 or 1). Presumably each of your 30 questions would be its own feature.
There are many classification techniques you can use. Support vector machines is popular as is maximum entropy. I haven't used the Java Machine Learning library, but at a glance I don't see either of these. The OpenNLP project has a maximum entropy implementation. LibSVM has a support vector machine implementation. You'll almost certainly have to modify your data to something that the library can understand.
Good luck!
Update: I agree with the other commenter that Russel and Norvig is a great AI book which discusses some of this. Bishop's "Pattern Recognition and Machine Learning" discusses classification issues in depth if you're interested in the down and dirty details.
Your task is classical for neural networks, which are intended first of all to solve exactly classification tasks. Neural network has rather simple realization in any language, and it is the "mainstream" of "machine learning", closer to AI than anything other.
You just implement (or get existing implementation) standart neural network, for example multilayered network with learning by error back propagation, and give it learning examples in cycle. After some time of such learning you will get it working on real examples.
You can read more about neural networks starting from here:
http://en.wikipedia.org/wiki/Neural_network
http://en.wikipedia.org/wiki/Artificial_neural_network
Also you can get links to many ready implementations here:
http://en.wikipedia.org/wiki/Neural_network_software