Develop Reference

How to take a webpage's content to create a native app for it? e.i not just a webview of the website

I want to create an app for a website that I don't own, like youtube or reddit. Every time I search for help, all I keep getting is how to create a webview. I've done that but that's not what I want. I want to take the website's content and create an app with it, like create my own buttons and such. I want to convert a website into an app.
I was thinking of somehow loading the website in the background and creating activities/buttons/intents that basically do what the webpage does but I don't know how to go about that or if there's a better way.
Thank you in advance.
Please let me know if clarification is needed.

Alright i got -3 votes for some reason and it has been 12 hours since i posted this question, i'm guessing the question has been consumed by the abyss of questions by now.
I made a reddit post and got some responses:
one user:
The term you’re looking for is “web scraping“ or “site scraping”. I think that’s probably what you need to research before you’re ready to handle how to make that into an app.
That’s a good starting point and once you have the content from scraping, making the app is no different than any other app.
second user:
If the website has a public (or private) API, you can use your buttons and networking calls to their API to build your client app. You are building a "YouTube client" in this case using the YouTube API.
If they dont have an API and you don't want to display their website, what you are doing is trying to reverse engineer their website and hijack the UI which is not a good practice and may even be illegal depending on the site and terms.
third user:
#second user is right on the money, but one thing I'll like to add is, the goal is to get data off of the website so that you can use in your app. An API is one (convenient) way of doing that but others are,
RSS or similar feed content (prefer this if there website supports)
HTML scraping
The above two will require you store the data somewhere else (and possible expose an API instead) because of technical reasons: you don't have access to older entries in RSS feeds and it's atrocious to keep on scraping everytime you need to access the website data. They can be done on the client (mobile app) but are best done on a server. This means you might need to get your hands dirty writing server code but you don't have to.
/thread
I'm going to do research on website scrapping and APIs to see which route i should go.
Hopefully this helped someone else.

How to make an API to interact with my app

How can I make an online server to access API's and handle requests? Then how can I have my android app access this online server to make requests?
I have an android app that I am building for a school project. This app will need to use multiple APIs to gather information to display to the user. However I don't want to allow people to decompile to source code and take my API keys. My plan is to build an online application that will do all of this work on the server side. This way the user only submits a query and the server sends information in response to their request. This will speed things up for the user and keep the keys from being accessed by other individuals.
Steps I think I need to take.
Set up server to access the API's
Make app access online server through HTTP requests
3....
4...
I have a very general concept in my head, but I'm not sure where to begin. If I'm wrong on any part of this question please correct me.
I do plan on moving this to the app store in the future.
edit: Do you know of tutorials that show the building of the API and then loading it to Amazon AWS or another server?

There are many options when building restful services for api consumption. You could start with php, which is the easiest to start with. Here is a nice tutorial that takes you through the initial stage all the way to the end of building login system for android using mysql and php as the server language. It contains the barebone details of setting up the infrastructure and logic. I think from there you manipulate and go further.
https://www.androidhive.info/2012/01/android-login-and-registration-with-php-mysql-and-sqlite/

check out java rest easy, it is a nice tool to use to build web API's. You can host it for cheap at red shift or amazon using their pay what you use billing.
The cheapest way for you to host your REST API is to use Amazon's API Gateway. You pay for what you use and pay nothing if it is not being used. API Gateway plays very nicely with Amazon's Lambda service that allows you to run discrete code units - again, you only pay for what you use. Lambda itself plays nicely with Amazon's pay-as-you go NoSQL datastores (SimpleDB and DynamoDB).

Facebook Messenger - ChatBot - wit.ai integration

I have my application running on AWS on tomcat instance with Java code with mongodb database.
I now need to integrate this with Facebook messenger ChatBot and wit.ai.
I am really struggling to get started, some sample code i have found but in different languages.
Can i run this and integrate with my portal running on Tomcat. it should call both Facebook messenger and wit api.
I need to get high level idea as to how to proceed.

To connect wit.ai to your facebook page, you just need to create an accound with wit.ai, and then create a new application (or use the default starting one).
Then you can go to the settings of the app, and you will se the Server Access Token code.
You need to paste that server access code in the field "Integrated NLP", that you will find in your Facebook for Developers account, then access the application that you want to use NLP, in the setting for Messenger. For that you must have the messenger app already created and configured with it's webhook wich I guess you have.
Once that's done, the Natural Language Processing (NLP) will be sent to your server automatically as another field in the message. And it's preety simple to understand and train from wit.ai
The messages of the user will be analyzed by wit.ai, and if it finds any entity of the default ones or one defined by you training the bot from wit.ai, will send it to messenger, and this one will insert the NLP in the message.
So you will recive a normal JSON message from messenger, with an extra field called NLP that will contain something like the following example:
"text":"reservar una pista","nlp":{"entities":{"tipo_cita":[{"confidence":0.98398202482107,"value":"pista","_entity":"tipo_cita"}]},"detected_locales":[{"locale":"es_XX","confidence":0.9935}]}}}]
My chatbot is in spanish, but there's an example of how you can see the NLP in the JSON message.
I introduced the text: "reservar una pista"
And it recognized the value: "pista" with a confidence of 0.9839. Wit also detected the language "es_XX" that's spanish with a confidence of 0.9935.

First of all think about what your bot should do. What functionality will it provide, which questions will be asked by users and how you would like to respond to that.
If you have a general idea about your problem space you can start thinking about the technical challenges.
Be aware that both the Messenger Platform and wit.ai are completely independent products and they provide an HTTP interface to their service. That means you can use any language you like to interact with these platforms. Your application acts as the middleman to these services.
Start off establishing an integration with the Messenger Platform so you that you are able to both receive and send messages.
For that you have to create a Facebook page which acts as the identity of your product. Users will find you in Messenger by your pages' name. You also need to create a Facebook app where you subscribe to your page and specify the webhook settings so that messages are relayed to your own backend service.
The documentation for this is really good and I advise you read it carefully. You can find it here.
After going through the docs you will have a good understanding of what you can do with the Messenger Platform and which types of messages you can send and receive.
Once your able to send and receive messages you can start to extract some sense out of them. For this you can use wit.ai, but there are also other services you might consider. E.g Google just released their own NLP platform which provides similar features. See here
Also there is api.ai
In general you have to send received messages to the service of your liking and get back structured information about what the intent of the user is and what values where extracted. With that information you can act accordingly.
If you want to stick with wit.ai go through their Getting Started guide and recipes section. That enabled me to use their platform.
I hope this gives you a general idea about how an integration could work. This is fairly high-level and much of the details (especially on the NLP side) depend on your specific use cases.

You can use a sample nodejs implementation of witai and facebook messenger from their official repo. First of all, you have to train the model of witai to understand expressions and extract entities. Then set up the messenger bot on fb and attach it to some fb page. Once you are able to get the messages from the webhook callback send them to the witai API. You will also have to define actions in your code for the witai to execute actions defined in the model.

Can an android application(client) communicate with a java web server using sockets?

I need an android application to communicate with a java web server using sockets, but can't seem to find any information regarding this. I have only been able to find information about java to java and android to android.

When I first tried doing this, I used the basic knock example taken from Lesson: All About Sockets, most likely written before the android OS existed, and adapted the server side to run on an android. That way I didn't have to make an user interface and could just focus on the communication. The code is here: EchoClient, KnockKnockServer, and KnockKnockClient.
The problems I initially ran into included:
Making sure that both sides knew which port to use.
Making sure I had set the permissions on the Android side to use the internet.
Making sure the client knew the ip of the server.
I posted my version of this code here. It worked one year, and then I had to fiddle with it again when I used it for my class this year. My students got it working, though. You'll get a security flag when you go to the site, since I don't want to pay for an officially registered certificate, but that is a rant for another day :-)

Here is an example how to use sockets in Android. With web server you should do the same. If you want to work with http protocol here is another link.

Connect two calls via web application

I need to add a functionality to my java-based web application that will allow users to click on a link and the application will automatically call the user and another party and connect them in a phone call.
Does anybody know what would this entail?
Thanks

It can be done with Twilio, and their new, easy Conferencing API. Trust me, it's really really simple. Another option might be CloudVox, but I haven't (formally) tried their service yet.

The World-Wide Web Consortium has an integrated set of speech interaction standards that you'll find interesting. There's a markup language called VoiceXML that is analogous to HTML in that web applications generate it. It differs from HTML in that it's specialized for temporally-based speech interactions instead of visual interactions. So instead of looking at a screen you listen to audio prompts and computer-generated speech. Instead of typing and mousing, you speak back and what you say is processed by a speech recognizer or recorded.
There are many companies using VoiceXML to automate voice response systems, and they handle billions of calls per year. You've probably talked to them many times without realizing it. One of the best companies in this space is Voxeo, and they have a developer site at http://evolution.voxeo.com/ that you can play with. Evolution lets you call your web application over an ordinary phone (or Skype). You actually talk to a VoiceXML-based web browser which will fetch a VoiceXML page from your Java application server, "play" it to you, listen to what you say, and then report that back to your app via a form submission, get the next page to render to you, etc.
Another related standard is CCXML, or Call Control XML. You use this to create teleconferences that may or may not include a voice response application.
So it sounds like in your case you want your standard web application to talk to a CCXML server and ask it establish call legs to the web user and to a customer service line. I know that Voxeo Evolution offers CCXML as well.
There are other good companies in this space too. One that comes to mind is TellMe, which was bought by Microsoft a year or two ago. These two companies (and others) offer professional services too.

So I wanted to write this up as an answer to the comment above. The Skype API provides a number of options for telephony in COM, Java and Python:
Skype4Java - https://developer.skype.com/wiki/Java_API
Skype4Py - https://developer.skype.com/wiki/Skype4Py
Skype4COM - https://developer.skype.com/Docs/Skype4COM
They provide a communication and command protocol layer for working with Skype, more info on the API here:
https://developer.skype.com/Docs/ApiDoc/Overview_of_the_Skype_API
It's kind of different for every platform, the Linux version is based on DBus or X11.

Try FreeSWITCH. I have done this before. Its pretty straight forward. Can be a bit hairy when you need to log call accounting and all those stuff. I hopefully would be able to provide you some guidelines and code samples, let me get home first. Cheers.
The good thing in using FreeSWITCH, you will be able to handle multiple calls, and quite a number of. You might need that in future.
Note: You have to use some kind of VoIP provider in order to do that. I was using Gizmo5 that time and it was pretty good.
Sorry buddy, lost the servlet code somewhere. But no worries it was a simple servlet. Fortunately, I had added my example Java code for XML-RPC, into the FreeSWITCH wiki, and actually that was the code my servlet was invoking down the road. Below is the snippet.
XmlRpcClientConfigImpl config = new XmlRpcClientConfigImpl();
XmlRpcClient client = new XmlRpcClient();
try {
config.setServerURL(new URL("http://localhost:8080/RPC2"));
config.setBasicUserName("freeswitch");
config.setBasicPassword("works");
client.setConfig(config);
// For external phone calls using VoIP. We will use something like below.
// new Object[]{"originate", "sofia/gateway/gizmo1/6098989898 &bridge(sofia/gateway/gizmo9/0116054545454)"}
// gizmo1, and gizmo9 are the accounts configured under freeswitch gateway configuration.
client.execute("freeswitch.api", new Object[]{"originate", "sofia/internal/1001 &park()"});
} catch (Exception ex) {
ex.printStackTrace();
}
Moreover, you need to configure few things prior doing this. You need to set up the gateway using your VoIP provider settings.
For FreeSWITCH related help, take a look at this SO Thread.

I know of 2 API providers that does what you need:
1) twilio - can connect to 2 or more parties using TwiML (their markup). example
2) Hoiio - very easy to use with 1 line of RESTful api call. example