Develop Reference

Clarification about REST request in Java and output/input streams

I have to implement a post request in plain Java.
I have read the following question:
How to make a post request to a json RESTful Service from Java?
this is a part of the example
String url = "https://myStore.com/REST-API/";
String requestBody = "{\"searchProduct\": \"" + searchProduct + "\"}";
URL obj = new URL(url);
HttpsURLConnection connection = (HttpsURLConnection) obj
.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type", "application/json");
OutputStream outputStream = connection.getOutputStream();
outputStream.write(requestBody.getBytes());
My question is: why the parameters are written on the output stream? As far as I know, output stream is for collecting the output of a request, not to make it.
So just a curiosity, consider that I am obviously not skilled on this.

First let explain how HttpConnectionURL works.
When you want to request data from a server,
you first create a connection to that server.
Then you write data to the connection (request)
and finally read data from the connection (response).
So to write data to the connection you get a reference to the Connection's OutputStream and write data to it.
OutputStreamWriter writer = new OutputStreamWriter(
connection.getOutputStream());
writer.write("message=" + message);
To read data from the connection you get a reference to the Connection's InputStream and read data from it.
InputStreamReader reader = new InputStreamReader(connection.getInputStream());
reader.read();
Generally you use OutputStream when data is flowing out of your program (to file,network etc.,) and InputStream when data is flowing into your program (from file,network etc.,).
I think this will give you the clarity you are looking for.
This answer explains in detail how HttpConnectionURL works

The goal of InputStream and OutputStream is to abstract streams. By stream, I mean the way of the processed data (Input of the program or Output)
If the application receives information from the stream, use the InputStream. If it sends data then OutputStream
InputStreamused to read data from a source.
var input = new FileInputStream("input.txt");// Read the data
OutputStreamused for writing data to a destination.
var output = new FileOutputStream("output.txt");// Write the data
You should read answers in the related question : There are more explanations.

Cannot upload file to a server

I'm using a function called UploadFFGS and this is its content:
URL url = new URL("http://linkedme.com/filebet.txt");
URLConnection ucn = url.openConnection();
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection = (HttpURLConnection) url.openConnection();
FileInputStream is = new FileInputStream("filebet.txt"); //before I download the same file because I must edit it and upload the new version
OutputStream ostream = connection.getOutputStream();
PrintWriter pwriter = new PrintWriter(ostream);
pwriter.print(jTextArea1.getText());
pwriter.close();
This program never uploads the file filebet I have on my desktop to my link (http://linkedme.com/filebet.txt). Any ideas? I call it in this way:
try {
UploadFFGS();
}
catch (MalformedURLException ex) {
Logger.getLogger(xGrep.class.getName()).log(Level.SEVERE, null, ex);
} catch (IOException ex) {
Logger.getLogger(xGrep.class.getName()).log(Level.SEVERE, null, ex);
}
Also, NetBeans gives me this error: "java.net.ProtocolException: cannot write to a URLConnection if doOutput=false - call setDoOutput(true)".

Your approach won't work because your API endpoint (most likely) is a regular file rather than an interpreted script. The endpoint must provide a API by means of which you upload a file (POST/PUT etc).

I have a different solution. Maybe this will be useful for someone.
Just have a look at your advanced proxy settings in your web browser.
System engineers in our company had changed the proxy settings but I was not aware of it.
This error cost me 3 work-days. I got this doOutput error while writing a ftp upload project in my company. I tried everything like adding conn.setDoOutput(true) or 'fifty shades' of similar solutions but non of them saved me.
But, after I changed my proxy settings to correct ones, the error dissapeared and now I am able to upload my files through ftp using urlConnection in java.
I used the code in the link below to make an upload process, and did not add anything except host, port, user and password.
http://www.ajaxapp.com/2009/02/21/a-simple-java-ftp-connection-file-download-and-upload/

reading bytes from web site

I am trying to create a proxy server.
I want to read the websites byte by byte so that I can display images and all other stuff. I tried readLine but I can't display images. Do you have any suggestions how I can change my code and send all data with DataOutputStream object to browser ?
try{
Socket s = new Socket(InetAddress.getByName(req.hostname), 80);
String file = parcala(req.url);
DataOutputStream out = new DataOutputStream(clientSocket.getOutputStream());
BufferedReader dis = new BufferedReader(new InputStreamReader(s.getInputStream()));
PrintWriter socketOut = new PrintWriter(s.getOutputStream());
socketOut.print("GET "+ req.url + "\n\n");
//socketOut.print("Host: "+req.hostname);
socketOut.flush();
String line;
while ((line = dis.readLine()) != null){
System.out.println(line);
}
}
catch (Exception e){}
}
Edited Part
This is what I should have to do. I can block banned web sites but can't allow other web sites in my program.
In the filter program, you will open a TCP socket at the specified port and wait for connections. If a
request comes (i.e. the client types a URL to access a web site), the application will process it to
decide whether access is allowed or not and then, using the same socket, it will send the reply back
to the client. After the client opened her connection to WebPolice (and her request has been checked
and is allowed), the real web page needs to be shown to the client. Therefore, since the user already gave her request, now it is WebPolice’s turn to forward the request so that the user can get the web page. Thus, WebPolice acts as a client and requests the web page. This means you need to open a connection to the web server (without closing the connection to the user), forward the request over this connection, get the reply and forward it back to the client. You will use threads to handle multiple connections (at the same time and/or at different times).

I don't know what exactly you're trying to do, but crafting an HTTP request and reading its response incorporates somewhat more than you have done here. Readline won't work on binary data anyway.
You can take a look at the URLConnection class (stolen here):
URL oracle = new URL("http://www.oracle.com/");
URLConnection yc = oracle.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream()));
Then you can read textual or binary data from the in object.

Read line will treat the line read as a String, so unless you want to mess around with conversions over to bytes, I wouldn't recommend that.
I would just read bytes until you can't read anymore, then write them out to a file, this should allow you to grab the images, keeping file headers intact which can be important when dealing with files other than text.
Hope this helps.

Instead of using BufferedReader you can try to use InputStream.
It has several methods for reading bytes.
http://docs.oracle.com/javase/6/docs/api/java/io/InputStream.html

Java: how to send POST request while displaying the page?

Here is my code:
try
{
ByteArrayOutputStream os = new ByteArrayOutputStream();
ImageIO.write(image, "png", os);
byte[] bytes = os.toByteArray();
os.flush();
os.close();
String code = encode(bytes);
URL base = applet.getCodeBase();
URL url = new URL(base.getProtocol(),
base.getHost(),
base.getPort(),
"/image.php?code=" + code);
HttpURLConnection c = (HttpURLConnection) url.openConnection();
c.setRequestMethod("POST");
c.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
c.setDoOutput(true);
DataOutputStream s = new DataOutputStream(c.getOutputStream());
s.writeBytes("code=" + code);
applet.getAppletContext().showDocument(url, "_blank");
}
catch (Exception e)
{
e.printStackTrace();
JOptionPane.showMessageDialog(
this,
e.toString(),
"Error",
JOptionPane.ERROR_MESSAGE);
}
By the time I use showDocument, the POST request is already done, so what I'm really doing is showing a blank page (instead, I want to show the image). The source of image.php is this:
<?php
$code = base64_decode($_GET["code"]);
header('Content-Type: image/png');
echo $code;
?>

You are using POST on Java and GET on PHP...

Your showDocument and the applet's POST request are completely independent. The POST request is done by your applet, and the result would be only usable inside your applet (but you are not reading it at all - and I'm not sure it is even sent).
showDocument, in contrast, always does a GET request - there is no way to instruct the browser to use POST here. You might be able to fabricate a POST request for a new HTML page by using the JavaScript bridge from your applet, though.
Theoretically, it should work anyways, as you send the image data as part of the URL, too, but there might be a length limit for the URL data in the Web server, or in the link from Java-Plugin to the browser.
You could instead encode your image in a data: URL, and use this for showDocument.
URL url = new URL("data:image/png;base64," + code);
(I did not test if Java's URL class actually accepts this. Please try and report. I suppose it is subject to the same browser URL length limits.)
An alternative would be having the server store the image (at least for some short time). Then you would use your POST from the applet to upload the data, get back (short) some unique identifier, which you then would pass to the showDocument URL.

how to click on a button through java?

I want to access forms on HTMl pages throught Java Programming Language without involving real browser in between.
At present I am doing it through HTML UNIT but it takes a bit more time to load a page. When it comes to accessing millions of page, then this extra bit time matters most.
Is there any other methods for doing this?

I've used something similar called httpunit before, but I have no idea how it compares performance wise.
If you have millions of pages to process, I would recommend throwing some more threads at it. Just a guess, but I think that if you scale this up to multiple threads, you'll run out of bandwidth before you run out of CPU power (in which case it won't matter how much faster it could be)

Accessing a web page using a browser, even HtmlUnit, is going to be slow. A better method is to test the layer just below the web interface, so that you don't need to access millions of pages -- instead you test enough to make sure that the web interface is using the lower layer correctly.

Most of the interaction in browser comes down to an HTTP GET or an HTTP POST.
You need to figure out exactly the operation you need, and then you can construct the URL and/or form data. Then you can use something like this:
try {
//Construct data
String data = URLEncoder.encode("key1", "UTF-8") + "=" + URLEncoder.encode("value1", "UTF-8"); data += "&" + URLEncoder.encode("key2", "UTF-8") + "=" + URLEncoder.encode("value2", "UTF-8");
// Send data
URL url = new URL("http://hostname:80/cgi");
URLConnection conn = url.openConnection(); conn.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(data);
wr.flush();
// Get the response
BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line; while ((line = rd.readLine()) != null) {
// Process line... }
wr.close();
rd.close();
} catch (Exception e) { }