Develop Reference

Is there a way to get the headers from Github csv raw data in java?

I'm currently trying to extract headers only in Strings.
The below is the method and I'm trying to get the headers from
https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv
not the data of the headers.
I'm using Commons-csv library .
I tried some codes that I found in stackoverflow but it prints the values of the header not the header itself.
how should I change my code to extract headers only with List of Strings
public void fetchVirusData() throws IOException, InterruptedException{
List<LocationStats> newStats=new ArrayList<>();
HttpClient client=HttpClient.newHttpClient();
HttpRequest request= HttpRequest.newBuilder()
.uri(URI.create(VIRUS_DATA_URL))
.build();
HttpResponse<String> httpResponse=client.send(request, HttpResponse.BodyHandlers.ofString());
StringReader csvBodyReader=new StringReader(httpResponse.body());
Iterable<CSVRecord> records = CSVFormat.DEFAULT.withFirstRecordAsHeader().parse(csvBodyReader);
//The code I found in stackoverflow that can extract headers but won't work
CSVParser parser = CSVParser.parse(csvBodyReader, CSVFormat.EXCEL.withFirstRecordAsHeader());
List<String> headers = parser.getHeaderNames();
for(String s:headers){
System.out.println("header test: "+s);
}
}

UPDATE
Try this
String fileName = "data.csv";
CSVReader reader = new CSVReader(new FileReader(fileName ));
// if the first line is the header
String[] header = reader.readNext();

Create a java REST client to call a spring boot REST API

I have a springboot project that takes only POST JSON String which I'm converting to HashMap using gson. I tested using Postman as a POST and add the body as props with a json string like {'fistname': 'John', 'lastname' : 'Doe'}, translates to props = {'fistname': 'John', 'lastname' : 'Doe'}. its working as expected
#RequestMapping(value = "/rest", method = RequestMethod.POST)
protected String parse(#RequestParam("props") String props) {
Gson gson = new Gson();
Map<String, String> params = new HashMap<String, String>();
params = gson.fromJson(props, Map.class);
// Rest of the process
}
On the other hand, I have a JavaEE project, which needs to call this API
protected void callREST() {
try {
String json = someClass.getDate() //retrieved from database which is stored as json structure
Map<String, String> props = gson.fromJson(json, Map.class);
URL url = new URL("http://localhost:9090/myApp/rest");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setDoOutput(true);
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Type", "application/json");
DataOutputStream wr = new DataOutputStream( conn.getOutputStream());
System.out.println(props.toString());
wr.writeBytes(json.toString());
wr.flush();
wr.close();
if(conn.getResponseCode() != HttpURLConnection.HTTP_CREATED) {
throw new RuntimeException("Failed :: HTTP error code : " + conn.getResponseCode());
}
BufferedReader br = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String output;
System.out.println("Output from Server ... \n");
while((output = br.readLine()) != null) {
System.out.println(output);
}
conn.disconnect();
} catch(Exception e) {
//print stack trace
}
}
I get Failed :: HTTP error code : 400. I suspect the spring boot is not receiving the data in props variable since its a POST request.
What should i add in the client code to pass the props and the data to make this call successful ?
Note: JavaEE is running on tomcat :8080, Springboot is running on
different tomcat :9090

#RequestParam means that server awaits param in request URL http://localhost:9090/myApp/rest?param=..... but in you client you are writing JSON in body of request.
Try to use #RequestBody annotation in your endpoint
protected String parse(#RequestBody String props) {...}

Your resources expects to get form parameters (i.e. key value pairs, using x-www-form-urlencoded encoding), where the value happens to be JSON (although what you posted is not valid JSON).
But your client Java code sets the content type to application/json, and sends the JSON as the body, instead of sending it as the value of the key "props" of a x-www-form-urlencoded body.
So that can't work.
If you can change the server, then do that. Accept JSON as the body directly:
#RequestMapping(value = "/rest", method = RequestMethod.POST)
public String parse(#RequestBody Map<String, String> map) {
...
}
If not, you'll need to send the correct key value pair, and make sure the value is properly url-encoded.

HttpURLConnection.getRequestProperties() throws IllegalStateException: "Already connected"

After executing a request I would like to check the request headers, but it doesn't work.
I call getRequestProperties() on an instance of sun.net.www.protocol.http.HttpURLConnection and I always get an IllegalStateException with the message "Already connected". As if I wanted to set request properties. But I only want to read them.
The responsible code for this behaviour is in the HttpUrlConnection:
http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7u40-b43/sun/net/www/protocol/http/HttpURLConnection.java#HttpURLConnection.getRequestProperties%28%29
public synchronized Map<String, List<String>> getRequestProperties() {
if (connected)
throw new IllegalStateException("Already connected");
// ...
}
Ok so maybe I should only read the request properties after disconnecting. But it turns out, disconnect() doesn't set connected to false. Although it should: http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7u40-b43/sun/net/www/protocol/http/HttpURLConnection.java#HttpURLConnection.disconnect%28%29
It also doesn't seem to make a difference if I read the stream to the end or not. Closing the InputStream before or after calling disconnect doesn't make a difference either.
I'm confused. Can you help me?
Why doesn't disconnect() set connected to false?
Why can't I read request properties while the urlConnection is connected?
How do you properly read request headers after the request?
The code to reproduce this is a Unit test for Android (I use Robolectric), but I think you can use it in a Java project as well and call it from main() after removing the test annotation:
/**
* Test if HttpUrlConnection works as expected, because in some cases it seems it doesn't
*
* #throws Exception
*/
#Test
public void testHttpUrlConnection() throws Exception
{
final URL url = new URL("http://www.stackoverflow.com");
final HttpURLConnection urlConnection = ( HttpURLConnection ) url.openConnection( );
urlConnection.setRequestMethod("GET");
InputStream is = null;
try
{
is = urlConnection.getInputStream();
assertEquals(200, urlConnection.getResponseCode());
}
catch (IOException ex)
{
is = urlConnection.getErrorStream( );
}
final String result = copyStreamToString(is); // some html response
// Streams must be closed before disconnecting (according to http://stackoverflow.com/a/11056207/3596676)
is.close();
assertTrue((Boolean) getFieldViaRecursiveReflection(urlConnection, "connected"));
// urlConnection should always be disconnected (according to http://developer.android.com/reference/java/net/HttpURLConnection.html)
urlConnection.disconnect();
assertFalse((Boolean) getFieldViaRecursiveReflection(urlConnection, "connected")); // AssertionError
// getRequestProperties throws IllegalStateException ("already connected")
Map<String, List<String>> requestProperties = urlConnection.getRequestProperties();
// do stuff with the properties
// return the result
}
private static String copyStreamToString( final InputStream is ) throws IOException
{
if ( is == null )
{
return "";
}
BufferedReader reader = new BufferedReader( new InputStreamReader( is ) );
String result = copyBufferedReaderToString( reader );
reader.close( );
return result;
}
private static String copyBufferedReaderToString( final BufferedReader bufferedReader ) throws IOException
{
StringBuffer sb = new StringBuffer( );
String line;
while ( ( line = bufferedReader.readLine( ) ) != null )
{
sb.append( line );
}
return sb.toString( );
}
private static Object getFieldViaRecursiveReflection(final Object object, final String attribute) throws Exception
{
return getFieldViaRecursiveReflection(object, object.getClass(), attribute);
}
private static Object getFieldViaRecursiveReflection(final Object object, final Class<?> c, final String attribute) throws Exception
{
try
{
final Field field = c.getDeclaredField(attribute);
field.setAccessible(true);
return field.get(object);
}
catch (NoSuchFieldException ex)
{
/* end of type hierarchy? */
Class<?> superClass = c.getSuperclass();
if (superClass == null)
{
throw ex;
}
else
{
return getFieldViaRecursiveReflection(object, superClass, attribute);
}
}
}

As no one posted an answer in the 2 months since I asked the question, but today I had to deal with the problem again and found a solution, I will answer my own question.
I can't answer all the questions in the answer (e.g. "why doesn't urlConnection.disconnect() set the connected attribute of urlConnection to false?"), but I found the solution for the main problem, which was that reading the headers of a request didn't work when the urlConnection was connected.
For some reason, which I can't remember, I wanted/needed to check the request headers after the request was done and the response was there. But I looked at the implementation of getRequestProperties() in sun.net.www.protocol.http.HttpURLConnection again (see the code here) and noticed that a method called filterAndAddHeaders gets called. So it seems that headers not only get read in that method, but set. I'm not sure why this is done in a getter method (the getRequestProperties() one), but it makes sense that when the request is already done, you should warn the user when he tries to add request headers - in this case with the IllegalStateException that bothered me so much.
To come to the solution:
I just moved the call to getRequestProperties() to before the request gets sent. And now everything works fine.
P.S.:
Please note that this is not all there is to it. One of my unit tests ran successfully even though I called getRequestProperties() after the request. In that case the urlConnection internal attribute connected was set to false. I haven't figured it all out, but it may have been related to the response status code 304 (not modified). Maybe this helps as a hint if you have to deal with this problem and for some reason can't move the getRequestProperties() call to before sending the request.

Just to clarify, backenddev says "I just moved the call ... to before the request gets sent", but when is that?
The error message is "already connected" so you may think that you have to get the parameters before the connection is opened, which is impossible (because the object that you need is created by that call). As he says, you have to get them before the request is sent.
The code below works for me. In my case, I'm making an https request. The TIMEOUT value is an integer (in milliseconds) and payload is a String containing the body. HTTP_FAILURE is the value 401. My call needed an authorization header, the value of which is not shown.
The getOutputStream() call sends the headers, and once you've done that, you can't change them (which is sensible) but you can't look at them either (which is not so sensible).
I had a very puzzling problem, so I wanted to display all of the parameters. (Actually, although I called setRequestProperty() three times, my debug only showed the Content-type value. The other properties are not parameters, apparently. Go figure!)
byte[] buffer = new byte[payload.length()];
buffer = payload.getBytes();
bout = new ByteArrayOutputStream();
bout.write(buffer);
byte[] payloadAsBytes = bout.toByteArray();
bout.close();
String contentLength = String.valueOf(payloadAsBytes.length);
String contentType = "application/xml; charset=utf-8";
URL url = new URL(serviceURL);
URLConnection connection = url.openConnection();
httpConn = (HttpsURLConnection)connection;
// Set the appropriate HTTP parameters.
httpConn.setRequestProperty("Content-Length", contentLength);
httpConn.setRequestProperty("Content-Type", contentType);
httpConn.setRequestProperty("Authorization", authHeader);
httpConn.setRequestMethod("POST");
httpConn.setDoOutput(true);
httpConn.setDoInput(true);
httpConn.setConnectTimeout(TIMEOUT);
httpConn.setReadTimeout(TIMEOUT);
// Display the params for debug
if (logger.isDebugEnabled()) {
logger.debug("{} - HTTP headers:", m);
Map<String, List<String>> props = httpConn.getRequestProperties();
Set<String> keys = props.keySet();
for (String key: keys) {
StringBuilder values = new StringBuilder();
for (String value: props.get(key)) {
values.append("\"").append(value).append("\" ");
}
logger.debug("{} - {}: {}", m, key, values.toString());
}
logger.debug("{} - request method {}", m, httpConn.getRequestMethod());
logger.debug("{} - url {}", m, httpConn.getURL().toString());
}
// Send the request
out = httpConn.getOutputStream(); // send the headers
out.write(payloadAsBytes); // send the body
out.close();
// check the response
if (httpConn.getResponseCode() == HTTP_FAILURE) {

Making a HTTPS Post request using XML in Java

I'm trying to use the API from Web Of Knowledge(WoK) to obtain some data. The documentation explain that you have to do POST Requests through HTTPS, sending a XML which contains the queries. But I only get the error 400 form server. (Bad Request)
Here is my code, I found it in Google and I make some fixes for my case.
public static void main(String[] args) throws Exception {
// Get target URL
String strURL = /*Here the Server URL*/;
// Get file to be posted
String strXMLFilename = "src/main/resources/xml/wosdata.xml";
File input = new File(strXMLFilename);
// Prepare HTTP post
PostMethod post = new PostMethod(strURL);
// Request content will be retrieved directly
// from the input stream
// Per default, the request content needs to be buffered
// in order to determine its length.
// Request body buffering can be avoided when
// content length is explicitly specified
post.setRequestEntity(new InputStreamRequestEntity(
new FileInputStream(input), input.length()));
// Specify content type and encoding
// If content encoding is not explicitly specified
// ISO-8859-1 is assumed
post.setRequestHeader(
"Content-type", "text/xml; charset=ISO-8859-1");
// Get HTTP client
HttpClient httpclient = new HttpClient();
// Execute request
try {
int result = httpclient.executeMethod(post);
// Display status code
System.out.println("Response status code: " + result);
// Display response
System.out.println("Response body: ");
System.out.println(post.getResponseBodyAsString());
}catch (Exception e) {
e.printStackTrace();
} finally {
// Release current connection to the connection pool
// once you are done
post.releaseConnection();
}
}

There is something wrong with the XML you are sending. You will have to look at server logs to find out exactly what, as 400 deliberately tells you as little as possible.

You should do it like this. First read the contents of the xml to String and do post using a StringRequestEntity.
// Get file to be posted
String strXMLFilename = "src/main/resources/xml/wosdata.xml";
StringBuilder contents = new StringBuilder();
try {
BufferedReader input = new BufferedReader(new FileReader(new File(strXMLFilename)));
try {
while (( line = input.readLine()) != null){
contents.append(line);
contents.append(System.getProperty("line.separator"));
}
}
finally {
input.close();
}
StringEntity requestEntity = new StringEntity(contents.toString());
post.setEntity(requestEntity);

read post request values HttpHandler

I'm writing a little Java app which implements an http service that receives http post commands from a client.
The class I'm using to implement all of this is HttpHandler and HttpServer in the com.sun.net. package.
Now I'm implementing an handle(HttpExchange exchange) function which handles the request, and I'm having truble reading the post values received by the request because the only access that I have to these values is via HttpExchange.getResponseBody() which is just an output stream.
I'm looking to parse text post values and uploaded files.

I have written classes that process multipart requests for my project Sceye-Fi, an HTTP server that uses the com.sun.net.httpserver classes that come with java 6, to receive photo uploads from an Eye-Fi card.
This can help with file uploads (multipart posts).
For a non-multipart post, you would need to do something like this:
// determine encoding
Headers reqHeaders = exchange.getRequestHeaders();
String contentType = reqHeaders.getFirst("Content-Type");
String encoding = "ISO-8859-1";
if (contentType != null) {
Map<String,String> parms = ValueParser.parse(contentType);
if (parms.containsKey("charset")) {
encoding = parms.get("charset");
}
}
// read the query string from the request body
String qry;
InputStream in = exchange.getRequestBody();
try {
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte buf[] = new byte[4096];
for (int n = in.read(buf); n > 0; n = in.read(buf)) {
out.write(buf, 0, n);
}
qry = new String(out.toByteArray(), encoding);
} finally {
in.close();
}
// parse the query
Map<String,List<String>> parms = new HashMap<String,List<String>>();
String defs[] = qry.split("[&]");
for (String def: defs) {
int ix = def.indexOf('=');
String name;
String value;
if (ix < 0) {
name = URLDecoder.decode(def, encoding);
value = "";
} else {
name = URLDecoder.decode(def.substring(0, ix), encoding);
value = URLDecoder.decode(def.substring(ix+1), encoding);
}
List<String> list = parms.get(name);
if (list == null) {
list = new ArrayList<String>();
parms.put(name, list);
}
list.add(value);
}

An alternative would be using HttpService from HttpCore.
There is a Basic HTTP server example in the documentation