I have written a function to read some data from an external API. What my function does is , it calls that API while reading a file from the disk. I want to optimize my code for large size of a file (35000 records). Could you please suggest me on this.
Following is my code.
public void readCSVFile() {
try {
br = new BufferedReader(new FileReader(getFileName()));
while ((line = br.readLine()) != null) {
String[] splitLine = line.split(cvsSplitBy);
String campaign = splitLine[0];
String adGroup = splitLine[1];
String url = splitLine[2];
long searchCount = getSearchCount(url);
StringBuilder sb = new StringBuilder();
sb.append(campaign + ",");
sb.append(adGroup + ",");
sb.append(searchCount + ",");
writeToFile(sb, getNewFileName());
}
} catch (Exception e) {
e.printStackTrace();
}
}
private long getSearchCount(String url) {
long recordCount = 0;
try {
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpGet getRequest = new HttpGet(
"api.com/querysearch?q="
+ url);
getRequest.addHeader("accept", "application/json");
HttpResponse response = httpClient.execute(getRequest);
if (response.getStatusLine().getStatusCode() != 200) {
throw new RuntimeException("Failed : HTTP error code : "
+ response.getStatusLine().getStatusCode());
}
BufferedReader br = new BufferedReader(new InputStreamReader(
(response.getEntity().getContent())));
String output;
while ((output = br.readLine()) != null) {
try {
JSONObject json = (JSONObject) new JSONParser()
.parse(output);
JSONObject result = (JSONObject) json.get("result");
recordCount = (long) result.get("count");
System.out.println(url + "=" + recordCount);
} catch (Exception e) {
System.out.println(e.getMessage());
}
}
httpClient.getConnectionManager().shutdown();
} catch (Exception e) {
e.getStackTrace();
}
return recordCount;
}
Since remote calls are slower than local disk access, you'll want to in some way parallelize or batch your remote calls. If you can't make batch calls to the remote API, but it allows multiple concurrent reads, then perhaps you want to use something like a thread pool to make the remote calls:
public void readCSVFile() {
// exception handling ignored for space
br = new BufferedReader(new FileReader(getFileName()));
List<Future<String>> futures = new ArrayList<Future<String>>();
ExecutorService pool = Executors.newFixedThreadPool(5);
while ((line = br.readLine()) != null) {
final String[] splitLine = line.split(cvsSplitBy);
futures.add(pool.submit(new Callable<String> {
public String call() {
long searchCount = getSearchCount(splitLine[2]);
return new StringBuilder()
.append(splitLine[0]+ ",")
.append(splitLine[1]+ ",")
.append(searchCount + ",")
.toString();
}
}));
}
for (Future<String> fs: futures) {
writeToFile(fs.get(), getNewFileName());
}
pool.shutdown();
}
Ideally, though, you'd really want to make a single batch read from the remote API if at all possible.
Related
I'm open for other ways to do this, but this is my code:
public class Client {
public static void main (String [] args) {
try(Socket socket = new Socket("localhost", 7789)) {
BufferedReader incoming = new BufferedReader(new InputStreamReader(socket.getInputStream()));
PrintWriter outgoing = new PrintWriter(socket.getOutputStream(),true);
StringBuilder sb = new StringBuilder();
Scanner scanner = new Scanner(System.in);
String send = "";
String response = "";
while (!send.equals("logout")){
System.out.println("Enter Command: ");
send = scanner.nextLine();
outgoing.println(send);
while ((response = incoming.readLine()) != null) {
System.out.println(response);
sb.append(response);
sb.append('\n');
}
}
} catch (IOException e) {
System.out.println("Client Error: "+ e.getMessage());
}
}
}
I do get response from the server, but the program is getting stuck in the inner while loop while ((response = incoming.readLine()) != null), so i can't enter a second command. how do i break the loop if the incoming response is done ?
The problem is that incoming.readLine() will only return null if the socket is closed, otherwise it will block and wait for more input from the server.
If you can change the server, you could add some marking that the request was fully processed and then check it like this while ((response = incoming.readLine()) != "--finished--").
If you cannot, try this:
while(response.isEmpty()){
if(incoming.ready()){ //check if there is stuff to read
while ((response = incoming.readLine()) != null){
System.out.println(response);
sb.append(response);
sb.append('\n');
}
}
}
I'm calling a BufferedReader to get a HTTP response body (if it exists) and stick it in one long string variable. Sometimes when I attempt to do this I get the error java.io.IOException: stream is closed sometimes when the while loop below starts to execute. I don't understand why. I'd like to make sure the object isn't null and that is has a response body before reading in the object.
BufferedReader readBuffer = null;
if (connection.getResponseCode() >= 200 && connection.getResponseCode() <= 299) {
readBuffer = new BufferedReader(new InputStreamReader(connection.getInputStream()));
} else {
readBuffer = new BufferedReader(new InputStreamReader(connection.getErrorStream()));
}
if(readBuffer != null) {
// Get the response body output from the server
StringBuilder calculatedOutput = new StringBuilder();
String rawOutputLine;
while ((rawOutputLine = readBuffer.readLine()) != null) {
calculatedOutput.append(rawOutputLine);
}
Logger.debug(String.format("BODY: %s", calculatedOutput.toString()));
readBuffer.close();
Try this to read the response:
BufferedReader readBuffer = null;
try {
if (connection.getResponseCode() >= 200 && connection.getResponseCode() <= 299) {
readBuffer = new BufferedReader(new InputStreamReader(connection.getInputStream()));
} else {
readBuffer = new BufferedReader(new InputStreamReader(connection.getErrorStream()));
}
// Get the response body output from the server
StringBuilder calculatedOutput = new StringBuilder();
String rawOutputLine;
while ((rawOutputLine = readBuffer.readLine()) != null) {
calculatedOutput.append(rawOutputLine);
}
Logger.debug(String.format("BODY: %s", calculatedOutput.toString()));
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
readBuffer.close();
} catch (IOException e) {
e.printStackTrace();
}
}
Try this code:
BufferedReader readBuffer = null;
if (connection.getResponseCode() >= 200 && connection.getResponseCode() <= 299) {
readBuffer = new BufferedReader(new InputStreamReader(connection.getInputStream()));
} else {
readBuffer = new BufferedReader(new InputStreamReader(connection.getErrorStream()));
}
if(readBuffer != null) {
// Get the response body output from the server
StringBuilder calculatedOutput = new StringBuilder();
String rawOutputLine;
if(readBuffer.ready()) {
while ((rawOutputLine = readBuffer.readLine()) != null) {
calculatedOutput.append(rawOutputLine);
}
}
Logger.debug(String.format("BODY: %s", calculatedOutput.toString()));
readBuffer.close();
I am working on an Android project. I am creating a graph and populating the graph with json content. My problem is that i keep getting this error and i am not sure why. java.lang.NegativeArraySizeException: -2
My Log.v shows the content of the array. So it's not empty. Maybe i am missing something.
I go through the rest api and add everything to the arraylist resModelList.
In the onPostExecute, I want to add my y-axis values to this array list yVals1.
This is where i get my error. (java.lang.NegativeArraySizeException: -2)
If i add the values like this, I get no error.
yVals1 = new ArrayList<Entry>();
yVals1.add(new Entry(1451606400, 10));
yVals1.add(new Entry(1454284800, 20));
yVals1.add(new Entry(1456790400, 30));
yVals1.add(new Entry(1459468800, 50));
My code
Global variables
ArrayList<ResultModel> resModelList;
ArrayList<Entry> yVals1;
Parse Json
//getResult
public class JSONTask extends AsyncTask<String, String, List<ResultModel>> {
#Override
protected List<ResultModel> doInBackground(String... params) {
BufferedReader reader = null;
HttpURLConnection connection = null;
try {
URL url = new URL(params[0]);
connection = (HttpURLConnection) url.openConnection();
connection.setRequestProperty("Cookie",session_name+"="+session_id);
connection.setRequestProperty("X-CSRF-Token", token);
//connection.setRequestProperty("Accept-Encoding", "identity");
connection.connect();
int length = connection.getContentLength();
InputStream stream = connection.getInputStream();
reader = new BufferedReader(new InputStreamReader(stream));
String line = "";
StringBuffer buffer = new StringBuffer();
while ((line = reader.readLine()) != null){
buffer.append(line);
}
Log.v("TESt", " " + length);
String finalJson = buffer.toString();
JSONArray parentArray = new JSONArray(finalJson);
resModelList = new ArrayList<>();
for(int i=0; i<parentArray.length(); i++){
JSONObject finalObject = parentArray.getJSONObject(i);
ResultModel resModel = new ResultModel();
resModel.setPost_date(finalObject.getString("post_date"));
resModel.setHow_much_has_ocd(finalObject.getString("how_much_has_ocd"));
resModelList.add(resModel);
}
return resModelList;
}catch (MalformedURLException e){
e.printStackTrace();
}catch (IOException e){
e.printStackTrace();
} catch (JSONException e) {
e.printStackTrace();
} finally {
if(connection != null) {
connection.disconnect();
}
try {
if(reader != null){
reader.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
return null;
}
#Override
protected void onPostExecute(List<ResultModel> result) {
super.onPostExecute(result);
if(!resModelList.isEmpty()){
//here is where i get my errors
yVals1 = new ArrayList<Entry>();
for (ResultModel ocd : resModelList){
int score = Integer.parseInt(ocd.getHow_much_has_ocd());
int timeStamp = Integer.parseInt(ocd.getPost_date());
//I get these log values
Log.v("Score: ", " " + score + " Timestamp: " + timeStamp);
yVals1.add(new Entry(timeStamp, score));
}
graph();
Log.v("Not Empty list", "");
}else {
Log.v("Empty list", "");
}
}
}
finalJson log.v
[{"post_date":"1481895820","did_you_avoid":"25","how_much_has_ocd":"81","how_would_you_rate":"82","overall_how_would":"35","were_there_any_distressing":"0","uid":"2"},{"post_date":"1481723564","did_you_avoid":"13","how_much_has_ocd":"10","how_would_you_rate":"13","overall_how_would":"16","were_there_any_distressing":"0","uid":"2"},{"post_date":"1481723488","did_you_avoid":"28","how_much_has_ocd":"56","how_would_you_rate":"75","overall_how_would":"32","were_there_any_distressing":"0","uid":"2"},{"post_date":"1481537274","did_you_avoid":"53","how_much_has_ocd":"59","how_would_you_rate":"15","overall_how_would":"71","were_there_any_distressing":"1","uid":"2"},{"post_date":"1481295470","did_you_avoid":"67","how_much_has_ocd":"64","how_would_you_rate":"66","overall_how_would":"57","were_there_any_distressing":"0","uid":"2"},{"post_date":"1481097609","did_you_avoid":"72","how_much_has_ocd":"85","how_would_you_rate":"62","overall_how_would":"64","were_there_any_distressing":"0","uid":"2"},{"post_date":"1480673252","did_you_avoid":"33","how_much_has_ocd":"69","how_would_you_rate":"84","overall_how_would":"37","were_there_any_distressing":"1","uid":"2"},
I am a beginner so it might just be simple mistake.
Thanks in advance
I found the error. It was my graph library.
I am using MPAndroidChart library and you need to sort the data. My back-end was sorted desc. I had to change it to Post date (asc).
Its related to this problem.
NegativeArraySizeException adding Scatter Data to a CombinedChart
I hope this helps someone else.
Am reading data from csv file , i have test for which this data will be the input .
i want it to run as tescase for every set of value. for that am using data provider
The problem is , it is taking only the last set row of data , please help me in debugging the code
For eg : if my csv has following data
name1 id1 text1
name2 id2 text2
name3 id3 text3
it taking only last row name3 id3 text3 and running the test only once not three times.
#DataProvider(name = "test")
public Object[][] provider( ) throws InterruptedException
{
Object[][] returnObject ;
String[] checkpoint = ReadfromCSV();
count = count + 1;
returnObject = new Object[][]{checkpoint };
return returnObject;
}
#Test(description = "Test", groups = "test" , dataProvider = "test")
public void compare(String val1,String val2,String val3,String val4,String val5,String val6,String val7,String val8,String val9,String val10,String val11 ) {
System.out.println("1:" + val1);
System.out.println("4:" + val2);
System.out.println("5:" + val3);
}
#SuppressWarnings("null")
public String[] ReadfromCSV() throws InterruptedException {
String[] data= null;
String csvFile = "F:/sample1.csv";
BufferedReader br = null;
String line = "";
String cvsSplitBy = ",";
try {
br = new BufferedReader(new FileReader(csvFile));
while ((line = br.readLine()) != null) {
// use comma as separator
data= line.split(cvsSplitBy);
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
if (br != null) {
try {
br.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
System.out.println("Done");
return data;
}
You should read entire file in data provider and return iterator of test cases. Here is some pseudocode for data provider. Notice that I used List<String []> to store test cases instead of Object[][]. This allows you do define test cases dynamically.
#DataProvider(name = "test")
public Iterator<Object []> provider( ) throws InterruptedException
{
List<Object []> testCases = new ArrayList<>();
String[] data= null;
//this loop is pseudo code
br = new BufferedReader(new FileReader(csvFile));
while ((line = br.readLine()) != null) {
// use comma as separator
data= line.split(cvsSplitBy);
testCases.add(data);
}
return testCases.iterator();
}
public String[][] ReadfromCSV() throws InterruptedException {
int count =0;
String[] data= null;
String returnObj[][] = null;
//System.out.println(System.getProperty("user.dir"));
String csvFile = System.getProperty("user.dir")+ "/src/test/resources/testdata.csv";
BufferedReader br = null;
String line = "";
String cvsSplitBy = ",";
ArrayList<String> content = new ArrayList<String>();
try {
//this loop is pseudo code
br = new BufferedReader(new FileReader(csvFile));
int datalength = 0;
int listsize =0;;
while ((line = br.readLine()) != null) {
// use comma as separator
content.add(line);
}
System.out.println(content);
listsize = content.size();
datalength = content.get(0).split(cvsSplitBy).length;
returnObj = new String[listsize][datalength];
for (int i = 0; i<listsize; i++) {
data = content.get(i).split(cvsSplitBy);
for (int j=0; j< datalength ; j++) {
returnObj[i][j] = data[j];
}
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
if (br != null) {
try {
br.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
System.out.println("Done");
return returnObj;
}}
I have to hit the public function(downloadDataQUANDL) about 1500 times in a loop to download data.
But my program after running for about 50-100 network calls, gets stuck randomly on one url.
Not sure why it is getting stuck as there is a maximum number of retries defined. So after exhausting all the attempts the program should skip and move to the next url.
Any ideas to fix the issue? please help
public List<Record> downloadDataQUANDL(String symbol){
List<Record> records = new ArrayList<Record>();
Calendar fromdate = Calendar.getInstance();
fromdate.add(Calendar.DATE, -365);
Date todate = new Date();
String url = getFeederURLQuandl(symbol, new Date(fromdate.getTimeInMillis()), todate);
InputStreamReader inputStreamReader = null;
BufferedReader bufferedReader = null;
try{
HttpResponse response = retryQuandl(url);
if(response.getStatusLine().getStatusCode() == 200){
inputStreamReader = new InputStreamReader(response.getEntity().getContent());
bufferedReader = new BufferedReader(inputStreamReader);
String line;
while ((line = bufferedReader.readLine()) != null) {
if (!Character.isDigit(line.charAt(0))) {
continue;
}
String[] values = line.split(",");
Date date = (Date) new SimpleDateFormat("yyyy-MM-dd").parse(values[0]);
records.add(new Record(date, Double.parseDouble(values[1]),
Double.parseDouble(values[2]), Double.parseDouble(values[3]),
Double.parseDouble(values[5]), Double.parseDouble(values[6])));
}
}else{
System.out.println("Data fetch failed for " + symbol);
return null;
}
}catch(Exception e){
System.out.println("Data fetch failed for " + symbol);
return null;
}
return records;
}
private HttpResponse retryQuandl(String url){
RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(30*1000).setConnectTimeout(30*1000).build();
HttpClient client = HttpClientBuilder.create().setDefaultRequestConfig(requestConfig).build();
HttpGet request = new HttpGet(url);
System.out.println(url);
int responsecode=0;
int nooftries = 1;
HttpResponse response=null;
while(responsecode != 200 && nooftries <= 5){
try{
response = client.execute(request);
responsecode = response.getStatusLine().getStatusCode();
}catch(Exception e){}
try {
Thread.sleep(nooftries * 1000);
} catch (InterruptedException e) {}
nooftries++;
}
return response;
}