Related
We have JAX RS implementation which needs to send back JSON output. But the response size is huge. And the client expects the same synchronously.
Hence I tried to use StreamingOutput... but the client is not really getting the data in chunks.
Below is sample snippet:
Server Side
streamingOutput = new StreamingOutput() {
#Override
public void write(OutputStream out) throws IOException, WebApplicationException {
JsonGenerator jsonGenerator = mapper.getFactory().createGenerator(out);
jsonGenerator.writeStartArray();
for(int i=0; i < 10; i++) {
jsonGenerator.writeStartObject();
jsonGenerator.writeStringField("Response_State", "Response State - " + i);
jsonGenerator.writeStringField("Response_Report", "Response Report - " + i);
jsonGenerator.writeStringField("Error_details", "Error Details - " + i);
jsonGenerator.writeEndObject();;
jsonGenerator.flush();
try {
Thread.currentThread().sleep(2000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
jsonGenerator.writeEndArray();
jsonGenerator.close();
}
};
return Response.status(200).entity(streamingOutput).build();
Client
HttpClient client = HttpClientBuilder.create().build();
HttpPost post = new HttpPost("http://localhost:8080/AccessData/FetchReport");
post.setHeader("Content-type", "application/json");
ResponseHandler<HttpResponse> responseHandler = new BasicResponseHandler();
StringEntity entity = new StringEntity(jsonRequest); //jsonRequest is request string
post.setEntity(entity);
HttpResponse response = client.execute(post);
BufferedReader buffReader = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
JsonParser jsonParser = new JsonFactory().createParser(buffReader);
while(jsonParser.nextToken() != JsonToken.END_OBJECT) {
System.out.println(jsonParser.getCurrentName() + ":" + jsonParser.getCurrentValue());
}
String output;
while((output = buffReader.readLine()) != null) {
System.out.println(output);
}
In the server side code, I am putting sleep call just to simulate a gap between chunks of data. What I need is that the client should receive chunks of data as and when it is thrown back by the server.
But here the client gets the response in entirety always.
Any possible solution?
Thanks in advance.
It looks like the client side is not implemented correctly: reading the array of the objects using the parser.
Also, I would like to recommend reading and writing a data transfer object instead of low level field-by-field reading and writing.
For the sake of completeness, here is a complete draft example that uses: Jersey 2.25.1, Jetty 9.2.14.v20151106.
Common
ResponseData class
import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;
public class ResponseData {
private final String responseState;
private final String responseReport;
private final String errorDetails;
#JsonCreator
public ResponseData(
#JsonProperty("Response_State") final String responseState,
#JsonProperty("Response_Report") final String responseReport,
#JsonProperty("Error_details") final String errorDetails) {
this.responseState = responseState;
this.responseReport = responseReport;
this.errorDetails = errorDetails;
}
public String getResponseState() {
return this.responseState;
}
public String getResponseReport() {
return this.responseReport;
}
public String getErrorDetails() {
return this.errorDetails;
}
#Override
public String toString() {
return String.format(
"ResponseData: responseState: %s; responseReport: %s; errorDetails: %s",
this.responseState,
this.responseReport,
this.errorDetails
);
}
}
Service
ServerProgram class
import java.net.URI;
import org.glassfish.jersey.jackson.JacksonFeature;
import org.glassfish.jersey.jetty.JettyHttpContainerFactory;
import org.glassfish.jersey.server.ResourceConfig;
public class ServerProgram {
public static void main(final String[] args) {
final URI uri = URI.create("http://localhost:8080/");
final ResourceConfig resourceConfig = new ResourceConfig(TestResource.class);
resourceConfig.register(JacksonFeature.class);
JettyHttpContainerFactory.createServer(uri, resourceConfig);
}
}
TestResource class
import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.io.OutputStream;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.WebApplicationException;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.StreamingOutput;
#Path("/")
public class TestResource {
#GET
#Produces(MediaType.APPLICATION_JSON)
public Response getData() {
final StreamingOutput streamingOutput = new JsonStreamingOutput();
return Response.status(200).entity(streamingOutput).build();
}
private static class JsonStreamingOutput implements StreamingOutput {
#Override
public void write(final OutputStream outputStream) throws IOException, WebApplicationException {
final ObjectMapper objectMapper = new ObjectMapper();
final JsonFactory jsonFactory = objectMapper.getFactory();
try (final JsonGenerator jsonGenerator = jsonFactory.createGenerator(outputStream)) {
jsonGenerator.writeStartArray();
for (int i = 0; i < 10; i++) {
final ResponseData responseData = new ResponseData(
"Response State - " + i,
"Response Report - " + i,
"Error Details - " + i
);
jsonGenerator.writeObject(responseData);
jsonGenerator.flush();
try {
Thread.currentThread().sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
jsonGenerator.writeEndArray();
}
}
}
}
Client
ClientProgram class
import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.core.JsonToken;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.BufferedInputStream;
import java.io.IOException;
import java.io.InputStream;
import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.core.MediaType;
import org.glassfish.jersey.client.ClientProperties;
public class ClientProgram {
public static void main(final String[] args) throws IOException {
Client client = null;
try {
client = ClientBuilder.newClient();
client.property(ClientProperties.READ_TIMEOUT, 10000);
try (final InputStream inputStream = client
.target("http://localhost:8080/")
.request(MediaType.APPLICATION_JSON)
.get(InputStream.class);
final BufferedInputStream bufferedInputStream = new BufferedInputStream(inputStream)) {
processStream(bufferedInputStream);
}
} finally {
if (client != null) {
client.close();
}
}
}
private static void processStream(final InputStream inputStream) throws IOException {
final ObjectMapper objectMapper = new ObjectMapper();
final JsonFactory jsonFactory = objectMapper.getFactory();
try (final JsonParser jsonParser = jsonFactory.createParser(inputStream)) {
final JsonToken arrayToken = jsonParser.nextToken();
if (arrayToken == null) {
// TODO: Return or throw exception.
return;
}
if (!JsonToken.START_ARRAY.equals(arrayToken)) {
// TODO: Return or throw exception.
return;
}
// Iterate through the objects of the array.
while (JsonToken.START_OBJECT.equals(jsonParser.nextToken())) {
final ResponseData responseData = jsonParser.readValueAs(ResponseData.class);
System.out.println(responseData);
}
}
}
}
Hope this helps.
I am working on an Android app (min API 8) and I want to make an activity where there is a preloader GIF running while some tasks are executed in the background.
These tasks involve connection to a database and queries. So in some way, I want to achieve something that involves techniques like you would use to make a progress bar.
I know I can't make a connection in the main thread in Java so I made a class that does this in another thread. The connection works fine, but I can't make the whole behaviour work properly. More precisely, if I use thread.join()then the main thread is blocked (which is the opposite of what I want) and if I don't use it, the code of the main thread goes on and finishes before the background process has terminated.
Can someone help me with this please ?
This is the class I created to manage POST queries to an URL :
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.Reader;
import java.io.UnsupportedEncodingException;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLEncoder;
import java.util.concurrent.atomic.AtomicReference;
public class Request
{
private URL m_url;
private StringBuilder m_parameters;
private HttpURLConnection m_connection;
public Request(String url)
{
try
{
m_url = new URL(url);
}
catch(MalformedURLException exception)
{
System.exit(1);
}
m_connection = null;
m_parameters = new StringBuilder();
}
public void put(String key, String value)
{
if(m_parameters.length() != 0)
m_parameters.append('&');
try
{
m_parameters.append(URLEncoder.encode(key, "UTF-8"));
m_parameters.append('=');
m_parameters.append(URLEncoder.encode(value, "UTF-8"));
}
catch(UnsupportedEncodingException exception)
{
System.exit(1);
}
}
private void sendRequest()
{
try
{
byte[] data = m_parameters.toString().getBytes("UTF-8");
m_connection = (HttpURLConnection) m_url.openConnection();
m_connection.setAllowUserInteraction(true);
m_connection.setRequestMethod("POST");
m_connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded;charset=utf-8");
m_connection.setRequestProperty("Content-Length", String.valueOf(data.length));
m_connection.setDoOutput(true);
m_connection.getOutputStream().write(data);
}
catch(Exception exception)
{
System.exit(1);
}
}
private String getResponse()
{
String response = "";
try
{
if(m_connection.getResponseCode() == HttpURLConnection.HTTP_OK)
{
Reader reader = new BufferedReader(new InputStreamReader(m_connection.getInputStream(), "UTF-8"));
int c;
do {
c = reader.read();
response += (char) c;
}
while (c >= 0);
}
}
catch(IOException exception)
{
System.exit(1);
}
m_connection.disconnect();
return response;
}
public String get()
{
final AtomicReference<String> response = new AtomicReference<>();
Thread thread = new Thread(new Runnable()
{
public void run()
{
sendRequest();
response.set(getResponse());
}
});
thread.start();
/*
try
{
thread.join();
}
catch(InterruptedException exception)
{
System.exit(1);
}
*/
return response.get();
}
}
And this is how I use it in the main thread (the activity) :
Request request = new Request("http://posttestserver.com/post.php?dump&html&sleep=5");
request.put("name", "bob");
String response = request.get();
Thanks.
Okay, I finally solved that problem using standard IntentServiceof Android :
https://developer.android.com/training/run-background-service/create-service.html
This tutorial helped a lot too :
http://code.tutsplus.com/tutorials/android-fundamentals-intentservice-basics--mobile-6183
Here is my new class Request:
import android.app.IntentService;
import android.content.Intent;
import android.os.Bundle;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.Reader;
import java.io.UnsupportedEncodingException;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLEncoder;
import java.util.Set;
public class Request extends IntentService
{
private URL m_url;
private StringBuilder m_parameters;
private HttpURLConnection m_connection;
public Request()
{
super("Request");
m_url = null;
m_connection = null;
m_parameters = new StringBuilder();
}
public void put(String key, String value)
{
if(m_parameters.length() != 0)
m_parameters.append('&');
try
{
m_parameters.append(URLEncoder.encode(key, "UTF-8"));
m_parameters.append('=');
m_parameters.append(URLEncoder.encode(value, "UTF-8"));
}
catch(UnsupportedEncodingException exception)
{
System.exit(1);
}
}
private void sendRequest()
{
try
{
byte[] data = m_parameters.toString().getBytes("UTF-8");
m_connection = (HttpURLConnection) m_url.openConnection();
m_connection.setAllowUserInteraction(true);
m_connection.setRequestMethod("POST");
m_connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded;charset=utf-8");
m_connection.setRequestProperty("Content-Length", String.valueOf(data.length));
m_connection.setDoOutput(true);
m_connection.getOutputStream().write(data);
}
catch(Exception exception)
{
System.exit(1);
}
}
private String getResponse()
{
String response = "";
try
{
if(m_connection.getResponseCode() == HttpURLConnection.HTTP_OK)
{
Reader reader = new BufferedReader(new InputStreamReader(m_connection.getInputStream(), "UTF-8"));
int c;
do {
c = reader.read();
response += (char) c;
}
while (c >= 0);
}
}
catch(IOException exception)
{
System.exit(1);
}
m_connection.disconnect();
return response;
}
protected void onHandleIntent(Intent intent)
{
Bundle bundle = intent.getExtras();
Set<String> keys = bundle.keySet();
for(String key : keys)
put(key, bundle.getString(key));
try
{
m_url = new URL(bundle.getString("url"));
}
catch(MalformedURLException exception)
{
System.exit(1);
}
sendRequest();
String response = getResponse();
Intent broadcast = new Intent();
broadcast.setAction(ReadyActivity.ResponseReceiver.m_broadcastKey);
broadcast.putExtra("response", response);
sendBroadcast(broadcast);
}
}
Here is the ResponseReceiversubclass of my main activity :
public class ResponseReceiver extends BroadcastReceiver
{
public static final String m_broadcastKey = "Uz258e3wZm77Z3Tdebn7PqgW3CLBJ8";
public void onReceive(Context context, Intent intent)
{
String response = intent.getStringExtra("response");
clear();
((TextView) m_widgets.get("text")).setText("Finally found someone !" + response);
show(m_widgets.get("text"));
((Button) m_widgets.get("button1")).setText("OK let's do this !");
show(m_widgets.get("button1"));
((Button) m_widgets.get("button2")).setText("Later ?");
show(m_widgets.get("button2"));
unregisterReceiver(m_receiver);
}
}
Then I also had to instantiate the ResponseReceiver in the activity:
private ResponseReceiver m_receiver;
...
IntentFilter filter = new IntentFilter(ResponseReceiver.m_broadcastKey);
m_receiver = new ResponseReceiver();
registerReceiver(m_receiver, filter);
And finally call the service :
Intent service = new Intent(this, Request.class);
ArrayList<CharSequence> parameters = new ArrayList<>();
service.putExtra("url", "http://posttestserver.com/post.php?dump&html&sleep=10");
service.putExtra("username", "bob");
service.putExtra("age", "20");
startService(service);
Maybe it will help someone in the future.
Here below is my client side script for WebSockets. In this code I have defined a WebSockets object with IP address and port.
Client Script:
var webSocket =
new WebSocket('ws://localhost:8080/Spring4JSONHandling/websocket');
webSocket.onmessage = function processMessage(message)
{
var jsonData = JSON.parse(message.data);
jsonforlogout = jsonData;
//var user=JSON.parse(username.data);
console.log(jsonData); //to print
if (jsonData.message != null){
var msgAdd = JSON.stringify(jsonData.message);
msgAdd = msgAdd.replace(/[&\/\\#,+()$~%.'"*?<>{}]/g, '');
alert("3"+msgAdd);
var xyz = msgAdd.split(" : ");
var logged = '${me}';
alert("logged" + logged);
alert("xyz[0]" + xyz[0]);
if (logged == xyz[0]){
alert("1");
$("#chat" + xyz[0]).chatbox("option", "boxManager").addMsg(xyz[0], xyz[1]);
} else{
alert("2");
$("#chat" + xyz[0]).chatbox("option", "boxManager").addMsg(xyz[0], xyz[1]);
}
}
/* if(jsonData.message=='close'){websocket.close();} */
if (jsonData.users != null)
{
document.getElementById("chatusers").innerHTML = "";
var loggedInUser = '${me}';
var i = 0;
while (i < jsonData.users.length)
{
var onlineAdd = JSON.stringify(jsonData.users[i]);
onlineAdd = onlineAdd.replace(/[&\/\\#,+()$~%.'":*?<>{}]/g, '');
if (loggedInUser != onlineAdd){
var clone = "<a href='#' onclick=\"clickFun('" + onlineAdd.trim() + "');\" class='usr' data-chat-id='chat" + onlineAdd + "' data-chat-fname='" + onlineAdd + "' data-chat-lname='" + onlineAdd + "'"
+ " data-chat-status='online' data-chat-alertmsg='' data-chat-alertshow='true' data-rel='popover-hover' data-placement='right' data-html='true'"
+ " data-content=\"<div class='usr-card'>"
+ "<img src='Spring4JSONHandling/resources/img/1.png' alt='Jessica Dolof'>"
+ "<div class='usr-card-content'>"
+ "<h3>Jessica Dolof</h3>"
+ "<p>Sales Administrator</p>"
+ "</div>"
+ "</div>\"><i></i>" + onlineAdd + "</a>";
$('#chatusers').append(clone);
}
i++;
}
}
//or(i = 0; i < responselist.data.length; i++) {
/* var i=0;
while(i<jsonData.users.length)
{
var comboitem = document.createElement("option");
comboitem.text = jsonData.users[i];//"utsav";//
comboitem.value = jsonData.users[i];//"10";
usercombo.options.add(comboitem);
i++;
} */
}
function sendMessage(txt) {
//alert(messageText.value+ " h1");
webSocket.send(txt);
// messageText.value = "";
}
window.onbeforeunload = function () {
webSocket.onclose = function() {};
webSocket.close();
};
Here is the code for the server end point. In this code I have defined a WebSockets object with a server endpoint.
Server Code:
package com.outbottle.controllers;
import java.io.IOException;
import java.io.StringWriter;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Set;
import javax.json.Json;
import javax.json.JsonArrayBuilder;
import javax.json.JsonObject;
import javax.json.JsonWriter;
import javax.servlet.http.HttpSession;
import javax.websocket.EndpointConfig;
import javax.websocket.OnClose;
import javax.websocket.OnMessage;
import javax.websocket.OnOpen;
import javax.websocket.Session;
import javax.websocket.server.ServerEndpoint;
#ServerEndpoint("/websocket")
public class WebSocketTest {
static Set<Session> chatroomUsers= Collections.synchronizedSet(new HashSet<Session>());
private final static HashMap<String, WebSocketTest> sockets = new HashMap<>();
private String myUniqueId;
private String getMyUniqueId() {
// unique ID from this class' hash code
return Integer.toHexString(this.hashCode());
}
#OnOpen
public void handleOpen(Session userSession) throws IOException {
chatroomUsers.add(userSession);
System.out.println("user added"); //user added
this.myUniqueId = this.getMyUniqueId();
System.out.println("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"+ myUniqueId);
WebSocketTest.sockets.put(this.myUniqueId, this);
System.out.println("###################################"+sockets);
}
#OnMessage
public void handleMessage(String message, Session userSession) throws IOException{
String username= (String) userSession.getUserProperties().get("username");
Iterator<Session> itr=chatroomUsers.iterator();
if(username==null){
userSession.getUserProperties().put("username", message);
userSession.getBasicRemote().sendText(buildJsonMessageData("System","You are now connected as "+message));
while (itr.hasNext()) (itr.next()).getBasicRemote().sendText(buildJsonUsersData()); }
else {
while (itr.hasNext()) {
itr.next().getBasicRemote().sendText(buildJsonMessageData(username,message));
}
}
}
#OnClose
public void handleClose(Session userSession) throws IOException {
// TODO Auto-generated method stub
System.out.println("user logout");
chatroomUsers.remove(userSession);
Iterator<Session> itr = chatroomUsers.iterator();
while (itr.hasNext()) (itr.next()).getBasicRemote().sendText(buildJsonUsersData());
}
private String buildJsonUsersData() {
Iterator<String> itr= getUserNames().iterator();
JsonArrayBuilder jsonArrayBuilder = Json.createArrayBuilder();
while (itr.hasNext()) jsonArrayBuilder.add((String)itr.next());
return Json.createObjectBuilder().add("users", jsonArrayBuilder).build().toString();
}
private String buildJsonMessageData(String username, String message)
{
JsonObject jsonObject=Json.createObjectBuilder().add("message", username+" : "+ message).build();
StringWriter stringWriter= new StringWriter();
try (JsonWriter jsonWriter = Json.createWriter(stringWriter)){
jsonWriter.write(jsonObject);
}
return stringWriter.toString();
}
private Set<String> getUserNames()
{
HashSet<String> returnSet = new HashSet<String>();
Iterator<Session> itr= chatroomUsers.iterator();
System.out.println("######################");
while (itr.hasNext())
{
returnSet.add(itr.next().getUserProperties().get("username").toString());
}
return returnSet;
}
}
The problem is when I send a message to the user on a different IP, all users with the same name get the message. I need to send the message to a particular session only.
It looks like you are iterating through all the sessions chatroomUsers in the handleMessage function.
If you only want to send the message to a particular user/session, do you not need to check for your username property here?
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 7 years ago.
Improve this question
I am developing a web crawler application. When i run the program i am getting these error messages below:
i've got these errors after running the program for more that 3 hours. I tried to allocate memory by changing eclipse.ini setting to 2048 MB of ram as it was answered in this topic but still get the same errors after 3 hours or less. I should run the program for more that 2-3 days non-stopping to get analyse the results.
Can you tell me what i am missing here to get these error below ?
These are my classes:
seeds.txt
http://www.stanford.edu
http://www.archive.org
WebCrawler.java
package pkg.crawler;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.PrintWriter;
import java.net.MalformedURLException;
import java.net.SocketTimeoutException;
import java.util.*;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.PriorityBlockingQueue;
import java.util.concurrent.TimeUnit;
import org.jsoup.HttpStatusException;
import org.jsoup.UnsupportedMimeTypeException;
import org.joda.time.DateTime;
public class WebCrawler {
public static Queue <LinkNodeLight> queue = new PriorityBlockingQueue <> (); // priority queue
public static final int n_threads = 5; // amount of threads
private static Set<String> processed = new LinkedHashSet <> (); // set of processed urls
private PrintWriter out; // output file
private PrintWriter err; // error file
private static Integer cntIntra = new Integer (0); // counters for intra- links in the queue
private static Integer cntInter = new Integer (0); // counters for inter- links in the queue
private static Integer dub = new Integer (0); // amount of skipped urls
public static void main(String[] args) throws Exception {
System.out.println("Running web crawler: " + new Date());
WebCrawler webCrawler = new WebCrawler();
webCrawler.createFiles();
try (Scanner in = new Scanner(new File ("seeds.txt"))) {
while (in.hasNext()) {
webCrawler.enque(new LinkNode (in.nextLine().trim()));
}
} catch (IOException e) {
e.printStackTrace();
return;
}
webCrawler.processQueue();
webCrawler.out.close();
webCrawler.err.close();
}
public void processQueue(){
/* run in threads */
Runnable r = new Runnable() {
#Override
public void run() {
/* queue may be empty but process is not finished, that's why we need to check if any links are being processed */
while (true) {
LinkNode link = deque();
if (link == null)
continue;
link.setStartTime(new DateTime());
boolean process = processLink(link);
link.setEndTime(new DateTime());
if (!process)
continue;
/* print the data to the csv file */
if (link.getStatus() != null && link.getStatus().equals(LinkNodeStatus.OK)) {
synchronized(out) {
out.println(getOutputLine(link));
out.flush();
}
} else {
synchronized(err) {
err.println(getOutputLine(link));
err.flush();
}
}
}
}
};
/* run n_threads threads which perform dequeue and process */
LinkedList <Thread> threads = new LinkedList <> ();
for (int i = 0; i < n_threads; i++) {
threads.add(new Thread(r));
threads.getLast().start();
}
for (Thread thread : threads) {
try {
thread.join();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
/* returns true if link was actually processed */
private boolean processLink(LinkNode inputLink) {
String url = getUrlGeneralForm(inputLink);
boolean process = true;
synchronized (processed) {
if (processed.contains(url)) {
process = false;
synchronized (dub) {dub++;}
} else
processed.add(url);
}
/* start processing only if the url have not been processed yet or not being processed */
if (process) {
System.out.println("Processing url " + url);
List<LinkNodeLight> outputLinks = parseAndWieghtResults(inputLink);
for (LinkNodeLight outputLink : outputLinks) {
String getUrlGeneralForumOutput = getUrlGeneralForm(outputLink);
/* add the new link to the queue only if it has not been processed yet */
process = true;
synchronized (processed) {
if (processed.contains(getUrlGeneralForumOutput)) {
process = false;
synchronized (dub) {dub++;}
}
}
if (process) {
enque(outputLink);
}
}
return true;
}
return false;
}
void enque(LinkNodeLight link){
link.setEnqueTime(new DateTime());
/* the add method requires implicit priority */
synchronized (queue) {
if (link.interLinks)
synchronized (cntInter) {cntInter++;}
else
synchronized (cntIntra) {cntIntra++;}
//queue.add(link, 100 - (int)(link.getWeight() * 100.f));
queue.add(link);
}
}
/**
* Picks an element from the queue
* #return top element from the queue or null if the queue is empty
*/
LinkNode deque(){
/* link must be checked */
LinkNode link = null;
synchronized (queue) {
link = (LinkNode) queue.poll();
if (link != null) {
link.setDequeTime(new DateTime());
if (link.isInterLinks())
synchronized (cntInter) {cntInter--;}
else
synchronized (cntIntra) {cntIntra--;}
}
}
return link;
}
private void createFiles() {
/* create output file */
try {
out = new PrintWriter(new BufferedWriter(new FileWriter("CrawledURLS.csv", false)));
out.println(generateHeaderFile());
} catch (IOException e) {
System.err.println(e);
}
/* create error file */
try {
err = new PrintWriter(new BufferedWriter(new FileWriter("CrawledURLSERROR.csv", false)));
err.println(generateHeaderFile());
} catch (IOException e) {
System.err.println(e);
}
}
/**
* formats the string so it can be valid entry in csv file
* #param s
* #return
*/
private static String format(String s) {
// replace " by ""
String ret = s.replaceAll("\"", "\"\"");
// put string into quotes
return "\"" + ret + "\"";
}
/**
* Creates the line that needs to be written in the outputfile
* #param link
* #return
*/
public static String getOutputLine(LinkNode link){
StringBuilder builder = new StringBuilder();
builder.append(link.getParentLink()!=null ? format(link.getParentLink().getUrl()) : "");
builder.append(",");
builder.append(link.getParentLink()!=null ? link.getParentLink().getIpAdress() : "");
builder.append(",");
builder.append(link.getParentLink()!=null ? link.getParentLink().linkProcessingDuration() : "");
builder.append(",");
builder.append(format(link.getUrl()));
builder.append(",");
builder.append(link.getDomain());
builder.append(",");
builder.append(link.isInterLinks());
builder.append(",");
builder.append(Util.formatDate(link.getEnqueTime()));
builder.append(",");
builder.append(Util.formatDate(link.getDequeTime()));
builder.append(",");
builder.append(link.waitingInQueue());
builder.append(",");
builder.append(queue.size());
/* Inter and intra links in queue */
builder.append(",");
builder.append(cntIntra.toString());
builder.append(",");
builder.append(cntInter.toString());
builder.append(",");
builder.append(dub);
builder.append(",");
builder.append(new Date ());
/* URL size*/
builder.append(",");
builder.append(link.getSize());
/* HTML file
builder.append(",");
builder.append(link.getFileName());*/
/* add HTTP error */
builder.append(",");
if (link.getParseException() != null) {
if (link.getParseException() instanceof HttpStatusException)
builder.append(((HttpStatusException) link.getParseException()).getStatusCode());
if (link.getParseException() instanceof SocketTimeoutException)
builder.append("Time out");
if (link.getParseException() instanceof MalformedURLException)
builder.append("URL is not valid");
if (link.getParseException() instanceof UnsupportedMimeTypeException)
builder.append("Unsupported mime type: " + ((UnsupportedMimeTypeException)link.getParseException()).getMimeType());
}
return builder.toString();
}
/**
* generates the Header for the file
* #param link
* #return
*/
private String generateHeaderFile(){
StringBuilder builder = new StringBuilder();
builder.append("Seed URL");
builder.append(",");
builder.append("Seed IP");
builder.append(",");
builder.append("Process Duration");
builder.append(",");
builder.append("Link URL");
builder.append(",");
builder.append("Link domain");
builder.append(",");
builder.append("Link IP");
builder.append(",");
builder.append("Enque Time");
builder.append(",");
builder.append("Deque Time");
builder.append(",");
builder.append("Waiting in the Queue");
builder.append(",");
builder.append("QueueSize");
builder.append(",");
builder.append("Intra in queue");
builder.append(",");
builder.append("Inter in queue");
builder.append(",");
builder.append("Dublications skipped");
/* time was printed, but no header was */
builder.append(",");
builder.append("Time");
/* URL size*/
builder.append(",");
builder.append("Size bytes");
/* HTTP errors */
builder.append(",");
builder.append("HTTP error");
return builder.toString();
}
String getUrlGeneralForm(LinkNodeLight link){
String url = link.getUrl();
if (url.endsWith("/")){
url = url.substring(0, url.length() - 1);
}
return url;
}
private List<LinkNodeLight> parseAndWieghtResults(LinkNode inputLink) {
List<LinkNodeLight> outputLinks = HTMLParser.parse(inputLink);
if (inputLink.hasParseException()) {
return outputLinks;
} else {
return URLWeight.weight(inputLink, outputLinks);
}
}
}
HTMLParser.java
package pkg.crawler;
import org.jsoup.Connection;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileOutputStream;
import java.io.FileWriter;
import java.io.IOException;
import java.io.OutputStreamWriter;
import java.io.PrintWriter;
import java.io.Writer;
import java.math.BigInteger;
import java.util.Formatter;
import java.util.HashMap;
import java.util.LinkedList;
import java.util.List;
import java.util.concurrent.TimeUnit;
import java.util.logging.Logger;
import java.security.*;
import java.nio.file.Path;
import java.nio.file.Paths;
public class HTMLParser {
private static final int READ_TIMEOUT_IN_MILLISSECS = (int) TimeUnit.MILLISECONDS.convert(30, TimeUnit.SECONDS);
private static HashMap <String, Integer> filecounter = new HashMap<> ();
public static List<LinkNodeLight> parse(LinkNode inputLink){
List<LinkNodeLight> outputLinks = new LinkedList<>();
try {
inputLink.setIpAdress(IpFromUrl.getIp(inputLink.getUrl()));
String url = inputLink.getUrl();
if (inputLink.getIpAdress() != null) {
url.replace(URLWeight.getHostName(url), inputLink.getIpAdress());
}
Document parsedResults = Jsoup
.connect(url)
.timeout(READ_TIMEOUT_IN_MILLISSECS)
.get();
inputLink.setSize(parsedResults.html().length());
/* IP address moved here in order to speed up the process */
inputLink.setStatus(LinkNodeStatus.OK);
inputLink.setDomain(URLWeight.getDomainName(inputLink.getUrl()));
if (true) {
/* save the file to the html */
String filename = parsedResults.title();//digestBig.toString(16) + ".html";
if (filename.length() > 24) {
filename = filename.substring(0, 24);
}
filename = filename.replaceAll("[^\\w\\d\\s]", "").trim();
filename = filename.replaceAll("\\s+", " ");
if (!filecounter.containsKey(filename)) {
filecounter.put(filename, 1);
} else {
Integer tmp = filecounter.remove(filename);
filecounter.put(filename, tmp + 1);
}
filename = filename + "-" + (filecounter.get(filename)).toString() + ".html";
filename = Paths.get("downloads", filename).toString();
inputLink.setFileName(filename);
/* use md5 of url as file name */
try (PrintWriter out = new PrintWriter(new BufferedWriter(new FileWriter(filename)))) {
out.println("<!--" + inputLink.getUrl() + "-->");
out.print(parsedResults.html());
out.flush();
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
String tag;
Elements tagElements;
List<LinkNode> result;
tag = "a[href";
tagElements = parsedResults.select(tag);
result = toLinkNodeObject(inputLink, tagElements, tag);
outputLinks.addAll(result);
tag = "area[href";
tagElements = parsedResults.select(tag);
result = toLinkNodeObject(inputLink, tagElements, tag);
outputLinks.addAll(result);
} catch (IOException e) {
inputLink.setParseException(e);
inputLink.setStatus(LinkNodeStatus.ERROR);
}
return outputLinks;
}
static List<LinkNode> toLinkNodeObject(LinkNode parentLink, Elements tagElements, String tag) {
List<LinkNode> links = new LinkedList<>();
for (Element element : tagElements) {
if(isFragmentRef(element)){
continue;
}
String absoluteRef = String.format("abs:%s", tag.contains("[") ? tag.substring(tag.indexOf("[") + 1, tag.length()) : "href");
String url = element.attr(absoluteRef);
if(url!=null && url.trim().length()>0) {
LinkNode link = new LinkNode(url);
link.setTag(element.tagName());
link.setParentLink(parentLink);
links.add(link);
}
}
return links;
}
static boolean isFragmentRef(Element element){
String href = element.attr("href");
return href!=null && (href.trim().startsWith("#") || href.startsWith("mailto:"));
}
}
Util.java
package pkg.crawler;
import java.util.Date;
import org.joda.time.DateTime;
import org.joda.time.format.DateTimeFormat;
import org.joda.time.format.DateTimeFormatter;
public class Util {
private static DateTimeFormatter formatter;
static {
formatter = DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss:SSS");
}
public static String linkToString(LinkNode inputLink){
return String.format("%s\t%s\t%s\t%s\t%s\t%s",
inputLink.getUrl(),
inputLink.getWeight(),
formatDate(inputLink.getEnqueTime()),
formatDate(inputLink.getDequeTime()),
differenceInMilliSeconds(inputLink.getEnqueTime(), inputLink.getDequeTime()),
inputLink.getParentLink()==null?"":inputLink.getParentLink().getUrl()
);
}
public static String linkToErrorString(LinkNode inputLink){
return String.format("%s\t%s\t%s\t%s\t%s\t%s",
inputLink.getUrl(),
inputLink.getWeight(),
formatDate(inputLink.getEnqueTime()),
formatDate(inputLink.getDequeTime()),
inputLink.getParentLink()==null?"":inputLink.getParentLink().getUrl(),
inputLink.getParseException().getMessage()
);
}
public static String formatDate(DateTime date){
return formatter.print(date);
}
public static long differenceInMilliSeconds(DateTime dequeTime, DateTime enqueTime){
return (dequeTime.getMillis()- enqueTime.getMillis());
}
public static int differenceInSeconds(Date enqueTime, Date dequeTime){
return (int)((dequeTime.getTime()/1000) - (enqueTime.getTime()/1000));
}
public static int differenceInMinutes(Date enqueTime, Date dequeTime){
return (int)((dequeTime.getTime()/60000) - (enqueTime.getTime()/60000));
}
}
URLWeight.java
package pkg.crawler;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.LinkedList;
import java.util.List;
import java.util.regex.Pattern;
public class URLWeight {
public static List<LinkNodeLight> weight(LinkNode sourceLink, List<LinkNodeLight> links) {
List<LinkNodeLight> interLinks = new LinkedList<>();
List<LinkNodeLight> intraLinks = new LinkedList<>();
for (LinkNodeLight link : links) {
if (isIntraLink(sourceLink, link)) {
intraLinks.add(link);
link.setInterLinks(false);
} else {
interLinks.add(link);
link.setInterLinks(true);
}
}
static boolean isIntraLink(LinkNodeLight sourceLink, LinkNodeLight link){
String parentDomainName = getHostName(sourceLink.getUrl());
String childDomainName = getHostName(link.getUrl());
return parentDomainName.equalsIgnoreCase(childDomainName);
}
public static String getHostName(String url) {
if(url == null){
// System.out.println("Deneme");
return "";
}
String domainName = new String(url);
int index = domainName.indexOf("://");
if (index != -1) {
domainName = domainName.substring(index + 3);
}
for (int i = 0; i < domainName.length(); i++)
if (domainName.charAt(i) == '?' || domainName.charAt(i) == '/') {
domainName = domainName.substring(0, i);
break;
}
/*if (index != -1) {
domainName = domainName.substring(0, index);
}*/
/* have to keep www in order to do replacements with IP */
//domainName = domainName.replaceFirst("^www.*?\\.", "");
return domainName;
}
public static String getDomainName(String url) {
String [] tmp= getHostName(url).split("\\.");
if (tmp.length == 0)
return "";
return tmp[tmp.length - 1];
}
}
PingTaskManager.java
package pkg.crawler;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class PingTaskManager {
private static ExecutorService executor = Executors.newFixedThreadPool(100);
public static void ping (LinkNode e) {
executor.submit(new PingTaks(e));
}
}
class PingTaks implements Runnable {
private LinkNode link;
public PingTaks( LinkNode link ) {
}
#Override
public void run() {
/* link.ping(); */
}
}
LinkNodeStatus.java
package pkg.crawler;
public enum LinkNodeStatus {
OK,
ERROR
}
LinkNodeLight.java
package pkg.crawler;
import org.joda.time.DateTime;
public class LinkNodeLight implements Comparable<LinkNodeLight> {
protected String url;
protected float weight;
protected DateTime enqueTime;
protected boolean interLinks;
public String getUrl() {
return url;
}
public float getWeight() {
return weight;
}
public void setWeight(float weight) {
this.weight = weight;
}
public DateTime getEnqueTime() {
return enqueTime;
}
public LinkNodeLight(String url) {
this.url = url;
}
public void setEnqueTime(DateTime enqueTime) {
this.enqueTime = enqueTime;
}
#Override
public int compareTo(LinkNodeLight link) {
if (this.weight < link.weight) return 1;
else if (this.weight > link.weight) return -1;
return 0;
}
}
LinkNode.java
package pkg.crawler;
import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.Socket;
import java.net.URL;
import java.net.UnknownHostException;
import java.util.Date;
import org.joda.time.DateTime;
public class LinkNode extends LinkNodeLight{
public LinkNode(String url) {
super(url);
}
private String tag;
private LinkNode parentLink;
private IOException parseException = null; // initialize parse Exception with null
private float weight;
private DateTime dequeTime;
private DateTime startTime;
private DateTime endTime;
private LinkNodeStatus status;
private String ipAdress;
private int size;
private String filename;
private String domain;
public DateTime getStartTime() {
return startTime;
}
public void setStartTime(DateTime startTime) {
this.startTime = startTime;
}
public DateTime getEndTime() {
return endTime;
}
public void setEndTime(DateTime endTime) {
this.endTime = endTime;
}
public DateTime getDequeTime() {
return dequeTime;
}
public String getTag() {
return tag;
}
public LinkNode getParentLink() {
return parentLink;
}
public Exception getParseException() {
return parseException;
}
public boolean hasParseException(){
return parseException!=null;
}
public void setDequeTime(DateTime dequeTime) {
this.dequeTime = dequeTime;
}
public void setTag(String tag) {
this.tag = tag;
}
public void setParentLink(LinkNode parentLink) {
this.parentLink = parentLink;
}
public void setParseException(IOException parseException) {
this.parseException = parseException;
}
#Override
public boolean equals(Object o) {
if (this == o) {
return true;
}
if (o == null || getClass() != o.getClass()) {
return false;
}
LinkNode link = (LinkNode) o;
if (url != null ? !url.equals(link.url) : link.url != null) {
return false;
}
return true;
}
#Override
public int hashCode() {
return url != null ? url.hashCode() : 0;
}
public long waitingInQueue(){
return Util.differenceInMilliSeconds( dequeTime,enqueTime );
}
public long linkProcessingDuration(){
return Util.differenceInMilliSeconds( endTime,startTime );
}
#Override
public String toString() {
StringBuilder sb = new StringBuilder("LinkNode{");
sb.append("url='").append(url).append('\'');
sb.append(", score=").append(weight);
sb.append(", enqueTime=").append(enqueTime);
sb.append(", dequeTime=").append(dequeTime);
sb.append(", tag=").append(tag);
if(parentLink!=null) {
sb.append(", parentLink=").append(parentLink.getUrl());
}
sb.append('}');
return sb.toString();
}
public void setStatus(LinkNodeStatus status) {
this.status = status;
}
public LinkNodeStatus getStatus(){
if (status == null) {
status = LinkNodeStatus.ERROR;
}
return status;
}
// check server link is it exist or not
/* this method gives fake errors
public LinkNodeStatus ping () {
boolean reachable = false;
String sanitizeUrl = url.replaceFirst("^https", "http");
try {
HttpURLConnection connection = (HttpURLConnection) new URL(sanitizeUrl).openConnection();
connection.setConnectTimeout(1000);
connection.setRequestMethod("HEAD");
int responseCode = connection.getResponseCode();
System.err.println(url + " " + responseCode);
reachable = (200 <= responseCode && responseCode <= 399);
} catch (IOException exception) {
}
return reachable?LinkNodeStatus.OK: LinkNodeStatus.ERROR;
}*/
public String getIpAdress() {
return ipAdress;
}
public void setIpAdress(String ipAdress) {
this.ipAdress = ipAdress;
}
/* methods for controlling url size */
public void setSize(int size) {
this.size = size;
}
public int getSize() {
return this.size;
}
public void setFileName(String filename) {
this.filename = filename;
}
public String getFileName() {
return this.filename;
}
public String getDomain() {
return domain;
}
public void setDomain(String domain) {
this.domain = domain;
}
}
I tried to allocate memory by changing eclipse.ini setting to 2048 MB of ram as it was answered in this topic but still get the same errors after 3 hours or less.
I hate to repeat myself(*), but in eclipse.ini you set up the memory for Eclipse, which has nothing to do with the memory for your crawler.
When using command line, you need to start it via java -Xmx2G pkg.crawler.WebCrawler.
When starting from Eclipse, you need to add -Xmx2G to the run configuration ("VM arguments" rather than "Program arguments").
(*) Link to a deleted question; requires some reputation to view.
I am using fineuploader within a struts system but am having trouble getting the file list in the server code.
My jsp contains the following code:
$("#fine-uploader").fineUploader({
debug: true,
request: {
endpoint: '/NRGI/act_MultiPhotoUpload.do'
},
deleteFile: {
enabled: true,
endpoint: '/uploads'
},
retry: {
enableAuto: true
}
});
with the following div near the bottom of the page:
<div id="fine-uploader"></div>
The action actMultiPhotoUpload points to a class via the struts.config.xml file:
<action path="/act_MultiPhotoUpload" name="FileUploadForm" scope="request" validate="true"
type="com.solarcentury.nrgi.document.action.MultiUploadAction"
input="/D5_PhotoLibrary.jsp">
</action>
The class MultiUploadAction is taken from your UploadReceiver and is as follows:
public class MultiUploadAction extends Action {
static Static_Env_Object seo = new Static_Env_Object();
private String UPLOAD_NOT_ALLOWED = "exe";
private EnvUtils eu;
// JUST FOR TESTING
private static final File UPLOAD_DIR = new File("Test/uploads");
private static File TEMP_DIR = new File("Test/uploadsTemp");
private static String CONTENT_LENGTH = "Content-Length";
private static int SUCCESS_RESPONSE_CODE = 200;
#Override
public ActionForward perform(ActionMapping mapping,
ActionForm form,
HttpServletRequest request,
HttpServletResponse response)
throws IOException, ServletException {
HttpSession session = request.getSession();
String sId = session.getId();
eu = new EnvUtils(seo.get_Env_Name(), this.getClass().getSimpleName());
/* **************************************** */
/* * The code for the session timeout check */
/* **************************************** */
if (session.getAttribute("SESS_User") == null) {
eu.log("NO SESSION", "Session timed out...");
return (mapping.findForward("globaltimeout"));
}
UserObject suo = new UserObj_Util(session).get_SessUser();
WebAlertMessages wam = new WebAlertMessages(request, suo.get_Language_ID());
DemonUtil du = new DemonUtil(seo.get_Env_Name());
// DateUtils dateUtil = new DateUtils();
RequestParser requestParser = null;
boolean isIframe = request.getHeader("X-Requested-With") == null || !request.getHeader("X-Requested-With").equals("XMLHttpRequest");
try
{
// resp.setContentType(isIframe ? "text/html" : "text/plain");
response.setContentType("text/plain");
response.setStatus(SUCCESS_RESPONSE_CODE);
// resp.addHeader("Access-Control-Allow-Origin", "http://192.168.130.118:8080");
// resp.addHeader("Access-Control-Allow-Credentials", "true");
// resp.addHeader("Access-Control-Allow-Origin", "*");
if (ServletFileUpload.isMultipartContent(request))
{
MultipartUploadParser multipartUploadParser = new MultipartUploadParser(request, TEMP_DIR, request.getSession().getServletContext());
requestParser = RequestParser.getInstance(request, multipartUploadParser);
writeFileForMultipartRequest(requestParser);
writeResponse(response.getWriter(), requestParser.generateError() ? "Generated error" : null, isIframe, false, requestParser);
}
else
{
requestParser = RequestParser.getInstance(request, null);
//handle POST delete file request
if (requestParser.getMethod() != null
&& requestParser.getMethod().equalsIgnoreCase("DELETE"))
{
String uuid = requestParser.getUuid();
handleDeleteFileRequest(uuid, response);
}
else
{
writeFileForNonMultipartRequest(request, requestParser);
writeResponse(response.getWriter(), requestParser.generateError() ? "Generated error" : null, isIframe, false, requestParser);
}
}
} catch (Exception e)
{
eu.log("UploadReceiver","Problem handling upload request" + e);
if (e instanceof MultiUploadAction.MergePartsException)
{
writeResponse(response.getWriter(), e.getMessage(), isIframe, true, requestParser);
}
else
{
writeResponse(response.getWriter(), e.getMessage(), isIframe, false, requestParser);
}
}
return (new ActionForward(mapping.getInput()));
}
And I use the MultipartUploadParser, RequestParser from the server java examples on the website.
When I step through the code, how ever many files I select to upload, the file list is always empty. Obviously I am doing something stupid here, but would appreciate any guidance please.
I have seen a similar support question where the asker was having trouble getting the filelist, also using struts, but there wasn't an answer against the question
Added on 14/11/2013
The full code is as follows:
The full listing of MultiUploadAction.java is as follows:
/*
* To change this template, choose Tools | Templates
* and open the template in the editor.
*/
package com.solarcentury.nrgi.document.action;
/**
*
* #author Roy
*/
import DemonWeb.DmForms.FileUploadForm;
import DemonWeb.DmLogic.DemonUtil;
import DemonWeb.DmLogic.Project;
import DemonWeb.DmSession.Static_Env_Object;
import DemonWeb.DmSession.UserObj_Util;
import DemonWeb.DmSession.UserObject;
import DemonWeb.DmSession.WebAlertMessages;
import DemonWeb.Utils.EnvUtils;
import com.solarcentury.nrgi.document.bean.Document;
import com.solarcentury.nrgi.document.logic.DocumentController;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.FilenameFilter;
import java.io.IOException;
import java.io.InputStream;
import java.io.PrintWriter;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.regex.Pattern;
import java.util.Enumeration;
import java.util.Collections;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.servlet.http.HttpSession;
import org.apache.commons.fileupload.servlet.ServletFileUpload;
import org.apache.commons.io.FileUtils;
import org.apache.commons.io.FilenameUtils;
import org.apache.commons.io.IOUtils;
import org.apache.struts.action.Action;
import org.apache.struts.action.ActionForm;
import org.apache.struts.action.ActionForward;
import org.apache.struts.action.ActionMapping;
import org.apache.struts.upload.FormFile;
/**
*
* #author ajantham
*/
public class MultiUploadAction extends Action {
static Static_Env_Object seo = new Static_Env_Object();
private String UPLOAD_NOT_ALLOWED = "exe";
private EnvUtils eu;
// JUST FOR TESTING
private static final File UPLOAD_DIR = new File("uploads");
private static File TEMP_DIR = new File("uploadsTemp");
private static String CONTENT_LENGTH = "Content-Length";
private static int SUCCESS_RESPONSE_CODE = 200;
#Override
public ActionForward perform(ActionMapping mapping,
ActionForm form,
HttpServletRequest request,
HttpServletResponse response)
throws IOException, ServletException {
HttpSession session = request.getSession();
String sId = session.getId();
eu = new EnvUtils(seo.get_Env_Name(), this.getClass().getSimpleName());
/* **************************************** */
/* * The code for the session timeout check */
/* **************************************** */
if (session.getAttribute("SESS_User") == null) {
eu.log("NO SESSION", "Session timed out...");
return (mapping.findForward("globaltimeout"));
}
UserObject suo = new UserObj_Util(session).get_SessUser();
WebAlertMessages wam = new WebAlertMessages(request, suo.get_Language_ID());
DemonUtil du = new DemonUtil(seo.get_Env_Name());
// DateUtils dateUtil = new DateUtils();
RequestParser requestParser = null;
boolean isIframe = request.getHeader("X-Requested-With") == null || !request.getHeader("X-Requested-With").equals("XMLHttpRequest");
try
{
// resp.setContentType(isIframe ? "text/html" : "text/plain");
response.setContentType("text/plain");
response.setStatus(SUCCESS_RESPONSE_CODE);
// resp.addHeader("Access-Control-Allow-Origin", "http://192.168.130.118:8080");
// resp.addHeader("Access-Control-Allow-Credentials", "true");
// resp.addHeader("Access-Control-Allow-Origin", "*");
if (ServletFileUpload.isMultipartContent(request))
{
MultipartUploadParser multipartUploadParser = new MultipartUploadParser(request, TEMP_DIR, request.getSession().getServletContext());
requestParser = RequestParser.getInstance(request, multipartUploadParser);
writeFileForMultipartRequest(requestParser);
writeResponse(response.getWriter(), requestParser.generateError() ? "Generated error" : null, isIframe, false, requestParser);
}
else
{
requestParser = RequestParser.getInstance(request, null);
//handle POST delete file request
if (requestParser.getMethod() != null
&& requestParser.getMethod().equalsIgnoreCase("DELETE"))
{
String uuid = requestParser.getUuid();
handleDeleteFileRequest(uuid, response);
}
else
{
writeFileForNonMultipartRequest(request, requestParser);
writeResponse(response.getWriter(), requestParser.generateError() ? "Generated error" : null, isIframe, false, requestParser);
}
}
} catch (Exception e)
{
eu.log("UploadReceiver","Problem handling upload request" + e);
if (e instanceof MultiUploadAction.MergePartsException)
{
writeResponse(response.getWriter(), e.getMessage(), isIframe, true, requestParser);
}
else
{
writeResponse(response.getWriter(), e.getMessage(), isIframe, false, requestParser);
}
}
return (new ActionForward(mapping.getInput()));
}
public void doDelete(HttpServletRequest req, HttpServletResponse resp) throws IOException
{
String uuid = req.getPathInfo().replaceAll("/", "");
handleDeleteFileRequest(uuid, resp);
}
private void handleDeleteFileRequest(String uuid, HttpServletResponse resp) throws IOException
{
FileUtils.deleteDirectory(new File(UPLOAD_DIR, uuid));
if (new File(UPLOAD_DIR, uuid).exists())
{
eu.log("UploadReceiver","couldn't find or delete " + uuid);
}
else
{
eu.log("UploadReceiver","deleted " + uuid);
}
resp.setStatus(SUCCESS_RESPONSE_CODE);
// resp.addHeader("Access-Control-Allow-Origin", "*");
}
private void writeFileForNonMultipartRequest(HttpServletRequest req, RequestParser requestParser) throws Exception
{
File dir = new File(UPLOAD_DIR, requestParser.getUuid());
dir.mkdirs();
String contentLengthHeader = req.getHeader(CONTENT_LENGTH);
long expectedFileSize = Long.parseLong(contentLengthHeader);
if (requestParser.getPartIndex() >= 0)
{
writeFile(req.getInputStream(), new File(dir, requestParser.getUuid() + "_" + String.format("%05d", requestParser.getPartIndex())), null);
if (requestParser.getTotalParts()-1 == requestParser.getPartIndex())
{
File[] parts = getPartitionFiles(dir, requestParser.getUuid());
File outputFile = new File(dir, requestParser.getFilename());
for (File part : parts)
{
mergeFiles(outputFile, part);
}
assertCombinedFileIsVaid(requestParser.getTotalFileSize(), outputFile, requestParser.getUuid());
deletePartitionFiles(dir, requestParser.getUuid());
}
}
else
{
writeFile(req.getInputStream(), new File(dir, requestParser.getFilename()), expectedFileSize);
}
}
private void writeFileForMultipartRequest(RequestParser requestParser) throws Exception
{
File dir = new File(UPLOAD_DIR, requestParser.getUuid());
dir.mkdirs();
if (requestParser.getPartIndex() >= 0)
{
writeFile(requestParser.getUploadItem().getInputStream(), new File(dir, requestParser.getUuid() + "_" + String.format("%05d", requestParser.getPartIndex())), null);
if (requestParser.getTotalParts()-1 == requestParser.getPartIndex())
{
File[] parts = getPartitionFiles(dir, requestParser.getUuid());
File outputFile = new File(dir, requestParser.getOriginalFilename());
for (File part : parts)
{
mergeFiles(outputFile, part);
}
assertCombinedFileIsVaid(requestParser.getTotalFileSize(), outputFile, requestParser.getUuid());
deletePartitionFiles(dir, requestParser.getUuid());
}
}
else
{
writeFile(requestParser.getUploadItem().getInputStream(), new File(dir, requestParser.getFilename()), null);
}
}
private void assertCombinedFileIsVaid(int totalFileSize, File outputFile, String uuid) throws MultiUploadAction.MergePartsException
{
if (totalFileSize != outputFile.length())
{
deletePartitionFiles(UPLOAD_DIR, uuid);
outputFile.delete();
throw new MultiUploadAction.MergePartsException("Incorrect combined file size!");
}
}
private static class PartitionFilesFilter implements FilenameFilter
{
private String filename;
PartitionFilesFilter(String filename)
{
this.filename = filename;
}
#Override
public boolean accept(File file, String s)
{
return s.matches(Pattern.quote(filename) + "_\\d+");
}
}
private static File[] getPartitionFiles(File directory, String filename)
{
File[] files = directory.listFiles(new MultiUploadAction.PartitionFilesFilter(filename));
Arrays.sort(files);
return files;
}
private static void deletePartitionFiles(File directory, String filename)
{
File[] partFiles = getPartitionFiles(directory, filename);
for (File partFile : partFiles)
{
partFile.delete();
}
}
private File mergeFiles(File outputFile, File partFile) throws IOException
{
FileOutputStream fos = new FileOutputStream(outputFile, true);
try
{
FileInputStream fis = new FileInputStream(partFile);
try
{
IOUtils.copy(fis, fos);
}
finally
{
IOUtils.closeQuietly(fis);
}
}
finally
{
IOUtils.closeQuietly(fos);
}
return outputFile;
}
private File writeFile(InputStream in, File out, Long expectedFileSize) throws IOException
{
FileOutputStream fos = null;
try
{
fos = new FileOutputStream(out);
IOUtils.copy(in, fos);
if (expectedFileSize != null)
{
Long bytesWrittenToDisk = out.length();
if (!expectedFileSize.equals(bytesWrittenToDisk))
{
eu.log("UploadReceiver","Expected file {} to be {} bytes; file on disk is {} bytes " + new Object[] { out.getAbsolutePath(), expectedFileSize, 1 });
out.delete();
throw new IOException(String.format("Unexpected file size mismatch. Actual bytes %s. Expected bytes %s.", bytesWrittenToDisk, expectedFileSize));
}
}
return out;
}
catch (Exception e)
{
throw new IOException(e);
}
finally
{
IOUtils.closeQuietly(fos);
}
}
private void writeResponse(PrintWriter writer, String failureReason, boolean isIframe, boolean restartChunking, RequestParser requestParser)
{
if (failureReason == null)
{
// if (isIframe)
// {
// writer.print("{\"success\": true, \"uuid\": \"" + requestParser.getUuid() + "\"}<script src=\"http://192.168.130.118:8080/client/js/iframe.xss.response.js\"></script>");
// }
// else
// {
writer.print("{\"success\": true}");
// }
}
else
{
if (restartChunking)
{
writer.print("{\"error\": \"" + failureReason + "\", \"reset\": true}");
}
else
{
// if (isIframe)
// {
// writer.print("{\"error\": \"" + failureReason + "\", \"uuid\": \"" + requestParser.getUuid() + "\"}<script src=\"http://192.168.130.118:8080/client/js/iframe.xss.response.js\"></script>");
// }
// else
// {
writer.print("{\"error\": \"" + failureReason + "\"}");
// }
}
}
}
private class MergePartsException extends Exception
{
MergePartsException(String message)
{
super(message);
}
}
}
The full source of MultipartUploadParser.java is as follows:
package com.solarcentury.nrgi.document.action;
import DemonWeb.DmSession.Static_Env_Object;
import DemonWeb.Utils.EnvUtils;
import org.apache.commons.fileupload.FileItem;
import org.apache.commons.fileupload.FileUploadException;
import org.apache.commons.fileupload.disk.DiskFileItemFactory;
import org.apache.commons.fileupload.servlet.FileCleanerCleanup;
import org.apache.commons.fileupload.servlet.ServletFileUpload;
import org.apache.commons.io.FileCleaningTracker;
import org.apache.commons.lang3.StringUtils;
import javax.servlet.ServletContext;
import javax.servlet.http.HttpServletRequest;
import java.io.File;
import java.io.IOException;
import java.io.UnsupportedEncodingException;
import java.util.*;
public class MultipartUploadParser
{
// final Logger log = LoggerFactory.getLogger(MultipartUploadParser.class);
static Static_Env_Object seo = new Static_Env_Object();
private EnvUtils eu;
private Map<String, String> params = new HashMap<String, String>();
private List<FileItem> files = new ArrayList<FileItem>();
// fileItemsFactory is a field (even though it's scoped to the constructor) to prevent the
// org.apache.commons.fileupload.servlet.FileCleanerCleanup thread from attempting to delete the
// temp file before while it is still being used.
//
// FileCleanerCleanup uses a java.lang.ref.ReferenceQueue to delete the temp file when the FileItemsFactory marker object is GCed
private DiskFileItemFactory fileItemsFactory;
public MultipartUploadParser(HttpServletRequest request, File repository, ServletContext context) throws Exception
{
this.eu = new EnvUtils(seo.get_Env_Name(), "MultipartUploadParser " + "1.0.0.0");
if (!repository.exists() && !repository.mkdirs())
{
throw new IOException("Unable to mkdirs to " + repository.getAbsolutePath());
}
fileItemsFactory = setupFileItemFactory(repository, context);
ServletFileUpload upload = new ServletFileUpload(fileItemsFactory);
List<FileItem> formFileItems = upload.parseRequest(request);
parseFormFields(formFileItems);
if (files.isEmpty())
{
eu.log("MultipartUploadParser","No files were found when processing the request. Debugging info follows");
writeDebugInfo(request);
throw new FileUploadException("No files were found when processing the request.");
}
else
{
writeDebugInfo(request);
}
}
private DiskFileItemFactory setupFileItemFactory(File repository, ServletContext context)
{
DiskFileItemFactory factory = new DiskFileItemFactory();
factory.setSizeThreshold(DiskFileItemFactory.DEFAULT_SIZE_THRESHOLD);
factory.setRepository(repository);
FileCleaningTracker pTracker = FileCleanerCleanup.getFileCleaningTracker(context);
factory.setFileCleaningTracker(pTracker);
return factory;
}
private void writeDebugInfo(HttpServletRequest request)
{
eu.log("MultipartUploadParser","-- POST HEADERS --");
for (String header : Collections.list((Enumeration<String>) request.getHeaderNames()))
{
eu.log("MultipartUploadParser", header + "header " + request.getHeader(header));
}
eu.log("MultipartUploadParser","-- POST PARAMS --");
for (String key : params.keySet())
{
eu.log("MultipartUploadParser", key + " key " + params.get(key));
}
}
private void parseFormFields(List<FileItem> items) throws UnsupportedEncodingException
{
for (FileItem item : items)
{
if (item.isFormField())
{
String key = item.getFieldName();
String value = item.getString("UTF-8");
if (StringUtils.isNotBlank(key))
{
params.put(key, StringUtils.defaultString(value));
}
}
else
{
files.add(item);
}
}
}
public Map<String, String> getParams()
{
return params;
}
public List<FileItem> getFiles()
{
if (files.isEmpty())
{
throw new RuntimeException("No FileItems exist.");
}
return files;
}
public FileItem getFirstFile()
{
if (files.isEmpty())
{
throw new RuntimeException("No FileItems exist.");
}
return files.iterator().next();
}
}
MultiUploadAction decides that the request isMultipartContent and so calls MultipartUploadParser. This class successfully
checks the directory structure and then uses its method ParseFormFields to buld up a list of files.
However it does not find any files or form fields, and so on line 62 of MultipartUploadParser files.isEmpty() is true,
and so an exception is thrown (line 70)
It doesn't matter how many files I select in the client, the file list is always empty.
Many thanks for your help in this - much appreciated
I know this is very old, but I ran into this same problem. In my implementation I had tried to adapt the example endpoint code into my Spring MVC project. I discovered that the Spring MVC framework was actually calling the same Apache ServletFileUpload.parseRequest BEFORE passing the request to my controller and detecting the multipart, fetching the params, etc. The framework was parsing the multipart request despite whether I was using MutlipartHttpServlet request, or simply HttpServletRequest as my signature for the controller method. When my code in the MultipartUploadParser hit ServletFileUpload.parseRequest again, this time it returned an empty List because they had already been parsed.
The endpoint example code in the git repo for Fine Uploader works as it is, it was just my adaptation with Spring MVC that didn't work.