My source function has a frequency control whereas I can adjust the data rate that it flushes data to the next operator. I was measuring the data rate per operator using Prometheus+Grafana. Then I started to generate data at the speed of 100 rec/sec. On the grafana dashboard was showing about 90 rec/sec. Then I increased the data rate to be 200 rec/sec. However, the Grafana dashboard is actually showing 12 rec/sec. I was imagining that the backpressure was holding the data. But the Flink dashboard does not show I am having backpressure.
So, when checked the Flink code for StreamSourceContexts.collect(T element) there is a sync block there. I suppose that it is there to ensure the orderliness of events. But, what if I call the StreamSourceContexts.collect(T element) inside my SourceFunction using a Future? Am I going to experience out of order on the events? Is there a Source Function that allows me to push events in an asynchronous way?
#Override
public void collect(T element) {
synchronized (lock) {
output.collect(reuse.replace(element));
}
}
My source function:
public class OrdersSource extends RichSourceFunction<Order> {
#Override
public void run(SourceContext<Order> sourceContext) {
try {
while (running) {
generateOrderItem(sourceContext);
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private void generateOrderItem(SourceContext<Order> sourceContext) {
try {
InputStream stream = new FileInputStream(dataFilePath);
BufferedReader reader = new BufferedReader(new InputStreamReader(stream, StandardCharsets.UTF_8));
long startTime = System.nanoTime();
String line = reader.readLine();
while (line != null) {
// I would like to put an async thread here
// Thread newThread = new Thread(() -> {
// sourceContext.collect(getOrderItem(line));
// });
// newThread.start();
sourceContext.collect(getOrderItem(line));
// sleep in nanoseconds to have a reproducible data rate for the data source
this.dataRateListener.busySleep(startTime);
// get start time and line for the next iteration
startTime = System.nanoTime();
line = reader.readLine();
}
reader.close();
reader = null;
stream.close();
stream = null;
} catch (FileNotFoundException e) {
System.err.println("Please make sure they are available at [" + dataFilePath + "].");
System.err.println(
" Follow the instructions at [https://docs.deistercloud.com/content/Databases.30/TPCH%20Benchmark.90/Data%20generation%20tool.30.xml?embedded=true] in order to download and create them.");
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Related
I've written an endless loop in which I want to send a User Message every 5 seconds. Therefore I wrote a thread which waits for 5 seconds and then sends the Message received by the readLine() Method. If the user doesn't give any input the loop doesn't go on because of the readLine() Method waiting for input. So how can I cancel the readLine() Method?
while (true) {
new Thread() {
#Override
public void run() {
try {
long startTime = System.currentTimeMillis();
while ((System.currentTimeMillis() - startTime) < 5000) {
}
toClient.println(serverMessage);
clientMessage = fromClient.readLine();
System.out.println(clientName + ": " + clientMessage);
} catch (IOException e) {
e.printStackTrace();
}
}
}.start();
serverMessage = input.readLine();
}
This looks to be a producer-consumer type problem and I would structure this entirely differently since this fromClient.readLine(); is blocking and thus should be performed within another thread.
So consider reading the user input in another thread into a data structure, a Queue<String> such as a LinkedBlockingQueue<String>, and then retrieve String elements from the queue in the code above every 5 seconds, or nothing if no elements are held in the queue.
Something like....
new Thread(() -> {
while (true) {
try {
blockingQueue.put(input.readLine());
} catch (InterruptedException | IOException e) {
e.printStackTrace();
}
}
}).start();
new Thread(() -> {
try {
while (true) {
try {
TimeUnit.SECONDS.sleep(5);
} catch (InterruptedException e) {
e.printStackTrace();
}
String input = blockingQueue.poll();
input = input == null ? "" : input;
toClient.println(input);
}
} catch (IOException e) {
e.printStackTrace();
}
}).start();
Side notes: don't call .stop() on a thread as that is a dangerous thing to do. Also avoid extending Thread.
I am trying to create a sort of console/terminal that allows the user to input a string, which then gets made into a process and the results are printed out. Just like a normal console. But I am having trouble managing the input/output streams. I have looked into this thread, but that solution sadly doesn't apply to my problem.
Along with the standard commands like "ipconfig" and "cmd.exe", I need to be able to run a script and use the same inputstream to pass some arguments, if the script is asking for input.
For example, after running a script "python pyScript.py", I should be able pass further input to the script if it is asking for it(example: raw_input), while also printing the output from the script. The basic behavior you would expect from a terminal.
What I've got so far:
import java.awt.BorderLayout;
import java.awt.Color;
import java.awt.Dimension;
import java.awt.event.KeyEvent;
import java.awt.event.KeyListener;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.io.OutputStreamWriter;
import javax.swing.JFrame;
import javax.swing.JPanel;
import javax.swing.JScrollPane;
import javax.swing.JTextPane;
import javax.swing.text.BadLocationException;
import javax.swing.text.Document;
public class Console extends JFrame{
JTextPane inPane, outPane;
InputStream inStream, inErrStream;
OutputStream outStream;
public Console(){
super("Console");
setPreferredSize(new Dimension(500, 600));
setLocationByPlatform(true);
setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
// GUI
outPane = new JTextPane();
outPane.setEditable(false);
outPane.setBackground(new Color(20, 20, 20));
outPane.setForeground(Color.white);
inPane = new JTextPane();
inPane.setBackground(new Color(40, 40, 40));
inPane.setForeground(Color.white);
inPane.setCaretColor(Color.white);
JPanel panel = new JPanel(new BorderLayout());
panel.add(outPane, BorderLayout.CENTER);
panel.add(inPane, BorderLayout.SOUTH);
JScrollPane scrollPanel = new JScrollPane(panel);
getContentPane().add(scrollPanel);
// LISTENER
inPane.addKeyListener(new KeyListener(){
#Override
public void keyPressed(KeyEvent e){
if(e.getKeyCode() == KeyEvent.VK_ENTER){
e.consume();
read(inPane.getText());
}
}
#Override
public void keyTyped(KeyEvent e) {}
#Override
public void keyReleased(KeyEvent e) {}
});
pack();
setVisible(true);
}
private void read(String command){
println(command);
// Write to Process
if (outStream != null) {
System.out.println("Outstream again");
BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(outStream));
try {
writer.write(command);
//writer.flush();
//writer.close();
} catch (IOException e1) {
e1.printStackTrace();
}
}
// Execute Command
try {
exec(command);
} catch (IOException e) {}
inPane.setText("");
}
private void exec(String command) throws IOException{
Process pro = Runtime.getRuntime().exec(command, null);
inStream = pro.getInputStream();
inErrStream = pro.getErrorStream();
outStream = pro.getOutputStream();
Thread t1 = new Thread(new Runnable() {
public void run() {
try {
String line = null;
while(true){
BufferedReader in = new BufferedReader(new InputStreamReader(inStream));
while ((line = in.readLine()) != null) {
println(line);
}
BufferedReader inErr = new BufferedReader(new InputStreamReader(inErrStream));
while ((line = inErr.readLine()) != null) {
println(line);
}
Thread.sleep(1000);
}
} catch (Exception e) {
e.printStackTrace();
}
}
});
t1.start();
}
public void println(String line) {
Document doc = outPane.getDocument();
try {
doc.insertString(doc.getLength(), line + "\n", null);
} catch (BadLocationException e) {}
}
public static void main(String[] args){
new Console();
}
}
I don't use the mentioned ProcessBuilder, since I do like to differentiate between error and normal stream.
UPDATE 29.08.2016
With the help of #ArcticLord we have achieved what was asked in the original question.
Now it is just a matter of ironing out any strange behavior like the non terminating process. The Console has a "stop" button that simply calls pro.destroy(). But for some reason this does not work for infinitely running processes, that are spamming outputs.
Console: http://pastebin.com/vyxfPEXC
InputStreamLineBuffer: http://pastebin.com/TzFamwZ1
Example code that does not stop:
public class Infinity{
public static void main(String[] args){
while(true){
System.out.println(".");
}
}
}
Example code that does stop:
import java.util.concurrent.TimeUnit;
public class InfinitySlow{
public static void main(String[] args){
while(true){
try {
TimeUnit.SECONDS.sleep(1);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println(".");
}
}
}
You are on the right way with your code. There are only some minor things you missed.
Lets start with your read method:
private void read(String command){
[...]
// Write to Process
if (outStream != null) {
[...]
try {
writer.write(command + "\n"); // add newline so your input will get proceed
writer.flush(); // flush your input to your process
} catch (IOException e1) {
e1.printStackTrace();
}
}
// ELSE!! - if no outputstream is available
// Execute Command
else {
try {
exec(command);
} catch (IOException e) {
// Handle the exception here. Mostly this means
// that the command could not get executed
// because command was not found.
println("Command not found: " + command);
}
}
inPane.setText("");
}
Now lets fix your exec method. You should use separate threads for reading normal process output and error output. Additionally I introduce a third thread that waits for the process to end and closes the outputStream so next user input is not meant for process but is a new command.
private void exec(String command) throws IOException{
Process pro = Runtime.getRuntime().exec(command, null);
inStream = pro.getInputStream();
inErrStream = pro.getErrorStream();
outStream = pro.getOutputStream();
// Thread that reads process output
Thread outStreamReader = new Thread(new Runnable() {
public void run() {
try {
String line = null;
BufferedReader in = new BufferedReader(new InputStreamReader(inStream));
while ((line = in.readLine()) != null) {
println(line);
}
} catch (Exception e) {
e.printStackTrace();
}
System.out.println("Exit reading process output");
}
});
outStreamReader.start();
// Thread that reads process error output
Thread errStreamReader = new Thread(new Runnable() {
public void run() {
try {
String line = null;
BufferedReader inErr = new BufferedReader(new InputStreamReader(inErrStream));
while ((line = inErr.readLine()) != null) {
println(line);
}
} catch (Exception e) {
e.printStackTrace();
}
System.out.println("Exit reading error stream");
}
});
errStreamReader.start();
// Thread that waits for process to end
Thread exitWaiter = new Thread(new Runnable() {
public void run() {
try {
int retValue = pro.waitFor();
println("Command exit with return value " + retValue);
// close outStream
outStream.close();
outStream = null;
} catch (InterruptedException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
});
exitWaiter.start();
}
Now this should work.
If you enter ipconfig it prints the command output, closes the output stream and is ready for a new command.
If you enter cmd it prints the output and let you enter more cmd commands like dir or cd and so on until you enter exit. Then it closes the output stream and is ready for a new command.
You may run into problems with executing python scripts because there are problems with reading Process InputStreams with Java if they are not flushed into system pipeline.
See this example python script
print "Input something!"
str = raw_input()
print "Received input is : ", str
You could run this with your Java programm and also enter the input but you will not see the script output until the script is finished.
The only fix I could find is to manually flush the output in the script.
import sys
print "Input something!"
sys.stdout.flush()
str = raw_input()
print "Received input is : ", str
sys.stdout.flush()
Running this script will bahave as you expect.
You can read more about this problem at
Java: is there a way to run a system command and print the output during execution?
Why does reading from Process' InputStream block altough data is available
Java: can't get stdout data from Process unless its manually flushed
EDIT: I have just found another very easy solution for the stdout.flush() problem with Python Scripts. Start them with python -u script.py and you don't need to flush manually. This should solve your problem.
EDIT2: We discussed in the comments that with this solution output and error Stream will be mixed up since they run in different threads. The problem here is that we cannot distinguish if output writing is finish when error stream thread comes up. Otherwise classic thread scheduling with locks could handle this situation. But we have a continuous stream until process is finished no matter if data flows or not. So we need a mechanism here that logs how much time has elapsed since last line was read from each stream.
For this I will introduce a class that gets an InputStream and starts a Thread for reading the incoming data. This Thread stores each line in a Queue and stops when end of stream arrives. Additionally it holds the time when last line was read and added to Queue.
public class InputStreamLineBuffer{
private InputStream inputStream;
private ConcurrentLinkedQueue<String> lines;
private long lastTimeModified;
private Thread inputCatcher;
private boolean isAlive;
public InputStreamLineBuffer(InputStream is){
inputStream = is;
lines = new ConcurrentLinkedQueue<String>();
lastTimeModified = System.currentTimeMillis();
isAlive = false;
inputCatcher = new Thread(new Runnable(){
#Override
public void run() {
StringBuilder sb = new StringBuilder(100);
int b;
try{
while ((b = inputStream.read()) != -1){
// read one char
if((char)b == '\n'){
// new Line -> add to queue
lines.offer(sb.toString());
sb.setLength(0); // reset StringBuilder
lastTimeModified = System.currentTimeMillis();
}
else sb.append((char)b); // append char to stringbuilder
}
} catch (IOException e){
e.printStackTrace();
} finally {
isAlive = false;
}
}});
}
// is the input reader thread alive
public boolean isAlive(){
return isAlive;
}
// start the input reader thread
public void start(){
isAlive = true;
inputCatcher.start();
}
// has Queue some lines
public boolean hasNext(){
return lines.size() > 0;
}
// get next line from Queue
public String getNext(){
return lines.poll();
}
// how much time has elapsed since last line was read
public long timeElapsed(){
return (System.currentTimeMillis() - lastTimeModified);
}
}
With this class we could combine the output and error reading thread into one. That lives while the input reading buffer threads live and have not comsumed data. In each run it checks if some time has passed since last output was read and if so it prints all unprinted lines at a stroke. The same with the error output. Then it sleeps for some millis for not wasting cpu time.
private void exec(String command) throws IOException{
Process pro = Runtime.getRuntime().exec(command, null);
inStream = pro.getInputStream();
inErrStream = pro.getErrorStream();
outStream = pro.getOutputStream();
InputStreamLineBuffer outBuff = new InputStreamLineBuffer(inStream);
InputStreamLineBuffer errBuff = new InputStreamLineBuffer(inErrStream);
Thread streamReader = new Thread(new Runnable() {
public void run() {
// start the input reader buffer threads
outBuff.start();
errBuff.start();
// while an input reader buffer thread is alive
// or there are unconsumed data left
while(outBuff.isAlive() || outBuff.hasNext() ||
errBuff.isAlive() || errBuff.hasNext()){
// get the normal output if at least 50 millis have passed
if(outBuff.timeElapsed() > 50)
while(outBuff.hasNext())
println(outBuff.getNext());
// get the error output if at least 50 millis have passed
if(errBuff.timeElapsed() > 50)
while(errBuff.hasNext())
println(errBuff.getNext());
// sleep a bit bofore next run
try {
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
System.out.println("Finish reading error and output stream");
}
});
streamReader.start();
// remove outStreamReader and errStreamReader Thread
[...]
}
Maybe this is not a perfect solution but it should handle the situation here.
EDIT (31.8.2016)
We discussed in comments that there is still a problem with the code while implementing a stop button that kills the started
process using Process#destroy(). A process that produces very much output e.g. in an infinite loop will
be destroyed immediately by calling destroy(). But since it has already produced a lot of output that has to be consumed
by our streamReader we can't get back to normal programm behaviour.
So we need some small changes here:
We will introduce a destroy() method to the InputStreamLineBuffer that stops the output reading and clears the queue.
The changes will look like this:
public class InputStreamLineBuffer{
private boolean emergencyBrake = false;
[...]
public InputStreamLineBuffer(InputStream is){
[...]
while ((b = inputStream.read()) != -1 && !emergencyBrake){
[...]
}
}
[...]
// exits immediately and clears line buffer
public void destroy(){
emergencyBrake = true;
lines.clear();
}
}
And some little changes in the main programm
public class ExeConsole extends JFrame{
[...]
// The line buffers must be declared outside the method
InputStreamLineBuffer outBuff, errBuff;
public ExeConsole{
[...]
btnStop.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent e) {
if(pro != null){
pro.destroy();
outBuff.destroy();
errBuff.destroy();
}
}});
}
[...]
private void exec(String command) throws IOException{
[...]
//InputStreamLineBuffer outBuff = new InputStreamLineBuffer(inStream);
//InputStreamLineBuffer errBuff = new InputStreamLineBuffer(inErrStream);
outBuff = new InputStreamLineBuffer(inStream);
errBuff = new InputStreamLineBuffer(inErrStream);
[...]
}
}
Now it should be able to destroy even some output spamming processes.
Note: I found out that Process#destroy() is not able to destroy child processes. So if you start cmd on windows
and start a java programm from there you will end up destroying the cmd process while the java programm is still running.
You will see it in the task manager. This problem could not be solved with java itself. it will need
some os depending external tools to get the pids of these processes and kill them manually.
Although #ArticLord solution is nice and neat, recently I faced the same kind of problem and came up with a solution that's conceptually equivalent, but slightly different in its implementation.
The concept is the same, namely "bulk reads": when a reader thread acquires its turn, it consumes all the stream it handles, and pass the hand only when it is done.
This guarantees the out/err print order.
But instead of using a timer-based turn assignment, I use a lock-based non-blocking read simulation:
// main method for testability: replace with private void exec(String command)
public static void main(String[] args) throws Exception
{
// create a lock that will be shared between reader threads
// the lock is fair to minimize starvation possibilities
ReentrantLock lock = new ReentrantLock(true);
// exec the command: I use nslookup for testing on windows
// because it is interactive and prints to stderr too
Process p = Runtime.getRuntime().exec("nslookup");
// create a thread to handle output from process (uses a test consumer)
Thread outThread = createThread(p.getInputStream(), lock, System.out::print);
outThread.setName("outThread");
outThread.start();
// create a thread to handle error from process (test consumer, again)
Thread errThread = createThread(p.getErrorStream(), lock, System.err::print);
errThread.setName("errThread");
errThread.start();
// create a thread to handle input to process (read from stdin for testing purpose)
PrintWriter writer = new PrintWriter(p.getOutputStream());
Thread inThread = createThread(System.in, null, str ->
{
writer.print(str);
writer.flush();
});
inThread.setName("inThread");
inThread.start();
// create a thread to handle termination gracefully. Not really needed in this simple
// scenario, but on a real application we don't want to block the UI until process dies
Thread endThread = new Thread(() ->
{
try
{
// wait until process is done
p.waitFor();
logger.debug("process exit");
// signal threads to exit
outThread.interrupt();
errThread.interrupt();
inThread.interrupt();
// close process streams
p.getOutputStream().close();
p.getInputStream().close();
p.getErrorStream().close();
// wait for threads to exit
outThread.join();
errThread.join();
inThread.join();
logger.debug("exit");
}
catch(Exception e)
{
throw new RuntimeException(e.getMessage(), e);
}
});
endThread.setName("endThread");
endThread.start();
// wait for full termination (process and related threads by cascade joins)
endThread.join();
logger.debug("END");
}
// convenience method to create a specific reader thread with exclusion by lock behavior
private static Thread createThread(InputStream input, ReentrantLock lock, Consumer<String> consumer)
{
return new Thread(() ->
{
// wrap input to be buffered (enables ready()) and to read chars
// using explicit encoding may be relevant in some case
BufferedReader reader = new BufferedReader(new InputStreamReader(input));
// create a char buffer for reading
char[] buffer = new char[8192];
try
{
// repeat until EOF or interruption
while(true)
{
try
{
// wait for your turn to bulk read
if(lock != null && !lock.isHeldByCurrentThread())
{
lock.lockInterruptibly();
}
// when there's nothing to read, pass the hand (bulk read ended)
if(!reader.ready())
{
if(lock != null)
{
lock.unlock();
}
// this enables a soft busy-waiting loop, that simultates non-blocking reads
Thread.sleep(100);
continue;
}
// perform the read, as we are sure it will not block (input is "ready")
int len = reader.read(buffer);
if(len == -1)
{
return;
}
// transform to string an let consumer consume it
String str = new String(buffer, 0, len);
consumer.accept(str);
}
catch(InterruptedException e)
{
// catch interruptions either when sleeping and waiting for lock
// and restore interrupted flag (not necessary in this case, however it's a best practice)
Thread.currentThread().interrupt();
return;
}
catch(IOException e)
{
throw new RuntimeException(e.getMessage(), e);
}
}
}
finally
{
// protect the lock against unhandled exceptions
if(lock != null && lock.isHeldByCurrentThread())
{
lock.unlock();
}
logger.debug("exit");
}
});
}
Note that both solutions, #ArticLord's and mine, are not totally starvation-safe, and chances (really few) are inversely proportional to consumers speed.
Happy 2016! ;)
I have a class called HomeView that is used to extend a Vaadin Designer HTML class. This class has a Vaadin table that takes input from an uploaded file. So far the file uploads fine and I can split the file up into lines for testing. I was trying to use Vaadin threads to lock the session and go to the UploadFile class in which I will split up the file and add to a row in the table. I would then unlock the session, exit back to the background thread and the UI should update the table with new rows. This is not happening with the code below.
public void uploadSucceeded(Upload.SucceededEvent succeededEvent) {
//upload notification for upload
new Notification("File Uploaded Successfully",
Notification.Type.HUMANIZED_MESSAGE)
.show(Page.getCurrent());
//create new class for parsing logic
uf = new UploadFile();
new Thread(new Runnable() {
#Override
public void run() {
try {
getSession().lock();
uf.parseFile();
getSession().unlock();
} catch (IOException e) {
new Notification("Could not parse file type",
e.getMessage(),
Notification.Type.ERROR_MESSAGE)
.show(Page.getCurrent());
}
catch (UnsupportedOperationException e) {
e.printStackTrace();
} catch (ReadOnlyException e) {
e.printStackTrace();
}
}
}).start();
//outputFile.delete();
}
});
UploadFile class
public class UploadFile extends HomeView {
/**
*
*/
private static final long serialVersionUID = 839096232794540854L;
public void parseFile() throws IOException {
//container.removeAllItems();
BufferedReader reader = null;
reader = new BufferedReader(new InputStreamReader(new FileInputStream(outputFile.getAbsolutePath()), StandardCharsets.UTF_8));
String line;
while ((line = reader.readLine()) != null)
{
System.out.println("before add:" + uploadTable.size());
container = uploadTable.getContainerDataSource();
container.addItem("row3");
Item item2 = container.getItem("row3");
Property property2 = item2.getItemProperty("name");
property2.setValue("hello");
uploadTable.setContainerDataSource(container);
System.out.println("after add:" + uploadTable.size());
}
reader.close();
}
}
If I take the code above and just put it in place of the method call, then the table updates fine. The table is updating the row count in the background, it's just not refreshing the view. What am I missing to make the UI refresh?
#Override
public void uploadSucceeded(Upload.SucceededEvent succeededEvent) {
//upload notification for upload
new Notification("File Uploaded Successfully",
Notification.Type.HUMANIZED_MESSAGE)
.show(Page.getCurrent());
//create new class for parsing logic
uf = new UploadFile();
new Thread(new Runnable() {
#Override
public void run() {
try {
getSession().lock();
BufferedReader reader = null;
reader = new BufferedReader(new InputStreamReader(new FileInputStream(outputFile.getAbsolutePath()), StandardCharsets.UTF_8));
String line;
while ((line = reader.readLine()) != null)
{
System.out.println("before add:" + uploadTable.size());
container = uploadTable.getContainerDataSource();
container.addItem("row3");
Item item2 = container.getItem("row3");
Property property2 = item2.getItemProperty("name");
property2.setValue("hello");
uploadTable.setContainerDataSource(container);
System.out.println("after add:" + uploadTable.size());
}
reader.close();
getSession().unlock();
} catch (IOException e) {
new Notification("Could not parse file type",
e.getMessage(),
Notification.Type.ERROR_MESSAGE)
.show(Page.getCurrent());
}
catch (UnsupportedOperationException e) {
e.printStackTrace();
} catch (ReadOnlyException e) {
e.printStackTrace();
}
}
}).start();
//outputFile.delete();
}
});
UI.getCurrent() helper uses a ThreadLocal variable to get the active UI and it only works in a code executed in UI thread (e.g. a init method or button click listener). Get the UI reference before constructing the Thread and use the access method around your code that modifies UI. Do not use getSession().lock() or similar, you'll most likely do something wrong with that. Here is a simple usage example that should help you to resolve your use case as well.
// Get the reference to UI to be modified
final UI ui = getUI();
new Thread() {
#Override
public void run() {
// Do stuff that don't affect UI state here, e.g. potentially
// slow calculation or rest call
final double d = 1*1;
ui.access(new Runnable() {
#Override
public void run() {
// This code here is safe to modify ui
Notification.show("The result of calculation is " + d);
}
});
}
}.start();
In addition to properly synchronised UI access you need to have properly working push connection or polling to get changes to the client. If you want to use "real push" you need to add the annotation and add vaadin-push module to your app. Simpler method (and most often just as good) is just to enable polling:
ui.setPollInterval(1000); // 1000ms polling interval for client
I have this code that downloads a web page:
HttpURLConnection connection;
private String downloadContent() {
InputStream content;
Source parser;
try {
content = connection.getInputStream(); //<--here is the download
parser = new Source(content);
content.close();
return parser.toString();
} catch (Exception e) {
return null;
}
}
While doing the download, I tried to get the amount of downloaded data, and if it reaches a limit, I stop the downloading, but I not found a way to do this. If someone know how to do, please tell me.
Now I want to limit the download time. Example: if the download pass 20 seconds, I stop it. I want to do this because my program it's an webcrawler and if by an error, it begins downloading a big file, it will stuck in the download, and is not this I want to do, so a filter in download by size is welcome, but as I don't know, a filter time will prevent this problem.
The proper way to achieve this is the following:
public class TimeOut {
public static class MyJob implements Callable<String> {
#Override
public String call() throws Exception {
// Do something
return "result";
}
}
public static void main(String[] args) {
Future<String> control
= Executors.newSingleThreadExecutor().submit(new MyJob());
try {
String result = control.get(5, TimeUnit.SECONDS);
} catch (TimeoutException ex) {
// 5 seconds expired, we cancel the job !!!
control.cancel(true);
}
catch (InterruptedException ex) {
} catch (ExecutionException ex) {
}
}
}
You can use AOP and a #Timeable annotation from jcabi-aspects (I'm a developer):
#Timeable(limit = 1, unit = TimeUnit.SECONDS)
String downloadContent() {
if (Thread.currentThread.isInterrupted()) {
throw new IllegalStateException("time out");
}
// download
}
Pay attention that you should check for isInterrupted() regularly and throw an exception when it is set to TRUE. This is the only way to terminate a thread in Java.
Also, for more detailed explanation, check this post: http://www.yegor256.com/2014/06/20/limit-method-execution-time.html
There is a specified class java.util.Timer that is intended to do the tasks you required.You can reference the API for more detail.
Life is messy. If you want to clean up after yourself, it takes some work.
private static final long TIMEOUT = TimeUnit.SECONDS.toMillis(20);
private String downloadContent() {
connection.setConnectTimeout(TIMEOUT); /* Set connect timeout. */
long start = System.nanoTime();
final InputStream content;
try {
content = connection.getInputStream();
} catch (IOException ex) {
return null;
}
/* Compute how much time we have left. */
final long delay = TIMEOUT -
TimeUnit.NANOS.toMillis(System.nanoTime() - time);
if (delay < 1)
return null;
/* Start a thread that can close the stream asynchronously. */
Thread killer = new Thread() {
#Override
public void run() {
try {
Thread.sleep(delay); /* Wait until time runs out or interrupted. */
} catch (InterruptedException expected) {
Thread.currentThread().interrupt();
}
try {
content.close();
} catch (IOException ignore) {
// Log this?
}
}
};
killer.start();
try {
String s = new Source(content).parser.toString();
/* Task completed in time; clean up immediately. */
killer.interrupt();
return s;
} catch (Exception e) {
return null;
}
}
You can't stop a running thread. What you can do, however:
1) Create a new thread and fetch the content from this thread. If the thread takes too long to answer, just go on and ignore its results. Downside of this approach: the background thread will still download the big file.
2) Use another HTTP connection API with more controls. I've used "Jakarta Commons HttpClient" a long time ago and was very pleased with its ability to timeout.
Here is a simplified version of my application showing what I'm doing.
/*
in my app's main():
Runner run = new Runner();
run.dowork();
*/
class Runner
{
private int totalWorkers = 2;
private int workersDone = 0;
public synchronized void workerDone()
{
workersDone++;
notifyAll();
}
public synchronized void dowork()
{
workersDone = 0;
//<code for opening a file here, other setup here, etc>
Worker a = new Worker(this);
Worker b = new Worker(this);
while ((line = reader.readLine()) != null)
{
//<a large amount of processing on 'line'>
a.setData(line);
b.setData(line);
while (workersDone < totalWorkers)
{
wait();
}
}
}
}
class Worker implements Runnable
{
private Runner runner;
private String data;
public Worker(Runner r)
{
this.runner = r;
Thread t = new Thread(this);
t.start();
}
public synchronized void setData(String s)
{
this.data = s;
notifyAll();
}
public void run
{
while (true)
{
synchronized(this)
{
wait();
//<do work with this.data here>
this.runner.workerDone();
}
}
}
}
The basic concept here is that I have a bunch of workers which all do some processing on an incoming line of data, all independently, and write out the data wherever they like - they do not need to report any data back to the main thread or share data with each other.
The problem that I'm having is that this code deadlocks. I'm reading a file of over 1 million lines and I'm lucky to get 100 lines into it before my app stops responding.
The workers, in reality, all do differing amounts of work so I want to wait until they all complete before moving to the next line.
I cannot let the workers process at different speeds and queue the data internally because the files I am processing are too large for this and won't fit in memory.
I cannot give each worker its own FileReader to independently get 'line', because I do a ton of processing on the line before the workers see it, and do not want to have to re-do the processing in each worker.
I know I'm missing some fairly simple aspect of synchronization in Java but I'm stuck at this point. If someone could explain what I'm doing wrong here I would appreciate it. I believe I'm misunderstanding some aspect of the synchronization but I'm out of ideas for attempting to fix it.
Working directly with synchronized, wait(), and notify() is definitely tricky.
Fortunately the Java Concurrency API provides some excellent control objects for this sort of thing that are much more intuitive. In particular, look at CyclicBarrier and CountDownLatch; one of them almost certainly will be what you're looking for.
You may also find a ThreadPoolExecutor to be handy for this situation.
Here's a simple example / conversion of your snippet that produces the following output (without deadlock, of course):
Read line: Line 1
Waiting for work to be complete on line: Line 1
Working on line: Line 1
Working on line: Line 1
Read line: Line 2
Waiting for work to be complete on line: Line 2
Working on line: Line 2
Working on line: Line 2
Read line: Line 3
Waiting for work to be complete on line: Line 3
Working on line: Line 3
Working on line: Line 3
All work complete!
public class Runner
{
public static void main(String args[]) {
Runner r = new Runner();
try {
r.dowork();
} catch (IOException e) {
// handle
e.printStackTrace();
}
}
CyclicBarrier barrier;
ExecutorService executor;
private int totalWorkers = 2;
public Runner() {
this.barrier = new CyclicBarrier(this.totalWorkers + 1);
this.executor = Executors.newFixedThreadPool(this.totalWorkers);
}
public synchronized void dowork() throws IOException
{
//<code for opening a file here, other setup here, etc>
//BufferedReader reader = null;
//String line;
final Worker worker = new Worker();
for(String line : new String[]{"Line 1", "Line 2", "Line 3"})
//while ((line = reader.readLine()) != null)
{
System.out.println("Read line: " + line);
//<a large amount of processing on 'line'>
for(int c = 0; c < this.totalWorkers; c++) {
final String curLine = line;
this.executor.submit(new Runnable() {
public void run() {
worker.doWork(curLine);
}
});
}
try {
System.out.println("Waiting for work to be complete on line: " + line);
this.barrier.await();
} catch (InterruptedException e) {
// handle
e.printStackTrace();
} catch (BrokenBarrierException e) {
// handle
e.printStackTrace();
}
}
System.out.println("All work complete!");
}
class Worker
{
public void doWork(String line)
{
//<do work with this.data here>
System.out.println("Working on line: " + line);
try {
Runner.this.barrier.await();
} catch (InterruptedException e) {
// handle
e.printStackTrace();
} catch (BrokenBarrierException e) {
// handle
e.printStackTrace();
}
}
}
}
IMHO you have improperly placed "workersDone = 0".
public synchronized void dowork()
{
// workersDone = 0;
//<code for opening a file here, other setup here, etc>
Worker a = new Worker(this);
Worker b = new Worker(this);
while ((line = reader.readLine()) != null)
{
workersDone = 0;
//<a large amount of processing on 'line'>
a.setData(line);
b.setData(line);
while (workersDone < totalWorkers)
{
wait();
}
}
}