I have a pipeline with quite a few steps (just above 15). I want to report failures everytime a DoFn fails. I started implementing it through TupleTags with code such as :
try {
... do stuff ...
c.output(successTag, ...);
} catch (Exception e) {
c.output(failureTag, new Failure(...));
}
But since my pipeline contains a lot of steps, this make the pipeline definition code quite hard to read / maintain.
Is there a more global way to achieve it ? Something like raising a custom exception which is handled globally at the pipeline level ?
What you are doing is the correct approach to catch errors and output them differently. You will need this on each step though. You could use a java pattern to reuse it if you prefer. Create a base class for all your ParDos and in processElement add the exception handling code. Then implement your processElement in a separate function (i.e. processElementImpl) which you call in processElement.
Related
I am not able to understand why Java doesn't allow to change exception message of an exception of type Exception (or its superclass Throwable) once it has been created. It allows to change the stackTrace using setStackTrace but not the message.
The use case I have seems genuine to me and would appreciate some alternative.
Use case
I have a controller service X that calls let's say 10 other dependent services. To make debugging easy, if a dependent service throws some exception we want to surface some identifiers from service X to its upstream to identify the failed request easily. For this we have requestId which we create and set for each dependency.
Now to avoid duplication across all dependent services and simplify code, we can create a common interceptor that allows us to do some work before and after each call. Something like -
// do some work
requestId = getRequestId(); // create or somehow get requestId
dependentService.setRequestId(requestId);
try {
dependentService.call();
}
catch (Exception e) {
e.setMessage(e.getMessage() + ... + requestId);
throw e;
}
//do some work
But Java doesn't allow us to set message. At the same time, we want to preserve the exception and its type (which could be any of the custom types defined by dependent services), so I don't want to do something like throw new Exception(e.getMessage() + ...)
It's not really what it's meant for, but you could attach another exception with addSuppressed:
} catch (Exception e) {
e.addSuppressed(new ExtraInfoException(...));
throw e;
}
where ... contains the extra info you want to include.
The advantage of this over adding to the exception message is that you can define your ExtraInfoException so that it has the info you want in fields, rather than having to parse it back out of the exception message.
With that said, a more idiomatic way to attach more exception info it would be:
} catch (Exception e) {
throw new ExtraInfoException(e, ...);
}
which has exactly the same advantage of allowing you to return structured information, with the additional advantage that you can catch ExtraInfoException directly, rather than catching Exception and then hunting for the extra info reflectively.
Why doesn't Java have setMessage in Exception/Throwable classes?
The answer to your question is that they (the library designers) did not think that changing a message on an exception was a useful thing to do.
To a large degree1, the designers have taken the view that they shouldn't design the APIs to directly support all possible use-cases ... including the obscure ones that almost nobody will encounter. Like your one2.
And in your case, there are other ways to achieve what you are trying to do; see the other answers.
I did a quick search of the Java bugs database to see if someone else had put in an RFE to request a setMessage method for Throwable or Exception. I couldn't find anything. If your requirement was even slightly common, there would probably be an RFE with an explanation of why it was declined.
1 - Obviously, there are exceptions to this, but that is beside the point.
2 - Obviously you would disagree that your use-case is obscure, but that that is also beside the point. The question is why they haven't implemented this, not whether they were wrong. (Asking / debating whether they were wrong is off-topic, because it is a matter of opinion.)
Resetting a message it's some kind of rewriting the history. You have a catch block when you catch exception and handle them. If you need to throw an exception during the handling, it's a different problem and an exception should be different.
} catch (SomeException e) {
// here we have SomeException and we want to handle it.
// if we can't we throw a new one, because we have a problem with handling.
// if the handling problem cause is SomeException we put it at the cause.
throw new AnotherException("with some message", e);
}
And in the stacks trace we will see that we have AnotherException because of SomeException which gives us information about the root of problem.
Just simply throw new instance like this:
try {
...
} catch(Exception ex) {
throw new Exception ("new message", ex);
}
I've been using Spark Dataset API to perform operations on a JSON to extract certain fields as needed. However, when the specification that I provide to let spark know what field to extract goes wrong, spark spits out an
org.apache.spark.sql.AnalysisException
How can unchecked runtime exceptions be handled in a distributed processing scenario like this ? I understand that throwing a try-catch would get things sorted but what is the recommended way to handle such a scenario
dataset = dataset.withColumn(current, functions.explode(dataset.col(parent + Constants.PUNCTUATION_PERIOD + child.substring(0, child.length() - 2))));
In scala, you should simply wrap the call in a Try and manage Failure. Something like:
val result = Try(executeSparkCode()) match {
case s: Success(_) => s;
case Failure(error: AnalysisException) => Failure(new MyException(error));
}
Note 1: If your question implies how to manage exception in scala, there are a lot of doc and post about this subject (i.e. don't throw). For example, you can check that answer (of mine)
Note 2: I don't have a scala dev env right here, so I didn't test this code)
In java there is a tricky situation however: the compiler doesn't expect an AnalysisException which is unchecked so you cannot catch this exception specifically. Probably some scala/java misunderstanding because scala doesn't track checked exceptions. What I did was:
try{
return executeSparkCode();
} catch (Exception ex) {
if(ex instanceOf AnalysisException){
throw new MyException(ex);
} else {
throw ex; // unmanaged exceptions
}
}
Note: In my case, I also tested the content of the error message for a specific exception that I must managed (i.e "path does not exist") in which case I return an empty dataset instead of throwing another exception. I was looking for a better solution and happened to get here...
I'm trying to define a Pentaho Kettle (ktr) transformation via code. I would like to add to the transformation a Text File Input Step: http://wiki.pentaho.com/display/EAI/Text+File+Input.
I don't know how to do this (note that I want to achieve the result in a custom Java application, not using the standard Spoon GUI). I think I should use the TextFileInputMeta class, but when I try to define the filename the trasformation doesn't work anymore (it seems empty in Spoon).
This is the code I'm using. I think the third line has something wrong:
PluginRegistry registry = PluginRegistry.getInstance();
TextFileInputMeta fileInMeta = new TextFileInputMeta();
fileInMeta.setFileName(new String[] {myFileName});
String fileInPluginId = registry.getPluginId(StepPluginType.class, fileInMeta);
StepMeta fileInStepMeta = new StepMeta(fileInPluginId, myStepName, fileInMeta);
fileInStepMeta.setDraw(true);
fileInStepMeta.setLocation(100, 200);
transAWMMeta.addStep(fileInStepMeta);
To run a transformation programmatically, you should do the following:
Initialise Kettle
Prepare a TransMeta object
Prepare your steps
Don't forget about Meta and Data objects!
Add them to TransMeta
Create Trans and run it
By default, each transformation germinates a thread per step, so use trans.waitUntilFinished() to force your thread to wait until execution completes
Pick execution's results if necessary
Use this test as example: https://github.com/pentaho/pentaho-kettle/blob/master/test/org/pentaho/di/trans/steps/textfileinput/TextFileInputTests.java
Also, I would recommend you create the transformation manually and to load it from file, if it is acceptable for your circumstances. This will help to avoid lots of boilerplate code. It is quite easy to run transformations in this case, see an example here: https://github.com/pentaho/pentaho-kettle/blob/master/test/org/pentaho/di/TestUtilities.java#L346
I'm currently dealing with exceptions handling and I'm wondering where should I catch them.
Here is an stack from the GWT app :
A helper with a method which can throws NumerFormatExeption
(FormHelper.java)
A widget which uses this helper (CostWidget.java)
A presenter which calls this widget to retrieve data (BuildingPresenter.java)
FormHelper.java
public static Integer prepareIntegerForDb(String string) {
return Integer.parseInt(string);
}
CostWidget.java
public DetailCostProxy getCostDetail() {
...
costDetail.setQuantity(FormHelper.prepareDoubleForBd(qtTextBox.getText()));
...
return costDetail;
}
public List<DetailCostProxy> getCostList() {
...
costDetails .add(ligneCout.getCostDetail());
...
}
BuildingPresenter.java
public void saveBuilding(final BuildingProxy inter, final CollectRequestContext savecontext) {
savecontext.save(display.getCostWidget().getCoutList()).fire(new Receiver<BuildingProxy >() {....
}
I am thinking about :
1) adding "throws NumberFormatException" to prepareIntegerForDb() in the helper
2) adding "throws NumberFormatException" to getCostDetail() in the widget
3) adding "throws NumberFormatException" to getCostList() in the widget
4) caching the exception in the presenter (in saveBuilding)
The aim is :
to log the exception
to provide the user with a message saying that something went wrong
What do you think about this approach considering that this in an example and I will have to apply this pattern into the entire app (more than 20 presenters).
Is my way a good way to handle exceptions in GWT ? or should I log the error directly in the helper or elsewhere ?
prepareIntegerForDB() should throw the exception. This happens automatically when Integer.parse() fails, and you do not have to actually throw the Exception.
getCostDetail() should explicitly catch and throw the exception, and possibly expand upon why it was thrown. Something like "The cost was not in a readable format". That method is responsible for only that one line.
getCostList() should catch and handle the exceptions. That method is responsible for an entire collection. If you do not handle the bad data here, you will lose the good data. Here is one way to handle the bad data.
public List<DetailCostProxy> getCostList() {
...
try {
DetailCostProxy cost = lineCount.getCostDetail()
costDetails.add(cost);
catch (NumberFormatException e) {
costDetails.add(null);
}
...
}
Finally, the method that displays your data to the user should interpret the data passed to it before displaying it. If you used my example above, this would be as simple as checking for null values.
What do you think about this approach considering that this in an
example and I will have to apply this pattern into the entire app
(more than 20 presenters).
Adding throws NumberFormatException declarations won't help you to "provide the user with a message saying that something went wrong". NumberFormatException-s are RuntimeException-s so the throws declaration won't even force to try/catch in the code that uses these methods.
Is my way a good way to handle exceptions in GWT ? or should I log the
error directly in the helper or elsewhere ?
4) catching the exception in the presenter (in saveBuilding)
The aim is :
to log the exception
to provide the user with a message saying that something went wrong
This question is not specific to GWT.
To catch the Exception is a good idea if you know how to deal with it.
If you signal the error to the user, you need to be able to have the user decide how to handle the issue (for example a pop-up message proposing two actions to resume the application execution).
Due to checked exceptions, we can have some problems in production having all exceptions caught in the right place and logged correctly.
I wonder if there is some opensource tool to help with auditing these problems.
For example, is there some AOP tool that would intercept all exceptions thrown and see if they are re-thrown, wrapped or logged? This would help identify the bad catches.
If you've decided that you would like to take the AOP route, the Spring Framework provides an easy to use AOP framework. Essentially, like much of Spring, you would use a combination of a xml config file and some java code to define the AOP functionality you are looking for.
In your case, I believe you would be looking to define an 'After Throwing Advice', in which you would of course have access to the exception thrown.
A good place to start in terms of documentation is the AOP Chapter in the Spring docs:
http://static.springsource.org/spring/docs/2.5.x/reference/aop.html
Oh, and I believe all Spring projects are open source as well =)
I know the question asks for an open source solution. I don't know of one but if the option is there then DynaTrace does exactly what you want. Good luck on your search.
There are tools such as FindBugs, PMD and Checkstyle which can identify some common Exception handling issues. I'm never seen a tool that specifically analyses your exception handling, if anyone knows I'll be interested!
I had this exact question, and I attempted to write something myself, and due to AOP nested proxying and lack of ability to use instrumenation / weaving, I gave up and just did a wide find and replace
One of he tools I did find back then for was AppSight by BMC software, but it's high cost was an issue
IntelliJ's Inspector can check code for many problems as you write it:
http://www.jetbrains.com/idea/documentation/inspections.jsp
But your problem sounds like it's more about education than technology. You need to educate your team on what proper exception handling means, when it should be done, etc. Tools will help, but not putting them into the code is the first place is better.
We use Spring aspects for our production systems to do logging, tracing, performance calculations, etc. Before, after, and exception advice work wonders - they keep the code in one place and give declarative flexibility as to where they are applied.
Just one caution: aspects aren't free. They add cost to each method you apply them to, so don't just pile them on. Moderation in all things is the key.
I didn't though about that yet but one solution, if you do not need to detect exceptions thrown on production envirionment, is to attach to your Java application a custom debugger that can be triggered whenever an exception is raised.
This french blog article talk about how to do it:
http://blog.xebia.fr/2011/12/12/legacy-code-gestion-des-exceptions-avec-jpda/
Here is the code:
Run with debug:
Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n
Connect to the JVM:
public static VirtualMachine connect(String port) throws IOException, IllegalConnectorArgumentsException {
AttachingConnector connector = null;
VirtualMachineManager vmManager = Bootstrap.virtualMachineManager();
for (Connector aconnector : vmManager.allConnectors()) {
if ("com.sun.jdi.SocketAttach".equals(aconnector.name())) {
connector = (AttachingConnector) aconnector;
break;
}
}
Map<String, Connector.Argument> args = connector.defaultArguments();
Connector.Argument pidArgument = args.get("port");
pidArgument.setValue(port);
return connector.attach(args);
}
Create your breakpoints. Exemple:
public static void createExceptionBreakPoint(VirtualMachine vm) {
EventRequestManager erm = vm.eventRequestManager();
List<ReferenceType> referenceTypes = vm.classesByName("java.lang.Throwable");
for (ReferenceType refType : referenceTypes){
ExceptionRequest exceptionRequest = erm.createExceptionRequest(refType, true, true);
exceptionRequest.setEnabled(true);
}
}
And then handle the exceptions:
public static void handleExceptionEvent(ExceptionEvent exceptionEvent) throws Exception {
ObjectReference remoteException = exceptionEvent.exception();
ThreadReference thread = exceptionEvent.thread();
List<Value> paramList = new ArrayList<Value>(1);
paramList.add(dumpFileName);
//crer un printStream dans la JVM cible
ObjectReference printStreamRef = printStreamClassType.newInstance(thread, printStreamConstructor, paramList,
ObjectReference.INVOKE_SINGLE_THREADED);
ReferenceType remoteType = remoteException.referenceType();
Method printStackTrace = (Method) remoteType.methodsByName("printStackTrace").get(1);
paramList.clear();
paramList.add(printStreamRef);
remoteException.invokeMethod(thread, printStackTrace, paramList, ObjectReference.INVOKE_SINGLE_THREADED);
Scanner scanner = new Scanner(new File(dumpFileName.value()));
while (scanner.hasNextLine()){
System.out.println(scanner.nextLine());
}
}
A bit heavy but it works, now how to catch the exceptions that are logged and the others?