Java - retrieving large amounts of data from a DB using iBatis - java

I need to extract data from a DB2 table, run some processing on each returned row and output to a flat file. I'm using iBatis but found that using the queryForList I started getting out of memory errors, I'll be looking at 100k+ rows of data increasing.
I've looked at using queryWithRowHandler instead but the iBatis RowHandler interface doesn't throw an exception from its handleRow function so if it gets an error I can't properly report it back and stop iterating the rest of the data. It looks like I can throw a RuntimeException but that doesn't strike me as a neat way of doing things.
I'd like to be able to stop processing while throwing a meaningful Exception indicating whether the error occurred on the data manipulation, the file access or whatever.
Has anyone had experience with this approach or have an alternative solution using iBatis. I know I could look to do this without iBatis, just using JDBC, but as iBatis is used for all other DB access in my app I'd like to avail of this architecture if possible.

1) Create your own RowHandler interface with checked Exceptions in signature:
public interface MySpecialRowHandler {
public void handleRow(Object row)
throws DataException, FileException, WhateverException;
}
2) Inherit (or even better, delegate ) from SqlMapDaoTemplate to add a new method that will manage your own handler with the same Exceptions in signature:
public class MySpecialTemplate extends SqlMapDaoTemplate {
...
public void queryWithRowHandler(String id,
final MySpecialRowHandler myRowHandler
) throws DataException, FileException, WhateverException {
// "holder" will hold the exception thrown by your special rowHandler
// both "holder" and "myRowHandler" need to be declared as "final"
final Set<Exception> holder = new HashSet<Exception>();
this.queryWithRowHandler(id,new RowHandler() {
public void handleRow(Object row) {
try {
// your own row handler is executed in IBatis row handler
myRowHandler.handleRow(row);
} catch (Exception e) {
holder.add(e);
}
}
});
// if an exception was thrown, rethrow it.
if (!holder.isEmpty()) {
Exception e = holder.iterator().next();
if (e instanceof DataException) throw (DataException)e;
if (e instanceof FileException) throw (FileException)e;
if (e instanceof WhateverException) throw (WhateverException)e;
// You'll need this, in case none of the above works
throw (RuntimeException)e;
}
}
}
3) Your business code will look like this:
// create your rowHandler
public class Db2RowHandler implements MySpecialRowHandler {
void handleRow(Object row) throws DataException, FileException, WhateverException {
// what you would have done in ibatis RowHandler, with your own exceptions
}
}
// use it.
MySpecialTemplate template = new MySpecialTemplate(daoManager);
try {
template.queryWithRowHandler("selectAllDb2", new Db2RowHandler());
} catch (DataException e) {
// ...
} catch (FileException e) {
...

Related

How to handle a PSQLException in java?

I have a unique constraint on one of my entities and whenever I get a PSQLException which occurs whenever that constraint is violated, I want to respond with a bad request.
This is my exception handler which I tried to implement:
#ControllerAdvice
public class DatabaseExceptionHandler {
#ExceptionHandler(value = PSQLException.class)
#ResponseStatus(HttpStatus.BAD_REQUEST)
public void handleDatabaseExceptions(PSQLException e) {
// i want to respond with a bad request only when this condition is satisfied
//
// if (e.getSQLState().equals("23505")) {
//
// }
}
}
And this is where the model is saved in db:
public DepartmentForHoliday setDepartment(DepartmentForHoliday department) {
if (department.getDepartmentId() == null) {
Department savedDepartment = new Department();
savedDepartment.setName(department.getName());
try {
departmentRepository.save(savedDepartment);
} catch (PSQLException e) {
/*here i have a compiler error which says that this exception is never thrown in the corresponding try block, but where ?*/
}
}
This is the exception that is thrown when I add a duplicate entry:
org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "uk_1t68827l97cwyxo9r1u6t4p7d"
Detail: Key (name)=(Tech) already exists.
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2458) ~[postgresql-9.4.1211.jre7.jar:9.4.1211.jre7]
How to handle PSQLExceptions ? Should I make my own exception as a wrapper or how to solve this problem ?
Key problem is that PSQLException is wrapped into some Spring exception (which I assume from your code); you have to unwrap it (for example using guava's Throwables):
public DepartmentForHoliday setDepartment(DepartmentForHoliday department) {
if (department.getDepartmentId() == null) {
Department savedDepartment = new Department();
savedDepartment.setName(department.getName());
try {
departmentRepository.save(savedDepartment);
} catch (RuntimeException e) {
Throwable rootCause = com.google.common.base.Throwables.getRootCause(e);
if (rootCause instanceof SQLException) {
if ("23505".equals(((SQLException) rootCause).getSQLState())) {
// do smth interesting :)
}
}
}
}
}
Once you do that you can throw your custom exception and handle it in DatabaseExceptionHandler
You are catching PSQLException. Instead of that, please catch SQLException. With SQLException you will can handle all this SQL exceptions.
You can check the SQLException knowledge at this link
Then in your code just treat the SQLException as you want. The most generic catch clause is the following one:
catch (SQLException e)
{
System.out.println("ERROR: Fetch statement failed: " +
e.getMessage());
}
With this code you are printing the exception. If you want more information, check this
This is quite late, but building on previous responses I was able to solve it as so:
try {
return this.projectRepository.saveAndFlush(patchedProjectEntity);
} catch (DataIntegrityViolationException e) {
if (e.getMostSpecificCause().getClass().getName().equals("org.postgresql.util.PSQLException") && ((SQLException) e.getMostSpecificCause()).getSQLState().equals("23505"))
throw new UniqueConstraintViolationException("", e.getMostSpecificCause());
throw e;
}
Where UniqueConstraintViolationException is a custom exception and handled with a spring controller advice.
You might as well register an exception handler for that wrapped exception (that #radek mentioned) directly.
In your case that's:
#ExceptionHandler(DataIntegrityViolationException::class)
protected fun handleDataIntegrityException(ex: DataIntegrityViolationException, request: WebRequest) : ResponseEntity<SomeBody>{
return ResponseEntity.badRequest().body(someBodyHere)
}
The error is converted within convertHibernateAccessException in org.springframework.orm.jpa.vendorHibernateJpaDialect, which has already processed away from PSQL. You can add a breakpoint there and follow the stacktrace.
There is a lot of proxy'ing happening under the hood, but the takeaway is that there is always a readable, expressive Exception to use directly.

Throwing exceptions through components, good practice?

When a sub method throws an exception, would encapsulation in a dedicated "package" exception be considered good pratice ?
public String doStuff() throws UtilsException {
try {
throw new NullPointerException("test");
} catch (NullPointerException e) {
throw new UtilsException("something occured", e);
}
}
//use this exception for all classes of this package / component
public class UtilsException extends Exception {
private static final long serialVersionUID = 1L;
public UtilsException() {
super();
}
public UtilsException(String message, Throwable cause) {
super(message, cause);
}
public UtilsException(String message) {
super(message);
}
public UtilsException(Throwable cause) {
super(cause);
}
}
Could Optional.empty() be an alternative to avoid throwing/catching of a complex app?
public Optional<String> doStuff() throws UtilsException {
try {
return Optional.of("ok");
} catch (NullPointerException e) {
LOG.error("Something append... {}", e.getMessage());
return Optional.empty();
}
}
First, you should never catch a NullPointerException (or runtime exceptions in general) an return someting else like you are doing.
Ok, maybe there are a very few cases where you need to do that (like a buggy third party api).
Exceptions like those (NullPointer, ClassCast, IllegalArgument, ect) happen when your program has a bug and you should let
them bubble up and handle them in some high order component of your program.
That being said, (and there comes the infamous phrase) it depends...
Exceptions are "responsible" for informing errors,thus they need to be informative for the caller will use them to decide what to do. Consider the following:
public void readFile(String path) throws IOException {
// read file content
return content;
}
try {
return readFile("foo.txt");
} catch(FileNotFound e) {
// For this specific scenario not finding the file is not a problem
return "";
} catch(IOException e) {
// This we are not expecting to happen, if the file exists we should be
// able to read it, otherwise we should inform the user.
log.error(e);
display("We had a problem reading the file, Check the file permissions and try again");
}
As you can see in the example above, you won't want to wrap the IOException in another exception in this case
because you will remove the client's ability to decide what to do when an error happened.
Also, note that the IOException is a form of "wrap" since exceptions are objects too you can use inheritance
to generalize what kind of errors your method throws and then throw more specific errors so the caller can
decide what to do.
When to wrap.
There are cases when wrapping exceptions is a good practice and is the way to go.
For example, if you are creating a lib whose main functionality is to get weather information.
For the first version you kept it simple and used a third party api to get the values for the day.
The main method of your api looks like this.
public Weather getWeather(Date day) throws HTTPException {
return weather.get(day);
}
Your api is doing pretty well but you noticed you're doing too much requests to the weather api and
you will have to start paying for it very soon. You then decided to cache the results in a database table
so you can reduce the amount of requests.
public Weather getWeather(Date day) throws HTTPException, SQLException {
Weather w = getFromCache(day);
if (w != null) {
return w;
} else {
return getAndCache(day);
}
}
Now you have a problem, you can't add this new exception to the throws statement because you will most certainly break
your api's users code.
And if you think about it, your api's users are no interested if you had problems getting the data from the wheter api or
from your cache, they just want to be informed of errors. This is a very good case to wrap those exceptions in
a more generic one, like WeatherFetchException.
As you can see, it really depends...
The rule of thumb to me is, keep your exceptions meaningful and if you want to wrap them, do only when
it makes sense and when it doesn't remove the caller's ability to handle errors.
Wrapping exceptions just for the sake of it is most definitely not a good practice.

ATG: Update Order Pattern

It has been known that you must use the following pattern in order to update an order in ATG Form-Handlers that doesn't inherit from the PurchaseProcessFormHanlder:
boolean acquireLock = false;
ClientLockManager lockManager = getLocalLockManager();
try {
acquireLock = !lockManager.hasWriteLock(profile.getRepositoryId(), Thread.currentThread());
if (acquireLock) {
lockManager.acquireWriteLock(profile.getRepositoryId(), Thread.currentThread());
}
boolean shouldRollback = false;
TransactionDemarcation transactionDemarcation = new TransactionDemarcation();
TransactionManager transactionManager = getTransactionManager();
transactionDemarcation.begin(transactionManager, TransactionDemarcation.REQUIRED);
try {
synchronized (getOrder()) {
...
...
...
}
} catch (final Exception ex) {
shouldRollback = true;
vlogError(ce, "There has been an exception during processing of order: {0}", getOrder().getId());
} finally {
try {
transactionDemarcation.end(shouldRollback);
} catch (final TransactionDemarcationException tde) {
vlogError(tde, "TransactionDemarcationException during finally: {0}", tde.getMessage());
} finally {
vlogDebug("Ending Transaction for orderId: {0}", order.getId());
}
}
} catch (final DeadlockException de) {
vlogError(de, "There has been an exception during processing of order: {0}", order.getId());
} catch (final TransactionDemarcationException tde) {
vlogError(tde, "There has been an exception during processing of order: {0}", order.getId());
} finally {
try {
if (acquireLock) {
lockManager.releaseWriteLock(getOrder().getProfileId(), Thread.currentThread(), true);
}
} catch (final Throwable th) {
vlogError(th, "There has been an error during release of write lock: {0}", th.getMessage());
}
}
In theory, any FormHandler that inherits from the PurchaseProcessFormHandler already implements the following steps OOTB:
Acquire LocalLockManager in order to avoid concurrent threads to modify the same order:
try {
acquireLock = !lockManager.hasWriteLock(profile.getRepositoryId(), Thread.currentThread());
if (acquireLock) {
lockManager.acquireWriteLock(profile.getRepositoryId(), Thread.currentThread());
}
} catch (final DeadlockException de) {
vlogError(de, "There has been an exception during processing of order: {0}", order.getId());
}
Create a new Transaction:
try {
TransactionDemarcation transactionDemarcation = new TransactionDemarcation();
TransactionManager transactionManager = getTransactionManager();
transactionDemarcation.begin(transactionManager, TransactionDemarcation.REQUIRED);
} catch (final TransactionDemarcationException tde) {
vlogError(tde, "There has been an exception during processing of order: {0}", order.getId());
}
Ending the Transaction being used:
try {
TransactionManager transactionManager = getTransactionManager();
Transaction transaction = transactionManager.getTransaction();
// If transaction is elegible for commiting:
transactionManager.commit();
transaction.commit();
// otherwise
transactionManager.rollback();
transaction.rollback();
} catch (final Exception ex) {
error = true;
vlogError(ex, "There has been an exception during processing of order: {0}", order.getId());
} finally {
// handle the error
}
Release the lock being used for the transaction:
finally {
ClientLockManager lockManager = getLocalLockManager();
lockManager.releaseWriteLock(profile.getRepositoryId(), Thread.currentThread(), true);
}
As per ATG documentation, the following methods implement the behaviour descripted above:
Method: beforeSet
Called before any setX methods on this form are set when a form that modifies properties of this form handler is submitted. Creates a transaction if necessary at the beginning of the form submission process, optionally obtaining a local lock to prevent multiple forms from creating transactions that may modify the same order.
Steps: 1 & 2
Method: afterSet
Called after any setX methods on this form are set when a form that modifies properties of this form handler is submitted. Commits or rolls back any transaction created in beforeSet, and releases any lock that was acquired at the time.
Steps: 3 & 4
Such as you will only have to handle the following procedures in order to update the order:
Syncronize the block of code that's going to be used for order updating in order to avoid thread concurrency.
synchronized (getOrder()) {
...
...
...
}
Perform order modifications:
synchronized (getOrder()) {
getOrder().setXXX();
getOrder().removeXXX();
}
Update the order (updateOrder pipeline chain will be invoked):
synchronized (getOrder()) {
...
...
...
getOrderManager().updateOrder(order);
}
This is pretty straightforward, unless you have to edit an order in any of the following scenarios:
Form handlers or custom form handler that are not in the PurchaseProcessFormHandler's hierachy.
Helpers or Tools classes.
Processors
ATG REST Web Services
&c
If so, you will have to implement the Transactional Pattern within your components.
Questions!
Is there any other pattern known to use instead of using the transactional pattern?
Would it be possible to implement/override the beforeSet & afterSet methods in FormHandlers just the same way ATG does it in PurchaseProcessFormHandler
Are you aware of any other approach?
The series of steps you have outlined above is the prescribed series of steps for updating an order.
Feel free to factor it out in any way you find useful. Just ensure that when you update an order, you, or your inherited code, have performed the requisite steps.
One common way that ATG does similar factoring is for a given method, say X(...), you would have a preX(...), doX(...), and postX(...) method. You can create an abstract class with all your boilerplate code in the preX() and postX() methods, maybe even declared final, and have the doX() declared abstract. Your component then will inherit from the abstract class and must implement the doX() method. You may need to handle exceptions explicitly as well.
This is, essentially, what the standard form handlers do (under different names).
For example;
public final void X(...) {
preX(...); // call the pre method
try {
doX(...); // call the do method
} catch (XException xe) {
// handle error
}
postX(...); // call the post method
}
protected final void preX(...) {
// do everything you need to do before your customer code
}
protected final void postX(...) {
// do everything you need to do after your customer code
}
protected abstract void doX(...) throws XException;
Another thing you could do, instead of inheriting from an abstract class, is to define an annotation that has all the boilerplate code.
A third thing you could do, in a similar way, but a lot harder to shoehorn into your ATG code, might be to define an aspect or a method invocation interceptor using third party frameworks.
However, once again, whatever you do, and however you do it, just ensure that you follow all the steps.

Where can I add exception handling in a Java class definition?

So, I had to create two files. One is a class definition. The other one uses the class' methods/fields.
(Artifact.java) Artifact Class definition:
public class Artifact {
int artNumber;
String arcName;
String artType;
int artYear;
double artWeight;
Artifact(int artNumber, String arcName, String artType, int artYear,double artWeight) {
this.artNumber = artNumber;
this.arcName = arcName;
this.artType = artType;
this.artYear = artYear;
this.artWeight = artWeight;
}
public void changeArtYear(int x) {
this.artYear = x;
}
public void changeArcName(String x) {
this.arcName = x;
}
public int getArtNumber() {
return artNumber;
}
public String getArcName() {
return arcName;
}
public String getArtType() {
return artType;
}
public int getArtYear() {
return artYear;
}
public double getArtWeight() {
return artWeight;
}
public String toString(){
return("The artifact #"+artNumber+" was discovered by "+arcName+". The artifact is made of "+artType+" and was discovered in "+artYear+". The artifact weighs "+artWeight+" kilograms.");
}
}
(ArtifactTester.java) Testing methods:
public class ArtifactTester {
public static void main(String[] args){
Artifact test = new Artifact(88888888,"ben","clay",1624,46.4);
System.out.println(test.toString()); //toString()
System.out.println(test.getArtWeight()); //getArtWeight()
System.out.println(test.getArtYear()); //getArtYear()
System.out.println(test.getArtType()); //getArtType()
System.out.println(test.getArcName()); //getArcName()
System.out.println(test.getArtNumber()); //getArtNumber()
test.changeArcName("zack");
test.changeArtYear(1400);
System.out.println(test.getArcName()); //getArcName()
System.out.println(test.getArtYear()); //getArtYear()
}
}
Anyways, my teacher to told me to add exception handling, but I am not sure where I would add exception handling.
Question: Is it possible to use exception handling in this situation?
Well to be blunt. Yes. Of course. You can use exception handling wherever and whenever you please (most of the time). Although, in this specific case I don't really see a good reason for it. But, I'll take your word for the need.
Now, as for where to handle exceptions, this is up to you. You can add exception handling in one of two places. You can either add exception handling when you call the methods like this:
try { //try executing a block of code which may throw exception
test.toString()
}
catch(Exception e) { //use Exception for all types of exceptions, or make it specific
//do something here if the exception is thrown
}
or you can excpetion handle in the methods themselves like so:
public void changeArtYear(int x) {
try{
this.artYear = x;
}
catch(Exception e){ //catch the exception that could be thrown
//do something
}
}
This should do the trick in your case if you want to add exception handling here. However, I would strongly urge you to learn exception handling and the different exceptions in Java, it is one of the most improtant fundamentals to programming in this language.
Also, let me point this out again: In this program, there is really no need to use exception handling except for practice. There is nothing here that would throw an exception for any reason. (Except maybe a NullPointerException if you passed a null parameter through one of your method calls)
Good Reference/Tutorial:
http://www.tutorialspoint.com/java/java_exceptions.htm
This site is an excellent java reference point in general, but specifically for your question today, this page shows you how to work with exceptions.
Is it possible to use exception handling in this situation?
I don't think so. You should probably go and ask your teacher.
In your code, Artifact is just a POJO (Plain Old Java Object). It would not throw any exceptions. All you do in the class is getters and setters, right? How can that throw any exceptions?
You can throw exceptions though. In your setters, you can check whether the argument is null before setting it to the fields. For example:
public void changeArcName(String x) {
if (x == null) throw new ArgumentException ("x is null!");
this.arcName = x;
}
Alternatively, you can just use brute force and use try...catch. like this:
Artifact test = new Artifact(88888888,"ben","clay",1624,46.4);
try {
System.out.println(test.toString()); //toString()
System.out.println(test.getArtWeight()); //getArtWeight()
System.out.println(test.getArtYear()); //getArtYear()
System.out.println(test.getArtType()); //getArtType()
System.out.println(test.getArcName()); //getArcName()
System.out.println(test.getArtNumber()); //getArtNumber()
test.changeArcName("zack");
test.changeArtYear(1400);
System.out.println(test.getArcName()); //getArcName()
System.out.println(test.getArtYear()); //getArtYear()
} catch (Exception ex) {
ex.printStackTrace ();
}
Warning: The catch block can never be reached!
I don't know whether the above is what your teacher wants. Just try both methods and hand it in and see what he/she says!

Capturing parameters of a method at runtime in Java

Our application uses several back-end services and we maintain wrappers which contain the methods to make the actual service calls. If any exception occurs in any of those methods while invoking a service, we throw a custom exception encapsulating the original exception as shown below.
interface IServiceA {
public void submit(String user, String attributes);
}
public class ServiceAWrapper implements IserviceA {
private ActualService getActualService() {
.....
}
public void submit(String user, String attributes) {
try {
Request request = new Request();
request.setUser(user);
request.setAttributes(attributes);
getActualService().call(request);
} catch(ServiceException1 e) {
throw new MyException(e, reason1);
} catch(ServiceException2 e) {
throw new MyException(e, reason2);
}
}
}
I would like to know if there's any framework that would allow me to
capture (and probably log) all the
parameters passed to my wrapper
methods at run-time; if the methods
are called.
capture the actual exception
object(MyException instance in above
example), if any thrown; so that I
could append the passed parameters
to the object at run-time.
I am currently exploring AspectJ to see if it can address my requirements, but I am not sure if it can be used to capture the parameters passed to methods at runtime and also to capture exception objects, if any occur.
Thanks.
With AspectJ, you can use around advice to execute advice instead of the code at the join point. You can then execute the actual join-point from within the advice by calling proceed. This would allow you to capture the input parameters, log them, and proceed to call the actual method.
Within the same advice you could capture any logs throw from the method, and inspect or log them before passing it back up to higher levels.
Matt B's answer is right. Specifically, you can do something like this:
aspect MonitorServiceCalls {
private final Logger LOG = LoggerFactory.getLog("ServiceCallLog");
Object around() throws MyException: call(public * *(..) throws MyException)
&& target(IServiceA+) {
MethodSignature msig = (MethodSignature)thisJoinPoint;
String fullMethName = msig.getMethod().toString();
try {
Object result = proceed();
LOG.info("Successful call to {} with arguments {}",
fullMethName,
thisJoinPoint.getArgs());
return result;
} catch(MyException e) {
LOG.warn("MyException thrown from {}: {}", msig.getMethod(), e);
throw e;
}
}
}
AspectJ is the right option. You will be able to get hold of the parameters by way of a JoinPoint object that will be passed to your advise methods. You can also get hold of the exception either by implementing an after throwing advise or an around advise.

Categories

Resources