Develop Reference

JUnit report to show test functionality, not coverage

One of the problems of a team lead is that people on the team (sometimes even including myself) often create JUnit tests without any testing functionality.
It's easily done since the developers use their JUnit test as a harness to launch the part of the application they are coding, and then either deliberately or forgetfully just check it in without any assert tests or mock verifies.
Then later it gets forgotten that the tests are incomplete, yet they pass and produce great code coverage. Running up the application and feeding data through it will create high code coverage stats from Cobertura or Jacoco and yet nothing is tested except its ability to run without blowing up - and I've even seen that worked-around with big try-catch blocks in the test.
Is there a reporting tool out there which will test the tests, so that I don't need to review the test code so often?
I was temporarily excited to find Jester which tests the tests by changing the code under test (e.g. an if clause) and re-running it to see if it breaks the test.
However this isn't something you could set up to run on a CI server - it requires set-up on the command line, can't run without showing its GUI, only prints results onto the GUI and also takes ages to run.

PIT is the standard Java mutation tester. From their site:
Mutation testing is conceptually quite simple.
Faults (or mutations) are automatically seeded into your code, then your tests are run. If your tests fail then the mutation is killed, if your tests pass then the mutation lived.
...
Traditional test coverage (i.e line, statement, branch etc) measures only which code is executed by your tests. It does not check that your tests are actually able to detect faults in the executed code. It is therefore only able to identify code the is definitely not tested.
The most extreme example of the problem are tests with no assertions. Fortunately these are uncommon in most code bases. Much more common is code that is only partially tested by its suite. A suite that only partially tests code can still execute all its branches (examples).
As it is actually able to detect whether each statement is meaningfully tested, mutation testing is the gold standard against which all other types of coverage are measured.
The quality of your tests can be gauged from the percentage of mutations killed.
It has a corresponding Maven plugin to make it simple to integrate as part of a CI build. I believe the next version will also include proper integration with Maven site reports too.
Additionally, the creator/maintainer is pretty active here on StackOverflow, and is good about responding to tagged questions.

As far as possible, write each test before implementing the feature or fixing the bug the test is supposed to deal with. The sequence for a feature or bug fix becomes:
Write a test.
Run it. At this point it will fail if it is a good test. If it does
not fail, change, replace, or add to it.
When you have a failing test, implement the feature it is supposed
to test. Now it should pass.

You have various options:
You probably could use some code analysis tool like checkstyle to verify that each test has an assertion. Or alternatively use a JUnit Rule to verify this, but both is easily tricked and works only on a superficial level.
Mutation testing as Jester does is again a technical solution which would work, and it seems #Tom_G has a tool that might work. But these tools are (in my experience) extremely slow, because the work by changing the code, running tests, analyzing result over and over again. So even tiny code bases take lots of time and I wouldn't even think about using it in a real project.
Code Reviews: such bad tests are easily caught by code reviews, and they should be part of every development process anyway.
All this still only scratches on the surface. The big question you should ponder is: why do developers feel tempted to create code just to start a certain part of the application? Why don't they write tests for what they want to implement, so there is almost no need for starting parts of the application. Get some training for automated unit testing and especially TDD/BDD, i.e. a process where you write the tests first.
In my experience it is very likely that you will hear things like: We can't test this because .... You need to find the real reason why the developers, can't or don't want to write these tests, which might or might not be the reasons they state. Then fix those reasons and those abominations of tests will go away all on their own.

What you are looking for is indeed mutation testing.
Regarding tool support, you might also want to look at the Major
mutation framework (mutation-testing.org), which is quite efficient and configurable. Major
uses a compiler-integrated mutator and gives you great control over
what should be mutated and tested. As far as I know Major does not yet
produce graphical reports but rather data (csv) files that you can
process or visualize in any way you want.

Sounds like you need to consider a coverage tool like Jacoco, the gradle plugin provides report on coverage. I also use the EclEmma Eclipse plugin to obtain the same results, but with a fairly nice integration in the IDE.
In my experience, Jacoco has provided acceptable numbers even when there are no-op unit test. As it seems able to accurately determine the tested code paths. No-op test get low or 0% coverage scores and the score increase as the test become more complete.
Update
To address the down-voter. Perhaps a more appropriate tool to address this is PMD. Can be used in an IDE or build system. With proper configuration and rule development it could be used to find these incomplete unit tests. I have used it in the past to find methods missing certain security related annotation in the past.

Why use JUnit for testing?

Maybe my question is a newbie one, but I can not really understand the circumstances under which I would use junit?
Whether I write simple applications or larger ones I test them with the System.out statements and it seams quite easy to me.
Why create test-classes with JUnit, unnecessary folders in the project if we still have to call the same methods, check what they return and we then have an overhead of annotating everything?
Why not write a class and test it at once with System.out but not create Test-classes?
PS. I have never worked on large projects I am just learning.
So what is the purpose?

That's not testing, that's "looking manually at output" (known in the biz as LMAO). More formally it's known as "looking manually for abnormal output" (LMFAO). (See note below)
Any time you change code, you must run the app and LMFAO for all code affected by those changes. Even in small projects, this is problematic and error-prone.
Now scale up to 50k, 250k, 1m LOC or more, and LMFAO any time you make a code change. Not only is it unpleasant, it's impossible: you've scaled up the combinations of inputs, outputs, flags, conditions, and it's difficult to exercise all possible branches.
Worse, LMFAO might mean visiting pages upon pages of web app, running reports, poring over millions of log lines across dozens of files and machines, reading generated and delivered emails, checking text messages, checking the path of a robot, filling a bottle of soda, aggregating data from a hundred web services, checking the audit trail of a financial transaction... you get the idea. "Output" doesn't mean a few lines of text, "output" means aggregate system behavior.
Lastly, unit and behavior tests define system behavior. Tests can be run by a continuous integration server and checked for correctness. Sure, so can System.outs, but the CI server isn't going to know if one of them is wrong–and if it does, they're unit tests, and you might as well use a framework.
No matter how good we think we are, humans aren't good unit test frameworks or CI servers.
Note: LMAO is testing, but in a very limited sense. It isn't repeatable in any meaningful way across an entire project or as part of a process. It's akin to developing incrementally in a REPL, but never formalizing those incremental tests.

We write tests to verify the correctness of a program's behaviour.
Verifying the correctness of a program's behaviour by inspecting the content of output statements using your eyes is a manual, or more specifically, a visual process.
You could argue that
visual inspection works, I check that the code does what it's meant to
do, for these scenarios and once I can see it's correct we're good to
go.
Now first up, it's great to that you are interested in whether or not the code works correctly. That's a good thing. You're ahead of the curve! Sadly, there are problems with this as an approach.
The first problem with visual inspection is that you're a bad welding accident away from never being able to check your code's correctness again.
The second problem is that the pair of eyes used is tightly coupled with the brain of the owner of the eyes. If the author of the code also owns the eyes used in the visual inspection process, the process of verifying correctness has a dependency on the knowledge about the program internalised in the visual inspector's brain.
It is difficult for a new pair of eyes to come in and verify the correctness of the code simply because they are not partnered up with brain of the original coder. The owner of the second pair of eyes will have to converse with original author of the code in order to fully understand the code in question. Conversation as a means of sharing knowledge is notoriously unreliable. A point which is moot if the Original Coder is unavailable to the new pair eyes. In that instance the new pair of eyes has to read the original code.
Reading other people's code that is not covered by unit tests is more difficult than reading code that has associated unit tests. At best reading other peoples code is tricky work, at its worst this is the most turgid task in software engineering. There's a reason that employers, when advertising job vacancies, stress that a project is a greenfield (or brand new) one. Writing code from scratch is easier than modifying existing code and thereby makes the advertised job appear more attractive to potential employees.
With unit testing we divide code up into its component parts. For each component we then set out our stall stating how the program should behave. Each unit test tells a story of how that part of the program should act in a specific scenario. Each unit test is like a clause in a contract that describes what should happen from the client code's point of view.
This then means that a new pair of eyes has two strands of live and accurate documentation on the code in question.
First they have the code itself, the implementation, how the code was done; second they have all of the knowledge that the original coder described in a set of formal statements that tell the story of how this code is supposed to behave.
Unit tests capture and formally describe the knowledge that the original author possessed when they implemented the class. They provide a description of how that class behaves when used by a client.
You are correct to question the usefulness of doing this because it is possible to write unit tests that are useless, do not cover all of the code in question, become stale or out of date and so on. How do we ensure that unit tests not only mimics but improves upon the process of a knowledgeable, conscientious author visually inspecting their code's output statements at runtime? Write the unit test first then write the code to make that test pass. When you are finished, let the computers run the tests, they're fast they are great at doing repetitive tasks they are ideally suited to the job.
Ensure test quality by reviewing them each time you touch off the code they test and run the tests for each build. If a test fails, fix it immediately.
We automate the process of running tests so that they are run each time we do a build of the project. We also automate the generation of code coverage reports that details what percentage of code that is covered and exercised by tests. We strive for high percentages. Some companies will prevent code changes from being checked in to source code control if they do not have sufficient unit tests written to describe any changes in behaviour to the code. Typically a second pair of eyes will review code changes in conjunction with the author of the changes. The reviewer will go through the changes ensure that the changes understandable and sufficiently covered by tests. So the review process is manual, but when the tests (unit and integration tests and possibly user acceptance tests) pass this manual review process the become part of the automatic build process. These are run each time a change is checked in. A continuous-integration server carries out this task as part of the build process.
Tests that are automatically run, maintain the integrity of the code's behaviour and help to prevent future changes to the code base from breaking the code.
Finally, providing tests allows you to aggressively re-factor code because you can make big code improvements safe in the knowledge that your changes do not break existing tests.
There is a caveat to Test Driven Development and that is that you have to write code with an eye to making it testable. This involves coding to interfaces and using techniques such as Dependency Injection to instantiate collaborating objects. Check out the work of Kent Beck who describes TDD very well. Look up coding to interfaces and study design-patterns

When you test using something like System.out, you're only testing a small subset of possible use-cases. This is not very thorough when you're dealing with systems that could accept a near infinite amount of different inputs.
Unit tests are designed to allow you to quickly run tests on your application using a very large and diverse set of different data inputs. Additionally, the best unit tests also account for boundary cases, such as the data inputs that lie right on the edge of what is considered valid.
For a human being to test all of these different inputs could take weeks whereas it could take minutes for a machine.
Think of it like this: You're also not "testing" something that will be static. Your application is most likely going through constant changes. Therefore, these unit tests are designed to run at different points in the compile or deployment cycle. Perhaps the biggest advantage is this:
If you break something in your code, you'll know about it right now, not after you deployed, not when a QA tester catches a bug, not when your clients have cancelled. You'll also have a better chance of fixing the glitch immediately, since it's clear that the thing that broke the part of the code in question most likely happened since your last compile. Thus, the amount of investigative work required to fix the problem is greatly reduced.

I added some other System.out can NOT do:
Make each test cases independent (It's important)
JUnit can do it: each time new test case instance will be created and #Before is called.
Separate testing code from source
JUnit can do it.
Integration with CI
JUnit can do it with Ant and Maven.
Arrange and combine test cases easily
JUnit can do #Ignore and test suite.
Easy to check result
JUnit offers many Assert methods (assertEquals, assertSame...)
Mock and stub make you focus on the test module.
JUnit can do: Using mock and stub make you setup correct fixture, and focus on the test module logic.

Unit tests ensure that code works as intended. They are also very helpful to ensure that the code still works as intended in case you have to change it later to build new functionalities to fix a bug. Having a high test coverage of your code allows you to continue developing features without having to perform lots of manual tests.
Your manual approach by System.out is good but not the best one.This is one time testing that you perform. In real world, requirements keep on changing and most of the time you make a lot of modificaiotns to existing functions and classes. So… not every time you test the already written piece of code.
there are also some more advanced features are in JUnit like like
Assert statements
JUnit provides methods to test for certain conditions, these methods typically start with asserts and allow you to specify the error message, the expected and the actual result
Some of these methods are
fail([message]) - Lets the test fail. Might be used to check that a certain part of the code is not reached. Or to have failing test before the test code is implemented.
assertTrue(true) / assertTrue(false) - Will always be true / false. Can be used to predefine a test result, if the test is not yet implemented.
assertTrue([message,] condition) - Checks that the boolean condition is true.
assertEquals([message,] expected, actual) - Tests whether two values are equal (according to the equals method if implemented, otherwise using == reference comparison). Note: For arrays, it is the reference that is checked, and not the contents, use assertArrayEquals([message,] expected, actual) for that.
assertEquals([message,] expected, actual, delta) - Tests whether two float or double values are in a certain distance from each other, controlled by the delta value.
assertNull([message,] object) - Checks that the object is null
and so on. See the full Javadoc for all examples here.
Suites
With Test suites, you can in a sense combine multiple test classes into a single unit so you can execute them all at once. A simple example, combining the test classes MyClassTest and MySecondClassTest into one Suite called AllTests:
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
import org.junit.runners.Suite.SuiteClasses;
#RunWith(Suite.class)
#SuiteClasses({ MyClassTest.class, MySecondClassTest.class })
public class AllTests { }

The main advantage of JUnit is that it is automated rather than you manually having to check with your print outs. Each test you write stays with your system. This means that if you make a change that has an unexpected side effect your test will catch it and fail rather than you having to remember to manually test everything after each change.

JUnit is a unit testing framework for the Java Programming Language. It is important in the test driven development, and is one of a family of unit testing frameworks collectively known as xUnit.
JUnit promotes the idea of "first testing then coding", which emphasis on setting up the test data for a piece of code which can be tested first and then can be implemented . This approach is like "test a little, code a little, test a little, code a little..." which increases programmer productivity and stability of program code that reduces programmer stress and the time spent on debugging.
Features
JUnit is an open source framework which is used for writing & running tests.
Provides Annotation to identify the test methods.
Provides Assertions for testing expected results.
Provides Test runners for running tests.
JUnit tests allow you to write code faster which increasing quality
JUnit is elegantly simple. It is less complex & takes less time.
JUnit tests can be run automatically and they check their own results and provide immediate feedback. There's no need to manually comb through a report of test results.
JUnit tests can be organized into test suites containing test cases and even other test suites.
Junit shows test progress in a bar that is green if test is going fine and it turns red when a test fails.

I have slightly different perspective of why JUnit is needed.
You can actually write all test cases yourself but it's cumbersome. Here are the problems:
Instead of System.out we can add if(value1.equals(value2)) and return 0 or -1 or error message. In this case, we need a "main" test class which runs all these methods and checks results and maintains which test cases failed and which are passed.
If you want to add some more tests you need to add them to this "main" test class as well. Changes to existing code. If you want to auto detect test cases from test classes, then you need to use reflection.
All your tests and your main class to run tests are not detected by eclipse and you need to write custom debug/run configurations to run these tests. You still don't see those pretty green/red colored outputs though.
Here is what JUnit is doing:
It has assertXXX() methods which are useful for printing helpful error messages from the conditions and communicating results to "main" class.
"main" class is called runner which is provided by JUnit, so we don't have to write any. And it detects the test methods automatically by reflection. If you add new tests with #Test annotation then they are automatically detected.
JUnit has eclipse integration and maven/gradle integration as well, so it is easy to run tests and you will not have to write custom run configurations.
I'm not an expert in JUnit, so that's what I understood as of now, will add more in future.

You cannot write any test case without using testing framework or else you will have to write your testing framewok to give justice to your test cases.
Here are some info about JUnit Framework apart from that you can use TestNG framework .
What is Junit?
Junit is widely used testing framework along with Java Programming Language. You can use this automation framework for both unit testing and UI testing.It helps us define the flow of execution of our code with different Annotations. Junit is built on idea of "first testing and then coding" which helps us to increase productivity of test cases and stability of the code.
Important Features of Junit Testing -
It is open source testing framework allowing users to write and run test cases effectively.
Provides various types of annotations to identify test methods.
Provides different Types of Assertions to verify the results of test case execution.
It also gives test runners for running tests effectively.
It is very simple and hence saves time.
It provides ways to organize your test cases in form of test suits.
It gives test case results in simple and elegant way.
You can integrate jUnit with Eclipse, Android Studio, Maven & Ant, Gradle and Jenkins

JUNIT is the method that is usually accepted by java developer.
Where they can provide similar expected input to the function and decide accordingly that written code is perfectly written or if test case fails then different approach may also need to implement.
JUNIT will make development fast and will ensure the 0 defects in the function.

JUNIT : OBSERVE AND ADJUST
Here is my perspective of JUNIT.
JUNIT can be used to,
1)Observe a system behaviour when a new unit is added in that system.
2)Make adjustment in the system to welcome the "new" unit in the system.
What? Exactly.
Real life eg.
When your relative visits your college hostel room,
1) You will pretend to be more responsible.
2) You will keep all things where they should be, like shoes in shoe rack not on chair, clothes in cupboard not on chair.
3) You will get rid of all the contraband.
4) you will start cleanUp in every device you posses.
In programming terms
System: Your code
UNIT: new functionality.
As JUNIT framework is used for JAVA language so JUNIT = JAVA UNIT (May be).
Suppose a you already have a well bulletproof code, but a new requirement came and you have to add the new requirement in your code. This new requirement may break your code for some input(testcase).
Easy way to adapt this change is using unit testing (JUNIT).
For that you should write multiple testcases for your code when you are building your codebase. And whenever a new requirement comes you just run all the test cases to see if any test case fails.
If No then you are a BadA** artist and you are ready to deploy the new code.
If any of the testcases fail then you change your code and again run testcases until you get the green status.

Getting into testing

I am at the stage now where I have a fairly good understanding of programming/development, using Java.
Could anyone tell me the best way for me to start using testing packages? I have looked at Hibernate but not surer where to go with it...
I use Eclipse 3.5 on Mac OS X. Is it a case of writing scripts to test methods? What is unit-testing? etc.
Where do I begin?
Many thanks. Alex

What is Unit Testing
Unit testing is writing code (i.e. test code) that passes known inputs into code under test and then validating the code under test returns expected outputs. It's the most granular testing you can perform on an application. To make it easier, usually a unit testing framework is used. For Java, JUnit is the most popular, but TestNG is also notable.
Getting Started
Unit testing frameworks provide tools for test execution, validation and results reporting. For your setup, Eclipse has built in support for JUnit. Eclipse is able to automatically detect tests, compile tests and code under test, execute tests, and report results within the IDE. Furthermore, failures are reported as clickable stack trace information that loads the corresponding file at the given line number.
Mock Objects
That you're also working with Hibernate, suggests you also investigate a mock object framework as well - such as jMock. Mock objects are usually substituted as part of a code under tests's composition and serve two purposes: (1) returning known outputs and (2) recording they've been called and how so that unit tests can introspect that information as part of validation.
The ability to use Mock objects to make testing easier is predicated on dependency injection. That is other entities that compose the object under test. The idea is decoupling dependencies (e.g. Hibernate) to focus on testing algorithms that manipulate that data you're working with.
Database
However, if you've got code that is not easily refactored, or perhaps you want to validate database code, you can also test Hibernate interaction as well. In that case you want a database in a known state. Three approaches come to mind:
Restoring a database backup at the beginning of each test execution.
Use dbunit, which provides its own mechanisms for maintain state.
Transactional locking with rollback. Wrap the entire case is wrapped with a try{} finally{}, where the latter always rolls back the transaction.

James Shore ("a thought leader in the Agile software development community") has a series of screen casts of him demonstrating Test Driven Development, using Eclipse.
http://jamesshore.com/Blog/Lets-Play/

While there are many ways to start testing, there is no "best" way so there's no point in looking for that as a starting point.
Search the web for a good tutorial on junit and do it. That will be the absolute best way to get started IMO. Don't get sidetracked with code coverage or integrating with Hudson or any of the other tasks that are on the periphery to testing. Focus on writing a handful (or 10) if tests first.
Once you understand the basics you can start looking at other tools to see if they meet your needs any better or worse than junit.

First up: Hibernate is not a testing package.
Now that's out of the way, I'd suggest you take a look at JUnit. Read up on unit testing first so you know what it is (the Wikipedia entry is a good place to start), then try the JUnit cookbook. Write some unit tests for a small piece of your code to see how it works, then move on to bigger chunks.
While you are at it, take a look at other development tools like Cobertura (for finding out how good your test coverage is) and static analysis tools like Findbugs and Checkstyle. These all integrate nicely with Ant and probably Eclipse, too.
If you are interested in improving your coding standards and build systems then I highly recommend using Ant, JUnit, Cobertura, Checkstyle and Findbugs together with a continuous integration server (e.g. Hudson or CruiseControl) and a version control system (e.g. git). With a toolkit like that you can't go wrong.
There are other frameworks out there (TestNG, Mockito etc) so take a look at them, too, and decide which you prefer (EDIT: And which work nicely together. Mockito + JUnit is a good combination.)

Reflection in unit tests for checking code coverage

Here's the scenario. I have VO (Value Objects) or DTO objects that are just containers for data. When I take those and split them apart for saving into a DB that (for lots of reasons) doesn't map to the VO's elegantly, I want to test to see if each field is successfully being created in the database and successfully read back in to rebuild the VO.
Is there a way I can test that my tests cover every field in the VO? I had an idea about using reflection to iterate through the fields of the VO's as part of the solution, but maybe you guys have solved the problem before?
I want this test to fail when I add fields in the VO, and don't remember to add checks for it in my tests.
dev environment:
Using JUnit, Hibernate/Spring, and Eclipse

Keep it simple: write one test per VO/DTO:
fill the VO/DTO with test data
save it
(optional: check everything has been correctly save at the database level, using pure JDBC)
load it
check that the loaded VO/DTO and the original one matches
Productive code will evolve and tests will need to be maintained as well. Making tests the simplest as possible, even if they are repetitive, is IMHO the best approach. Over-engineering the tests or testing framework itself to make tests generic (e.g. by reading fields with reflection and filling VO/DTO automatically) leads to several problems:
time spent to write the test is higher
bug might be introduced in the test themselves
maintenance of the test is harder because they are more sophisticated
tests are harder to evolve, e.g. the generic code will maybe not work for new kinds of VO/DTO that differ slightly from the other and will be introduced later (it's just an example)
tests can not be used easily as example of how the productive code works
Test and productive code are very different in nature. In productive code, you try to avoid duplication and maximize reuse. Productive code can be complicated, because it is tested. On the other hand, you should try to have tests as simple as possible, and duplication is ok. If a duplicated portion is broken, the test will fail anyway.
When productive code change, this may require several tests to be trivially changed. With the problem that tests are seen as boring piece of code. But I think that's the way they should be.
If I however got your question wrong, just let me know.

I would recommend cobertura for this task.
You will get a complete code coverage report after you run your tests and if you use the cobertura-check ant task you can add checks for the coverage and stop the ant call with the property haltonfailure.

You could make it part of the validation of the VO. If the fields aren't set when you use a getter it can throw an exception.

How to best test Java code?

I have been working on a comparatively large system on my own, and it's my first time working on a large system(dealing with 200+ channels of information simultaneously). I know how to use Junit to test every method, and how to test boundary conditions. But still, for system test, I need to test all the interfacing and probably so some stress test as well (maybe there are other things to do, but I don't know what they are). I am totally new to the world of testing, and please give me some suggestions or point me to some info on how a good code tester would do system testing.
PS: 2 specific questions I have are:
how to test private functions?
how to testing interfaces and avoid side effects?

Here are two web sites that might help:
The first is a list of open source Java tools. Many of the tools are addons to JUnit that allow either easier testing or testing at a higher integration level.
Depending on your system, sometimes JUnit will work for system tests, but the structure of the test can be different.
As for private methods, check this question (and the question it references).
You cannot test interfaces (as there is no behavior), but you can create an abstract base test classes for testing that implementations of an interface follow its contract.
EDIT: Also, if you don't already have unit tests, check out Working Effectivly with Legacy Code; it is a must for testing code that is not set up well for testing.

Mocking is a good way to be able to simulate system tests in unit testing; by replacing (mocking) the resources upon which the other component depends, you can perform unit testing in a "system-like" environment without needing to have the entire system constructed to do it.
As to your specific questions: generally, you shouldn't be using unit testing to test private functions; if they're private, they're private to the class. If you need to test something, test a public method which uses that private method to do something. Avoiding side effects that can be potentially problematic is best done using either a complete test environment (which can easily be wiped back to a "virgin" state) or using mocking, as described above. And testing interfaces is done by, well, testing the interface methods.

Firstly, if you already have a large system that doesn't have any unit tests, and you're planning on adding some, then allow me to offer some general advice.
From maintaining the system and working with it, you'll probably already know the areas of the system which tend to be buggiest, which tend to change often and which tend not to change very much. If you don't, you can always look through the source control logs (you are using source control, right?) to find out where most of the bug fixes and changes are concentrated. Focus your testing efforts on these classes and methods. There's a general rule called the 80/20 rule which is applicable to a whole range of things, this being one of them.
It says that, roughly on average, you should be able to cover 80 percent of the offending cases by doing just 20% of the work. That is, by writing tests for just 20% of the code, you can probably catch 80% of the bugs and regressions. That's because most of the fragile code, commonly changed code and worst offending code makes up just 20% of the codebase. In fact, it may be even less.
You should use junit to do this and you should use something like JMock or some other mocking library to ensure you're testing in isolation. For system testing/integration testing, that is, testing things while they're working together, I can recommend FitNesse. I've had good experience with it in the past. It allows you to write your test in a web browser using simple table-like layouts, where you can easily define your inputs and expected outputs. All you have to do is write a small backing class called a Fixture, which handles the creation of the components.

Private functions will be tested when the public functions that call them. Your testing of the public function only cares that the result returned is correct.
When dealing with API (to other packages or URLS or even to file/network/database) you should mock them. A good unit test should run in a few milliseconds not in seconds. Mocking is the only way to do that. It means that bugs between packages can be dealt with a lot easier than logical bugs at the functional level. For Java easymock is a very good mocking framework.

You may have a look on this list : Tools for regression testing / test automation of database centric java application? for a list of interesting tools.
As you seem to already use Junit extensively it means that you're already "test infected", that is a good point...
In my personal experience, the most difficult thing to manage is data. I mean, controlling very acutely the data agaisnt which the tests are runned.

The lists of tools given before are useful. From personal experience these are the tools I find useful:
Mocking - Mockito is an excellent implementation and has clever techniques to ensure you only have to mock the methods you really care about.
Database testing - DBunit is indespensible for setting up test data and verifying database interactions.
Stress testing - Jmeter - once you see passed the slightly clunky gui this is a very robust tool for setting up scenarios and running stress tests.
As for general approach start by trying to get tests running for the usual "happy paths" through your application these can form a basis for regression testing and performance testing. Once this is complete you can start looking at edge cases and error scenarios.
Although this level of testing should be secondary to good unit testing.
Good luck!