Develop Reference

Is there any need to switch to modules when migrating to Java 9 or later?

We're currently migrating from Java 8 to Java 11. However, upgrading our services was less painful, than we anticipated. We basically only had to change the version number in our build.gradle file and the services were happily up and running. We upgraded libraries as well as (micro) services that use those libs. No problems until now.
Is there any need to actually switch to modules? This would generate needless costs IMHO. Any suggestion or further reading material is appreciated.
To clarify, are there any consequences if Java 9+ code is used without introducing modules? E.g. can it become incompatible with other code?

No.
There is no need to switch to modules.
There has never been a need to switch to modules.
Java 9 and later releases support traditional JAR files on the
traditional class path, via the concept of the unnamed module, and will
likely do so until the heat death of the universe.
Whether to start using modules is entirely up to you.
If you maintain a large legacy project that isn’t changing very much,
then it’s probably not worth the effort.
If you work on a large project that’s grown difficult to maintain over
the years then the clarity and discipline that modularization brings
could be beneficial, but it could also be a lot of work, so think
carefully before you begin.
If you’re starting a new project then I highly recommend starting with
modules if you can. Many popular libraries have, by now, been upgraded
to be modules, so there’s a good
chance that all of the dependencies that you need are already available
in modular form.
If you maintain a library then I strongly recommend that you
upgrade it to be a module if you haven’t done so already, and if all of
your library’s dependencies have been converted.
All this isn’t to say that you won’t encounter a few stumbling blocks
when moving past Java 8. Those that you do encounter will, however,
likely have nothing to do with modules per se. The most common
migration problems that we’ve heard about since we released Java 9 in
2017 have to do with changes to the syntax of the version
string and to the removal or
encapsulation of internal APIs
(e.g., sun.misc.Base64Decoder) for which public, supported
replacements have been available for years.

I can only tell you my organization opinion on the matter. We are in the process of moving to modules, for every single project that we are working on. What we are building is basically micro-services + some client libraries. For micro-services the transition to modules is somehow a lower priority: the code there is already somehow isolated in the docker container, so "adding" modules in there does not seem (to us) very important. This work is being picked up slowly, but it's low priority.
On the other hand, client libraries is an entirely different story. I can not tell you the mess we have sometimes. I'll explain one point that I hated before jigsaw. You expose an interface to clients, for everyone to use. Automatically that interface is public - exposed to the world. Usually, what I do, is have then some package-private classes, that are not exposed to the clients, that use that interface. I don't want clients to use that, it is internal. Sounds good? Wrong.
The first problem is that when those package-private classes grow, and you want more classes, the only way to keep everything hidden is to create classes in the same package:
package abc:
-- /* non-public */ Usage.java
-- /* non-public */ HelperUsage.java
-- /* non-public */ FactoryUsage.java
....
When it grows (in our cases it does), those packages are way too big. Moving to a separate package you say? Sure, but then that HelperUsage and FactoryUsage will be public and we tried to avoid that from the beginning.
Problem number two: any user/caller of our clients can create the same package name and extend those hidden classes. It happened a few times to us already, fun times.
modules solves this problem in a beautiful way : public is not really public anymore; I can have friend access via exports to directive. This makes our code lifecycle and management much easier. And we get away from classpath hell. Of course maven/gradle handle that for us, mainly, but when there is a problem, the pain will be very real. There could be many other examples, too.
That said, transition is (still) not easy. First of all, everyone on the team needs to be aligned; second there are hurdles. The biggest two I still see is: how do you separate each module, based on what, specifically? I don't have a definite answer, yet. The second is split-packages, oh the beautiful "same class is exported by different modules". If this happens with your libraries, there are ways to mitigate; but if these are external libraries... not that easy.
If you depend on jarA and jarB (separate modules), but they both export abc.def.Util, you are in for a surprise. There are ways to solve this, though. Somehow painful, but solvable.
Overall, since we migrated to modules (and still do), our code has become much cleaner. And if your company is "code-first" company, this matters. On the other hand, I have been involved in companies were this was seen as "too expensive", "no real benefit" by senior architects.

Java reusablitity

What is the proper way to make java reusable components that is capable of used in various applications. I mean for example I'm doing a application that has its own user interface and database and etc. If I want to make this app reusable for many other applications as a component of other applications. For example one feature of my first app may needed by other app. So how to make it possible for other apps to use this feature of my app without changing my original code. What are the proper ways to achieve this re usability.

Write something simple which does what it does very well. Document it and unit test it and make it open source.

Have the the reusable components in another project (e.g. "common") and package them as .jar. Then include that jar in the projects where it's needed.
Extracting a separate project might be tricky though. You should observer the following:
the common components should not be dependent on anything in the higher level of abstraction (i.e. you services must not have any UI-related dependencies)
the internals of the components must not be visible to the application using them. I.e. your jar should expose a minimum API.
You have a couple of options for the mechanics of packaging:
simple IDE-dependant packaging - declare a inter-project dependency. On build export the jar and put on the classpath of the client application
Maven/Ivy - install the dependency in a repository (local or remote) and use the dependency resolution mechanisms of maven/ivy

This is a rather broad question. As such, I am offering broad suggestions:
Know your OO basics. Inheritance, encapsulation, polymorphism. It gets crazier from there on out.
Learn about design patterns, start observing them in applications you already use.
Look at popular open libraries to see how they implement patterns and modules.
Try things in sandbox projects. Grow your knowledge in clean environments.
Since you mention Java, check out the Spring Framework.
Hope that helps.

You need code in such a way that your components are loosely coupled. Then the re-usability is very much high. Take a look at this and this.

Sun Microsystems, the creators of the Java language, have at last recognized this need, and have released the Java Beans Component Architecture. Java Beans are, quite simply, reusable controls written in Java, for Java application development.
Beans are "capsules" of code, each designed for a specific purpose. The advantage of Java Beans over standard programming controls is that Beans are independent. They are not specific to operating systems or development environments. A Bean created in one development environment can be easily copied and modified by another. This allows Java Beans greater flexibility in enterprise computing, as components are easily shared between developers.

Why use many sub-projects and dependencies over packages?

I’ve mostly in my career worked in small or mid-sized java projects. I recently saw a huge project comprising of 30 projects in eclipse. I don’t really get concept of creating many small projects and then maintain inter-project dependencies. When do we prefer this over simply organizing stuff in packages?
I guessed it’s a maven thing (have mostly been using Ant). I’ve been reading up on Maven’s concept of modules as well – I saw some links on net recommending creation of different modules for web, dao and service layers under a parent module. Is it really a common/best practice?
With or without maven – does such division really makes life easier? Isn’t it more compact to have everything in a single project with well-defined packages structure for different layers?

It's common to split up projects into API, implementation, web, etc. components–when there's a need to do so. Large projects are just that: large.
There are benefits to keeping components separate"
Re-use functionality (e.g., the web layer uses the service layer
Package individual components (e.g., ship jus the API to a client)
Version sub-components; define their version dependencies
You can do all the same stuff with one giant project, but it's more difficult to determine what goes where, and why. Life is easier when those lines of demarcation are clearly defined.
How much easier depends on the project, but when you're dealing with hundreds of thousands of lines of code, occasionally millions, breaking that stuff up saves huge headaches.

Why choose to create a separate module in maven? To help you with your development. There really is no other reason.
The are a number of different reasons why you may want to create a separate module:
Separation of concerns: yes, you can do this with packages, but if it's in a separate module, then it can be compiled separately, and you can reduce the amount of tangle[*] in your packages.
The modules are managed by different teams, with release cycles of their own.
More understandable code: If all of your dao code is in one module, and all of your web in another, you can test them separately.
A module can be a separate deployable entity. I have a project which has two web apps, 5 batches and two other core modules (one core for the webapp and one core for the batches). I can now build and deploy each module separately.
The modules are published and used externally. If this is true, then you want the minimum amount of 'other' code in this module.
You choose to break up into modules for the same reasons as you would for separating into packages, but at a higher level, at a group of packages.
30 does seem excessive. But there may be good reasons for it. It's up to you and your project to decide what is the right level for the number of modules.
Personally, I try not to split excessively, unless there is a very good reason to do so.
[*] Tangle: mess which describes the links between packages. Package A uses B, which uses C, which uses both A and B. Something which doesn't help understanding.

I think excessive modularisation is akin to over engineering. The best route in my opinion is to start with one module/project and keep with that until such point as it becomes obvious to everyone involved that part of this existing module would benefit from being extracted into its own module. Yes, that means extra work at that point, but for me I'd rather do the work then than endlessly battle with unneeded complexity, in terms of my builds and my development environment, for benefits which are never actually realised.
Unfortunately there seems to be a tendency at the start of a projects to modularise to the nth degree before even a single line of code is written.

there is another advantage to this.. if i have to deploy on 'selected' components i dont waste time and resources deploying dependencies that I dont need.

Can OSGi help reduce complexity?

I saw lots of presentations on OSGi and i think it sounds promising for enforcing better modularization. Apparently "hotdeployment" and "running different versions of x in parallel" are mayor selling points too.
I wonder whether what OSGi promises to solve is even an issue...? It reminded me of the early days of OO when similar claims were maid:
When OO was new, the big argument was reusability. It was widely claimed that when using OO, one would only have to "write once" and could then "use everywhere".
In practice I only saw this working for some pretty low level examples. I think the reason for this is that writing reusable code is hard. Not technically but from a interface design point of view. You have to anticipate how future clients will want to use your classes and take the right choices up front. This is difficult by definition and thus the potential reusability benefit often failed to deliver.
With OSGi, I have the suspicion that here again we could fall for promises, potential solutions for problems that we don't really have. Or if we have them, we don't have them in a big enough quantity and severity that would justify to buy into OSGi for help. "Hotdeployment" for example of a subset of modules is definitely a great idea, but how often does it really work? How often not because it turned out you got the modularization wrong for the particular issue? How about model entities that are shared between multiple modules? Do these modules all have to be changed at the same time? Or do you flatten your objects to primitives and use only those in inter-module communication, in order to be able to keep interface contracts?
The hardest problem when applying OSGi is, I would presume, to get the modularization "right". Similar to getting the interfaces of your classes right in OO, with OSGi, the problem stays the same, on a bigger scale this time, the package or even service level.
As you might have guessed, I'm currently trying to evaluate OSGi for use in a project. The major problem we have, is increasing complexity as the codebase grows and I would like to break the system up in smaller modules that have less and more defined interactions.
Given no framework can ever help deciding what to modularize, has OSGi ever payed off for you?
Has it made your life easier when working in teams?
Has it helped to reduce bug count?
Do you ever successfully "hotdeploy" major components?
Does OSGi help to reduce complexity over time?
Did OSGi keep its promises?
Did it fulfill your expectations?
Thanks!

OSGi pays off because it enforces modularization at runtime, something you previously did not have, often causing the design on paper and implementation to drift apart. This can be a big win during development.
It definitely helps make it easier to work in teams, if you let teams focus on a single module (possibly a set of bundles), and if you get your modularization right. One could argue that one can do the same thing with a build tool like Ant+Ivy or Maven and dependencies, the granularity of dependencies that OSGi uses is in my opinion far superior, not causing the typical "dragging in everything plus the kitchen sink" that JAR level dependencies cause.
Modular code with less dependencies tends to lead to cleaner and less code, in turn leading to less bugs that are easier to test for and solve. It also promotes designing components as simple and straightforward as possible, whilst at the same time having the option to plug in more complicated implementations, or adding aspects such as caching as separate components.
Hot deployment, even if you do not use it at runtime, is a very good test to validate if you modularized your application correctly. If you cannot start your bundles in a random order at all, you should investigate why. Also, it can make your development cycle a lot quicker if you can update an arbitrary bundle.
As long as you can manage your modules and dependencies, big projects stay manageable and can be easily evolved (saving you from the arguably bad "complete rewrite").
The downside of OSGi? It's a very low-level framework, and whilst it solves the problems it is intended for quite well, there are things that you still need to resolve yourself. Especially if you come from a Java EE environment, where you get free thread-safety and some other concepts that can be quite useful if you need them, you need to come up with solutions for these in OSGi yourself.
A common pitfall is to not use abstractions on top of OSGi to make this easier for the average developer. Never ever let them mess with ServiceListeners or ServiceTrackers manually. Carefully consider what bundles are and are not allowed to do: Are you willing to give developers access to the BundleContext or do you hide all of this from them by using some form of declarative model.

I've worked with OSGi for some years now (although in the context of an eclipse project, not in a web project). It is clear that the framework does not free you from thinking how to modularize. But it enables you to define the rules.
If you use packages and defines (In a design document? Verbal?) that certain packages may not access classes in other packages, without an enforcement of this constraint, it will be broken. If you hire new developers they don't know the rules. They WILL break the rules. With OSGi you can define the rules in code. For us this was a big win, as it has helped us to maintain the architecture of our system.
OSGi does not reduce complexity. But it definitely helps to handle it.

I am using OSGI for over 8 years now, and every time I dive in a non-OSGI project I get the feeling over overspeeding without a seatbelt on.
OSGI makes project setup and deployment harder, and forces you to think about modularization upfront, but gives you the easy of mind of enforcing the rules at runtime.
Take maven apache camel as an example. When you create a new maven project and add apache camel as a dependency, the applications seems to have all its dependencies, and you will only notice the ClassNotFoundExceptions at runtime, which is bad. When you run in an OSGI container and load the apache camel modules, the modules with unmet dependencies are not started, and you know upfront what the problem is.
I also use the hot-deployment all the time, and update parts of my application on the fly without the need for a restart.

I used OSGI in one project (I admit - not very much). It provides good promises, but as #Arne said, you still need to think on your own about how you modularize.
OSGI did help our project because it made the architecture more stable. Breaking the modularization is more "difficult", so decisions that we made regarding how to modularize stayed valid for a longer time.
To put it differently - without OSGI, and under time pressure to deliver, sometimes you or your team members make compromises, shortcuts and other hacks, and the the original intent of the architecture is lost.
So OSGI didn't reduce the complexity per se, but it protected it from growing unnecessarily over time. I guess that is a good thing :)
I haven't used the hot deploy feature, so I can't comment about that.
To answer your last point, it did meet my expectations, but it required a learning curve and some adaption, and the payoff is only for long-term.
(as a side note, your question reminds me a bit of the adage that "maven is the awt of build systems")

OSGi does NOT pay off. The fact is OSGi is not easy to use and at the end of the day or year depending on how long it takes you to get things working, it does not add value:
Your application will not be more modular overall, on the contrary, It ends being more exposed and not isolated from other applications since it is a share everything instead of share nothing arch.
Versioning is pushed further down the stack, you wrestle with maven transitive dependencies only to do that again at runtime in OSGI.
Most libraries are designed to work as libraries in the application classloader not as bundles with their own classloader.
Maybe appropriate for plugin architectures where third party developers need to be sandboxed or maybe it is just EJB2.0 all over again.
I added the following slides and I will follow up with example code to demonstrate how to work successfully with OSGi if it is forced on you.
http://www.slideshare.net/ielian/tdd-on-osgi

No, OSGI will make you grey early.

How should I start when developing a system based on modules or plugins?

I intend to develop a system that is entirely based on modules. The system base should have support for finding out about plugins, starting them up and being able to provide ways for those modules to communicate. Ideally, one should be able to put in new modules and yank out unused modules at will, and modules should be able to use each other's funcionality if it is available.
This system should be used as a basis for simulation systems where a lot of stuff happens in different modules, and other modules might want to do something based on that.
The system I intend to develop is going to be in Java. The way I see it, I intend to have a folder with a subfolder for each module that includes a XML that describes the module with information such as name, maybe which events it might raise, stuff like that. I suppose I might need to write a custom ClassLoader to work this stuff out.
The thing is, I don't know if my idea actually holds any water and, of course, I intend on building a working prototype. However, I never worked on a truly modular system before, and I'm not really sure what is the best way to take on this problem.
Where should I start? Are there common problems and pitfalls that are found while developing this kind of system? How do I make the modules talk with each other while maintaining isolation (i.e, you remove a module and another module that was using it stays sane)? Are there any guides, specifications or articles I can read that could give me some ideas on where to start? It would be better if they were based on Java, but this is not a requirement, as what I'm looking for right now are ideas, not code.
Any feedback is appreciated.

You should definitely look at OSGi. It aims at being the component/plugin mechanism for Java. It allows you to modularize your code (in so-called bundles) and update bundles at runtime. You can also completely hide implementation packages from unwanted access by other bundles, eg. only provide the API.
Eclipse was the first major open-source project to implement and use OSGi, but they didn't fully leverage it (no plugin installations/updates without restarts). If you start from scratch though, it will give you a very good framework for a plugin system.
Apache Felix is a complete open-source implementation (and there are others, such as Eclipse Equinox).

Without getting into great detail, you should be looking at Spring and a familiarization with OSGI or the Eclipse RCP frameworks will also give you some fundamental concepts you will need to keep in mind.

Another option is the ServiceLoader added in Java 1.6.

They are many way to do it but something simple can be by using Reflection. You write in your XML file name of file (that would be a class in reallity). You can than check what type is it and create it back with reflection. The class could have a common Interface that will let you find if the external file/class is really one of your module. Here is some information about Reflexion.
You can also use a precoded framework like this SourceForge onelink text that will give you a first good step to create module/plugin.