Develop Reference

Is Maven similar to npm?

As I have worked with npm which looks for dependencies in package.json file and download it for you. Similarly, I see a pom.xml file in Java project. Does maven looks in this file and download dependencies for me. Can I pass around this pom.xml file like package.json, rather than giving the dependency jars ? Are these tools similar and just build for different platforms ?

Same tool, different language?
Maven is the most popular build and dependency resolution tool for Java, just like NPM is for JS. But it's not just the same tool for a different language. There are obviously huge differences between Java and JS builds, and these differences are directly visible in the way Maven operates. For example, while many JS tools rely on Git to do some heavy-lifting, Maven works with custom filesystem-based Maven repositories, as Maven predates Git and needs to handle binary artifacts, which Git historically didn't handle well. In Maven there's a clear separation between sources and binaries, while they are often the same thing in JS world.
Maven basics
Maven in its purest form follows a declarative model, where pom.xml (similar to package.json) defines different properties of the build, but contains no scripts. The disadvantage is it can be a challenge to fine-tune some aspects of the build without using scripts as you have to rely on plugins. The advantage is it can be easier to understand other builds just by looking at pom.xml, as they usually follow the same approach without too much customization. Gradle is a popular Groovy-based tool built on top of Maven standards and conventions, and is specifically designed to simplify pom.xml and break this "no script" barrier.
Referencing your dependencies
Similarly to package.json, you don't work with pom.xml of your dependency directly, but rather define dependency coordinates and let your build tool handle the rest. In Maven the basic form of these coordinates is GAV (groupId, artifactId, version).
Flat dependency tree?
Based on comments in the other answer, Maven provides "flat dependency tree", not "nested dependency tree" that NPM provides by default. Maven does not allow multiple versions of the same dependency. If it happens that different versions are requested, Maven uses dependency resolution to pick a single version. This means that sometimes your transitive dependencies will get a different version than they require, but there are ways to manage this. However, this limitation comes from Java, not Maven, as (normally) in Java a class loader will only provide access to a single class definition even if multiple definitions are found on the classpath. Since Java is not particularly good at handling this, Maven tries to avoid this scenario in the first place.
Note: since npm v3 the dependencies are flatten. The alternative package manager yarn also does the same.
Maturity
Furthermore, Maven is considerably older than NPM, has a larger user base, huge number of custom plugins, and so far could probably be considered more mature overall. Sometimes Maven is used for non-Java or even polyglot projects, as there are plugins for handling other languages or specific environments, such as Android. There are plugins that bridge Maven and other build tools, such as frontend-maven-plugin that actually handles multiple JS build tools.

Yes they are similar in the context that their main purpose is to provide a way describing the project dependencies, instead of keeping them within the project code, and their secondary purpose is to provide developers with an easy way to perform, define and share dev-time/build-time tasks. Both of the above are expressed inside a descriptor file.
Now deciding which one to use is, most of the times, straightforward because it depends on the primary language you are working on. A rough grouping is:
java: maven
javascript/typescript: npm
Below I provide a detailed explanation of the common features and differences. I use | to separate between maven | npm terms respectively:
Common features:
Both tools support dynamic fetch of dependencies ( artifacts | packages ) based on a descriptor file pom.xml|package.json, and also allow you to deploy | publish your own artifacts | packages.
They both have a default public repository | registry ( http://repo.maven.apache.org/maven2/ | https://registry.npmjs.org), but 3rd-party can also be used (via settings.xml|.npmrc ).
They both support the concept of build-level dependencies (plugins | devDependencies used in scripts). *Maven supports provided dependencies also but this does not seem to apply to npm, since javascript is rarely deployed into containers.
They both support dependency namespacing: groupId|scope
Differrences:
maven has an additional local repository(cache):
No need to fetch again the same dependency for differrent projects.
Artifacts that are installed locally, are automatically accessible by other local projects.
dependencies from a project build in maven are downloaded in <homedir>/.m2. With npm they are downloaded in <projectdir>/node_modules.
Building in maven is commonly a one-step process: mvn package (fetch deps , build). In npm it is a 2-step process: npm install (fetch deps) , npm build (build)
maven defines build lifecycles (for building,testing,deploying) consisted of phases, to which default operations(plugin goals) attach, based on differrent packaging options(.jar,.war,.ear e.t.c). You can then overwrite these operations, or inject new ones (via the plugin system). This provides kind of an out-of-the box solution for build,docgen,test,deploy e.t.c.
npm approach is more simplistic ( see: scripts)
Due to the above, npm is labeled as a package-management tool for javascript while maven is labeled as a build-automation and dependency-management tool for java.
In maven setting-up the build process more commonly involves editing the pom.xml.
In npm it involves writing code or configuring complementary build tools like gulp,webpack e.t.c
For some reason version ranges defined by users in npm modules, are much more loose than in maven. This can cause issues with transitive dependencies, that is why an additional file was recently added: package-lock.json
With npm it is much more straightforward to start a new project: npm init. With maven, you need to know how to write a minimal pom.xml, or read about archetypes.
In general it is much more common to edit pom.xml than package.json. E.g. adding dependencies in maven is done manually (or via IDE) while in npm via command line.
As with all build tools, you can call one tool from inside the other, but I think its much more common to call npm from inside maven, than the opposite.
npm supports dev,production builds. In maven this needs to be defined through profiles.

yes. it's a similar packaging tool for java. look for gradle also which gives you more liberty with groovy language, but for start you can use maven to organize your dependencies. you include them as tags there and maven does the job for you.
it traverses the dependency tree and downloads all the appropriate jars.

Yes, same with gradle, but they are not user friendly as npm.

Build multiple java projects with dynamic dependencies

I have multiple java projects in a folder. Also there is a second folder with libraries, that might be used as build dependencies from the projects. The projects may also have dependencies to other Projects. What's the best approach to build all projects ?
In other words I want to build the projects without explicit telling their dependencies.I think the biggest problem is the dependecy between the projects.

There are multiple build systems that are available that you may use. Maven has a complete dependency system built into it. Almost all third party open source jars are directly accessible via the World Wide Maven repository system. Basically, you describe the jar you need (groupId, artifactId, and version) and Maven will automatically fetch it for you. Not only that, but Maven also will build your project without having to create a build file. Instead, you have to describe your project in a project object model (a pom.xml file) and Maven will download everything you need, including all compilers, etc.
Almost all new projects use Maven, but Maven has a few downsides:
Since you don't control a build process, it can sometimes feel like poking a prodding a black box to get the build to work the way you want.
Documentation can be scant -- especially if you're moving beyond basic Java compiles.
You usually have to arrange your project in a specific layout. For example, source files should go under src/main/java while JUnit tests are under src/test/java. You don't have to follow the recommended layout, but then you'd have to modify the pom.xml file this way and that to get your build to work. That defeats the whole purpose of the pom.xml in the first place.
If you already have another build system setup (like Ant), you lose everything. There's no easy way to move from Ant to Maven.
The other is called Ant with Ivy. Ivy uses Ant for building, but can access Maven's world wide repository system for third party dependencies. It's a great compromise if you already are heavily invested in Ant. I also find Ant with Ivy to be better documented than Maven (although that's not too difficult). There's an excellent chapter going over the basics of Ivy in Manning Publication's Ant in Action.
With either process, I would recommend that you build a company wide Maven repository using either Nexus or Artifactory. This way, any proprietary third party jars (like Oracle jars) can also be stored in your company wide Maven repository since they won't be in the standard World Wide Maven repository.
By the way, if this is a company wide effort, and you are moving multiple Ant projects into Ivy, I have an Ivy project I use in Github that makes things easier.
Oh, there's a third possibility called Gradle which I know nothing about. I also believe it can use the World Wide Maven repository. It's based on Groovy which is based on Java syntax, and that's about all I can say. Maybe others can fill you in on the details. The Gradle group contends it solves a lot of problems of both Ant/Ivy and Maven.

Whatever tool you use, if you have various projects interdependent, you need to be clear on the independent ones which will be built first before building the dependent projects. You need to have a clear dependency structure for your projects.

You can do this with Apache Ivy. You can lay out the locations for you common libraries, define published artifacts and inter-dependencies in an ivy.xml document in each project, and let a top-level Ant build with the Ivy tasks figure out what the build order should be based on those dependencies.

What is the most elegant solution to managing various Java external libraries?

Perhaps the reason I stalled learning Java until now is because I HATE how Java handles external libraries. I'm stuck keeping them in one place, adding them individually, fixing problems with versioning and every time I move/rename them, and copying and writing the classpath over and over each time I release a Java application.
There has to be an elegant solution to all of this. I keep all of my libraries (regardless of task, platform, or other) in their own little folder inside a "lib" folder in my development folder, kind of like this:
Dev
-lib
+JS-jQuery
+Flex-Degrafa
-Java-Xerces
+Xerces-1.2.3
+More libraries
I can use either Netbeans or Eclipse for Java dev, but none of them provide a very streamlined (and not to mention idiot-proof) way of managing all of these.
A nudge in the right direction or an online article/tutorial on this would be greatly appreciated.

You can either use Ant + Ivy or Maven to manage your library dependencies.

If it is only dependency management you're after and you're happy with the rest of your build process, I would use Ivy, as it can unobtrusively manage your dependencies, leaving your existing build process intact. There is a plugin for Eclipse called IvyIDE that contributes your dependencies via a classpath container.
Maven 2 has a steeper learning curve but provides a much richer set of functionality for building your projects and Eclipse integration through m2eclipse or IAM.
Personally I use Maven as I have a large number of projects to work with and Maven is particularly suited to efficient development across lots of projects.
Have a look at the introductory documentation to see what works for you.
Ivy Tutorial
Maven Getting Started Guide

Netbeans 6.7.1's Maven support is quite good and comes out of the box with the IDE.
The Eclipse addon was frustrating enough that I gave Netbeans another try.
A third choice besides ChssPly76's options is to use Ant with the Maven Ant Tasks. I don't know if I'd call any of these solutions particularly "elegant," but they do spare you the need to manage your own lib/ directory and classpath variables.

If you're working on Linux you can install Java libraries with APT or RPM.
Otherwise, I normally check precompiled JARs into a lib directory in my project's version control repository and make sure the names of the JAR files include full version information. E.g. lib/foo-1.5.6.jar, not lib/foo.jar.
To avoid having to manually set the classpath before running your app, you can set the classpath in the Manifests of the JARs themselves to define the dependencies of each JAR file. The JVM will follow all the dependencies when loading classes.

Maven is often more trouble than it's worth, but the ability to open a maven project directly into IDEs such as IntelliJ is excellent. For example, IntelliJ will download all dependencies and have them available without having to run a build first, or an mvn command and then a project refresh. It also isn't necessary to re-generate the project every time a dependency is added. I work with a number of Eclipse developers who switched to IntelliJ for this alone.
However, one shortfall of Maven is that many libraries (or versions of libraries) are not available on public repositories. Therefore it is often necessary to set up a local repository such as archiva. In ant, it would just be a matter of adding it to the lib directory in the repository.
Maven can also attack when you need to do something that maven doesn't directly support via a plugin. What would normally be a few lines of ant can often turn into a morning's worth of work.
Finally, buildr is an excellent way of using Maven's dependency management and plugins, while also supporting ad-hoc tasks.

Any best practices or tools for Ivy repositories?

we use Gradle for building Java projects and at the moment we have Ivy repositories to store third-party artifacts and also to publish our own artifacts into (repo is build using Gant scripts and the Ivy ANT tasks). but repo management is basic.
Gradle is able to work with a maven repo as well, so switching to a Maven artifact manager like Archiva or Nexus is an option, but perhaps unnecessary. do you know any tools or best practices than can help us in building and maintaining Ivy repos?
just to be clear: we have already read the tutorials and more and understand how to do it, but it's still basic to maintain.

In the past, I've only used an ivy repository for small private repositories publishing artifacts using simple low level protocols like an FTP site. (All the site needs is a versioned directory layout and an ivy.xml file describing the arifacts)
The maven based repository infrastructure is now so pervasive, with some many projects using it, it's almost pointless to promote an alternative repository management standard.
Sonatype (company behind Maven) make their repository product, Nexus, available to all, because it's in everyone's interest to keep the band-width requirements to Maven central under control.
Thankfully, ivy plays nice with Maven meaning you can take advantage of the best of both worlds.

I've created a blog entry about my Ivy repository layout and choices. You might have different requirements, but I think it is always good to check other's solutions to get some ideas...

There's a community project called Ivy Roundup that aims to build a consistent, up-to-date ivy repo of common third party libraries. It may be a good idea to match the naming conventions used there, or even better, just get modules from there using the <ivy:install> task.

In my opinion there isn't much in Ivy's repository to work with because it just works. What you can't do with Ivy's Ant tasks you can do directly from the file system, simple as that.
Admittedly something like changing the artefact name can be difficult but then again that's something you shouldn't do anyway.

What I am typically doing in practice when I need a third-party library is to search for it in the MVN repository and then click on the "Ivy" tab to get the Ivy dependency for my ivy.xml.

Maven replacement?

What would you suggest as a replacement to the Maven Java build toolset? Just plain Ant scripts? SCons?

It depends on what you use anyway. Maven 1? Maven 2? I find Maven 2 decently documented, and quite powerful, when used in combination with Ant tasks and some home baked Java plugins.
There are Ant tasks to use Maven features from Ant: http://maven.apache.org/ant-tasks/index.html . So you don't really need to write your own "library".
And you can call Ant from Maven: http://maven.apache.org/plugins/maven-antrun-plugin/ .
You can also script things using Groovy for example: http://groovy.codehaus.org/GMaven+-+Executing+Groovy+Code .
Or just write Java code for the custom bits you need: http://maven.apache.org/guides/plugin/guide-java-plugin-development.html .
There's no reason not to mix and match, when you can do it ;)

While I like the idea of Maven, I always find myself fighting it for anything but the simplest configurations. Its architecture is plugin-based, and the quality of the plugins varies widely. Documentation is sketchy at best; half the time I can't figure out if the doc I'm reading is for Maven 1 or 2. And how often have you had to resort to using the maven-ant-plugin to do something basic?
I've been using Apache Ivy for about a year now, and I'm pretty happy with it. There is a bit of a learning curve though. But there's an active mailing list, and the author is very responsive.
What I ended up doing was writing a "library" of a few simple Ant targets such as "resolve", "compile", "install", etc. that expect things to be in standard Maven places (such as src/main/java) and then <import> the targets into my various projects. This way you get the best of both worlds- Maven-style configuration, and all the power of Ant.

I'm fond of using Ant with Ivy. Ivy uses Maven repositories to fetch library dependencies without forcing you to change your entire build system to suit Maven.

I've been using buildr for some of my projects. Very terse and readable build scripts (no xml) and it uses maven repos (with trivial effort, maven 1 and maven 2).
I was looking for something post-maven1 to work on for some projects and m2 wasn't all that appealing. Maven had already left all m1 users high and dry with the need to redo all of their build systems (and some of mine cannot be expressed in m2 as far as I can tell).

I really like using ant4eclipse - you set up your project dependencies in eclipse as normal and you can write a single ant script that'll build all the projects in the right order.
For third-party libs, you can either set up a single eclipse project containing (and exporting) all of your jars [I DON'T RECOMMEND THIS], or a separate project for each third-party lib, exporting the jar. [I recommend the latter b/c you can track project->third-party deps and upgrade parts when needed.]

I would go with buildr, if i am expecting complex build targets in my build. The benefit is, you are not dependent on any particular library at all. You have all the APIs from Ruby to do whatever you want; unlike Ivy in which case you are still tied up to ANT. Effort of migration to Ivy is worth if you want to introduce library dependency to an already established ANT based build mechanism which does all the operations you want to do.

There are at least three different parts of Maven: the repository, the implementation independent lifecyle and the conventions for a default project. Using Ivy provides only the repository.
I've not used it but I think that easyant is designed to be a more complete replacement to Maven. It uses Ivy as the repository but then adds conventions and standard modules.