How to make an executable jar? - java

I'm creating a new web application using Maven. I have got some code from 'spring's guides online which creates a database. However for some reason, the code is never being ran.
In my pom.xml, I have included the following code:
<properties>
<start-class>hello.Application</start-class>
</properties>
And this is the 'Application' class which I got from the spring guides.
package hello;
import java.sql.ResultSet;
public class Application {
public static void main(String args[]) {
// simple DS for test (not for production!)
SimpleDriverDataSource dataSource = new SimpleDriverDataSource();
dataSource.setDriverClass(org.h2.Driver.class);
dataSource.setUsername("sa");
dataSource.setUrl("jdbc:h2:mem");
dataSource.setPassword("");
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
System.out.println("Creating tables");
jdbcTemplate.execute("drop table customers if exists");
jdbcTemplate.execute("create table customers(" +
"id serial, first_name varchar(255), last_name varchar(255))");
String[] names = "John Woo;Jeff Dean;Josh Bloch;Josh Long".split(";");
for (String fullname : names) {
String[] name = fullname.split(" ");
System.out.printf("Inserting customer record for %s %s\n", name[0], name[1]);
jdbcTemplate.update(
"INSERT INTO customers(first_name,last_name) values(?,?)",
name[0], name[1]);
}
System.out.println("Querying for customer records where first_name = 'Josh':");
List<Customer> results = jdbcTemplate.query(
"select * from customers where first_name = ?", new Object[] { "Josh" },
new RowMapper<Customer>() {
#Override
public Customer mapRow(ResultSet rs, int rowNum) throws SQLException {
return new Customer(rs.getLong("id"), rs.getString("first_name"),
rs.getString("last_name"));
}
});
for (Customer customer : results) {
System.out.println(customer);
}
}
}
My Project structure is as follows:
Project name
src
webapp
hello
application.java
I am pretty new to this but I just can't see why its not finding the application.java file.
Any ideas would be appreciated.
EDIT: This is the whole pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org
/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>embed.tomcat.here</groupId>
<artifactId>EmbedTomcatNew</artifactId>
<packaging>war</packaging>
<version>0.0.1-SNAPSHOT</version>
<name>EmbedTomcatNew Maven Webapp</name>
<url>http://maven.apache.org</url>
<properties>
<start-class>hello.Application</start-class>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<finalName>EmbedTomcatNew</finalName>
<plugins>
<plugin>
<groupId>org.apache.tomcat.maven</groupId>
<artifactId>tomcat7-maven-plugin</artifactId>
<version>2.2</version>
<configuration>
<port>9966</port>
</configuration>
</plugin>
</plugins>
</build>
</project>
EDIT:
I should also mention that it is running a jsp file - which only has 'hello world' in it that prints to the browser - so it is running but I want it to run the java class first.


WAR
You are building a .war file, as evidenced in the <packaging>war</packaging> definition, which is only deployable to a Web Application container. There is no startup class, and as well documented on stackoverflow there is do way to control the order of startup in most web app containers.
JAR
You have to change your project to be an executable .jar and specify the main class in the Manifest in the jar plugin configuration options. Just setting some random property isn't going to do anything.
You probably want to use the shade plugin to bundle all the transient dependencies into a monolithic .jar as well otherwise you have an classpath installation nightmare on your hands.
Here is an example, running this from the src/main/webapp dir is a bad non-portable idea, that should be passed in as an argument.
import java.io.File;
import org.apache.catalina.startup.Tomcat;
public class Main {
public static void main(String[] args) throws Exception {
String webappDirLocation = "src/main/webapp/";
Tomcat tomcat = new Tomcat();
//The port that we should run on can be set into an environment variable
//Look for that variable and default to 8080 if it isn't there.
String webPort = System.getenv("PORT");
if(webPort == null || webPort.isEmpty()) {
webPort = "8080";
}
tomcat.setPort(Integer.valueOf(webPort));
tomcat.addWebapp("/", new File(webappDirLocation).getAbsolutePath());
System.out.println("configuring app with basedir: " + new File("./" + webappDirLocation).getAbsolutePath());
tomcat.start();
tomcat.getServer().await();
}
}

Related

How to upload Folder in google cloud using java?

Previously I was working in AWS and I am new in Google Cloud, in AWS there was a way to upload directories/folder to bucket. I have done bit of research for uploading directory/folder in Google Cloud bucket but couldn't find. Can someone help me, how to achieve this in Google Cloud using Java.

There is no embedded functions in the Google Cloud Storage client library (or even in the API) to perform this automatically. You have to recursively upload all your files, managing yourselves the folders tree exploration.
With the gcloud CLI, you can use the command gsutil cp -r .... The -r stands for "recursive" and it performs exactly the same operation.

In Google Cloud Storage client library for java is not built-in the functionality to upload folders, but I crafted this java code to upload folders to GCS, I used Open JDK 8 and Debian
App.java
package com.example.app;
import com.google.cloud.storage.BlobId;
import com.google.cloud.storage.BlobInfo;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.io.File;
public class App {
public static void main(String[] args) throws IOException {
//directory that you want to upload
String dir = "/home/user/repo";
// get the name of the parent directory
String[] path = dir.split("/");
String folder = path[path.length - 1];
//get files in main directory
File[] files = new File(dir).listFiles();
// define your projectID & bucket name
String bucket = "myawesomefolder";
String projectId = "myawesomeprojectID";
Storage storage = StorageOptions.newBuilder().setProjectId(projectId).build().getService();
System.out.println("Uploading folder: " + folder);
uploadFolder(files, folder, bucket, storage);
}
static void uploadFolder(File[] files, String folder, String bucket, Storage storage) throws IOException {
for (File file : files) {
if (!file.isHidden()) {
// if it is a directory read the files within the subdirectory
if (file.isDirectory()) {
String[] lpath = file.getAbsolutePath().split("/");
String lfolder = lpath[lpath.length - 1];
String xfolder = folder + "/" + lfolder;
uploadFolder(file.listFiles(), xfolder, bucket, storage); // Calls same method again.
} else {
// add directory/subdirectory to the file name to create the file structure
BlobId blobId = BlobId.of(bucket, folder + "/" + file.getName());
//prepare object
BlobInfo blobInfo = BlobInfo.newBuilder(blobId).build();
// upload object
storage.create(blobInfo, Files.readAllBytes(Paths.get(file.getAbsolutePath())));
System.out.println("Uploaded: gs://" + bucket + "/" + folder + "/" + file.getName());
}
}
}
}
}
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>testGcs</artifactId>
<version>1.0-SNAPSHOT</version>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>7</source>
<target>7</target>
</configuration>
</plugin>
</plugins>
</build>
<properties>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-storage</artifactId>
<version>1.111.2</version>
</dependency>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-nio</artifactId>
<version>0.121.2</version>
</dependency>
</dependencies>
</project>

Not able to parse Protobuf in java

I have two protobuf files. I have to compare the contents of both of them in order to proceed further with the code. For this, i am trying to parse a protobuf file but some how i am not able to get the various message types and other information within the .proto file. I have to do all this in java.
Code snippets:
package com.example.demo;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import com.google.protobuf.DescriptorProtos;
import com.google.protobuf.DescriptorProtos.FileDescriptorProto;
import com.google.protobuf.Descriptors;
import com.google.protobuf.Descriptors.FileDescriptor;
import com.google.protobuf.InvalidProtocolBufferException;
public class TestProto {
public static FileDescriptorProto parseProto(InputStream protoStream)
throws InvalidProtocolBufferException, Descriptors.DescriptorValidationException {
DescriptorProtos.FileDescriptorProto descriptorProto = null;
try {
descriptorProto = FileDescriptorProto.parseFrom(protoStream);
} catch (IOException e) {
e.printStackTrace();
}
return descriptorProto;
}
public static InputStream readProto(File filePath) {
InputStream is = null;
Reader reader = null;
try {
is = new FileInputStream(filePath);
reader = new InputStreamReader(is);
int data = reader.read();
while (data != -1) {
System.out.print((char) data);
data = reader.read();
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return is;
}
public static void main(String args[]) {
InputStream protoStream = readProto(new File("D:/PROTOBUF CONVERTER/default.proto"));
Descriptors.FileDescriptor fileDescriptor = null;
DescriptorProtos.FileDescriptorProto fileDescriptorProto = null;
try {
fileDescriptorProto = parseProto(protoStream);
fileDescriptor = FileDescriptor.buildFrom(fileDescriptorProto, new FileDescriptor[] {}, true);
System.out.println("\n*******************");
System.out.println(fileDescriptor.getFullName());
System.out.println(fileDescriptor.getName());
System.out.println(fileDescriptor.getPackage());
System.out.println(fileDescriptor.getClass());
System.out.println(fileDescriptor.getDependencies());
System.out.println(fileDescriptor.toProto());
System.out.println(fileDescriptor.getServices());
System.out.println(fileDescriptor.getMessageTypes());
System.out.println(fileDescriptor.getOptions());
} catch(Exception e) {
e.printStackTrace();
}
}
}
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.3.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.springboot</groupId>
<artifactId>demo</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>demo</name>
<description>Demo project for Spring Boot</description>
<properties>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.xolstice.maven.plugins</groupId>
<artifactId>protobuf-maven-plugin</artifactId>
<version>0.6.1</version>
</dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>3.5.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/commons-io/commons-io -->
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.6</version>
</dependency>
</dependencies>
<build>
<finalName>ProtobufParseDemo</finalName>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<inherited>true</inherited>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
default.proto
syntax = "proto3";
package tutorial;
option java_package = "com.example.tutorial";
option java_outer_classname = "AddressBookProtos";
message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
required string number = 1;
optional PhoneType type = 2 [default = HOME];
}
repeated PhoneNumber phones = 4;
}
message AddressBook {
repeated Person people = 1;
}
I can see the protofile data on the console due to code line "System.out.print((char) data);". However, i am not able to see any output in the sysout of the FileDescriptors.
I am new to Protocol buffers.
Questions:
what I am trying to do, is it relevant OR I am making some mistake?
Is there any other method to do this in Java?
I have seen some answers, like the one here Protocol Buffers: How to parse a .proto file in Java.
It says that the input to the parseFrom method should be of binary type i.e. a compiled schema. Is there a way in which we can obtain the compiled version of the .proto file in java code (not in command line)?
Ok, to be more clear on this, I have to compare two .proto files.
First would be the one which is already uploaded with the ML model
and
Second would be the one which is to be uploaded for the same ML model.
If there are differences in the input or output message types of the two .proto files, then accordingly i have to increment the version number of the model.
I have found solutions where the proto is converted to proto descriptors and then converted to byte array and further passed tp parsrFrom method. Can't this process of converting .proto to proto.desc, be done via java code ?
Point to keep in mind here is that, i do not have the proto files in my classpath and giving the address in pom.xml (that of input and output directories) is not possible here as i have to download the old proto and compare it with the new proto to be uploaded as mentioned above.

Unable to connect to mongo database using Java, OSGI, Karaf

I've installed the mongo driver in my running Karaf server:
bundle:install -s wrap:mvn:org.mongodb/mongo-java-driver/3.6.3
I'm simply trying to connect to the DB and log the databases I have. Currently running out of the box local instance. Below is the code I wrote to demo this in OSGI/Karaf. I'm using the mvn bundle plugin.
I created a database under the alias osgiDatabase
I'm running my debugger and the failure happens during the instantiation of the MongoClient() but not understanding what I could be doing wrong.
This works when I don't use Karaf. The only error I get is Activator start error in bundle
POM
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.qa</groupId>
<artifactId>board</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>bundle</packaging>
<dependencies>
<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongo-java-driver</artifactId>
<version>3.6.3</version>
</dependency>
<dependency>
<groupId>org.osgi</groupId>
<artifactId>org.osgi.core</artifactId>
<version>6.0.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
<extensions>true</extensions>
<configuration>
<instructions>
<Import-Package>com.mongodb, org.osgi.framework</Import-Package>
<Bundle-Activator>Connection.Activator</Bundle-Activator>
<Export-Package>*</Export-Package>
</instructions>
</configuration>
</plugin>
</plugins>
</build>
</project>
DBUtil
package Connection;
import com.mongodb.MongoClient;
import com.mongodb.client.MongoDatabase;
import java.util.List;
public class DBUtil {
MongoClient client;
MongoDatabase database;
public DBUtil() {
}
public DBUtil(String databaseName) {
if (client == null) {
client = new MongoClient();
database = client.getDatabase(databaseName);
}
}
/**
* Allows you to reveal all databases under the current connection
*/
public void showDatabases() {
if (client == null) {
throw new NullPointerException();
}
List<String> databases = client.getDatabaseNames();
for (String db : databases) {
System.out.println("The name of the database is: " + db);
}
}
}
Activator
package Connection;
import org.osgi.framework.BundleActivator;
import org.osgi.framework.BundleContext;
public class Activator implements BundleActivator {
public void start(BundleContext bundleContext) throws Exception {
DBUtil util = new DBUtil("osgiDatabase");
// util.showDatabases();
System.out.println("Working");
}
public void stop(BundleContext bundleContext) throws Exception {
System.out.println("Bundle disabled");
}
}

Your Import-Package configuration looks wrong. If you configure it explicitly like this you switch off the auto detection of needed packages. So it is very likely you are missing some packages your code needs.
Instead try to only configure the activator and leave the rest on defaults.
To get better logs you should use a try catch in your Activator an log the exception using slf4j. So you get some more information what is wrong.

Hadoop: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected

My MapReduce jobs runs ok when assembled in Eclipse with all possible Hadoop and Hive jars included in Eclipse project as dependencies. (These are the jars that come with single node, local Hadoop installation).
Yet when trying to run the same program assembled using Maven project (see below) I get:
Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
This exception happens when program is assembled using the following Maven project:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.bigdata.hadoop</groupId>
<artifactId>FieldCounts</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>FieldCounts</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-jobclient</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.hive.hcatalog</groupId>
<artifactId>hcatalog-core</artifactId>
<version>0.12.0</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>16.0.1</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<source>${jdk.version}</source>
<target>${jdk.version}</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>attached</goal>
</goals>
<phase>package</phase>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>com.bigdata.hadoop.FieldCounts</mainClass>
</manifest>
</archive>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
* Please advise where and how to find compatible Hadoop jars? *
[update_1]
I am running Hadoop 2.2.0.2.0.6.0-101
As I have found here: https://github.com/kevinweil/elephant-bird/issues/247
Hadoop 1.0.3: JobContext is a class
Hadoop 2.0.0: JobContext is an interface
In my pom.xml I have three jars with version 2.2.0
hadoop-hdfs 2.2.0
hadoop-common 2.2.0
hadoop-mapreduce-client-jobclient 2.2.0
hcatalog-core 0.12.0
The only exception is hcatalog-core which version is 0.12.0, I could not find any more recent version of this jar and I need it!
How can I find which of these 4 jars produces java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected ?
Please, give me an idea how to solve this. (The only solution I see is to compile everything from source!)
[/update_1]
Full text of my MarReduce Job:
package com.bigdata.hadoop;
import java.io.IOException;
import java.util.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.*;
import org.apache.hadoop.util.*;
import org.apache.hcatalog.mapreduce.*;
import org.apache.hcatalog.data.*;
import org.apache.hcatalog.data.schema.*;
import org.apache.log4j.Logger;
public class FieldCounts extends Configured implements Tool {
public static class Map extends Mapper<WritableComparable, HCatRecord, TableFieldValueKey, IntWritable> {
static Logger logger = Logger.getLogger("com.foo.Bar");
static boolean firstMapRun = true;
static List<String> fieldNameList = new LinkedList<String>();
/**
* Return a list of field names not containing `id` field name
* #param schema
* #return
*/
static List<String> getFieldNames(HCatSchema schema) {
// Filter out `id` name just once
if (firstMapRun) {
firstMapRun = false;
List<String> fieldNames = schema.getFieldNames();
for (String fieldName : fieldNames) {
if (!fieldName.equals("id")) {
fieldNameList.add(fieldName);
}
}
} // if (firstMapRun)
return fieldNameList;
}
#Override
protected void map( WritableComparable key,
HCatRecord hcatRecord,
//org.apache.hadoop.mapreduce.Mapper
//<WritableComparable, HCatRecord, Text, IntWritable>.Context context)
Context context)
throws IOException, InterruptedException {
HCatSchema schema = HCatBaseInputFormat.getTableSchema(context.getConfiguration());
//String schemaTypeStr = schema.getSchemaAsTypeString();
//logger.info("******** schemaTypeStr ********** : "+schemaTypeStr);
//List<String> fieldNames = schema.getFieldNames();
List<String> fieldNames = getFieldNames(schema);
for (String fieldName : fieldNames) {
Object value = hcatRecord.get(fieldName, schema);
String fieldValue = null;
if (null == value) {
fieldValue = "<NULL>";
} else {
fieldValue = value.toString();
}
//String fieldNameValue = fieldName+"."+fieldValue;
//context.write(new Text(fieldNameValue), new IntWritable(1));
TableFieldValueKey fieldKey = new TableFieldValueKey();
fieldKey.fieldName = fieldName;
fieldKey.fieldValue = fieldValue;
context.write(fieldKey, new IntWritable(1));
}
}
}
public static class Reduce extends Reducer<TableFieldValueKey, IntWritable,
WritableComparable, HCatRecord> {
protected void reduce( TableFieldValueKey key,
java.lang.Iterable<IntWritable> values,
Context context)
//org.apache.hadoop.mapreduce.Reducer<Text, IntWritable,
//WritableComparable, HCatRecord>.Context context)
throws IOException, InterruptedException {
Iterator<IntWritable> iter = values.iterator();
int sum = 0;
// Sum up occurrences of the given key
while (iter.hasNext()) {
IntWritable iw = iter.next();
sum = sum + iw.get();
}
HCatRecord record = new DefaultHCatRecord(3);
record.set(0, key.fieldName);
record.set(1, key.fieldValue);
record.set(2, sum);
context.write(null, record);
}
}
public int run(String[] args) throws Exception {
Configuration conf = getConf();
args = new GenericOptionsParser(conf, args).getRemainingArgs();
// To fix Hadoop "META-INFO" (http://stackoverflow.com/questions/17265002/hadoop-no-filesystem-for-scheme-file)
conf.set("fs.hdfs.impl",
org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
conf.set("fs.file.impl",
org.apache.hadoop.fs.LocalFileSystem.class.getName());
// Get the input and output table names as arguments
String inputTableName = args[0];
String outputTableName = args[1];
// Assume the default database
String dbName = null;
Job job = new Job(conf, "FieldCounts");
HCatInputFormat.setInput(job,
InputJobInfo.create(dbName, inputTableName, null));
job.setJarByClass(FieldCounts.class);
job.setMapperClass(Map.class);
job.setReducerClass(Reduce.class);
// An HCatalog record as input
job.setInputFormatClass(HCatInputFormat.class);
// Mapper emits TableFieldValueKey as key and an integer as value
job.setMapOutputKeyClass(TableFieldValueKey.class);
job.setMapOutputValueClass(IntWritable.class);
// Ignore the key for the reducer output; emitting an HCatalog record as
// value
job.setOutputKeyClass(WritableComparable.class);
job.setOutputValueClass(DefaultHCatRecord.class);
job.setOutputFormatClass(HCatOutputFormat.class);
HCatOutputFormat.setOutput(job,
OutputJobInfo.create(dbName, outputTableName, null));
HCatSchema s = HCatOutputFormat.getTableSchema(job);
System.err.println("INFO: output schema explicitly set for writing:"
+ s);
HCatOutputFormat.setSchema(job, s);
return (job.waitForCompletion(true) ? 0 : 1);
}
public static void main(String[] args) throws Exception {
String classpath = System.getProperty("java.class.path");
//System.out.println("*** CLASSPATH: "+classpath);
int exitCode = ToolRunner.run(new FieldCounts(), args);
System.exit(exitCode);
}
}
And class for complex key:
package com.bigdata.hadoop;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.io.WritableComparable;
import com.google.common.collect.ComparisonChain;
public class TableFieldValueKey implements WritableComparable<TableFieldValueKey> {
public String fieldName;
public String fieldValue;
public TableFieldValueKey() {} //must have a default constructor
//
public void readFields(DataInput in) throws IOException {
fieldName = in.readUTF();
fieldValue = in.readUTF();
}
public void write(DataOutput out) throws IOException {
out.writeUTF(fieldName);
out.writeUTF(fieldValue);
}
public int compareTo(TableFieldValueKey o) {
return ComparisonChain.start().compare(fieldName, o.fieldName)
.compare(fieldValue, o.fieldValue).result();
}
}

Hadoop has gone through a huge code refactoring from Hadoop 1.0 to Hadoop 2.0. One side effect
is that code compiled against Hadoop 1.0 is not compatible with Hadoop 2.0 and vice-versa.
However source code is mostly compatible and thus one just need to recompile code with target
Hadoop distribution.
The exception "Found interface X, but class was expected" is very common when you're running
code that is compiled for Hadoop 1.0 on Hadoop 2.0 or vice-versa.
You can find the correct hadoop version used in the cluster, then specify that hadoop version in the pom.xml file Build your project with the same version of hadoop used in the cluster and deploy it.

You need to recompile "hcatalog-core" to support Hadoop 2.0.0.
Currently "hcatalog-core" only supports Hadoop 1.0

Obviously, you have versions incompatibility between you Hadoop and Hive versions. You need to upgrade (or downgrade) your Hadoop version or Hive version.
This is due the incompatibility between Hadoop 1 and Hadoop 2.

Look for entries like this
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
</dependency>
in your pom.xml.
These define the hadoop version to use. Change them or remove them as per your requirements.

Even I ran through this problem.
Was trying use HCatMultipleInputs with hive-hcatalog-core-0.13.0.jar. We are using hadoop 2.5.1.
The following code change helped me fix the issue:
//JobContext ctx = new JobContext(conf,jobContext.getJobID());
JobContext ctx = new Job(conf);

Debugging a ClassNotFoundException throw my a Maven Mojo when using #requiresDependencyResolution in a Mojo

A ClassNotFoundException is being thrown in a plugin I've developed. The class which can't be founf definitely exists and its associated project is included as a dependency in the executing project's pom.xml file as follows:
<dependency>
<groupId>com.example</groupId>
<artifactId>project-one</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
My plugin is included the executing pom.xml as follows:
<build>
<plugins>
<plugin>
<groupId>com.example</groupId>
<artifactId>project-two-plugin</artifactId>
<version>1.0</version>
<executions>
<execution>
<configuration>
<customSettingOne>
setting
</customSettingOne>
</configuration>
<phase>prepare-package</phase>
<goals>
<goal>some-task</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
My plugin class is as follows:
/**
* #goal some-task
*
* #requiresDependencyResolution compile
*/
public class MyPluginMojo extends AbstractMojo {
/**
* Directory to save the CSV files to.
*
* #parameter alias="customSettingOne"
* #required
*/
private File customSettingOne;
}
I have tried this code using:
Apache Maven 2.2.1 (r801777; 2009-08-06 20:16:01+0100)
and the embedded version used by Eclipse m2e
Embedded (3.0.2/1.0.200.20111228-1245
I get a ClassNotFoundException when my plugin code tried to load the class from ProjectOne.
Anyone have any ideas how I can get to the bottom of this? Is it possible to inspect or dump out the classpath being used in the plugin?

I would check here first:
Guide to Maven Classloading
and if that doesn't help, maybe a bit of diagnostic code like the following:
package stackoverflow;
import java.net.URL;
import java.net.URLClassLoader;
public class PrintClassLoader {
public static void main(String[] args) {
PrintClassLoader pcl = new PrintClassLoader();
pcl.printClassLoader(pcl.getClass().getClassLoader());
}
public void printClassLoader(ClassLoader classLoader) {
if (null == classLoader) {
return;
}
System.out.println("--------------------");
System.out.println(classLoader);
if (classLoader instanceof URLClassLoader) {
URLClassLoader ucl = (URLClassLoader) classLoader;
int i = 0;
for (URL url : ucl.getURLs()) {
System.out.println("url[" + (i++) + "]=" + url);
}
}
printClassLoader(classLoader.getParent());
}
}
For example will print something like:
--------------------
sun.misc.Launcher$AppClassLoader#35ce36
url[0]=file:/D:/dev/workspaces/3.6/all/Z_temp/target/classes/
url[1]=file:/D:/dev/.m2/repository/javax/mail/mail/1.4/mail-1.4.jar
url[2]=file:/D:/dev/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar
url[3]=file:/D:/dev/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar
--------------------
sun.misc.Launcher$ExtClassLoader#757aef
url[0]=file:/C:/java/jdk/jdk1.6.0_31/jre/lib/ext/dnsns.jar
url[1]=file:/C:/java/jdk/jdk1.6.0_31/jre/lib/ext/localedata.jar
url[2]=file:/C:/java/jdk/jdk1.6.0_31/jre/lib/ext/sunjce_provider.jar
url[3]=file:/C:/java/jdk/jdk1.6.0_31/jre/lib/ext/sunmscapi.jar
url[4]=file:/C:/java/jdk/jdk1.6.0_31/jre/lib/ext/sunpkcs11.jar

Categories

Resources