I have a json file and want to convert it to parquet file.
I know how to convert from json file to java objects but still need the conversion from java objects to parquet file.
I don't want to convert using AVRO/ Protocol Buffers/Thrift/etc.. and also, don't want to use the Apache drill- just a java code.
Im using Java project with maven, my pom.xml look like this:
<dependencies>
<dependency>
<groupId>org.apache.parquet</groupId>
<artifactId>parquet-common</artifactId>
<version>1.8.1</version>
</dependency>
<dependency>
<groupId>org.apache.parquet</groupId>
<artifactId>parquet-encoding</artifactId>
<version>1.8.1</version>
</dependency>
<dependency>
<groupId>org.apache.parquet</groupId>
<artifactId>parquet-column</artifactId>
<version>1.8.1</version>
</dependency>
<dependency>
<groupId>org.apache.parquet</groupId>
<artifactId>parquet-hadoop</artifactId>
<version>1.8.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>0.20.2</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.2.0</version>
</dependency>
</dependencies>
Means I don't have a problem using hadoop.
So I found this link (the last comment):
https://groups.google.com/forum/#!topic/parquet-dev/lfWanFOc040
The above link has what I want, but I have a problem with 2 imports there and can't find the correct dependencies for them.
Those are the 2 imports I can't find:
import org.apache.hadoop.hive.ql.io.parquet.writable.BinaryWritable;
import org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport;
will appreciate your help for finding the dependencies.
THANX:)
OK I found the answer:
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>0.13.0</version>
</dependency>
enjoy!
Related
I am developing jira server plugin which uses a 3rd party library which contains jaxb as a runtime dependency. Following is the pom xml file of that 3rd party library
<dependencies>
<dependency>
<groupId>org.threeten</groupId>
<artifactId>threetenbp</artifactId>
<version>1.3.5</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>io.gsonfire</groupId>
<artifactId>gson-fire</artifactId>
<version>1.8.0</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
<version>2.2.12</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>io.swagger</groupId>
<artifactId>swagger-annotations</artifactId>
<version>1.5.20</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.5</version>
<scope>compile</scope>
</dependency>
And also a jaxb version (v2.3.0) is available in the jira server runtime. I am getting the following error
ClassCastException: attempting to cast jar:file:/C:/JiraServer/Jira%20Software/Atlassian/Jira/atlassian-jira/WEB-INF/lib/jaxb-api-2.3.0.jar!/javax/xml/bind/JAXBContext.class to
bundle://252.0:18/javax/xml/bind/JAXBContext.class. Please make sure that you are specifying the proper ClassLoader.
I tried to exclude the jaxb from the 3rd party library that I am using, but it did not worked for me. Can someone please advice how to resolve this issue.
Thanks!
Mifraz
I am developing a NiFi processor and want to test it with JUnit, but one of the properties of Processor is a ControllerService, which is DBCPConnectionPool, but I don't know how to add this service to the JUnit Test?
I try to new DBCPConnectionPool(), but there no object named like that, although I have added all dependencies to my pom file, like the followings:
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-dbcp-service-api</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-utils</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-standard-services-api-nar</artifactId>
<type>nar</type>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-dbcp-service-api</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-mock</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-api</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-processor-utils</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-utils</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-dbcp-service-api</artifactId>
</dependency>
runner.addControllerService(...);
Please give some recommendation about how to write this unit test, thanks.
It looks like you included the nifi-dbcp-service-api module twice, but one of those should be nifi-dbcp-service, which contains the implementation classes.
Then look at DBCPServiceTest for examples of how to use the DBCPConnectionPool in a unit test.
I just upgraded Tapestry 5.3.8 to 5.4.0, and then I found the #Secured annotation has an issue. If I go to page which has this #Secured, the exception org/apache/tapestry5/services/TransformMethodSignature and could not initialize class nu.localhost.tapestry5.springsecurity.services.internal.TransformConstants2 will thrown (with no detail). Is it no longer supported in 5.4.0? Or do I need to upgrade something in my pom.xml?
pom.xml
<dependency>
<groupId>org.apache.tapestry</groupId>
<artifactId>tapestry-core</artifactId>
<version>${tapestry-release-version}</version>
</dependency>
<dependency>
<groupId>org.apache.tapestry</groupId>
<artifactId>tapestry-hibernate</artifactId>
<version>${tapestry-release-version}</version>
</dependency>
<dependency>
<groupId>org.apache.tapestry</groupId>
<artifactId>tapestry-upload</artifactId>
<version>${tapestry-release-version}</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-c3p0</artifactId>
<version>4.2.0.Final</version>
</dependency>
<dependency>
<groupId>com.mchange</groupId>
<artifactId>c3p0</artifactId>
<version>0.9.5.2</version>
</dependency>
<dependency>
<groupId>org.apache.tapestry</groupId>
<artifactId>tapestry-spring</artifactId>
<version>${tapestry-release-version}</version>
</dependency>
<dependency>
<groupId>org.apache.tapestry</groupId>
<artifactId>tapestry-kaptcha</artifactId>
<version>${tapestry-release-version}</version>
</dependency>
<dependency>
<groupId>org.got5</groupId>
<artifactId>tapestry5-jquery</artifactId>
<version>4.1.1</version>
</dependency>
<dependency>
<groupId>com.github.lltyk</groupId>
<artifactId>tapestry-spring-security</artifactId>
<version>3.0.3</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
<version>4.3.0.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
<version>4.3.0.RELEASE</version>
<scope>test</scope>
</dependency>
Well, after struggling for days, I'm finally figured out that this library:
<dependency>
<groupId>com.github.lltyk</groupId>
<artifactId>tapestry-spring-security</artifactId>
<version>3.0.3</version>
</dependency>
Can not keep up with Tapesty 5.4, because it has a class: nu.localhost.tapestry5.springsecurity.services.internal.TransformConstants2 that refers to org.apache.tapestry5.services.TransformMethodSignature which is not available any longer in Tapestry 5.4.
I've tried to upgrade tapestry-spring-security to highest version: 3.0.5, but still can't keep up with Tapestry 5.4, so the only way to solve it, is to remove it. And to replace all the functionalities which was provided by tapestry-spring-security, I need to add 4 new libs to pom (in your case you might need to add more or less than 4): spring-security-core: 4.0.0.M1, spring-security-web: 4.2.3.RELEASE, spring-dao: 2.0.8, servlet-api: 2.5
In my case, I need to copy the code from the original source for these classes: LogoutService (interface), LogoutServiceImpl, IfRole and IfLoggedIn. Create the files and paste the code.
I have written an integration test for ElasticSearch by extending ElasticsearchIntegrationTest. Top section of the test below:
import org.junit.*;
import static org.junit.Assert.assertEquals;
import org.elasticsearch.test.ElasticsearchIntegrationTest;
public class ProductSearchTest extends ElasticsearchIntegrationTest
I'm seeing this error from JUnit when trying to run the test via maven
java.lang.NoSuchMethodError: com.carrotsearch.randomizedtesting.RandomizedContext.runWithPrivateRandomness(Lcom/carrotsearch/randomizedtesting/Randomness;Ljava/util/concurrent/Callable;)Ljava/lang/Object;
at __randomizedtesting.SeedInfo.seed([9DE685AB75B54F0A:10B1B129F9E3CB67]:0)
I've included the dependencies in the pom as described on the elasticsearch site
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-test-framework</artifactId>
<version>4.10.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>1.4.0</version>
<scope>test</scope>
<type>test-jar</type>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>1.4.0</version>
<scope>test</scope>
</dependency>
Has anyone seen this before? It looks like i'm missing a dependency (i've tried adding a couple of the carrotsearch maven dependencies into the pom without success). Alternatively, does anyone have the integration tests working with the versions I am using, so at least if I expend hours trying to locate the dependency issue I can be confident it will work once identified.
Thank you for your time.
Indeed, it seems there is a mismatch between the randomizedtesting jar Lucene uses and the one ES uses. Try this in your pom.xml:
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-test-framework</artifactId>
<version>4.10.2</version>
<scope>test</scope>
<exclusions>
<exclusion>
<artifactId>randomizedtesting-runner</artifactId>
<groupId>
com.carrotsearch.randomizedtesting
</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>1.4.0</version>
<scope>test</scope>
<type>test-jar</type>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>1.4.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.carrotsearch.randomizedtesting</groupId>
<artifactId>randomizedtesting-runner</artifactId>
<version>2.1.10</version>
<scope>test</scope>
</dependency>
I am trying to get a clean code sample of Google API OAuth2 authentication.
My code is mostly based on the explanation at https://code.google.com/p/google-api-java-client/wiki/OAuth2
I deviate from this tutorial because I want my libraries be managed by Maven and because I want to create a project without Google Apps Engine dependencies.
Each time I try to the run the project on my Glassfish server, I get the following error, probably indicating a dependency conflict.
java.lang.NoSuchMethodError: com.google.api.client.json.JsonFactory.fromInputStream(Ljava/io/InputStream;Ljava/lang/Class;)Ljava/lang/Object;
The relevant piece of my pom.xml:
<dependencies>
<dependency>
<groupId>com.google.api-client</groupId>
<artifactId>google-api-client</artifactId>
<version>1.14.1-beta</version>
</dependency>
<dependency>
<groupId>com.google.http-client</groupId>
<artifactId>google-http-client</artifactId>
<version>1.14.1-beta</version>
</dependency>
<dependency>
<groupId>com.google.api-jackson2</groupId>
<artifactId>google-api-jackson2</artifactId>
<version>1.14.1-beta</version>
</dependency>
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-calendar</artifactId>
<version>v3-rev34-1.14.1-beta</version>
</dependency>
<dependency>
<groupId>com.google.oauth-client</groupId>
<artifactId>google-oauth-client</artifactId>
<version>1.14.1-beta</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.1.3</version>
</dependency>
<dependency>
<groupId>com.google.oauth-client</groupId>
<artifactId>google-oauth-client-servlet</artifactId>
<version>1.14.1-beta</version>
</dependency>
<dependency>
<groupId>javax.persistence</groupId>
<artifactId>persistence-api</artifactId>
<version>1.0.2</version>
</dependency>
<dependency>
<groupId>javax.validation</groupId>
<artifactId>validation-api</artifactId>
<version>1.1.0.CR2</version>
</dependency>
<dependency>
<groupId>javax.faces</groupId>
<artifactId>jsf-api</artifactId>
<version>2.1</version>
</dependency>
<dependency>
<groupId>javax.enterprise</groupId>
<artifactId>cdi-api</artifactId>
<version>1.1-PRD</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<version>3.0.1</version>
</dependency>
<dependency>
<groupId>javax.annotation</groupId>
<artifactId>jsr250-api</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>javax</groupId>
<artifactId>javaee-web-api</artifactId>
<version>6.0</version>
</dependency>
</dependencies>
Please help me find the root cause of this error, because the used dependencies seem just fine to me and there's no similar problem case I could find.
The correct dependency for jackson2 is:
<dependency>
<groupId>com.google.http-client</groupId>
<artifactId>google-http-client-jackson2</artifactId>
<version>1.14.1-beta</version>
</dependency>
Once I changed that, the project built successfully, and I was able to call the method in question just fine.
There could be something else on the classpath that's interfering. I suggest you run the program with -verbose: class (see here for a more detailed explanation), which will show any conflicts.