I am getting below error when doing count on the parquet file,
java.lang.NoSuchMethodError: org.apache.parquet.schema.Types$MessageTypeBuilder.addFields([Lorg/apache/parquet/schema/Type;)Lorg/apache/parquet/schema/Types$GroupBuilder;
at org.apache.spark.sql.execution.datasources.parquet.CatalystReadSupport$.clipParquetSchema(CatalystReadSupport.scala:117)
Detailed error as below:
ERROR org.apache.spark.util.SparkUncaughtExceptionHandler - Uncaught exception in thread Thread[Executor task launch worker-4,5,main]
java.lang.NoSuchMethodError:
org.apache.parquet.schema.Types$MessageTypeBuilder.addFields([Lorg/apache/parquet/schema/Type;)Lorg/apache/parquet/schema/Types$GroupBuilder;
at
org.apache.spark.sql.execution.datasources.parquet.CatalystReadSupport$.clipParquetSchema(CatalystReadSupport.scala:117)
at org.apache.spark.sql.execution.datasources.parquet.CatalystReadSupport.init(CatalystReadSupport.scala:70)
at org.apache.spark.sql.execution.datasources.parquet.SpecificParquetRecordReaderBase.initialize(SpecificParquetRecordReaderBase.java:135)
at org.apache.spark.sql.execution.datasources.parquet.UnsafeRowParquetRecordReader.initialize(UnsafeRowParquetRecordReader.java:130)
at org.apache.spark.sql.execution.datasources.parquet.UnsafeRowParquetRecordReader.tryInitialize(UnsafeRowParquetRecordReader.java:117)
at org.apache.spark.rdd.SqlNewHadoopRDD$$anon$1.<init>(SqlNewHadoopRDD.scala:169)
at org.apache.spark.rdd.SqlNewHadoopRDD.compute(SqlNewHadoopRDD.scala:126)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Couldnt figure out the issue. the dataframe has schemas as expected. Can you anyone show me the direction here
Parquet related jars will not be present under lib folder for spark 1.6.1.
Download the jar from here. Unzip the downloaded folder and place the jar inside lib folder present within spark folder i.e $SPARK_HOME/lib.
This should solve your problem.
Related
2022-07-05 14:40:46,411 [Catalina-utility-2] INFO org.apache.catalina.core.ContainerBase.[Catalina].[site].[/bk]- 2 Spring WebApplicationInitializers detected on classpath
2022-07-05 14:40:46,614 [Catalina-utility-2] ERROR org.apache.catalina.startup.HostConfig- Error deploying web application archive [/home/user/tomcat/webapps/site/bk.war]
java.lang.IllegalStateException: Error starting child
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:716) ~[catalina.jar:9.0.16]
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690) ~[catalina.jar:9.0.16]
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:695) ~[catalina.jar:9.0.16]
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:978) [catalina.jar:9.0.16]
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1850) [catalina.jar:9.0.16]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) [tomcat-util.jar:9.0.16]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) [?:?]
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:773) [catalina.jar:9.0.16]
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:427) [catalina.jar:9.0.16]
at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1621) [catalina.jar:9.0.16]
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:305) [catalina.jar:9.0.16]
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123) [catalina.jar:9.0.16]
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1144) [catalina.jar:9.0.16]
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1346) [catalina.jar:9.0.16]
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1350) [catalina.jar:9.0.16]
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1328) [catalina.jar:9.0.16]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:9.0.16]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[site].StandardContext[/bk]]
at org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:441) ~[catalina.jar:9.0.16]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:198) ~[catalina.jar:9.0.16]
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:713) ~[catalina.jar:9.0.16]
... 24 more
Implying that you didn't move anything else in the configuration:
From the Tomcat 9.0 documentation (Emphasis mine)
Copy unpacked directory hierarchy into a subdirectory in directory $CATALINA_BASE/webapps/. Tomcat will assign a context path to your application based on the subdirectory name you choose. We will use this technique in the build.xml file that we construct, because it is the quickest and easiest approach during development. Be sure to restart Tomcat after installing or updating your application.
Copy the web application archive file into directory $CATALINA_BASE/webapps/. When Tomcat is started, it will automatically expand the web application archive file into its unpacked form, and execute the application that way. This approach would typically be used to install an additional application, provided by a third party vendor or by your internal development staff, into an existing Tomcat installation. NOTE - If you use this approach, and wish to update your application later, you must both replace the web application archive file AND delete the expanded directory that Tomcat created, and then restart Tomcat, in order to reflect your changes.
From the trace, you have your .war file in /home/user/tomcat/webapps/site/. In other words, you are mixing the options Tomcat gives you.
Either
Move your .war file to /home/user/tomcat/webapps/
Move the .war content into /home/user/tomcat/webapps/site/
Note that you for (1) you will need to access your app as http://localhost:8080/bk/ but for (2) it would be http://localhost:8080/site/.
I have a Clojure project using Leiningen that needs access to the Firebase Admin SDK. So I'm trying to use the Java version from Google's Maven. What ends up happening when I run lein repl is it starts to download all of the requirements for that particular jar but then crashes with the following error(s):
Could not find artifact com.google.firebase:firebase-database:jar:19.0.2 in central (https://repo1.maven.org/maven2/)
Could not find artifact com.google.firebase:firebase-database:jar:19.0.2 in clojars (https://repo.clojars.org/)
Could not find artifact com.google.firebase:firebase-database:jar:19.0.2 in google (https://maven.google.com/)
This could be due to a typo in :dependencies, file system permissions, or network issues.
If you are behind a proxy, try setting the 'http_proxy' environment variable.
Exception in thread "Thread-1" clojure.lang.ExceptionInfo: Could not resolve dependencies {:suppress-msg true, :exit-code 1}
at clojure.core$ex_info.invokeStatic(core.clj:4617)
at clojure.core$ex_info.invoke(core.clj:4617)
at leiningen.core.classpath$get_dependencies_STAR_.invokeStatic(classpath.clj:311)
at leiningen.core.classpath$get_dependencies_STAR_.invoke(classpath.clj:265)
at clojure.lang.AFn.applyToHelper(AFn.java:165)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invokeStatic(core.clj:646)
at clojure.core$apply.invoke(core.clj:641)
at clojure.core$memoize$fn__14312.doInvoke(core.clj:6109)
at clojure.lang.RestFn.invoke(RestFn.java:457)
at leiningen.core.classpath$get_dependencies$fn__9133.invoke(classpath.clj:332)
at leiningen.core.classpath$get_dependencies.invokeStatic(classpath.clj:330)
at leiningen.core.classpath$get_dependencies.doInvoke(classpath.clj:324)
at clojure.lang.RestFn.invoke(RestFn.java:445)
at clojure.lang.AFn.applyToHelper(AFn.java:160)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.core$apply.invokeStatic(core.clj:652)
at clojure.core$apply.invoke(core.clj:641)
at leiningen.core.classpath$resolve_managed_dependencies.invokeStatic(classpath.clj:441)
at leiningen.core.classpath$resolve_managed_dependencies.doInvoke(classpath.clj:428)
at clojure.lang.RestFn.invoke(RestFn.java:445)
at leiningen.core.eval$prep.invokeStatic(eval.clj:85)
at leiningen.core.eval$prep.invoke(eval.clj:73)
at leiningen.core.eval$eval_in_project.invokeStatic(eval.clj:362)
at leiningen.core.eval$eval_in_project.invoke(eval.clj:356)
at leiningen.repl$server$fn__12516.invoke(repl.clj:244)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invokeStatic(core.clj:646)
at clojure.core$apply.invoke(core.clj:641)
at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1890)
at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1881)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at clojure.core$apply.invokeStatic(core.clj:650)
at clojure.core$apply.invoke(core.clj:641)
at clojure.core$bound_fn_STAR_$fn__13275.doInvoke(core.clj:1912)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.run(AFn.java:22)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.eclipse.aether.resolution.DependencyResolutionException: Could not find artifact com.google.firebase:firebase-database:jar:19.0.2 in central (https://repo1.maven.org/maven2/)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:355)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
at cemerick.pomegranate.aether$resolve_dependencies_STAR_.invokeStatic(aether.clj:806)
at cemerick.pomegranate.aether$resolve_dependencies_STAR_.doInvoke(aether.clj:707)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invokeStatic(core.clj:646)
at clojure.core$apply.invoke(core.clj:641)
at cemerick.pomegranate.aether$resolve_dependencies.invokeStatic(aether.clj:815)
at cemerick.pomegranate.aether$resolve_dependencies.doInvoke(aether.clj:809)
at clojure.lang.RestFn.invoke(RestFn.java:2793)
at leiningen.core.classpath$get_dependencies_STAR_.invokeStatic(classpath.clj:273)
... 38 more
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not find artifact com.google.firebase:firebase-database:jar:19.0.2 in central (https://repo1.maven.org/maven2/)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:422)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:224)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:338)
... 53 more
Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact com.google.firebase:firebase-database:jar:19.0.2 in central (https://repo1.maven.org/maven2/)
at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed(ArtifactTransportListener.java:48)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:365)
at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:75)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute(BasicRepositoryConnector.java:583)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get(BasicRepositoryConnector.java:259)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:498)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:399)
... 55 more
My project.clj is as follows:
(defproject xxxxx "0.1.0-SNAPSHOT"
:description "FIXME: write description"
:url "http://example.com/FIXME"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:dependencies [[org.clojure/clojure "1.9.0"]
[http-kit "2.3.0"]
[datascript "0.18.7"]
[com.google.firebase/firebase-database "19.0.2"]]
:main ^:skip-aot xxxxx.xx
:target-path "target/%s"
:profiles {:uberjar {:aot :all}}
:repositories [["google" "https://maven.google.com/"]])
Actual project name has been replaced with x's of course.
Any ideas of what this problem could be? I'm on Ubuntu 18.04 by the way.
com.google.firebase:firebase-database:jar:19.0.2 is not the dependency for the Firebase Admin SDK. That's the Firebase Realtime Database Android client SDK.
As shown in the documentation, the latest version of the Firebase Admin SDK for Java is com.google.firebase:firebase-admin:6.11.0.
I was working on my Play project when I got BSOD. After computer rebooted I started getting the following exception when trying to run any activator command (like activator clean or activator run):
[info] Done updating.
Error wrapping InputStream in GZIPInputStream: java.util.zip.ZipException: Not in GZIP format
at sbt.ErrorHandling$.translate(ErrorHandling.scala:10)
at sbt.WrapUsing.open(Using.scala:34)
at sbt.Using.apply(Using.scala:23)
at sbt.IO$$anonfun$gzipFileIn$1.apply(IO.scala:863)
at sbt.IO$$anonfun$gzipFileIn$1.apply(IO.scala:862)
at sbt.Using.apply(Using.scala:24)
at sbt.IO$.gzipFileIn(IO.scala:862)
at sbt.Sync$.readUncaught(Sync.scala:88)
at sbt.Sync$.readInfo(Sync.scala:84)
at sbt.Sync$$anonfun$apply$1.apply(Sync.scala:28)
at sbt.Sync$$anonfun$apply$1.apply(Sync.scala:22)
at sbt.Defaults$$anonfun$copyResourcesTask$1.apply(Defaults.scala:941)
at sbt.Defaults$$anonfun$copyResourcesTask$1.apply(Defaults.scala:937)
at scala.Function4$$anonfun$tupled$1.apply(Function4.scala:35)
at scala.Function4$$anonfun$tupled$1.apply(Function4.scala:34)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
at sbt.std.Transform$$anon$4.work(System.scala:63)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.Execute.work(Execute.scala:237)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.zip.ZipException: Not in GZIP format
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:165)
at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:79)
at sbt.Using$$anonfun$gzipInputStream$1.apply(Using.scala:84)
at sbt.Using$$anonfun$gzipInputStream$1.apply(Using.scala:84)
at sbt.Using$$anon$1.openImpl(Using.scala:51)
at sbt.WrapUsing$$anonfun$open$2.apply(Using.scala:34)
at sbt.ErrorHandling$.translate(ErrorHandling.scala:10)
at sbt.WrapUsing.open(Using.scala:34)
at sbt.Using.apply(Using.scala:23)
at sbt.IO$$anonfun$gzipFileIn$1.apply(IO.scala:863)
at sbt.IO$$anonfun$gzipFileIn$1.apply(IO.scala:862)
at sbt.Using.apply(Using.scala:24)
at sbt.IO$.gzipFileIn(IO.scala:862)
at sbt.Sync$.readUncaught(Sync.scala:88)
at sbt.Sync$.readInfo(Sync.scala:84)
at sbt.Sync$$anonfun$apply$1.apply(Sync.scala:28)
at sbt.Sync$$anonfun$apply$1.apply(Sync.scala:22)
at sbt.Defaults$$anonfun$copyResourcesTask$1.apply(Defaults.scala:941)
at sbt.Defaults$$anonfun$copyResourcesTask$1.apply(Defaults.scala:937)
at scala.Function4$$anonfun$tupled$1.apply(Function4.scala:35)
at scala.Function4$$anonfun$tupled$1.apply(Function4.scala:34)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
at sbt.std.Transform$$anon$4.work(System.scala:63)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.Execute.work(Execute.scala:237)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[error] (root/compile:copyResources) Error wrapping InputStream in GZIPInputStream: java.util.zip.ZipException: Not in GZIP format
I tried all workarounds I managed to find. I deleted the "project" and all "target" folders in the project I tried to run and the project it depends on. I deleted ".sbt" folder in "C:\Users...". I even cleaned TEMP directory. Nothing helped - I still get this error after "Loading project definition from.." and "Resolving..." messages in the log. What else can be done to solve the issue?
Try to clean the project's target directory
cd ${your_project}
rm -fr project/target
rm -fr project/project/target
I've solved the problem - there were some other dependent projects, when I cleaned them the issue disapppeared.
When I'm running MLlib for files > 1 partition in our cluster I get the following exception:
16/08/14 12:43:23 WARN TaskSetManager: Lost task 2.0 in stage 2.1 (TID
49, da06.qcri.org): FetchFailed(BlockManagerId(3, da08.qcri.org,
33322), shuffleId=0, mapId=5, reduceId=2, message=
org.apache.spark.shuffle.FetchFailedException: Failed to connect to
da08.qcri.org:33322 at
org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:323)
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:300)
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:51)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at
scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) at
scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at
org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at
org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:152)
at
org.apache.spark.Aggregator.combineCombinersByKey(Aggregator.scala:58)
at
org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:83)
at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:98) at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at
org.apache.spark.scheduler.Task.run(Task.scala:89) at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by:
java.io.IOException: Failed to connect to ***.org:33322 at
org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:216)
at
org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:167)
at
org.apache.spark.network.netty.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferService.scala:90)
at
org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140)
at
org.apache.spark.network.shuffle.RetryingBlockFetcher.access$200(RetryingBlockFetcher.java:43)
at
org.apache.spark.network.shuffle.RetryingBlockFetcher$1.run(RetryingBlockFetcher.java:170)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3
more
Caused by: java.nio.channels.UnresolvedAddressException at
sun.nio.ch.Net.checkAddress(Net.java:123) at
sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:621) at
io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:209)
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:207)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1097)
at
io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:471)
at
io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:456)
at
io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47)
at
io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:471)
at
io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:456)
at
io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50)
at
io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:471)
at
io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:456)
at
io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:438)
at
io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:908)
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:203)
at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:166) at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
... 1 more
In the slaves configuration file, I have the nodes by IP and not by hostname. In addition, when I ping the machines from the master node using the hostname, it does not seem to have any issue.
Anyone had a similar issue or has some idea on how to solve it?
I m new in Android Studio. I m using Android Studio 1.0 RC 2 Build #AI-135.1611497 and JDK: 1.8.0_25. Trying to import project, and getting error. Allready tried delete .gradle.. Gradle is setted to Use default gradle wrapper. Any idea to solve it?
2014-12-02 10:52:19,643 [3351320] INFO - s.plugins.gradle.GradleManager - Instructing gradle to use java from C:\Program Files\Java\jdk1.8.0_25\
2014-12-02 10:52:19,647 [3351324] INFO - s.plugins.gradle.GradleManager - Instructing gradle to use java from C:\Program Files\Java\jdk1.8.0_25\
2014-12-02 10:52:19,649 [3351326] INFO - ls.idea.gradle.util.GradleUtil - Looking for embedded Maven repo at 'C:\Users\tomas\Android-Studio\gradle\m2repository'
2014-12-02 10:52:19,652 [3351329] INFO - .project.GradleExecutionHelper - Passing command-line args to Gradle Tooling API: [-Pandroid.injected.build.model.only=true, -Pandroid.injected.invoked.from.ide=true, --init-script, C:\Users\tomas\AppData\Local\Temp\asLocalRepo4461828038903261980.gradle]
2014-12-02 10:52:21,777 [3353454] INFO - .project.GradleProjectResolver - Gradle project resolve error
org.gradle.tooling.BuildException: Could not run build action using Gradle distribution 'https://services.gradle.org/distributions/gradle-2.2.1-all.zip'.
at org.gradle.tooling.internal.consumer.ResultHandlerAdapter.onFailure(ResultHandlerAdapter.java:57)
at org.gradle.tooling.internal.consumer.async.DefaultAsyncConsumerActionExecutor$1$1.run(DefaultAsyncConsumerActionExecutor.java:57)
at org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at org.gradle.tooling.internal.consumer.BlockingResultHandler.getResult(BlockingResultHandler.java:46)
at org.gradle.tooling.internal.consumer.DefaultBuildActionExecuter.run(DefaultBuildActionExecuter.java:46)
at org.jetbrains.plugins.gradle.service.project.GradleProjectResolver.doResolveProjectInfo(GradleProjectResolver.java:186)
at org.jetbrains.plugins.gradle.service.project.GradleProjectResolver.access$300(GradleProjectResolver.java:64)
at org.jetbrains.plugins.gradle.service.project.GradleProjectResolver$ProjectConnectionDataNodeFunction.fun(GradleProjectResolver.java:361)
at org.jetbrains.plugins.gradle.service.project.GradleProjectResolver$ProjectConnectionDataNodeFunction.fun(GradleProjectResolver.java:333)
at org.jetbrains.plugins.gradle.service.project.GradleExecutionHelper.execute(GradleExecutionHelper.java:203)
at org.jetbrains.plugins.gradle.service.project.GradleProjectResolver.resolveProjectInfo(GradleProjectResolver.java:116)
at org.jetbrains.plugins.gradle.service.project.GradleProjectResolver.resolveProjectInfo(GradleProjectResolver.java:64)
at com.intellij.openapi.externalSystem.service.remote.RemoteExternalSystemProjectResolverImpl$1.produce(RemoteExternalSystemProjectResolverImpl.java:41)
at com.intellij.openapi.externalSystem.service.remote.RemoteExternalSystemProjectResolverImpl$1.produce(RemoteExternalSystemProjectResolverImpl.java:37)
at com.intellij.openapi.externalSystem.service.remote.AbstractRemoteExternalSystemService.execute(AbstractRemoteExternalSystemService.java:59)
at com.intellij.openapi.externalSystem.service.remote.RemoteExternalSystemProjectResolverImpl.resolveProjectInfo(RemoteExternalSystemProjectResolverImpl.java:37)
at com.intellij.openapi.externalSystem.service.remote.wrapper.ExternalSystemProjectResolverWrapper.resolveProjectInfo(ExternalSystemProjectResolverWrapper.java:49)
at com.intellij.openapi.externalSystem.service.internal.ExternalSystemResolveProjectTask.doExecute(ExternalSystemResolveProjectTask.java:48)
at com.intellij.openapi.externalSystem.service.internal.AbstractExternalSystemTask.execute(AbstractExternalSystemTask.java:137)
at com.intellij.openapi.externalSystem.service.internal.AbstractExternalSystemTask.execute(AbstractExternalSystemTask.java:123)
at com.intellij.openapi.externalSystem.util.ExternalSystemUtil$2.execute(ExternalSystemUtil.java:406)
at com.intellij.openapi.externalSystem.util.ExternalSystemUtil$3$1.run(ExternalSystemUtil.java:474)
at com.intellij.openapi.progress.impl.ProgressManagerImpl$TaskRunnable.run(ProgressManagerImpl.java:471)
at com.intellij.openapi.progress.impl.ProgressManagerImpl$6.run(ProgressManagerImpl.java:281)
at com.intellij.openapi.progress.impl.ProgressManagerImpl$2.run(ProgressManagerImpl.java:178)
at com.intellij.openapi.progress.ProgressManager.executeProcessUnderProgress(ProgressManager.java:209)
at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:212)
at com.intellij.openapi.progress.impl.ProgressManagerImpl.runProcess(ProgressManagerImpl.java:171)
at com.intellij.openapi.application.impl.ApplicationImpl$10$1.run(ApplicationImpl.java:645)
at com.intellij.openapi.application.impl.ApplicationImpl$8.run(ApplicationImpl.java:419)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at com.intellij.openapi.application.impl.ApplicationImpl$1$1.run(ApplicationImpl.java:149)
Caused by: org.gradle.internal.exceptions.LocationAwareException: A problem occurred evaluating project ':app'.
at org.gradle.initialization.DefaultExceptionAnalyser.transform(DefaultExceptionAnalyser.java:77)
at org.gradle.initialization.MultipleBuildFailuresExceptionAnalyser.transform(MultipleBuildFailuresExceptionAnalyser.java:47)
at org.gradle.initialization.StackTraceSanitizingExceptionAnalyser.transform(StackTraceSanitizingExceptionAnalyser.java:30)
at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:108)
at org.gradle.initialization.DefaultGradleLauncher.getBuildAnalysis(DefaultGradleLauncher.java:97)
at org.gradle.launcher.exec.InProcessBuildActionExecuter$DefaultBuildController.configure(InProcessBuildActionExecuter.java:84)
at org.gradle.tooling.internal.provider.ClientProvidedBuildAction.run(ClientProvidedBuildAction.java:43)
at org.gradle.tooling.internal.provider.ClientProvidedBuildAction.run(ClientProvidedBuildAction.java:31)
at org.gradle.tooling.internal.provider.ConfiguringBuildAction.run(ConfiguringBuildAction.java:119)
at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:36)
at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:26)
at org.gradle.launcher.daemon.server.exec.ExecuteBuild.doBuild(ExecuteBuild.java:47)
at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:34)
at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:119)
at org.gradle.launcher.daemon.server.exec.WatchForDisconnection.execute(WatchForDisconnection.java:35)
at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:119)
at org.gradle.launcher.daemon.server.exec.ResetDeprecationLogger.execute(ResetDeprecationLogger.java:24)
at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:119)
at org.gradle.launcher.daemon.server.exec.StartStopIfBuildAndStop.execute(StartStopIfBuildAndStop.java:33)
at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:119)
at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:71)
at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:69)
at org.gradle.util.Swapper.swap(Swapper.java:38)
at org.gradle.launcher.daemon.server.exec.ForwardClientInput.execute(ForwardClientInput.java:69)
at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:119)
at org.gradle.launcher.daemon.server.exec.LogToClient.doBuild(LogToClient.java:60)
at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:34)
at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:119)
at org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment.doBuild(EstablishBuildEnvironment.java:70)
at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:34)
at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:119)
at org.gradle.launcher.daemon.server.exec.DaemonHygieneAction.execute(DaemonHygieneAction.java:39)
at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:119)
at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy$1.run(StartBuildOrRespondWithBusy.java:46)
at org.gradle.launcher.daemon.server.DaemonStateCoordinator$1.run(DaemonStateCoordinator.java:246)
at org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:64)
Caused by: org.gradle.api.GradleScriptException: A problem occurred evaluating project ':app'.
at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:54)
at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:148)
at org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:39)
at org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:26)
at org.gradle.configuration.project.ConfigureActionsProjectEvaluator.evaluate(ConfigureActionsProjectEvaluator.java:34)
at org.gradle.configuration.project.LifecycleProjectEvaluator.evaluate(LifecycleProjectEvaluator.java:59)
at org.gradle.api.internal.project.AbstractProject.evaluate(AbstractProject.java:504)
at org.gradle.api.internal.project.AbstractProject.evaluate(AbstractProject.java:83)
at org.gradle.execution.TaskPathProjectEvaluator.configureHierarchy(TaskPathProjectEvaluator.java:47)
at org.gradle.configuration.DefaultBuildConfigurer.configure(DefaultBuildConfigurer.java:35)
at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:129)
at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:106)
... 32 more
Caused by: org.gradle.process.internal.ExecException: A problem occurred starting process 'command 'git''
at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:196)
at org.gradle.process.internal.DefaultExecHandle.failed(DefaultExecHandle.java:325)
at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:83)
... 1 more
Caused by: net.rubygrapefruit.platform.NativeException: Could not start 'git'
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
at net.rubygrapefruit.platform.internal.WindowsProcessLauncher.start(WindowsProcessLauncher.java:22)
at net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:65)
... 1 more
Caused by: java.io.IOException: Cannot run program "git" (in directory "C:\Users\tomas\Eman\CUCKOO-AN"): CreateProcess error=2, The system cannot find the file specified
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
... 4 more
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified
... 5 more
The problem was on a path to Git.
in Android studio
Goto file --->settings-->versioncontrol--->Git--> give the path of git.exe
Correct this one:
C:\Program Files (x86)\Git\cmd\git.exe
user credentials
Goto file--->settings-->GitHub-> give username and password
Try to comment all 'git' commands in your build.gradle file
Add the path to the Git binary folder (the folder containing git.exe) to your system PATH.
This resolve the problem for me.
Since android studio is still referring to jdk1.7.0_40 change its path
Goto Files->Other Settings->Default Project Structure.
Under Platform Settings Select Android sdk...under JDK
location change C:\Program Files (x86)\Java\jdk1.7.0_40 to C:\Program Files\Java\1.8.0_25.
This should solve your problem.
I faced the same issue while working with a new project in gradle, initially it was working fine but after initalizing project directory as git repo IntelliJ IDEA started throwing this error, removing .git folder worked for me and then re-initalizing project dir as git repo worked fine second time
I had this problem a while back and I found out Kaspersky antivirus was moving some files to quarantine. Check your antivirus and restore any wrongly flagged files