Develop Reference

How to connect to Google big query with spark using java locally?

I am trying to connect to Google big query using spark in java, but I am unable to find accurate documentation for the same.
I tried: https://cloud.google.com/dataproc/docs/tutorials/bigquery-connector-spark-example
and
https://github.com/GoogleCloudPlatform/spark-bigquery-connector#compiling-against-the-connector
My code:
sparkSession.conf().set("credentialsFile", "/path/OfMyProjectJson.json");
Dataset<Row> dataset = sparkSession.read().format("bigquery").option("table","myProject.myBigQueryDb.myBigQuweryTable")
.load();
dataset.printSchema();
But this is throwing exception:
Exception in thread "main" java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider com.google.cloud.spark.bigquery.BigQueryRelationProvider could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:247)
at scala.collection.TraversableLike$class.filter(TraversableLike.scala:259)
at scala.collection.AbstractTraversable.filter(Traversable.scala:104)
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:614)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:190)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:164)
at com.mySparkConnector.getDataset(BigQueryFetchClass.java:12)
Caused by: java.lang.IllegalArgumentException: A project ID is required for this service but could not be determined from the builder or the environment. Please set a project ID using the builder.
at com.google.cloud.spark.bigquery.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:142)
at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:285)
at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.BigQueryOptions.<init>(BigQueryOptions.java:91)
at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.BigQueryOptions.<init>(BigQueryOptions.java:30)
at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.BigQueryOptions$Builder.build(BigQueryOptions.java:86)
at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.BigQueryOptions.getDefaultInstance(BigQueryOptions.java:159)
at com.google.cloud.spark.bigquery.BigQueryRelationProvider$.$lessinit$greater$default$2(BigQueryRelationProvider.scala:29)
at com.google.cloud.spark.bigquery.BigQueryRelationProvider.<init>(BigQueryRelationProvider.scala:40)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
... 15 more
My json file contains project_id
I tried searching for possible solutions but am unable to find any, hence please help me with finding a solution to this exception, or else any documentation on how to connect to big query with spark.

I've got exactly the same error with DataProcPySparkOperator operator in the Airflow. The fix was to provide
dataproc_pyspark_jars='gs://spark-lib/bigquery/spark-bigquery-latest_2.12.jar'
instead of
dataproc_pyspark_jars='gs://spark-lib/bigquery/spark-bigquery-latest.jar'
I guess in your case it should be passed as a command line argument like
--jars=gs://spark-lib/bigquery/spark-bigquery-latest_2.12.jar

Recently a PR handling this issue has been merged to the the spark-bigquery-connector, a new version of the connector will be released soon.
A simple solution for now is to add the environment variable GOOGLE_APPLICATION_CREDENTIALS=/path/OfMyProjectJson.json to the spark runtime.

Javafx NullPointerException when Playing Sounds [duplicate]

This question already has an answer here:
How do I determine the correct path for FXML files, CSS files, Images, and other resources needed by my JavaFX Application?
(1 answer)
Closed 2 years ago.
I am working on a Javafx game and have been attempting to implement some sound using the following code:
URL laserResource = getClass().getResource("LaserBlast.wav");
AudioClip laserPlayer = new AudioClip(laserResource.toString());
This code is instantiated in the main class before any methods. I have a module called JavaFxImages which contains all my resources including my sounds and I have been successfully able to access images for ImageView from the folder through code such as Image asteroid = new Image("Asteroid.png");.
However, when I run my game I get a NullPointerException at the line for my audio clip. I have spent the past many hours researching and trying every different way to access this resource. I've tried many permutations of the URL ("JavaFxImages/LaserBlast.wav", Desktop/JavaFxImages/LaserBlast.wav", etc.), but every time it has the same error. I'm really at a loss for what is wrong with the code. I would greatly appreciate any help that can be given.
Thanks in advance!
The full error message is given below.
Error Message:
Exception in Application constructor
Exception in thread "main" java.lang.RuntimeException: Unable to construct Application instance: class sample.Main
at com.sun.javafx.application.LauncherImpl.launchApplication1(LauncherImpl.java:907)
at com.sun.javafx.application.LauncherImpl.lambda$launchApplication$156(LauncherImpl.java:182)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at com.sun.javafx.application.LauncherImpl.lambda$launchApplication1$162(LauncherImpl.java:819)
at com.sun.javafx.application.PlatformImpl.lambda$runAndWait$176(PlatformImpl.java:326)
at com.sun.javafx.application.PlatformImpl.lambda$null$174(PlatformImpl.java:295)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.javafx.application.PlatformImpl.lambda$runLater$175(PlatformImpl.java:294)
at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)
Caused by: java.lang.NullPointerException
at sample.Main.<init>(Main.java:76)
... 10 more

Your error mostly happens on Itellij IDEA - Javafx Project
Instead of:
URL laserResource = getClass().getResource("LaserBlast.wav");
AudioClip laserPlayer = new AudioClip(laserResource.toString());
Use:
//By default your Package_NAME is: sample
File laserResource = new File("src/Package_NAME/LaserBlast.wav");
AudioClip laserPlayer = new AudioClip(laserResource.toURI().toString());

AccessControlException when adding code to MouseMotionListener Java

Recently ive been working on some listeners in an applet in java. I have had no problems with keylisteners and so I have begun working with mousemotionlisteners. However, as soon as I added code to the MouseMoved() method, I got an AccessControlException when I ran. The code compiled fine. Here is the code in the MouseMotionListener:
public void mouseMoved( MouseEvent e){
mouseX = e.getXOnScreen();
mouseY = e.getYOnScreen();
move();
}
I am doing this on my home computer, but this same code works on the computer at my school. Here is the exception error:
java.security.AccessControlException: access denied ("java.awt.AWTPermission" "watchMousePointer")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
at java.security.AccessController.checkPermission(AccessController.java:559)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at java.awt.MouseInfo.getPointerInfo(MouseInfo.java:79)
at Test.<init>(Test.java:35)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.lang.Class.newInstance(Class.java:374)
at sun.applet.AppletPanel.createApplet(AppletPanel.java:793)
at sun.applet.AppletPanel.runLoader(AppletPanel.java:722)
at sun.applet.AppletPanel.run(AppletPanel.java:379)
at java.lang.Thread.run(Thread.java:744)
Does anybody know why I cannot run this program? I've read about giving myself permissions but nobody explains how to do it.
Thanks!

Get rid of the SecurityManager, or grant yourself that permission in your .policy file.

Exception in thread "main" java.lang.NoClassDefFoundError with NetBeans

Im getting this Exception with my project,after I run the project X times, I get it, but it is not always in the same class, sometimes it happens with other classes, or being more especific, with other frames, as far as I have seen, I have got it only with my frames.
when I get this error, I only retype the package of the frame that throws the exception and done!, but of course, that is not a solution.
Exception in thread "main" java.lang.NoClassDefFoundError: xoxo/Vistas/VistaLogin$1
at xoxo.Vistas.VistaLogin.initComponents(VistaLogin.java:49)
at xoxo.Vistas.VistaLogin.<init>(VistaLogin.java:19)
at xoxo.Controlador.ControladorLogin.<clinit>(ControladorLogin.java:17)
at xoxo.princ.Principal.main(Principal.java:7)
Caused by: java.lang.ClassNotFoundException: xoxo.Vistas.VistaLogin$1
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
... 4 more
thanks for the help.

See my answer of this i am sure that is Desired Solution. and if You sure 100% that there is exist same class which is shown in error.
This is my ans
Try it that will be helpful.

Java Stackoverflow Error

Hi sometimes i encoutner this error after restarsting sun application server 8.2 domain anyone have any idea what could be the reason?
javax.servlet.ServletException at
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:255) at
javax.servlet.http.HttpServlet.service(HttpServlet.java:860) at
sun.reflect.GeneratedMethodAccessor156.invoke(Unknown
Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at
java.lang.reflect.Method.invoke(Method.java:585) at
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:249) at
java.security.AccessController.doPrivileged(Native
Method) at
javax.security.auth.Subject.doAsPrivileged(Subject.java:517) at
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:282) at
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:165) at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:257) at
org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:55) at
org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:161) at
java.security.AccessController.doPrivileged(Native
Method) at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157) at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:723) at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:484) at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:417) at
org.apache.catalina.core.ApplicationDispatcher.access$000(ApplicationDispatcher.java:80) at
org.apache.catalina.core.ApplicationDispatcher$PrivilegedForward.run(ApplicationDispatcher.java:95) at
java.security.AccessController.doPrivileged(Native
Method) at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:313) at
org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:448) at
org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:339) at
org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:282) at
org.apache.catalina.core.StandardHostValve.postInvoke(StandardHostValve.java:184) at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:552) at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:132) at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:551) at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:933) at
org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:189) at
com.sun.enterprise.web.connector.grizzly.ProcessorTask.doProcess(ProcessorTask.java:604) at
com.sun.enterprise.web.connector.grizzly.ProcessorTask.process(ProcessorTask.java:475) at
com.sun.enterprise.web.connector.grizzly.ReadTask.executeProcessorTask(ReadTask.java:371) at
com.sun.enterprise.web.connector.grizzly.ReadTask.doTask(ReadTask.java:264) at
com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:281) at
com.sun.enterprise.web.connector.grizzly.WorkerThread.run(WorkerThread.java:83)-----
Root Cause
-----java.lang.StackOverflowError at java.security.AccessController.doPrivileged(Native
Method) at
com.sun.security.auth.PolicyFile.getPermissions(PolicyFile.java:818) at
javax.security.auth.SubjectDomainCombiner$3.run(SubjectDomainCombiner.java:357) at
java.security.AccessController.doPrivileged(Native
Method) at
javax.security.auth.SubjectDomainCombiner.combineJavaxPolicy(SubjectDomainCombiner.java:353) at
javax.security.auth.SubjectDomainCombiner.combine(SubjectDomainCombiner.java:191) at
java.security.AccessControlContext.goCombiner(AccessControlContext.java:390) at
java.security.AccessControlContext.optimize(AccessControlContext.java:304) at
java.security.AccessController.checkPermission(AccessController.java:426) at
java.lang.SecurityManager.checkPermission(SecurityManager.java:532) at
java.security.Security.getProperty(Security.java:724) at
com.sun.security.auth.PolicyFile.initPolicyFile(PolicyFile.java:356) at
com.sun.security.auth.PolicyFile.init(PolicyFile.java:269) at
com.sun.security.auth.PolicyFile.getPermissions(PolicyFile.java:869) at
com.sun.security.auth.PolicyPermissions.init(PolicyFile.java:1416) at
com.sun.security.auth.PolicyPermissions.elements(PolicyFile.java:1429) at
javax.security.auth.SubjectDomainCombiner.combineJavaxPolicy(SubjectDomainCombiner.java:366) at
javax.security.auth.SubjectDomainCombiner.combine(SubjectDomainCombiner.java:191)
… ad inifinitum

Infinite recursion.
This is evident by your long, repeating backtrace. The fact that you receive a stack overflow error also supports this.

My guess is something with your security policy is triggering an infinitely recursive operation. Though there does not appear to be any application code involved, so it's hard to say for sure. Is the repeating section always bounded between calls to SubjectDomainCombiner?
One thing you can do you confirm it's truly an infinitely recursive problem is to increase the stack size. You should see the looping section expand to fill the remainder of the space when it does crash. Otherwise, you're just running low on stack... it does happen, especially when you've already got 30 or 40 frames on the stack from the application server framework itself.
The default stack size is 256k.
You can increase it by applying the JVM argument, for example -Xss2M to get 2MB of stack.
Do this just to confirm the problem, not as a permanent solution.