Develop Reference

Method showString([class java.lang.Integer, class java.lang.Integer, class java.lang.Boolean]) does not exist in PySpark

This is the snippet:
from pyspark import SparkContext
from pyspark.sql.session import SparkSession
sc = SparkContext()
spark = SparkSession(sc)
d = spark.read.format("csv").option("header", True).option("inferSchema", True).load('file.csv')
d.show()
After this runs into the error:
An error occurred while calling o163.showString. Trace:
py4j.Py4JException: Method showString([class java.lang.Integer, class java.lang.Integer, class java.lang.Boolean]) does not exist
All the other methods work well. Tried researching alot but in vain. Any lead will be highly appreciated

This is an indicator of a Spark version mismatch. Before Spark 2.3 show method took only two arguments:
def show(self, n=20, truncate=True):
since 2.3 it takes three arguments:
def show(self, n=20, truncate=True, vertical=False):
In your case Python client seems to invoke the latter one, while the JVM backend uses the older version.
Since SparkContext initialization undergone significant changes in 2.4, which would cause failure on SparkContext.__init__, you're likely using:
2.3.x Python library.
2.2.x JARs.
You can confirm that by checking versions directly from your session, Python:
sc.version
vs. JVM:
sc._jsc.version()
Problems like this, are usually a result of misconfigured PYTHONPATH (either directly, or by using pip installed PySpark on top per-existing Spark binaries) or SPARK_HOME.

On spark-shell console, enter the variable name and see the data type.
As an alternative, you can tab twice after variable named. and it will show necessary function which could be applied.
Example of a DataFrame object.
res23: org.apache.spark.sql.DataFrame = [order_id: string, book_name: string ... 1 more field]

Can javac stack frame operand verification types be disabled or ignored at class load in java 7?

The problem I am facing can be distilled down to this simple example. Compile this with Oracle java 7 jdk, and then attempt to run it with IBM jre 7 or jre 8. It fails with NoClassDefFoundError on java.lang.AbstractStringBuilder.
package org.example;
public class Example {
public static void main(String[] args) {
StringBuffer sb = new StringBuffer("got it") ;
System.out.println(" works "+sb);
StringBuilder sb2 = new StringBuilder("other") ;
sb2.append("killer") ;
sb2.length() ;
System.out.println("builder "+sb2);
System.out.println( sb == null ? sb2 : sb);
}
}
Related to http://chrononsystems.com/blog/java-7-design-flaw-leads-to-huge-backward-step-for-the-jvm the java 7 compiler adds stack frames to the byte code which include type verification for the stack operands.
Byte code for this example compiled with Oracle jdk includes:
frame_type = 255 /* full_frame */
offset_delta = 0
locals = [ class "[Ljava/lang/String;", class java/lang/StringBuffer, class java/lang/StringBuilder ]
stack = [ class java/io/PrintStream, class java/lang/AbstractStringBuilder ]
While bytecode from IBM jdk compiler is:
frame_type = 255 /* full_frame */
offset_delta = 0
locals = [ class "[Ljava/lang/String;", class java/lang/StringBuffer, class java/lang/StringBuilder ]
stack = [ class java/io/PrintStream, class java/io/Serializable ]
The stack frame from Oracle resolves to the base class AbstractStringBuilder class. That class does not exist in IBM JRE rt.jar. We use Oracle JDK in our build process & I’m reluctant to change that. This java 7 type verification seems to create non-portable bytecode.
Is there a way to disable this type verification checking at either compile time or runtime?
Thanks

Why would you try to compile and run Java in different Vendors. I see imports are different. Something might have gone wrong somewhere. Oracle JDK has StringBuilder as well
Try using java.lang.StringBuilder instead of StringBuilder and share what output you are getting.
java.lang.StringBuffer sb = new java.lang.StringBuffer("got it") ;
System.out.println(" works "+sb);
java.lang.StringBuilder sb2 = new java.lang.StringBuilder("other") ;

Thanks for the advice & help. After posting this, we found the root cause. we were using an older version of aspect-j to instrument the classes before packaging them in our application. Sorry, I never found why the aspect j instrumentation added AbstractStringBuilder to the altered class file.
We were no longer requiring aspect-j. So removing that solved the problem.
Thanks for the help

Why does java.lang.VerifyError mean?

While I've tried to use the following code snippet with the Groovy in-operator explanation the VerifyError has occured. Have you guys any idea about?
The code and console output is below.
class Hello extends ArrayList {
boolean isCase(Object val) {
return val == 66
}
static void main(args) {
def myList = new Hello()
myList << 55
assert 66 in myList
assert !myList.contains(66)
}
}
The error log:
Exception in thread "main" java.lang.VerifyError: (class: Hello, method: super$1$stream signature: ()Ljava/util/stream/Stream;) Illegal use of nonvirtual function call
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:259)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:116)
The code origin from the topic How does the Groovy in operator work?.
Update:
Groovy Version: 1.8.6 JVM: 1.6.0_45 Vendor: Sun Microsystems Inc. OS: Linux

Check this out.
It's for Java, but generally problem is, that you are using wrong library versions. The class is there, but different version than expected.
http://craftingjava.blogspot.co.uk/2012/08/3-reasons-for-javalangverfiyerror.html
Probably you have messed up Groovy or Java SDK installations.

IncompatibleClassChangeError in Java

After learning Lambda Expressions in Java, I tried to practice some simple examples. But in my first example only I am getting the following error.
Exception in thread "main" java.lang.IncompatibleClassChangeError
at java.lang.invoke.MethodHandleNatives.linkMethodHandleConstant(MethodHandleNatives.java:384)
at com.example.lambda.HelloLambda.main(HelloLambda.java:15)
Caused by: java.lang.NoSuchMethodException: no such method: java.lang.invoke.LambdaMetafactory.metaFactory(Lookup,String,MethodType,MethodHandle,MethodHandle,MethodType)CallSite/invokeStatic
at java.lang.invoke.MemberName.makeAccessException(MemberName.java:763)
at java.lang.invoke.MemberName$Factory.resolveOrFail(MemberName.java:880)
at java.lang.invoke.MethodHandles$Lookup.resolveOrFail(MethodHandles.java:1019)
at java.lang.invoke.MethodHandles$Lookup.linkMethodHandleConstant(MethodHandles.java:1284)
at java.lang.invoke.MethodHandleNatives.linkMethodHandleConstant(MethodHandleNatives.java:382)
... 1 more
Caused by: java.lang.NoSuchMethodError: java.lang.invoke.LambdaMetafactory.metaFactory(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/CallSite;
at java.lang.invoke.MethodHandleNatives.resolve(Native Method)
at java.lang.invoke.MemberName$Factory.resolve(MemberName.java:852)
at java.lang.invoke.MemberName$Factory.resolveOrFail(MemberName.java:877)
... 4 more
The error seems an error because of backward compatibility issue. But don't know how to fix this. Many answers in StackOverFlow suggested Recompilation could fix this issue. But still I am getting this error.
My code
package com.example.lambda;
public class HelloLambda {
static String firstname = "ChanSek";
static String lastname = "Nayak";
interface HelloService {
String hello();
}
public static void main(String[] args) {
HelloService helloService =
() -> {String hello="Hello " + firstname + " " + lastname;
return hello;};
System.out.println(helloService.hello());
}
}
The code compiles fine. But running gives the above mentioned error.
I am using JDK1.8.0 snapshot.

Is it possibly because of this?
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8019635
The way lambdas have been done in Java 8 has changed, very recently (07/2013), in a not backwards compatible way.
If you've somehow managed to compile with a compiler which talks PRE beta 103 lambdas, but are running with a JRE of POST 103 lambdas, you'll have problems.
(The hint for me here is that the metafactory name used to be mixed case, but in java 1.8.0 beta 103 appears to be lower case - you're searching for mixed case, but not finding it.)

Preserving parameter/argument names in compiled Java classes

When I compile something like this:
public class MyClass
{
void myMethod(String name, String options, String query, String comment)
{
...
}
}
and compile this to a class file, it seems that argument names are lost. That is, when some other Java code references MyClass and wants to call or overwrite myMethod, my IDE (currently Eclipse) seems to get this method signature from the class-file:
void myMethod(String arg0, String arg1, String arg2, String arg3);
I know that Eclipse (and possibly other IDEs too) allows me to provide a link to the source or the Javadoc (as Bishiboosh pointed out) of MyClass and can take advantage of this. But I'm curious if there is some way to tell javac to include the names into the class-file, so that users of that class can see the argument names even if they only have the class file.
Solution for classes
When I compile a class with java -g:vars, the names of parameters are included in the class file. -g:vars seems to be equivalent to Eclipse -> project properties -> Java compiler -> Add variable attributes to generated class files.
This solution was suggested by several authors, but the answer from Nick finally made me believe.
On my machine, Eclipse sometimes used this info, sometimes it didn't, which was probably my fault or a bug in Eclipse, but not a problem with the class files or the compile. Anyway, now I know that the information is definitely present.
But no solution for interfaces
While this works (kind of) fine for classes, it's not working for interfaces.
For me, the logical reason seems to be, that -g:vars only provides the names of local variables, which is what the documentation for javac also states. In the body of a method, it's parameters are very similar to local variables, thus they are covered by -g:vars. interface methods don't have bodies, so they can't have local variables.
My initial question only asked for classes, because I was not aware that there might be any difference.
Class file format
As gid pointed out, the class file format does not support storrage of parameter names. I found a section in the class file spec that descibes a data struture which should holf the parameter names of methods, but this is definitely not used when compiling interfaces.
When compiling a class, I can't tell if the mentioned data structure is used, or if Eclipse infers the parameter names from the usage of parameters inside the method body. An expert could clarify this, but it's not that relevant I think.

To preserve names in the class file for debugging purposes try project properties, Java compiler, then "Add variable attributes to generated class files" (See Eclipse Help).
Compiling the following source:
public class StackOverflowTest {
public void test(String foo, String bar) {
// blah
}
}
Is decompiled into:
// Compiled from StackOverflowTest.java (version 1.5 : 49.0, super bit)
public class StackOverflowTest {
// Method descriptor #6 ()V
// Stack: 1, Locals: 1
public StackOverflowTest();
0 aload_0 [this]
1 invokespecial java.lang.Object() [8]
4 return
Line numbers:
[pc: 0, line: 1]
Local variable table:
[pc: 0, pc: 5] local: this index: 0 type: StackOverflowTest
// Method descriptor #15 (Ljava/lang/String;Ljava/lang/String;)V
// Stack: 0, Locals: 3
public void test(java.lang.String foo, java.lang.String bar);
0 return
Line numbers:
[pc: 0, line: 4]
Local variable table:
[pc: 0, pc: 1] local: this index: 0 type: StackOverflowTest
[pc: 0, pc: 1] local: foo index: 1 type: java.lang.String
[pc: 0, pc: 1] local: bar index: 2 type: java.lang.String
}
See the parameter names are preserved in the class files.
I would suggest you look into how your source is being compiled, which version it is compiled for etc.
EDIT:
Ah, I see this is different for interfaces - they don't seem to have this information available for the debugger which I guess makes sense. I don't think there'll be a way round this, if you just want to see the parameter names when you're editing source you'll need to go the javadoc route as Nagrom_17 suggests (attach the source).

You don't specially need the source to make arg names appear in Eclipse...If you specify the Javadoc, Eclipse will display the args.

It might help to compile with debug support, which stores all names in the .class file.
though I don't know whether Eclipse takes that into account.

Eclipse will pick up the names of arguments if you include debug information in the class file: javac -g:vars should be enough.

There is no support in the class file data structure for storing the parameter names to any method, no matter what javac options you use.
In order to see the original names in an IDE you have to supply them with either the javadoc or the source.
If you have a particular need to get at them at runtime it is possible to add annotations to parameters, but you'll have to create your own as there isn't a standard set to use.
Sorry can't be more helpful
EDIT:
I stand completely corrected...the class file format does clearly have space for named parameters (JLS 4.7)
What I can't see is how the hell you can get at them using java.lang.reflect.*

You don't need a separate Javadoc file you can create 'inline' javadocs in Eclipse using a special comment with two asterisks(*) after the first slash of a multi-line comment.
example code:
public class MyClass
{
/**
* documentation of your method
*
* #param name a String describing the name
* #param options used to describe current option
* #param query
* #param comment
* #return void
*/
void myMethod(String name, String options, String query, String comment)
{
...
}
}