Develop Reference

Hive UDF in Java fails when creating a table

What is the difference between those two queries:
SELECT my_fun(col_name) FROM my_table;
and
CREATE TABLE new_table AS SELECT my_fun(col_name) FROM my_table;
Where my_fun is a java UDF.
I'm asking, because when I create new table (second query) I receive a java error.
Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator initialization failed
...
Caused by: org.apache.hadoop.hive.ql.exec.UDFArgumentException: Unable to instantiate UDF implementation class com.company_name.examples.ExampleUDF: java.lang.NullPointerException
I found that the source of error is line in my java file:
encoded = Files.readAllBytes(Paths.get(configPath));
But the question is why it works when table is not created and fails if table is created?

The problem might be with the way you read the file. Try to pass the file path as the second argument in the UDF, then read as follows
private BufferedReader getReaderFor(String filePath) throws HiveException {
try {
Path fullFilePath = FileSystems.getDefault().getPath(filePath);
Path fileName = fullFilePath.getFileName();
if (Files.exists(fileName)) {
return Files.newBufferedReader(fileName, Charset.defaultCharset());
}
else
if (Files.exists(fullFilePath)) {
return Files.newBufferedReader(fullFilePath, Charset.defaultCharset());
}
else {
throw new HiveException("Could not find \"" + fileName + "\" or \"" + fullFilePath + "\" in inersect_file() UDF.");
}
}
catch(IOException exception) {
throw new HiveException(exception);
}
}
private void loadFromFile(String filePath) throws HiveException {
set = new HashSet<String>();
try (BufferedReader reader = getReaderFor(filePath)) {
String line;
while((line = reader.readLine()) != null) {
set.add(line);
}
} catch (IOException e) {
throw new HiveException(e);
}
}
The full code for different generic UDF that utilizes file reader can be found here

I think there are several points unclear, so this answer is based on assumptions.
First of all, it is important to understand that hive currently optimize several simple queries and depending on the size of your data, the query that is working for you SELECT my_fun(col_name) FROM my_table; is most likely running locally from the client where you are executing the job, that is why you UDF can access your config file locally available, this "execution mode" is because the size of your data. CTAS trigger a job independent on the input data, this job runs distributed in the cluster where each worker fail accessing your config file.
It looks like you are trying to read your configuration file from the local file system, not from the HDSFS Files.readAllBytes(Paths.get(configPath)), this means that your configuration has to either be replicated in all the worker nodes or be added previously to the distributed cache (you can use add file from this, doc here. You can find another questions here about accessing files from the distributed cache from UDFs.
One additional problem is that you are passing the location of your config file through an environment variable which is not propagated to worker nodes as part of your hive job. You should pass this configuration as a hive config, there is an answer for accessing Hive Config from UDF here assuming that you are extending GenericUDF.

Has anyone done a FitNesse TEST that queries to REAL Database? I can't make Fitnesse connect successfully

I'm asking because ALL examples I find in Google, are the same from the Fitnesse tutorial: a very simple query to a list or array in memory, NOT A REAL Database.
Yes, Fixtures never have to deal with that, but how am I supposed to test my fixtures if I can't even make the connection to the DB in a simulation of an "API"?
What I'm trying to simulate is the call from a FitNesse Fixture to query in Java into a PostgreSQL database/table. In this simple example I'm trying to obtain, at least one column from one row, in one table. When I execute the code, it runs perfectly by it's own. The problem is when trying to execute from Fitnesse through the fixture. It always fails with a ClassNotFoundException, when calling the JDBC driver. This doesn't happen by running the code by it's own.
Here is the code that does the query:
package queriespackage;
import java.sql.*;
public class testQuery01 {
public static Connection openDBConnection(){
Connection connectionString = null;
try {
String dbhost = "SOMEURL";//Redacted
String port = "SOMEPORT";//Redacted
String dbname = "THEDBNAME";//Redacted
String username = "SOMEUSER";//Redacted
String password = "SOMEPASSWORD";//Redacted
String driverJDBC = "org.postgresql.Driver";
Class.forName(driverJDBC);
connectionString = DriverManager.getConnection("jdbc:postgresql://" + dbhost + ":" + port + "/" + dbname,username,password);
connectionString.setAutoCommit(false);
} catch (SQLException e) {
e.printStackTrace(System.err);
System.exit(0);
} catch (Exception e) {
System.err.println( e.getClass().getName()+": "+ e.getMessage() );
System.exit(0);
};
return connectionString;
};
public static ResultSet executeQuery(Connection connectionString, int intAccountId) throws SQLException{
Statement querySession = connectionString.createStatement();
//The query string
String queryString = "SELECT DISTINCT "
+ "account_search.account_id,"
+ "account_search.account_name"
+ " FROM account_search "
+ " WHERE"
+ " account_search.account_id = "+ intAccountId
+ "LIMIT 1";
ResultSet queryResult = querySession.executeQuery(queryString);
return queryResult;
};
public static String processQueryResult(ResultSet queryResult) throws SQLException{
String strQueryValueReturned = null;
while (queryResult.next()) {
strQueryValueReturned = queryResult.getString("account_name");
};
return strQueryValueReturned;
};
public static boolean closeDBConnection(Connection connectionString){
try {
if(connectionString!=null){
connectionString.close();
}
} catch (SQLException e) {
System.err.println( e.getClass().getName()+": "+ e.getMessage() );
System.exit(0);
};
return true;
};
public static String testQuery(int intAccountId) throws SQLException, ClassNotFoundException{
boolean bolConnectionStatus = false;
String strValueReturned = null;
Connection connectionString = openDBConnection();
if(connectionString != null){
ResultSet qryQueryResult = executeQuery(connectionString, intAccountId);
strValueReturned = processQueryResult(qryQueryResult);
bolConnectionStatus = closeDBConnection(connectionString);
if(!bolConnectionStatus){
System.exit(0);
}
}else{
System.exit(0);
};
return strValueReturned;
};
};
If I add a Main method to that code, passing it the argument value for "intAccountId", it successfully returns the name of the account "account_name", just as expected.
Now here's the Fixture that should be called by the FitNesse test:
package fixturespackage;
import java.sql.SQLException;
import queriespackage.testQuery01;
public class testFixture01{
private int Int_AccountId;
//Fixture Constructor (setter)
public testFixture01(int Int_AccountId){
this.Int_AccountId = Int_AccountId;
};
public String query() throws SQLException, ClassNotFoundException{
return testQuery01.testQuery(Int_AccountId);
};
};
Just as the FitNesse guide says, there must be a "query" method, that does the actual call to the interface in the DB. I had to add a constructor instead of the "setter", because FitNesse actually demands it: "Could not invoke constructor for fixturepackage.testFixture01"
Here's the FitNesse page:
!***> System Variables
!define TEST_SYSTEM {slim}
!path C:\FitnessTest\bin
*!
|Query: fixturespackage.testFixture01|21 |
|Str_AccountName |
|SomeName |
Here's a Screenshot of my BuildPath, so you can see I have the JDBC Library for Java 8, JDK 1.8, JRE 1.8... and the "org.postgresql.Driver.class" is included in the project.
This is the error I receive, when running from FitNesse:
This is the error I get, when debugging the line where FitNesse failed by using Inspect tool:
... and YES, I also tried by hard coding the name of the JDBC:
I have searched a lot for a REAL LIFE example, both here, the FitNesse Guide and Google.
The FitNesse Guide might be extensive, but let's be sincere, it's full of "dirty word here", unrealistic and incomplete examples and missing a lot of information.
So, asking again, has anyone done a REAL LIFE test making queries, using FitNesse, that could help me find out what am I doing wrong?

I have to admit I've only done limited database tests with FitNesse, but I have used them (to query DB2).
I did not use query tables (or wrote my own fixtures to query), but instead used jdbcslim in combination with script tables and scenario's.
That fact that the driver class cannot be found suggests that although the jar is present on the classpath in your IDE it is not available when FitNesse is running your fixture code.
I notice you specify the classpath as a single directory in the wiki. In Java that means that all class files should be in that directory (as .class files, in the right subdirectory for their defined package). It will not pick up any jars (or zips) in that directory. Did you unpack your database driver's jar to that directory? If not, you need to add a !path line pointing to the jar (so the entire path including the filename) with the database driver.
Now listing every jar you need can quickly become cumbersome, so you can also use wildcards. I tend to copy all the jars I need to a single directory, that also contains my fixture .class files, and add a single !path line loading all jars in that directory.
So if you also copied your database driver to the directory in you question you could ensure it, and your own fixture, to be available via
!path C:\FitnessTest\bin
!path C:\FitnessTest\bin\*.jar

Alt-codes only work in Java string when run within Netbeans

I have a small java program that reads a given file with data and converts it to a csv file.
I've been trying to use the arrow symbols: ↑, ↓, → and ← (Alt+24 to 27) but unless the program is run from within Netbeans (Using F6), they will always come out as '?' in the resulting csv file.
I have tried using the unicodes, eg "\u2190" but it makes no difference.
Anyone know why this is happening?
As requested, here is a sample code that gives the same issue. This wont work when run using the .jar file, just creating a csv file containing '?', however running from within Netbeans works.
import java.io.FileNotFoundException;
import java.io.PrintWriter;
public class Sample {
String fileOutName = "testresult.csv";
/**
* #param args the command line arguments
*/
public static void main(String[] args) throws FileNotFoundException {
Sample test = new Sample();
test.saveTheArrow();
}
public void saveTheArrow() {
try (PrintWriter outputStream = new PrintWriter(fileOutName)) {
outputStream.print("←");
outputStream.close();
}
catch (FileNotFoundException e) {
// Do nothing
}
}
}

new PrintWriter(fileOutName) uses the default charset of the JVM - you may have different defaults in Netbeans and in the console.
Google Sheet uses UTF_8 according to this thread so it would make sense to save your file using that character set:
Files.write(Paths.get("testresult.csv"), "←".getBytes(UTF_8));

Using the "<-" character in your editor is for sure not the desired byte 0x27.
Use
outputStream.print( new String( new byte[] { 0x27}, StandardCharsets.US_ASCII);

Elevate Java application while running

A nasty problem popped out with my software. I am making a program that interacts with another existing software (a game). User has reported that he runs the game with administrator privileges and under that circumstances, my program stops working for him.
Short investigation revealed that some people really need to run the game under administrator account and some don't. It would be great if my program would be able to detect this and warn user if the game is running under administrator account:
If the user clicks "Elevate", I'd like to ask windows to elevate the java.exe running my jar file and invoke the typical UAC dialog.
Obviously, this time the question would not be about java updater but JRE
My question is: Is this possible? Can windows elevate my java.exe instance's privilege? Does java have a way to do it? Or can I use command line command?
I want to avoid restarting the program (though it wouldn't probably be such a big deal).
Edit:
If you look in the comments, you'll see that there's no avoiding the restart of an application - process can only start elevated, not become elevated. This kinda shifts the question, unfortunately. Basically, it now sounds more like: "How to restart my application with admin rights?". Unless, of course, there's a trick like two java.exe sharing one jar...

If still of interest: In Windows 7 my JavaElevator works. It elevates a running Java process when used in the main method of the Java application. Simply add -elevate as last program parameter and use the elevator in the main method.
The elevator class:
package test;
import com.sun.jna.Native;
import com.sun.jna.platform.win32.Kernel32;
import com.sun.jna.platform.win32.Kernel32Util;
import com.sun.jna.platform.win32.ShellAPI;
import com.sun.jna.platform.win32.WinDef;
/**
* Elevates a Java process to administrator rights if requested.
*/
public class JavaElevator {
/** The program argument indicating the need of being elevated */
private static final String ELEVATE_ARG = "-elevate";
/**
* If requested, elevates the Java process started with the given arguments to administrator level.
*
* #param args The Java program arguments
* #return The cleaned program arguments
*/
public static String[] elevate(String[] args) {
String[] result = args;
// Check for elevation marker.
boolean elevate = false;
if (args.length > 0) {
elevate = args[args.length - 1].equals(ELEVATE_ARG);
}
if (elevate) {
// Get the command and remove the elevation marker.
String command = System.getProperty("sun.java.command");
command = command.replace(ELEVATE_ARG, "");
// Get class path and default java home.
String classPath = System.getProperty("java.class.path");
String javaHome = System.getProperty("java.home");
String vm = javaHome + "\\bin\\java.exe";
// Check for alternate VM for elevation. Full path to the VM may be passed with: -Delevation.vm=...
if (System.getProperties().contains("elevation.vm")) {
vm = System.getProperty("elevation.vm");
}
String parameters = "-cp " + classPath;
parameters += " " + command;
Shell32.INSTANCE.ShellExecute(null, "runas", vm, parameters, null, 0);
int lastError = Kernel32.INSTANCE.GetLastError();
if (lastError != 0) {
String errorMessage = Kernel32Util.formatMessageFromLastErrorCode(lastError);
errorMessage += "\n vm: " + vm;
errorMessage += "\n parameters: " + parameters;
throw new IllegalStateException("Error performing elevation: " + lastError + ": " + errorMessage);
}
System.exit(0);
}
return result;
}
}
Usage in the main method of the Java application:
public static void main(String[] args) {
String[] args1 = JavaElevator.elevate(args);
if (args1.length > 0) {
// Continue as intended.
...
I know, this is a very basic implementation - sufficient for one of my daily hiccups: Starting an elevated process from Eclipse. But maybe it points someone in some dicrection...

As has been pointed in comments, sadly the Java (or any other process) cannot be elevated while running. While in the case of JWM, it could be theoretically possible to move whole program context from normal user java.exe to elevated one, I don't think it's possible. I hope some day someone will come and tell me I'm wrong.
Surprisingly, even with restart in place, this was a tricky task that took me a while to figure out.
The non java part
First, how do we exactly run a program elevated from command line? There's an answer and you can see it's not simple. But we can break it to this VBS script:
Set UAC = CreateObject("Shell.Application")
UAC.ShellExecute "program name", "command line parameters", "working directory", "runas", 1
Soon, it also turns out that we won't have any success running java.exe from VBS script. In the end, I decided to run a helper batch file. Finally, here (answer to question in the last link) we have a complete set of two scripts which really run the given .jar file elevated. Here's improved version that allows quick testing by drag'n'dropping the Jar file on it:
' Require first command line parameter
if WScript.Arguments.Count = 0 then
MsgBox("Jar file name required.")
WScript.Quit 1
end if
' Get the script location, the directorry where it's running
Set objShell = CreateObject("Wscript.Shell")
strPath = Wscript.ScriptFullName
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objFile = objFSO.GetFile(strPath)
strFolder = objFSO.GetParentFolderName(objFile)
'MsgBox(strFolder)
' Create the object that serves as runnable something
Set UAC = CreateObject("Shell.Application")
' Args:
' path to executable to run
' command line parameters - first parameter of this file, which is the jar file name
' working directory (this doesn't work but I use it nevertheless)
' runas command which invokes elevation
' 0 means do not show the window. Normally, you show the window, but not this console window
' which just blinks and disappears anyway
UAC.ShellExecute "run-normally.bat", WScript.Arguments(0), strFolder, "runas", 0
WScript.Quit 0
The Java part
Java part is more straightforward. What we need to do is to open new process and execute the prepared scripts in it.
/**
* Start this very jar file elevated on Windows. It is strongly recommended to close any existing IO
* before calling this method and avoid writing anything more to files. The new instance of this same
* program will be started and simultaneous write/write or read/write would cause errors.
* #throws FileNotFoundException if the helper vbs script was not found
* #throws IOException if there was another failure inboking VBS script
*/
public void StartWithAdminRights() throws FileNotFoundException, IOException {
//The path to the helper script. This scripts takes 1 argument which is a Jar file full path
File runAsAdmin = new File("run-as-admin.vbs");;
//Our
String jarPath;
//System.out.println("Current relative path is: " + s);
try {
jarPath = "\""+new File(Main.class.getProtectionDomain().getCodeSource().getLocation().toURI().getPath()).getAbsolutePath()+"\"";
} catch (URISyntaxException ex) {
throw new FileNotFoundException("Could not fetch the path to the current jar file. Got this URISyntax exception:"+ex);
}
//If the jar path was created but doesn't contain .jar, we're (most likely) not running from jar
//typically this happens when running the program from IDE
//These 4 lines just serve as a fallback in testing, should be deleted in production
//code and replaced with another FileNotFoundException
if(!jarPath.contains(".jar")) {
Path currentRelativePath = Paths.get("");
jarPath = "\""+currentRelativePath.toAbsolutePath().toString()+"\\AutoClient.jar\"";
}
//Now we check if the path to vbs script exists, if it does we execute it
if(runAsAdmin.exists()) {
String command = "cscript \""+runAsAdmin.getAbsolutePath()+"\" "+jarPath;
System.out.println("Executing '"+command+"'");
//Note that .exec is asynchronous
//After it starts, you must terminate your program ASAP, or you'll have 2 instances running
Runtime.getRuntime().exec(command);
}
else
throw new FileNotFoundException("The VBSScript used for elevation not found at "+runAsAdmin.getAbsolutePath());
}

This is my version. It creates a VBScript script, then executes it. This only works if the program that is being run is in a jar file, so you will have to run your IDE as administrator to actually test your program.
public static void relaunchAsAdmin() throws IOException {
relaunchAsAdmin(ThisClass.class); //Change ThisClass to the class that this method is in
}
public static void relaunchAsAdmin(Class<?> clazz) throws IOException {
if(isCurrentProcessElevated()) {
return;
}
final String dir = System.getProperty("java.io.tmpdir");
final File script = new File(dir, "relaunchAsAdmin" + System.nanoTime() +
".vbs");
try {
script.createNewFile();
OutputStreamWriter osw = new OutputStreamWriter(new FileOutputStream(script));
osw.append("Set s=CreateObject(\"Shell.Application\")" + ln + "s.ShellExecute \"" +
System.getProperty("java.home") + "\\bin\\java.exe" + "\",\"-jar \"\"" +
new File(clazz.getProtectionDomain().getCodeSource(
).getLocation().toURI()).getAbsolutePath() + "\"\"\",,\"runas\",0" +
ln + "x=createObject(\"scripting.fileSystemObject\").deleteFile(" +
"WScript.scriptfullname)");
osw.close();
if(System.getenv("processor_architecture").equals("x86")) {
Runtime.getRuntime().exec("C:\\Windows\\System32\\wscript.exe \"" +
script.getAbsolutePath() + "\"");
} else {
Runtime.getRuntime().exec("C:\\Windows\\SysWoW64\\wscript.exe \"" +
script.getAbsolutePath() + "\"");
}
} catch(URISyntaxException e) {
e.printStackTrace();
}
Runtime.getRuntime().exit(0);
}
Note that it is a bit messy. I have been using this method before, so it has been line wrapped to 100 characters (except the comment I wrote for this answer). The
isCurrentProcessElevated()
method will have to be implemented in one way or another. You could try using JNI, or you could use a pure Java method, such as writing in the Program Files or System32 directory and seeing if it failed.
Obviously, this solution will only work on Windows. I never needed to elevate on Linux or Mac systems (mainly because I don't have any Mac systems, and I don't use Linux - I just play with it).

Android file copying using Runtime.exec() not working

I am trying to copy a file 'project.jpg' from my /sdcard to /sdcard/temp/ folder, but for some reason , the file isn't getting copied. I am testing using a virtual device and have transferred the file 'project.jpg' via the adb shell. The function used to copy the file is,
public void $copyFile()
{
try
{
cpSrc = escapePath(this.cpSrc);
cpDest = escapePath(this.cpDest);
Log.d("$copyFile()","cpSrc = "+cpSrc);
Log.d("$copyFile()","cpDest = "+cpDest);
String destination = getFilename(cpDest,extractFilename(cpSrc));
Runtime.getRuntime().exec("dd in="+cpSrc+" of="+destination);
Log.d("$copyFile()","executed command : 'dd in="+cpSrc+" of="+destination+"'");
displayToast("File Copied Sucessfully.");
clearAllModes();
return;
}
catch(Exception e)
{
displayToast("$copyFile Error : "+e);
this.clearCopyBuffer();
clearAllModes();
}
}
where escapePath() is used to escape space characters(if any) in the given paths. I got the debug Logs as follows,
cpSrc = /sdcard/project.jpg
cpDest = /sdcard/temp
executed command : 'dd in=/sdcard/project.jpg of=/sdcard/temp/project.jpg
Could anyone point the error in the code,
BTW suggestions for other ways of coping files/folders ? it would be helpful as i am trying my hand at a file manager.