Develop Reference

What is the default value for spark.sql.columnNameOfCorruptRecord?

I have read the documentation but can not get spark.sql.columnNameOfCorruptRecord default value even with google searching.
The second question - how PERMISSIVE mode works when spark.sql.columnNameOfCorruptRecord is empty or null?

According to the code (19/01/2021) it's _corrupt_record:
val COLUMN_NAME_OF_CORRUPT_RECORD = buildConf("spark.sql.columnNameOfCorruptRecord")
.doc("The name of internal column for storing raw/un-parsed JSON and CSV records that fail " +
"to parse.")
.version("1.2.0")
.stringConf
.createWithDefault("_corrupt_record")
Regarding how PERMISSIVE mode works, you can see this in FailSafeParser[T]:
def parse(input: IN): Iterator[InternalRow] = {
try {
rawParser.apply(input).toIterator.map(row => toResultRow(Some(row), () => null))
} catch {
case e: BadRecordException => mode match {
case PermissiveMode =>
Iterator(toResultRow(e.partialResult(), e.record))
case DropMalformedMode =>
Iterator.empty
case FailFastMode =>
throw new SparkException("Malformed records are detected in record parsing. " +
s"Parse Mode: ${FailFastMode.name}. To process malformed records as null " +
"result, try setting the option 'mode' as 'PERMISSIVE'.", e)
}
}
private val toResultRow: (Option[InternalRow], () => UTF8String) => InternalRow = {
if (corruptFieldIndex.isDefined) {
(row, badRecord) => {
var i = 0
while (i < actualSchema.length) {
val from = actualSchema(i)
resultRow(schema.fieldIndex(from.name)) = row.map(_.get(i, from.dataType)).orNull
i += 1
}
resultRow(corruptFieldIndex.get) = badRecord()
resultRow
}
} else {
(row, _) => row.getOrElse(nullResult)
}
}
If it isn't specified, it'll fallback to the default value defined in the configuration.

Getting last similar item out of Map in Java

I have a Map that has been alphabetically sorted by converting it using TreeMap.
The Map contains both a String (installer file name) and Path (installer path on file system) for instance
Map installers;
I need to obtain the most recent installer file name. However, regex seems like it'd be too complicated.
The code I have currently to display the installers and their paths is this:
Map<String, Path> installers = findInstallers();
Set s = installers.entrySet();
Iterator it = s.iterator();
while(it.hasNext()) {
Map.Entry entry = (Map.Entry) it.next();
String installerFile = (String) entry.getKey();
Path installerPath = (Path) entry.getValue();
System.out.println(installerFile + " ==> " + installerPath.toString());
}
System.out.println("================================");
private Map<String, Path> findInstallers() {
HashMap<String, Path> installerPathMap = new HashMap<>();
try {
Path productReleasePath = Paths.get("C:", "test");
List<Path> allPaths = Files.walk(productReleasePath)
.filter(Files::isRegularFile)
.collect(Collectors.toList());
allPaths.forEach(path -> {
if (!path.toFile().getName().toLowerCase().endsWith(".log")) {
String installerFiileName = path.toFile().getName();
installerPathMap.put(installerFiileName, path);
}
});
} catch (IOException e) {
e.printStackTrace();
}
return new TreeMap<>(installerPathMap);
}
This is a sample output:
Client_1.exe ==> C:\test\build_1\Win32\Client_1.exe
Client_5.exe ==> C:\test\build_5\Win32\Client_5.exe
Client_6.exe ==> C:\test\build_6\Win32\Client_6.exe
Server_1.exe ==> C:\test\build_1\Win64\Server_1.exe
Server_2.exe ==> C:\test\build_2\Win64\Server_2.exe
Server_Linux_1.tar.gz ==> C:\test\build_1\Linux32\Server_Linux_1.tar.gz
Server_Linux_2.tar.gz ==> C:\test\build_2\Linux32\Server_Linux_1.tar.gz
================================
I need to shorten my Map to only contain the highest key and it's value pair, so the output is similar to this:
Client_6.exe ==> C:\test\build_6\Win32\Client_6.exe
Server_2.exe ==> C:\test\build_2\Win64\Server_2.exe
Server_Linux_2.tar.gz ==> C:\test\build_2\Linux32\Server_Linux_1.tar.gz
================================
Any help would be greatly appreciated.

If you add the paths to a map using the root of the installer name as a key (i.e. the part before the underscore), and discard the lowest version when there is a key collision, you'll get what you want.
Note that sorting the names alphabetically won't work because version 9 will sort after 10, so you'll have to extract the version and do a numeric comparison.
I'm not certain of your naming convention, but the helper functions in the following example should be easy enough to modify if my assumptions aren't correct.
public class InstallerList {
public static void main(String[] args) throws IOException {
Path productReleasePath = Paths.get("C:", "test");
Collection<Path> installers = Files.walk(productReleasePath)
.filter(Files::isRegularFile)
.filter(p -> !p.getFileName().toString().endsWith(".log"))
// Collect files with the highest version
.collect(Collectors.toMap(
// Key is installer name *without* version
InstallerList::extractName,
// Value mapper; identity mapping to the path
p -> p,
// Select newest version when there is a collision
InstallerList::newer
))
.values();
for (Path path : installers) {
System.out.println(path.getFileName() + " ==> " + path);
}
}
// Extract the root name of an installer from a path (up to but not including the last '_')
public static String extractName(Path path) {
String fileName = path.getFileName().toString();
int i = fileName.lastIndexOf('_');
if (i < 0) {
throw new IllegalArgumentException(fileName);
}
return fileName.substring(0, i);
}
// Return the path with the highest version number
public static Path newer(Path p1, Path p2) {
return extractVersion(p1) > extractVersion(p2) ? p1 : p2;
}
// Extract a version number from a path (number between the last '_' and the following '.')
private static int extractVersion(Path path) {
String fileName = path.getFileName().toString();
int i = fileName.lastIndexOf('_');
if (i < 0) {
throw new IllegalArgumentException(fileName);
}
int j = fileName.indexOf('.', i);
if (j < 0) {
throw new IllegalArgumentException(fileName);
}
return Integer.parseInt(fileName.substring(i + 1, j));
}
}

How do I query existing applications on JBoss AS 7?

I'm working on automatically deploying to a remote JBoss AS 7.1.1 server from a Jenkins build server as part of a build pipeline and have a small jar file that I call from ant (based on this).
My question is how do I find out if an application is already installed? Doing a deploy plan will fail if the application is already deployed(I could catch the exception that's thrown but that's not great).

You can read the resource before doing the deploy. From there you can either redploy it or do nothing.
Here is an example that would work on a standalone server.
private boolean exists(final ModelControllerClient client, final String deploymentName) {
final ModelNode op = new ModelNode();
op.get(OP).set("read-children-names");
op.get("child-type").set(ClientConstants.DEPLOYMENT);
final ModelNode result;
try {
result = client.execute(op);
// Check to make sure there is an outcome
if (result.hasDefined(ClientConstants.OUTCOME)) {
if (result.get(ClientConstants.OUTCOME).asString().equals(ClientConstants.SUCCESS)) {
final List<ModelNode> deployments = (result.hasDefined(ClientConstants.RESULT) ? result.get(ClientConstants.RESULT).asList() : Collections.<ModelNode>emptyList());
for (ModelNode n : deployments) {
if (n.asString().equals(deploymentName)) {
return true;
}
}
} else if (result.get(ClientConstants.OUTCOME).asString().equals(ClientConstants.FAILED)) {
throw new IllegalStateException(String.format("A failure occurred when checking existing deployments. Error: %s",
(result.hasDefined(ClientConstants.FAILURE_DESCRIPTION) ? result.get(ClientConstants.FAILURE_DESCRIPTION).asString() : "Unknown")));
}
} else {
throw new IllegalStateException(String.format("An unexpected response was found checking the deployment. Result: %s", result));
}
} catch (IOException e) {
throw new IllegalStateException(String.format("Could not execute operation '%s'", op), e);
}
return false;
}
If you're using maven, there is a maven plugin you could use too.

An alternative:
ModelNode res = AS7CliUtils.executeRequest("/deployment=* /:read-resource", ctx.getAS7Client() );
{
"outcome" => "success",
"result" => [{
"address" => [("deployment" => "jboss-as-wicket-ear-ear.ear")],
"outcome" => "success",
"result" => {
"content" => [{"hash" => bytes { ... }}],
"enabled" => true,
"name" => "jboss-as-wicket-ear-ear.ear",
"persistent" => true,
"runtime-name" => "jboss-as-wicket-ear-ear.ear",
"subdeployment" => {
"jboss-as-wicket-ear-ejb.jar" => undefined,
"jboss-as-wicket-ear-war.war" => undefined
},
"subsystem" => {"datasources" => undefined}
}
}]
}
JBoss AS CLI client lib contains some API for that, can't find it right now.
This is a primitive implementation of query parsing (doesn't support nested values and doesn't care about escaping etc.).
/**
* Parse CLI command into a ModelNode - /foo=a/bar=b/:operation(param=value,...) .
*
* TODO: Support nested params.
*/
public static ModelNode parseCommand( String command ) {
return parseCommand( command, true );
}
public static ModelNode parseCommand( String command, boolean needOp ) {
String[] parts = StringUtils.split( command, ':' );
if( needOp && parts.length < 2 ) throw new IllegalArgumentException("Missing CLI command operation: " + command);
String addr = parts[0];
ModelNode query = new ModelNode();
// Addr
String[] partsAddr = StringUtils.split( addr, '/' );
for( String segment : partsAddr ) {
String[] partsSegment = StringUtils.split( segment, "=", 2);
if( partsSegment.length != 2 ) throw new IllegalArgumentException("Wrong addr segment format - need '=': " + command);
query.get(ClientConstants.OP_ADDR).add( partsSegment[0], partsSegment[1] );
}
// No op?
if( parts.length < 2 ) return query;
// Op
String[] partsOp = StringUtils.split( parts[1], '(' );
String opName = partsOp[0];
query.get(ClientConstants.OP).set(opName);
// Op args
if( partsOp.length > 1 ){
String args = StringUtils.removeEnd( partsOp[1], ")" );
for( String arg : args.split(",") ) {
String[] partsArg = arg.split("=", 2);
query.get(partsArg[0]).set( unquote( partsArg[1] ) );
}
}
return query;
}// parseCommand()

Pretty print for a groovy ConfigObject?

I have this groovy program that creates a groovy configuration file, using a ConfigObject. Once the ConfigObject is set up, it is written to a file using:
myFile.withWriter {writer -> myConfigObject.writeTo(writer)}
This results in each property of the ConfigObject being written on a single line. So for instance a map will be printed as:
graphs=[["type":"std", "host":"localhost", "name":"cpurawlinux"], ["type":"std", "host":"localhost", "name":"memory"], ["type":"std", "host":"localhost", "name":"udp"] ... ]
which is quite unreadable if someone has to take a look at it.
Is there a way to get a more friendly output? Something like that would be great:
graphs=[
["type":"std", "host":"localhost", "name":"cpurawlinux"],
["type":"std", "host":"localhost", "name":"memory"],
["type":"std", "host":"localhost", "name":"udp"]
...
]
I know I could create my own writeTo, but isn't there already something in Groovy for that?

Unfortunately, you'll need to write your own writeTo as you say.
If you have a config file with structure like:
graphs {
a=["type":"std", "host":"localhost", "name":"cpurawlinux"]
b=["type":"std", "host":"localhost", "name":"memory"]
}
Then writeTo will write it out with structure, but if your config file is just a big old list of things, it will write it out as a big old list

if it helps anyone, i had the same question and wrote this...not pretty (ha), but works:
def prettyPrint(properties, level=1, stringBuilder = new StringBuilder()) {
return properties.inject(stringBuilder) { sb, name, value ->
sb.append("\n").append("\t" * level).append(name)
if (!(value instanceof Map) && !(value instanceof List)) {
return sb.append("=").append(value)
} else {
return prettyPrint(properties.getProperty(name), level+1, sb)
}
}
}

Based on mike's answer above:
def prettyPrint
prettyPrint = {obj, level = 0, sb = new StringBuilder() ->
def indent = { lev -> sb.append(" " * lev) }
if(obj instanceof Map){
sb.append("{\n")
obj.each{ name, value ->
if(name.contains('.')) return // skip keys like "a.b.c", which are redundant
indent(level+1).append(name)
(value instanceof Map) ? sb.append(" ") : sb.append(" = ")
prettyPrint(value, level+1, sb)
sb.append("\n")
}
indent(level).append("}")
}
else if(obj instanceof List){
sb.append("[\n")
obj.each{ value ->
indent(level+1)
prettyPrint(value, level+1, sb).append(",")
sb.append("\n")
}
indent(level).append("]")
}
else if(obj instanceof String){
sb.append('"').append(obj).append('"')
}
else {
sb.append(obj)
}
}
For an input like:
{
grails {
scaffolding {
templates.domainSuffix = "Instance"
}
enable {
native2ascii = true
blah = [ 1, 2, 3 ]
}
mime.disable.accept.header.userAgents = [ "WebKit", "Presto", "Trident" ]
}
}
Produces:
{
grails {
scaffolding {
templates {
domainSuffix = "Instance"
}
}
enable {
native2ascii = true
blah = [
1,
2,
3,
]
}
mime {
disable {
accept {
header {
userAgents = [
"WebKit",
"Presto",
"Trident",
]
}
}
}
}
}
}

Why does my fork/join deadlock?

Consider the following snipped of code, which calculates the size of all paths given.
def pathSizes = []
paths.each { rootPath ->
pathSizes.addAll(
withPool { pool ->
runForkJoin(rootPath) { path ->
def headSizes = [:]
println path
def lines = ["ls", "-al", path].execute().text.readLines()
(0..<3).each { lines.remove(0) }
lines.each { line ->
def fields = line.split(/\s+/)
if (fields[0] =~ /^d/)
forkOffChild("$path/${fields.last()}")
else {
def userName = fields[2]
def fileSize = fields[4] as long
if (headSizes[userName] == null)
headSizes[userName] = fileSize
else
headSizes[userName] += fileSize
}
}
quietlyJoin()
System.gc()
def shallowSizes =
headSizes.collectEntries { userName, fileSize ->
def childResult =
childrenResults.sum {
it.shallowSizes[userName] ? it.shallowSizes[userName] : 0
} ?: 0
return [userName, fileSize + childResult]
}
def deepSizes =
childrenResults.sum { it.deepSizes ?: [] } +
shallowSizes.collect { userName, fileSize ->
[userName: userName, path: path, fileSize: fileSize]
}
return [shallowSizes: shallowSizes, deepSizes: deepSizes]
}.deepSizes
})
}
Why does this snippet of code deadlock? There are no interactions between threads except possibly with the system call and other parts of the Java framework. If the system calls are the problem, then how can I fix it, without removing the system calls (they are slow, hence the need to parallelize)?