Develop Reference

What is the default value for spark.sql.columnNameOfCorruptRecord?

I have read the documentation but can not get spark.sql.columnNameOfCorruptRecord default value even with google searching.
The second question - how PERMISSIVE mode works when spark.sql.columnNameOfCorruptRecord is empty or null?

According to the code (19/01/2021) it's _corrupt_record:
val COLUMN_NAME_OF_CORRUPT_RECORD = buildConf("spark.sql.columnNameOfCorruptRecord")
.doc("The name of internal column for storing raw/un-parsed JSON and CSV records that fail " +
"to parse.")
.version("1.2.0")
.stringConf
.createWithDefault("_corrupt_record")
Regarding how PERMISSIVE mode works, you can see this in FailSafeParser[T]:
def parse(input: IN): Iterator[InternalRow] = {
try {
rawParser.apply(input).toIterator.map(row => toResultRow(Some(row), () => null))
} catch {
case e: BadRecordException => mode match {
case PermissiveMode =>
Iterator(toResultRow(e.partialResult(), e.record))
case DropMalformedMode =>
Iterator.empty
case FailFastMode =>
throw new SparkException("Malformed records are detected in record parsing. " +
s"Parse Mode: ${FailFastMode.name}. To process malformed records as null " +
"result, try setting the option 'mode' as 'PERMISSIVE'.", e)
}
}
private val toResultRow: (Option[InternalRow], () => UTF8String) => InternalRow = {
if (corruptFieldIndex.isDefined) {
(row, badRecord) => {
var i = 0
while (i < actualSchema.length) {
val from = actualSchema(i)
resultRow(schema.fieldIndex(from.name)) = row.map(_.get(i, from.dataType)).orNull
i += 1
}
resultRow(corruptFieldIndex.get) = badRecord()
resultRow
}
} else {
(row, _) => row.getOrElse(nullResult)
}
}
If it isn't specified, it'll fallback to the default value defined in the configuration.

BungeeCord - Detected an authenticated player in offline mode

I've been trying for a few days how to detect if the account of a player is authenticated Mojang while he is in OfflineMode.
Why do I want to do that?
Currently, I have a basic management system that consists of checking if the player's nickname exists in the mojang database, if there is setOnlineMode set to true otherwise it is set to false.
The system allows to display the skin of the player and his UUID but the problem is that if the player considered offline purchases a Premium account with the same pseudonym, he does not have his skin or his real UUID because the setOnlineMode is set to false to prevent the loss of its progress.
My goal is to make a system that detects that an offline user has just logged in with an authenticated minecraft account so that the server can offer him an automatic transfer of his progress to his new authentic UUID.
I did some research is try as for example here, i removed the onlinemode condition to allow checking if the player was authenticated then here I deleted the disconnect if the player was not valid. Which gave me a wonderful mistake.
13:13:31 [GRAVE] [Arbi13_] -> UpstreamBridge - encountered exception io.netty.handler.codec.EncoderException: java.lang.IllegalArgumentException: Cannot get ID for packet class net.md_5.bungee.protocol.packet.SetCompression in phase GAME with direction TO_CLIENT
at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:125)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:801)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814)
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794)
at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1066)
at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:305)
at net.md_5.bungee.netty.ChannelWrapper.write(ChannelWrapper.java:60)
at net.md_5.bungee.UserConnection$1.sendPacket(UserConnection.java:148)
at net.md_5.bungee.UserConnection.setCompressionThreshold(UserConnection.java:697)
at net.md_5.bungee.connection.InitialHandler$6$1.run(InitialHandler.java:523)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IllegalArgumentException: Cannot get ID for packet class net.md_5.bungee.protocol.packet.SetCompression in phase GAME with direction TO_CLIENT
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:399)
at net.md_5.bungee.protocol.Protocol$DirectionData.getId(Protocol.java:462)
at net.md_5.bungee.protocol.MinecraftEncoder.encode(MinecraftEncoder.java:23)
at net.md_5.bungee.protocol.MinecraftEncoder.encode(MinecraftEncoder.java:9)
at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:107)
... 15 more
#Override
public void handle(LoginRequest loginRequest) throws Exception
{
Preconditions.checkState( thisState == State.USERNAME, "Not expecting USERNAME" );
this.loginRequest = loginRequest;
if ( getName().contains( "." ) )
{
disconnect( bungee.getTranslation( "name_invalid" ) );
return;
}
if ( getName().length() > 16 )
{
disconnect( bungee.getTranslation( "name_too_long" ) );
return;
}
int limit = BungeeCord.getInstance().config.getPlayerLimit();
if ( limit > 0 && bungee.getOnlineCount() > limit )
{
disconnect( bungee.getTranslation( "proxy_full" ) );
return;
}
// If offline mode and they are already on, don't allow connect
// We can just check by UUID here as names are based on UUID
if ( !isOnlineMode() && bungee.getPlayer( getUniqueId() ) != null )
{
disconnect( bungee.getTranslation( "already_connected_proxy" ) );
return;
}
Callback<PreLoginEvent> callback = new Callback<PreLoginEvent>()
{
#Override
public void done(PreLoginEvent result, Throwable error)
{
if ( result.isCancelled() )
{
disconnect( result.getCancelReasonComponents() );
return;
}
if ( ch.isClosed() )
{
return;
}
unsafe().sendPacket( request = EncryptionUtil.encryptRequest() );
thisState = State.ENCRYPT;
}
};
// fire pre login event
bungee.getPluginManager().callEvent( new PreLoginEvent( InitialHandler.this, callback ) );
}
#Override
public void handle(final EncryptionResponse encryptResponse) throws Exception
{
Preconditions.checkState( thisState == State.ENCRYPT, "Not expecting ENCRYPT" );
SecretKey sharedKey = EncryptionUtil.getSecret( encryptResponse, request );
BungeeCipher decrypt = EncryptionUtil.getCipher( false, sharedKey );
ch.addBefore( PipelineUtils.FRAME_DECODER, PipelineUtils.DECRYPT_HANDLER, new CipherDecoder( decrypt ) );
BungeeCipher encrypt = EncryptionUtil.getCipher( true, sharedKey );
ch.addBefore( PipelineUtils.FRAME_PREPENDER, PipelineUtils.ENCRYPT_HANDLER, new CipherEncoder( encrypt ) );
String encName = URLEncoder.encode( InitialHandler.this.getName(), "UTF-8" );
MessageDigest sha = MessageDigest.getInstance( "SHA-1" );
for ( byte[] bit : new byte[][]
{
request.getServerId().getBytes( "ISO_8859_1" ), sharedKey.getEncoded(), EncryptionUtil.keys.getPublic().getEncoded()
} )
{
sha.update( bit );
}
String encodedHash = URLEncoder.encode( new BigInteger( sha.digest() ).toString( 16 ), "UTF-8" );
String preventProxy = ( ( BungeeCord.getInstance().config.isPreventProxyConnections() ) ? "&ip=" + URLEncoder.encode( getAddress().getAddress().getHostAddress(), "UTF-8" ) : "" );
String authURL = "https://sessionserver.mojang.com/session/minecraft/hasJoined?username=" + encName + "&serverId=" + encodedHash + preventProxy;
Callback<String> handler = new Callback<String>()
{
#Override
public void done(String result, Throwable error)
{
if ( error == null )
{
LoginResult obj = BungeeCord.getInstance().gson.fromJson( result, LoginResult.class );
if ( obj != null && obj.getId() != null )
{
loginProfile = obj;
name = obj.getName();
uniqueId = Util.getUUID( obj.getId() );
authenticated = true;
finish();
return;
}
if(isOnlineMode()) {
disconnect(bungee.getTranslation("offline_mode_player"));
return;
}
finish();
return;
} else
{
disconnect( bungee.getTranslation( "mojang_fail" ) );
bungee.getLogger().log( Level.SEVERE, "Error authenticating " + getName() + " with minecraft.net", error );
}
}
};
HttpClient.get( authURL, ch.getHandle().eventLoop(), handler );
}

I dont think this is possible due to Minecraft not exchanging session details. I know servers who are running a second proxy with online mode enabled for the premium users to handle the session details leading to the same bukkit servers "behind".

You might be looking for this: https://www.spigotmc.org/resources/fastlogin.14153/
I have not used it yet, but according to the reviews, it still works in the latest version.
Also, it's open source, you might be able to peek into the code and see how it's done.

how can i refactor a method with retry policy. without using a 3rd party

I wonder how can i refactor this method with retry lambda [not using a 3rd party retry engine].
(somthing like a lambda that gets number of attempts that will passed as parameter into this method and done this same thing in more functional style
public static Connection createDbConnection( JobConfiguration jobConfiguration, ArrayList<String> hostsIps ) {
Connection retConnection = null;
Properties connectionProperties = new Properties();
connectionProperties.put( "user", jobConfiguration.getDbConfiguration().getUserName() );
connectionProperties.put( "password", jobConfiguration.getDbConfiguration().getPassword() );
connectionProperties.put( "ConnectionLoadBalance", 1 );
int attempts = 0;
while ( ++attempts <= hostsIps.size() ) {
try {
String connectionString = String.format( "jdbc:vertica://%s:%s/%s",
hostsIps.get( attempts - 1),
jobConfiguration.getDbConfiguration().getPort(),
jobConfiguration.getDbConfiguration().getDbName());
retConnection = DriverManager.getConnection
(
connectionString,
connectionProperties
);
retConnection.setAutoCommit( false );
break;
} catch ( Exception ex ) {
LOGGER.error( "Failed to connect to node - {}" + hostsIps.get( attempts - 1 ) + ex );
}
}
return retConnection;
}
can you help?
thanks

The main thing that will simplify your code is to extract all the code inside your while loop and put it in a separate method, e.g. "tryConnect". Then you could use a lambda, if desired, as follows:
Optional<Connection> retConnection = hostIps.stream().sequential()
.map(ip -> tryConnect(ip, jobConfiguration, connectionProperties))
.filter(Objects::nonNull).findFirst();
return retConnection.orElse(null);

Android Studio cordova Push plugin

Can someone help me? I'm new to Android studio.
After I installed the plugin push in my project, I received the following error message:
Error:Execution failed for task ':dexDebug'.
com.android.ide.common.process.ProcessException: org.gradle.process.internal.ExecException: Process 'command '/Library/Java/JavaVirtualMachines/jdk1.7.0_09.jdk/Contents/Home/bin/java'' finished with non-zero exit value 2
I use Android studio 1.3.2 with Cordova https://github.com/phonegap-build/PushPlugin.
I've been searching for days, but can not find a solution
I hope you mean this file
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
*/
import java.util.regex.Pattern
import groovy.swing.SwingBuilder
String doEnsureValueExists(filePath, props, key) {
if (props.get(key) == null) {
throw new GradleException(filePath + ': Missing key required "' + key + '"')
}
return props.get(key)
}
String doGetProjectTarget() {
def props = new Properties()
file('project.properties').withReader { reader ->
props.load(reader)
}
return doEnsureValueExists('project.properties', props, 'target')
}
String[] getAvailableBuildTools() {
def buildToolsDir = new File(getAndroidSdkDir(), "build-tools")
buildToolsDir.list()
.findAll { it ==~ /[0-9.]+/ }
.sort { a, b -> compareVersions(b, a) }
}
String doFindLatestInstalledBuildTools(String minBuildToolsVersion) {
def availableBuildToolsVersions
try {
availableBuildToolsVersions = getAvailableBuildTools()
} catch (e) {
println "An exception occurred while trying to find the Android build tools."
throw e
}
if (availableBuildToolsVersions.length > 0) {
def highestBuildToolsVersion = availableBuildToolsVersions[0]
if (compareVersions(highestBuildToolsVersion, minBuildToolsVersion) < 0) {
throw new RuntimeException(
"No usable Android build tools found. Highest installed version is " +
highestBuildToolsVersion + "; minimum version required is " +
minBuildToolsVersion + ".")
}
highestBuildToolsVersion
} else {
throw new RuntimeException(
"No installed build tools found. Please install the Android build tools version " +
minBuildToolsVersion + " or higher.")
}
}
// Return the first non-zero result of subtracting version list elements
// pairwise. If they are all identical, return the difference in length of
// the two lists.
int compareVersionList(Collection aParts, Collection bParts) {
def pairs = ([aParts, bParts]).transpose()
pairs.findResult(aParts.size()-bParts.size()) {it[0] - it[1] != 0 ? it[0] - it[1] : null}
}
// Compare two version strings, such as "19.0.0" and "18.1.1.0". If all matched
// elements are identical, the longer version is the largest by this method.
// Examples:
// "19.0.0" > "19"
// "19.0.1" > "19.0.0"
// "19.1.0" > "19.0.1"
// "19" > "18.999.999"
int compareVersions(String a, String b) {
def aParts = a.tokenize('.').collect {it.toInteger()}
def bParts = b.tokenize('.').collect {it.toInteger()}
compareVersionList(aParts, bParts)
}
String getAndroidSdkDir() {
def rootDir = project.rootDir
def androidSdkDir = null
String envVar = System.getenv("ANDROID_HOME")
def localProperties = new File(rootDir, 'local.properties')
String systemProperty = System.getProperty("android.home")
if (envVar != null) {
androidSdkDir = envVar
} else if (localProperties.exists()) {
Properties properties = new Properties()
localProperties.withInputStream { instr ->
properties.load(instr)
}
def sdkDirProp = properties.getProperty('sdk.dir')
if (sdkDirProp != null) {
androidSdkDir = sdkDirProp
} else {
sdkDirProp = properties.getProperty('android.dir')
if (sdkDirProp != null) {
androidSdkDir = (new File(rootDir, sdkDirProp)).getAbsolutePath()
}
}
}
if (androidSdkDir == null && systemProperty != null) {
androidSdkDir = systemProperty
}
if (androidSdkDir == null) {
throw new RuntimeException(
"Unable to determine Android SDK directory.")
}
androidSdkDir
}
def doExtractIntFromManifest(name) {
def manifestFile = file(android.sourceSets.main.manifest.srcFile)
def pattern = Pattern.compile(name + "=\"(\\d+)\"")
def matcher = pattern.matcher(manifestFile.getText())
matcher.find()
return Integer.parseInt(matcher.group(1))
}
def doPromptForPassword(msg) {
if (System.console() == null) {
def ret = null
new SwingBuilder().edt {
dialog(modal: true, title: 'Enter password', alwaysOnTop: true, resizable: false, locationRelativeTo: null, pack: true, show: true) {
vbox {
label(text: msg)
def input = passwordField()
button(defaultButton: true, text: 'OK', actionPerformed: {
ret = input.password;
dispose();
})
}
}
}
if (!ret) {
throw new GradleException('User canceled build')
}
return new String(ret)
} else {
return System.console().readPassword('\n' + msg);
}
}
def doGetConfigXml() {
def xml = file("res/xml/config.xml").getText()
// Disable namespace awareness since Cordova doesn't use them properly
return new XmlParser(false, false).parseText(xml)
}
def doGetConfigPreference(name, defaultValue) {
name = name.toLowerCase()
def root = doGetConfigXml()
def ret = defaultValue
root.preference.each { it ->
def attrName = it.attribute("name")
if (attrName && attrName.toLowerCase() == name) {
ret = it.attribute("value")
}
}
return ret
}
// Properties exported here are visible to all plugins.
ext {
// These helpers are shared, but are not guaranteed to be stable / unchanged.
privateHelpers = {}
privateHelpers.getProjectTarget = { doGetProjectTarget() }
privateHelpers.findLatestInstalledBuildTools = { doFindLatestInstalledBuildTools('19.1.0') }
privateHelpers.extractIntFromManifest = { name -> doExtractIntFromManifest(name) }
privateHelpers.promptForPassword = { msg -> doPromptForPassword(msg) }
privateHelpers.ensureValueExists = { filePath, props, key -> doEnsureValueExists(filePath, props, key) }
// These helpers can be used by plugins / projects and will not change.
cdvHelpers = {}
// Returns a XmlParser for the config.xml. Added in 4.1.0.
cdvHelpers.getConfigXml = { doGetConfigXml() }
// Returns the value for the desired <preference>. Added in 4.1.0.
cdvHelpers.getConfigPreference = { name, defaultValue -> doGetConfigPreference(name, defaultValue) }
}

That plugin is deprecated, try using the new one that replaces that one.
https://github.com/phonegap/phonegap-plugin-push

The error is the result of conflicting plugin jar libraries.
Look for libs like com.android.support.v4.jar and similar...
To get around this you need to search for (forked) plugins that already resolved the conflict.
Sadly enough simply deinstalling the "Problem-Plugin" often does not remove the error because there are dependencies left overs somewhere in the files used by cordove build
In that case search manually for these left overs (for example in ../your_project/package.json) and remove all occurences of the deinstalled plugin.

java neo4j check if a relationship exist

excluse if this is a duplicate, although I did not find the answer so far.
I have an application that creates nodes and relationships via cypher statement against the REST-API. I create relationships with the below code:
public URI createRelationship(GraphNodeTypes sourceType, URI sourceNode,
GraphNodeTypes targetType, URI targetNode,
GraphRelationshipTypes relationshipType, String[] jsonAttributes) {
URI relationShipLocation = null;
String cypherArt = getNodeIdFromLocation(sourceNode)+"-[:"+relationshipType+"]->"+getNodeIdFromLocation(targetNode);
logger.info("creating relationship ({}:{}) -[:{}]-> ({}:{})",
sourceType,
getNodeIdFromLocation(sourceNode),
relationshipType,
targetType,
getNodeIdFromLocation(targetNode));
try {
URI finalUrl = new URI( sourceNode.toString() + "/relationships" );
String cypherStatement = generateJsonRelationship( targetNode,
relationshipType,
jsonAttributes );
logger.trace("sending CREATE RELATIONSHIP cypher as {} to endpoint {}", cypherStatement, finalUrl);
WebResource resource = Client.create().resource( finalUrl );
ClientResponse response = resource
.accept( MediaType.APPLICATION_JSON )
.type( MediaType.APPLICATION_JSON )
.entity( cypherStatement )
.post( ClientResponse.class );
String responseEntity = response.getEntity(String.class).toString();
int responseStatus = response.getStatus();
logger.trace("POST to {} returned status code {}, returned data: {}",
finalUrl, responseStatus,
responseEntity);
// first check if the http code was ok
HttpStatusCodes httpStatusCodes = HttpStatusCodes.getHttpStatusCode(responseStatus);
if (!httpStatusCodes.isOk()){
if (httpStatusCodes == HttpStatusCodes.FORBIDDEN){
logger.error(HttpErrorMessages.getHttpErrorText(httpStatusCodes.getErrorCode()));
} else {
logger.error("Error {} sending data to {}: {} ", response.getStatus(), finalUrl, HttpErrorMessages.getHttpErrorText(httpStatusCodes.getErrorCode()));
}
} else {
JSONParser reponseParser = new JSONParser();
Object responseObj = reponseParser.parse(responseEntity);
JSONObject jsonResponseObj = responseObj instanceof JSONObject ?(JSONObject) responseObj : null;
if(jsonResponseObj == null)
throw new ParseException(0, "returned json object is null");
//logger.trace("returned response object is {}", jsonResponseObj.toString());
try {
relationShipLocation = new URI((String)((JSONObject)((JSONArray)((JSONObject)((JSONArray)((JSONObject)((JSONArray)jsonResponseObj.get("results")).get(0)).get("data")).get(0)).get("rest")).get(0)).get("self"));
} catch (Exception e) {
logger.warn("CREATE RELATIONSHIP statement did not return a self object, returning null -- error was {}", e.getMessage());
relationShipLocation = null;
}
}
} catch (Exception e) {
logger.error("could not create relationship ");
}
return relationShipLocation;
}
private static String generateJsonRelationship( URI endNode,
GraphRelationshipTypes relationshipType, String[] jsonAttributes ) {
StringBuilder sb = new StringBuilder();
sb.append( "{ \"to\" : \"" );
sb.append( endNode.toString() );
sb.append( "\", " );
sb.append( "\"type\" : \"" );
sb.append( relationshipType.toString() );
if ( jsonAttributes == null || jsonAttributes.length < 1 ){
sb.append( "\"" );
} else {
sb.append( "\", \"data\" : " );
for ( int i = 0; i < jsonAttributes.length; i++ ) {
sb.append( jsonAttributes[i] );
if ( i < jsonAttributes.length - 1 ){
// Miss off the final comma
sb.append( ", " );
}
}
}
sb.append( " }" );
return sb.toString();
}
My problem is that I would like to check if a given relationship of given type already exists between two nodes PRIOR creating it.
Can someone tell me, how to query for a relationship???
With nodes I do a MATCH like this:
MATCH cypher {"statements": [ {"statement": "MATCH (p:SOCIALNETWORK {sn_id: 'TW'} ) RETURN p", "resultDataContents":["REST"]} ] }
against the endpoint
http://localhost:7474/db/data/transaction/<NUMBER>
How would I construct the statement to check for a relationship, say between node 6 and 5 or whatever?
Thanks in advance,
Chris

You might want to consider doing this through cypher, and using the MERGE/ON CREATE/ON MATCH keywords.
For example, you could do something like this:
create (a:Person {name: "Bob"})-[:knows]->(b:Person {name: "Susan"});
MATCH (a:Person {name: "Bob"}), (b:Person {name: "Susan"})
MERGE (a)-[r:knows]->(b)
ON CREATE SET r.alreadyExisted=false
ON MATCH SET r.alreadyExisted=true
RETURN r.alreadyExisted;
This MATCH/MERGE query that I provide here will return true or false, depending on whether the relationship already existed or not.
Also, FWIW it looks like the code you're using to accumulate JSON via StringBuilder objects is likely to be ponderous and error prone. There are plenty of good libraries like Google GSON that will do JSON for you, so you can create JSON objects, arrays, primitives, and so on -- then let the library worry about serializing it properly to a string . This tends to make your code a lot cleaner, easier to maintain, and when you mess something up about your JSON formatting (we all do), it's way easier to find than when you accumulate strings like that.

In Java
Relationship getRelationshipBetween(Node n1, Node n2) { // RelationshipType type, Direction direction
for (Relationship rel : n1.getRelationships()) { // n1.getRelationships(type,direction)
if (rel.getOtherNode(n1).equals(n2)) return rel;
}
return null;
}