Develop Reference

In Java, how do I extract the domain of a URL?

I'm using Java 8. I want to extract the domain portion of a URL. Just in case I'm using the word "domain" incorrectly, what i want is if my server name is
test.javabits.com
I want to extract "javabits.com". Similarly, if my server name is
firstpart.secondpart.lastpart.org
I want to extract "lastpart.org". I tried the below
final String domain = request.getServerName().replaceAll(".*\\.(?=.*\\.)", "");
but its not extracting the domain properly. Then I tried what this guy has in his site -- https://www.mkyong.com/regular-expressions/domain-name-regular-expression-example/, e.g.
private static final String DOMAIN_NAME_PATTERN = "^((?!-)[A-Za-z0-9-]{1,63}(?<!-)\\.)+[A-Za-z]{2,6}$";
but that is also not extracting what I want. How can I extract the domain name portion properly?

Summary: Do not use regex for this. Use whois.
If I try to extrapolate from your question, to find out what you really want to do, I guess you want to find the domain belonging to some non-infrastructural owner from the host part of a URL. Additionally, from the tag of your question, you want to do it with the help of a regex.
The task you are undertaking is at best impractical, but probably impossible.
There are a number of corner cases that you would have to weed out. Apart from the list of infrastructural domains kindly provided by Lennart in https://publicsuffix.org/list/public_suffix_list.dat, you also have the cases of an empty host field in the URL or an IP-address forming the host part.
So, is there a better approach to this? Of course there is. What you do want to do is query a public database for the data you need. The protocol for such queries is called WHOIS.
Apache Commons provide an easy way to access WHOIS information in the WhoisClient. From there you can query the domain field, and find some more information that may be useful to you.
It shouldn't be harder than
import org.apache.commons.net.whois.WhoisClient;
import java.io.IOException;
public class CommonsTest {
public static void main(String args) {
WhoisClient c = new WhoisClient();
try {
c.connect(WhoisClient.DEFAULT_HOST);
System.out.println(c.query(URL));
c.disconnect();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Using this will get you the whois information aboutt he domain you are asking for. If the domain is uregistered, that is, is a private domain, as in the case of www.stackexchange.com you will get an error saying no domain is registered. Remove the first part of the address and try again. Once you found the registered domain, you will also find the registrar and the registrer.
Now, unfortunately, whois is not as simple as one would think. Read further on https://manpages.debian.org/jessie/whois/whois.1.en.html for an elaboration on how to use it and what information you can expect from different sources.
Also, check related questions here.

try it like this:
String parts[] = longDomain.split(".");
String domain = parts[parts.length-2] + "." + [parts.length -1];

Spark: Ignoring or handling DataSet select errors

I'm testing some prototype application. We have json data with nested fields. I'm trying to pull some field using following json and code:
Feed: {name: "test",[Record: {id: 1 AllColumns: {ColA: "1",ColB: "2"}}...]}
Dataset<Row> completeRecord = sparkSession.read().json(inputPath);
final Dataset<Row> feed = completeRecord.select(completeRecord.col("Feed.Record.AllColumns"));
I have around 2000 files with such records. I have tested some files individually and they are working fine. But for some file I am getting below error on second line:
org.apache.spark.sql.AnalysisException: Can't extract value from
Feed#8.Record: need struct type but got string;
I'm not sure what is going on here. But I would like to either handle this error gracefully and log which file has that record. Also, is there any way to ignore this and continue with rest of the files?

Answering my own question based on what I have learned. There are couple of ways to solve it. Spark provides options to ignore corrupt files and corrupt records.
To ignore corrupt files one can set following flag to true:
spark.sql.files.ignoreCorruptFiles=true
For more fine grained control and to ignore bad records instead of ignoring the complete file. You can use one of three modes that Spark api provides.
According to DataFrameReader api
mode (default PERMISSIVE): allows a mode for dealing with corrupt
records during parsing.
PERMISSIVE : sets other fields to null when it
meets a corrupted record, and puts the malformed string into a new
field configured by columnNameOfCorruptRecord. When a schema is set by
user, it sets null for extra fields.
DROPMALFORMED : ignores the whole
corrupted records.
FAILFAST : throws an exception when it meets
corrupted records.
PERMISSIVE mode worked really well for me but when I provided my own schema Spark filled missing attributes with null instead of marking it corrupt record.

The exception says that one of the json files differs in its structure and that the path Feed.Record.AllColumns does not exist in this specific file.
Based on this method
private boolean pathExists(Dataset<Row> df, String path) {
try {
df.apply(path);
return true;
}
catch(Exception ex){
return false;
}
}
you can decide if you execute the select or log an error message:
if(pathExists(completeRecord, "Feed.Record.AllColumns") {
final Dataset<Row> feed = completeRecord.select(completeRecord.col("Feed.Record.AllColumns"));
//continue with processing
}
else {
//log error message
}

AggregationBuilders throws an error JsonGenerationException

I am using elasticsearch API (elasticsearch5.2.1.jar)to do the aggregation query as follows,
StatsAggregationBuilder statAggBuild = AggregationBuilders.stats("agg").field("ipAddress");
This returns the follows exception
{ "error" : "JsonGenerationException[Can not write a field name, expecting a value]"}
Am I doing anything wrong here ?

It is not causing any issue while setting this Object in the request. It returns the response as expected
It looks the actual problem exist in the toString method, Which throws the error when we try to look what exist in the project

Objectify: Unable to evaluate the expression Method threw 'com.googlecode.objectify.LoadException' exception

i am learning how to store and retrieve data with the google app engine and objectify, and set up a test project in intellij-idea. i created a simple entity that looks like this:
ContactType
#Entity
public class ContactType {
#Id
public Long id;
public String name;
public ContactType(String name){
this.name = name;
}
}
before i start testing i delete all saved instances i created before in my servlet:
deleting old data
Objectify ofy = ObjectifyService.ofy();
ObjectifyService.register(ContactType.class);
List<Key<ContactType>> contactTypes = ofy.load().type(ContactType.class).keys().list();
ofy.delete().keys(contactTypes).now();
after that i save this entity like this:
saving new data
ContactType contactType1 = new ContactType("contactType1");
ContactType contactType2 = new ContactType("contactType2");
ofy.save().entity(contactType1 ).now();
ofy.save().entity(contactType2 ).now();
then i retrieve the objects i just saved like this:
retrieving data
List<ContactType> list= ofy
.load()
.type(ContactType.class)
.list();
and get the 2 expected objects.
but when i comment out the lines that delete and save the old entries, and just want to retrieve the entries that i saved last time (and which i can still see in the development console), and inspect the returned entries with the intellij-idea debugger, i just get this small error message and no stacktrace in the console at all.:
debugging error message
Unable to evaluate the expression Method threw 'com.googlecode.objectify.LoadException' exception.
and when i change the "view as" option from "list" to "toString" in the intellij-idea debugger i get only following information:
so my questions are:
how can i save and retrieve data with objectify?
how can i see a detailed error stacktrace when something goes wrong?

i finally fixed this problem. when i try to get the size of the returned list and put a try/catch around it then i get an error message that my entity does not have a default constructor without parameters.
try{
List<ContactType> list= ofy
.load()
.type(ContactType.class)
.list();
int size = list.size();
}catch(LoadException e){
String message = e.getMessage();
}
after adding following construtor everything works fine:
public ContactType(){
name = "";
}

In answer to part 2 of your question, the stacktrace thrown by Objectify includes all the information you need in the wrapped exception. Whatever is catching and logging the exception is apparently suppressing the exception message. This is not default behavior of the GAE dev environment, so I don't know what's going on.

Searching a key word with java client programme from wsdl

Heyy I have a problem
I generated a client programme using http://api.search.live.net/search.wsdl this service for searching a key words..
I generated client by help of Eclipse-web project from this service.
I have done searched on this service(live.net) but ı can't show on the console. How ı can do that?
public static void main(String[] args) throws RemoteException {
LiveSearchPortTypeProxy bb=new LiveSearchPortTypeProxy();
SearchRequest request=new SearchRequest();
SearchRequestType1 bbs=new SearchRequestType1();
aas.setParameters(request);
sorgu.setAppId("*****************************************"); //you can take this ID from live service for using this service
sorgu.setSources(new SourceType[]{SourceType.Web});
sorgu.setQuery("keyword");
SearchResponseType0 cevap= bb.search(bbs);
}

Have a look at the cevap variable and its public get methods in a debugger, and see what you have.
You will most likely end up with an instance of org.w3c.Node which represents the root of the parsed response. This can then be treated like any other DOM tree.