Develop Reference

XPath Filter by Date

I need to filter the records by date through JAVA class:
public static final String DATE_TIME = "DateTime"
Date dateNow = setTimeToZero(Calendar.getInstance().getTime());
String date = DATE_TIME_FORMAT.format(dateNow);
Request request = table.createRequest();
DATE_FILTER = XPathFilter.newFilter("date-greater-than(" + Root_Table_DateField.format() + '$date')");
request.setXPathFilter(DATE_FILTER);
request.setXPathParameter(DATE_TIME, date);
RequestResult = reqResult = request.execute();
The field I am trying to access is define as DateTime, but I don't want the time now, so I set the time to zero, so I can filter for all field with date greater than dd-MM-yyyT00:00:00:000
But it return a predicate error: PredicateException: Invalid XPath expresion ./dateTime = 26-06-2020T00:00:00:000 - Unexpected 'T00:00:00.000'
Any clue?
Thank you

Try to add the date formatted like that (xs:date("2020-06-26Z")
date-greater-than(xs:date("2004-12-25-12:00"), (xs:date("2020-06-26Z"))
Reference: https://www.w3.org/TR/xpath-functions-31/

Apache Flink: Wierd FlatMap behaviour

I'm ingesting a stream of data into Flink. For each 'instance' of this data, I have a timestamp. I can detect if the machine I'm getting the data from is 'producing' or 'not producing', this is done via a custom flat map function that's located in it's own static class.
I want to calculate how long the machine has been producing / not producing.
My current approach is collecting the production and non production timestamps in two plain lists. For each 'instance' of the data, I calculate the current production/non-production duration by subtracting the latest timestamp from the earliest timestamp. This is giving me incorrect results, though. When the production state changes from producing to non producing, I clear the timestamp list for producing and vice versa, so that if the production starts again, the duration starts from zero.
I've looked into the two lists I collect the respective timestamps in and I see things I don't understand. My assumption is that, as long as the machine 'produces', the first timestamp in the production timestamp list stays the same, while new timestamps are added to the list per new instance of data.
Apparantly, this assumption is wrong since I get seemingly random timestamps in the lists. They are still correctly ordered, though.
Here's my code for the flatmap function:
public static class ImaginePaperDataConverterRich extends RichFlatMapFunction<ImaginePaperData, String> {
private static final long serialVersionUID = 4736981447434827392L;
private transient ValueState<ProductionState> stateOfProduction;
SimpleDateFormat dateFormat = new SimpleDateFormat("dd.MM.yyyy HH:mm:ss.SS");
DateFormat timeDiffFormat = new SimpleDateFormat("dd HH:mm:ss.SS");
String timeDiffString = "00 00:00:00.000";
List<String> productionTimestamps = new ArrayList<>();
List<String> nonProductionTimestamps = new ArrayList<>();
public String calcProductionTime(List<String> timestamps) {
if (!timestamps.isEmpty()) {
try {
Date firstDate = dateFormat.parse(timestamps.get(0));
Date lastDate = dateFormat.parse(timestamps.get(timestamps.size()-1));
long timeDiff = lastDate.getTime() - firstDate.getTime();
if (timeDiff < 0) {
System.out.println("Something weird happened. Maybe EOF.");
return timeDiffString;
}
timeDiffString = String.format("%02d %02d:%02d:%02d.%02d",
TimeUnit.MILLISECONDS.toDays(timeDiff),
TimeUnit.MILLISECONDS.toHours(timeDiff) % TimeUnit.HOURS.toHours(1),
TimeUnit.MILLISECONDS.toMinutes(timeDiff) % TimeUnit.HOURS.toMinutes(1),
TimeUnit.MILLISECONDS.toSeconds(timeDiff) % TimeUnit.MINUTES.toSeconds(1),
TimeUnit.MILLISECONDS.toMillis(timeDiff) % TimeUnit.SECONDS.toMillis(1));
} catch (ParseException e) {
e.printStackTrace();
}
System.out.println("State duration: " + timeDiffString);
}
return timeDiffString;
}
#Override
public void open(Configuration config) {
ValueStateDescriptor<ProductionState> descriptor = new ValueStateDescriptor<>(
"stateOfProduction",
TypeInformation.of(new TypeHint<ProductionState>() {}),
ProductionState.NOT_PRODUCING);
stateOfProduction = getRuntimeContext().getState(descriptor);
}
#Override
public void flatMap(ImaginePaperData ImaginePaperData, Collector<String> output) throws Exception {
List<String> warnings = new ArrayList<>();
JSONObject jObject = new JSONObject();
String productionTime = "0";
String nonProductionTime = "0";
// Data analysis
if (stateOfProduction == null || stateOfProduction.value() == ProductionState.NOT_PRODUCING && ImaginePaperData.actSpeedCl > 60.0) {
stateOfProduction.update(ProductionState.PRODUCING);
} else if (stateOfProduction.value() == ProductionState.PRODUCING && ImaginePaperData.actSpeedCl < 60.0) {
stateOfProduction.update(ProductionState.NOT_PRODUCING);
}
if(stateOfProduction.value() == ProductionState.PRODUCING) {
if (!nonProductionTimestamps.isEmpty()) {
System.out.println("Production has started again, non production timestamps cleared");
nonProductionTimestamps.clear();
}
productionTimestamps.add(ImaginePaperData.timestamp);
System.out.println(productionTimestamps);
productionTime = calcProductionTime(productionTimestamps);
} else {
if(!productionTimestamps.isEmpty()) {
System.out.println("Production has stopped, production timestamps cleared");
productionTimestamps.clear();
}
nonProductionTimestamps.add(ImaginePaperData.timestamp);
warnings.add("Production has stopped.");
System.out.println(nonProductionTimestamps);
//System.out.println("Production stopped");
nonProductionTime = calcProductionTime(nonProductionTimestamps);
}
// The rest is just JSON stuff
Do I maybe have to hold these two timestamp lists in a ListState?
EDIT: Because another user asked, here is the data I'm getting.
{'szenario': 'machine01', 'timestamp': '31.10.2018 09:18:39.432069', 'data': {1: 100.0, 2: 100.0, 101: 94.0, 102: 120.0, 103: 65.0}}
The behaviour I expect is that my flink program collects the timestamps in the two lists productionTimestamps and nonProductionTimestamps. Then I want my calcProductionTime method to subtract the last timestamp in the list from the first timestamp, to get the duration between when I first detected the machine is "producing" / "not-producing" and the time it stopped "producing" / "not-producing".

I found out that the reason for the 'seemingly random' timestamps is Apache Flink's parallel execution. When the parallelism is set to > 1, the order of events isn't guaranteed anymore.
My quick fix was to set the parallelism of my program to 1, this guarantees the order of events, as far as I know.

Java - Improving iteration through large String List

I have to parse files which has around 50000 lines and has to iterate through each line, parse, create a List and save to database. Initially I thought the time taken is because of reading the file. But the file is actually read within a second. But the parsing of data takes long time.
public static final String record = "dlrCode,partNumber,5,0.00,5000.00,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0,0.00,0";
public static final String COMMA = ",";
public static final String QUOTES = "\"";
public static final String EMPTY_STRING = "";
public static void main(String[] args){
List<String> recordsList = new ArrayList<String>();
Date time = new Date();
Part partVO = null;
PartHistory partHistoryVO = null;
List<PartHistory> partHistoryList = null;
List<Part> partsList = new ArrayList<Part>();
int splitLength = 0;
Calendar cal = Calendar.getInstance();
int historySplitCount = 0;
int monthCountReverse = 0;
//add 20000 records to list
for(int i=0; i<20000; i++){
recordsList.add(record);
}
System.out.println("Added in "+((new Date()).getTime() - time.getTime()) +" ms");
//reset time
time = new Date();
//parse records
for(String sCurrentLine : recordsList){
partVO = new Part();
partHistoryList = new ArrayList<PartHistory>();
//Parsing inventory information
partVO.setDealerCode(sCurrentLine.split(COMMA)[0]);
partVO.setPartNumber(sCurrentLine.split(COMMA)[1]);
partVO.setDmsMfId(sCurrentLine.split(COMMA)[2]);
partVO.setQtyOnHand(Math.round(Float.parseFloat(sCurrentLine.split(COMMA)[3])));
partVO.setDealerNet(Float.parseFloat(sCurrentLine.split(COMMA)[4]));
//Parsing history information
//starting from the 6th record as the first 5 records are used above
historySplitCount = 5;
//to subtract one month from current date
monthCountReverse = -1;
splitLength = sCurrentLine.split(COMMA).length;
while(splitLength>=(historySplitCount+1)){
partHistoryVO = new PartHistory();
//subtract one month from current date
cal.add(Calendar.MONTH, monthCountReverse);
partHistoryVO.setMonth(cal.get(Calendar.MONTH)+1);
partHistoryVO.setYear(cal.get(Calendar.YEAR));
partHistoryVO.setLineHitsMonthly(Math.round(Float.parseFloat(sCurrentLine.split(COMMA)[historySplitCount])));
historySplitCount++;
partHistoryVO.setQuantity(Math.round(Float.parseFloat(sCurrentLine.split(COMMA)[historySplitCount])));
historySplitCount++;
partHistoryList.add(partHistoryVO);
}
partVO.setHistoryList(partHistoryList);
partsList.add(partVO);
}
System.out.println("Iterated in "+((new Date()).getTime() - time.getTime()) +" ms");
}
Output
Added in 15 ms
Iterated in 12823 ms
Can the iteration time be improved and brought under atleast 5 seconds?

You're calling
sCurrentLine.split(COMMA)
several times in your code. Make a
final String[]
variable the first time you call it in the loop and use that instead thereafter and it'll get that many times faster.

For each line, you call the split() function multiple times, sCurrentLine.split(COMMA)[0],
a better way is to split it once and store into an array
String[] elements = sCurrentLine.split(COMMA);
dealerCode = elements[0];
partNumber = elements[1];
FYI, to count how much time spent, you can also use System.currentTimeMillis(), this does not need to create a new Date instance :)
long timeStarts = System.currentTimeMillis();
//loop goes here
long timeTook = System.currentTimeMillis() - timeStarts;

How to find the difference of two timestamps in java?

I have an ArrayList including several number of time-stamps and the aim is finding the difference of the first and the last elements of the ArrayList.
String a = ArrayList.get(0);
String b = ArrayList.get(ArrayList.size()-1);
long diff = b.getTime() - a.getTime();
I also converted the types to int but still it gives me an error The method getTime is undefined for the type String.
Additional info :
I have a class A which includes
String timeStamp = new SimpleDateFormat("ss S").format(new Date());
and there is a class B which has a method private void dialogDuration(String timeStamp)
and dialogueDuration method includes:
String a = timeSt.get(0); // timeSt is an ArrayList which includes all the timeStamps
String b = timeSt.get(timeSt.size()-1); // This method aims finding the difference of the first and the last elements(timestamps) of the ArrayList (in seconds)
long i = Long.parseLong(a);
long j = Long.parseLong(b);
long diff = j.getTime()- i.getTime();
System.out.println("a: " +i);
System.out.println("b: " +j);
And one condition is that the statement(String timeStamp = new SimpleDateFormat("ss S").format(new Date());) wont be changed in class A. And an object of class B is created in class A so that it invokes the dialogueDuration(timeStamp) method and passes the values of time-stamps to class B.
My problem is this subtraction does not work, it gives an error cannot invoke getTime() method on the primitive type long. It gives the same kind of error also for int and String types?
Thanks a lot in advance!

Maybe like this:
SimpleDateFormat dateFormat = new SimpleDateFormat("ss S");
Date firstParsedDate = dateFormat.parse(a);
Date secondParsedDate = dateFormat.parse(b);
long diff = secondParsedDate.getTime() - firstParsedDate.getTime();

Assuming you have Timestamp objects or Date Objects in your ArrayList you could do:
Timestamp a = timeSt.get(0);
Timestamp b = timeSt.get(timeSt.size()-1);
long diff = b.getTime() - a.getTime();

You can calculate the difference with the both following methods(also you can modify the mentioned methods to return difference as 'millisecond', 'day', 'month', etc by adding additional if statement or using switch case):
private Long calculateDifference(String date1, String date2, String value) {
Timestamp date_1 = stringToTimestamp(date1);
Timestamp date_2 = stringToTimestamp(date2);
long milliseconds = date_1.getTime() - date_2.getTime();
if (value.equals("second"))
return milliseconds / 1000;
if (value.equals("minute"))
return milliseconds / 1000 / 60;
if (value.equals("hours"))
return milliseconds / 1000 / 3600;
else
return new Long(999999999);
}
private Timestamp stringToTimestamp(String date) {
try {
SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
Date parsedDate = dateFormat.parse(date);
return new Timestamp(parsedDate.getTime());
} catch (Exception e) {
return null;
}
}
For example:
calculateDifference("2021-10-20 10:00:01", "2021-10-20 10:15:01", "minute");
will return '-15'
or
calculateDifference("2021-10-20 12:00:01", "2021-10-20 10:15:01", "minute");
will return '105'

You should make your ArrayList x to an ArrayList<TimeStamp> x. Subsequently, your method get(int) will return an object of type TimeStamp (instead of a type String). On a TimeStamp you are allowed to invoke getTime().
By the way, do you really need java.sql.TimeStamp? Maybe a simple Date or Calendar is easier and more appropriate.

String to date with no format specified

I want to convert a string into a date, this is simple. But what I'd like to do it without knowing the date format.
Here is a situation: say I have 100 dates and all are in the same format but I'd like to write a Java program to find out this format for me. The result of this program should give me a list of all the possible formats.
For example:
06-06-2006
06-06-2009
...
06-13-2001 <- 99th record
the result of this will give me date format can be mm-dd-yyyy
If the 99th record also was 06-06-2006 the result should be mm-dd-yyyy and dd-mm-yyyy.
Can someone please help me with an example?

Seems sensible to create a set of formats you know about (DATE_FORMATS) and then test each line to see which formats understand every line. You should end up with a set of possibilities.
public class DateFormatDetector {
private static final Set<String> DATE_FORMATS = new HashSet<String>();
static {
DATE_FORMATS.add("yyyy-MM-dd");
DATE_FORMATS.add("dd-MM-yyyy");
DATE_FORMATS.add("MM-dd-yyyy");
}
public static Set<String> getPossibleDateFormats(List<String> dates) {
Set<SimpleDateFormat> candidates = new HashSet<SimpleDateFormat>();
for (String df : DATE_FORMATS) {
SimpleDateFormat candidate = new SimpleDateFormat(df);
candidate.setLenient(false);
candidates.add(candidate);
}
for (String date : dates) {
Iterator<SimpleDateFormat> it = candidates.iterator();
while (it.hasNext()) {
SimpleDateFormat candidate = it.next();
try {
// try to parse the string as a date
candidate.parse(date);
}
catch (ParseException e) {
// failed to parse, so this format is not suitable
it.remove();
}
}
}
Set<String> results = new HashSet<String>();
for (SimpleDateFormat candidate : candidates)
results.add(candidate.toPattern());
return results;
}
}

Try to use SimpleDateFormat prepare all possible formats and calculate parsed result.

The solution could be functional Java as described for example in the stack overflow