i know this topic isn't new, though i have to dig it up again.
I already searched the Web numerous times (including some Threads here on stackoverflow) but haven't found a satisfying answer so far.
(Amongst others I checked
Parsing Ambiguous Dates in Java and
http://www.coderanch.com/t/375367/java/java/Handling-Multiple-Date-Formats-Elegantly
I am currently writing a Dateparser in Java, which takes a date and generates a format-String which can be used by SimpleDateFormat for parsing the date.
The dates are parsed via regex (yes, it's an ugly one xD) from Logfiles (IBM Websphere, Tomcat, Microsoft Exchange, ....). Because we have customers in (at least 2) different Locales, there is no way to simply "throw" the String against the parse-method of SimpleDateFormat and expect it to work properly.
Furthermore, there is the problem with the position of day and month (i.e. formats "dd/MM/yyyy" or "MM/dd/yyyy") which cannot be solved if i don't have at least two datasets where the day-digit has changed..
So my current approach would be storing the dateformats for a specific software installed at a specific customer's systems in a database (mysql / xml / ... ) and forcing the user to at least specify customername and softwarename so there is enough context to break down the amount of possibilites the format may be given in.
This "subset" then would be used to try to parse the logfiles of the specified software.
(The subset is stored in a HashMap in a HashMap in the form
HashMap> map;
The Integer-Key is the length of the formatstring and the String Key of the second Hashmap specifies a datesignature only containing the separating characters.
(i.e. ".. ::." for a date with format "dd.MM.yyyy 11:11:11.111")
I also take into account the value of the digits, i.e. a digit > 12 has to be a day because there is no 13th month. But this only works reliably for Date-Strings later than the 12th of a month..
Is there any chance to avoid implementing prior knowledge about the environment out of which the logfile came, thus enabling the parser to reliably parse one date without having to refer a second datestring for comparison?
I'm stuck on that for almost 3 months now -.-
Any suggestions would be very welcome =)
Edit:
Okay guys this thread can be closed. I now came up with a different solution for my specific problem. For those who are interested:
I am writing a Logreader in Java. As we have regular maintenance I have to read many logfiles.
But it's not just the plain text information that's written in the file.
Imagine a server just having crashed, it's sunday night and the next person to notice is the head of the IT dpt of the customer. Then on the following day I have to to maintenance and check the logfiles. Judging by content, everything seemed okay, nothing unusual. Half an hour after sending the maintenance report I receive a mail with the above mentioned head of it dpt ranting, that the server had crashed and it seemed to go unnoticed.
The point is, you can't keep track over content and the timestamps for logfiles with several thousand lines. So i developed a component which reads a logfile and calculates the time between two different log-entrys. Each logline got parsed into a java.util.Date to later get the Date as Timestamp for high resolution regarding the log-intervals. The differences i then threw onto a linegraph, which makes longer timeouts between two loglines visible as a big spike relating to the rest of the file.
My solution now will be to completely throw away the date-half of the String and insert a dummy-Date with a predefined format. The date only has to change if the Hour and minute approach 23:59.
The original date later is presented on the graph with the "fake-data" lying beneath.
I thank all of you for your suggestions and feedback =)
(And I hope my English has been understandable so far ;) )
My suggestion is to store all dates as 'ambiguous' until such time that the ambiguity can be resolved. (This assumes that a particular customer will always supply data in the same format.) As soon as you get a log from a customer for which you can unambiguously identify the date format, you would then be able to retrospectively apply this format to previously files.
To do this, you would need a table mapping each customer to their date format with some marker (e.g. NULL) to indicate that format is not yet established. You will probably also need to create your own date representation such that you can model these ambiguous dates.
So, as an example, if the possible date formats are:
dd/mm/yyyy
mm/dd/yyyy
yyyy/mm/dd
yyyy/dd/mm
Given dates, you should always be able to identify the year (permitting two digit years would make this problem considerably harder). So you should be able to map dates as follows:
25/01/2011 -> UNAMBIGUOUS_DD_MM_YYYY
12/01/2011 -> AMBIGUOUS_XX_XX_YYYY
2011/03/03 -> AMBIGUOUS_YYYY_XX_XX
03/30/2011 -> UNAMBIGUOUS_MM_DD_YYYY
If possible, you can ask the customers to pass the dateformat string also along with their actual date strings.
i.e. in their log files, they would need to have one more column
..... , '03/11/2011' , 'MM/DD/YYYY' , ...
I think the strategy you are going for (i.e. analysing a bigger set of data) is the best you can get.
From a single line of logfile you will never know if 3/5/11 is the 3rd of may in 2011 or the 5th of march in 2011. (I guess there might also be locales that might interpret this as 11th of may in 2003...)
I had these problems myself some time ago, and i also could only try to introduce some sort of context by either looking at numbers>12, or what changes quickest (must be "day"). But you already stated that yourself...
Related
I want to parse some dates in Java, but the format is not defined and could be a lot of them (any ISO-8601 format which is already a lot, Unix timestamp in any unit, and more)
Here are some samples :
1970-01-01T00:00:00.00Z
1234567890
1234567890000
1234567890000000
2021-09-20T17:27:00.000Z+02:00
The perfect parsing might be impossible because of ambiguous cases but, a solution to parse most of the common dates with some logical might be achievable (for example timestamps are considered in seconds / milli / micro / nano in order to give a date close to the 2000 era, dates like '08/07/2021' could have a default for month and day distinction).
I didn't find any easy way to do it in Java while in python it is kind of possible (not working on all my samples but at least some of them) using infer_datetime_format of panda function to_datetime (https://pandas.pydata.org/docs/reference/api/pandas.to_datetime.html).
Are there some easy approach in Java?
Well, first of all, I agree with rzwitserloot here that date parsing in free format is extremely difficult and full of ambiguities. So you are skating on thin ice and will eventually run into trouble if you just assume that a user input will be correctly parsed the way you think it will.
Nevertheless, we could make it work if I assume either of the following:
You simply don't care if it will be parsed incorrectly; or
You are doing this for fun or for learning purposes; or
You have a banner, saying:
If the parsing goes wrong, it's your fault. Don't blame us.
Anyway, the DateTimeFormatterBuilder is able to build a DateTimeFormatter which could be able to parse a lot of different patterns. Since a formatter supports optional parsing, it could be instructed to try to parse a certain value, or skip that part if no valid value could be found.
For instance, this builder is able to parse a fairly wide range of ISO-like dates, with many optional parts:
DateTimeFormatterBuilder builder = new DateTimeFormatterBuilder()
.appendPattern("uuuu-M-d")
.optionalStart()
.optionalStart().appendLiteral(' ').optionalEnd()
.optionalStart().appendLiteral('T').optionalEnd()
.appendValue(ChronoField.HOUR_OF_DAY)
.optionalStart()
.appendLiteral(':')
.appendValue(ChronoField.MINUTE_OF_HOUR)
.optionalStart()
.appendLiteral(':')
.appendValue(ChronoField.SECOND_OF_MINUTE)
.optionalStart()
.appendFraction(ChronoField.NANO_OF_SECOND, 1, 9, true)
.optionalEnd()
.optionalEnd()
.optionalEnd()
.appendPattern("[XXXXX][XXXX][XXX][XX][X]")
.optionalEnd();
DateTimeFormatter formatter = builder.toFormatter(Locale.ROOT);
All of the strings below can be successfully parsed by this formatter.
Stream.of(
"2021-09-28",
"2021-07-04T14",
"2021-07-04T14:06",
"2001-09-11 00:00:15",
"1970-01-01T00:00:15.446-08:00",
"2021-07-04T14:06:15.2017323Z",
"2021-09-20T17:27:00.000+02:00"
).forEach(testcase -> System.out.println(formatter.parse(testcase)));
Als you can see, with optionalStart() and optionalEnd(), you could define optional portions of the format.
There are many more patterns you probably want to parse. You could add those patterns to the abovementioned builder. Alternatively, the appendOptional​(DateTimeFormatter) method could be used to include multiple builders.
The perfect parsing might be impossible because of ambiguous cases but, a solution to parse most of the common dates with some logical might be achievable
Sure, and such wide-ranging guesswork should most definitely not be part of a standard java.* API. I think you're also wildly underestimating the ambiguity. 1234567890? It's just flat out incorrect to say that this can reasonably be parsed.
You are running into many, many problems here:
Java in general prefers throwing an error instead of guessing. This is inherent in the language (java has few optional syntax constructs; semicolons aren't optional, () for method invocations are not optional, java intentionally does not have 'truthy/false', i.e. if (foo) is only valid if foo is an expression of the boolean type, unlike e.g. python where you can stick anything in there and there's a big list of what counts as falsy, with the rest being considering truthy. When in rome, be like the romans: If this tenet annoys you, well, either learn to love it, begrudgingly accept it, or program in another language. This idea is endemic in the entire ecosystem. For what it is worth, given that debugging tends to take far longer than typing the optional constructs, java is objectively correct or at least making rational decisions for being like this.
Either you can't bring in the notion that 'hey, this number is larger than 12, therefore it cannot possibly be the month', or, you have to accept that whether a certain date format parsers properly depends on whether the day-of-month value is above or below 12. I would strongly advocate that you avoid a library that fails this rule like the plague. What possible point is there, in the end? "My app will parse your date correctly, but only for about 3/5ths of all dates?" So, given that you can't/should not take that into account, 1234567890, is that seconds-since-1970? milliseconds-since-1970? Is that the 12th of the 34th month of the year 5678, the 90th hour, and assumed zeroes for minutes, seconds, and millis? If a library guesses, that library is wrong, because you should not guess unless you're 95%+ sure.
The obvious and perennial "do not guess" example is, of course, 101112. Is that November 10th, 2012 (european style)? Is that October 11th, 2012 (American style), or is that November 12th, 2010 (ISO style)? These are all reasonable guesses and therefore guessing is just wrong here. Do. Not. Guess. Unless you're really sure. Given that this is a somewhat common way to enter dates, thus: Guessing at all costs is objectively silly (see above). Guessing only when it's pretty clear and erroring out otherwise is mostly useless, given that ambiguity is so easy to introduce.
The concept of guessing may be defensible but only with a lot more information. For example, if you give me the input '101112100000', there's no way it's correct to guess here. But if you also tell me that a human entered this input, and that human is clearly clued into, say, german locale, then I can see the need to be able to turn that into '10th of november 2012, 10 o'clock in the morning': Interpreting as seconds or millis since some epoch is precluded by the human factor, and the day-month-year order by locale.
You asked:
Are there some easy approach in Java?
This entire question is incorrect. The in Java part needs to be stripped from this question, and then the answer is a simple: No. There is no simple way to parse strings into date/times without a lot more information than just the input string. If another library says they can do that, they are lying, or at least, operating under a list of cultural and source assumptions as long as my leg, and you should not be using that library.
I don't know any standard library with this functionality, but you can always use DateTimeFormatter class and guess the format looping over a list of predefined formats, or using the ones provides by this class.
This is a typichal approximation of what you want to archive.
Here you can see and old implementation https://balusc.omnifaces.org/2007/09/dateutil.html
FTA (https://github.com/tsegall/fta) is designed to solve exactly this problem (among others). It currently parses thousands of formats and does not do it via a predefined set, so typically runs extremely quickly. In this example we explicitly set the DateResolutionMode, however, it will default to something intelligent based on the Locale. Here is an example:
import com.cobber.fta.dates.DateTimeParser;
import com.cobber.fta.dates.DateTimeParser.DateResolutionMode;
public abstract class Simple {
public static void main(final String[] args) {
final String[] samples = { "1970-01-01T00:00:00.00Z", "2021-09-20T17:27:00.000Z+02:00", "08/07/2021" };
final DateTimeParser dtp = new DateTimeParser().withDateResolutionMode(DateResolutionMode.MonthFirst).withLocale(Locale.ENGLISH);
for (final String sample : samples)
System.err.printf("Format is: '%s'%n", dtp.determineFormatString(sample));
}
}
Which will give the following output:
Format is: 'yyyy-MM-dd'T'HH:mm:ss.SSX'
Format is: 'yyyy-MM-dd'T'HH:mm:ss.SSSX'
Format is: 'MM/dd/yyyy'
Scanner.class give many useful tools to analyse numbers, like .hasNext_() and .next_() methods which can be triggered to specific traitments.
Is somewhere something like this, for Date : in a first time it would be to detect dates in a String, and adapts response to the format (mm/dd/yy|dd/mm/yy|yyyymmdd, are seconds inside, are letters inside, is four digits year, is negative ...) .
Edit for responses
Scanner give more than SimpleDateFormat it is able to return MatchResult and all is already done (and tested), Is such work is already done, or have I to developp it (re-invent wheel?)
Negative date for before JC periods
mm/dd/yy are from English, dd/mm/yy from other European, yyyymmdd from OS, so finding those differences gives different responses to apply.
Edit2
Calendar is a pretty tool, just know if somebody have already add on it the method above.
Take a look at Natty library. It's pretty cool. It can parse different kinds of dates including "today" "tommorow" "next Friday". Drawbacks is that you cannot really customize it. It has predefined rules for parsing dates and if you want to add/modify them you have to get source code and then compile your own version of natty.
I'm trynig to write a proto file that has a Date field which is not defined as a type into Protocol buffer.
I have read the following post but I couldn't figure out a proper solution that suits me :
What the best ways to use decimals and datetimes with protocol buffers?.
I'm trying to convert the proto file to a java .
My answer in the linked post relates mainly to protobuf-net; however, since you are coming at this from java I would recommend: keep it simple.
For dates, I would suggest just using the time (perhaps milliseconds) into an epoch (1 Jan 1970 is traditional). For times, just the size in that same unit (milliseconds etc). For decimal, maybe use fixed point simply by scaling - so maybe treat 1.05 as the long 1050 and assert always exactly 3dp (hence fixed point).
This is simple and pragmatic, and covers most common scenarios without making things complicated.
I'm not sold on this idea, but I'm really not sold on the idea of storing dates (which aren't instants in time) as a timestamp, so here's my suggestion.
Convert your date into a human-readable integer (e.g. 2014-11-3 becomes 20141103) and store this integer value. It contains exactly the data you need, is simple to create and parse, and takes up minimal space. Additionally, it is ordered and has a one-to-one mapping of dates to valid values (granted, invalid numbers are possible, such as 20149999, but these are easy to detect). In contrast, there are approximately 86400 valid timestamps that represent each day.
NB: There is a discussion on DBA SE criticizing this method of date storage, but in that context a specialized date type exists, which obviously isn't the case here.
I want to represent a set of labelled instances (data) in a file to be fed in to LibSVM as training data. For the problem mentioned in this question. It will include,
Login date
Login time
Location (country code?)
Day of the week
Authenticity (0 - Non Authentic, 1 - Authentic) - The Label
How can I format this data to be input to the SVM?
Are you asking about the data format or how to convert the data? For the latter you're going to have to experiment to find the right way to do this. The general idea is to convert your data into a nominal or ordinal value attribute. Some of these are simple - #4, #6 - some of these are going to be tough - #1-#3.
For example, you could represent #1 as three attributes of day, month and year, or just one by converting it to a UNIX like timestamp.
The IP is even harder - there's no straightforward way to convert that into a meaningful ordinal value. Using every IP as a nominal attribute might not be useful depending on your problem.
Once you figure this out, convert your data, check the LibSVM docs. The general format is followed by : i.e., +1 1:0 2:0 .. etc
I believe there is an unstated assumption in the previous answers. The unstated assumption is that users of libSVM know that they should avoid putting categorical data into the classifier.
For example, libSVM will not know what to do with country codes. If you are trying to predict which visitors are most likely to buy something on your site then you could have problems if USA is between Chad and Niger in your country code list. The bulge from USA will likely skew predictions for the countries located near it.
To fix this I would create one category for each country under consideration (and perhaps an 'other' category). Then for each instance you want to classify, I would set all the country categories to zero except the one to which the instance belongs. (To do this with the libSVM sparse file format, this isn't really a big deal).
Is there any way in Java to guess the date format when it is not explicitly defined?
For example a user types in 11Mar09 or 11-09-2009 or 11/09/2009 or 11-09 what is the best way of parsing this to a Date object without either a bunch of try catch or raising an error?
I don't think you want to do this, especially based on your examples, but if you must, I think your best bet will be to use something like Apache's DateUtils in commons-lang:
String[] datePatterns = new String[] {
"ddMMMyy", // ex. 11Mar09
"dd-MM-yyyy", // ex. 11-09-2009
"dd/MM/yyyy", // ex. 11/09/2009
"dd-MM" // ex. 11-09
}
Date date = DateUtils.parseDate(stringDate, datePatterns);
Unfortunately dates like the fourth one above will be problematic - is "11/09" September 11th, November 9th, September 2011, November 2009, or something else?
My recommendation is don't. Use a date picker or an explicitely noted format. Guessing will lead to all kinds of problems, easily including, if the date is a critical one, litigation.
If you have to guess, provide some form of feedback that is non-ambiguous, something like a confirmation page that says "Your flight will be booked on the 9th of November, 2009. Is this correct?".
You could have a bunch of regular expressions and cycle through until you find a match. I think you could also have a bunch of DateFormats and cycle through them, catching exceptions on ones that failed.
The first avoids using exceptions for non-exceptional cases, the second is maybe better from a design point of view in that you're using a date parsing framework for what it was designed for. But overall, I don't think either approach is necessarily "best" or "worst" per se-- more a matter of personal preferences/beliefs.
As the domain of possible date-strings are infinite, I don't see how it could be possible to recognize them all. You can however pick a subset for pattern-matching.
You give no clues about your user-interface, but the best approach here would be to help the user input the date. For example with a pop-up calendar or just forcing a predefined format.