Develop Reference

Elastic Search and Y10k (years with more than 4 digits)

I discovered this issue in connection with Elastic Search queries, but since the ES date format documentation links to the API documentation for the java.time.format.DateTimeFormatter class, the problem is not really ES specific.
Short summary: We are having problems with dates beyond year 9999, more exactly, years with more than 4 digits.
The documents stored in ES have a date field, which in the index descriptor is defined with format "date", which corresponds to "yyyy-MM-dd" using the pattern language from DateTimeFormatter. We are getting user input, validate the input using org.apache.commons.validator.DateValidator.isValid also with the pattern "yyyy-MM-dd" and if valid, we create an ES query with the user input. This fails with an execption if the user inputs something like 20202-12-03. The search term is probably not intentional, but the expected behaviour would be not to find anything and not that the software coughs up an exception.
The problem is that org.apache.commons.validator.DateValidator is internally using the older SimpleDateFormat class to verify if the input conforms to the pattern and the meaning of "yyyy" as interpreted by SimpleDateFormat is something like: Use at least 4 digits, but allow more digits if required. Creating a SimpleDateFormat with pattern "yyyy-MM-dd" will thus both parse an input like "20202-07-14" and similarly format a Date object with a year beyond 9999.
The new DateTimeFormatter class is much more strict and means with "yyyy" exactly four digits. It will fail to parse an input string like "20202-07-14" and also fail to format a Temporal object with a year beyond 9999. It is worth to notice that DateTimeFormatter is itself capable of handling variable-length fields. The constant DateTimeFormatter.ISO_LOCAL_DATE is for example not equivalent to "yyyy-MM-dd", but does, conforming with ISO8601, allow years with more than four digits, but will use at least four digits. This constant is created programmatically with a DateTimeFormatterBuilder and not using a pattern string.
ES can't be configured to use the constants defined in DateTimeFormatter like ISO_LOCAL_DATE, but only with a pattern string. ES also knows a list of predefined patterns, occasionally the ISO standard is also referred to in the documentation, but they seem to be mistaken and ignore that a valid ISO date string can contain five digit years.
I can configure ES with a list of multiple allowed date patterns, e.g "yyyy-MM-dd||yyyyy-MM-dd". That will allow both four and five digits in the year, but fail for a six digit year. I can support six digit years by adding yet another allowed pattern: "yyyy-MM-dd||yyyyy-MM-dd||yyyyyy-MM-dd", but then it fails for seven digit years and so on.
Am I overseeing something, or is it really not possible to configure ES (or a DateTimeFormatter instance using a pattern string) to have a year field with at least four digits (but potentially more) as used by the ISO standard?

Edit
ISO 8601
Since your requirement is to conform with ISO 8601, let’s first see what ISO 8601 says (quoted from the link at the bottom):
To represent years before 0000 or after 9999, the standard also
permits the expansion of the year representation but only by prior
agreement between the sender and the receiver. An expanded year
representation [±YYYYY] must have an agreed-upon number of extra year
digits beyond the four-digit minimum, and it must be prefixed with a +
or − sign instead of the more common AD/BC (or CE/BCE) notation; …
So 20202-12-03 is not a valid date in ISO 8601. If you explicitly inform your users that you accept, say, up to 6 digit years, then +20202-12-03 and -20202-12-03 are valid, and only with the + or - sign.
Accepting more than 4 digits
The format pattern uuuu-MM-dd formats and parses dates in accordance with ISO 8601, also years with more than four digits. For example:
DateTimeFormatter dateFormatter = DateTimeFormatter.ofPattern("uuuu-MM-dd");
LocalDate date = LocalDate.parse("+20202-12-03", dateFormatter);
System.out.println("Parsed: " + date);
System.out.println("Formatted back: " + date.format(dateFormatter));
Output:
Parsed: +20202-12-03
Formatted back: +20202-12-03
It works quite similarly for a prefixed minus instead of the plus sign.
Accepting more than 4 digits without sign
yyyy-MM-dd||yyyyy-MM-dd||yyyyyy-MM-dd||yyyyyyy-MM-dd||yyyyyyyy-MM-dd||yyyyyyyyy-MM-dd
As I said, this disagrees with ISO 8601. I also agree with you that it isn’t nice. And obviously it will fail for 10 or more digits, but that would fail for a different reason anyway: java.time handles years in the interval -999 999 999 through +999 999 999. So trying yyyyyyyyyy-MM-dd (10 digit year) would get you into serious trouble except in the corner case where the user enters a year with a leading zero.
I am sorry, this is as good as it gets. DateTimeFormatter format patterns do not support all of what you are asking for. There is no (single) pattern that will give you four digit years in the range 0000 through 9999 and more digits for years after that.
The documentation of DateTimeFormatter says about formatting and parsing years:
Year: The count of letters determines the minimum field width below which padding is used. If the count of letters is two, then a
reduced two digit form is used. For printing, this outputs the
rightmost two digits. For parsing, this will parse using the base
value of 2000, resulting in a year within the range 2000 to 2099
inclusive. If the count of letters is less than four (but not two),
then the sign is only output for negative years as per
SignStyle.NORMAL. Otherwise, the sign is output if the pad width is
exceeded, as per SignStyle.EXCEEDS_PAD.
So no matter which count of pattern letters you go for, you will be unable to parse years with more digits without sign, and years with fewer digits will be formatted with this many digits with leading zeroes.
Original answer
You can probably get away with the pattern u-MM-dd. Demonstration:
String formatPattern = "u-MM-dd";
DateTimeFormatter dateFormatter = DateTimeFormatter.ofPattern(formatPattern);
LocalDate normalDate = LocalDate.parse("2020-07-14", dateFormatter);
String formattedAgain = normalDate.format(dateFormatter);
System.out.format("LocalDate: %s. String: %s.%n", normalDate, formattedAgain);
LocalDate largeDate = LocalDate.parse("20202-07-14", dateFormatter);
String largeFormattedAgain = largeDate.format(dateFormatter);
System.out.format("LocalDate: %s. String: %s.%n", largeDate, largeFormattedAgain);
Output:
LocalDate: 2020-07-14. String: 2020-07-14.
LocalDate: +20202-07-14. String: 20202-07-14.
Counter-intuituvely but very practically one format letter does not mean 1 digit but rather as many digits as it takes. So the flip side of the above is that years before year 1000 will be formatted with fewer than 4 digits. Which, as you say, disagrees with ISO 8601.
For the difference between pattern letter y and u for year see the link at the bottom.
You might also consider one M and/or one d to accept 2020-007-014, but again, this will cause formatting into just 1 digit for numbers less than 10, like 2020-7-14, which probably isn’t what you want and again disagrees with ISO.
Links
Years section of Wikipedia article: ISO 8601
Documentation of DateTimeFormatter
uuuu versus yyyy in DateTimeFormatter formatting pattern codes in Java?

Maybe this will work:
[uuuu][uuuuu][...]-MM-dd
Format specifiers placed between square brackets are optional parts. Format specifiers inside brackets can be repeated to allow for multiple options to be accepted.
This pattern will allow a year number of either four or five digits, but rejects all other cases.
Here is this pattern in action. Note that this pattern is useful for parsing a string into a LocalDate. However, to format a LocalDate instance into a string, the pattern should be uuuu-MM-dd. That is because the two optional year parts cause the year number to be printed twice.
Repeating all possible year number digit counts, is the closest you can get in order to make it work the way you expect it to work.
The problem with the current implementation of DateTimeFormatter is that when you specify 4 or more u or ys, the resolver will try to consume exactly that number of year digits. However, with less than 4, then the resolver will try to consume as many as possible. I do not know whether this behavior is intentional.
So the intended behavior can be achieved with a formatter builder, but not with a pattern string. As JodaStephen once pointed out, "patterns are a subset of the possible formatters".
Maybe the characters #, { and }, which are reserved for future use, will be useful in this regard.

Update
You can use DateTimeFormatterBuilder#appendValueReduced to restrict the number of digits in a year in the range of 4-9 digits.
import java.time.LocalDate;
import java.time.format.DateTimeFormatter;
import java.time.format.DateTimeFormatterBuilder;
import java.time.temporal.ChronoField;
public class Main {
public static void main(String[] args) {
DateTimeFormatter formatter = new DateTimeFormatterBuilder()
.appendValueReduced(ChronoField.YEAR, 4, 9, 1000)
.appendPattern("-MM-dd")
.toFormatter();
String[] dateStrArr = { "2017-10-20", "20171-10-20", "201712-10-20", "2017123-10-20" };
for (String dateStr : dateStrArr) {
System.out.println(LocalDate.parse(dateStr, formatter));
}
}
}
Output:
2017-10-20
+20171-10-20
+201712-10-20
+2017123-10-20
Original answer
You can use the pattern [uuuu][u]-MM-dd where [uuuu] conforms to a 4-digit year and [u] can cater to the requirement of any number of digits allowed for a year.
Demo:
import java.time.LocalDate;
import java.time.format.DateTimeFormatter;
public class Main {
public static void main(String[] args) {
DateTimeFormatter formatter = DateTimeFormatter.ofPattern("[uuuu][u]-MM-dd");
String[] dateStrArr = { "2017-10-20", "20171-10-20", "201712-10-20", "2017123-10-20" };
for (String dateStr : dateStrArr) {
System.out.println(LocalDate.parse(dateStr, formatter));
}
}
}
Output:
2017-10-20
+20171-10-20
+201712-10-20
+2017123-10-20

How to convert two digit year to full year using Java 8 time API

I wish to remove the Joda-Time library from my project.
I am trying to convert a two digit year to full year. The following code from Joda-Time can fulfil the purpose. Below is the following code of joda-time
DateTimeFormatter TWO_YEAR_FORMATTER = DateTimeFormat.forPattern("yy");
int year = LocalDate.parse("99"", TWO_YEAR_FORMATTER).getYear();
System.out.println(year);
Output: 1999
This is the output that I expect and that makes sense in my situation. However, when I try the same procedure with java.time API, it produces a DatetimeParseException. Below is the following code of java.time API:
DateTimeFormatter TWO_YEAR_FORMATTER = DateTimeFormatter.ofPattern("yy");
int year = LocalDate.parse("99", TWO_YEAR_FORMATTER).getYear();
System.out.println(year);
Stacktrace:
Exception in thread "main" java.time.format.DateTimeParseException: Text '99' could not be parsed: Unable to obtain LocalDate from TemporalAccessor: {Year=2099},ISO of type java.time.format.Parsed
at java.base/java.time.format.DateTimeFormatter.createError(DateTimeFormatter.java:2017)
at java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1952)
at java.base/java.time.LocalDate.parse(LocalDate.java:428)
at scratch.main(scratch.java:10)
Caused by: java.time.DateTimeException: Unable to obtain LocalDate from TemporalAccessor: {Year=2099},ISO of type java.time.format.Parsed
at java.base/java.time.LocalDate.from(LocalDate.java:396)
at java.base/java.time.format.Parsed.query(Parsed.java:235)
at java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1948)
... 2 more
I failed to see the reason of the stacktrace. It would be nice if someone could help me out to understand the following scenario and also explain how to convert a two digit year to full year using Java 8 time API.

The problem is that you can't parse a year on its own into a LocalDate. A LocalDate needs more information than that.
You can use the parse method of the formatter, which will give you a TemporalAccessor, and then get the year field from that:
int year = TWO_YEAR_FORMATTER.parse("99").get(ChronoField.YEAR);
System.out.println(year);
Addressing the discrepancy between the two: these are two distinct APIs. Yes, they are very similar, and the java.time package was informed by design decisions of JodaTime, but it was never intended to be a drop-in replacement for it.
See this answer if you would like to change the pivot year (by default '99' will resolve to 2099 and not 1999).

Why it didn’t work
java.time doesn’t supply default values for month and day of month the way it seems that Joda-Time does.
The message says that it cannot obtain a LocalDate from a year alone, or conversely, it is missing month and day (you may supply your own default values, though, as demonstrated in Mikhail Kholodkov’s answer).
Generally this behaviour is here to help us: it reminds us to supply all the values needed, and makes it clear from the code if any default values are used, and which.
What to do instead
Just use the Year class of java.time. First declare
public static final DateTimeFormatter TWO_YEAR_FORMATTER = new DateTimeFormatterBuilder()
.appendValueReduced(ChronoField.YEAR, 2, 2, 1950)
.toFormatter();
Then use
int year = Year.parse("99", TWO_YEAR_FORMATTER).getValue();
System.out.println(year);
Output
1999
Insert your desired base value where I put 1950 to specify a year within 99 years (inclusive) from that year. If you want the year to be in the past including this year, you may use:
private static final Year base = Year.now(ZoneId.of("Asia/Kolkata")).minusYears(99);
public static final DateTimeFormatter TWO_YEAR_FORMATTER = new DateTimeFormatterBuilder()
.appendValueReduced(ChronoField.YEAR, 2, 2, base.getValue())
.toFormatter();
BTW, don’t take my word for it, but I think Joda-Time uses current year minus 30 years as base or pivot. If this is so, using 30 instead of 99 in the last snippet will be the most compatible (so will also give you the 1999 you expected).

You can't create a LocalDate because the String you give doesn't provide the information needed to fill all required fields.
You can create a Year though
Year parsed = Year.parse("99", TWO_YEAR_FORMATTER);
int year = parsed.getValue();

You need to provide default values for DAY_OF_MONTH and MONTH_OF_YEAR
DateTimeFormatter TWO_YEAR_FORMATTER = new DateTimeFormatterBuilder()
.appendPattern("yy")
.parseDefaulting(ChronoField.MONTH_OF_YEAR, 1)
.parseDefaulting(ChronoField.DAY_OF_MONTH, 1)
.toFormatter();
In addition,
Java-8 uses the range 2000-2099 per default, not like SimpleDateFormat
the range -80 years until +20 years relative to today.
Full answer
Since you're parsing years only, in order to have '99' as '1999' your code should be like this:
DateTimeFormatter TWO_YEAR_FORMATTER = new DateTimeFormatterBuilder()
.appendPattern("")
.parseDefaulting(ChronoField.MONTH_OF_YEAR, 1)
.parseDefaulting(ChronoField.DAY_OF_MONTH, 1)
.appendValueReduced(ChronoField.YEAR_OF_ERA, 2, 2, LocalDate.now().minusYears(80))
.toFormatter();

You could parse the year directly using formatter.parse(input, Year::from) :
import java.time.*;
import java.time.format.*;
class Main {
public static void main(String[] args) {
DateTimeFormatter TWO_YEAR_FORMATTER = DateTimeFormatter.ofPattern("yy");
Year year = TWO_YEAR_FORMATTER.parse("99", Year::from);
System.out.println(year);
}
}
Note that this will output 2099. To output 1999, you should use a specific formatter like the one suggested by Ole V.V. in their answer.

Java MM/dd/yy simple date DateTimeFormatter not working for greater than 2037

Actually, I found problem with MM/dd/yy date format :
If enter year greater than 37 then the year format reflects as 1937.
i.e, if I enter input as 02/05/37 then when I am printing this date into console the date changes into 02/05/1937.
if he entered less than 02/05/37 then working fine.
Date startDate = new SimpleDateFormat("dd/MM/yy").parse("01/01/47");
System.out.println(startDate);

Assuming you're using SimpleDateFormat: It's conform specifications that 02/05/37 is parsed as 02/05/1937. At least for the next year or so...
Java's SimpleDateFormat has to decide in which century your date should be. It does this by adjusting dates to be within 80 years before and 20 years after the time the SimpleDateFormat instance is created. 2037 is within 80 years before the current date (2016), so it uses a time in the past.

The other answers are correct. You need to understand SimpleDateFormat behavior for assuming your intended century.
You are using old outmoded classes. The new recommended classes have a different behavior on this issue.
java.time
The java.time framework built into Java 8 and later supplants the old java.util.Date & SimpleDateFormat classes.
The behavior about assuming century is different. In the DateTimeFormatter class, a two-digit century is interpreted as being in the 21st century, resulting in a year within the range 2000 to 2099 inclusive.
The java.time classes include LocalDate for representing a date-only value without time-of-day and without time zone.
String input = "02/01/47";
DateTimeFormatter formatter = DateTimeFormatter.ofPattern( "dd/MM/yy" );
LocalDate localDate = LocalDate.parse( input , formatter );
2047-01-02
By the way, a tip: Avoid two-digit years if at all possible. The confusion and trouble induced is not worth the savings of two bytes and a smudge of toner.

If you don't supply century info, then it has to make an assumption and it quite reasonably assumes that you are going to want mostly dates in the past, with some scope for future dates, but not too far, as it's more likely that you'll want prior dates, such as birth dates, etc. And people quite commonly live up to about 80 years of age. So far more dates will be in the past for any given current date, than future ones, based on this assumption.
From the spec...
For parsing with the abbreviated year pattern ("y" or "yy"),
SimpleDateFormat must interpret the abbreviated year relative to some
century. It does this by adjusting dates to be within 80 years before
and 20 years after the time the SimpleDateFormat instance is created.
For example, using a pattern of "MM/dd/yy" and a SimpleDateFormat
instance created on Jan 1, 1997, the string "01/11/12" would be
interpreted as Jan 11, 2012 while the string "05/04/64" would be
interpreted as May 4, 1964. During parsing, only strings consisting of
exactly two digits, as defined by Character.isDigit(char), will be
parsed into the default century. Any other numeric string, such as a
one digit string, a three or more digit string, or a two digit string
that isn't all digits (for example, "-1"), is interpreted literally.
So "01/02/3" or "01/02/003" are parsed, using the same pattern, as Jan
2, 3 AD. Likewise, "01/02/-3" is parsed as Jan 2, 4 BC.
Otherwise, calendar system specific forms are applied. For both
formatting and parsing, if the number of pattern letters is 4 or more,
a calendar specific long form is used. Otherwise, a calendar specific
short or abbreviated form is used.
So, if you to do something with this, then you'll need to check if the formatted date is prior to today's date (or some other cut off that you choose) and just add 100 years to the given date, if you wish to only have future dates or beyond a different cut off from the default one.

Two letter year, shall it be allowed?

Recently after library updagrade of Apache POI, I upgraded some other API as well. The other library I used read all cell contents as String and then I had to parse this string into Date.
The problem occurred when user started entering date as dd-mm-yy, the year appeared as 00yy AD.
As per documentation of SimpleDateFormat
For parsing, if the number of pattern letters is more than 2, the year
is interpreted literally, regardless of the number of digits. So using
the pattern "MM/dd/yyyy", "01/11/12" parses to Jan 11, 12 A.D.
So the question is, is it a better to enter the four letter year over two letter year?
The another question is what is best way to predict the year if its in two letter format.
Since the issue will come while parsing below year
Bond Start Date : 12-Jan-98 (1998)
Bond End Date : 12-Jan-70 (2070)
Regards,
Hanumant

It is not clear what you are asking.
If you are asking how to specify a date format that accepts 2 digit years (only) and interprets them conventionally, then you should use "dd-mm-yy".
If you are asking how to specify a date format that accepts 2 digit years and interprets them conventionally, AND ALSO handles 4 (or more) digit years, then you can't. As the javadoc says, if you use "dd-mm-yyyy", 2 digit years are interpreted as years in the first century AD.
One possible solution is to use TWO formats. First attempt to parse using "dd-mm-yy", and if that fails, try "dd-mm-yyyy".
But this is a hack ... and problematic if the user might actually need to enter a historical date.
If you are asking what you should do, then I'd recommend moving away from ambiguous ad-hoc formats that force you to (effectively) guess what the user means.
If the user has to enter dates / times in a character-based form, require them to use one of the ISO 8601 formats, and be strict when parsing the user-supplied date/time strings.
Otherwise, provide the user with a date picker widget.
The another question is what is best way to predict the year if its in two letter format.
Well this is the nub of the problem isn't it! In the 20th century, we all knew what a 2 digit year meant. You just slapped 19 on the front. (Ah ... those were the good old days!)
Now it is necessary to use a different heuristic. And the heuristic that SimpleDateFormat uses is described by the javadoc thus:
"For parsing with the abbreviated year pattern ("y" or "yy"), SimpleDateFormat must interpret the abbreviated year relative to some century. It does this by adjusting dates to be within 80 years before and 20 years after the time the SimpleDateFormat instance is created. For example, using a pattern of "MM/dd/yy" and a SimpleDateFormat instance created on Jan 1, 1997, the string "01/11/12" would be interpreted as Jan 11, 2012 while the string "05/04/64" would be interpreted as May 4, 1964."
The heuristic is 80 years before to 20 years after "now". So actually 12-Jan-98 is in 1998 and 12-Jan-70 is in 1970 ... if you parse using a SimpleDateFormat with a "yy" format.
If you need the dates to mean something else, then you will need to use a different date parser. For example, if you use the Joda-time libraries, it is possible to specify the "pivot year"; i.e. the middle year of the century in which 2-digit years fall.
Reference:
Joda-time Freaky Formatters

java.text.ParseException: Unparseable date: yyyy-MM-dd HH:mm:ss.SSSSSS

I am getting ParseException for the following code
String dateStr = "2011-12-22 10:56:24.389362";
String formatStr = "yyyy-MM-dd HH:mm:ss.SSSSSS";
Date testDate = null;
SimpleDateFormat sdf= new SimpleDateFormat(formatStr);
sdf.setLenient(false);
testDate = sdf.parse(dateStr);
System.out.println("CHECK DATE " + sdf.format(testDate));
Exception in thread "main" java.text.ParseException: Unparseable date: "2011-12-22 10:56:24.389362"
at java.text.DateFormat.parse(DateFormat.java:337)
If I comment out the line sdf.setLenient(false), then I see a time difference in the ouput
CHECK DATE 2011-12-22 11:02:53.000362
What am I doing wrong??

'S' is for millisecond. There are 1000 (0 to 999) milliseconds in a second. 389362 is greater than 999. The extra 389000 milliseconds are getting converted to 389 seconds, or 6 minutes 29 seconds and added to the time.

The S format specifier refers to milliseconds. When you allow lenient parsing, the last part is interpreted as 389362 milliseconds. When that's added to the date so far, the last 3 digits (actually, the value % 1000) become the actual milliseconds, and you wind up with a date about 389 seconds (~6 1/2 minutes) later than you're expecting. (With strict parsing, the parser knows that 389362 milliseconds doesn't make sense, so it throws an error.)
The simplest way around that, if you can guarantee the date will always look like that, would be to chop the last 3 digits off. (This will about half the time give you a date that's off by a millisecond. But that's better than having to write a date parser...)

Your date input for milliseconds is incorrect. It should be:-
String dateStr = "2011-12-22 10:56:24.389";
You also do not need the extra number of "S"s in the pattern. The following should suffice:
String formatStr = "yyyy-MM-dd HH:mm:ss.S";
It is clearly mentioned in the java docs for presentation type of Number:
Number: For formatting, the number of pattern letters is the minimum
number of digits, and shorter numbers are zero-padded to this amount.
For parsing, the number of pattern letters is ignored unless it's
needed to separate two adjacent fields.
It works when you set lenient to be true (or comment out the line which defaults it true) since you are asking the parser to be not strict about the parsing. From java docs on setLenient():-
Specify whether or not date/time parsing is to be lenient. With
lenient parsing, the parser may use heuristics to interpret inputs
that do not precisely match this object's format. With strict parsing,
inputs must match this object's format.

S is only to be used for milliseconds. If you want microseconds, you will have to write your own parser.

Use toISOString('HH:mm:ss.S') to get milliseconds (3 digits), then complete as you need with 0.
For example:
new Date().toISOString('HH:mm:ss.S')
returns "2012-02-10T12:16:39.124Z"