How can I make sure my entire development environment around PostgreSQL is not messing about with local timezones. For simplicity I need to be 100% sure that each and every time(stamp) value is UTC. When I inserted a row with timestamp without time zone (!) using the CURRENT_TIMESTAMP function I had to realize this was not the case, even though I never ever specified any time zone information.
Is there any step-by-step manual that helps me get rid of time zones?
This requires understanding first. I wrote a comprehensive answer about how PostgreSQL handles timestamps and time zones here:
Ignoring timezones altogether in Rails and PostgreSQL
You cannot "not" have a time zone. You can operate with the type timestamp [without time zone], but you'd still have a time zone in your client.
Your statement:
When I inserted a row with timestamp without time zone (!) using the CURRENT_TIMESTAMP function ...
is a contradictio in adjecto. CURRENT_TIMESTAMP returns a timestamp with time zone (!). If you just cast it (or have it coerced automatically) into timestamp [without time zone], the time zone offset is truncated instead of applied. You get local time (whatever the current time zone setting of the session is) instead of UTC. Consider:
SELECT CURRENT_TIMESTAMP AT TIME ZONE 'UTC'
,CURRENT_TIMESTAMP::timestamp
Unless your local time zone setting is 'UTC' or something like 'London', the two expressions return different values.
If you want to save the literal value you see in your time zone, use one of:
SELECT CURRENT_TIMESTAMP::timestamp
,now()::timestamp
,LOCALTIMESTAMP;
If you want to save the point in time as it would be represented in UTC, use one of:
SELECT CURRENT_TIMESTAMP AT TIME ZONE 'UTC'
,now() AT TIME ZONE 'UTC;
You have fallen victim to a major misconception: Time stamps do not contain any time zone information! See my other answer here for details. In other words, your entire development environment already doesn't use time zones. The only thing you need to ensure is that when a text representation of the time is converted to a time stamp (and vice versa), the thing doing the converting knows what time zone the text representation was expressed in. For everything else, time zones are irrelevant.
I blame Sun for this! They thought it would be convenient for developers if they included methods for converting a time stamp to/from text inside the timestamp object itself (first with Date and then with Calendar). Since this conversion required a time zone, they thought it would be extra convenient if that same class stored the time zone, so you wouldn't have to pass it every time when doing a conversion. This fostered one of the most pervasive (and damaging) misconceptions in Java ever. I don't know what excuse people who develop in other languages have. Maybe they're just dumb.
Declare date columns "timestamptz" or "timestamp with time zone".
Are you also asking about converting existing data not stored with timestamps?
Wrong type, use TIMEZONE WITH TIME ZONE
timestamp without time zone
The TIMESTAMP WITHOUT TIME ZONE data type in both Postgres and the SQL standard represents a date and a time-of-day but without any concept of time zone or offset-from-UTC. So this type cannot represent a moment, is not a point on the timeline.
Any time zone or offset information you submit with a value to a column of this type will be ignored.
When tracking specific moments, use the other type, TIMESTAMP WITH TIME ZONE. In Postgres, any time zone or offset information you submit with a value to a column of this type will be used to adjust into UTC (and then discarded).
For simplicity I need to be 100% sure that each and every time(stamp) value is UTC.
Then use a column of type TIMESTAMP WITH TIME ZONE.
s there any step-by-step manual that helps me get rid of time zones?
You do not want to get rid of time zones (and offsets), as that would mean you would be left with an ambiguous date and time-of-day. For example, noon on the 23rd of January this year fails to tell us if you mean noon in Tokyo Japan, noon in Toulouse France, or noon in Toledo Ohio US. Those are all different moments, all several hours apart.
java.time
With JDBC 4.2, we can exchanged java.time objects rather than the terrible legacy date-time classes.
OffsetDateTime odt = OffsetDateTime.now( ZoneOffset.UTC ) ;
myPreparedStatement.setObject( … , odt ) ;
Retrieval.
OffsetDateTime odt = myResultSet.getObject( … , OffsetDateTime.class ) ;
These values will all be in UTC, having an offset of zero hours-minutes-seconds.
Beware of middleware
Beware that many tools and middleware, such as PgAdmin, will lie to you. In a well-intentioned anti-feature, they apply a default time zone to the data pulled from the database. Values of type TIMESTAMP WITH TIME ZONE are always stored in UTC in Postgres, always. But your tool may report that value as if in America/Montreal, or Pacific/Auckland, or any other default time zone.
I recommend always setting the default time zone in such tools to UTC.
Related
I have a simple spring boot REST API application, using plain jdbc to fetch data from a MSSQL DB. I am trying to figure out how best to retrieve a DATETIME2 column from the DB (which stores no timezone info), and serialize it as a UTC timestamp (and treat it as such in general in code).
My DB server timezone is set to UTC. I know that everything stored to this column is stored as UTC and I cannot change the column type unfortunately. It's a bit of a legacy DB, so all the dates will need to fetch will have this same problem, hence looking for a clean neat and tidy solution.
Ideally in my Java app, I would ideally like all my "date" fields to be of type java.time.Instant, since it is easy to handle and will serialize to json looking something like "someDate": "2022-05-30T15:04:06.559896Z".
The options as I see them are:
Use a custom RowMapper to do something like myModel.setDate(rs.getTimestamp("Date").toLocalDateTime().toInstant(ZoneOffset.UTC));, but this just seems verbose. I suppose I could tuck it away in some utility class static function?
Use LocalDateTime everywhere and do myModel.setDate(rs.getTimestamp("Date").toLocalDateTime()). But then Jackson will serialize it without timezone information.
Set the whole app timezone to UTC on startup. But this could be changed by other code, and from what I read is generally a bad idea.
Caveat: I am not a user of Spring.
moment versus not-a-moment
You need to get clear on one fundamental issue with date-time handling: moment versus not-a-moment.
By “moment” I mean a specific point on the timeline. Without even thinking about time zones and such, we all know that time flows forward, one moment at a time. Each moment is simultaneous for everyone around the world (sticking with Newtonian time, ignoring Einstein Relativity here 😉). To track a moment in Java, use Instant, OffsetDateTime, or ZonedDateTime. These are three different ways to represent a specific point on the timeline.
By “not-a-moment” I mean a date with a time-of-day, but lacking the context of a time zone or offset-from-UTC. If I were to say to you, “Call me at noon tomorrow” without the context of a time zone, you would have no way of knowing if you should call at noon time in Tokyo Japan, noon time in Toulouse France, or noon time in Toledo Ohio US — three very different moments, several hours apart. For not-a-moment, use LocalDateTime.
So never mix LocalDateTime with the other three classes, Instant, OffsetDateTime, or ZonedDateTime. You would be mixing your apples with your oranges.
You said:
I would ideally like all my "date" fields to be of type java.time.Instant
Yes, I would agree on generally using Instant as the member field on any Java object tracking a moment. This is generally a good idea — but only for moments. For not-a-moment, as discussed above, you should use LocalDateTime instead.
TIMESTAMP WITH TIME ZONE
Another issue, Instant was not mapped in JDBC 4.2 and later. Some JDBC drivers may optionally handle an Instant object, but doing so is not required.
So convert your Instant to a OffsetDateTime. The OffsetDateTime class is mapped in JDBC to a database column of a type akin to the SQL-standard type TIMESTAMP WITH TIME ZONE.
OffsetDateTime odt = instant.atOffset( Offset.UTC ) ;
Writing to database.
myPreparedStatement.setObject( … , odt ) ; // Pass your `OffsetDateTime` object.
Retrieval.
OffsetDateTime odt = myResultSet.getObject( … , OffsetDateTime.class ) ;
TIMESTAMP WITHOUT TIME ZONE
For database columns of a type akin to the SQL-standard type TIMESTAMP WITHOUT TIME ZONE, use LocalDateTime class.
Writing to database.
myPreparedStatement.setObject( … , ldt ) ; // Pass your `LocalDateTime` object.
Retrieval.
LocalDateTime ldt = myResultSet.getObject( … , LocalDateTime.class ) ;
Specify time zone
You said:
My DB server timezone is set to UTC.
That should be irrelevant. Always write your Java code in such as way as to not rely on the JVM’s current default time zone, the host OS’ current default time zone, or the database’s current default time zone. All of those lay outside your control as a programmer.
Specify your desired/expected time zone explicitly.
Retrieve a moment from the database, and adjust into a desired time zone.
OffsetDateTime odt = myResultSet.getObject( … , OffsetDateTime.class ) ;
ZoneId z = ZoneId.of( "Africa/Tunis" ) ;
ZonedDateTime zdt = odt.atZoneSameInstant( z ) ;
Generate text localized to the user's preferred locale.
Locale locale = Locale.JAPAN ;
DateTimeFormatter f = DateTimeFormatter.ofLocalizedDateTime( FormatStyle.LONG ).withLocale( locale ) ;
String output = zdt.format( f ) ;
DATETIME2 in MS SQL Server
The type DATETIME2 type in MS SQL Server stores a date with time-of-day, but lacks the context of a time zone or offset-from-UTC.
That is exactly the wrong type to use for storing a moment. As discussed above, that type is akin to the SQL standard type TIMESTAMP WITHOUT TIME ZONE, and maps to the Java class LocalDateTime.
You seem to understand that fact given your comment:
I know that everything stored to this column is stored as UTC and I cannot change the column type unfortunately. It's a bit of a legacy DB …
Let me point out that you do not know the values in that column are intended to represent a moment as seen with an offset of zero. You can expect that, and hope so. But without using the protection of the database’s type system, you cannot be certain. Every user, every DBA, and every SysAdmin must have always been aware of this unfortunate scenario, and must have always done the right thing. You’ll need lots of luck with that.
I must mention that the ideal solution is to refactor your database, to correct this wrong choice of data type for that column. But I understand this could be a burdensome and challenging fix.
So given this unfortunate scenario without a fix being feasible, what to do?
Options 1, 2, & 3 you listed
Option 1
Regarding your option # 1, yes that makes sense to me. Except two things:
I would change the name of your model method to be more precise: setInstant. Or use a descriptive business name such as setInstantWhenContractGoesIntoEffect.
Never use the awful legacy date-time classes in Java such as Timestamp. Change this:
myModel.setDate(rs.getTimestamp("Date").toLocalDateTime().toInstant(ZoneOffset.UTC));
… to:
myModel
.setInstantWhenContractGoesIntoEffect
(
resultSet
.getObject( "Date" , LocalDateTime.class ) // Returns a `LocalDateTime` object.
.toInstant( ZoneOffset.UTC ) // Returns an `Instant` object.
)
;
Option 2
As for your option # 2, I am not quite sure what you have in mind. But my impression is that would be the wrong way to go. I believe the best approach, for long-term maintenance without "technical debt", and for avoiding confusing and mishaps, is to “tell the truth”. Do not pretend to have a zebra when you actually have donkey. So:
On the database side, be clear and explicit that you have a date with time but lack the context of an offset. Add lots of documentation to explain that this is based on a faulty design, and that we are intend to store moments as seen in UTC.
On the app side, the Java side, deal only with Instant, OffsetDateTime, and ZonedDateTime objects, because within the data model we are representing moments. So use classes that represent a moment. So no use of LocalDateTime where you really mean a specific point on the timeline.
Obviously, there is some kind of a dividing line between your database side and your app side. Crossing that line is where you must convert between your Java type for a moment and your database type faking it as a moment. Where you draw that line, that transition zone, is up to you.
Option 3
As for your option # 3, yes that would be a very bad idea.
Setting such a default is not reliable. Any SysAdmin, or even an unfortunate OS update, could change the OS’s current default time zone. Like wise for the database’s current default time zone. And likewise for the JVM’s current default time zone.
So you end up three default time zones that could be changing, with each affecting various parts of your environment. And changing the current default time zone in any of those places immediately affects all other software depending on that default, not just your particular app.
As mentioned above, I recommend just the opposite: Code without any reliance on default time zones anywhere.
The one place for accessing a default time zone is maybe for presentation to the user. But even then, if the context in crucial, you must confirm the desired/expected time zone with the user. And where you do make use of a current default time zone, do so explicitly rather than implicitly. That is, make explicit calls such as ZoneId.getSystemDefault() rather than using omitted optional arguments.
I'm not sure I see a problem.
Instant values are UTC by definition, and java.sql.Timestamps have no zone other than the one implied by the database setting. You know the database is strictly UTC. This is lucky for you since it eliminates one error-prone conversion. Then, reading java.sql.Timestamps and keeping them as Instants at runtime is trivial, given java.sql.Timestamp#toInstant(). DON'T convert through LocalDateTime.
This has nothing to do with setting any "default" timezone in your application. Design and write your code so that internally (i.e. runtime memory and database) you deal ONLY with UTC (i.e. instants). The only point at which you should convert instants to anything local is at external interface points... i.e.
when outputting date/time values, either for human consumption or for other software that expects a specific timezone.
when reading date/time values from the user or another program (for which you will need to know any implied zone if it's not explicit)
Leave your "default" timezone as whatever is given to you by your environment. Then, no matter where your code is running, it will produce meaningful local dates/times.
Establish a strict rule that you deal only with UTC internally. This will make reasoning about your code MUCH simpler in the long run.
I guess the only real stumbling block is realizing that things depending on local conditions, such as day boundaries, have to be done in the local zone... but write your code to "think" UTC internally.
I did read timestamps-and-time-zones-in-postgresql and understood that a timestampTZ is stored as a UTC-timestamp with any timezone/offset converted away and lost.
So, when loading a JPA/Hibernate Entity with an OffsetDateTime-field, bound to such a timestampTZ field, where does the Offset come from?
Is it always converted into the JDBC-Connection's Timezone?
This is a kind of information-truncation where we actually lose the original timezone-information, thus we are set back to whatever timezone the JDBC-connection is bound to and thus, we are required to store the timezone-information additionally if we'd needed that?
If all of the above holds, wouldn't it be much clearer/precise to use Instant instead of OffsetBigTime, which represents an UTC-point-in-time exactly like timestampTZ is doing?
Then I would have to at least apply the "proper timezone" explicitly in code and not have it applied "magically" by some DB-connection...
when loading an JPA/Hibernate Entity with a OffsetDateTime-field, bound to such an timestampTZ field, where does the Offset come from
While I do not use JPA or Hibernate (I use straight JDBC), I would expect that you receive an OffsetDateTime where the offset is zero hours-minutes-seconds ahead/behind UTC. We might refer to this as “at UTC” for short.
You can see for yourself. Retrieve an OffsetDateTime object, and call toString. If the resulting text shows +00:00 or Z at the end, hat means an offset of zero.
we are required to store the timezone-information additionally if we'd needed that?
Yes, if you care about the time zone or offset submitted to the database, you must save that information in a second column yourself with your own extra programming. By default, Postgres uses the submitted zone or offset info to adjust into UTC, then discards that zone or offset info.
I expect most business apps do not care what the original zone or offset was. Keep in mind that the moment, the point on the timeline, is not changed. Only the wall-clock time appears different. Postgres is like a person in Iceland (where UTC is their year-round permanent time zone) receiving a call from someone in Tokyo or Montréal. If both persons look up at the clock on their walls, the person in Tokyo sees a time of day several hours ahead of the Postgres person in Iceland. The Montréal person sees a time of day on the clock hanging on heir own wall to be hours behind that of the Postgres person in Iceland.
When you retrieve that OffsetDateTime object with an offset of zero, you can easily adjust into any time zone you desire.
OffsetDateTime odt = myResultSet.getObject( … , OffsetDateTime.class ) ;
ZoneId z = ZoneId.of( "Asia/Tokyo" ) ;
ZonedDateTime zdt = odt.atZoneSameInstant( z ) ;
wouldn't it be much more clearer/precise to use Instant
Yes!!
For whatever reasons, the folks controlling the JDBC API made the odd choice in JDBC 4.2 to require support for OffsetDateTime but not require support for Instant or ZonedDateTime. These other two classes are used much more often in most apps I imagine. So the logic of their choice escapes me.
Your JDBC driver may support Instant or ZonedDateTime. Just try it and see. The JDBC 4.2 API does not forbid such support; the API makes no mention of those types. Therefore support of Instant or ZonedDateTime is optional.
Then I would have to at least apply the "proper timezone" explicitly in code and not have it applied "magically" by some db-connection...
If you are retrieving java.time objects through JDBC 4.2 compliant drivers, I would be very surprised to see them applying a zone or offset to retrieved values. I expect you will only receive OffsetDateTime objects with an offset of zero. But I do not recall his behavior being mandated in the specification one way or the other. So always test the behavior of your particular JDBC driver.
Beware that retrieving values as text, or using other middleware tools such as PgAdmin, may well inject some default zone or offset. While well-intentioned, I consider this an anti-feature, creating the illusion of a particular zone having been saved in the database when it was not in fact.
What Timezone is used to unmarshal an OffsetDateTime from a Postgres timestampTZ?
Firstly, know that an offset is merely a number of hours, minutes, and seconds ahead or behind the prime meridian. A time zone is much more. A time zone is a history of the past, present, and future changes to the offset used by the people of a particular region.
So your title’s wording is contradictory. There is no time zone involved with an OffsetDateTime, thus the name. For a time zone, use the ZonedDateTime class.
I'm using IntelliJ (Kotlin and Java language), I'm trying to get a report of time by using query from my database and send that to the browser (I'm using Postman to see the result).
When I debug my code and go through the result from the query the timezone is ok like the way I want it, its even show -7 (the difference time between me and the UTC) but when it comes to the browser (Postman) it's showing UTC time and +0000, for example:
"date": "2017-10-12T15:00:33.000+0000"
instead of
"date": "2017-10-12T15:00:33.000+0007".
I tried many options and waste around 6 hours to find the solution but nothing work.
Postgres stores a TIMESTAMP WITH TIME ZONE in UTC, discarding the passed zone after using it to make the adjustment into UTC. Note that this is Postgres-specific behavior – databases vary widely in their date-time handling, and the SQL spec barely touches on the subject.
As commented by Marlowe, if you need to remember the time zone captured from data-entry, you will need to store that in another column. Capture a proper time zone name in the format of continent/region, such as America/Montreal, Africa/Casablanca, or Pacific/Auckland. Never use the 3-4 letter abbreviation such as EST or IST as they are not true time zones, not standardized, and not even unique(!).
➠ Here's the rub: A Postgres session in an interactive tool such as pgAdmin dynamically applies a default time zone after fetching the UTC value. While well-intentioned, this is an anti-feature in my opinion as it obscures the true nature of the stored data.
Fetch the value in UTC using the modern java.time classes.
Instant instant = myResultSet.getObject( … , Instant.class ) ;
Adjust to your desired time zone.
ZoneId z = ZoneId.of( "Asia/Kolkata" ) ;
ZonedDateTime zdt = instant.atZone( z ) ;
Generate a string for the web browser using the DateTimeFormatter class. Note the automatic localization features there.
All this has been covered many times on Stack Overflow. Search to learn more.
Solution: so i found the solution to my problem here:
Overriding BeanPropertyRowMapper to Support JodaTime DateTime
in shortly, timestamp type doesn't work well with timezone so i changed it to DateTime in Joda and wrapped the result of the query by costumed Bean (from the link i posted)
If we have timestamps that contain the timezone info, like 2017-07-03T17:30:00-04:00, and parse it into java.Date or joda.DateTime.
Does it contains timezone information ?
I am asking this because i want to compare two different date instance. So if it does not contain timezone information, the day difference will be wrong with different timezones
UPDATE:
I run a quick unit test to verify, first convert date instance to milliseconds and convert back to TimeUnit after subtract these two milliseconds. The hours are different for different timezone
Both java.util.Date and Joda-Time have been supplanted by the java.time classes.
Your input string 2017-07-03T17:30:00-04:00 is in standard ISO 8601 format and has an offset-from-UTC at the end. That -04:00 means the string represents a moment four hours behind UTC.
This offset is not a time zone. A time zone is a history of offsets for a particular region. For example, America/Barbados or America/New_York.
Parse your string as an java.time.OffsetDateTime object.
OffsetDateTime odt = OffsetDateTime.parse( "2017-07-03T17:30:00-04:00" );
odt.toString(): 2017-07-03T17:30:00-04:00
You may compare OffsetDateTime instances by calling the methods IsEqual, isBefore, and isAfter.
To see the same simultaneous moment in UTC, extract an Instant.
Instant instant = odt.toInstant() ;
instant.toString(): 2017-07-03T21:30:00Z
The Z on the end is short for Zulu and means UTC.
It is going to depend on what type of DateTime you use, as of Java 8 you have these options:
A LocalDate or LocalDateTime. It is going to discard time zone information, you will wind up with a value that is 'valid' only for the local timezone. This value is ambiguous without some context as to the specific timezone of the server process which generated the value.
A ZonedDate or ZonedDateTime. This one preserves the time zone. Comparison is still going to be ambiguous: you have issues like DST or calendaring changes to contend with (depending on the range of datetime which you need to be compatible with). For sorting/comparison purposes you would probably want to convert it to a reference timescale, which is why:
An Instant represents a particular moment in time, on the absolute timescale of UTC. Any Instant is directly comparable with any other Instant and any ambiguity in values is resolved by the definition of Instant. Input values will be converted to the matching counterparts in UTC, so the original timezone (if any) will be lost even if the absolute time value will be preserved correctly. Instant is therefore not a good choice if you rely on the timezone to make decisions about location or locale, for instance.
I am trying to convert the ZonedDateTime to a Date. Looks like in the conversion, it looses the time zone and gets my local time.
System.out.println("zoneDate1::::::::::"+ZonedDateTime.now(ZoneId.of("America/Chicago")));
System.out.println("zoneDate1:Date::::::::::"+Date.from(ZonedDateTime.now(ZoneId.of("America/Chicago")).toInstant()));
The above outputs as below:
zoneDate1::::::::::2016-04-15T17:35:06.357-05:00[America/Chicago]
zoneDate1:Date::::::::::Fri Apr 15 18:35:06 EDT 2016
Is this because this is a Date type? How would i go about doing this kind of conversion and conserve the zoned time?
What is the problem? What did you expect? I see no misbehavior.
Your java.time type (ZonedDateTime) is assigned a time zone of America/Chicago.
Your JVM apparently has an assigned time zone related to east coast of North America, the clue being the EDT value seen in string output. The toString method on java.util.Date applies your JVM’s current default time zone when generating its textual representation of the date-time value. Poorly designed, this behavior is trying to be helpful but is actually confusing because you cannot actually get or set this time zone on the java.util.Date object.
At any rate, the east coast of North America (such as America/New_York time zone) is an hour ahead of America/Chicago. So you are seeing 17:xx:xx time for Chicago and 18:xx:xx for Eastern Daylight Saving Time. These values are correct.
You should call java.util.TimeZone.getDefault() when investigating the behavior of the old date-time classes.
java.time
The bigger problem is that you are even using these old date-time classes such as java.util.Date/.Calendar. They are poorly designed, confusing, and troublesome. Avoid these old classes altogether. They have been supplanted in Java 8 and later by the java.time framework.
Also, avoid using 3-4 letter zone abbreviations like EDT. These are neither standardized nor unique. Use proper time zone names in continent/region format.
Instant
To capture the current date-time in java.time, just use Instant. This class captures a moment on the timeline in UTC. Do most of your work in UTC. No need for time zones unless expected by your user when displayed in the user interface.
Instant now = Instant.now();
Database
To send to your database, first make sure you have defined the column in the table as something along the line of the SQL standard TIMESTAMP WITH TIME ZONE. By the way, support for date-time types various among databases with some doing a much better job than others.
Hopefully JDBC drivers will be updated someday to directly handle the java.time types. Until then, we must convert into a java.sql type when transferring data to/from a database. The old java.sql classes have new methods to facilitate these conversions.
java.sql.Timestamp
For a date-time value like Instant, we need the java.sql.Timestamp class and its from( Instant ) method.
java.sql.Timestamp ts = java.sql.Timestamp.from( now );
Avoid working in java.sql.Timestamp as it is part of the old poorly-designed mess that is the early Java date-time classes. Use them only for database transfer, then shift into java.time immediately.
Instant instant = ts.toInstant();
So simple, no time zones or offset-from-UTC involved. The Instant, java.sql.Timestamp, and database storage are all in UTC.
ZonedDateTime
When you do need to shift into some locality’s wall-clock time, apply a time zone.
ZoneId zoneId = ZoneId.of( "America/Chicago" ); // Or "America/New_York" and so on.
ZonedDateTime zdt = ZonedDateTime.ofInstant( instant , zoneId );
Huh? Date doesn't have time zones so, this is probably why it's failing. Maybe this is what you're looking for:
Date.from(java.time.ZonedDateTime.now().toInstant());
If your database allows you to store the timestamp along with the timezone, you should go ahead and save it as a timestamp.
If not, I would recommend that you store the date-time as per your timezone (or GMT). Add an additional column in the table to hold the value of the user's timezone.
When you fetch the value from the database, you can convert it to the user's timezone. Avoid storing just the date.