I need timestamp format for my dates in database.
For now i have joda.Datetime in database , but also in my restApi application.
I tried to create a new column , and converted the existing joda.Datetime in the other column time.LocalDateTime. Also I replaced in all code joda.DateTime with time.LocalDateTime.
It works, but when i make a get call in postman, i received a json like:
{
seconds: x1,
minutes: x2,
hours: x3,
days: x4,
........
}
I think i need a convertor, to show the timestamp as "dd-mm-yy hh-mm-ss"
I want to have timestamp format in database to be able to execute SQL standard operation and named queries on time.
In my database I have bytea type for dates. I use PostgreSQL with DBeaver.
Is this the right way, or you could recommend me another option?
Is this the right way, or you could recommend me another option?
Without experience with PostgreSQL I should say that bytea is the wrong datatype for your dates or timestamps. timestamp with time zone is good and supports SQL operations and queries. It further has the advantage that you can store OffsetDateTime (perhaps even Instant, I am not sure) directly, so you avoid formatting your timestamp for storing it. That’ll be recommended.
For a time stamp to be a time stamp date and time of day is not enough. Date and time of day will be interpreted differently in different time zones (and is even ambiguous in the fall when summer time ends and the clocks are turned backward). As far as I have understood timestamp with time zone will make sure that time stamps are stored in UTC, so will be unambiguous points in time. In Java the Instant class represents a point in time independently of time zone, so is good for timestamps. Some JDBC drivers allow you to store an Instant directly into a timestamp with time zone column, others require you to convert to OffsetDateTime first. In the latter case use
OffsetDateTime dateTimeForDatabase = yourInstant.atOffset(ZoneOffset.UTC);
Edit: Please note that the with time zone bit is a bit of a lie, as #Jon Skeet points out in a comment. The database doesn’t store a time zone or offset, it only makes sure that dates and times are stored in UTC for removing ambiguity about the point in time.
Link: Date/Time Types in the PostgreSQL docs
Related
In my PostgreSQL database i have a Timestamp column where I store data in UTC,
I fetch this date as LocalDateTime in my Java code.
When I generate an excel file I want to convert this date to the user time zone,
I found this solution:
localDateTimeFromDB.toInstant(ZoneOffset.UTC)
.atZone(ZoneId.of(userTimeZone))
.format(DateTimeFormatter.ofPattern("dd/MM/yyyy HH:mm:ss")));
It works but I want to know if is it the optimize solution ?
You could skip the manual imtermediate conversion to an Instant, but I doubt this will optimize your approach significantly.
Here's some slightly different code of a very similar length producing the same result (,I think. You haven't shown yours so far):
localDateTimeFromDb.atZone(ZoneOffset.UTC)
.withZoneSameInstant(ZoneId.of(userTimeZone))
.format(DateTimeFormatter.ofPattern("dd/MM/yyyy HH:mm:ss"));
I have a MySQL database which is storing a datetime value, let's say 2020-10-11 12:00:00. (yyyy-mm-dd hh:mm:ss format)
The type of this date (in mysql) is DATETIME
When I retrieve this data in my controller, it has the java 7 type "Date". But it adds a timezone CEST due to my locale I suspect. Here I already find confusing that when displaying this date which is not supposed to have a timezone attached it actually has... and the debugger says it is "2020-10-11 12:00:00 CEST".
My problem is that date was not stored with the CEST timezone. It was stored with the America/New_York one, for example. EDIT: What I mean with this line, is that the date was stored from new york using the timezone of new york. So, it was really 12:00:00 AM there, but here in Madrid it was 18:00:00 PM. I need that 18:00:00.
So in New York, someone did an insert at that time. Which means that the time in Europe was different. I need to calculate which time was in Europe when in America was 12AM. But my computer keeps setting that date to CEST when I retrieve it so all my parsing attempts are failing... This was my idea:
Date testingDate // This date is initialized fetching the "2020-10-11 12:00:00" from mySql
Calendar calendar = new GregorianCalendar()
calendar.setTime(testingDate)
calendar.setTimeZone(TimeZone.getTimeZone("America/New_York")
SimpleDateFormat localDateFormatter = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss")
TimeZone localTimeZone = TimeZone.getTimeZone("Europe/Madrid")
localDateFormatter.setTimeZone(localTimeZone)
String localStringDate = localDateFormatter.format(calendar.getTime())
Date newDate = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").parse(localStringDate)
Here my idea is that: I create a brand new calendar, put on it the time that I had on America and I also say hey this calendar should have the America Timezone. So when I get the time of it using a formatter from Europe it should add the corresponding hours to it. It makes a lot of sense in my head but it is just not working in the code D: And I really don't want to calculate the time difference by myself and adding or substracting the hours because that would look extremely hardcoded in my opinion.
Can any one give me some ideas of what I'm interpreting wrong or how should I tackle this problem in a better way?
Important: I'm using java 7 and grails 2.3.6.
My problem is that date was not stored with the CEST timezone. It was stored with the America/New_York one, for example.
From what I know of MySQL, this is impossible.
Calendar calendar = new GregorianCalendar()
No, don't. The calendar API is a disaster. Use java.time, the only time API in java that actually works and isn't completely broken / very badly designed. If you can't (java 7 is extremely out of date and insecure, you must upgrade!), there's the jsr310 backport. Add that dependency and use it.
Let me try to explain how to understand time first, because otherwise, any answer to this question cannot be properly understood:
About time!
There are 3 completely different concepts and they are all often simplified to mean 'time', but you shouldn't simplify them - these 3 different ideas are not really related and if you ever confuse one for another, problems always occur. You cannot convert between these 3 concepts unless you do so deliberately!
"solarflares time": These describe the moment in time as a universal global concept that something occurred or will occur. "That solar flare was observed at X" is a 'solarflares' time. Best way to store this is millis since epoch.
"appointment time": These describe a specific moment in time as it was or will be in some localized place, but stated in a globally understandable way. "We have a zoom meeting next tuesday at 5" is one of these. It's not actually constant, because locales can decide to adopt new timezones or e.g. move the 'switch date' for daylight savings. For example, if you have an appointment at your dentist on 'november 5th, at 17:00, 2021', and you want to know how many hours are left until your appointment starts, then the value should not change just because you flew to another timezone and are looking at this number from there. However, it should change if the country where you made the appointment in decided to abolish daylight savings time. That's the difference between this one and the 'solarflares' one. This one can still change due to political decisions.
"wake-up-alarm time": These describe a more mutable concept: Some way humans refer to time which doesn't refer to any specific instant or is even trying to. Think "I like to wake up at 8", and thus the amount of time until your alarm will go off next is continually in flux if you are travelling across timezones.
Now, on to your question:
I have a MySQL database which is storing a datetime value, let's say 2020-10-11 12:00:00. (yyyy-mm-dd hh:mm:ss format)
Not so fast. What exact type does that column have? What is in your CREATE TABLE statement? The key thing to figure out here is what is actually stored on disk? Is it solarflare, appointment, or wakeup-alarm? There's DATE, DATETIME and TIMESTAMP, and over the years, mysql has significantly changed how these things are stored.
I believe that, assuming you are using the modern takes on storage (So, newish mysql and no settings to explicitly emulate old behaviour), e.g. a DATETIME stores sign, year, day, hour, minute, and second under the hood, which means it is wakeup alarm style: There is no timezone info in this, therefore, the actual moment in time is not set at all and depends on who is asking.
Contrast to TIMEZONE which is stored as UTC epoch seconds, so it's solarflares time, and it doesn't include any timezone at all. You'd have to store that separately. As far as I know, the most useful of the 3 time representations (appointment time) is not a thing in mysql. That's very annoying; mysql tends to be, so perhaps par for the course.
In java, all 3 concepts exist:
solarflares time is java.time.Instant. java.util.Date, java.sql.Timestamp, System.currentTimeMillis() are also solarflares time. That 'Date' is solarflares timestamp is insane, but then there is a reason that API was replaced.
appointment time is java.time.ZonedDateTime
wakeup-alarm time is java.time.LocalDateTime.
When I retrieve this data in my controller, it has the java 7 type "Date".
Right. So, solarflares time.
Here's the crucial thing:
If the type of time stored in MySQL does not match the type of time on the java side, pain happens.
It sure sounds like you have wakeup-alarm time on disk, and it ends up on java side as solarflares time. That means somebody involved a timezone conversion. Could have happened internally in mysql, could have happened in flight between mysql and the jdbc driver (mysql puts it 'on the wire' converted), or the jdbc driver did it to match java.sql.Timestamp.
The best solution is not to convert at all, and the only real way to do that is to either change your mysql table def to match java's, so, make that CREATE TABLE (foo TIMESTAMP), as TIMESTAMP is also solarflares time, or, to use, at the JDBC level, not:
someResultSet.getTimestamp(col);
as that returns solarflares time, but:
someResultSet.getObject(col, LocalDateTime.class);
The problem is: Your JDBC driver may not support this. If it doesn't, it's a crappy JDBC driver, but that happens sometimes.
This is still the superior plan - plan A. So do not proceed to the crappy plan B alternative unless there is no other way.
Plan B:
Acknowledge that conversion happens and that this is extremely annoying and errorprone. So, make sure you manage it, carefully and explicitly: Make sure the proper SET call is set up so that mysql's sense of which timezone we are at matched. Consider adding storing the timezone as a column in your table if you really need appointment time. etcetera.
Thanks to #rzwitserloot I was able to find out a solution.
First I'll get the data from the database. I'll get rid of any timezone added by the driver / mysql by converting it to a LocalDateTime. Then, I'll create a new ZonedDateTime using the Timezone that was used when storing the data in the database.
Once I have a ZonedDateTime, it is time to convert it using my current timezone. I'll get a new ZonedDateTime object with the proper time.
Then I just add a few more lines to convert it back to my main "Date" class:
I've used the ThreeTen backport as suggested.
Date dateMySQL //Initialized with the date from mysql
Calendar calendar = new GregorianCalendar()
calendar.setTime(dateMySQL)
org.threeten.bp.LocalDateTime localDateTime = org.threeten.bp.LocalDateTime.of(calendar.get(Calendar.YEAR), calendar.get(Calendar.MONTH)+1,
calendar.get(Calendar.DAY_OF_MONTH), calendar.get(Calendar.HOUR_OF_DAY), calendar.get(Calendar.MINUTE),
calendar.get(Calendar.SECOND))
String timezone //Initialized with the timezone from mysql (Ex: "America/New_York")
ZonedDateTime zonedDateTime = ZonedDateTime.of(localDateTime, ZoneId.of(timezone))
ZonedDateTime utcDate = zonedDateTime.withZoneSameInstant(ZoneId.of("Europe/Madrid"))
calendar.setTimeInMillis(utcDate.toInstant().toEpochMilli())
Date desiredDate = calendar.time
dateMySQL: "2020-10-11 10:00:00" // CEST due to my driver
timezone: "America/New_York"
desiredDate: "2020-10-11 19:00:00" // CEST Yay!
Introduction
I realise similar questions have been asked here before. But I still needed to do a bit more testing for my use case - and am I little surprised/confused by the outcome.
The following is a description (with code) of the test and results.
I'm working on an application that:
Records event history (things that happened at a point in time).
This history is immutable. Once an event is written, we do not change it.
We report on this history.
Schedules future, possibly recurrent, actions. For example, "every Friday do this check."
So a clear understanding of time data is very important.
I am trying understand the timestamp types in MySQL (datetime and timestamp) and how they interact with Java code and ISO8601 timestrings.
I've read the MySQL docs around its Timestamp type - how it converts a time value to store it as UTC and, on retrieval, converts that UTC value to the server or session time zone. That sounded like good "point-in-time" storage.
The datetime MySQL type is bit more opaque. This article (http://code.openark.org/blog/mysql/timestamp-vs-datetime-which-should-i-be-using) considers it to be not so different from a string.
I repeated the "SELECT NOW() + 0;" query and got the same kind of answer as the article.
To try to clarify things, I wrote a small Java class that wrote to a single database table comprised of 3 columns:
tz (String denoting the standard timezone id). UTC is used the tests - but it is not a timezone.
mydatetime (Datetime column)
mytimestamp (Timestamp column)
Java code
The Java code is located here: https://gist.github.com/iliomad/c6d8a50613fa99e2d079b8ad3b9eca4d
Test 1
I created 3 different ISO8601 timestamp strings - all for the same point in time - but with different offsets/timezones.
Each was written the to database, retrieved and printed out.
To start with I set the MySQL database server time zone to "+00:00".
Output
Writing times to the database.
UTC
2018-04-13T11:12:00Z, 1523617920
Europe/Amsterdam
2018-04-13T13:12:00+02:00, 1523617920
Asia/Calcutta
2018-04-13T16:42:00+05:30, 1523617920
Reading stored times from database.
Description, Datetime column (epoch seconds), Datetime column (as ISO string), Timestamp column (as epoch seconds), Timestamp column (as ISO string)
UTC, 1523617920, 2018-04-13T11:12:00Z, 1523617920, 2018-04-13T11:12:00Z
Europe/Amsterdam, 1523617920, 2018-04-13T13:12+02:00[Europe/Amsterdam], 1523617920, 2018-04-13T13:12+02:00[Europe/Amsterdam]
Asia/Calcutta, 1523617920, 2018-04-13T16:42+05:30[Asia/Calcutta], 1523617920, 2018-04-13T16:42+05:30[Asia/Calcutta]
All looks good. The epoch seconds, when put into something like https://www.epochconverter.com/, agree with the ISO8601 time strings.
Test 2
I then changed the timezone of the MySQL server from UTC (+00:00) to Europe/Amsterdam (+02:00) and read the stored times again.
Output
Reading stored times from database.
Description, Datetime column (epoch seconds), Datetime column (as ISO string), Timestamp column (as epoch seconds), Timestamp column (as ISO string)
UTC, 1523617920, 2018-04-13T11:12:00Z, 1523625120, 2018-04-13T13:12:00Z
Europe/Amsterdam, 1523617920, 2018-04-13T13:12+02:00[Europe/Amsterdam], 1523625120, 2018-04-13T15:12+02:00[Europe/Amsterdam]
Asia/Calcutta, 1523617920, 2018-04-13T16:42+05:30[Asia/Calcutta], 1523625120, 2018-04-13T18:42+05:30[Asia/Calcutta]
My expectation was the Datetime column would be affected - thinking it did not store timezone information. However, it was the Timestamp column that moved.
Conclusion
We will not routinely be changing our server timezone.
I'm trying to see which MySQL data type best represents a point in time - something I can count on as accurate and unchanging (and not head-wrecking). On the basis of the above, if I supply point-in-time info in the form of an ISO8601 string, the Datetime MySQL type retains the supplied information.
It's perfectly possible that my test code is incorrect and/or my interpretation of the results is.
Could do with a little guidance here.
In the test case above, is the Datetime MySQL type doing a better job of storing the point in time information?
After a bit more tweaking, I got a Java class that manages to store and retrieve point-in-time information. I'm still a bit jittery on this - so many parts want to have a say in the time and that makes me uncomfortable.
The class is on the original gist here: https://gist.github.com/iliomad/c6d8a50613fa99e2d079b8ad3b9eca4d (and revisions show changes).
I can change the MySQL server timezone to anything I wish. I still get out what I put in.
A query directly in the MySQL query manager also behaves - the timestamp column as unix epoch holds steady. That's what I expect.
select tz, mydatetime, unix_timestamp(mydatetime) as mydatetime_as_epoch, unix_timestamp(mytimestamp) from basic_time;
tz = timezone id (String); mydatetime (Datetime column); mytimestamp (Timestamp column)
Timestamp MySQL type is a timestamp. I don't need to mess with it.
Java 8 java.sql.Timestamp class fromInstant and toInstant methods claim to maintain the same point in time.
I tried Instant on the setObject and getObject preparedStatement methods. MySQL connector/J just does not accept them - I dug into the source code for a look.
Datetime MySQL type I'm still a bit confused by (I don't know what it is) but knowing the timezone I can maintain the point-in-time information.
I tried both the MySQL Connector/J 5.1.46 and 8.0.11 versions. Same behaviour for each.
The JDBC parameter useLegacyDatetimeCode=false seems to be important. I need to run more tests with and without this to confirm.
Not quite up to analysing this completely just yet. Not fun trying to figure this out.
I am wondering how the conversion works in this. MySQL server (5.6) treats TIMESTAMP as zone-adjusted (and internally stored in/retrieved from UTC). It also treats DATETIME as having no zone.
On the Java side, I am recommended to read into java.sql.Timestamp in either case. Is there a zone-type conversion taking place (when going through MySQL-connector 5.1.37) from MySQL's DATETIME to java.sql.Timestamp (such as to apply the client system zone) ?
In the end, there is only one zone for my server and clients, and so I maintain a specific ZoneId (in app code) to get to ZonedDateTime. But I would like to work with ZonedDateTime, going back and forth to the database stored as DATETIME. A simple example of conversion will be appreciated!
Let's address each question you have. First: Is there a zone-type conversion taking place (when going through MySQL-connector 5.1.37) from MySQL's DATETIME to java.sql.Timestamp (such as to apply the client system zone)?
First off, I presume that you are using the getTimestamp(int) method from the connector. I could not find an official source that showed me an enlightening answer; however, there was this question in which the answer stated: When you call getTimestamp(), MySQL JDBC driver converts the time from GMT into default timezone if the type is timestamp. It performs no such conversion for other types.
However, in this version of the method, it uses an underlying Calendar to convert the Timestamp to the TimeZone specified, if the underlying database doesn't store time zone information. This may be the solution to your second question, as long as you knew the time zone at which the value was stored (which you do). But if it is not, it seems that with the first method there is no conversion taking place, at least when it retrieves the DATETIME. Speaking about your second question:But I would like to work with ZonedDateTime, going back and forth to the database stored as DATETIME.
It makes me think that there is a way to do this as long as you knew which time zone you are converting from. As we have previously stated, you and your clients are only working with one ZoneId, which is totally fine. However, this answer is provided to work with more time zones. Multiple ZoneId's can be achieved if you were to store the ZoneId of the connection in the database; retrieving it as well as the DATETIME and finally processing these values into a ZonedDateTime. You could store the ZoneIds into the database using the ID's of the ZoneId class (if you wanted to).
Timestamp t = resultSet.getTimestamp(timestampColumnId);
ZoneId zoneId = ZoneId.of(resultSet.getString(zoneColumnId), ZoneId.SHORT_IDS);
ZonedDateTime d = ZonedDateTime.ofInstant(t.toInstant(), zoneId);
Or, you could just store the DATETIME as a TIMESTAMP in the database as ZZ Coder suggests in his answer stated above. But, you could just use the ZoneId you have hard-coded as such:
Timestamp t = resultSet.getTimestamp(timestampColumnId);
ZonedDateTime d = ZonedDateTime.ofInstant(t.toInstant(), zoneId);
EDIT
Looking at the source code, on get or set calls using the getTimestamp(int, Calendar) or the setTimestamp(int, Timestamp, Calendar) function, the timezone of the Calendar is used. However, in some cases with TIMESTAMP, when a Calendar is not used, the JDBC then uses the time zone of the server. And according to the original poster, it worked (see comment below).
What is the proper way to save date or time based data in the database?
What are the proper "field mappings" for java to postgresql(or to some other database)?
That data should be stored in utc format without timezones.
-> timestamp and date based stuff fails in here, those will add current timezone (http://docs.oracle.com/javase/7/docs/api/java/util/Date.html)
-> what are the other options?
should I use "plain epoch/integer" column and other column for timezone? But then I cannot use all the functions etc. that the database is providing for me.
I could use hibernate with some jodatime magic, but in my current stack I don't have hibernate in use.
Possible solutions:
1). Change the computer/java timezone -> java will in the UCT (eg. export TZ="GMT" or -Duser.timezone=UCT)
2). Use epoch/Integer/Long values in date/time fields / types -> works but now I cannot use build in database functions.
3). Use Jodatime with custom hibernate datatypes?
4). Use Java8 new time and date apis?
In most cases, it is best to use the Postgres data type timestamptz (short for timestamp with time zone) when dealing with multiple time zones or when you want to save all timestamps as UTC.
Don't let the name mislead you, the time zone is not actually saved. But (as opposed to timestamp [without time zone]) the time zone from textual input is taken into account as modifier to compute the actual UTC timestamp value, which is saved.
On output, the text representation of the value is formatted according to your current time zone setting: timestamp is shifted and the according time zone modifier attached to it.
Note that timestamps without appended time zone are interpreted according to the current time zone setting of your session. If you want to enter a literal UTC value disregarding the current time zone, it has to be:
'2014-08-21 16:39:09+0'::timestamptz
not:
'2014-08-21 16:39:09'::timestamptz -- would assume current time zone
Detailed explanation in this related answer:
Ignoring timezones altogether in Rails and PostgreSQL
As you say, it would be best to store dates as UTC on the database. In oracle you can use a DATE or TIMESTAMP datatype. You can then use the java layer to present your dates in local time to the user and with a java.sql.timestamp column. Joda is essentially built in to the latest version of java so definitely use that for any conversions etc. The alternative would be to store timestamp with timezone in oracle and perhaps use oracle date functions in your sql and stored procedures to convert the date as required. We do the former, but it may depend on your team (db people vs java people) and your audience - are there likely to be lots of different timezones in the user base or is timing on the DST changeover going to break your app.
If you can describe particular situations you are concerned about I'm sure someone will help out. Storing your data in UTC will at least ensure that your data is solid but may require many conversions in the presentation layer.
There is not a single correct way but I my opinion is that you should store time as a long unix timestamp and single dates as epoch days. Java 8 has nice functions to deal with them. Avoid locking yourself with jodatime and hibernate just to manage a date.
What do you mean functions that the database is providing you?
You can always do SQL selects with integers and long since they are called from your program.
If you need a lot of manual use of the database (not programmatic) then you may want to use human readable dates.