I want to use Instant type to put it to MySQL database (timestamp field). Unfortunately, when using POJO and Record#from(), jOOQ doesn't let me do that for some reason. I have the following to my gradle configuration:
forcedTypes {
forcedType {
name = "Instant"
types = "timestamp"
}
}
The code is being generated correctly, but doesn't work and gives me errors in runtime:
Data truncation: Incorrect datetime value: '2021-01-16 05:01:25.457+00:00' for column `test`.`messages`.`time_sent` at row 1
I tried to add my own converter and binder, but they don't seem to work if name is set in gradle config, as a warning appears during build. But without name I can't get jOOQ to generate Instant for timestamp field.
Is there a way to use Instant with SQL timestamp when using jOOQ's POJO?
MySQL's timestamp data type is a bit, well, idiotic.
It stores time as seconds since the epoch (jan 1st 1970, UTC, midnight), in a 32-bit integer. This has a few issues:
There is no room for milliseconds. Your Instant has .457 milliseconds and this needs to be stripped out, which is why JOOQ is refusing to do this; that's destroying data. I assume you don't care about those millis, but JOOQ doesn't know that. You can try to strip the millis out of the instant, if you must, JOOQ would presumably allow saving an Instant, if that Instant has the property that it the epochmillis that it wraps around is divisible by 1000 (has no millis part). Except, that was annoying, so at some point, eventhough it's stored as 32-bit seconds-since-epoch, the data type now also contains, separately, a fractional second, from 0 to 999999. Either you have an old version of MySQL or the underlying table structure that doesn't allow this, or JOOQ isn't aware that MySQL does at least support millis.
At 2038-01-19 03:14:07 UTC, your app explodes. That's it. That's the last timestamp representable by a MySQL TIMESTAMP object. We're less than 18 years removed from this. Therefore this datatype is effectively unusable, and you should use something else. (This may not sound believable to you. Peruse MySQL8's user manual §11.2.2 if you need some convincing, straight from the source). Java's core instant storage system doesn't suffer from the dreaded Y2K38, as java uses millis-since-epoch in 64-bit; we got a few billion years to go before that runs out of numbers.
Note that the printed message is a bit misleading. Instants are stored as milliseconds since epoch and do not have a timezone, that +00.00 in the printout suggests that it does. It doesn't - and thus the fact that mysql's TIMESTAMP type also doesn't isn't the problem.
Solutions
The best solution, by far, is to use a database engine that isn't broken like this. For example, try postgres.
A distant second solution is to peruse JOOQ issue #9665 where #lucaseder (core contributor of JOOQ; he's the genius doing the magic over there :P) says it doesn't work yet, but there's some code there you may be able to use.
Note that if you actually care about zones, this becomes a lot more complicated. There are 3 different ways to represent time; these are distinct and cannot be converted to each other without additional info and caveats; most tools incorrectly silently convert them anyway, and pain ensues when this happens. Thus, you may want to think about which of the 3 kinds of time you have here:
solarflares time: A moment time something happened or will happen, such as a solarflare. If some political entity decides to change timezone, it has no effect. The 'time until event occurs' / 'time since event occurred' goes up by 1 second every second no matter what happens with timezones. In java terms, java.time.Instant.
appointment time: I call my barber in Amsterdam and I have an appointment on Jan 5th, 2023, 14:00. You'd think this is like solarflares time, but, no. If the dutch parliament slams the gavel and adopts a law that the Netherlands will no longer observe daylight savings and will remain in summertime, then the moment that gavel comes down, the # of seconds until my appointment goes up by an hour (is it it down by an hour?). This is not exotic at all - see EU Directive 2000/84/EC - it is, in fact, likely. Solarflares time should not shift like this, and appointment time does. Best represented as year+month+day+hour+minute+second+millis+a full zone (which is Europe/Amsterdam, not silly useless stuff like +0800 or PST). In java terms, ZonedDateTime.
Alarmclock time: Just years, months, day, hour, minute, second. That's it - it doesn't represent anything particular but just the concept. If I set my alarm to wake me up at 08 in the morning and I take a trip across the pacific, the time until my alarm goes off should change drastically as I move timezones. In java terms, LocalDateTime and friends.
I'm assuming you have a solarflares time (for example, to track 'user X change their password at this time'). This answer assumes you meant that. If you did not, my advice would change a bit. Mostly that whole 'do not use mysql' thing would become even stronger: What you really want is the datatype TIMESTAMP WITH TIME ZONE such as postgres has.
Related
A client's API that I am integrating with returns a time with the time zone as a short id (3 letter id). e.g.
9:00 MST
13:30 EDT
17:00 MDT
I realize this is bad practice (especially with the daylight savings), and short ids should be avoided, but unfortunately I have no way to get the client to change their api. Is there a clean way to convert these short ids to a property ZoneId? ZoneId.SHORT_IDS looks like the closest, but it doesn't include daylights savings. Worst case, I think I can just make my own map of short ids to ZoneIds since it's limited to the US
Time with zone makes no sense
13:30 EDT
A time-of-day plus a time zone makes no sense. Without the context of a date, there is no real meaning here.
The SQL-92 standard declares a TIME WITH TIME ZONE type, but neglects to define its meaning. I am not alone in being mystified by the SQL committee’s intention. And also, importantly, that data type is a misnomer: the SQL standard means offset when they say “time zone”.
Java offers an OffsetTime class. I presume this class exists merely for mapping through JDBC to that screwy SQL type of TIME WITH TIME ZONE. The Javadoc offers no real explanation as to the meaning of this class.
So what is your goal? I suggest you edit your Question to indicate what kind of processing you intend to do with such inputs. Perhaps we can guide you from there.
Pseudo time zones are not unique
You asked:
Is there a clean way to convert these short ids to a property ZoneId?
No.
These 2-4 character pseudo time zones are not defined, are not standardized, and are not unique.
CST — Do you mean Central Standard Time in the Americas? Or do you mean China Standard Time? Or Cuba Standard Time?
IST — Do you mean India Standard Time? Or Ireland Standard Time?
PST — Pacific Standard Time or Pitcairn Standard Time?
BST — Bangladesh Standard Time, Bougainville Standard Time, or British Summer Time?
AMST — Amazon Summer Time or Armenia Summer Time?
So, no you cannot cleanly determine a time zone from a pseudo zone. You can only guess. If you know for certain the domain of possible values you expect to receive, and you are certain as to their intended actual time zone, then you can create your own mapping.
But even then, there is no class in Java nor data type in SQL to represent a date with time zone. You would have to convert that time zone to an offset from UTC. And for that conversion, you would need a date to determine a moment to determine the offset in use at that moment by the people of that zone… which leads us back in a circle to point at the top of this Answer: A time with only a zone (or offset), but without the context of a date, makes no sense.
Reject senseless data
You said:
but unfortunately I have no way to get the client to change their api
Providing a time-of-day with an ambiguous pseudo time zone is like providing a money amount tagged with "dollar" while asking to convert to "francs".
We don't know if "dollar" means Canada Dollar (CAD), United States Dollar (USD), or a Australian dollar (AUD), or any of the twenty currencies with that name.
We don't know if franc means the franc CFA in West Africa, the Swiss franc, or some other currency with that name.
We don't have a date on which to look up a conversion rate.
So what would you do with such a request? What can you do? You would have to either ask for much more explicit definitions, or you would have to refuse the request.
I have a MySQL database which is storing a datetime value, let's say 2020-10-11 12:00:00. (yyyy-mm-dd hh:mm:ss format)
The type of this date (in mysql) is DATETIME
When I retrieve this data in my controller, it has the java 7 type "Date". But it adds a timezone CEST due to my locale I suspect. Here I already find confusing that when displaying this date which is not supposed to have a timezone attached it actually has... and the debugger says it is "2020-10-11 12:00:00 CEST".
My problem is that date was not stored with the CEST timezone. It was stored with the America/New_York one, for example. EDIT: What I mean with this line, is that the date was stored from new york using the timezone of new york. So, it was really 12:00:00 AM there, but here in Madrid it was 18:00:00 PM. I need that 18:00:00.
So in New York, someone did an insert at that time. Which means that the time in Europe was different. I need to calculate which time was in Europe when in America was 12AM. But my computer keeps setting that date to CEST when I retrieve it so all my parsing attempts are failing... This was my idea:
Date testingDate // This date is initialized fetching the "2020-10-11 12:00:00" from mySql
Calendar calendar = new GregorianCalendar()
calendar.setTime(testingDate)
calendar.setTimeZone(TimeZone.getTimeZone("America/New_York")
SimpleDateFormat localDateFormatter = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss")
TimeZone localTimeZone = TimeZone.getTimeZone("Europe/Madrid")
localDateFormatter.setTimeZone(localTimeZone)
String localStringDate = localDateFormatter.format(calendar.getTime())
Date newDate = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").parse(localStringDate)
Here my idea is that: I create a brand new calendar, put on it the time that I had on America and I also say hey this calendar should have the America Timezone. So when I get the time of it using a formatter from Europe it should add the corresponding hours to it. It makes a lot of sense in my head but it is just not working in the code D: And I really don't want to calculate the time difference by myself and adding or substracting the hours because that would look extremely hardcoded in my opinion.
Can any one give me some ideas of what I'm interpreting wrong or how should I tackle this problem in a better way?
Important: I'm using java 7 and grails 2.3.6.
My problem is that date was not stored with the CEST timezone. It was stored with the America/New_York one, for example.
From what I know of MySQL, this is impossible.
Calendar calendar = new GregorianCalendar()
No, don't. The calendar API is a disaster. Use java.time, the only time API in java that actually works and isn't completely broken / very badly designed. If you can't (java 7 is extremely out of date and insecure, you must upgrade!), there's the jsr310 backport. Add that dependency and use it.
Let me try to explain how to understand time first, because otherwise, any answer to this question cannot be properly understood:
About time!
There are 3 completely different concepts and they are all often simplified to mean 'time', but you shouldn't simplify them - these 3 different ideas are not really related and if you ever confuse one for another, problems always occur. You cannot convert between these 3 concepts unless you do so deliberately!
"solarflares time": These describe the moment in time as a universal global concept that something occurred or will occur. "That solar flare was observed at X" is a 'solarflares' time. Best way to store this is millis since epoch.
"appointment time": These describe a specific moment in time as it was or will be in some localized place, but stated in a globally understandable way. "We have a zoom meeting next tuesday at 5" is one of these. It's not actually constant, because locales can decide to adopt new timezones or e.g. move the 'switch date' for daylight savings. For example, if you have an appointment at your dentist on 'november 5th, at 17:00, 2021', and you want to know how many hours are left until your appointment starts, then the value should not change just because you flew to another timezone and are looking at this number from there. However, it should change if the country where you made the appointment in decided to abolish daylight savings time. That's the difference between this one and the 'solarflares' one. This one can still change due to political decisions.
"wake-up-alarm time": These describe a more mutable concept: Some way humans refer to time which doesn't refer to any specific instant or is even trying to. Think "I like to wake up at 8", and thus the amount of time until your alarm will go off next is continually in flux if you are travelling across timezones.
Now, on to your question:
I have a MySQL database which is storing a datetime value, let's say 2020-10-11 12:00:00. (yyyy-mm-dd hh:mm:ss format)
Not so fast. What exact type does that column have? What is in your CREATE TABLE statement? The key thing to figure out here is what is actually stored on disk? Is it solarflare, appointment, or wakeup-alarm? There's DATE, DATETIME and TIMESTAMP, and over the years, mysql has significantly changed how these things are stored.
I believe that, assuming you are using the modern takes on storage (So, newish mysql and no settings to explicitly emulate old behaviour), e.g. a DATETIME stores sign, year, day, hour, minute, and second under the hood, which means it is wakeup alarm style: There is no timezone info in this, therefore, the actual moment in time is not set at all and depends on who is asking.
Contrast to TIMEZONE which is stored as UTC epoch seconds, so it's solarflares time, and it doesn't include any timezone at all. You'd have to store that separately. As far as I know, the most useful of the 3 time representations (appointment time) is not a thing in mysql. That's very annoying; mysql tends to be, so perhaps par for the course.
In java, all 3 concepts exist:
solarflares time is java.time.Instant. java.util.Date, java.sql.Timestamp, System.currentTimeMillis() are also solarflares time. That 'Date' is solarflares timestamp is insane, but then there is a reason that API was replaced.
appointment time is java.time.ZonedDateTime
wakeup-alarm time is java.time.LocalDateTime.
When I retrieve this data in my controller, it has the java 7 type "Date".
Right. So, solarflares time.
Here's the crucial thing:
If the type of time stored in MySQL does not match the type of time on the java side, pain happens.
It sure sounds like you have wakeup-alarm time on disk, and it ends up on java side as solarflares time. That means somebody involved a timezone conversion. Could have happened internally in mysql, could have happened in flight between mysql and the jdbc driver (mysql puts it 'on the wire' converted), or the jdbc driver did it to match java.sql.Timestamp.
The best solution is not to convert at all, and the only real way to do that is to either change your mysql table def to match java's, so, make that CREATE TABLE (foo TIMESTAMP), as TIMESTAMP is also solarflares time, or, to use, at the JDBC level, not:
someResultSet.getTimestamp(col);
as that returns solarflares time, but:
someResultSet.getObject(col, LocalDateTime.class);
The problem is: Your JDBC driver may not support this. If it doesn't, it's a crappy JDBC driver, but that happens sometimes.
This is still the superior plan - plan A. So do not proceed to the crappy plan B alternative unless there is no other way.
Plan B:
Acknowledge that conversion happens and that this is extremely annoying and errorprone. So, make sure you manage it, carefully and explicitly: Make sure the proper SET call is set up so that mysql's sense of which timezone we are at matched. Consider adding storing the timezone as a column in your table if you really need appointment time. etcetera.
Thanks to #rzwitserloot I was able to find out a solution.
First I'll get the data from the database. I'll get rid of any timezone added by the driver / mysql by converting it to a LocalDateTime. Then, I'll create a new ZonedDateTime using the Timezone that was used when storing the data in the database.
Once I have a ZonedDateTime, it is time to convert it using my current timezone. I'll get a new ZonedDateTime object with the proper time.
Then I just add a few more lines to convert it back to my main "Date" class:
I've used the ThreeTen backport as suggested.
Date dateMySQL //Initialized with the date from mysql
Calendar calendar = new GregorianCalendar()
calendar.setTime(dateMySQL)
org.threeten.bp.LocalDateTime localDateTime = org.threeten.bp.LocalDateTime.of(calendar.get(Calendar.YEAR), calendar.get(Calendar.MONTH)+1,
calendar.get(Calendar.DAY_OF_MONTH), calendar.get(Calendar.HOUR_OF_DAY), calendar.get(Calendar.MINUTE),
calendar.get(Calendar.SECOND))
String timezone //Initialized with the timezone from mysql (Ex: "America/New_York")
ZonedDateTime zonedDateTime = ZonedDateTime.of(localDateTime, ZoneId.of(timezone))
ZonedDateTime utcDate = zonedDateTime.withZoneSameInstant(ZoneId.of("Europe/Madrid"))
calendar.setTimeInMillis(utcDate.toInstant().toEpochMilli())
Date desiredDate = calendar.time
dateMySQL: "2020-10-11 10:00:00" // CEST due to my driver
timezone: "America/New_York"
desiredDate: "2020-10-11 19:00:00" // CEST Yay!
What is the proper way to save date or time based data in the database?
What are the proper "field mappings" for java to postgresql(or to some other database)?
That data should be stored in utc format without timezones.
-> timestamp and date based stuff fails in here, those will add current timezone (http://docs.oracle.com/javase/7/docs/api/java/util/Date.html)
-> what are the other options?
should I use "plain epoch/integer" column and other column for timezone? But then I cannot use all the functions etc. that the database is providing for me.
I could use hibernate with some jodatime magic, but in my current stack I don't have hibernate in use.
Possible solutions:
1). Change the computer/java timezone -> java will in the UCT (eg. export TZ="GMT" or -Duser.timezone=UCT)
2). Use epoch/Integer/Long values in date/time fields / types -> works but now I cannot use build in database functions.
3). Use Jodatime with custom hibernate datatypes?
4). Use Java8 new time and date apis?
In most cases, it is best to use the Postgres data type timestamptz (short for timestamp with time zone) when dealing with multiple time zones or when you want to save all timestamps as UTC.
Don't let the name mislead you, the time zone is not actually saved. But (as opposed to timestamp [without time zone]) the time zone from textual input is taken into account as modifier to compute the actual UTC timestamp value, which is saved.
On output, the text representation of the value is formatted according to your current time zone setting: timestamp is shifted and the according time zone modifier attached to it.
Note that timestamps without appended time zone are interpreted according to the current time zone setting of your session. If you want to enter a literal UTC value disregarding the current time zone, it has to be:
'2014-08-21 16:39:09+0'::timestamptz
not:
'2014-08-21 16:39:09'::timestamptz -- would assume current time zone
Detailed explanation in this related answer:
Ignoring timezones altogether in Rails and PostgreSQL
As you say, it would be best to store dates as UTC on the database. In oracle you can use a DATE or TIMESTAMP datatype. You can then use the java layer to present your dates in local time to the user and with a java.sql.timestamp column. Joda is essentially built in to the latest version of java so definitely use that for any conversions etc. The alternative would be to store timestamp with timezone in oracle and perhaps use oracle date functions in your sql and stored procedures to convert the date as required. We do the former, but it may depend on your team (db people vs java people) and your audience - are there likely to be lots of different timezones in the user base or is timing on the DST changeover going to break your app.
If you can describe particular situations you are concerned about I'm sure someone will help out. Storing your data in UTC will at least ensure that your data is solid but may require many conversions in the presentation layer.
There is not a single correct way but I my opinion is that you should store time as a long unix timestamp and single dates as epoch days. Java 8 has nice functions to deal with them. Avoid locking yourself with jodatime and hibernate just to manage a date.
What do you mean functions that the database is providing you?
You can always do SQL selects with integers and long since they are called from your program.
If you need a lot of manual use of the database (not programmatic) then you may want to use human readable dates.
I have 2 different computers, each with different TimeZone.
In one computer im printing System.currentTimeMillis(), and then prints the following command in both computers:
System.out.println(new Date(123456)); --> 123456 stands for the number came in the currentTimeMillis in computer #1.
The second print (though typed hardcoded) result in different prints, in both computers.
why is that?
How about some pedantic detail.
java.util.Date is timezone-independent. Says so right in the javadoc.
You want something with respect to a particular timezone? That's java.util.Calendar.
The tricky part? When you print this stuff (with java.text.DateFormat or a subclass), that involves a Calendar (which involves a timezone). See DateFormat.setTimeZone().
It sure looks (haven't checked the implementation) like java.util.Date.toString() goes through a DateFormat. So even our (mostly) timezone-independent class gets messed up w/ timezones.
Want to get that timezone stuff out of our pure zoneless Date objects? There's Date.toGMTString(). Or you can create your own SimpleDateFormatter and use setTimeZone() to control which zone is used yourself.
why is that?
Because something like "Oct 4th 2009, 14:20" is meaningless without knowing the timezone it refers to - which you can most likely see right now, because that's my time as I write this, and it probably differs by several hours from your time even though it's the same moment in time.
Computer timestamps are usually measured in UTC (basically the timezone of Greenwich, England), and the time zone has to be taken into account when formatting them into something human readable.
Because that milliseconds number is the number of milliseconds past 1/1/1970 UTC. If you then translate to a different timezone, the rendered time will be different.
e.g. 123456 may correspond to midday at Greenwich (UTC). But that will be a different time in New York.
To confirm this, use SimpleDateFormat with a time zone output, and/or change the timezone on the second computer to match the first.
javadoc explains this well,
System.currentTimeMillis()
Note that while the unit of time of the return value is a millisecond, the granularity of the value depends on the underlying operating system and may be larger. For example, many operating systems measure time in units of tens of milliseconds.
See https://docs.oracle.com/javase/7/docs/api/java/util/Date.html#toString().
Yes, it's using timezones. It should also print them out (the three characters before the year).
I need help. I have been trying to figure out why java util date is 5 hours behind after converting from C# ticks.
in C#, the date is 6/8/2013 11:02:07 AM, I convert this date into ticks then pass it to java as long.
code snippet:
taken:
- long TICKS_AT_EPOCH = 621355968000000000L;
- long TICKS_PER_MILLISECOND = 10000;
java.util.Date date = new java.util.Date((ctime - TICKS_AT_EPOCH) / TICKS_PER_MILLISECOND);
Now java util date is Sat Jun 08 06:02:07 CDT 2013
Notice that the hour is 5 hours difference.
Any suggestions why?
You are constructing a java.util.Date based on milliseconds since 1/1/1970 UTC. You appear to be correcting from the fact that .net's System.DateTime.Ticks are based on 1/1/0001 and are 10,000 ticks to a millisecond. That is correct, but you have forgotten to adjust to UTC.
In .Net, the value coming from DateTime.Ticks is highly dependent on the DateTime.Kind property. There are three possible kinds of DateTime values.
DateTimeKind.Utc - This kind means that the value represents UTC time. It usually comes from a call to DateTime.UtcNow, but can also be constructed directly, and often is. For example, you might be retrieving UTC times from a database. You can feed the ticks from here directly into your conversion, and it will work.
DateTimeKind.Local - This usually comes from a call to DateTime.Now. The values are representative of the local time zone. You will need to convert to UTC before checking the ticks. You can do the following:
DateTime dt = DateTime.Now;
int utcTicks = dt.ToUniversalTime().Ticks;
Be aware that if the time happens during a daylight saving "fall-back" style transition, the result might be incorrect. The DateTime class has no idea about time zones. It just reflects the current local clock. If the value in dt is ambiguous, ToUniversalTime() will assume that the value is representative of standard time, even if you just retrieved it while in daylight time. This is just one of the many confusing and probablematic aspects of DateTime in .net.
DateTimeKind.Unspecified - This is the most common kind of DateTime you will encounter, and usually comes from DateTime.Parse() or a constructor like new DateTime(...). Unfortunately, there is nothing in here that will tell you about the time zone these dates are representative of. You can still try calling .ToUniversalTime(), but the framework will make the assumption that these times are representative of your local time zone, as if the kind was Local. That assumption could be completely wrong, depending on how you sourced the data. There really is no safe way to transform an Unspecified DateTime to a UTC value (ticks or otherwise).
There are some solutions, such as using DateTimeOffset instead of DateTime, or using the Noda Time library instead of the built-in types. You can read more about these problems here and here.
The time is not 5 hours behind, it's exactly the same time. The problem is with the way you print it.
You need to tell C# and Java to use the same time-zone when converting the date to string. One of them is using UTC and the other CDT.
java.util.date automatically corrects for your time zone. See this question: How to set time zone of a java.util.Date?
The ctime is UTC (Universal Coordinated Time), which is a time standard referenced to Greenwich. You're expressing your time in Central time. There's your difference.