I'm using Cassandra with spark, and I have a JavaRDD<Srring> containing 6 columns separated by , like this:
header: canal,action,time,tiemend,client
I have created a table Mytable with 6 columns:
CREATE TABLE IF NOT EXISTS ref_event_by_user_session (
canal TEXT,
action TEXT,
time timestamp,
timeend timestamp,
Client INT,
PRIMARY KEY(canal, action, time, timeend)
);
and now I want to save my JavaRDD in my Cassandra table using javaFunctions().saveToCassandra,
but I don't know how to use it. Could you please show me how to do it?
I've found the solution.
If it can help someone :)
I just create a java class table2 with canal String,
action String,
time Date,
timeend Date,
Client String,
and then :
JavaRDD<Table2> table2 = session_7.map(new Function<Tuple2<KeyTable2, ValueTable2>, Table2>() {
#Override
public Table2 call(Tuple2<KeyTable2, ValueTable2> keyTable2ValueTable2Tuple2) throws Exception {
SimpleDateFormat formatter = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS");
Date startdate = formatter.parse(keyTable2ValueTable2Tuple2._1.time.replace("+","."));
Date endtdate = formatter.parse(keyTable2ValueTable2Tuple2._1.endtime.replace("+","."));
Table2 t = new Table2(keyTable2ValueTable2Tuple2._2.canal,keyTable2ValueTable2Tuple2._2.motif,startdate,endtdate,keyTable2ValueTable2Tuple2._1.client);
return t;
}
});
javaFunctions(table2).writerBuilder("Schema", "Table", mapToRow(Table2.class)).saveToCassandra();
Note : you must add the getters in the class even if the differents variables are public.
Related
My Java application uses a MySQL database that I am trying to convert to SQLite. I was able to convert the database itself just fine, but am having an issue with the timestamps that are stored within the database.
Specifically, my timestamps were stored in the default MySQL format:
2018-04-14 16:33:00
Now, when trying to read the timestamp in SQLite, I am getting the following error:
ParseException: Unparseable date: "2018-04-14 16:33:00" does not match (\p{Nd}++)\Q-\E(\p{Nd}++)\Q-\E(\p{Nd}++)\Q \E(\p{Nd}++)\Q:\E(\p{Nd}++)\Q:\E(\p{Nd}++)\Q.\E(\p{Nd}++)
Is there an efficient way to handle this without actually modifying the format that is stored in the database?
Here is the DDL for one of the tables where this problem exists:
create table agent_list_updates
(
update_id integer not null
primary key
autoincrement,
agent_count integer not null,
update_timestamp timestamp default CURRENT_TIMESTAMP,
user_id integer not null
constraint fk_agent_list_updates_users
references users
)
;
create index idx_agent_list_updates_idx_agent_list_updates_user_id
on agent_list_updates (user_id)
;
The query:
SELECT
agent_count,
update_timestamp
FROM agent_list_updates;
And the Java code that actually causes the error:
if (resultSet.next()) {
return resultSet.getTimestamp("update_timestamp");
} else {
return null;
}
You have to define date format, and then use it to parse your timestamp as a String from resultSet. An example:
SimpleDateFormat formatter = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
while(resultSet.next()) {
Date date = formatter.parse(resultSet.getString("update_timestamp"));
System.out.println(date);
}
because there is no default constructor for Timestamp, or you can do it with the method:
new Timestamp(System.currentTimeMillis());
but Date does not provide any such method directly.
Hello I have and SLQLite database in which I have table water_logs
CREATE TABLE water_logs(
_id INTEGER PRIMARY KEY AUTOINCREMENT,
amount REAL NOT NULL,
icon INTEGER NOT NULL,
date INTEGER NOT NULL);
I store date in milliseconds.
Calendar cal = Calendar.getInstance();
cal.getTimeInMillis();
My problem is I want to get the day from the my date column using strftime function. The problem is tjat java calendar timestamp is different from SLQLite time stamp
1436859563832 --> result from cal.getTimeInMillis();
1436607407--> SELECT strftime('%s','now')
What I'm actually trying to do is to group records by day. The following SQL query works just fine if value of SELECT strftime('%s','now') is paste in the date column
SELECT SUM(amount), date(`date`) FROM water_logs
GROUP BY date(`date`, 'unixepoch')
Seems to me that you are using 2 different value types.
When you use
Calendar cal = Calendar.getInstance();
long time = cal.getTimeInMillis();
The output value is in Milliseconds, as described here.
While when you use
strftime('%s','now')
The output value is in Seconds, as described here.
So, that might be the cause for the mismatch between the two values.
Of course that the value in seconds might undergo some rounding which might change its value a little.
I will try to provide you the best way to store Dates in SQLite database.
1) Always use integers to store the dates.
2) Use this utility method to store the dates into the database,
public static Long saveDate(Date date) {
if (date != null) {
return date.getTime();
}
return null;
}
Like,
ContentValues values = new ContentValues();
values.put(COLUMN_NAME, saveDate(entity.getDate()));
long id = db.insertOrThrow(TABLE_NAME, null, values);
3) Use this utility method to load date,
public static Date loadDate(Cursor cursor, int index) {
if (cursor.isNull(index)) {
return null;
}
return new Date(cursor.getLong(index));
}
like,
entity.setDate(loadDate(cursor, INDEX));
4) You can also order the data by date using simple ORDER clause,
public static final String QUERY = "SELECT table._id, table.dateCol FROM table ORDER BY table.dateCol DESC";
//...
Cursor cursor = rawQuery(QUERY, null);
cursor.moveToFirst();
while (!cursor.isAfterLast()) {
// Process results
}
I have a simple table in my database:
CREATE TABLE [InformacjeZDziekanatu] (
[_id] INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
[DataWstawienia] DATE NOT NULL,
[DataModyfikacji] DATE NOT NULL,
[Tresc] VARCHAR2 NOT NULL);
In my application only what I want to do is to get value of DataWstawienia column.
I am using such method:
try {
ResultSet result = stat.executeQuery("select * from InformacjeZDziekanatu order by _id desc limit 5");
int id;
Date dataWst;
Date dataMod;
String tresc;
for(int j = 0 ; j < 5 ; j++) {
result.next();
Object[] lista = new Object[4];
id = result.getInt("_id");
dataWst = result.getDate("DataWstawienia");
dataMod = result.getDate("DataModyfikacji");
tresc = result.getString("Tresc");
lista[0] = id;
lista[1] = dataWst;
lista[2] = dataMod;
lista[3] = tresc;
dane[j] = lista;
}
}
All dates in DataWstawienia column are today's dates but using this method above I get 1970-01-01 date all the time.
What did I do wrong?
SQLIte does not have a DATE data type. Dates need to either be stored as integers (number of seconds since Unix Epoch is common) or as ISO 8601 times (e.g. 2013-10-01T12:12:12). If you say DATE in your SQLite schema you will store values as strings. Your solution of changing the type of dateWst to String acknowledges the fact that you are storing dates as strings and not as an internal date type.
I've looked around and could not seem to find this asked specifically, on SO, but I've found similar questions like this one and lots of questions regarding SQL itself or C#...
Here is what I am seeing:
MapSqlParameterSource parameterSource = new MapSqlParameterSource();
//parameterSource.addValue( "insertDate", DateTime.now().minusHours( 1 ).toGregorianCalendar(), Types.TIME );
parameterSource.addValue( "insertDate", new Timestamp( DateTime.now().minusHours( 1 ).getMillis() ) );
List< String > contents =
_simpleJdbcTemplate
.query(
"SELECT TOP (200) inserteddatetime, Contents FROM dbo.tsomeTable WHERE (ServiceName = 'something') and inserteddatetime > :insertDate",
new RowMapper< String >() {
#Override
public String mapRow( final ResultSet rs, final int rowNum ) throws SQLException {
System.out.println( rs.getTimestamp( "inserteddatetime" ) );
return rs.getString( "Contents" );
}
}, parameterSource );
The query "hangs"\does nothing\never returns if:
I use the uncommented Timestamp object (as presented above)
I replace the parameterSource object with DateTime.now().minusHours( 1 ).toGregorianCalendar() or DateTime.now().minusHours( 1 ).toGregorianCalendar().getTime()
I try the commented out line, but change the type to Timestamp
So, here is my question or questions...
Is there a known bug\issue with querying on datetime columns in sql server?
Why do I have to use Time and not Timestamp?
I suspect that Spring is converting the date objects over to Timestamp when I query with the object directly (as demonstrated in #2).
TIMESTAMP is a server-generated value used to help with data consistency, its not a simple data type, so for storing datetime value, avoid TIMESTAMP datatype.
Also, SQL Server TIMESTAMP is confusing, and it's deprecated. It is replaced by the ROWVERSION keyword, which should reduce confusion.
I am struggling trying to persist a map to SQLserver with the code below and keep getting the following error:
* Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Invalid object name 'OnlineReport_availabilities'. *
I use Play framework JPA for this have have managed to persist Maps before with similar code. I tried manually creating the table in the db to see if there was any issue with the name but all seems ok.
I am doing something non conventional or incorrect here?
#Entity
public class OnlineReport extends Model {
#Temporal(TemporalType.DATE)
public Date date;
#ElementCollection
public Map<Date, Double> availabilities;
public OnlineReport(){
this.date = new Date();
this.availabilities = new HashMap<Date, Double>();
}
public void addAvailability(double availability){
TreeSet<Date> set = new TreeSet(availabilities.entrySet());
Date lastEntry = null;
if(!set.isEmpty())
lastEntry = set.last();
Date now = new Date();
if(lastEntry != null){
//Add availibility every 10mn
if(DateUtil.getMinutesBetween(lastEntry, now) >= 10){
availabilities.put(now, availability);
save();
}
} else {
availabilities.put(now, availability);
save();
}
}
}
Update.
Running JPA in debugSQL I have noticed the following errors:
ERROR ~ Unsuccessful: create table OnlineReport_availabilities (OnlineReport_id numeric(19,0) not null, availabilities double precision null, availabilities_KEY datetime null, primary key (OnlineReport_id, availabilities_KEY))
ERROR ~ Cannot define PRIMARY KEY constraint on nullable column in table 'OnlineReport_availabilities'.
I am under the impression I am missing some annotation definition of 'availabilities'.
Likely you do not have such a table or it is not accessible via connection in use. Are object (tables, fields etc) names case sensitive in db?
So, after your update we know why you do not have table. Looks like it fails to create not null -constraint to the column that is part of the key. Dialect fails for SQL Server, because your mapping is minimal, but correct. You can try to force it by adding following annotation:
#MapKeyColumn(columnDefinition = "datetime not null")
Of course having datetime as part of the key will produce problems if it is possible that there is more than one entry per millisecond.