I have the following method where I am doing db insertions. I want to perform the inserts in a transaction.
Meaning when there are 100 values in records, I wan to insert them all and commit once.
How could I amend the following such that I can get record.value() info for each insert queries below.
This would essentially equate to having andThen() a 100 times but of course I do not want to write andThen() a 100 times nor do I know the number of records which can vary.
To note: Using RX Java 1.
Please advice. Thank you.
public Observable<?> insert(Observable<Record<String, String>> records) {
// I am looking for a way to get this record.value() into the following return block.
records.flatMap(record -> {
String value = record.value();
return null;
});
return client.rxGetConnection()
// making it transactional by setting to false
.flatMap(connection -> connection.rxSetAutoCommit(false)
// was looking to insert above records and flatMap operations here but it is not possible from what I have explored.
.toCompletable()
// .andThen(connection.rxExecute("INSERT (name) VALUES " + record.value()) // trying to achieve this, to be able to get record.value for each insert
.andThen(connection.rxExecute("INSERT (name) VALUES some_value"))
.flatMap(rows -> connection.rxCommit())).toObservable();
}
I am making the following query which works and updates the info into the database as expected. But is there a way I can get an output from Single < UpdateResult >?
public Single<UpdateResult> update(UpdateEventRequest request) {
return client.rxUpdateWithParams(
"UPDATE mydb.sample SET sta_cd=?, some_ts=current_timestamp WHERE id=? RETURNING sta_cd",
new JsonArray(Arrays.asList(sta_cd, id)));
}
From the following e variable, I was hoping to get the value "10012". But it doesn't seem possible. Tried with map, flatmap and just see options available in e. The only result data in e is 'keys' which is an empty list and 'updated' which is an integer value of 1. My DB is postgres and was expecting results from from Single < UpdateResult > since am using 'RETURNING' in the query.
I have done the same for an insert operation which works but that is via the method rxQueryWithParams() and that returns a Single < ResultSet > instead. Thus wondering if this is even possible. Been having a look at docs and maybe this is not possible as an update query is returning a Single < UpdateResult > . Looking for advice if this is possible, to return data from an update query or a way around this. Please advice. Thanks.
Single<UpdateResult> result = someClass.update("10012", "78632");
result.subscribe(
e -> {
System.out.println("success: " + e); // I land here as expected
},
error -> {
System.out.println("error: " + error);
}
);
Because you are using RETURNING in these commands, treat these INSERT and UPDATE commands as queries.
Run them through rxQueryWithParams() so you can retrieve the results.
When you run rxUpdateWithParams(), the UpdateResult contains only the number of rows affected.
I want to use this SQL query:
String hql = "select e from " + Terminals.class.getName() + " e WHERE e.merchantId IN :merchant_ids";
TypedQuery<Terminals> query = entityManager.createQuery(hql, Terminals.class).setParameter("merchant_ids", merchant_ids);
List<Terminals> merchants = query.getResultList();
But I get error: the right syntax to use near ') So IN clause list into IN (....) can't be empty. Is there some solution to this problem?
It is allowable and even very fine not executing the query:
if (merchant_ids.isEmpty()) {
return new ArrayList<>();
} else {
String hql = "select e from " + Terminals.class.getName()
+ " e WHERE e.merchantId IN :merchant_ids";
return entityManager.createQuery(hql, Terminals.class)
.setParameter("merchant_ids", merchant_ids)
.getResultList();
}
I do not know what would happen if one would pass null instead of an empty list;
SQL ... IN NULL could do. On the other hand it might do a full table scan in order to return 0 results.
If x IN() would not result in 0 records (when there is an OR ...) then:
if (merchant_ids.isEmpty()) {
merchant_ids.add(-1);
String hql = "select e from " + Terminals.class.getName() + ...
Very often, I used to stuck this kind of case. I couldn't find out a proper solution. Since you are using Spring JPA But I have some workaround to suggest to you.
Implement EntityManger and create your SQL queries in runtime. So you can populate your where cause and everything.
Like this: entityManager.createNativeQuery(sql.toString())
Implement if-else block. Check if the list is empty or not, if false call actual query (with IN block) or else write another query without IN block.
Again I am telling, this may not be a proper solution. But I see this is proper workaround.
I am not familiar with hibernate but since it is an SQL error, the following should work :
TypedQuery<Terminals> query = entityManager
.createQuery(hql, Terminals.class)
.setParameter("merchant_ids",merchant_ids.size()==0?null:merchant_ids);
But as #Richard Barker mentioned , best solution is to not even execute the query when the list is empty.
You will even save on the unnecessary database call , when you already know that the query is not going to return anything.
I followed #Rambler's suggestion and created a method to return a null:
public static <T> Collection<T> nullIfEmpty(Collection<T> collection) {
return (collection == null || collection.isEmpty()) ? null : collection;
}
This was easier to add in place, but I agree that it is better to not make the call to the database.
I need to calculate average occupancy for selected day of week (eg. all Fridays - for each minute). I didn't find any JPQL/Querydsl solution for this problem because of lack of Date/Time functions. So I'm trying to make use of Java Streams. My (simplified) object:
class Occupancy {
private LocalDateTime timeStamp;
private int occupied;
}
my repo:
#Query("select o from Occupancy o")
public Stream<Occupancy> streamAllOccupancies();
sample:
try ( Stream<Occupancy> stream = repository.streamAllOccupancies()) {
Function<Occupancy,LocalTime> OccupancyMinutesGrouping = (Occupancy o) -> {
return o.getDateTime().toLocalTime().truncatedTo(ChronoUnit.MINUTES);
};
Map<LocalTime,Double> avgMap = stream
.filter( o -> o.getDateTime().getDayOfWeek() == DayOfWeek.MONDAY) //example
.collect(
Collectors.groupingBy(
OccupancyMinutesGrouping,
Collectors.averagingInt(Occupancy::getOccupied)
)
);
}
It works - but is it possible change this map into list of my occupancy objects:
new Occupancy( localTime, averagedOccupancy );
I'm also worried about stream efficiency - it has to process all records from the database. How does the stream work with jpa repo? First SQL gets all the records - then stream processes it? Or are they processed sequentially on every record? Maybe the best solution is to use Native SQL query insted of Stream? Any ideas will be very helpful...
As for conversion to the List<Occupancy>, please note that occupied field is of int type while average could be non-integral. So I assume that the Occupancy class is defined this way:
class Occupancy {
private LocalDateTime timeStamp;
private double occupied;
public Occupancy(LocalDateTime ts, double occ) {
this.timeStamp = ts;
this.occupied = occ;
}
}
Now you can just create one more stream from the resulting map:
List<Occupancy> occupancies = avgMap.entrySet().stream()
.map(e -> new Occupancy(e.getKey(), e.getValue()))
.collect(Collectors.toList());
It seems that intermediate Map is unavoidable (at least if your stream is not already sorted by LocalTime).
As for memory usage: it depends on the underlying JDBC driver. The resulting stream indeed reads the underlying ResultSet row-by-row, but it's JDBC-specific how many rows are prebuffered at once. For example, it's known that MySQL driver by default retrieves complete ResultSet into the memory, so you may need some query hint like this:
#QueryHints(value = #QueryHint(name = HINT_FETCH_SIZE, value = "" + Integer.MIN_VALUE))
See this blog post for details.
Also note that if your JDBC driver actually fetches the data row-by-row from the server (without buffering), this actually might have worse performance as you may need more round-trips between DBMS and your application (this might be especially crucial if DBMS server is located on different machine). So consult your JDBC driver documentation for additional details.
I know that the only really correct way to protect SQL queries against SQL injection in Java is using PreparedStatements.
However, such a statement requires that the basic structure (selected attributes, joined tables, the structure of the WHERE condition) will not vary.
I have here a JSP application that contains a search form with about a dozen fields. But the user does not have to fill in all of them - just the one he needs. Thus my WHERE condition is different every time.
What should I do to still prevent SQL injection?
Escape the user-supplied values? Write a wrapper class that builds a PreparedStatement each time? Or something else?
The database is PostgreSQL 8.4, but I would prefer a general solution.
Thanks a lot in advance.
Have you seen the JDBC NamedParameterJDBCTemplate ?
The NamedParameterJdbcTemplate class
adds support for programming JDBC
statements using named parameters (as
opposed to programming JDBC statements
using only classic placeholder ('?')
arguments.
You can do stuff like:
String sql = "select count(0) from T_ACTOR where first_name = :first_name";
SqlParameterSource namedParameters = new MapSqlParameterSource("first_name", firstName);
return namedParameterJdbcTemplate.queryForInt(sql, namedParameters);
and build your query string dynamically, and then build your SqlParameterSource similarly.
I think that fundamentally, this question is the same as the other questions that I referred to in my comment above, but I do see why you disagree — you're changing what's in your where clause based on what the user supplied.
That still isn't the same as using user-supplied data in the SQL query, though, which you definitely want to use PreparedStatement for. It's actually very similar to the standard problem of needing to use an in statement with PreparedStatement (e.g., where fieldName in (?, ?, ?) but you don't know in advance how many ? you'll need). You just need to build the query dynamically, and add the parameters dynamically, based on information the user supplied (but not directly including that information in the query).
Here's an example of what I mean:
// You'd have just the one instance of this map somewhere:
Map<String,String> fieldNameToColumnName = new HashMap<String,String>();
// You'd actually load these from configuration somewhere rather than hard-coding them
fieldNameToColumnName.put("title", "TITLE");
fieldNameToColumnName.put("firstname", "FNAME");
fieldNameToColumnName.put("lastname", "LNAME");
// ...etc.
// Then in a class somewhere that's used by the JSP, have the code that
// processes requests from users:
public AppropriateResultBean[] doSearch(Map<String,String> parameters)
throws SQLException, IllegalArgumentException
{
StringBuilder sql;
String columnName;
List<String> paramValues;
AppropriateResultBean[] rv;
// Start the SQL statement; again you'd probably load the prefix SQL
// from configuration somewhere rather than hard-coding it here.
sql = new StringBuilder(2000);
sql.append("select appropriate,fields from mytable where ");
// Loop through the given parameters.
// This loop assumes you don't need to preserve some sort of order
// in the params, but is easily adjusted if you do.
paramValues = new ArrayList<String>(parameters.size());
for (Map.Entry<String,String> entry : parameters.entrySet())
{
// Only process fields that aren't blank.
if (entry.getValue().length() > 0)
{
// Get the DB column name that corresponds to this form
// field name.
columnName = fieldNameToColumnName.get(entry.getKey());
// ^-- You'll probably need to prefix this with something, it's not likely to be part of this instance
if (columnName == null)
{
// Somehow, the user got an unknown field into the request
// and that got past the code calling us (perhaps the code
// calling us just used `request.getParameterMap` directly).
// We don't allow unknown fields.
throw new IllegalArgumentException(/* ... */);
}
if (paramValues.size() > 0)
{
sql.append("and ");
}
sql.append(columnName);
sql.append(" = ? ");
paramValues.add(entry.getValue());
}
}
// I'll assume no parameters is an invalid case, but you can adjust the
// below if that's not correct.
if (paramValues.size() == 0)
{
// My read of the problem being solved suggests this is not an
// exceptional condition (users frequently forget to fill things
// in), and so I'd use a flag value (null) for this case. But you
// might go with an exception (you'd know best), either way.
rv = null;
}
else
{
// Do the DB work (below)
rv = this.buildBeansFor(sql.toString(), paramValues);
}
// Done
return rv;
}
private AppropriateResultBean[] buildBeansFor(
String sql,
List<String> paramValues
)
throws SQLException
{
PreparedStatement ps = null;
Connection con = null;
int index;
AppropriateResultBean[] rv;
assert sql != null && sql.length() > 0);
assert paramValues != null && paramValues.size() > 0;
try
{
// Get a connection
con = /* ...however you get connections, whether it's JNDI or some conn pool or ... */;
// Prepare the statement
ps = con.prepareStatement(sql);
// Fill in the values
index = 0;
for (String value : paramValues)
{
ps.setString(++index, value);
}
// Execute the query
rs = ps.executeQuery();
/* ...loop through results, creating AppropriateResultBean instances
* and filling in your array/list/whatever...
*/
rv = /* ...convert the result to what we'll return */;
// Close the DB resources (you probably have utility code for this)
rs.close();
rs = null;
ps.close();
ps = null;
con.close(); // ...assuming pool overrides `close` and expects it to mean "release back to pool", most good pools do
con = null;
// Done
return rv;
}
finally
{
/* If `rs`, `ps`, or `con` is !null, we're processing an exception.
* Clean up the DB resources *without* allowing any exception to be
* thrown, as we don't want to hide the original exception.
*/
}
}
Note how we use information the user supplied us (the fields they filled in), but we didn't ever put anything they actually supplied directly in the SQL we executed, we always ran it through PreparedStatement.
The best solution is to use a middle that does data validation and binding and acts as an intermediary between the JSP and the database.
There might be a list of column names, but it's finite and countable. Let the JSP worry about making the user's selection known to the middle tier; let the middle tier bind and validate before sending it on to the database.
Here is a useful technique for this particular case, where you have a number of clauses in your WHERE but you don't know in advance which ones you need to apply.
Will your user search by title?
select id, title, author from book where title = :title
Or by author?
select id, title, author from book where author = :author
Or both?
select id, title, author from book where title = :title and author = :author
Bad enough with only 2 fields. The number of combinations (and therefore of distinct PreparedStatements) goes up exponentially with the number of conditions. True, chances are you have enough room in your PreparedStatement pool for all those combinations, and to build the clauses programatically in Java, you just need one if branch per condition. Still, it's not that pretty.
You can fix this in a neat way by simply composing a SELECT that looks the same regardless of whether each individual condition is needed.
I hardly need mention that you use a PreparedStatement as suggested by the other answers, and a NamedParameterJdbcTemplate is nice if you're using Spring.
Here it is:
select id, title, author
from book
where coalesce(:title, title) = title
and coalesce(:author, author) = author
Then you supply NULL for each unused condition. coalesce() is a function that returns its first non-null argument. Thus if you pass NULL for :title, the first clause is where coalesce(NULL, title) = title which evaluates to where title = title which, being always true, has no effect on the results.
Depending on how the optimiser handles such queries, you may take a performance hit. But probably not in a modern database.
(Though similar, this problem is not the same as the IN (?, ?, ?) clause problem where you don't know the number of values in the list, since here you do have a fixed number of possible clauses and you just need to activate/disactivate them individually.)
I'm not confident if there is a quote() method, which was widely used in PHP's PDO. This would allow you a more flexible query building approach.
Also, one of the possible ideas could be creating special class, which would process filter criterias and would save into a stack all placeholders and their values.