ps = (PreparedStatement) connection.prepareStatement(
"SELECT nm.id, nid.key, nm.name, nm.languageCode FROM odds.name as nm JOIN (odds.name_id as nid)\r\n"
+ "ON (nm.id = nid.id) where nm.name like '%' and nid.key not like \"vhc%\" and nid.key not like \"vdr%\" and nid.key not like \"vto%\" and nid.key not like \"vbl%\"\r\n"
+ "and nid.key not like \"vf%\" and nid.key not like \"vfl%\" and nid.key not like \"vsm%\" and nid.key not like \"rgs%\"\r\n"
+ "and nid.key not like \"srrgs%\" and nm.typeId=8 and nm.sourceId=-1 and nm.languageCode = 'en'");
for(Entry <String, Tag> e : allTags.entrySet()) {
ResultSet rs = ps.executeQuery();
while(rs.next()) {
if(rs.getString("name").equals(e.getValue().getTranslation(Language.EN))) {
e.getValue().setAlternativeKey(rs.getString("name"));
break;
}
}
}
);
Do you have any Ideas how I can do this a way faster. I'll try to find a string in the database and add an extra information to my object. But I have to do this for 1265 objects, so the program runs about 80 seconds.
Thanks in advance!
First of all, when tackling performance problems, get yourself a profiling tool that tells you where you're spending the time, how often a given method is called and so on.
But I think the case is clear enough to give some more specific hints.
You're executing your PreparedStatement over and over again, once for every entry in allTags.entrySet(), always giving you the same results, and inside in software you filter out the lines you're interested in. So you're doing the same query 1265 times, correct?
And it's puzzling me what you're doing inside the while(rs.next()) loop. Effectively, your code does (after introducing some local variables, moving constant values out of loops, ...):
for(Entry <String, Tag> e : allTags.entrySet()) {
Tag tag = e.getValue();
String translation = tag.getTranslation(Language.EN);
ResultSet rs = ps.executeQuery();
while(rs.next()) {
if(rs.getString("name").equals(translation)) {
tag.setAlternativeKey(translation);
break;
}
}
}
So, the only role of the query result seems to be to decide whether the alternative key should be set (if the translation of your Tag shows up as name in the ResultSet) - the value is already fixed by the result of the method call getTranslation(Language.EN), independent of any database result.
I'd suggest to do one execution of your query, collecting the name values in a HashSet names, and after that do the allTags loop setting the translation if the translation is contained in your names set. That should give the same result as your code, and probably much faster.
Open the DB-client of your choice (e.g. HeidiSQL) and do an
explain [the select statement that is originally executed]
That way MySQL explains to you what it's doing when trying to create the result and where time gets lost.
From there you can go on e.g. creating indizes or changing your query to make use of existing ones.
BTW:
nm.name like '%'
looks strange. Is that a variant of
is not null
The latter might be faster. If the texts in the other like-statements are always the same, a better performance might be achieved by checking these conditions when inserting the data, add columns of type int or boolean and save the result of this check as integer/boolean in addition to the text itself. Checking against a fixed numeric value is way faster than text searches.
Related
Probably the only way to do this is to keep a list of static final Strings but if anyone has any idea it is welcome.
I have a POJO and a database table. The columns(properties) are the same as the class members of the POJO. I want to avoid (it is a fairly big program) to write getColumn("username") but instead write something like getColumn(UserPOJO.getUsername().getActualMemberNameInAString())
The only way I can think of is to have a UserPOJO class with just static variables and update it when I add members in the class. The second option is to have all the properties as enums but that is bad I think in a number of levels.
What I would like to have is a way to get the member name that corresponds to the getter method. Some kind of reflection?
Problem to be solved is typos in a huge codebase. I overemphasize here because I have received comments and answers that are not what I am asking for.
Note that I am not using any framework or ORM tool. I am using a Graph database that maps all columns to a Vertex and then you need to do .getProperty(string s) to get an Object. And I do this many times in the code
My understanding of this question is you want minimum code change when your underlaying Database Table get changed.
so let me put it as , say I have a table Info as
CREATE TABLE Info (
field1 INTEGER ,
field2 VARCHAR(1)
);
and I am expecting field3 could be added in future then I need a solution where
I do not need to add extra code for getting information for field3.
Please confirm if the understanding is correct ? if yes then you can use below strategy
/**
* generic Method for executing Any RAW SQL Query.
* for e.g. sql = "select * from Info";
*/
public List<Map<String,String>> executeQuery(String sql) throws Exception {
logger.info("executing SQL = "+sql);
List<Map<String,String>> resultList = new ArrayList<Map<String, String>>();
Statement statement=null;
ResultSet resultSet=null;
ResultSetMetaData metaData = null;
try{
statement = connection.createStatement();
resultSet = statement.executeQuery(sql);
metaData = resultSet.getMetaData();
int noOfColumns = metaData.getColumnCount();
while(resultSet.next()){
Map<String,String> row = new HashMap<String, String>();
for(int i =1;i<=noOfColumns;i++){
row.put(metaData.getColumnName(i), resultSet.getString(metaData.getColumnName(i)));
}
resultList.add(row);
}
} catch (SQLException e) {
throw new Exception(" Exception while executing query on database. Reason "+e.getMessage());
}
finally {
closeResources(statement, resultSet);
}
return resultList;
}
for input String sql = "select * from Info"; Here you will get all records of Info Table with key-value pair where key suggests columnName and value suggests value of that column.
And this is as good as dynamic ,even if we add extra columns we will get that information in our Map also it does not require to use any framework
You have multiple options:
If you only need the name once, I wouldn't bother to create constants for them.
In one of your comments you stated you will use it a lot and in different places, you could create a class that stores all those strings as constants. At least you will avoid typos, and your code wont be littered with strings.
You could use reflection but that would overcomplicate things (in my opinion).
I wouldn't exclude using enums as an option. This SO answer explains it better, then I could do...
I want to pass a List of String coming from request parameter to the preparedStatement.setObject() as a single parameter.Here I have coverted list of Objects to a single String.
So while passing this converted String to setObject method it is converting ' to \'.
So my query is like :
select * from category where category IN (?)
for (int counter = 0; (!sqlParams.isEmpty()) && counter < sqlParams.size(); counter++) {
System.out.println(sqlParams.get(counter));
stmt.setObject(counter + 1, sqlParams.get(counter));
System.out.println(stmt.toString());
}
here sqlParams.get(counter) is giving following value to me.
'Adult', 'Classic', 'Fantasy', 'Mystery'
but wen i am using stmt.setObject(), and printing the values of stmt, it is showing following value :
'\'Adult\', \'Classic\', \'Fantasy\', \'Mystery\''
So at query formation is something like this :
SELECT * FROM mytable WHERE category IN ('Adult\', \'Classic\', \'Fantasy\', \'Mistry\'');
There are other ways to solve these approach too, such as passing individual String and then query execution for each individual String.but It will increase time complexity for my code.
Can anyone Suggest me the solution for this?
JDBC prepared statements cannot be used when the number of parameters is variable, like here. The ? in the query is expanded into a single value, not to several separated by commas.
What you can do is construct an sql with the appropriate number of ?s and then set each parameter in a loop.
I'm making a java application for a toy store and using the MS-Access database.
I'm using the operator "LIKE" to search for products in database using the user's keyword.
E.g. in Access:
SELECT * FROM tblToy WHERE toyName LIKE '*puppy*' OR toyBrand LIKE '*puppy*'
this gives me the desired result in access.
But in java when i run this same query it returns null:
String query = "puppy";
sql = "SELECT * FROM tblToy WHERE toyName LIKE '*" + query+"*' "+
"OR toyBrand LIKE '*" + query + "*'";
rs = db.executeQuery(sql);
while(rs.next()){
String name = rs.getString("toyName");
return name;
}
return null;
Can anyone help me on this? I know it must be something simple which I'm missing out now but I just don't know what to do. Would appreciate your guys help.
I think with Java, you need to escape single quotes, so try using \' for all your single quotes, then try % instead of * as someone else mentioned, since % is the wildcard for SQL.
There are two possibilities for wildcards according to where you are running the query, * or %. In this case, you need %
Hello guys I am having some problems with exact matches while doing a NamedQuery.
I am currently using something like this:
#NamedQuery(name = MyClass.GET_ENTRY_BY_NAME, query = "select e from Entry e where e.name =:"+ Entry.NAME )
...
Query query = em.createNamedQuery(MyClass.GET_ENTRY_BY_NAME);
query.setParameter(Entry.NAME, myEntry.getName());
It works for most cases, however I noticed that in case the user pass the file name with an space at the end, the namedQuery ignores that character. For example:
Query query = em.createNamedQuery(MyClass.GET_ENTRY_BY_NAME);
query.setParameter(Entry.NAME, myEntry.getName()+ " ");
Will return the same result as the query before. Bypassing my 'valid entry' validation. In other words I'd like the query to return no entry at all and treat the error later on.
One workaround I could think of, is to put single quotes surrounding my parameter in the namedQuery, like this:
#NamedQuery(name = MyClass.GET_ENTRY_BY_NAME, query = "select e from entry e where e.name =':"+ Entry.NAME "'")
However it will trash my code in case the String contains single quotes in it...
Any ideas guys?
I guess this happens because your database field is declared as CHAR(...), and therefore stored values are padded with whitespaces which are not taken into account by = operation.
So, you may either declare your database field as VARCHAR(...) or use a built-in trim function:
query = "select e from Entry e where trim(trailing from e.name) =:"+ Entry.NAME
I did some research in JPA and found out that it does some automatic trimming for CHARs, I am not sure if this behaves the same with Strings, but since it is happening to me... I believe so. The only way to bypass it is by setting some attribute within the session DatabaseLogin object (see http://www.eclipse.org/eclipselink/api/1.1/org/eclipse/persistence/sessions/DatabaseLogin.html#setShouldTrimStrings) .
Well I didn't want to be messing up with the session properties so I decided to make some sort of check and throwing the same exception as the NoResultException catch does in my code.
I basically took the result from the database and compared the field with the String I used:
query.setParameter(Entry.NAME, myEntry.getName());
...
if(!StringUtils.equals(result.getName(), myEntry.getName()){
do a cool throw just like NoResultException Catch
}
I also had to include the Trim function axtavt! This is just to make sure that if the database has a column with trailing spaces and it matches the parameter given by the user, it will be included as a valid answer. For example:
Database entry: Name = "Flavio " - Trimmed with Function = "Flavio".
Parameter passed: Name = "Flavio " - Trimmed by JPA automatic function = "Flavio".
If it isnt trimmed at all it will just Compare "Flavio " with "Flavio", returning NoResult when it was supposed to return that Entry.
Nasty workaround, but as long as there is no other way to stop the auto-trimming we will have to just make use of this sort of things.
Thanks for all the other answers!!
having major issues with my query processing time :(
i think it is because the query is getting recompiled evrytime. but i dont see any way around it.
the following is the query/snippet of code:
private void readPerformance(String startTime, String endTime,
String performanceTable, String interfaceInput) throws SQLException, IOException {
String interfaceId, iDescp, iStatus = null;
String dtime, ingress, egress, newLine, append, routerId= null;
StringTokenizer st = null;
stmtD = conn.createStatement(java.sql.ResultSet.TYPE_FORWARD_ONLY,
java.sql.ResultSet.CONCUR_READ_ONLY);
stmtD.setFetchSize(Integer.MIN_VALUE);
BufferedReader interfaceRead = new BufferedReader(new FileReader(interfaceInput));
BufferedWriter pWrite = new BufferedWriter(new FileWriter("performanceInput.txt"));
while((newLine = interfaceRead.readLine())!= null){
st = new StringTokenizer(newLine,",");
while(st.hasMoreTokens()){
append = st.nextToken()+CSV+st.nextToken()+st.nextToken()+CSV+st.nextToken();
System.out.println(append +" ");
iStatus = st.nextToken().trim();
interfaceId = st.nextToken().trim();
append = append + CSV+iStatus+CSV+interfaceId;
System.out.println(append +" ");
pquery = " Select d.dtime,d.ifInOctets, d.ifOutOctets from "+performanceTable+"_1_60" +" AS d Where d.id = " +interfaceId
+ " AND dtime BETWEEN " +startTime+ " AND "+ endTime;
rsD = stmtD.executeQuery(pquery);
/* interface query*/
while(rsD.next()){
dtime = rsD.getString(1);
ingress= rsD.getString(2);
egress = rsD.getString(3);
pWrite.write(append + CSV + dtime+CSV+ingress+CSV+egress+NL);
}//end while
}//end while
}// end while
pWrite.close();
interfaceRead.close();
rsD.close() ;
stmtD.close();
}
my interfaceId value keeps changing. so i have put the query inside the loop resulting in recompilation of query multiple times.
is there any betetr way? can i sue stored procedure in java? if so how? do not have much knowledge of it.
current processing time is almost 60 mins (:(()!!! Text file getting generated is over 300 MB
Please help!!!
Thank you.
You can use a PreparedStatement and paramters, which may avoid recompiling the query. Since performanceTable is constant, this can be put into the prepared query. The remaining variables, used in the WHERE condition, are set as parameters.
Outside the loop, create a prepared statement, rather than a regular statement:
PreparedStatement stmtD = conn.prepareStatement(
"Select d.dtime,d.ifInOctets, d.ifOutOctets from "+performanceTable+"_1_60 AS d"+
" Where d.id = ? AND dtime BETWEEN ? AND ?");
Then later, in your loop, set the parameters:
stmtD.setInteger(1, interfaceID);
stmtD.setInteger(2, startTime);
stmtD.setInteger(3, endTime);
ResultSet rsD = stmtD.executeQuery(); // note no SQL passed in here
It may be a good idea to also check the query plan from MySQL with EXPLAIN to see if that is part of the bottleneck also. Also, there is quite a bit of diagnostic string concatenation going on in the function. Once the query is working, removing that may also improve performance.
Finally, note that even if the query is fast, network latency may slow things down. JDBC provides batch execution of multiple queries to help reduce overall latency per statement. See addBatch/executeBatch on Connection.
More information required but I can offer some general questions/suggestions. It may have nothing to do with the compilation of the query plan (that would be unusual)
Are the id and dtime columns indexed?
How many times does a query get executed in the 60mins?
How much time does each query take?
If the time per query is large then the problem is the query execution itself, not the compilation. Check the indexes as described above.
If there are many many many queries then it might be the sheer volume of queries that is causing the problem. Using PreparedStatement (see mdma's answer) may help. Or you can try and batch the interfaceIDs you want by using an "in" statement and running a query for every 100 interfaceIDs rather than one for each.
EDIT: As a matter of good practice you should ALWAYS use PreparedStatement as it will correctly handle datatypes such as dates so you don't have to worry about formatting them into correct SQL syntax. Also prevents SQL injection.
From the looks of things you are kicking off multiple select queries (even 100's based on your file size)
Instead of doing that, from your input file create a comma delimited list of all the interfaceId values and then make 1 SQL call using the "IN" keyword. You know the performanceTable, startTime and endTime arent changing so the query would look something like this
SELECT d.dtime,d.ifInOctets, d.ifOutOctets
FROM MyTable_1_60 as d
WHERE dtime BETWEEN '08/14/2010' AND '08/15/2010'
AND d.id IN ( 10, 18, 25, 13, 75 )
Then you are free to open your file, dump the result set in one swoop.