Database search with lucene - java

I'm using Lucene for querying a website's database but i'm experiencing some problems. I don't actually know if the problems come from indexing or searching (more precisely the construction of queries). Well, as far as i'm aware, when searching in several SQL database tables its better to use more than one document for each table (i followed these tutorials:
http://kalanir.blogspot.pt/2008/06/indexing-database-using-apache-lucene.html
http://www.lucenetutorial.com/techniques/indexing-databases.html
http://www.youtube.com/watch?v=jTDTYdU6nTc
) which are close to what i want to do. In fact, in my case i have to search in 3 tables which are all related because each one specifies the above level (e.g.: product -> type -> color). Thus, my indexing was something like this:
String sql = "select c.idConteudo as ID, c.designacao as DESIGNACAO, cd.texto as DESCRICAO, ctf.webTag as TAG from Conteudo c, ConteudoDetalhe cd, ConteudoTipoFormato ctf where c.idConteudo = cd.idConteudo AND cd.idConteudoTipoFormato = ctf.idConteudoTipoFormato;";
Statement stmt = connection.createStatement();
ResultSet rs = stmt.executeQuery(sql);
Document document;
while (rs.next())
{
String S = new String();
S += IndexerCounter;
document = new Document();
document.add(new Field("ID_ID",S, Field.Store.YES, Field.Index.NO));
document.add(new Field("ID CONTEUDO", rs.getString("ID"), Field.Store.YES, Field.Index.NO));
document.add(new Field("DESIGNACAO", rs.getString("DESIGNACAO"), Field.Store.NO, Field.Index.TOKENIZED));
document.add(new Field("DESCRICAO", rs.getString("DESCRICAO"), Field.Store.NO, Field.Index.TOKENIZED));
document.add(new Field("TAG", rs.getString("TAG"), Field.Store.NO, Field.Index.TOKENIZED));
try{
writer.addDocument(document);
}catch(CorruptIndexException e){
}catch(IOException e){
}catch(Exception e){ } //just for knowing if something is wrong
IndexerCounter++;
}
If i output the results they are something like this:
ID: idConteudo: designacao: texto: webTag
1:1:Xor:xor 1 Descricao:x or
2:1:Xor:xor 2 Descricao:xis Or
3:1:Xor:xor 3 Descricao:exor
4:2:And:and 1 Descricao:and
5:2:And:and 2 Descricao:&
6:2:And:and 3 Descricao:ande
7:2:And:and 4 Descricao:a n d
8:2:And:and 5 Descricao:and,
9:3:Nor:nor 1 Descricao:nor
10:3:Nor:nor 2 Descricao:not or
What i really want is to make a query for (for example Xor) and search it in the created documents for it. Thus my searching method is something like this:
Constructor:
public Spider(String Query, String Pathh) {
String[] Q;
QueryFromUser = new String();
QueryFromUser = Query;
QueryToSearch1 = new String();
QueryToSearch2 = new String();
Path = Pathh;
try {
try {
Class.forName("com.mysql.jdbc.Driver");
} catch (ClassNotFoundException e) {
e.printStackTrace();
return;
}
try {
connection = DriverManager.getConnection("jdbc:mysql://localhost:3306/mydb", "root", "");
} catch (SQLException e) {
e.printStackTrace();
return;
}
Q = Query.split(" ");
//NOTE: the AND word enables the search engine to search by the various words in a query
for (int i = 0; i < Q.length; i++) {
if ((Q.length - i) > 1) //prevents the last one to take a AND
{
QueryToSearch1 += Q[i] + " AND ";
} else {
QueryToSearch1 += Q[i];
}
}
for (int i = 0; i < Q.length; i++) {
QueryToSearch2 += "+" + Q[i];
}
try {
SEARCHING_CONTENT();
} catch (ClassNotFoundException ex) {
Logger.getLogger(Spider.class.getName()).log(Level.SEVERE, null, ex);
} catch (InstantiationException ex) {
Logger.getLogger(Spider.class.getName()).log(Level.SEVERE, null, ex);
} catch (IllegalAccessException ex) {
Logger.getLogger(Spider.class.getName()).log(Level.SEVERE, null, ex);
} catch (SQLException ex) {
Logger.getLogger(Spider.class.getName()).log(Level.SEVERE, null, ex);
} catch (ParseException ex) {
Logger.getLogger(Spider.class.getName()).log(Level.SEVERE, null, ex);
}
SEARCHING_WEB(); //not for using now
} catch (CorruptIndexException ex) {
Logger.getLogger(Spider.class.getName()).log(Level.SEVERE, null, ex);
} catch (IOException ex) {
Logger.getLogger(Spider.class.getName()).log(Level.SEVERE, null, ex);
}
The idea is that QueryToSearch1 and QueryToSearch2 has the commands (i saw it on an online tutorial, don't quite remember where) AND and +. Thus, to a query "not or" from the user, what will be searched it will be "not AND or" for searching for the two words simultaneously and "+not+or" for searching the two words separetly. This is one of my doubts, i don't really know if the construction of lucene queries are like this. The fact is that, in the method Querying:
private void SEARCHING_CONTENT() throws CorruptIndexException, IOException, ClassNotFoundException, InstantiationException, IllegalAccessException, SQLException, ParseException {
Querying(QueryToSearch1); // search for the whole phrase
Querying(QueryToSearch2); //search by individual words
//Querying(QueryFromUser); //search by individual words
}
private void Querying(String QueryS) throws CorruptIndexException, IOException, ClassNotFoundException, InstantiationException, IllegalAccessException, SQLException, ParseException {
searcher = new IndexSearcher(IndexReader.open(Path + "/INDEX_CONTENTS"));
query = new QueryParser("TAG", new StopWords()).parse(QueryS);
query.toString();
hits = searcher.search(query);
pstmt = connection.prepareStatement(sql);
for (int i = 0; i < hits.length(); i++) {
id = hits.doc(i).get("TAG");
pstmt.setString(1, id);
displayResults(pstmt);
}
}
there are no hits on the documents for the query. It is important to say that in the following line:
query = new QueryParser("TAG", new StopWords()).parse(QueryS);
the StopWords is a class i made that extents StandardAnalyser but its a new class with words i specified (for NOT removing important on my search words like or or and - in this case those words may be important).
The problem is, as i told. There are no hits when the search is performed. I'm not sure if this is because of the indexing or because of the construction of the queries to search (if the queries are bad constructed, thence, there are no hits).
I would apreciatte any help from anyone. I would gladly provide more information if needed.
Thanks a lot.

Easy first move for you - use Luke (https://code.google.com/p/luke/) to look on your index. You could run your queries from Luke to check, do they find something, or not.
Luke is pretty easy to understand, since it have very usefull UI (https://code.google.com/p/luke/source/browse/wiki/img/overview.png)

Related

Retry the preparedstatement batches if an exception (SQLException | BatchUpdateException) occurs - Java

I had a situation where my code was getting hit by the deadlock issue with SQL Server for some transactions. So, I implemented a retry logic to overcome the same. Now, I'm facing a new problem. The problem is, whenever it retries, the batch which was tried to execute will be empty/cleared after recovering from the exception. This is causing missing data inserts/updates. Please help me with this.
Summary:
Retry the preparedstatement batches if an exception (SQLException | BatchUpdateException) occurs
Current Implementation:
do {
try {
if (isInsert) {
int[] insertRows = psInsert.executeBatch();
psInsert.clearBatch();
System.out.println("insertRowsSuccess:" + Arrays.toString(insertRows));
} else {
int[] updateRows = psUpdate.executeBatch();
psUpdate.clearBatch();
System.out.println("updateRowsSuccess:" + Arrays.toString(updateRows));
}
break;
} catch (BatchUpdateException e) {
conn.rollback();
if (++count == maxTries) {
System.out.println(e.getMessage());
getFailedRecords(e, operation);
}
} catch (SQLException e) {
if (++count == maxTries) {
System.err.format("SQL State: %s\n%s", e.getSQLState(), e.getMessage());
}
}
System.out.println("Tries:" + count);
} while (true);
private static void getFailedRecords(BatchUpdateException ex, String operation) {
int[] updateCount = ex.getUpdateCounts();
ArrayList<Integer> failedRecsList = new ArrayList<Integer>();
int failCount = 0;
for (int i = 0; i < updateCount.length; i++) {
if (updateCount[i] == Statement.EXECUTE_FAILED) {
failCount++;
failedRecsList.add(i);
}
}
System.out.println(operation + " Failed Count: " + failCount);
System.out.println(operation + " FailedRecordsIndex:" + failedRecsList);
}
After execute, irrespective of success or failure of the batch, the batch will be cleared. You will need to repopulate the batch before you can retry.
I think you should just move the clearBatch() outside of the try. If you have to recheck whether it's an insert or update, so be it. If psInsert and psUdate are the same class, or derive from the same base class (and it sounds like they should), you can easily call it in a separate one liner method.
I also think if you submit this question to code review, you'd get some good suggestions to improve the code. I'm not saying it's terrible, but I think there's room for improvement in the underlying model. I'm not familiar enough with Java, though.

SQLiteException: near "null": syntax error (code 1): , while compiling: create table

Please help me, i get this errors when i run my app use Idea
here is screen of error
http://prntscr.com/en3lcu
here is part of code where create table
private String getTableDDL(final Class<? extends GlassContract.Table> table) {
return getTableDDL(table, GlassContract.getTableName(table));
}
private String getTableDDL(final Class<? extends GlassContract.Table> table, String tableName) {
final StringBuilder sql = new StringBuilder(128);
sql.append("create table ").append(tableName).append(" (");
for (final Field field : table.getFields()) {
if (field.getName().startsWith("_") || field.isAnnotationPresent(Deprecated.class))
continue;
try {
sql.append(field.get(null));
} catch (Exception ignore) {
}
try {
final Field type = table.getDeclaredField("_SQL_" + field.getName() + "_TYPE");
sql.append(' ').append(type.get(null));
} catch (Exception ignore) {
sql.append(" TEXT");
}
sql.append(',');
}
try {
final Field type = table.getDeclaredField("_PK_COMPOSITE");
sql.append("PRIMARY KEY(").append(type.get(null)).append(")");
sql.append(',');
} catch (Exception ignore) {
// ignore
}
try {
final Field type = table.getDeclaredField("_UNIQUE_COMPOSITE");
sql.append("UNIQUE(").append(type.get(null)).append(")");
sql.append(',');
} catch (Exception ignore) {
// ignore
}
sql.setLength(sql.length() - 1); // chop off last comma
sql.append(')');
Log.v(TAG, "DDL for " + table.getSimpleName() + ": " + sql);
return sql.toString();
}
I please help me, because I break my head))
Are you really trying to create text fields in your table that are named null?
Even if this works (and I am not sure it does), you are duplicating this and creating two identically named fields called null
The first field in that CREATE TABLE statement doesn't have a proper name (it is "null"). That's why it blows up.
There's more fields with illegal names as well.

Java DOM XML Parser - printed XML contains one tag instead of multiple ones

I have written some code loading records from MySQL and parsing them and saving into XML file. What I wanted to achieve was somethin like this:
<part><row>.....</row><row>...</row></part>
Now I have:
<part><row>...</row></part>
Here is screen showing my poroblem:
Here is Java code section which parses data into XML Document structure:
private void callSPInParOrWithout(final Document doc,
final Connection conn) {
ResultSet rs = null;
CallableStatement cs = null;
try {
// <part>
Element part = doc.createElement("part");
doc.appendChild(part);
cs = conn.prepareCall("{CALL getBrandRows(?)}");
cs.setString(1, "Brand#13");
boolean results = cs.execute();
while (results) {
// <row>
Element row = doc.createElement("row");
part.appendChild(row);
rs = cs.getResultSet();
while (rs.next()) {
// <p_partkey>
Element pPartKey = doc.createElement("p_partkey");
pPartKey.appendChild(doc.createTextNode(Integer.toString(
rs.getInt("p_partkey"))));
row.appendChild(pPartKey);
// <p_name>
Element pName = doc.createElement("p_name");
pName.appendChild(doc.createTextNode(rs.getString(
"p_name")));
row.appendChild(pName);
// <p_mfgr>
Element pMfgr = doc.createElement("p_mfgr");
pMfgr.appendChild(doc.createTextNode(rs.getString(
"p_mfgr")));
row.appendChild(pMfgr);
// <p_brand>
Element pBrand = doc.createElement("p_brand");
pBrand.appendChild(doc.createTextNode(rs.getString(
"p_brand")));
row.appendChild(pBrand);
// <p_type>
Element pType = doc.createElement("p_type");
pType.appendChild(doc.createTextNode(rs.getString(
"p_type")));
row.appendChild(pType);
// <p_size>
Element pSize = doc.createElement("p_size");
pSize.appendChild(doc.createTextNode(Integer.toString(
rs.getInt("p_size"))));
row.appendChild(pSize);
// <p_container>
Element pContainer = doc.createElement("p_container");
pContainer.appendChild(doc.createTextNode(rs.getString(
"p_container")));
row.appendChild(pContainer);
// <p_retailprice>
Element pRetailPrice = doc.createElement("p_retailprice");
pRetailPrice.appendChild(doc.createTextNode(
Float.toString(rs.getFloat("p_retailprice"))));
// <p_comment>
Element pComment = doc.createElement("p_comment");
pComment.appendChild(doc.createTextNode(rs.getString(
"p_comment")));
row.appendChild(pComment);
}
results = cs.getMoreResults();
}
} catch (SQLException e) {
JOptionPane.showMessageDialog(null, e.getMessage(),
"Exception occured", JOptionPane.ERROR_MESSAGE);
} finally {
try {
if (rs != null) rs.close();
if (cs != null) cs.close();
} catch (SQLException e) {
}
}
}
As you can see in the screenshot it works well but I made some mistake with creating <row> tag.
You are creating the row element only once, before iterating over all the rows in the ResultSet.
To get the expected result, create the row element inside the while (rs.next()) loop, and move part.appendChild(row); to the end of that loop's body.
CallableStatement#execute() indicates if a result set is available. CallableStatement#getMoreResults() indicates if another result set is available; there is no use in calling it unless your statement returns multiple result sets (see also this answer). If your CallableStatement returns only one result set, you can safely use an if in place of the outer while, and remove the call to getMoreResults().

Why my query result returns me false

I'm new with java and sql query and for the user connexion, I connect to the DB and check if the login exists. Here is what I do :
requete = "SELECT Login,Password,DroitModifAnnuaire,DroitRecepteurDem,DroitResponsableDem,PiloteIso,Administrateur,DroitNews,DroitTenues,DroitEssai,Nom,Prenom FROM Annuaire WHERE Login='"
+ (request.getParameter("login") + "'");
instruction = connexion.createStatement();
jeuResultats = instruction.executeQuery(requete);
try{
jeuResultats.next();
} catch (SQLException e) {
e.printStackTrace();
}
if (jeuResultats.next() == false) {
loadJSP("/index.jsp", request, reponse);
}else {
loadJSP("/views/menu.jsp", request, reponse);
}
The login that I enter is good but it redirect me to index.jspand I have the error : the result set has no current row
I tried to search answer to this error but I didn't found. So why it returns me false ? While when I do System.out.println(jeuResultats.getString(1)); the login is printed.
jeuResultats.next(); moves your result to the next row. You start with 0th row, i.e. when you call .next() it reads the first row, then when you call it again, it tries to read the 2nd row, which does not exist.
Some additional hints, not directly related to the question:
Java Docs are a good place to start Java 8 ResultSet, for e.x., perhaps ResultSet.first() method may be more suited for your use.
Since you are working with resources, take a look at try-with-resources syntax. Official tutorials are a good starting point for that.
Also take a look at prepared statement vs Statement. Again, official guide is a good place to start
Make the below changes in you code. Currently the next() method is shifting result list to fetch the data at 1st index, whereas the data is at the 0th Index:
boolean result = false;
try{
result = jeuResultats.next();
} catch (SQLException e) {
e.printStackTrace();
}
if (!result) {
loadJSP("/index.jsp", request, reponse);
}else {
loadJSP("/views/menu.jsp", request, reponse);
}
Replace your code by below code:
requete = "SELECT Login, Password, DroitModifAnnuaire, DroitRecepteurDem, DroitResponsableDem, PiloteIso, Administrateur, DroitNews, DroitTenues, DroitEssai, Nom, Prenom FROM Annuaire WHERE Login = '"
+ (request.getParameter("login") + "'");
instruction = connexion.createStatement();
jeuResultats = instruction.executeQuery(requete);
try{
if (jeuResultats.next()) {
loadJSP("/index.jsp", request, reponse);
} else {
loadJSP("/views/menu.jsp", request, reponse);
}
} catch (SQLException e) {
e.printStackTrace();
}

This codes can't check NoResultException

This is the Code I was made. It shouldn't get and enter to NoResultException, but it doesn't as expected. There is an unused data. I try to print out, here is the output : "[ ]"
private void deleteButtonActionPerformed(java.awt.event.ActionEvent evt) {
try {
int row = tableDataRangka.getSelectedRow();
String idRangka = tableDataRangka.getValueAt(row, 0).toString();
System.out.println( relasiRumahkayuRangkaDAO.getRelasiByIdRangka(idRangka).toString() );
} catch (NoResultException nre) {
// Doing something..
} catch (Exception ex) {
Logger.getLogger(MasterDataProjectUI.class.getName()).log(Level.SEVERE, null, ex);
}
}
Here is the code of method "getRelasiByIdRangka" :
public List<RelasiRumahKayuRangka> getRelasiByIdRangka(String idRangka) throws Exception{
initEntityManager();
List<RelasiRumahKayuRangka> rrDrs = new ArrayList<>();
Query q = em.createNamedQuery("RelasiRumahKayuRangka.findByIdRangka");
q.setParameter("idRangka", idRangka);
rrDrs.addAll(q.getResultList());
closeEntityManager();
return rrDrs;
}
And this one is the JPA query, findByIdRangka :
#NamedQuery(name = "RelasiRumahKayuRangka.findByIdRangka", query = "SELECT r FROM RelasiRumahKayuRangka r WHERE r.relasiRumahKayuRangkaPK.idRangka = :idRangka"),
Do you guys know the solution, so the code can be catched by NoResultException ?
In your code. if the result set is empty then list will also be empty.
Make use of that situation and throw a desired exception.
In your getRelasiByIdRangka(String idRangka) add following changes.
public List<RelasiRumahKayuRangka>
getRelasiByIdRangka( String idRangka ) throws Exception {
initEntityManager();
// rest of the code here
// ...
// check if some results are found or not
if( rrDrs.isEmpty() ) { // or ( rrDrs.size() == 0 )
throw new NoResultException( "No results found" );
} // if empty
// this should be a filled in list
return rrDrs;
} // end of method

Categories

Resources