Develop Reference

Backslash not working for postgres query from jdbc [duplicate]

Unable to use copy command with jdbc Postgres. Whats wrong with the below code snippet sample.
public boolean loadReportToDB(String date) {
// TODO Auto-generated method stub
Connection connection = DBUtil.getConnection("POSTGRESS");
String fileName = "C:/_0STUFF/NSE_DATA/nseoi_" + date + ".csv";
String sql = "\\copy fno_oi FROM 'C:\\_0STUFF\\NSE_DATA\\nseoi_27102017.csv' DELIMITER ',' CSV header";
try {
PreparedStatement ps = connection.prepareStatement(sql);
System.out.println("query"+ps.toString());
int rowsaffected = ps.executeUpdate();
System.out.println("INT+" + rowsaffected);
return true;
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return false;
}
org.postgresql.util.PSQLException: ERROR: syntax error at or near "\"
Position: 1
at org.
if we use
String sql = "copy fno_oi FROM 'C:\\_0STUFF\\NSE_DATA\\nseoi_27102017.csv' DELIMITER ',' CSV header";
then no rows are updated
postgres version postgresql-10.0-1-windows-x64

This works for me:
try (Connection conn = DriverManager.getConnection(connUrl, myUid, myPwd)) {
long rowsInserted = new CopyManager((BaseConnection) conn)
.copyIn(
"COPY table1 FROM STDIN (FORMAT csv, HEADER)",
new BufferedReader(new FileReader("C:/Users/gord/Desktop/testdata.csv"))
);
System.out.printf("%d row(s) inserted%n", rowsInserted);
}
Using copyIn(String sql, Reader from) has the advantage of avoiding issues where the PostgreSQL server process is unable to read the file directly, either because it lacks permissions (like reading files on my Desktop) or because the file is not local to the machine where the PostgreSQL server is running.

As your input file is stored locally on the computer running your Java program you need to use the equivalent of copy ... from stdin in JDBC because copy can only access files on the server (where Postgres is running).
To do that use the CopyManager API provided by the JDBC driver.
Something along the lines:
Connection connection = DBUtil.getConnection("POSTGRES");
String fileName = "C:/_0STUFF/NSE_DATA/nseoi_" + date + ".csv";
String sql = "copy fno_oi FROM stdin DELIMITER ',' CSV header";
BaseConnection pgcon = (BaseConnection)conection;
CopyManager mgr = new CopyManager(pgcon);
try {
Reader in = new BufferedReader(new FileReader(new File(fileName)));
long rowsaffected = mgr.copyIn(sql, in);
System.out.println("Rows copied: " + rowsaffected);
} catch (SQLException e) {
e.printStackTrace();
}

Append row in Sql without replacing previous row java

So i have database with value like this...
i'm trying to append the value by using insert into without replacing it,the data from this txt file...
but when i reload/refresh the database there is no new data being appended into the database...,
here is my code....
public static void importDatabase(String fileData){
try{
File database = new File(fileData);
FileReader fileInput = new FileReader(database);
BufferedReader in = new BufferedReader(fileInput);
String line = in.readLine();
line = in.readLine();
String[] data;
while (line != null){
data = line.split(",");
int ID = Integer.parseInt(data[0]);
String Nama = data[1];
int Gaji = Integer.parseInt(data[2]);
int Absensi = Integer.parseInt(data[3]);
int cuti = Integer.parseInt(data[4]);
String Status = data[5];
String query = "insert into list_karyawan values(?,?,?,?,?,?)";
ps = getConn().prepareStatement(query);
ps.setInt(1,ID);
ps.setString(2,Nama);
ps.setInt(3,Gaji);
ps.setInt(4,Absensi);
ps.setInt(5,cuti);
ps.setString(6,Status);
line = in.readLine();
}
ps.executeUpdate();
ps.close();
con.close();
System.out.println("Database Updated");
in.close();
}catch (Exception e){
System.out.println(e);
}
}
When i run it, it shows no error but the data never get into database, where did i go wrong?.,...

Auto-commit mode is enabled by default.
The JDBC driver throws a SQLException when a commit or rollback operation is performed on a connection that has auto-commit set to true.
Symptoms of the problem can be unexpected application behavior
update the JVM configuration for the ActiveMatrix BPM node to use the following Oracle connection property:
autoCommitSpecCompliant=false Try once
Note:I am not able to put as comment so i posted as a answer

java - Jar file can't be executed on another pc

I have an executable jar file I compiled from my program and I ran it on my PC. It works perfectly fine when I ran it in my command prompt using java -jar [nameofjar.jar]
However, I tried testing it on another pc. Using command prompt to run the same jar file, it throws an error:
D:\QA06122018_2>java -jar Indexing.jar
java.lang.NullPointerException
at IndexDriver.processText(IndexDriver.java:81)
at IndexDriver.index(IndexDriver.java:140)
at Main.main(Main.java:44).....
Both PC are using the same operating system and settings.
I even looked at the code regarding the error and there doesn't seem to be any problem with it. Ran fine on my IDE.
Is there anything I might overlooked?
EDIT:
The code :
public PreparedStatement preparedStatement = null;
MysqlAccessIndex con = new MysqlAccessIndex();
public Connection con1 = con.connect();
String path1;
public void index() throws Exception {
// Connection con1 = con.connect();
try {
Statement statement = con1.createStatement();
ResultSet rs = statement.executeQuery("select * from filequeue where Status='Active' LIMIT 5");
while (rs.next()) {
// get the filepath of the PDF document
path1 = rs.getString(2);
int getNum = rs.getInt(1);
Statement test = con1.createStatement();
test.executeUpdate("update filequeue SET STATUS ='Processing' where UniqueID="+getNum);
try {
// call the index function
PDDocument document = PDDocument.load(new File(path1),MemoryUsageSetting.setupTempFileOnly());
if (!document.isEncrypted()) {
PDFTextStripper tStripper = new PDFTextStripper();
for(int p=1; p<=document.getNumberOfPages();++p) {
tStripper.setStartPage(p);
tStripper.setEndPage(p);
try {
String pdfFileInText = tStripper.getText(document);
processText(pdfFileInText);
System.out.println("Page "+p+" done");
}catch (Exception e){
e.printStackTrace();
Statement statement1 = con1.createStatement();
statement1.executeUpdate("update filequeue SET Error ='E0003' where UniqueID="+getNum);
statement1.executeUpdate("update filequeue SET Status ='Error' where UniqueID="+getNum);
con1.commit();
con1.close();
}
}
}
// After completing the process, update status: Complete
Statement pre= con1.createStatement();
pre.executeUpdate("update filequeue SET STATUS ='Complete' where UniqueID="+getNum);
// con1.commit();
preparedStatement.close();
document.close();
System.out.println("Successfully commited changes to the database!");
con1.commit();
// con1.close();
// updateComplete_DB(getNum);
} catch (Exception e) {
try {
System.err.println(e);
Statement statement1 = con1.createStatement();
statement1.executeUpdate("update filequeue SET STATUS ='Error' where UniqueID="+getNum);
statement1.executeUpdate("update filequeue SET Error ='E0002' where UniqueID="+getNum);
con1.commit();
// add rollback function
rollbackEntries();
}catch (Exception e1){
System.out.println("Could not rollback updates :" + e1.getMessage());
}
}
// con1.close();
}
}catch(Exception e){
e.printStackTrace();
//System.out.println("lalala");
}
//con1.commit();
con1.close();
}
Calling the method:
public void processText(String text) throws SQLException {
String lines[] = text.split("\\r?\\n");
for (String line : lines) {
String[] words = line.split(" ");
String sql="insert IGNORE into test.indextable values (?,?);";
preparedStatement = con1.prepareStatement(sql);
int i=0;
for (String word : words) {
// check if one or more special characters at end of string then remove OR
// check special characters in beginning of the string then remove
// insert every word directly to table db
word=word.replaceAll("([\\W]+$)|(^[\\W]+)", "");
preparedStatement.setString(1, path1);
preparedStatement.setString(2, word);
preparedStatement.executeUpdate();
}
}
preparedStatement.close();
}

The root cause is that there were no lines to process.
You appear to only create prepared statements inside the for (String line : lines) { loop. But you only close the last statement you created (outside that loop).
When you don't have any lines, preparedStatement is null, because you never created one.
Even when you have lines to process, you are creating lots of prepared statements but only closing the last one.
You should probably create one prepared statement at the start of the method and reuse it for the whole method, closing it at the end.

Correct way to use copy Postgres jdbc

Unable to use copy command with jdbc Postgres. Whats wrong with the below code snippet sample.
public boolean loadReportToDB(String date) {
// TODO Auto-generated method stub
Connection connection = DBUtil.getConnection("POSTGRESS");
String fileName = "C:/_0STUFF/NSE_DATA/nseoi_" + date + ".csv";
String sql = "\\copy fno_oi FROM 'C:\\_0STUFF\\NSE_DATA\\nseoi_27102017.csv' DELIMITER ',' CSV header";
try {
PreparedStatement ps = connection.prepareStatement(sql);
System.out.println("query"+ps.toString());
int rowsaffected = ps.executeUpdate();
System.out.println("INT+" + rowsaffected);
return true;
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return false;
}
org.postgresql.util.PSQLException: ERROR: syntax error at or near "\"
Position: 1
at org.
if we use
String sql = "copy fno_oi FROM 'C:\\_0STUFF\\NSE_DATA\\nseoi_27102017.csv' DELIMITER ',' CSV header";
then no rows are updated
postgres version postgresql-10.0-1-windows-x64

This works for me:
try (Connection conn = DriverManager.getConnection(connUrl, myUid, myPwd)) {
long rowsInserted = new CopyManager((BaseConnection) conn)
.copyIn(
"COPY table1 FROM STDIN (FORMAT csv, HEADER)",
new BufferedReader(new FileReader("C:/Users/gord/Desktop/testdata.csv"))
);
System.out.printf("%d row(s) inserted%n", rowsInserted);
}
Using copyIn(String sql, Reader from) has the advantage of avoiding issues where the PostgreSQL server process is unable to read the file directly, either because it lacks permissions (like reading files on my Desktop) or because the file is not local to the machine where the PostgreSQL server is running.

As your input file is stored locally on the computer running your Java program you need to use the equivalent of copy ... from stdin in JDBC because copy can only access files on the server (where Postgres is running).
To do that use the CopyManager API provided by the JDBC driver.
Something along the lines:
Connection connection = DBUtil.getConnection("POSTGRES");
String fileName = "C:/_0STUFF/NSE_DATA/nseoi_" + date + ".csv";
String sql = "copy fno_oi FROM stdin DELIMITER ',' CSV header";
BaseConnection pgcon = (BaseConnection)conection;
CopyManager mgr = new CopyManager(pgcon);
try {
Reader in = new BufferedReader(new FileReader(new File(fileName)));
long rowsaffected = mgr.copyIn(sql, in);
System.out.println("Rows copied: " + rowsaffected);
} catch (SQLException e) {
e.printStackTrace();
}

Improve speed of SQL Inserts into XML column from JDBC (SQL Server)

I am currently writing a Java program which loops through a folder of around 4000 XML files.
Using a for loop, it extracts the XML from each file, assigns it to a String 'xmlContent', and uses the PreparedStatement method setString(2,xmlContent) to insert the String into a table stored in my SQL Server.
The column '2' is a column called 'Data' of type XML.
The process works, but it is slow. It inserts about 50 rows into the table every 7 seconds.
Does anyone have any ideas as to how I could speed up this process?
Code:
{ ...declaration, connection etc etc
PreparedStatement ps = con.prepareStatement("INSERT INTO Table(ID,Data) VALUES(?,?)");
for (File current : folder.listFiles()){
if (current.isFile()){
xmlContent = fileRead(current.getAbsoluteFile());
ps.setString(1, current.getAbsoluteFile());
ps.setString(2, xmlContent);
ps.addBatch();
if (++count % batchSize == 0){
ps.executeBatch();
}
}
}
ps.executeBatch(); // performs insertion of leftover rows
ps.close();
}
private static String fileRead(File file){
StringBuilder xmlContent = new StringBuilder();
FileReader fr = new FileReader(file);
BufferedReader br = new BufferedReader(fr);
String strLine = "";
br.readLine(); //removes encoding line, don't need it and causes problems
while ( (strLine = br.readLine() ) != null){
xmlContent.append(strLine);
}
fr.close();
return xmlContent.toString();
}

Just from a little reading and a quick test - it looks like you can get a decent speedup by turning off autoCommit on your connection. All of the batch query tutorials I see recommend it as well. Such as http://www.tutorialspoint.com/jdbc/jdbc-batch-processing.htm
Turn it off - and then drop an explicit commit where you want (at the end of each batch, at the end of the whole function, etc).
conn.setAutoCommit(false);
PreparedStatement ps = // ... rest of your code
// inside your for loop
if (++count % batchSize == 0)
{
try {
ps.executeBatch();
conn.commit();
}
catch (SQLException e)
{
// .. whatever you want to do
conn.rollback();
}
}

Best make the read and write parallel.
Use one thread to read the files and store in a buffer.
Use another thread to read from the buffer and execute queries on database.
You can use more than one thread to write to the database in parallel. That should give you even better performance.
I would suggest you follow this MemoryStreamMultiplexer approach where you can read the XML files in one thread and store in a buffer and then use one or more thread to read from the buffer and execute against database.
http://www.codeproject.com/Articles/345105/Memory-Stream-Multiplexer-write-and-read-from-many
It is a C# implementation, but you get the idea.