Android - Importing multiple .CSV Files to multiple tables in SQLite database - java

I have in my Android device a folder with multiple .CSV files.
I want to import all of them to my SQLite Database, but each file must be a different table.
All .CSV file are simple. They have just one column.
Example:
File.CSV
12345
123
00000000
AnotherFile.CSV
XXXXX
ZZZZZZZZZZ
FFFF
Here is my method and it is not working. I could not understand why:
#TargetApi(Build.VERSION_CODES.M)
public void importaTabelas() {
//Check the read permission
if (checkSelfPermission(Manifest.permission.READ_EXTERNAL_STORAGE) == PackageManager.PERMISSION_GRANTED) {
try {
//Check if the folder exists
File importDir = new File (Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS) + "/ENEL/IMPORTADOS/");
if (!importDir.exists())
{
importDir.mkdirs();
}
//Read all file names
for (File f : importDir.listFiles()) {
if (f.isFile()) {
//Put the files names into variable nomeArq
nomeArq = f.getName();
//Take off the file extension .csv
if (nomeArq.indexOf(".") > 0)
nomeArq = nomeArq.substring(0, nomeArq.lastIndexOf("."));
SQLiteDatabase db = this.banco.getWritableDatabase();
try {
//Create table with the name of the .csv file
String criaTab = "CREATE TABLE IF NOT EXISTS " + nomeArq + " (id integer PRIMARY KEY AUTOINCREMENT, codigo varchar (50))";
db.execSQL(criaTab);
db.close();
} catch (SQLException e) {
e.printStackTrace();
}
//String for the file location
String fn = importDir + "/" + nomeArq + ".csv";
//Reads the file
FileReader fileReader = new FileReader(fn);
BufferedReader buffer = new BufferedReader(fileReader);
//ContentValues contentValues = new ContentValues();
String line = "";
//db.beginTransaction();
while ((line = buffer.readLine()) != null) {
//String[] colums = line.split("\t");
//String[] colums = line.split(";");
Toast.makeText(this, line, Toast.LENGTH_SHORT).show();
//contentValues.put("codigo", line);
//db.insert(nomeArq, null, contentValues);
db.execSQL("INSERT INTO " + nomeArq + " (codigo) VALUES ('" + line + "')");
}
//db.setTransactionSuccessful();
//db.endTransaction();
}
}
} catch (Exception e) {
Toast.makeText(this, "Catch!", Toast.LENGTH_SHORT).show();
e.printStackTrace();
}
}
else {
requestPermissions(new String[]{Manifest.permission.READ_EXTERNAL_STORAGE}, 1);
}
}
Would you help me to make it works? Thanks!


I refactored your code into a bunch of smaller methods. Each method is responsible for one thing, which is good practice, try to do that any time you can.
I only did a few changes:
The line creating the FileReader now uses the file (less error prone)
Changed the way you insert by creating a single insert query so you have less database accesses (ie like this: INSERT INTO table(codigo) VALUES ('XXXX'), ('ZZZZ'), ('FFFF'); )
Changed the text in the Toasts to identify better where your error comes from.
Try it out and see if you can find your error better.
(I did not try to compile the code so you might have to tweak it a little but should be fine overall)
main import method:
public void importaTabelas() {
//Check the read permission
if (checkSelfPermission(Manifest.permission.READ_EXTERNAL_STORAGE) == PackageManager.PERMISSION_GRANTED) {
try {
makeDirs();
//Read all file names
for (File f : importDir.listFiles()) {
if (f.isFile()) {
importFile(f);
}
}
} catch (IOException e) {
Toast.makeText(this, "Could not import tables! " + e.getMessage(), Toast.LENGTH_SHORT).show();
e.printStackTrace();
}
} else {
requestPermissions(new String[]{Manifest.permission.READ_EXTERNAL_STORAGE}, 1);
}
}
Make the directories
private void makeDirs() {
//Check if the folder exists
File importDir = new File (Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS) + "/ENEL/IMPORTADOS/");
if (!importDir.exists()) {
importDir.mkdirs();
}
}
Import a single file
private void importFile(File f) {
try {
SQLiteDatabase db = this.banco.getWritableDatabase();
//Put the files names into variable nomeArq
String nomeArq = f.getName();
//Take off the file extension .csv
if (nomeArq.indexOf(".") > 0)
nomeArq = nomeArq.substring(0, nomeArq.lastIndexOf("."));
createTable(db, nomeArq);
String insertQuery = buildImportQuery(f, nomeArq);
db.execSQL(insertQuery);
} catch (SQLException e) {
Toast.makeText(this, "Could not import file. " + e.getMessage(), Toast.LENGTH_SHORT).show();
e.printStackTrace();
}
}
Builds the insert query for a specific file
private String buildImportQuery(File f, String nomeArq) {
StringBuilder sb = new StringBuilder();
try {
//Reads the file
FileReader fileReader = new FileReader(f);
BufferedReader buffer = new BufferedReader(fileReader);
String line;
sb.append("INSERT INTO " + nomeArq + " (codigo) VALUES ");
boolean addComma = false;
while ((line = buffer.readLine()) != null) {
if(line.length() > 0) {
if(addComma) {
sb.append(",");
}
sb.append("('" + line.trim() + "')");
addComma = true;
}
}
sb.append(";");
} catch (IOException e) {
Toast.makeText(this, "Could not write query. " + e.getMessage(), Toast.LENGTH_SHORT).show();
e.printStackTrace();
}
return sb.toString();
}
Creates a single table
private void createTable(SQLiteDatabase db, String tableName) {
try {
//Create table with the name of the .csv file
String criaTab = "CREATE TABLE IF NOT EXISTS " + tableName + " (id integer PRIMARY KEY AUTOINCREMENT, codigo varchar (50))";
db.execSQL(criaTab);
db.close();
} catch (Exception e) {
Toast.makeText(this, "Could not create table " + tableName + "." + e.getMessage(), Toast.LENGTH_SHORT).show();
e.printStackTrace();
}
}

Related

Doesn't insert csv file records in java (jdbc)

I have an error when my CSVLoader file does not insert the data in my database, I get this in my console
FieldName (UploadDownloadFileServlet) = fileName
FileName (UploadDownloadFileServlet) = prueba2.csv
ContentType (UploadDownloadFileServlet) = application/vnd.ms-excel
Size in bytes (UploadDownloadFileServlet) = 47
Absolute Path at server (UploadDownloadFileServlet) = C:\Users\SISTEMAS\workspaceEclipse.metadata.plugins\org.eclipse.wst.server.core\tmp0\wtpwebapps\SAC-sin-impresion\uploadedCsvFiles\prueba2.csv
Query: INSERT INTO MATRICULA(C:UsersSISTEMASworkspaceEclipse.metadata.pluginsorg.eclipse.wst.server.coretmp0wtpwebappsSAC-sin-impresionuploadedCsvFilesprueba2.csv) VALUES(?)
[CSVLoader]: 0 records loaded into MATRICULA DB table
THIS IS MY CODE:
public void loadCSV(InputStream csvFile, String tableName,
boolean truncateBeforeLoad) throws Exception {
CSVReader csvReader = null;
if(null == this.connection) {
throw new Exception("Not a valid connection.");
}
try {
/* Modified by rammar.
*
* I was having issues with the CSVReader using the "\" to escape characters.
* A MySQL CSV file contains quote-enclosed fields and non-quote-enclosed NULL
* values written as "\N". The CSVReader was removing the "\". To detect "\N"
* I must remove the escape character, and the only character you can replace
* it with that you are pretty much guaranteed will not be used to escape
* text is '\0'.
* I read this on:
* http://stackoverflow.com/questions/6008395/opencsv-in-java-ignores-backslash-in-a-field-value
* based on:
* http://sourceforge.net/p/opencsv/support-requests/5/
*/
// PREVIOUS VERSION: csvReader = new CSVReader(new FileReader(csvFile), this.seprator);
csvReader = new CSVReader(new InputStreamReader(csvFile), this.seprator, '"', '\0');
} catch (Exception e) {
e.printStackTrace();
throw new Exception("Error occured while executing file. "
+ e.getMessage());
}
String[] headerRow = csvReader.readNext();
if (null == headerRow) {
throw new FileNotFoundException(
"No columns defined in given CSV file." +
"Please check the CSV file format.");
}
String questionmarks = StringUtils.repeat("?,", headerRow.length);
questionmarks = (String) questionmarks.subSequence(0, questionmarks
.length() - 1);
/* NOTE from Ron: Header column names must match SQL table fields */
String query = SQL_INSERT.replaceFirst(TABLE_REGEX, tableName);
query = query
.replaceFirst(KEYS_REGEX, StringUtils.join(headerRow, ","));
query = query.replaceFirst(VALUES_REGEX, questionmarks);
System.out.println("Query: " + query); // Modified by rammar to suppress output
String[] nextLine;
Connection con = null;
PreparedStatement ps = null;
try {
con = this.connection;
con.setAutoCommit(false);
ps = con.prepareStatement(query);
if(truncateBeforeLoad) {
//delete data from table before loading csv
con.createStatement().execute("DELETE FROM " + tableName);
}
final int batchSize = 1000;
int count = 0;
Date date = null;
while ((nextLine = csvReader.readNext()) != null) {
if (null != nextLine) {
int index = 1;
for (String string : nextLine) {
date = DateUtil.convertToDate(string);
if (null != date) {
ps.setDate(index++, new java.sql.Date(date
.getTime()));
} else {
/* Section modified by rammar to allow NULL values
* to be input into the DB. */
if (string.length() > 0 && !string.equals("\\N")) {
ps.setString(index++, string);
} else {
ps.setNull(index++, Types.VARCHAR);
//ps.setString(index++, null); // can use this syntax also - not sure which is better
}
}
}
ps.addBatch();
}
if (++count % batchSize == 0) {
ps.executeBatch();
}
}
ps.executeBatch(); // insert remaining records
System.out.println("[" + this.getClass().getSimpleName() + "]: " +
count + " records loaded into " + tableName + " DB table");
con.commit();
} catch (Exception e) {
con.rollback();
e.printStackTrace();
throw new Exception(
"Error occured while loading data from file to database."
+ e.getMessage());
} finally {
/*if (null != ps)
ps.close();
*/
/*if (null != con)
con.close();*/
csvReader.close();
}
}

Request parameters coming from jsps are changed when two different users access the code same time

public String generateDataPDF() {
System.out.println("Inside generate PDF");
String filePath = "";
HttpSession sess = ServletActionContext.getRequest().getSession();
try {
sess.setAttribute("msg", "");
if (getCrnListType().equalsIgnoreCase("F")) {
try {
filePath = getModulePath("CRNLIST_BASE_LOCATION") + File.separator + getCrnFileFileName();
System.out.println("File stored path : " + filePath);
target = new File(filePath);
FileUtils.copyFile(crnFile, target);
} catch (Exception e) {
System.out.println("File path Exception " + e);
}
}
System.out.println("Values from jsp are : 1)Mode of Generation : " + getCrnListType() + " 2)Policy Number : " + getCrnNumber() + " 3)Uploaded File Name : " + getCrnFileFileName() + " 4)LogoType : " + getLogoType()
+ " 5)Output Path : " + getOutputPath() + " 6)Type of Generation : " + getOptionId() + " 7)PDF Name : " + getPdfName());
String srtVAL = "";
String arrayVaue[] = new String[]{getCrnListType(), getCrnListType().equalsIgnoreCase("S") ? getCrnNumber() : filePath, getLogoType().equalsIgnoreCase("WL") ? "0" : "1",
getOutputPath(), getGenMode(), getRenType()};
//INS DB Connection
con = getInsjdbcConnection();
ArrayList selectedCRNList = new ArrayList();
String selectedCRNStr = "";
selectedCRNStr = getSelectedVal(selectedCRNStr, arrayVaue[1]);
String[] fileRes = selectedCRNStr.split("\\,");
if (fileRes[0].equalsIgnoreCase("FAIL")) {
System.out.println("fileRes is FAIL beacause of other extension file.");
sess.setAttribute("pr", "Please upload xls or csv file.");
return SUCCESS;
}
System.out.println("List file is : " + selectedCRNStr);
String st[] = srtVAL.split("[*]");
String billDateStr = DateUtil.getStrDateProc(new Date());
Timestamp strtPasrsingTm = new Timestamp(new Date().getTime());
String minAMPM = DateUtil.getTimeDate(new Date());
String str = "";
String batchID = callSequence();
try {
System.out.println("Inside Multiple policy Generation.");
String userName=sess.getAttribute("loginName").toString();
String list = getProcessesdList(userName);
if (list != null) {
System.out.println("list is not null Users previous data is processing.....");
//setTotalPDFgNERATEDmSG("Data is processing please wait.");
sess.setAttribute("pr","Batch Id "+list+" for User " + userName + " is currently running.Please wait till this Process complete.");
return SUCCESS;
}
String[] policyNo = selectedCRNStr.split("\\,");
int l = 0, f = 0,counter=1;
for (int j = 0; j < policyNo.length; j++,counter++) {
String pdfFileName = "";
int uniqueId=counter;
globUniqueId=uniqueId;
insertData(batchID, new Date(), policyNo[j], getOptionId(), userName,uniqueId);
System.out.println("Executing Proc one by one.");
System.out.println("policyNo[j]" + policyNo[j]);
System.out.println("getOptionId()" + getOptionId());
System.out.println("seqValue i.e batchId : " + batchID);
}
str = callProcedure(policyNo[j], getOptionId(), batchID);
String[] procResponse = str.split("\\|");
for (int i = 0; i < procResponse.length; i++) {
System.out.println("Response is : " + procResponse[i]);
}
if (procResponse[0].equals("SUCCESS")) {
Generator gen = new Generator();
if (getPdfName().equalsIgnoreCase("true")) {
System.out.println("Checkbox is click i.e true");
pdfFileName = procResponse[1];
} else {
System.out.println("Checkbox is not click i.e false");
String POLICY_SCH_GEN_PSS = getDetailsForFileName(userName, policyNo[j], batchID);
String[] fileName = POLICY_SCH_GEN_PSS.split("\\|");
if (getLogoType().equals("0") || getLogoType().equals("2")) {
System.out.println("If logo is O or 1");
pdfFileName = fileName[1];
} else if (getLogoType().equals("1")) {
System.out.println("If logo is 2");
pdfFileName = fileName[0];
}
}
b1 = gen.genStmt(procResponse[1], procResponse[2], "2", getLogoType(), "0", pdfFileName,"1",userName,batchID);
l++;
updateData(uniqueId,batchID, "Y");
} else {
f++;
updateData(uniqueId,batchID, "F");
}
}
sess.setAttribute("pr","Total "+l+" "+getGenericModulePath("PDF_RES1") + " " + " " + getGenericModulePath("PDF_RES2") + " " + f);
}catch (Exception e) {
updateData(globUniqueId,batchID, "F");
System.out.println("Exception in procedure call");
setTotalPDFgNERATEDmSG("Fail");
e.printStackTrace();
sess.setAttribute("pr", "Server Error.");
return SUCCESS;
}
}catch (Exception ex) {
ex.printStackTrace();
sess.setAttribute("pr", "Server Error.");
return SUCCESS;
}
System.out.println("Above second return");
return SUCCESS;
}
GenerateDataPDf method generates PDF based on the parameters i.e ProductType(GenMode),CrnList(uploaded in excel file...)Code works fine when only single user generates PDF. But If two different User(User and roles are assigned in application) start the process same time request paraeters are overridden then! Suppose first user request pdf for 50 customers for product 1. User1's process is still running and second user request for product2. Now User1's pdf are generated but for product2.....! Here batchId is unique for every single request.One table is maintained where batch_id,all pdf,generation flags are mainained there. How do I solve this?

As per your comment, this is what I would do, It's probably not the best way to do !
Firstly : Create a function to collet all your data at the beginning. You should not modify/update/create anything when you are generating a PDF. IE : array/list collectPDFData() wich should retourn an array/list.
Secondly : Use a synchronized methods like synchronized boolean generatePDF(array/list)
"Synchronized" methods use monitor lock or intrinsic lock in order to manage synchronization so when using synchronized, each method share the same monitor of the corresponding object.
NB : If you use Synchronize, it's probably useless to collect all your data in a separate way, but I think it's a good practice to make small function dedicated to a specific task.
Thus, your code should be refactored a little bit.

Android SQLite cursor returns 33 results, but debug only outputs 17

Hi folks I've got a strange cade. I'm trying to debug the SQLite DB in an app. If I do a query SELECT * from table I get 33 results, but if I iterate over the cursor it ends at result #17.
Here's my debug class (the method in question is public static void WriteToFile(Cursor cursor, String query , String tables, String uri)) :
package com.s2u.android.ps;
import java.io.File;
import java.io.FileWriter;
import java.io.PrintWriter;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Calendar;
import java.util.List;
import android.content.Context;
import android.database.Cursor;
import android.database.sqlite.SQLiteDatabase;
import android.os.Environment;
import android.util.Log;
import com.s2u.android.ps.BO.App;
import com.s2u.android.ps.BO.AppMember;
import com.s2u.android.ps.datamodel.DatabaseManager;
import com.s2u.android.ps.networkApis.AndroidLog;
import com.s2u.android.ps.networkApis.AppConfig;
public class DebugToFile {
private static String TAG = "DebugToFile";
private static File path = new File(Environment.getExternalStorageDirectory() + "/ps_debug");
public static void WriteToFile(String lines , String tag)
{
WriteToFile(lines , tag , "txt");
}
public static void WriteToFile(String lines , String tag , String ext)
{
if (!Validate("WriteToFile(String lines)"))
return;
File file = new File(path, tag + "_" + GetDateStampTens() + "." + ext);
try
{
FileWriter fw = new FileWriter(file.getAbsolutePath() , true);
PrintWriter pw = new PrintWriter(fw);
pw.println(GetDateStamp() + " - " + lines);
pw.println();
pw.flush();
pw.close();
//Log.e(TAG, "WriteToFile(String lines) - " + file.toString());
}
catch (Exception e)
{
Log.e(TAG, "WriteToFile(String lines) failed!", e);
}
}
public static void WriteToFileAppMember(ArrayList<AppMember> appMembers , String tag)
{
if (!Validate("WriteToFile(AppMember)"))
return;
File file = new File(path, tag + "_" + GetDateStampTens() + ".csv");
try
{
FileWriter fw = new FileWriter(file.getAbsolutePath() , true);
PrintWriter pw = new PrintWriter(fw);
pw.println(GetDateStamp() + " - " + "AppMembers");
boolean doOnce = true;
for(com.s2u.android.ps.BO.AppMember appMember : appMembers)
{
if (doOnce)
{
doOnce = false;
pw.println(appMember.getCsvLabels());
}
pw.println(appMember.getCsvString());
}
pw.println();
pw.flush();
pw.close();
//Log.e(TAG, "WriteToFile(String lines) - " + file.toString());
}
catch (Exception e)
{
Log.e(TAG, "WriteToFile(String lines) failed!", e);
}
}
public static void WriteToFileAppMember(List<AppMember> appMembers , String tag)
{
if (!Validate("WriteToFile(AppMember)"))
return;
File file = new File(path, tag + "_" + GetDateStampTens() + ".csv");
try
{
FileWriter fw = new FileWriter(file.getAbsolutePath() , true);
PrintWriter pw = new PrintWriter(fw);
pw.println(GetDateStamp() + " - " + "AppMembers");
boolean doOnce = true;
for(com.s2u.android.ps.BO.AppMember appMember : appMembers)
{
if (doOnce)
{
doOnce = false;
pw.println(appMember.getCsvLabels());
}
pw.println(appMember.getCsvString());
}
pw.println();
pw.flush();
pw.close();
//Log.e(TAG, "WriteToFile(String lines) - " + file.toString());
}
catch (Exception e)
{
Log.e(TAG, "WriteToFile(String lines) failed!", e);
}
}
public static void WriteToFileApps(List<App> apps , String tag)
{
if (!Validate("WriteToFile(AppMember)"))
return;
File file = new File(path, tag + "_" + GetDateStampTens() + ".csv");
try
{
FileWriter fw = new FileWriter(file.getAbsolutePath() , true);
PrintWriter pw = new PrintWriter(fw);
pw.println(GetDateStamp() + " - " + "App objects");
boolean doOnce = true;
for(com.s2u.android.ps.BO.App app : apps)
{
if (doOnce)
{
doOnce = false;
pw.println(app.getCsvLabels());
}
pw.println(app.getCsvString());
}
pw.println();
pw.flush();
pw.close();
//Log.e(TAG, "WriteToFile(String lines) - " + file.toString());
}
catch (Exception e)
{
Log.e(TAG, "WriteToFile(String lines) failed!", e);
}
}
public static void WriteToFile(Cursor cursor, String query , String tables, String uri)
{
if (!Validate("WriteToFile(cursor)"))
return;
File file = new File(path, uri + "_" + GetDateStampTens() + ".csv");
try
{
FileWriter fw = new FileWriter(file.getAbsolutePath(), true);
PrintWriter pw = new PrintWriter(fw);
int resultCount = cursor.getCount();
pw.println("time: " + GetDateStamp());
pw.println("tables: " + tables);
pw.println("query: " + query);
pw.println("result count: " + Integer.toString(resultCount));
int row = 0;
String labels = "row,";
int startPosition = cursor.getPosition();
cursor.moveToPosition(-1);
while (cursor.moveToNext())
{
int colCount = cursor.getColumnCount();
row++;
if (row >= resultCount)
{
pw.println("Error! rows >= cursor count -- at row : " + Integer.toString(row) );
break;
}
StringBuilder line = new StringBuilder(512);
if (colCount <= 0)
pw.println("Empty row?");
for(int i = 0; i < colCount; i++)
{
if (row == 1)
{
labels += cursor.getColumnName(i) + "[" + GetCursorFieldTypeString(cursor, i) + "]";
if (i < colCount - 1)
labels += ",";
}
if (i == 0)
line.append(Integer.toString(row) + ",");
line.append(GetCursorString(cursor, i));
if (i < colCount - 1)
{
line.append(",");
}
}
if (row == 1)
pw.println(labels);
pw.println(line.toString());
cursor.moveToNext();
if (row > 100)
{
pw.println("max rows output - stopped at row: " + Integer.toString(row));
break;
}
}
pw.println("END");
pw.println();
pw.flush();
pw.close();
//Log.e(TAG, "WriteToFile(cursor) - " + file.toString());
cursor.moveToPosition(startPosition);
}
catch (Exception e)
{
Log.e(TAG, "WriteToFile(cursor) failed!", e);
}
}
private static boolean Validate(String methodName)
{
if (!AppConfig.isTestBuild())
{
Log.i(TAG, methodName + " - this is not a test build!");
return false;
}
if (!isExternalStorageWritable())
{
AndroidLog.e(TAG, methodName + " - external storage not accessible");
return false;
}
if (!path.exists())
{
path.mkdir();
if (!path.exists())
{
AndroidLog.e(TAG, methodName + " - directory doesn't exist and couldn't create it: " + path.toString());
return false;
}
}
return true;
}
private static String GetDateStamp()
{
Calendar c = Calendar.getInstance();
SimpleDateFormat df = new SimpleDateFormat("yyyyMMdd-kk:mm:ss.SSS");
String date = df.format(c.getTime());
return date;
}
private static String GetDateStampTens()
{
String date = GetDateStamp();
date = date.substring(0,date.length() - 1) + "0";
return date;
}
private static boolean isExternalStorageWritable() {
String state = Environment.getExternalStorageState();
if (Environment.MEDIA_MOUNTED.equals(state)) {
return true;
}
return false;
}
private static String GetCursorString(Cursor cursor, Integer i)
{
String result = "undefined";
switch(cursor.getType(i))
{
case Cursor.FIELD_TYPE_NULL:
result = "NULL";
break;
case Cursor.FIELD_TYPE_BLOB:
result = "BLOB length: " + Integer.toString(cursor.getBlob(i).length);
break;
case Cursor.FIELD_TYPE_FLOAT:
result = Float.toString(cursor.getFloat(i));
break;
case Cursor.FIELD_TYPE_INTEGER:
result = Integer.toString(cursor.getInt(i));
break;
case Cursor.FIELD_TYPE_STRING:
result = cursor.getString(i);
break;
default:
result = "undefined cursor value type(" + Integer.toString(cursor.getType(i)) + ") -- try getString: " + cursor.getString(i);
}
result.replace("", " ");
return result;
}
private static String GetCursorFieldTypeString(Cursor cursor, Integer i)
{
String result = "UNK";
switch(cursor.getType(i))
{
case Cursor.FIELD_TYPE_NULL:
result = "NULL";
break;
case Cursor.FIELD_TYPE_BLOB:
result = "BLOB";
break;
case Cursor.FIELD_TYPE_FLOAT:
result = "F";
break;
case Cursor.FIELD_TYPE_INTEGER:
result = "INT";
break;
case Cursor.FIELD_TYPE_STRING:
result = "STR";
break;
default:
result = "UNK(" + Integer.toString(cursor.getType(i)) + ") ";
}
return result;
}
public static String AppListTypeToString(int appListType)
{
if (appListType == 0)
return "kAppListMain";
else if (appListType == 1)
return "kAppListProfile";
else if (appListType == 2)
return "kAppListPromoted";
return "unknown list type int: " + Integer.toString(appListType);
}
public static void DumpDatabaseToFiles(DatabaseManager db)
{
SQLiteDatabase readableDb = db.getReadableDatabase();
DumpDatabaseToFiles(readableDb);
}
public static void DumpDatabaseToFiles(SQLiteDatabase db)
{
if (!Validate("DumpDatabaseToFiles"))
return;
Cursor c = db.rawQuery("SELECT name FROM sqlite_master WHERE type='table'", null);
if (c.getCount() <= 0)
{
WriteToFile("table name count: " + Integer.toString(c.getCount()) , "dbdump_err");
c.close();
return;
}
//AndroidLog.i(TAG , "DumpDB table count: " + Integer.toString(c.getCount()));
List<String> tableNames = new ArrayList<String>();
if (c.moveToFirst())
{
while(!c.isAfterLast())
{
tableNames.add(c.getString(c.getColumnIndex("name")));
c.moveToNext();
}
}
c.close();
for (int i = 0; i < tableNames.size(); i++)
{
String table = tableNames.get(i);
c = db.rawQuery("SELECT * FROM " + table + " LIMIT 100 ", null);
WriteToFile(c, "all" , table, table);
c.close();
}
}
}
The output csv file is:
tables: app - from AppDAO.bulkInsertApp
query: SELECT * FROM app
result count: 33
row,_id[INT],packageName[STR],appName[STR],iconUrl1[STR],iconUrl2[NULL],publisher[STR],publisherEmail[NULL],price[INT],currency[STR],version[STR],category[STR],releaseDate[NULL],updatedOn[NULL],hasTried[INT],promo_url[NULL],promoParam[NULL],promoValueKey[NULL]
1,8192,com.shared2you.android.powerslyde,Powerslyde,https://lh5.ggpht.com/1qigt9Zz7oh5kTFiIS9ukJljVTm7W-Ur34XzcaQhFjc9GlMzATJ-ATRwYB6gxQhscHEU=w300,NULL,Shared2you, Inc.,NULL,0,, 1.08 ,Lifestyle,NULL,NULL,1,NULL,NULL,NULL
2,8219,com.android.providers.downloads.ui,com.android.providers.downloads.ui,NULL,NULL,NULL,NULL,NULL,,NULL,NULL,NULL,NULL,1,NULL,NULL,NULL
3,8225,com.google.android.apps.maps,Maps,https://lh3.ggpht.com/JW-F0fkeBHpKyh8lDcyQ7CveTRynYGByVBH9hUqnJxw4x64ORhoFJISdOWhekULemw0=w300,NULL,Google Inc.,NULL,0,, Varies with devic,Travel & Local,NULL,NULL,1,NULL,NULL,NULL
4,8231,com.android.vending,com.android.vending,NULL,NULL,NULL,NULL,NULL,,NULL,NULL,NULL,NULL,1,NULL,NULL,NULL
5,8246,com.google.android.apps.magazines,Google Play Newsstand,https://lh5.ggpht.com/rowOPaiODov-bNG7rnD6awPZwLnOc7Vzab-29GpfvB6jfE8DhOR42owBqAmLUXj-W2sI=w300,NULL,Google Inc.,NULL,0,, 3.1.0 ,News & Magazines,NULL,NULL,1,NULL,NULL,NULL
6,8248,com.google.android.gm,Gmail,https://lh4.ggpht.com/Ebn-CW55BnkwG7ng5nuGpijVpJeabTa-uPijd4keKbHpedz29SvDj3EZkfr20ZZzznE=w300,NULL,Google Inc.,NULL,0,, Varies with devic,Communication,NULL,NULL,1,NULL,NULL,NULL
7,8250,com.google.android.music,Google Play Music,https://lh6.ggpht.com/5opWBg-m6yFcjWzJz1LlT05YIf2Alyiy9YtpQm1f6U42LXWmCvB54M1zEkV9-hCaoTc=w300,NULL,Google Inc.,NULL,0,, Varies with devic,Music & Audio,NULL,NULL,1,NULL,NULL,NULL
8,8253,com.google.android.videos,Google Play Movies & TV,https://lh5.ggpht.com/fFPQTALNNU4xflvbazvbwPL5o4X3a_CqYHUWIh4FXmfU78aSSuP1OMkGXhXouxXzWPov=w300,NULL,Google Inc.,NULL,0,, Varies with devic,Media & Video,NULL,NULL,1,NULL,NULL,NULL
9,8312,com.android.chrome,Chrome Browser - Google,https://lh6.ggpht.com/lum4KYB0TtgvR-8vRMUZ_JhRnMQ4YqBIR0yjspc4ETsM9iJ8-4YHZ0s0HO9i0ez_=w300,NULL,Google Inc.,NULL,0,, Varies with devic,Communication,NULL,NULL,1,NULL,NULL,NULL
10,8316,com.google.android.calendar,Google Calendar,https://lh5.ggpht.com/qgUPYBPSTb61cPrijI9YXV3BEy00t5bhoBugDpEXTdEsQEv9B9-j8_ZDs_ClQzPbskc=w300,NULL,Google Inc.,NULL,0,, 201308023 ,Productivity,NULL,NULL,1,NULL,NULL,NULL
11,8433,com.estrongs.android.pop,ES File Explorer File Manager,https://lh5.ggpht.com/P31CiAbF5UMC1wbJxv2sPT4tSLLqfqUZPp8N0ATEaA0ZeMxXv_NjVDiswVKjeUUSS2w=w300,NULL,ES APP Group,NULL,0,, Varies with devic,Productivity,NULL,NULL,1,NULL,NULL,NULL
12,8867,com.devhd.feedly,Feedly,https://lh4.ggpht.com/rkouDgWbT3WNztDRa5QvnN8SatDK3zeHHwOMHZbiu2Vlf3-9hLlmH89W9gJpGEtxo3U=w300,NULL,Feedly Team,NULL,0,, 18.1.2 ,News & Magazines,NULL,NULL,1,NULL,NULL,NULL
13,8917,com.google.android.email,com.google.android.email,NULL,NULL,NULL,NULL,NULL,,NULL,NULL,NULL,NULL,1,NULL,NULL,NULL
14,12113,com.google.android.play.games,Google Play Games,https://lh5.ggpht.com/tkg8ndU21RjzO5WSz7JRpYJ35P-oDTm0md2sNwvVoBtQ0kE_ORHhorrzQWcjVTevxP8_=w300,NULL,Google Inc.,NULL,0,, 1.1.04 ,Entertainment,NULL,NULL,1,NULL,NULL,NULL
15,87853,com.google.android.apps.docs.editors.sheets,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,1,NULL,NULL,NULL
16,87862,com.google.android.apps.photos,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,1,NULL,NULL,NULL
17,87867,com.umfersolutions.eatthiszombies,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,1,NULL,NULL,NULL
END
Thanks!

You are advancing the cursor position two times, one in
while (cursor.moveToNext())
and the other one at the end of the loop in
pw.println(line.toString());
cursor.moveToNext();
Thats why you always will get half of the results, since at the end you move it one position, and then at then when checking the while condition it will advance again, so its reading position 0, then position 2, then 4...and so on...

Duplicate cursor.moveToNext() in the loop:
while (cursor.moveToNext())
{
...
pw.println(line.toString());
cursor.moveToNext();
...
}

Java CSVReader ignore commas in double quotes

I have a CSV file that I am having trouble parsing. I am using the opencsv library. Here is what my data looks like and what I am trying to achieve.
RPT_PE,CLASS,RPT_MKT,PROV_CTRCT,CENTER_NM,GK_TY,MBR_NM,MBR_PID
"20150801","NULL","33612","00083249P PCP602","JOE SMITH ARNP","NULL","FRANK, LUCAS E","50004655200"
The issue I am having is the member name ("FRANK, LUCAS E") is being split into two columns and the member name should be one. Again I'm using opencsv and a comma as the separator. Is there any way I can ignore the commas inside the double-quotes?
public void loadCSV(String csvFile, String tableName,
boolean truncateBeforeLoad) throws Exception {
CSVReader csvReader = null;
if (null == this.connection) {
throw new Exception("Not a valid connection.");
}
try {
csvReader = new CSVReader(new FileReader(csvFile), this.seprator);
} catch (Exception e) {
e.printStackTrace();
throw new Exception("Error occured while executing file. "
+ e.getMessage());
}
String[] headerRow = csvReader.readNext();
if (null == headerRow) {
throw new FileNotFoundException(
"No columns defined in given CSV file."
+ "Please check the CSV file format.");
}
String questionmarks = StringUtils.repeat("?,", headerRow.length);
questionmarks = (String) questionmarks.subSequence(0, questionmarks
.length() - 1);
String query = SQL_INSERT.replaceFirst(TABLE_REGEX, tableName);
System.out.println("Base Query: " + query);
String headerRowMod = Arrays.toString(headerRow).replaceAll(", ]", "]");
String[] strArray = headerRowMod.split(",");
query = query
.replaceFirst(KEYS_REGEX, StringUtils.join(strArray, ","));
System.out.println("Add Headers: " + query);
query = query.replaceFirst(VALUES_REGEX, questionmarks);
System.out.println("Add questionmarks: " + query);
String[] nextLine;
Connection con = null;
PreparedStatement ps = null;
try {
con = this.connection;
con.setAutoCommit(false);
ps = con.prepareStatement(query);
if (truncateBeforeLoad) {
//delete data from table before loading csv
con.createStatement().execute("DELETE FROM " + tableName);
}
final int batchSize = 1000;
int count = 0;
Date date = null;
while ((nextLine = csvReader.readNext()) != null) {
System.out.println("Next Line: " + Arrays.toString(nextLine));
if (null != nextLine) {
int index = 1;
for (String string : nextLine) {
date = DateUtil.convertToDate(string);
if (null != date) {
ps.setDate(index++, new java.sql.Date(date
.getTime()));
} else {
ps.setString(index++, string);
}
}
ps.addBatch();
}
if (++count % batchSize == 0) {
ps.executeBatch();
}
}
ps.executeBatch(); // insert remaining records
con.commit();
} catch (SQLException | IOException e) {
con.rollback();
e.printStackTrace();
throw new Exception(
"Error occured while loading data from file to database."
+ e.getMessage());
} finally {
if (null != ps) {
ps.close();
}
if (null != con) {
con.close();
}
csvReader.close();
}
}
public char getSeprator() {
return seprator;
}
public void setSeprator(char seprator) {
this.seprator = seprator;
}
public char getQuoteChar() {
return quoteChar;
}
public void setQuoteChar(char quoteChar) {
this.quoteChar = quoteChar;
}
}

Did you try the the following?
CSVReader reader = new CSVReader(new FileReader("yourfile.csv"), ',');
I wrote a following program and it works for me, I got the following result:
[20150801] [NULL] [33612] [00083249P PCP602] [JOE SMITH ARNP] [NULL]
[FRANK, LUCAS E] [50004655200]
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import au.com.bytecode.opencsv.CSVReader;
public class CVSTest {
/**
* #param args
*/
public static void main(String[] args) {
CSVReader reader = null;
try {
reader = new CSVReader(new FileReader(
"C:/Work/Dev/Projects/Pure_Test/Test/src/cvs"), ',');
} catch (FileNotFoundException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
String[] nextLine;
try {
while ((nextLine = reader.readNext()) != null) {
// nextLine[] is an array of values from the line
System.out.println("[" + nextLine[0] + "] [" + nextLine[1]
+ "] [" + nextLine[2] + "] [" + nextLine[3] + "] ["
+ nextLine[4] + "] [" + nextLine[5] + "] ["
+ nextLine[6] + "] [" + nextLine[7] + "]");
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}

According to the documentation, you can supply custom separator and quote characters in the constructor, which should deal with it:
CSVReader(Reader reader, char separator, char quotechar)
Construct your reader with , as separator and " as quotechar.

It is simple to load your CSV as an SQL table into HSQLDB, then select rows from the table to insert into another database. HSQLDB handles commas inside quotes. You need to define your text source as "quoted". See this:
http://hsqldb.org/doc/2.0/guide/texttables-chapt.html

Your case should be handled out of the box with no special configuration required.
If you can't make it work, then just switch to uniVocity-parsers to do this for you - it's twice as fast in comparison to OpenCSV, requires much less code and is packed with features.
CsvParserSettings settings = new CsvParserSettings(); // you have many configuration options here - check the tutorial.
CsvParser parser = new CsvParser(settings);
List<String[]> allRows = parser.parseAll(new FileReader(new File("C:/Work/Dev/Projects/Pure_Test/Test/src/cvs")));
Disclosure: I am the author of this library. It's open-source and free (Apache V2.0 license).

Update MySQL table using data from a text file through Java

I have a text file with four lines, each line contains comma separated values like below file
My file is:
Raj,raj34#myown.com,123455
kumar,kumar#myown.com,23453
shilpa,shilpa#myown.com,765468
suraj,suraj#myown.com,876567
and I have a MySQL table which contains four fields
firstname lastname email phno
---------- ---------- --------- --------
Raj babu raj34#hisown.com 2343245
kumar selva kumar#myown.com 23453
shilpa murali shilpa#myown.com 765468
suraj abd suraj#myown.com 876567
Now I want to update my table using the data in the above text file through Java.
I have tried using bufferedReader to read from the file and used split method using comma as delimiter and stored it in array. But it is not working. Any help appreciated.
This is what I have tried so far
void readingFile()
{
try
{
File f1 = new File("TestFile.txt");
FileReader fr = new FileReader(f1);
BufferedReader br = new BufferedReader(fr);
String strln = null;
strln = br.readLine();
while((strln=br.readLine())!=null)
{
// System.out.println(strln);
arr = strln.split(",");
strfirstname = arr[0];
strlastname = arr[1];
stremail = arr[2];
strphno = arr[3];
System.out.println(strfirstname + " " + strlastname + " " + stremail +" "+ strphno);
}
// for(String i : arr)
// {
// }
br.close();
fr.close();
}
catch(IOException e)
{
System.out.println("Cannot read from File." + e);
}
try
{
st = conn.createStatement();
String query = "update sampledb set email = stremail,phno =strphno where firstname = strfirstname ";
st.executeUpdate(query);
st.close();
System.out.println("sampledb Table successfully updated.");
}
catch(Exception e3)
{
System.out.println("Unable to Update sampledb table. " + e3);
}
}
and the output i got is:
Ganesh Pandiyan ganesh1#myown.com 9591982389
Dass Jeyan jeyandas#myown.com 9689523645
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
Gowtham Selvan gowthams#myown.com 9894189423
at TemporaryPackages.FileReadAndUpdateTable.readingFile(FileReadAndUpdateTable.java:35)
at TemporaryPackages.FileReadAndUpdateTable.main(FileReadAndUpdateTable.java:72)
Java Result: 1
#varadaraj:
This is the code of yours....
String stremail,strphno,strfirstname,strlastname;
// String[] arr;
Connection conn;
Statement st;
void readingFile()
{
try {
BufferedReader bReader= new BufferedReader(new FileReader("TestFile.txt"));
String fileValues;
while ((fileValues = bReader.readLine()) != null)
{
String[] values=fileValues .split(",");
strfirstname = values[0];
// strlastname = values[1];
stremail = values[1];
strphno = values[2];
System.out.println(strfirstname + " " + strlastname + " " + stremail +" "+ strphno);
}
bReader.close();
} catch (IOException e) {
System.out.println("File Read Error");
}
// for(String i : arr)
// {
// }
try
{
st = conn.createStatement();
String query = "update sampledb set email = stremail,phno =strphno where firstname = strfirstname ";
st.executeUpdate(query);
st.close();
System.out.println("sampledb Table successfully updated.");
}
catch(Exception e3)
{
System.out.println("Unable to Update sampledb table. " + e3);
}
}

What you are having looks like a CSV file, you may consider libraries like Super CSV to help you in reading and parsing the file.

you are getting ArrayIndexOutOfBoundException upon trying to access at index 1 , i.e at lastname field value, so check whether you have no data at index 1 for any of the list elements in your text file

try this
public class FileReaderTesting {
static String stremail;
static String strphno;
static String strfirstname;
static String strlastname;
static Connection conn;
static Statement st;
public static void main(String[] args) {
try {
BufferedReader bReader= new BufferedReader(new FileReader("C:\\fileName.txt"));
String fileValues;
while ((fileValues = bReader.readLine()) != null)
{
String[] values=fileValues .split(",");
strfirstname = values[0];
// strlastname = values[1];
stremail = values[1];
strphno = values[2];
System.out.println(strfirstname + " " + stremail +" "+ strphno);
st = conn.createStatement();
String query = "update sampledb set email = '"+stremail+"',pno = '"+strphno+"' where firstname = '"+strfirstname+"' ";
System.out.println(query);
st.executeUpdate(query);
st.close();
System.out.println("sampledb Table successfully updated.");
}
bReader.close();
} catch (IOException e) {
System.out.println("File Read Error");
}
catch(Exception e3)
{
System.out.println("Unable to Update sampledb table. " + e3);
}
}
}

Categories

Resources