Im trying to parse a pipe delimited file and insert fields into a table. when i start the application nothing happens in my DB. My DB has 4 columns (account_name, command_name, and system_name, CreateDt). The file i am parsing has the date in the first row then extra data. The rows following i only need the first 3 fields in each the rest is extra data. the last row is the row count. i skipped the inserting date because for now but want to get back to it after at least able to insert the first 3 fields. I have little experience with parsing a file and storing data in a DB and have looked through jdbc examples to get to this point but im struggling and am sure there is a better way.
File Example
20200310|extra|extra|extra||
Mn1223|01192|windows|extra|extra|extra||
Sd1223|02390|linux|extra|extra|extra||
2
table format
account_name command_name system_name createDt
Mn1223 01192 windows 20200310
Sd1223 02390 linux 20200310
Code to parse and insert into DB
public List insertZygateData (List<ZygateEntity> parseData) throws Exception {
String filePath = "C:\\DEV\\Test_file.xlsx";
List<String> lines = Files.readAllLines(Paths.get(filePath));
// remove date and amount
lines.remove(0);
lines.remove(lines.size() - 1);
for (ZygateEntity zygateInfo : parseData){
new MapSqlParameterSource("account_name", zygateInfo.getAccountName())
.addValue("command_name", zygateInfo.getCommandName())
.addValue("system_name", zygateInfo.getSystemName())
.getValues();
}
return lines.stream()
.map(s -> s.split("[|]")).map(val -> new ZygateEntity(val[0],val[1],val[2])).collect(Collectors.toList());
}
public boolean cleantheTable() throws SQLException {
String sql = "INSERT INTO Landing.midrange_xygate_load (account_name,command_name,system_name)"+
"VALUES (:account_name,:command_name,:system_name)";
boolean truncated = false;
Statement stmt = null;
try {
String sqlTruncate = "truncate table Landing.midrange_xygate_load";
jdbcTemplate.execute(sqlTruncate);
truncated = true;
} catch (Exception e) {
e.printStackTrace();
truncated = false;
return truncated;
} finally {
if (stmt != null) {
jdbcTemplate.execute(sql);
stmt.close();
}
}
log.info("Clean the table return value :" + truncated);
return truncated;
}
}
Entity/Model
public ZygateEntity(String accountName, String commandName, String systemName){
this.accountName=accountName;
this.commandName=commandName;
this.systemName=systemName;
}
//getters and setters
#Override
public String toString() {
return "ZygateEntity [accountName=" + accountName + ", commandName=" + commandName + ", systemName=" + systemName + ", createDt=" + createDt +"]";
}
}
Taking a look at what you've provided, it seems you have a jumbled collection of bits of code, and while most of it is there, it's not all there and not quite all in the right order.
To get some kind of clarity, try to break down what it is you're doing into separate steps, and have a method that focuses on each step. In particular, you write
Im trying to parse a pipe delimited file and insert fields into a table
This naturally breaks down into two parts:
parsing the pipe-delimited file, and
inserting fields into a table.
For the first part, you seem to have most of the parts already in your insertZygateData method. In particular, this line reads all the lines of a file into a list:
List<String> lines = Files.readAllLines(Paths.get(filePath));
These lines then remove the first and last lines from the list of lines read:
// remove date and amount
lines.remove(0);
lines.remove(lines.size() - 1);
You then have some code that looks a bit out of place: this seems to be something to do with inserting into the database, but we haven't created our list of ZygateEntity objects as we haven't yet finished reading the file. Let's put this for loop to one side for the moment.
Finally, we take the list of lines we read, split them using pipes, create ZygateEntity objects from the parts and create a List of these objects, which we then return.
return lines.stream()
.map(s -> s.split("[|]")).map(val -> new ZygateEntity(val[0],val[1],val[2])).collect(Collectors.toList());
Putting this lot together, we have a useful method that parses the file, completing the first part of the task:
private List<ZygateEntity> parseZygateData() throws IOException {
String filePath = "C:\\DEV\\Test_file.xlsx";
List<String> lines = Files.readAllLines(Paths.get(filePath));
// remove date and amount
lines.remove(0);
lines.remove(lines.size() - 1);
return lines.stream()
.map(s -> s.split("[|]")).map(val -> new ZygateEntity(val[0],val[1],val[2])).collect(Collectors.toList());
}
(Of course, we could add a parameter for the file path to read, but in the interest of getting something working, it's OK to stick with the current hard-coded file path.)
So, we've got our list of ZygateEntity objects. How do we write a method to insert them into the database?
We can find a couple of the ingredients we need in your code sample. First, we need the SQL statement to insert the data. This is in your cleanThetable method:
String sql = "INSERT INTO Landing.midrange_xygate_load (account_name,command_name,system_name)"+
"VALUES (:account_name,:command_name,:system_name)";
We then have this loop:
for (ZygateEntity zygateInfo : parseData){
new MapSqlParameterSource("account_name", zygateInfo.getAccountName())
.addValue("command_name", zygateInfo.getCommandName())
.addValue("system_name", zygateInfo.getSystemName())
.getValues();
}
This loop creates a MapSqlParameterSource out of each ZygateEntity object, and then converts it to a Map<String, Object> by calling the getValues() method. But then it does nothing with this value. Effectively you're creating these objects and getting rid of them again without doing anything with them. This isn't ideal.
A MapSqlParameterSource is used with a Spring NamedParameterJdbcTemplate. Your code mentions a jdbcTemplate, which appears to be a field within the class that parses data and inserts into the database, but you don't show the full code of this class. I'm going to have to assume it's a NamedParameterJdbcTemplate rather than a 'plain' JdbcTemplate.
A NamedParameterJdbcTemplate contains a method update that takes a SQL string and a SqlParameterSource. We have a SQL string, and we're creating MapSqlParameterSource objects, so we can use these to carry out the insert. There's not a lot of point in creating one of these MapSqlParameterSource objects only to convert it to a map, so let's remove the call to getValues().
So, we now have a method to insert the data into the database:
public void insertZygateData(List<ZygateEntity> parseData) {
String sql = "INSERT INTO Landing.midrange_xygate_load (account_name,command_name,system_name)"+
"VALUES (:account_name,:command_name,:system_name)";
for (ZygateEntity zygateInfo : parseData){
SqlParameterSource source = new MapSqlParameterSource("account_name", zygateInfo.getAccountName())
.addValue("command_name", zygateInfo.getCommandName())
.addValue("system_name", zygateInfo.getSystemName());
jdbcTemplate.update(sql, source);
}
}
Finally, let's take a look at your cleanThetable method. As with the others, let's keep it focused on one task: it looks like at the moment you're trying to delete the data out of the table and then insert it in the same method, but let's have it just focus on deleting the data as we've now got a method to insert the data.
We can't immediately get rid of the String sql = ... line, because the finally block in your code uses it. If stmt is not null, then you attempt to run the INSERT statement and then close stmt.
However, stmt is never assigned any value other than null, so it remains null. stmt != null is therefore always false, so the INSERT statement never runs. Your finally block never does anything, so you would be best off removing it altogether. With your finally block gone, you can also get rid of your local variable stmt and the sql string, leaving us with a method whose focus is to truncate the table:
public boolean cleantheTable() throws SQLException {
boolean truncated = false;
try {
String sqlTruncate = "truncate table Landing.midrange_xygate_load";
jdbcTemplate.execute(sqlTruncate);
truncated = true;
} catch (Exception e) {
e.printStackTrace();
truncated = false;
return truncated;
}
log.info("Clean the table return value :" + truncated);
return truncated;
}
I'll leave it up to you to write the code that calls these methods. I wrote some code for this purpose, and it ran successfully and inserted into a database.
So, in summary, no data was being written to your database because you were never making a call to the database to insert any. In your insertZygateData method you were creating the parameter-source objects but not doing anything useful with them, and in your cleanThetable method, it looked like you were trying to insert data, but your line jdbcTemplate.execute(sql) that attempted to do this never ran. Even if stmt wasn't null, this line wouldn't work as you didn't pass the parameter values in anywhere: you would get an exception from the database as it would be expecting values for the parameters but you never gave it any.
Hopefully my explanation gives you a way of getting your code working and helps you understand why it wasn't.
I have to Update the Xml document object generated using Apache XMLbeans.There are two ways I am trying to update and save the document.
Step 1 : I am parsing the document and updating with the new values and saving with the parsed document itself.
private boolean updateContact(ContactType contacts, String contactFilePath, String name) throws Exception {
ContactsDocument contactDoc = ContactsDocument.Factory.parse(new File(contactFilePath));
ContactType contact = contactDoc.getContactType();
contact.setName(name);
contactDoc.save(new File(contactFilePath) , XmlUtils.getDefaultFileSavingOptions());
}
Step 2 : I am passing the updated document type and creating new instance of the xml document and saving with the updated type.
private boolean writeContact(ContactType contactType, String contactFilePath) throws Exception {
ContactsDocument contactsDoc = ContactsDocument.Factory.newInstance();
contactsDoc.setContactType(contactType);
contactsDoc.save(new File(contactFilePath), XmlUtils.getDefaultFileSavingOptions());
}
The step 2 is working but i want to know, will step 1 work ? and which is the efficient way of doing it for this scenario.
The Step 1 works perfect with the XML default file saving options and it does not modifies or removes any of the existing namespaces present in the file.
private boolean updateContact(ContactType contacts, String contactFilePath, String name) throws Exception {
ContactsDocument contactDoc = ContactsDocument.Factory.parse(new File(contactFilePath));
ContactType contact = contactDoc.getContactType();
contact.setName(name);
contactDoc.save(new File(contactFilePath) , XmlUtils.getDefaultFileSavingOptions());
}
It is also good approach to parse the file and save the changes on top of it, Instead of parse and instantiating the xml document for every updates.
I am using Couchbase Lite SDK for android and saving an object instance of MyClass as a document in the database. MyClass has an attribute that stores the date in the java.util.Date. During run time, I fetch all the instances of MyClass saved in the database and store them in the ArrayList<MyClass>. When I insert a new document into the database and read the values from the database to show all the entered instances, the date field saved in the database is retrieved as a Long when I next try to fetch the details from the database. The code I use to load the details from the database is:
Code snippet 1:
for (Field field: fields) {
field.setAccessible(true);
if (properties.containsKey(field.getName())) {
if ("date".equals(field.getName())) {
Log.d("DebugTag", properties.get(field.getName()) + "");
long dateLong = (Long) properties.get(field.getName());
details.setDate(new Date(dateLong));
} else {
field.set(details, properties.get(field.getName()));
}
} else if("_id".equals(field.getName())) {
details.set_id(document.getId());
} else {
final String msg = "Field " + field.getName() + " not present in document ";
Log.e(TAG, msg);
}
}
You can see that I have added an additional check in case the field is date. This works perfectly fine. So, I save a new entry to database and come back to the page where I see all the entries made into the database.
Now, I have implemented a new functionality to update the details of a record in the database. For updating the record I have the following implementation:
Code snippet 2:
public static boolean updateDocument(Database database, String docID, Map<String, Object> map) {
if (null == database || null == map || null == docID) {
return false;
}
boolean success = true;
Document document = database.getDocument(docID);
try {
// Have to put in the last revision id as well to update the document.
// If we do not do this, this will throw exception.
map.put("_rev", document.getProperty("_rev"));
// Putting new properties in the document ...
document.putProperties(map);
} catch (CouchbaseLiteException e) {
Log.e(TAG, "Error putting property", e);
success = false;
}
return success;
}
After doing this when I try to reload the items, it gives me exception while reading the date field in my Code snippet 1 saying that the Date object cannot be typecasted as Long and the application crashes. Now, when I again open the application, it works perfectly fine with all the changes to the edited entry reflecting correctly. Can anyone let me know the reason for this? I suspect, until we close the database connection, the changes are not committed to the actual database location and the date field in the updated entry is kept in the cache as the Date object in my case.
PS: Although, I have found a workaround for this by setting the date as a Long object in the payload (map in function updateDocument() in Code snippet 2), it would still be interesting to understand the problem I faced.
Looking at this further, this could be reasons related to auto-boxing where long to Long conversion of primitive types to the object wrapper class is crashing the application when trying to save.
Have you tried:
long value = Long.parseLong((String)...);
More specifically in Code snippet 1:
long dateLong = Long.parseLong((String)properties.get(field.getName()));
I have a Javabean with a date_observed field. Now I am trying to create a form where a user can search for entries between a start and end date.
Should I create another Javabean that extends this bean to have a start and end date field so that I can populate these field from request parameter?
I'd like to cleanly pass a bean to my Dao for SQL string generation and also have a way to do form validation if they enter incorrect date format.
Typically I would do
public void processDate_observed(HttpServletRequest request, Comment comment) {
String _date = FormUtil.getFieldValue(request, FIELD_DATE_OBSERVED);
if (!"".equals(_date) && _date != null) {
try {
Date date = DateUtil.parse(_date);
com.setDate_observed(date);
} catch (ParseException e) {
setError(FIELD_DATE_OBSERVED, e.getMessage());
}
}
}
But my Comment Javabean does not have fields for start_date and end_date
And for dao.search(comment)
public List<Evaluatee> search(Comment comment) {
SQL = ".... where date_observed > ? AND date_observed <= ?
ps.setDate(1, comment.EXTENDED FIELD???)
ps.setDate(2, comment.EXTENDED FIELD???)
...
}
What is the best practice here? Creat a whole new Javabean, extend my original bean or pass along the two date fields to Dao? But then how do you pass form validation back to form if I don't have the dates in a bean?
Is a good practice that in your DAO class you have a serch method with a start and end date as parameters.
public List<Evaluatee> search(String startDate, String endDate) {
///CODE GOES HERE
}
It's not necessary that the DAOs needs to have a Comment object as an argument, the rule is that for each table of the data base need to be a class with the same fields as in the table.
I have a field in my dataset defined as:
DataSourceDateTimeField dateField = new DataSourceDateTimeField("date");
and a StaticTextItem in DynamicForm defined as
StaticTextItem dateItem = new StaticTextItem("date", "Date");
I played a lot with different combinations of setDateFormatter, but datetime values are still rendered as something like 2011-08-23T20:00:00 (datasource receives it in json as a yyyy-MM-dd'T'HH:mm:ss.SSSZ field).
Is there some easy way to output datetime values to StaticTextItem? I assume using DynamicForm.fetchData().
UPD1. Data example.
Table row in PgSQL:
1 | "2011-09-29 12:10:05.010276+04" | "Europe"
Data sent by REST service:
{
"location":"Europe",
"id":1,
"date":"2011-09-29T08:10:05.010+0000"
}
Data fetched by SGWT from REST service (I dumped it with JSON.encode(XMLTools.selectObjects(data, "/").getJavaScriptObject()) in my implementation of transformResponse()) :
{
"location":"Europe",
"id":1,
"date":"2011-09-29T08:10:05"
}
Value as rendered in StaticTextField:
2011-09-29T08:10:05
So datetime values returned by server seem to conform the standard and also I have no warnings in Developer Console.
UPD2. May be I'm doing something wrong in my transformResponse() ?
protected void transformResponse(DSResponse response, DSRequest request, Object data) {
String json = JSON.encode(XMLTools.selectObjects(data, "/").getJavaScriptObject());
SC.logWarn("Response received");
SC.logWarn(json);
Record[] records = jsonToRecords(json);
//safe HTML text values
for (Record rec: records) {
for (DataSourceField field: getFields()) {
if (field.getType() == FieldType.TEXT) {
String textVal = rec.getAttribute(field.getName());
if (textVal != null) {
textVal = SafeHtmlUtils.htmlEscape(textVal);
}
rec.setAttribute(field.getName(), textVal);
}
}
}
response.setData(records);
response.setStartRow(0);
response.setEndRow(records.length);
response.setTotalRows(records.length);
}
/**
* Converts JS-array into SmartGWT records
*/
public static native ListGridRecord[] jsonToRecords(String jsonString) /*-{
var json = eval(jsonString);
return #com.smartgwt.client.widgets.grid.ListGrid::convertToListGridRecordArray(Lcom/google/gwt/core/client/JavaScriptObject;)(json);
}-*/;
Your problem is most likely that your date value is still a String, not an instance of Date. It should have been parsed into a Date when the JSON response was received - if it was not, there will be a warning in the SmartGWT Developer Console tell you so.
The format expected for date values in documented under DataSource.dataFormat - XML Schema date/datetime format is expected by default, and you can provide a FieldValueParser if you are unable to make your server produce this format.