Im trying to parse a pipe delimited file and insert fields into a table. when i start the application nothing happens in my DB. My DB has 4 columns (account_name, command_name, and system_name, CreateDt). The file i am parsing has the date in the first row then extra data. The rows following i only need the first 3 fields in each the rest is extra data. the last row is the row count. i skipped the inserting date because for now but want to get back to it after at least able to insert the first 3 fields. I have little experience with parsing a file and storing data in a DB and have looked through jdbc examples to get to this point but im struggling and am sure there is a better way.
File Example
20200310|extra|extra|extra||
Mn1223|01192|windows|extra|extra|extra||
Sd1223|02390|linux|extra|extra|extra||
2
table format
account_name command_name system_name createDt
Mn1223 01192 windows 20200310
Sd1223 02390 linux 20200310
Code to parse and insert into DB
public List insertZygateData (List<ZygateEntity> parseData) throws Exception {
String filePath = "C:\\DEV\\Test_file.xlsx";
List<String> lines = Files.readAllLines(Paths.get(filePath));
// remove date and amount
lines.remove(0);
lines.remove(lines.size() - 1);
for (ZygateEntity zygateInfo : parseData){
new MapSqlParameterSource("account_name", zygateInfo.getAccountName())
.addValue("command_name", zygateInfo.getCommandName())
.addValue("system_name", zygateInfo.getSystemName())
.getValues();
}
return lines.stream()
.map(s -> s.split("[|]")).map(val -> new ZygateEntity(val[0],val[1],val[2])).collect(Collectors.toList());
}
public boolean cleantheTable() throws SQLException {
String sql = "INSERT INTO Landing.midrange_xygate_load (account_name,command_name,system_name)"+
"VALUES (:account_name,:command_name,:system_name)";
boolean truncated = false;
Statement stmt = null;
try {
String sqlTruncate = "truncate table Landing.midrange_xygate_load";
jdbcTemplate.execute(sqlTruncate);
truncated = true;
} catch (Exception e) {
e.printStackTrace();
truncated = false;
return truncated;
} finally {
if (stmt != null) {
jdbcTemplate.execute(sql);
stmt.close();
}
}
log.info("Clean the table return value :" + truncated);
return truncated;
}
}
Entity/Model
public ZygateEntity(String accountName, String commandName, String systemName){
this.accountName=accountName;
this.commandName=commandName;
this.systemName=systemName;
}
//getters and setters
#Override
public String toString() {
return "ZygateEntity [accountName=" + accountName + ", commandName=" + commandName + ", systemName=" + systemName + ", createDt=" + createDt +"]";
}
}
Taking a look at what you've provided, it seems you have a jumbled collection of bits of code, and while most of it is there, it's not all there and not quite all in the right order.
To get some kind of clarity, try to break down what it is you're doing into separate steps, and have a method that focuses on each step. In particular, you write
Im trying to parse a pipe delimited file and insert fields into a table
This naturally breaks down into two parts:
parsing the pipe-delimited file, and
inserting fields into a table.
For the first part, you seem to have most of the parts already in your insertZygateData method. In particular, this line reads all the lines of a file into a list:
List<String> lines = Files.readAllLines(Paths.get(filePath));
These lines then remove the first and last lines from the list of lines read:
// remove date and amount
lines.remove(0);
lines.remove(lines.size() - 1);
You then have some code that looks a bit out of place: this seems to be something to do with inserting into the database, but we haven't created our list of ZygateEntity objects as we haven't yet finished reading the file. Let's put this for loop to one side for the moment.
Finally, we take the list of lines we read, split them using pipes, create ZygateEntity objects from the parts and create a List of these objects, which we then return.
return lines.stream()
.map(s -> s.split("[|]")).map(val -> new ZygateEntity(val[0],val[1],val[2])).collect(Collectors.toList());
Putting this lot together, we have a useful method that parses the file, completing the first part of the task:
private List<ZygateEntity> parseZygateData() throws IOException {
String filePath = "C:\\DEV\\Test_file.xlsx";
List<String> lines = Files.readAllLines(Paths.get(filePath));
// remove date and amount
lines.remove(0);
lines.remove(lines.size() - 1);
return lines.stream()
.map(s -> s.split("[|]")).map(val -> new ZygateEntity(val[0],val[1],val[2])).collect(Collectors.toList());
}
(Of course, we could add a parameter for the file path to read, but in the interest of getting something working, it's OK to stick with the current hard-coded file path.)
So, we've got our list of ZygateEntity objects. How do we write a method to insert them into the database?
We can find a couple of the ingredients we need in your code sample. First, we need the SQL statement to insert the data. This is in your cleanThetable method:
String sql = "INSERT INTO Landing.midrange_xygate_load (account_name,command_name,system_name)"+
"VALUES (:account_name,:command_name,:system_name)";
We then have this loop:
for (ZygateEntity zygateInfo : parseData){
new MapSqlParameterSource("account_name", zygateInfo.getAccountName())
.addValue("command_name", zygateInfo.getCommandName())
.addValue("system_name", zygateInfo.getSystemName())
.getValues();
}
This loop creates a MapSqlParameterSource out of each ZygateEntity object, and then converts it to a Map<String, Object> by calling the getValues() method. But then it does nothing with this value. Effectively you're creating these objects and getting rid of them again without doing anything with them. This isn't ideal.
A MapSqlParameterSource is used with a Spring NamedParameterJdbcTemplate. Your code mentions a jdbcTemplate, which appears to be a field within the class that parses data and inserts into the database, but you don't show the full code of this class. I'm going to have to assume it's a NamedParameterJdbcTemplate rather than a 'plain' JdbcTemplate.
A NamedParameterJdbcTemplate contains a method update that takes a SQL string and a SqlParameterSource. We have a SQL string, and we're creating MapSqlParameterSource objects, so we can use these to carry out the insert. There's not a lot of point in creating one of these MapSqlParameterSource objects only to convert it to a map, so let's remove the call to getValues().
So, we now have a method to insert the data into the database:
public void insertZygateData(List<ZygateEntity> parseData) {
String sql = "INSERT INTO Landing.midrange_xygate_load (account_name,command_name,system_name)"+
"VALUES (:account_name,:command_name,:system_name)";
for (ZygateEntity zygateInfo : parseData){
SqlParameterSource source = new MapSqlParameterSource("account_name", zygateInfo.getAccountName())
.addValue("command_name", zygateInfo.getCommandName())
.addValue("system_name", zygateInfo.getSystemName());
jdbcTemplate.update(sql, source);
}
}
Finally, let's take a look at your cleanThetable method. As with the others, let's keep it focused on one task: it looks like at the moment you're trying to delete the data out of the table and then insert it in the same method, but let's have it just focus on deleting the data as we've now got a method to insert the data.
We can't immediately get rid of the String sql = ... line, because the finally block in your code uses it. If stmt is not null, then you attempt to run the INSERT statement and then close stmt.
However, stmt is never assigned any value other than null, so it remains null. stmt != null is therefore always false, so the INSERT statement never runs. Your finally block never does anything, so you would be best off removing it altogether. With your finally block gone, you can also get rid of your local variable stmt and the sql string, leaving us with a method whose focus is to truncate the table:
public boolean cleantheTable() throws SQLException {
boolean truncated = false;
try {
String sqlTruncate = "truncate table Landing.midrange_xygate_load";
jdbcTemplate.execute(sqlTruncate);
truncated = true;
} catch (Exception e) {
e.printStackTrace();
truncated = false;
return truncated;
}
log.info("Clean the table return value :" + truncated);
return truncated;
}
I'll leave it up to you to write the code that calls these methods. I wrote some code for this purpose, and it ran successfully and inserted into a database.
So, in summary, no data was being written to your database because you were never making a call to the database to insert any. In your insertZygateData method you were creating the parameter-source objects but not doing anything useful with them, and in your cleanThetable method, it looked like you were trying to insert data, but your line jdbcTemplate.execute(sql) that attempted to do this never ran. Even if stmt wasn't null, this line wouldn't work as you didn't pass the parameter values in anywhere: you would get an exception from the database as it would be expecting values for the parameters but you never gave it any.
Hopefully my explanation gives you a way of getting your code working and helps you understand why it wasn't.
I have a query which selects persons from a table.
SelectConditionStep<PersonRecord> select = context
.selectFrom(Tables.PERSON)
.where(Tables.PERSON.ISDELETED.eq(false));
if(searchValue != null && searchValue.length() > 0){
select.and(Tables.PERSON.LASTNAME.likeIgnoreCase(String.format("%%%s%%", searchValue)));
}
List<PersonRecord> dbPersons = select
.orderBy(Tables.PERSON.LASTNAME, Tables.PERSON.FIRSTNAME, Tables.PERSON.ID)
.limit(length).offset(start)
.fetch();
This code works pretty well. Because I display the data in a datatables table I need to have optional / dynamic sorting capability. I did not find a solution so far.
found the solution myself now:
Collection<SortField<?>> sortFields = new ArrayList<>();
sortFields.add(Tables.PERSON.FIRSTNAME.asc());
List<PersonRecord> dbPersons = select
.orderBy(sortFields)
.limit(length).offset(start)
.fetch();
Does anybody know is it possible to write in JOOQ select with a set of predefined values? I need it for insert if not exists.
For example,
INSERT INTO test
(text)
SELECT '1234567890123456789'
WHERE
NOT EXISTS (
SELECT id FROM test WHERE text = '1234567890123456789'
);
I've found the answer by myself:
List<Param<?>> params = new LinkedList<>();
params.add(DSL.val("1234567890123456789"));
List<Field<?>> fields = new LinkedList<>();
fields.add(TEST.TEXT);
SelectConditionStep<Record1<TEXT>> notExistsSelect = context.select(TEST.TEXT).from(TEST).where(TEST.TEXT.eq("1234567890123456789"));
SelectConditionStep<Record> insertIntoSelect = context.select(params).whereNotExists(notExistsSelect);
context.insertInto(TEST, fields).select(insertIntoSelect).execute();
But it would be great, if we had an ability to do it via InsertQuery. I've not found the way to do it.
I am trying to use querydsl for building dynamic queries for dynamic schemas. I am trying to get just the query instead of having to actually execute it.
So far I have faced two issues:
- The schema.table notation is absent. Instead I only get the table name.
- I have been able to get the query but it separates out the variables and puts '?' instead which is understandable. But I am wondering if there is some way to get fully materialized query including the parameters.
Here is my current attempt and result(I am using MySQLTemplates to create the configuration):
private SQLTemplates templates = new MySQLTemplates();
private Configuration configuration = new Configuration(templates);
String table = "sometable"
Path<Object> userPath = new PathImpl<Object>(Object.class, table);
StringPath usernamePath = Expressions.stringPath(userPath, "username");
NumberPath<Long> idPath = Expressions.numberPath(Long.class, userPath, "id");
SQLQuery sqlQuery = new SQLQuery(connection, configuration)
.from(userPath).where(idPath.eq(1l)).limit(10);
String query = sqlQuery.getSQL(usernamePath).getSQL();
return query;
And what I get is:
select sometable.username
from sometable
where sometable.id = ?
limit ?
What I wanted to get was:
select sometable.username
from someschema.sometable
where sometable.id = ?
limit ?
Update: I came up with this sort of hack to get parameters materialized(Not ideal and would love better solution) But still could not get Schema.Table notation to work:
Hack follows. Please suggest cleaner QueryDsl way of doing it:
String query = cleanQuery(sqlQuery.getSQL(usernamePath));
private String cleanQuery(SQLBindings bindings){
String query = bindings.getSQL();
for (Object binding : bindings.getBindings()) {
query = query.replaceFirst("\\?", binding.toString());
}
return query;
}
To enable schema printing use the following pattern
SQLTemplates templates = MySQLTemplates.builder()
.printSchema()
.build();
SQLTemplates subclasses were used before, but since some time the builder pattern is the official way to customize the templates http://www.querydsl.com/static/querydsl/3.3.1/reference/html/ch02s03.html#d0e904
And to enable direct serialization of literals use
//configuration level
configuration.setUseLiterals(true);
//query level
configuration.setUseLiterals(true);
Here is a full example
// configuration
SQLTemplates templates = MySQLTemplates.builder()
.printSchema()
.build();
Configuration configuration = new Configuration(templates);
// querying
SQLQuery sqlQuery = new SQLQuery(connection, configuration)
.from(userPath).where(idPath.eq(1l)).limit(10);
sqlQuery.setUseLiterals(true);
String query = sqlQuery.getSQL(usernamePath).getSQL();
If you always just want the SQL query string out, move setUseLiterals from query to configuration.
Concerning the usage of Querydsl expressions the usage of code generation like documented here is advised http://www.querydsl.com/static/querydsl/3.3.1/reference/html/ch02s03.html
It will make your code typesafe, compact and readable.
If you want to try Querydsl without code generation you can replace
Path<Object> userPath = new PathImpl<Object>(Object.class, variable);
with
Path<Object> userPath = new RelationalPathBase<Object>(Object.class, variable, schema, table);
When working with QueryDSL, you must provide a template for the database platform to build the query for. I see you are already are doing this here:
private SQLTemplates templates = new MySQLTemplates();
private Configuration configuration = new Configuration(templates);
To make the schema name appear in the generated query, the only way I have found to do this is (there may be an easier way) is to extend the template class and explicitly call this.setPrintSchema(true); inside the constructor. Here is a class that should work for MySql:
import com.mysema.query.sql.MySQLTemplates;
public class NewMySqlTemplates extends MySQLTemplates {
public NewMySqlTemplates() {
super('\\', false);
}
public NewMySqlTemplates(boolean quote) {
super('\\', quote);
}
public NewMySqlTemplates(char escape, boolean quote) {
super(escape, quote);
this.setPrintSchema(true);
}
}
Then simply use this NewMySqlTemplates class in place of the MySQLTemplates class like this:
private SQLTemplates templates = new NewMySQLTemplates();
private Configuration configuration = new Configuration(templates);
I have this working using PostgresTemplates, so I may have a typo or mistake in the NewMySqlTemplates class above, but you should be able to get it to work. Good luck!
Here is my select statement
SelectQuery<Record> selectQueryPayment = transaction.selectQuery();
selectQueryPayment.addSelect(AccountPayment.ACCOUNT_PAYMENT.PAYMENT_NUMBER,AccountPayment.ACCOUNT_PAYMENT.PAYMENT_AMOUNT,AccountPayment.ACCOUNT_PAYMENT.PAYMENT_TYPE,
AccountPayment.ACCOUNT_PAYMENT.PAYMENT_DATE,AccountPayment.ACCOUNT_PAYMENT.PAYMENT_AMOUNT,AccountPayment.ACCOUNT_PAYMENT.PAYMENT_AMOUNT.subtract(AccountPayment.ACCOUNT_PAYMENT.AMOUNT_REFUNDED.add(AccountPayment.ACCOUNT_PAYMENT.AMOUNT_APPLIED)));
Here you can see a complex select with some calculation
ACCOUNT_PAYMENT.PAYMENT_AMOUNT.subtract(AccountPayment.ACCOUNT_PAYMENT.AMOUNT_REFUNDED.add(AccountPayment.ACCOUNT_PAYMENT.AMOUNT_APPLIED))
How to create Alias for this? And then get back data from it?
Ok I got the solution we can use this
AccountPayment.ACCOUNT_PAYMENT.PAYMENT_AMOUNT.subtract(AccountPayment.ACCOUNT_PAYMENT.AMOUNT_REFUNDED.add(AccountPayment.ACCOUNT_PAYMENT.AMOUNT_APPLIED)).as("OverPayment")
We have to add as("Alias Name") and getting value back we have to use
Result<Record> resultPayment = selectQueryPayment.fetch();
for(Record record : resultPayment){
feeAmount = resultPayment.getValues("OverPayment");
}