Is there a way to dynamically specify a file name in the LOAD DATA INFILE? Can it be parameterized like for instance (syntax may be incorrect) LOAD DATA INFILE '$filename'?
A citation from MySQL documentation:
The LOAD DATA INFILE statement reads rows from a text file into a table at a very high speed. The file name must be given as a literal string.
That means that it can not be a parameter of a prepared statement. But no one forbids to make the string interpolation while the statement is just a string in your PHP code.
Unfortunately, this feature is not yet supported in MySQL and is currently listed as bug/feature request #39115 http://bugs.mysql.com/bug.php?id=39115
Or you can make a temporary copy of the file (BATCH example):
LOAD_DATA.bat
COPY %1 TempFileToLoad.csv
mysql --user=myuser --password=mypass MyDatabase < ScriptLoadMyDatabase.sql
DEL TempFileToLoad.csv
the SQL (for info) :
ScriptLoadMyDatabase.sql
load data infile 'TempFileToLoad.csv' IGNORE
into table tLoad
FIELDS TERMINATED BY ';' OPTIONALLY ENCLOSED BY '"'
lines terminated by '\r\n'
IGNORE 1 LINES
(#DateCrea, NomClient, PrenomClient, TypeMvt, #Montant, NumeroClient)
set DateCrea = str_to_date(#DateCrea, '%Y-%m-%d'), Montant = (round(#Montant / 1000)*2) ;
And finished to put a link to the BAT file in SendTo windows folder.
If you're asking if it can be used in a script; you can do some thing like this with php:
<?php
$mysqli = new mysqli("host", "user", "pwd", "db");
/* check connection */
if (mysqli_connect_errno()) {
printf("Connect failed: %s\n", mysqli_connect_error());
exit();
}
$sql = "CREATE TABLE number1 (id INT PRIMARY KEY auto_increment,data TEXT)";
if ($result = $mysqli->query($sql)) {
} else {
printf("<br>%s",$mysqli->error);
}
$host = $_SERVER['HTTP_HOST'];
$uri = rtrim(dirname($_SERVER['PHP_SELF']), '/\\');
$filename = "data.csv";
$sql = "LOAD DATA LOCAL INFILE '$host$uri$filename' INTO TABLE number1";
if ($result = $mysqli->query($sql)) {
} else {
printf("<br>%s",$mysqli->error);
}
// Close the DB connection
$mysqli->close();
exit;
%>
If the file is in the same folder as the script just use $filename a instead of $host$uri$filename. I put this together quick from a couple scripts I'm using, sorry if it doesn't work without debug, but it should be pretty close. It requires mysqli.
Related
I am trying to execute COPY INTO statement in Java code like this:
copy into s3://snowflake
from "TEST"."PUBLIC"."USER_TABLE_TEMP"
storage_integration = s3_int
file_format = CSV_TEST;
And it works fine.
Is there any way to add this file_format in Java code, so there is no need to set it up in Snowflake?
For example, SQL code of file_format that I have set in Snowflake is
ALTER FILE FORMAT "TEST"."PUBLIC".CSV_TEST SET COMPRESSION = 'NONE' FIELD_DELIMITER =
',' RECORD_DELIMITER = '\n' SKIP_HEADER = 0 FIELD_OPTIONALLY_ENCLOSED_BY = 'NONE'
TRIM_SPACE = TRUE ERROR_ON_COLUMN_COUNT_MISMATCH = FALSE ESCAPE = 'NONE'
ESCAPE_UNENCLOSED_FIELD = '\134' DATE_FORMAT = 'AUTO' TIMESTAMP_FORMAT = 'AUTO' NULL_IF = ('\\N');
Is there any way to write this as Java code?
UPDATE
Here is the code where I am using copy into statement:
String q = "COPY INTO s3://snowflake/"+ userId +" from \"EPICEROS\".\"PUBLIC\".\"USER_TABLE_TEMP\" storage_integration = s3_int file_format = CSV_TEST OVERWRITE=TRUE;";
jdbcTemplatePerBrand.get(brand).query(q, s -> {});
So how can I apply like file_format created on execution of query?
You are wanting an EXTERNAL STAGE
Which you would create like:
CREATE STAGE awesome_stange_name
URL = 's3://snowflake'
FILE_FORMAT = test.public.csv_test
and then you can copy to it:
COPY INTO #awesome_stange_name
FROM test.public.user_table_temp;
This means if the user doing the copy has permission to use the stage, then they can, without need to have access to the security tokens needed to working with that secure location.
Is there any way to write this as Java code?
In Snowflake, creating and altering file formats is done through SQL. You can simply execute a SQL statement through a JDBC connection in Java.
Just change your alter to a create if the file format is not already created:
CREATE FILE FORMAT "TEST"."PUBLIC".CSV_TEST COMPRESSION = 'NONE' FIELD_DELIMITER =
',' RECORD_DELIMITER = '\n' SKIP_HEADER = 0 FIELD_OPTIONALLY_ENCLOSED_BY = 'NONE'
TRIM_SPACE = TRUE ERROR_ON_COLUMN_COUNT_MISMATCH = FALSE ESCAPE = 'NONE'
ESCAPE_UNENCLOSED_FIELD = '\134' DATE_FORMAT = 'AUTO' TIMESTAMP_FORMAT = 'AUTO' NULL_IF = ('\\N');
Assign that to a String variable like sql and just run it like any other statement using JDBC:
ResultSet rs = stmt.executeQuery(sql);
You can then have a line rs.next(); and read from the first ordinal column or the column name status (in lowercase) to get the success/failure message.
This is the solution that I found for my question.
To be able to write file_format from code and not create one in Snowflake I did like this:
copy into s3://snowflake
from "TEST"."PUBLIC"."USER_TABLE_TEMP"
storage_integration = s3_int
OVERWRITE = TRUE
file_format = (type = csv compression = 'none' file_extension ='csv'
FIELD_OPTIONALLY_ENCLOSED_BY = '"'
NULL_IF = ()
single = true
max_file_size = 4900000000;
I also added OVERWRITE = TRUE which means that if my file exists alredy in S3, overwrite it with new one.
single = true and max_file_size = 4900000000 means that I am allowing to export files big to 5 GB. If I haven't added these two, my one big file would be separated in few smaller .csv files, which I did not want.
I have a login page written in php and I want to pass the query to a jar file ,the java jar file should check the query sent by php for some keywords ,can someone help me please!
thanks.
this is a part of my php code
<?php
$username=$_POST['username'];
$password=$_POST['password'];
if($username&&$password)
{
$connect=mysql_connect("localhost","root","123") or die("Couldnt connect to Database");
mysql_select_db("test") or die("Couldnt find Database");
$query = mysql_query("SELECT * FROM users WHERE username='$username'");
exec("java -jar QuerySignature.jar $query");
$numrows = mysql_num_rows($query);
if($numrows!==0)
{
while($row = mysql_fetch_assoc($query))
{
$dbusername = $row['username'];
$dbpassword=$row['password'];
}
if($username==$dbusername&&$password==$dbpassword )
{
echo "You are logged in!";
I excuted the jar file and sent the $query as an argument I want the java program to read the query it self NOT the return value
exec("java -jar QuerySignature.jar $query");
Am I doing right here?
and this is my java program
QuerySignature.java
public static void main (String a[])
{ QuerySignature q1= new QuerySignature();
String s = "select * from employee where id=20 and name='Angel' or age=20";
String QueryKey = q1.Msg_Init(s);
try {
q1.Msg_Md5(QueryKey);
} catch (Exception e) {}
I want to know, how to execute the passed argument from php instead of this line in my java program:
"select * from employee where id=20 and name='Angel' or age=20";
Sorry ,I know I am being annoying here ..but help me out and thank u.
I'm trying to read text files .txt with more than 10.000 lines per file, splitting them and inserting the data in Access database using Java and UCanAccess. The problem is that it becomes slower and slower every time (as the database gets bigger).
Now after reading 7 text files and inserting them into database, it would take the project more than 20 minutes to read another file.
I tried to do just the reading and it works fine, so the problem is the actual inserting into database.
N.B: This is my first time using UCanAccess with Java because I found that the JDBC-ODBC Bridge is no longer available. Any suggestions for an alternative solution would also be appreciated.
If your current task is simply to import a large amount of data from text files straight into the database, and it does not require any sophisticated SQL manipulations, then you might consider using the Jackcess API directly. For example, to import a CSV file you could do something like this:
String csvFileSpec = "C:/Users/Gord/Desktop/BookData.csv";
String dbFileSpec = "C:/Users/Public/JackcessTest.accdb";
String tableName = "Book";
try (Database db = new DatabaseBuilder()
.setFile(new File(dbFileSpec))
.setAutoSync(false)
.open()) {
new ImportUtil.Builder(db, tableName)
.setDelimiter(",")
.setUseExistingTable(true)
.setHeader(false)
.importFile(new File(csvFileSpec));
// this is a try-with-resources block,
// so db.close() happens automatically
}
Or, if you need to manually parse each line of input, insert a row, and retrieve the AutoNumber value for the new row, then the code would be more like this:
String dbFileSpec = "C:/Users/Public/JackcessTest.accdb";
String tableName = "Book";
try (Database db = new DatabaseBuilder()
.setFile(new File(dbFileSpec))
.setAutoSync(false)
.open()) {
// sample data (e.g., from parsing of an input line)
String title = "So, Anyway";
String author = "Cleese, John";
Table tbl = db.getTable(tableName);
Object[] rowData = tbl.addRow(Column.AUTO_NUMBER, title, author);
int newId = (int)rowData[0]; // retrieve generated AutoNumber
System.out.printf("row inserted with ID = %d%n", newId);
// this is a try-with-resources block,
// so db.close() happens automatically
}
To update an existing row based on its primary key, the code would be
Table tbl = db.getTable(tableName);
Row row = CursorBuilder.findRowByPrimaryKey(tbl, 3); // i.e., ID = 3
if (row != null) {
// Note: column names are case-sensitive
row.put("Title", "The New Title For This Book");
tbl.updateRow(row);
}
Note that for maximum speed I used .setAutoSync(false) when opening the Database, but bear in mind that disabling AutoSync does increase the chance of leaving the Access database file in a damaged (and possibly unusable) state if the application terminates abnormally while performing the updates.
Also, if you need to use slq/ucanaccess, you have to call setAutocommit(false) on the connection at the begin, and do a commit each 200/300 record. The performances will improve drammatically (about 99%).
I tried to use the DataStax Java Driver, but then i found out it does not support the COPY command, does anyone know other methods of exporting data using Java? Thanks.
For example i have created event table:
cqlsh:kunderatest> describe TABLE event ;
CREATE TABLE event (
id text,
log text,
timstamp bigint,
PRIMARY KEY (id)
)
and inserted three record
cqlsh:kunderatest> INSERT INTO event (id, log , timstamp ) VALUES ( '1', 'my first log' , 12345678);
cqlsh:kunderatest> INSERT INTO event (id, log , timstamp ) VALUES ( '2', 'my second log' , 12345679);
cqlsh:kunderatest> INSERT INTO event (id, log , timstamp ) VALUES ( '3', 'my third log' , 12345680);
1) First you can do it by using CQLSH client. Now you can export the data of event table into any file (in this case it is log.txt) by executing following command.
cqlsh:kunderatest> COPY kunderatest.sample (id, name, age, address) TO './log.txt' WITH DELIMITER = '|' AND QUOTE = '''' AND ESCAPE = '''' AND NULL = '<null>';
3 rows exported in 0.042 seconds.
You can validate the command output by verify log.txt file. Hope it will help you.
2) Second you can also use Runtime utility of Java to execute the export command in order to achieve the goal.
create a file (let say command.txt) and paste the following export command into that file.
COPY kunderatest.sample (id, name, age, address) TO './log.txt' WITH DELIMITER = '|' AND QUOTE = '''' AND ESCAPE = '''' AND NULL = '<null>'
after creating the file and adding the above command into that file do the following it will export the data into file which is given in export command.
String exportCommand = cassandraHome + "bin/cqlsh " + hostname + " " + rpcPort + " -f command.txt"; // file which holds export command
Runtime runtime = Runtime.getRuntime();
Process process = runtime.exec(exoprtCommand);
// for keep tracking the log, you can do following.
InputStream is = process.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line = null;
while ((line = br.readLine()) != null)
{
}
Note: cassandrahome is path of cassandra package directory. in my case it is /usr/local/apache-cassandra-2.0.6
Spaces not changing to underscored when sent from Java-->PHP-->SQL
Java code:
String urlString = "http://www.mysite.com/auth/verifyuser.php?name="+name.toLowerCase().replace(" ","_");
PHP code:
$name = mysql_real_escape_string($_GET['name']);
$name = str_replace(' ', '_', $name);
$query = "select * from authinfo where name LIKE '$name'";
mysql_query($query);
$num = mysql_affected_rows();
if ($num > 0) {
echo '1';
} else {
echo '0';
}
when I implement a test log on the SQL database, it somehow still seems to show up with spaces instead of underscores(even though I replace it in Java and PHP) and the PHP file returns '0' rather than '1'. I've heard the issue might be whitespaces? It seems to happen to only certain users, mostly mac users.
If your php file is returning a 0, that means your query is not getting executed. Where are you establishing a connection with the database before executing the query?
Remark: where name = '$name'
mysql_affected_rows concerns INSERT, UPDATE and DELETE.
$r = mysql_query($query);
$num = mysql_num_rows($r);
It's unsafe to pass raw name into URL without encoding it.
String urlString = "http://www.example.com/auth/verifyuser.php?name=" + URLEncoder.encode(name.toLowerCase(), "UTF-8");
In PHP you can obtain data:
$name = urldecode($_GET['name']);