I am trying to execute COPY INTO statement in Java code like this:
copy into s3://snowflake
from "TEST"."PUBLIC"."USER_TABLE_TEMP"
storage_integration = s3_int
file_format = CSV_TEST;
And it works fine.
Is there any way to add this file_format in Java code, so there is no need to set it up in Snowflake?
For example, SQL code of file_format that I have set in Snowflake is
ALTER FILE FORMAT "TEST"."PUBLIC".CSV_TEST SET COMPRESSION = 'NONE' FIELD_DELIMITER =
',' RECORD_DELIMITER = '\n' SKIP_HEADER = 0 FIELD_OPTIONALLY_ENCLOSED_BY = 'NONE'
TRIM_SPACE = TRUE ERROR_ON_COLUMN_COUNT_MISMATCH = FALSE ESCAPE = 'NONE'
ESCAPE_UNENCLOSED_FIELD = '\134' DATE_FORMAT = 'AUTO' TIMESTAMP_FORMAT = 'AUTO' NULL_IF = ('\\N');
Is there any way to write this as Java code?
UPDATE
Here is the code where I am using copy into statement:
String q = "COPY INTO s3://snowflake/"+ userId +" from \"EPICEROS\".\"PUBLIC\".\"USER_TABLE_TEMP\" storage_integration = s3_int file_format = CSV_TEST OVERWRITE=TRUE;";
jdbcTemplatePerBrand.get(brand).query(q, s -> {});
So how can I apply like file_format created on execution of query?
You are wanting an EXTERNAL STAGE
Which you would create like:
CREATE STAGE awesome_stange_name
URL = 's3://snowflake'
FILE_FORMAT = test.public.csv_test
and then you can copy to it:
COPY INTO #awesome_stange_name
FROM test.public.user_table_temp;
This means if the user doing the copy has permission to use the stage, then they can, without need to have access to the security tokens needed to working with that secure location.
Is there any way to write this as Java code?
In Snowflake, creating and altering file formats is done through SQL. You can simply execute a SQL statement through a JDBC connection in Java.
Just change your alter to a create if the file format is not already created:
CREATE FILE FORMAT "TEST"."PUBLIC".CSV_TEST COMPRESSION = 'NONE' FIELD_DELIMITER =
',' RECORD_DELIMITER = '\n' SKIP_HEADER = 0 FIELD_OPTIONALLY_ENCLOSED_BY = 'NONE'
TRIM_SPACE = TRUE ERROR_ON_COLUMN_COUNT_MISMATCH = FALSE ESCAPE = 'NONE'
ESCAPE_UNENCLOSED_FIELD = '\134' DATE_FORMAT = 'AUTO' TIMESTAMP_FORMAT = 'AUTO' NULL_IF = ('\\N');
Assign that to a String variable like sql and just run it like any other statement using JDBC:
ResultSet rs = stmt.executeQuery(sql);
You can then have a line rs.next(); and read from the first ordinal column or the column name status (in lowercase) to get the success/failure message.
This is the solution that I found for my question.
To be able to write file_format from code and not create one in Snowflake I did like this:
copy into s3://snowflake
from "TEST"."PUBLIC"."USER_TABLE_TEMP"
storage_integration = s3_int
OVERWRITE = TRUE
file_format = (type = csv compression = 'none' file_extension ='csv'
FIELD_OPTIONALLY_ENCLOSED_BY = '"'
NULL_IF = ()
single = true
max_file_size = 4900000000;
I also added OVERWRITE = TRUE which means that if my file exists alredy in S3, overwrite it with new one.
single = true and max_file_size = 4900000000 means that I am allowing to export files big to 5 GB. If I haven't added these two, my one big file would be separated in few smaller .csv files, which I did not want.
Related
Is there a way to dynamically specify a file name in the LOAD DATA INFILE? Can it be parameterized like for instance (syntax may be incorrect) LOAD DATA INFILE '$filename'?
A citation from MySQL documentation:
The LOAD DATA INFILE statement reads rows from a text file into a table at a very high speed. The file name must be given as a literal string.
That means that it can not be a parameter of a prepared statement. But no one forbids to make the string interpolation while the statement is just a string in your PHP code.
Unfortunately, this feature is not yet supported in MySQL and is currently listed as bug/feature request #39115 http://bugs.mysql.com/bug.php?id=39115
Or you can make a temporary copy of the file (BATCH example):
LOAD_DATA.bat
COPY %1 TempFileToLoad.csv
mysql --user=myuser --password=mypass MyDatabase < ScriptLoadMyDatabase.sql
DEL TempFileToLoad.csv
the SQL (for info) :
ScriptLoadMyDatabase.sql
load data infile 'TempFileToLoad.csv' IGNORE
into table tLoad
FIELDS TERMINATED BY ';' OPTIONALLY ENCLOSED BY '"'
lines terminated by '\r\n'
IGNORE 1 LINES
(#DateCrea, NomClient, PrenomClient, TypeMvt, #Montant, NumeroClient)
set DateCrea = str_to_date(#DateCrea, '%Y-%m-%d'), Montant = (round(#Montant / 1000)*2) ;
And finished to put a link to the BAT file in SendTo windows folder.
If you're asking if it can be used in a script; you can do some thing like this with php:
<?php
$mysqli = new mysqli("host", "user", "pwd", "db");
/* check connection */
if (mysqli_connect_errno()) {
printf("Connect failed: %s\n", mysqli_connect_error());
exit();
}
$sql = "CREATE TABLE number1 (id INT PRIMARY KEY auto_increment,data TEXT)";
if ($result = $mysqli->query($sql)) {
} else {
printf("<br>%s",$mysqli->error);
}
$host = $_SERVER['HTTP_HOST'];
$uri = rtrim(dirname($_SERVER['PHP_SELF']), '/\\');
$filename = "data.csv";
$sql = "LOAD DATA LOCAL INFILE '$host$uri$filename' INTO TABLE number1";
if ($result = $mysqli->query($sql)) {
} else {
printf("<br>%s",$mysqli->error);
}
// Close the DB connection
$mysqli->close();
exit;
%>
If the file is in the same folder as the script just use $filename a instead of $host$uri$filename. I put this together quick from a couple scripts I'm using, sorry if it doesn't work without debug, but it should be pretty close. It requires mysqli.
I'm currently working with Cucumber and Java. I would like to retrieve that path of a file from ITestResult.
I'm currently retrieving the parameters with:
Object[] test = testResult.getParameters();
However the only thing I can access would seem the be the first objects name and nothing else.
test = {Object[1]#1492}
0 = {CucumberFeatureWrapper#1493} "Links at EDM Documents View,"
cucumberFeature = {CucumberFeature#1516}
path = "test/01-automation.feature"
feature = {Feature#1518}
cucumberBackground = null
currentStepContainer = {CucumberScenario#1519}
cucumberTagStatements = {ArrayList#1520} size = 1
i18n = {I18n#1521}
currentScenarioOutline = null
I cannot see anyway of retrieving path = "test/01-automation.feature" under cucumber feature.
Have you tried something like ((CucumberFeatureWrapper)test[0]).getCucumberFeature().getPath()?
Does anybody know is it possible to write in JOOQ select with a set of predefined values? I need it for insert if not exists.
For example,
INSERT INTO test
(text)
SELECT '1234567890123456789'
WHERE
NOT EXISTS (
SELECT id FROM test WHERE text = '1234567890123456789'
);
I've found the answer by myself:
List<Param<?>> params = new LinkedList<>();
params.add(DSL.val("1234567890123456789"));
List<Field<?>> fields = new LinkedList<>();
fields.add(TEST.TEXT);
SelectConditionStep<Record1<TEXT>> notExistsSelect = context.select(TEST.TEXT).from(TEST).where(TEST.TEXT.eq("1234567890123456789"));
SelectConditionStep<Record> insertIntoSelect = context.select(params).whereNotExists(notExistsSelect);
context.insertInto(TEST, fields).select(insertIntoSelect).execute();
But it would be great, if we had an ability to do it via InsertQuery. I've not found the way to do it.
I tried to use the DataStax Java Driver, but then i found out it does not support the COPY command, does anyone know other methods of exporting data using Java? Thanks.
For example i have created event table:
cqlsh:kunderatest> describe TABLE event ;
CREATE TABLE event (
id text,
log text,
timstamp bigint,
PRIMARY KEY (id)
)
and inserted three record
cqlsh:kunderatest> INSERT INTO event (id, log , timstamp ) VALUES ( '1', 'my first log' , 12345678);
cqlsh:kunderatest> INSERT INTO event (id, log , timstamp ) VALUES ( '2', 'my second log' , 12345679);
cqlsh:kunderatest> INSERT INTO event (id, log , timstamp ) VALUES ( '3', 'my third log' , 12345680);
1) First you can do it by using CQLSH client. Now you can export the data of event table into any file (in this case it is log.txt) by executing following command.
cqlsh:kunderatest> COPY kunderatest.sample (id, name, age, address) TO './log.txt' WITH DELIMITER = '|' AND QUOTE = '''' AND ESCAPE = '''' AND NULL = '<null>';
3 rows exported in 0.042 seconds.
You can validate the command output by verify log.txt file. Hope it will help you.
2) Second you can also use Runtime utility of Java to execute the export command in order to achieve the goal.
create a file (let say command.txt) and paste the following export command into that file.
COPY kunderatest.sample (id, name, age, address) TO './log.txt' WITH DELIMITER = '|' AND QUOTE = '''' AND ESCAPE = '''' AND NULL = '<null>'
after creating the file and adding the above command into that file do the following it will export the data into file which is given in export command.
String exportCommand = cassandraHome + "bin/cqlsh " + hostname + " " + rpcPort + " -f command.txt"; // file which holds export command
Runtime runtime = Runtime.getRuntime();
Process process = runtime.exec(exoprtCommand);
// for keep tracking the log, you can do following.
InputStream is = process.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line = null;
while ((line = br.readLine()) != null)
{
}
Note: cassandrahome is path of cassandra package directory. in my case it is /usr/local/apache-cassandra-2.0.6
Spaces not changing to underscored when sent from Java-->PHP-->SQL
Java code:
String urlString = "http://www.mysite.com/auth/verifyuser.php?name="+name.toLowerCase().replace(" ","_");
PHP code:
$name = mysql_real_escape_string($_GET['name']);
$name = str_replace(' ', '_', $name);
$query = "select * from authinfo where name LIKE '$name'";
mysql_query($query);
$num = mysql_affected_rows();
if ($num > 0) {
echo '1';
} else {
echo '0';
}
when I implement a test log on the SQL database, it somehow still seems to show up with spaces instead of underscores(even though I replace it in Java and PHP) and the PHP file returns '0' rather than '1'. I've heard the issue might be whitespaces? It seems to happen to only certain users, mostly mac users.
If your php file is returning a 0, that means your query is not getting executed. Where are you establishing a connection with the database before executing the query?
Remark: where name = '$name'
mysql_affected_rows concerns INSERT, UPDATE and DELETE.
$r = mysql_query($query);
$num = mysql_num_rows($r);
It's unsafe to pass raw name into URL without encoding it.
String urlString = "http://www.example.com/auth/verifyuser.php?name=" + URLEncoder.encode(name.toLowerCase(), "UTF-8");
In PHP you can obtain data:
$name = urldecode($_GET['name']);