I tried to use the DataStax Java Driver, but then i found out it does not support the COPY command, does anyone know other methods of exporting data using Java? Thanks.
For example i have created event table:
cqlsh:kunderatest> describe TABLE event ;
CREATE TABLE event (
id text,
log text,
timstamp bigint,
PRIMARY KEY (id)
)
and inserted three record
cqlsh:kunderatest> INSERT INTO event (id, log , timstamp ) VALUES ( '1', 'my first log' , 12345678);
cqlsh:kunderatest> INSERT INTO event (id, log , timstamp ) VALUES ( '2', 'my second log' , 12345679);
cqlsh:kunderatest> INSERT INTO event (id, log , timstamp ) VALUES ( '3', 'my third log' , 12345680);
1) First you can do it by using CQLSH client. Now you can export the data of event table into any file (in this case it is log.txt) by executing following command.
cqlsh:kunderatest> COPY kunderatest.sample (id, name, age, address) TO './log.txt' WITH DELIMITER = '|' AND QUOTE = '''' AND ESCAPE = '''' AND NULL = '<null>';
3 rows exported in 0.042 seconds.
You can validate the command output by verify log.txt file. Hope it will help you.
2) Second you can also use Runtime utility of Java to execute the export command in order to achieve the goal.
create a file (let say command.txt) and paste the following export command into that file.
COPY kunderatest.sample (id, name, age, address) TO './log.txt' WITH DELIMITER = '|' AND QUOTE = '''' AND ESCAPE = '''' AND NULL = '<null>'
after creating the file and adding the above command into that file do the following it will export the data into file which is given in export command.
String exportCommand = cassandraHome + "bin/cqlsh " + hostname + " " + rpcPort + " -f command.txt"; // file which holds export command
Runtime runtime = Runtime.getRuntime();
Process process = runtime.exec(exoprtCommand);
// for keep tracking the log, you can do following.
InputStream is = process.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line = null;
while ((line = br.readLine()) != null)
{
}
Note: cassandrahome is path of cassandra package directory. in my case it is /usr/local/apache-cassandra-2.0.6
Related
I am trying to execute COPY INTO statement in Java code like this:
copy into s3://snowflake
from "TEST"."PUBLIC"."USER_TABLE_TEMP"
storage_integration = s3_int
file_format = CSV_TEST;
And it works fine.
Is there any way to add this file_format in Java code, so there is no need to set it up in Snowflake?
For example, SQL code of file_format that I have set in Snowflake is
ALTER FILE FORMAT "TEST"."PUBLIC".CSV_TEST SET COMPRESSION = 'NONE' FIELD_DELIMITER =
',' RECORD_DELIMITER = '\n' SKIP_HEADER = 0 FIELD_OPTIONALLY_ENCLOSED_BY = 'NONE'
TRIM_SPACE = TRUE ERROR_ON_COLUMN_COUNT_MISMATCH = FALSE ESCAPE = 'NONE'
ESCAPE_UNENCLOSED_FIELD = '\134' DATE_FORMAT = 'AUTO' TIMESTAMP_FORMAT = 'AUTO' NULL_IF = ('\\N');
Is there any way to write this as Java code?
UPDATE
Here is the code where I am using copy into statement:
String q = "COPY INTO s3://snowflake/"+ userId +" from \"EPICEROS\".\"PUBLIC\".\"USER_TABLE_TEMP\" storage_integration = s3_int file_format = CSV_TEST OVERWRITE=TRUE;";
jdbcTemplatePerBrand.get(brand).query(q, s -> {});
So how can I apply like file_format created on execution of query?
You are wanting an EXTERNAL STAGE
Which you would create like:
CREATE STAGE awesome_stange_name
URL = 's3://snowflake'
FILE_FORMAT = test.public.csv_test
and then you can copy to it:
COPY INTO #awesome_stange_name
FROM test.public.user_table_temp;
This means if the user doing the copy has permission to use the stage, then they can, without need to have access to the security tokens needed to working with that secure location.
Is there any way to write this as Java code?
In Snowflake, creating and altering file formats is done through SQL. You can simply execute a SQL statement through a JDBC connection in Java.
Just change your alter to a create if the file format is not already created:
CREATE FILE FORMAT "TEST"."PUBLIC".CSV_TEST COMPRESSION = 'NONE' FIELD_DELIMITER =
',' RECORD_DELIMITER = '\n' SKIP_HEADER = 0 FIELD_OPTIONALLY_ENCLOSED_BY = 'NONE'
TRIM_SPACE = TRUE ERROR_ON_COLUMN_COUNT_MISMATCH = FALSE ESCAPE = 'NONE'
ESCAPE_UNENCLOSED_FIELD = '\134' DATE_FORMAT = 'AUTO' TIMESTAMP_FORMAT = 'AUTO' NULL_IF = ('\\N');
Assign that to a String variable like sql and just run it like any other statement using JDBC:
ResultSet rs = stmt.executeQuery(sql);
You can then have a line rs.next(); and read from the first ordinal column or the column name status (in lowercase) to get the success/failure message.
This is the solution that I found for my question.
To be able to write file_format from code and not create one in Snowflake I did like this:
copy into s3://snowflake
from "TEST"."PUBLIC"."USER_TABLE_TEMP"
storage_integration = s3_int
OVERWRITE = TRUE
file_format = (type = csv compression = 'none' file_extension ='csv'
FIELD_OPTIONALLY_ENCLOSED_BY = '"'
NULL_IF = ()
single = true
max_file_size = 4900000000;
I also added OVERWRITE = TRUE which means that if my file exists alredy in S3, overwrite it with new one.
single = true and max_file_size = 4900000000 means that I am allowing to export files big to 5 GB. If I haven't added these two, my one big file would be separated in few smaller .csv files, which I did not want.
I have a Script order.pl it has 3 variables
$dbcount=$ARGV[0];
if($dbcount == ""){$dbcount = 196001;}
$Num_Batches =$ARGV[1];
if($Num_Batches == ""){$Num_Batches=1;}
print "batches:$Num_Batches\t";
$TimeStamp = $ARGV[2];
if($TimeStamp == ""){$TimeStamp = "";}
$DBFetch = 'java GetWOConfHold_Auto '. $dbcount." ".$TimeStamp ;
print "DBFetch:$DBFetch\n";
print "timestamp :$TimeStamp";
system($DBFetch);
Here the java file is GetWOConfHold_Auto.java and i want to sent both dbcount and empty String to the GetWoConfHol_Auto java file as a command line arguments but
while running it is showing
perl order.pl 196000 1
batches:1 timestamp: dbcount:196000 DBFetch:java GetWOConfHold_Auto 196000
timestamp :args[0] 19600
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
at GetWOConfHold_Auto.main(GetWOConfHold_Auto.java:18)
no of picking updates data : 0
No data.exiting
I tried to print the timestamp but it is showing
timestamp :args[0] 19600 not the empty String
and in java it is stopping at the main method signature
I want to sent the empty String to java file as a command line argument. and i check if timestamp is null just go and pick the value from the properties file
In the string variant of system, you need to quote the empty string.
Have you tried the list version of system?
my $DBFetch = system 'java', 'GetWOConfHold_Auto' ,$dbcount, $TimeStamp;
Is there a way to dynamically specify a file name in the LOAD DATA INFILE? Can it be parameterized like for instance (syntax may be incorrect) LOAD DATA INFILE '$filename'?
A citation from MySQL documentation:
The LOAD DATA INFILE statement reads rows from a text file into a table at a very high speed. The file name must be given as a literal string.
That means that it can not be a parameter of a prepared statement. But no one forbids to make the string interpolation while the statement is just a string in your PHP code.
Unfortunately, this feature is not yet supported in MySQL and is currently listed as bug/feature request #39115 http://bugs.mysql.com/bug.php?id=39115
Or you can make a temporary copy of the file (BATCH example):
LOAD_DATA.bat
COPY %1 TempFileToLoad.csv
mysql --user=myuser --password=mypass MyDatabase < ScriptLoadMyDatabase.sql
DEL TempFileToLoad.csv
the SQL (for info) :
ScriptLoadMyDatabase.sql
load data infile 'TempFileToLoad.csv' IGNORE
into table tLoad
FIELDS TERMINATED BY ';' OPTIONALLY ENCLOSED BY '"'
lines terminated by '\r\n'
IGNORE 1 LINES
(#DateCrea, NomClient, PrenomClient, TypeMvt, #Montant, NumeroClient)
set DateCrea = str_to_date(#DateCrea, '%Y-%m-%d'), Montant = (round(#Montant / 1000)*2) ;
And finished to put a link to the BAT file in SendTo windows folder.
If you're asking if it can be used in a script; you can do some thing like this with php:
<?php
$mysqli = new mysqli("host", "user", "pwd", "db");
/* check connection */
if (mysqli_connect_errno()) {
printf("Connect failed: %s\n", mysqli_connect_error());
exit();
}
$sql = "CREATE TABLE number1 (id INT PRIMARY KEY auto_increment,data TEXT)";
if ($result = $mysqli->query($sql)) {
} else {
printf("<br>%s",$mysqli->error);
}
$host = $_SERVER['HTTP_HOST'];
$uri = rtrim(dirname($_SERVER['PHP_SELF']), '/\\');
$filename = "data.csv";
$sql = "LOAD DATA LOCAL INFILE '$host$uri$filename' INTO TABLE number1";
if ($result = $mysqli->query($sql)) {
} else {
printf("<br>%s",$mysqli->error);
}
// Close the DB connection
$mysqli->close();
exit;
%>
If the file is in the same folder as the script just use $filename a instead of $host$uri$filename. I put this together quick from a couple scripts I'm using, sorry if it doesn't work without debug, but it should be pretty close. It requires mysqli.
I want to fetch the app category from play store through its unique identifier i.e. package name, I am using the following code but does not return any data. I also tried to use this AppsRequest.newBuilder().setAppId(query) still no help.
Thanks.
String AndroidId = "dead000beef";
MarketSession session = new MarketSession();
session.login("email", "passwd");
session.getContext().setAndroidId(AndroidId);
String query = "package:com.king.candycrushsaga";
AppsRequest appsRequest = AppsRequest.newBuilder().setQuery(query).setStartIndex(0)
.setEntriesCount(10).setWithExtendedInfo(true).build();
session.append(appsRequest, new Callback<AppsResponse>() {
#Override
public void onResult(ResponseContext context, AppsResponse response) {
String response1 = response.toString();
Log.e("reponse", response1);
}
});
session.flush();
Use this script:
######## Fetch App names and genre of apps from playstore url, using pakage names #############
"""
Reuirements for running this script:
1. requests library
Note: Run this command to avoid insecureplatform warning pip install --upgrade ndg-httpsclient
2. bs4
pip install requests
pip install bs4
"""
import requests
import csv
from bs4 import BeautifulSoup
# url to be used for package
APP_LINK = "https://play.google.com/store/apps/details?id="
output_list = []; input_list = []
# get input file path
print "Need input CSV file (absolute) path \nEnsure csv is of format: <package_name>, <id>\n\nEnter Path:"
input_file_path = str(raw_input())
# store package names and ids in list of tuples
with open(input_file_path, 'rb') as csvfile:
for line in csvfile.readlines():
(p, i) = line.strip().split(',')
input_list.append((p, i))
print "\n\nSit back and relax, this might take a while!\n\n"
for package in input_list:
# generate url, get html
url = APP_LINK + package[0]
r = requests.get(url)
if not (r.status_code==404):
data = r.text
soup = BeautifulSoup(data, 'html.parser')
# parse result
x = ""; y = "";
try:
x = soup.find('div', {'class': 'id-app-title'})
x = x.text
except:
print "Package name not found for: %s" %package[0]
try:
y = soup.find('span', {'itemprop': 'genre'})
y = y.text
except:
print "ID not found for: %s" %package[0]
output_list.append([x,y])
else:
print "App not found: %s" %package[0]
# write to csv file
with open('results.csv', 'w') as fp:
a = csv.writer(fp, delimiter=",")
a.writerows(output_list)
This is what i did, best and easy solution
https://androidquery.appspot.com/api/market?app=your.unique.package.name
Or otherwise you can get source html and get the string out of it ...
https://play.google.com/store/apps/details?id=your.unique.package.name
Get this string out of it - use split or substring methods
<span itemprop="genre">Sports</span>
In this case sports is your category
use android-market-api it will gives all information of application
I'm trying to call a bash script within a oracle database through a java script. To test it I just tried a basic script :
#!/bin/bash
echo "It works !"
And the java script that I use is :
import java.lang.*;
import java.io.*;
public class UAM_TOOLS{
public static String Toto () throws IOException {
String[] unixCommand = {"/home/oz380/toto.sh"};
String pwd;
Process p = Runtime.getRuntime().exec(unixCommand);
BufferedReader input =
new BufferedReader
(new InputStreamReader(p.getInputStream()));
pwd = input.readLine();
input.close();
return pwd;
}
};
I granted all the permissions that had to be granted and I created the function in my database :
SQL> CREATE OR REPLACE FUNCTION TOPI RETURN VARCHAR2
2 as language java
3 name 'UAM_TOOLS.Toto() return java.lang.String';
4 /
But then when I call the function :
select TOPI from dual;
or :
SQL> set serveroutput on;
SQL> DECLARE
2 G VARCHAR2(50);
3 BEGIN
4 G := UAM.TOPI;
5 DBMS_OUTPUT.PUT_LINE(G);
6 END;
7 /
It doesn't work and prints the error :
ORA-29541: class UAM.UAM_TOOLS could not be resolved
I don't really understand what the problem can be. If anyone does I would be really thankful.
Before the
CREATE FUNCTION
step you need to compile your class at command line
$>javac UAM_TOOLS.java
or using an IDE sth like Eclipse
that will generate compiled class with .class extension. For your case it will be UAM_TOOLS.class
And you still need to upload it to database on command line where the host which db runs on it
$>loadjava -user yourUserName/youPass#Yourdb UAM_TOOLS.class
after that 2 step you can resume with create function step.