Hi all after getting some advice, I am attempting to use the filewriter method in order to export my google analytics queries that i got to a CSV file format here is what i have so far
private static void printGaData(GaData results) {
try {
PrintWriter pw = new PrintWriter(BufferedWriter(new FileWriter("data.csv")));
}
catch(Exception e) {
e.printStackTrace();
}
System.out.println(
"printing results for profile: " + results.getProfileInfo().getProfileName());
if (results.getRows() == null || results.getRows().isEmpty()) {
System.out.println("No results Found.");
} else {
// Print column headers.
for (ColumnHeaders header : results.getColumnHeaders()) {
System.out.printf(header.getName() + ", ");
}
System.out.println();
// Print actual data.
for (List<String> row : results.getRows()) {
for (String column : row) {
pw.printf(row + ", ");
}
pw.println();
}
pw.println();
}
}
}
doesnt output any data and keeps saying that the pw is non extent and stuff like that
Your PrintWriter is inside the try catch block. If you define it outside like
PrintWriter pw = null;
try {
pw = new PrintWriter(BufferedWriter(new FileWriter("data.csv")));
} catch (Exception e) {
// handle exception
}
then it will be available to the rest of your code.
Related
I am trying to write a method which recursively gathers data from files, and writes erroneous data to an error file. See code block:
public static LinkedQueue<Stock> getStockData(LinkedQueue<Stock> stockQueue, String startPath) throws Exception {
File dir = new File(getValidDirectory(startPath));
try (PrintStream recordErrors = new PrintStream(new File("EODdataERRORS.txt"))) {
for (File name : dir.listFiles()) {
if (name.isDirectory()) {
getStockData(stockQueue, name.getPath());
}
else if (name.canRead()) {
Scanner readFile = new Scanner(name);
readFile.nextLine();
while (readFile.hasNext()) {
String line = readFile.nextLine();
String[] lineArray = line.split(",+");
if (lineArray.length == 8) {
try {
Stock stock = new Stock(name.getName().replaceAll("_+(.*)", ""));
stock.fromRecord(lineArray);
stockQueue.enqueue(stock);
}
catch (Exception ex) {
recordErrors.println(line + " ERROR: " + ex.getMessage());
System.err.println(line + " ERROR: " + ex.getMessage());
}
}
else {
recordErrors.println(line + " ERROR: Invalid record length.");
System.err.println(line + " ERROR: Invalid record length.");
}
}
}
}
}
catch (FileNotFoundException ex) {
System.err.println("FileNotFoundException. Please ensure the directory is configured properly.");
}
return stockQueue;
}
However, the error file is always blank.
I've tried calling the .flush() and .close() methods. System.err is outputting so I know the code is being run. I've tried instantiating the PrintStream outside of the try-with-resources, no change.
I've tried calling the method at earlier points in the code (i.e. right after instantiation of the printStream, and in the if{} block) and it does output to the error file. It's only within the catch{} and else{} blocks (where I actually need it to work) that it refuses to print anything. I've also tried storing the error data and using a loop after the blocks to print the data and it still won't work. See code block:
public static LinkedQueue<Stock> getStockData(LinkedQueue<Stock> stockQueue, String startPath) throws Exception {
File dir = new File(getValidDirectory(startPath));
LinkedQueue errors = new LinkedQueue();
try (PrintStream recordErrors = new PrintStream(new File("EODdataERRORS.txt"))) {
for (File name : dir.listFiles()) {
if (name.isDirectory()) {
getStockData(stockQueue, name.getPath());
}
else if (name.canRead()) {
Scanner readFile = new Scanner(name);
readFile.nextLine();
while (readFile.hasNext()) {
String line = readFile.nextLine();
String[] lineArray = line.split(",+");
if (lineArray.length == 8) {
try {
Stock stock = new Stock(name.getName().replaceAll("_+(.*)", ""));
stock.fromRecord(lineArray);
stockQueue.enqueue(stock);
}
catch (Exception ex) {
errors.enqueue(line + " ERROR: " + ex.getMessage());
System.err.println(line + " ERROR: " + ex.getMessage());
}
}
else {
errors.enqueue(line + " ERROR: Invalid record length.");
System.err.println(line + " ERROR: Invalid record length.");
}
}
}
}
while (!errors.isEmpty()) {
recordErrors.println(errors.dequeue());
}
}
catch (FileNotFoundException ex) {
System.err.println("FileNotFoundException. Please ensure the directory is configured properly.");
}
return stockQueue;
}
Edit
Code has been edited to show instantiation of the PrintStream only once. The error persists. I am sorry there is no Repex, I cannot recreate this error except for in this specific case.
I have solved the issue. I'm not really sure what the issue was, but it appears to have something to do with the PrintStream being instantiated in a method other than the main{} method. This would also explain why I was unable to recreate this error, try as I might.
As such, I've solved it by simply storing the errors in a list and printing them in the main{} method.
public static void getStockData(LinkedQueue<Stock> stockQueue, LinkedQueue<String> errorQueue, String startPath) {
File dir = new File(getValidDirectory(startPath));
try {
for (File name : dir.listFiles()) {
if (name.isDirectory()) {
getStockData(stockQueue, errorQueue, name.getPath());
}
else if (name.canRead()) {
Scanner readFile = new Scanner(name);
readFile.nextLine();
while (readFile.hasNext()) {
String line = readFile.nextLine();
String[] lineArray = line.split(",+");
if (lineArray.length == 8) {
try {
Stock stock = new Stock(name.getName().replaceAll("_+(.*)", ""));
stock.fromRecord(lineArray);
stockQueue.enqueue(stock);
}
catch (Exception ex) {
errorQueue.enqueue(line + "; ERROR: " + ex.getMessage());
System.err.println(line + "; ERROR: " + ex.getMessage());
}
}
else {
errorQueue.enqueue(line + "; ERROR: Invalid record length.");
System.err.println(line + "; ERROR: Invalid record length.");
}
}
}
}
}
catch (FileNotFoundException ex) {
System.err.println("FileNotFoundException. Please ensure the directory is configured properly.");
}
}
This has the downside of taking up more memory, but I see no other way to get this to work the way I want it to. Thanks for the help!
I have a problem on my code; basically I have an array containing some key:
String[] ComputerScience = { "A", "B", "C", "D" };
And so on, containing 40 entries.
My code reads 900 pdf from 40 folder corresponding to each element of ComputerScience, manipulates the extracted text and stores the output in a file named A.txt , B.txt, ecc ...
Each folder "A", "B", ecc contains 900 pdf.
After a lot of documents, an exception "Too many open files" is thrown.
I'm supposing that I am correctly closing files handler.
static boolean writeOccurencesFile(String WORDLIST,String categoria, TreeMap<String,Integer> map) {
File dizionario = new File(WORDLIST);
FileReader fileReader = null;
FileWriter fileWriter = null;
try {
File cat_out = new File("files/" + categoria + ".txt");
fileWriter = new FileWriter(cat_out, true);
} catch (IOException e) {
e.printStackTrace();
}
try {
fileReader = new FileReader(dizionario);
} catch (FileNotFoundException e) { }
try {
BufferedReader bufferedReader = new BufferedReader(fileReader);
if (dizionario.exists()) {
StringBuffer stringBuffer = new StringBuffer();
String parola;
StringBuffer line = new StringBuffer();
int contatore_index_parola = 1;
while ((parola = bufferedReader.readLine()) != null) {
if (map.containsKey(parola) && !parola.isEmpty()) {
line.append(contatore_index_parola + ":" + map.get(parola).intValue() + " ");
map.remove(parola);
}
contatore_index_parola++;
}
if (! line.toString().isEmpty()) {
fileWriter.append(getCategoryID(categoria) + " " + line + "\n"); // print riga completa documento N x1:y x2:a ...
}
} else { System.err.println("Dictionary file not found."); }
bufferedReader.close();
fileReader.close();
fileWriter.close();
} catch (IOException e) { return false;}
catch (NullPointerException ex ) { return false;}
finally {
try {
fileReader.close();
fileWriter.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return true;
}
But the error still comes. ( it is thrown at:)
try {
File cat_out = new File("files/" + categoria + ".txt");
fileWriter = new FileWriter(cat_out, true);
} catch (IOException e) {
e.printStackTrace();
}
Thank you.
EDIT: SOLVED
I found the solution, there was, in the main function in which writeOccurencesFile is called, another function that create a RandomAccessFile and doesn't close it.
The debugger sais that Exception has thrown in writeOccurencesFile but using Java Leak Detector i found out that the pdf were already opened and not close after parsing to pure text.
Thank you!
Try using this utility specifically designed for the purpose.
This Java agent is a utility that keeps track of where/when/who opened files in your JVM. You can have the agent trace these operations to find out about the access pattern or handle leaks, and dump the list of currently open files and where/when/who opened them.
When the exception occurs, this agent will dump the list, allowing you to find out where a large number of file descriptors are in use.
i have tried using try-with resources; but the problem remains.
Also running in system macos built-in console print out a FileNotFound exception at the line of FileWriter fileWriter = ...
static boolean writeOccurencesFile(String WORDLIST,String categoria, TreeMap<String,Integer> map) {
File dizionario = new File(WORDLIST);
try (FileWriter fileWriter = new FileWriter( "files/" + categoria + ".txt" , true)) {
try (FileReader fileReader = new FileReader(dizionario)) {
try (BufferedReader bufferedReader = new BufferedReader(fileReader)) {
if (dizionario.exists()) {
StringBuffer stringBuffer = new StringBuffer();
String parola;
StringBuffer line = new StringBuffer();
int contatore_index_parola = 1;
while ((parola = bufferedReader.readLine()) != null) {
if (map.containsKey(parola) && !parola.isEmpty()) {
line.append(contatore_index_parola + ":" + map.get(parola).intValue() + " ");
map.remove(parola);
}
contatore_index_parola++;
}
if (!line.toString().isEmpty()) {
fileWriter.append(getCategoryID(categoria) + " " + line + "\n"); // print riga completa documento N x1:y x2:a ...
}
} else {
System.err.println("Dictionary file not found.");
}
}
}
} catch (IOException e) {
e.printStackTrace();
}
return true;
}
This is the code that i am using now, although the bad managing of Exception, why the files seem to be not closed?
Now i am making a test with File Leak Detector
Maybe your code raises another exception that you are not handling. Try add catch (Exception e) before finally block
You also can move BufferedReader declaration out the try and close it in finally
I am writing a method that takes in a List of Twitter Status objects as a parameter, opens a log file containing String represenatations of Tweets, checks if any of the String representations of the Status objects are already written to the file - if so, it removes them from the list, if not it appends the Status to the file.
Everything is working up until I attempt to write to the file. Nothing is being written at all. I am led to believe that it is due to the method having the file open in two different places: new File("tweets.txt") and new FileWriter("tweets.txt, true).
Here is my method:
private List<Status> removeDuplicates(List<Status> mentions) {
File mentionsFile = new File("tweets.txt");
try {
mentionsFile.createNewFile();
} catch (IOException e1) {
// Print error + stacktrace
}
List<String> fileLines = new ArrayList<>();
try {
Scanner scanner = new Scanner(mentionsFile);
while (scanner.hasNextLine()) {
fileLines.add(scanner.nextLine());
}
scanner.close();
} catch (FileNotFoundException e) {
// Print error + stacktrace
}
List<Status> duplicates = new ArrayList<>();
for (Status mention : mentions) {
String mentionString = "#" + mention.getUser().getScreenName() + " \"" + mention.getText() + "\" (" + mention.getCreatedAt() + "\")";
if (fileLines.contains(mentionString)) {
duplicates.add(mention);
} else {
try {
Writer writer = new BufferedWriter(new FileWriter("tweets.txt", true));
writer.write(mentionString);
} catch (IOException e) {
// Print error + stacktrace
}
}
}
mentions.removeAll(duplicates);
return mentions;
}
I wrote here few thoughts looking your code.
Remember to always close the object Reader and Writer.
Have a look at try-with-resources statement :
try (Writer writer = new BufferedWriter(new FileWriter("tweets.txt", true))) {
writer.write(mentionString);
} catch (IOException e) {
// Print error + stacktrace
}
To read an entire file in a List<String>:
List<String> lines = Files.readAllLines(Paths.get("tweets.txt"), StandardCharsets.UTF_8);
And again, I think it's a bad practice write in the same file you're reading of.
I would suggest to write in a different file if you don't have a particular constraint.
But if you really want have this behavior there are few alternative.
Create a temporary file as output and, when you process is successfully completed, than move it to the old one using Files.move(from, to).
So I'm trying to import a CSV file into my MySQL database through my Java program. The program imports everything that's in the file, like it's suppose to, but the first row, it send to the end of the table, and the program see it's there, but if I search for that nr, it says it doesn't exists. And if I go directly to the database table and edit the nr(if the nr is 137, and I edit and write 137 again) the program recognize that nr, and if I search for it, it will find, and the database table organizes itself and sends that entry where is suppose to be.
I just don't see any logic in this. I someone could help me out, I'd appreciated.
LOAD DATA INFILE 'C:\\Users\\carla.DESKTOP-9364K9K\\Desktop\\Alunos_1.csv'
INTO TABLE utentes character set utf8
FIELDS TERMINATED BY ','
(NrProcesso, Nome, #Nome_Resumido, Ano, Turma, #Subsidio, #Nome_EE, #NIF, #email, #Obs)
SET
Subsidio = IF(#Subsidio='','Nenhum',#Subsidio),
Nome_Resumido = IF(#Nome_Resumido='',NULL,#Nome_Resumido),
Nome_EE = IF(#Nome_EE='',NULL,#Nome_EE),
NIF = IF(#NIF = '', NULL,#NIF),
email = IF(#email='',NULL,#email),
Obs = IF(#Obs='',NULL,#Obs);
Thanks in advance.
You have do do something to check cell/column value and form a sql to inject in MySQL.
public List<Object> getRecordingsListFromCsv(String csvFileLocation, String mp3FileLocation, String mp3FileLocation2, String saveFileLocation, ChannelSftp sftp) {
Map<String, File> recordingsFilesMap = null;
BufferedReader br = null;
List<String> errorFilesList = new ArrayList<>();
List<Object> tempList = new LinkedList<>();
try {
csvRows = 0;
recordingsFilesMap = new LinkedHashMap<>();
br = new BufferedReader(new FileReader(csvFileLocation));
String line = br.readLine();
scriptLog.info("\n" + csvFileLocation + " loaded. Parsing File...");
while ((line = br.readLine()) != null) {
String[] csvArray = parseCsvLineToArray(line);
// System.out.println(Arrays.asList(csvArray) + "\n\n");
if (csvArray[0].trim().isEmpty()) {
continue;
}
/* Do your stuff here */
csvRows++;
}
} catch (FileNotFoundException e) {
scriptLog.error("\n---ERROR---\n FILE NOT FOUND: " + csvFileLocation);
String errorStr = "Type=" + e.toString();
errorStr += "StackTrace=" + Arrays.toString(e.getStackTrace());
scriptLog.error(errorStr);
} catch (IOException e) {
String errorStr = "Type=" + e.toString();
errorStr += "StackTrace=" + Arrays.toString(e.getStackTrace());
scriptLog.error(errorStr);
} finally {
if (br != null) {
try {
br.close();
} catch (IOException e) {
System.out.println(e.toString());
}
}
}
Hope it will help you at some extent!!
Hi all I am in the process of trying to export data that i queried from google analytics api to a csv file format using JAVA. I am very new to java and have looked at using things like supercsv and some other csv converting programs. However I am looking at the code and feel like you can just simply output the data to a csv format. If anyone has suggestions it would be awesome!
private static GaData executeDataQuery(Analytics analytics, String profileId) throws IOException {
return analytics.data().ga().get("ga:" + profileId, // Table Id. ga: + profile id.
"today", // Start date.
"today", // End date.
"ga:pageviews, ga:visits, ga:uniquePageviews") // Metrics.
.setDimensions("")
.setSort("-ga:visits")
.setFilters("ga:medium==organic")
.setMaxResults(25)
.execute();
}
That is my query
private static void printGaData(GaData results) {
System.out.println(
"printing results for profile: " + results.getProfileInfo().getProfileName());
if (results.getRows() == null || results.getRows().isEmpty()) {
System.out.println("No results Found.");
} else {
// Print column headers.
for (ColumnHeaders header : results.getColumnHeaders()) {
System.out.printf("%30s", header.getName());
}
System.out.println();
// Print actual data.
for (List<String> row : results.getRows()) {
for (String column : row) {
System.out.printf("%30s", column);
}
System.out.println();
}
System.out.println();
}
}
}
This is the part that i think i need to modify in order to output it to csv.
Thanks all
ok so i have changed it to use buffered writer for the CSV conversion so far i have it so..
public static void main(String[] args) throws IOException{
BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
BufferedWriter out = new BufferedWriter(new FileWriter("c://data.csv"));
try {
String inputLine = null;
do {
inputLine=in.readLine();
out.write(inputLine);
out.newLine();
} while (!inputLine.equalsIgnoreCase("eof"));
System.out.print("Write Successful");
} catch(IOException e1) {
System.out.println("Error during reading/writing");
} finally {
out.close();
in.close();
}
}
for the first part of the writer...
private static void printGaData(GaData results) {
System.out.println(
"printing results for profile: " + results.getProfileInfo().getProfileName());
if (results.getRows() == null || results.getRows().isEmpty()) {
System.out.println("No results Found.");
} else {
// Print column headers.
for (ColumnHeaders header : results.getColumnHeaders()) {
pwt.print(header.getName() + ", ");
}
pw.println();
// Print actual data.
for (List<String> row : results.getRows()) {
for (String column : row) {
pw.print(column + ", ");
}
pw.println();
}
System.out.println();
}
}
}
giving me errors saying that it doesnt read it. Anyone wanna give me some pointers?
getting errors that says pw cannot be resolved :/
Well where you loop through the columns just add a comma.
for(String column : row) {
System.out.println(column + ", ");
}
You might need to modify it as this code adds a comma at the end of each element, including the last one. You would need to find out if column is the last element of row, and only add a comma if it ISN'T.
You could connect it to a FileWriter (through BufferedWriter) and output it to a .csv file if you wanted. Put this code at the beginning of your method
try {
PrintWriter pw = new PrintWriter(BufferedWriter(new FileWriter("data.csv")));
}
catch(Exception e) {
e.printStackTrace();
}
and then to print each line your loop would look like this
for (List<String> row : results.getRows()) {
for (String column : row) {
pw.print(row + ",");
}
pw.println();
}
again you would need to find out if column is the last element of row.
You can export data from Google Analytics to BigQuery and then to csv using this tool - https://www.owox.com/products/bi/pipeline/google-analytics-to-google-bigquery-streaming/
Without Java.