How to load multiple csv into multiple tables using sql loader - java

I use SQL loader to load multiple CSV files into multiple tables.
Example: I have the a.csv, b.csv, c.csv files loaded in the Employee table, and the d.csv, e.csv, and f.csv files load into the Student table.
I have written a Java script to implement SQL loader
#RestController
public class FileUploadController {
// Using ServletContext
#Autowired
ServletContext context;
String folderUpload = context.getRealPath("/WEB-INF/uploaded");
Process p = Runtime.getRuntime().exec(new String[] { "cmd", "/C", "all.bat" }, null,
new File(folderUpload));
String sqlldrCmd = "Sqlldr baotrung/baotrung1192 control = " + folderUpload + "/full.ctl"
+ "log=d:/bt.log skip=1";
System.out.println(sqlldrCmd.replace("\\", "/"));
System.out.println("SQLLDR Started ....... ");
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec(sqlldrCmd.replace("\\", "/"));
System.out.println(proc.waitFor());
System.out.println("SQLLDR Ended ........ ");
I know in SQL loadder there is an option to load:
load data
INFILE 'loader2.csv'
INTO TABLE articles_formatted
APPEND
FIELDS TERMINATED BY ','
(article_id SEQUENCE (MAX, 1),
author CHAR (30)
format,
pub_date SYSDATE,
title,
ext_fname FILLER CHAR (80)
text LOBFILE (ext_fname) TERMINATED BY EOF)
However, if I use the load data command
INFILE only loads a file into a table. If I want to continue loading I have to continue writing that command. This is impossible to do because my system has too many CSV files. I have an idea of ​​joining the CSV file into a single CSV file but my CSV files are for multiple tables rather than one table so the work This connection is not implemented. I have two questions:
How to determine which CSV file is loaded into the table when the
number of CSV is so large. I made the CSV name match the table name
but it did not fit my system.
How to load multiple CSV into multiple tables. Is there a way to
insert all the files in my directory into the LOAD DATA INFILE
statement and load it one by one?
I have written a Java to do this problem but it can not be read sequentially
package com.baotrung;
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.Stream;
public class ListFile {
public static void main(String[] args) {
File dir = new File("E:\\xls");
File[] files = dir.listFiles((d, name) -> name.endsWith(".csv"));
for (File file : files) {
file.getName();
}
Path content = Paths.get("E://xls//final.ctl");
Stream<String> stream;
try {
stream = Files.lines(content);
String line1 = Files.readAllLines(Paths.get("E://xls//final.ctl")).get(1);
System.out.println(line1);
List<String> replaced = stream.map(line -> line.replace(line1, files[0].getName().toString()))
.collect(Collectors.toList());
Files.write(content, replaced);
stream.close();
System.out.println("Find and Replace done!!!");
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
The code executes the list of CSV files in the directory, then opens the final.ctl file and executes the contents of the CSV file. But I can not duplicate it to make the week of the files in the folder.
How can I read the files at [0] files in turn.
Please help.

If you merge all your CSV files, I'd suggest you to set - in its first column (or an identifier) a value that distinguishes records. For example:
first;1;little;20;111 --> should go to table called "first"
first;2;foot;30;111
second;100;donald;50;43 --> should go to table called "second"
Then you'd utilize control file's WHEN clause (search for "Loading records based on condition", here) and specify which identifier's records go to which table. The following example is adjusted from Case Study 5: Loading data into multiple tables:
LOAD DATA
INFILE ...
INTO TABLE first
WHEN identifier= 'first'
(empno POSITION(1:4) INTEGER EXTERNAL,
ename POSITION(6:15) CHAR,
deptno POSITION(17:18) CHAR,
mgr POSITION(20:23) INTEGER EXTERNAL)
INTO TABLE second
WHEN identifier= 'second'
(empno POSITION(1:4) INTEGER EXTERNAL,
projno POSITION(25:27) INTEGER EXTERNAL)
INTO TABLE third
WHEN identifier= 'third'
(empno POSITION(1:4) INTEGER EXTERNAL,
projno POSITION(29:31 INTEGER EXTERNAL)
I guess that it should work just fine. Moreover, now you know the keywords to search for (WHEN, MULTIPLE TABLES) so that you could Google yourself for some more info. Good luck!

Related

Need to Create in-memory file structure hierarchy in Java

I need to write a Java program which reads data from a file called input.txt, parses it, and figures out the total size of storage consumed by each directory in this system.
The size of storage consumed by any directory is defined as the sum of the size of this directory, sizes of all the files in this directory and the total storage sizes consumed by all the directories in this directory. The program should write the name of each directory, and the total storage consumed by it to a file called output.txt.
Consider the following input file:
Name | Type | Size | Owner | Directory
root , directory , 128 , admin , NONE
users , directory , 512 , admin , root
navin , directory , 1024 , navin , users
navin.jpg , photo , 128000 , navin , navin
This contains information about files stored in a file system. Each line corresponds to one file, and the fields are separated by commas. "The first field contains the filename, the second contains the file type, the third field is the size of the file in bytes, the fourth field is the username of the owner of the file, and the last field is the name of the parent directory of this file" (i.e. the name of the directory in which this file is located.) Note: the special parent directory name NONE indicates that this file is the root directory of the filesystem.
Also, for the purposes of this program, assume that all file/directory names are unique in the system.
I have tried below code:
package com.threads;
import java.io.*;
public class TotalDirectorySize {
public static void main(String[] args) {
BufferedReader br = null;
try {
String sCurrentLine;
br = new BufferedReader(new FileReader("C:\\input.txt"));
while ((sCurrentLine = br.readLine()) != null) {
System.out.println("Row values from file" + " " + sCurrentLine);
String[] values = sCurrentLine.split(",");
for (String val : values)
System.out.println("Values are" + " " + val);
createDirectory(values);
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (br != null)
br.close();
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
}
Here I am trying to fetch five values in an array and want to create specific directory based on the last value in array. I am thinking of creating an in-memory structure but not sure how to use data structures java. Please suggest a good method to do it.

How to remove garbage value from a file name which has been created by using createTempFile() method

I have used File.createTempFile() method to create temp file but as its output it appends the garbage value with the file name too. I used the method for uploading zipfile, but unable to delete those appended garbage value. For further functionality I need the exact name of file.
Kindly help...
Highly appreciate your response.
My concern is, as code stated by niiraj874u, I am getting the the File name : tmp4501156806082176909.txt
But I want only tmp.txt How can I remove appended numeric value?
You can use java.io.File.getName() method to get name of file..
import java.io.File;
import java.io.IOException;
public class FileDemo {
public static void main(String[] args) {
File f = null;
// creates temporary file
try {
f = File.createTempFile("tmp", ".txt", new File("D:/"));
} catch (IOException e) {
e.printStackTrace();
}
// prints name of temp file
System.out.println("File name: "+f.getName());
// prints absolute path
System.out.println("File path: "+f.getAbsolutePath());
}
}
this will print like
File name: tmp4501156806082176909.txt
File path: D:\tmp4501156806082176909.txt
It sounds like you don't need a temp file. The purpose of the "garbage" is to protect two or more instances of the app from overwriting each other. In this case use system.getProperty("java.io.temp") to get the temp dir.

Netbeans Java: Where to put my CSV file?

I followed a tutorial to create some simple code to output the contents of a csv file. However, I always get the following message:
java.io.FileNotFoundException: Data.csv (The system cannot find the file specified)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at java.util.Scanner.<init>(Scanner.java:656)
at testing.csv.files.Test.main(Test.java:26)
BUILD SUCCESSFUL (total time: 0 seconds)
So I guess this means that the program is running, but it can't find my csv file. Basically, I just dragged and dropped it from my desktop into the "Source Packages" file in my Java Project, which is where my Test.java file is. I've also tried putting it in the "testing.csv.files", but that did not work either. Neither did putting it in the "Test Packages".
I've ran out of ideas. Where am I supposed to put this csv file?
here is my code:
package testing.csv.files;
import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;
import java.util.logging.Level;
import java.util.logging.Logger;
public class Test {
public static void main(String[] args) {
//.csv comma separated values
String fileName = "Data.csv";
File file = new File(fileName); // TODO: read about File Names
try {
Scanner inputStream = new Scanner(file);
while (inputStream.hasNext()){
String data = inputStream.next();
System.out.println(data);
}
inputStream.close();
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
You could try pointing to the full path of the file.
For example: String fileName = "Desktop/Data.csv";
FYI - You can copy the full path of a file by right clicking on a file while holding shift, then selecting: "copy as path".
Put your csv file(i.e Data.csv) in your project folder then it will work properly i tried your code it works fine for me

how to read a text file using scanner in Java?

This is my code to read a text file. When I run this code, the output keeps saying "File not found.", which is the message of FileNotFoundException. I'm not sure what is the problem in this code.
Apparently this is part of the java. For the whole java file, it requires the user to input something and will create a text file using the input as a name.
After that the user should enter the name of the text file created before again (assume the user enters correctly) and then the program should read the text file.
I have done other parts of my program correctly, but the problem is when i enter the name again, it just can not find the text file, eventhough they are in the same folder.
public static ArrayList<DogShop> readFile()
{
try
{ // The name of the file which we will read from
String filename = "a.txt";
// Prepare to read from the file, using a Scanner object
File file = new File(filename);
Scanner in = new Scanner(file);
ArrayList<DogShop> shops = new ArrayList<DogShop>();
// Read each line until end of file is reached
while (in.hasNextLine())
{
// Read an entire line, which contains all the details for 1 account
String line = in.nextLine();
// Make a Scanner object to break up this line into parts
Scanner lineBreaker = new Scanner(line);
// 1st part is the account number
try
{ int shopNumber = lineBreaker.nextInt();
// 2nd part is the full name of the owner of the account
String owner = lineBreaker.next();
// 3rd part is the amount of money, but this includes the dollar sign
String equityWithDollarSign = lineBreaker.next();
int total = lineBreaker.nextInt();
// Get rid of the dollar sign;
// we use the subtring method from the String class (see the Java API),
// which returns a new string with the first 'n' characters chopped off,
// where 'n' is the parameter that you give it
String equityWithoutDollarSign = equityWithDollarSign.substring(1);
// Convert this balance into a double, we need this because the deposit method
// in the Account class needs a double, not a String
double equity = Double.parseDouble(equityWithoutDollarSign);
// Create an Account belonging to the owner we found in the file
DogShop s = new DogShop(owner);
// Put money into the account according to the amount of money we found in the file
s.getMoney(equity);
s.getDogs(total);
// Put the Account into the ArrayList
shops.add(s);
}
catch (InputMismatchException e)
{
System.out.println("File not found1.");
}
catch (NoSuchElementException e)
{
System.out.println("File not found2");
}
}
}
catch (FileNotFoundException e)
{
System.out.println("File not found");
} // Make an ArrayList to store all the accounts we will make
// Return the ArrayList containing all the accounts we made
return shops;
}
If you are working in some IDE like Eclipse or NetBeans, you should have that a.txt file in the root directory of your project. (and not in the folder where your .class files are built or anywhere else)
If not, you should specify the absolute path to that file.
Edit:
You would put the .txt file in the same place with the .class(usually also the .java file because you compile in the same folder) compiled files if you compile it by hand with javac. This is because it uses the relative path and the path tells the JVM the path where the executable file is located.
If you use some IDE, it will generate the compiled files for you using a Makefile or something similar for Windows and will consider it's default file structure, so he knows that the relative path begins from the root folder of the project.
Well.. Apparently the file does not exist or cannot be found. Try using a full path. You're probably reading from the wrong directory when you don't specify the path, unless a.txt is in your current working directory.
I would recommend loading the file as Resource and converting the input stream into string. This would give you the flexibility to load the file anywhere relative to the classpath
If you give a Scanner object a String, it will read it in as data. That is, "a.txt" does not open up a file called "a.txt". It literally reads in the characters 'a', '.', 't' and so forth.
This is according to Core Java Volume I, section 3.7.3.
If I find a solution to reading the actual paths, I will return and update this answer. The solution this text offers is to use
Scanner in = new Scanner(Paths.get("myfile.txt"));
But I can't get this to work because Path isn't recognized as a variable by the compiler. Perhaps I'm missing an import statement.
This should help you..:
import java.io.*;
import static java.lang.System.*;
/**
* Write a description of class InRead here.
*
* #author (your name)
* #version (a version number or a date)
*/
public class InRead
{
public InRead(String Recipe)
{
find(Recipe);
}
public void find(String Name){
String newRecipe= Name+".txt";
try{
FileReader fr= new FileReader(newRecipe);
BufferedReader br= new BufferedReader(fr);
String str;
while ((str=br.readLine()) != null){
out.println(str + "\n");
}
br.close();
}catch (IOException e){
out.println("File Not Found!");
}
}
}
Just another thing... Instead of System.out.println("Error Message Here"), use System.err.println("Error Message Here"). This will allow you to distinguish the differences between errors and normal code functioning by displaying the errors(i.e. everything inside System.err.println()) in red.
NOTE: It also works when used with System.err.print("Error Message Here")

Store and retrieve word documents with MySQL

I need to store and retrieve MS Word documents into MySQL 5.1 with Servlets. I've the code to upload a file, but I don't know can I feed into the table. I've used BLOB for the field I've to insert .doc files.
Here's my code snippet to upload files:
protected void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
response.setContentType("text/html;charset=UTF-8");
PrintWriter out = response.getWriter();
try {
// get access to file that is uploaded from client
Part p1 = request.getPart("file");
String type=p1.getContentType();
String name=p1.getName();
long size = p1.getSize();
InputStream is = p1.getInputStream();
//FileInputStream fis = is.
// read filename which is sent as a part
Part p2 = request.getPart("name");
Scanner s = new Scanner(p2.getInputStream());
String filename = s.nextLine(); // read filename from stream
// get filename to use on the server
String outputfile = this.getServletContext().getRealPath(filename); // get path on the server
FileOutputStream os = new FileOutputStream (outputfile);
// write bytes taken from uploaded file to target file
int ch = is.read();
while (ch != -1) {
os.write(ch);
ch = is.read();
}
os.close();
out.println("<h3>File : '" + name + "' Type : '" + type + "' "
+ "of Size : " + ((double) size/1024) + "KB uploaded successfully!</h3>");
}
catch(Exception ex) {
out.println("Exception -->" + ex.getMessage());
}
finally {
out.close();
}
}
Here, I've used Servlets 3.0 feature for uploading a file...
My table schema :
resources
- UserID [varchar(15)]
- Document [mediumblob]
Could anyone help me how can I store the document into the table and though BLOB is a type representing binary data, how can I retrieve as a Word Document (*.doc)?
I agree with Archimedix... Instead of putting them into MySQL as BLOB, you can store the file on the disk and store its path in MYSQL as TEXT field. This way your retrieval time will be low. If you are space conscious then you can zip the doc and save it on the disk and on request uncompress and send it.
UPDATE
From your code it appears that you already have the handle of the file and you are able to save it on the server.
Now to save space you can zip it using default java zip utility.
You might face a problem when two people upload two different files with the same name. To avoid scenarios like this you can either rename your archived document with an uuid (use java 6 uuid class) or you can generate SHA1 for that file and use that for name.
Now you can use the absolute path of the archived (and renamed file) for storing in the MySQL.
Instead of table schema
resources
UserID [varchar(15)]
Document [mediumblob]
You can use this
resources
UserID [varchar(15)]
Document [varchar(512)]
So for a query like this:
Select Document from table Documents WHERE UserID = 'abcd';
you will now get an absolute path for the zipped file. Uncompress this file and send it.
A partial answer on storing the Word documents in files:
You don't need any additional column to save the file name as the document's record ID can serve as the file name.
When saving a new document, do in a database transaction so that you can undo the process when something goes wrong.
In pseudo code, this would look like this:
begin transaction;
try {
save new record for document;
save Word document in predefined directory, using record's ID as the filename;
} catch (Exception e) {
rollback transaction;
throw e; // Rethrow exception
}
commit transaction;
The code above assumes that an exception is thrown when an error occurs.

Categories

Resources