I have star schema model in which Server Table contains information about server name. Information Table contains information that I want for specific server. And Actual Data Table contains information about which server contains which information.
Server Table
Information Table
Actual Data Table
Now the problem that I am having is- I am trying to insert Data into the Data Table using JDBC. But I am unsure how should I add data into Actual Data Table in star schema model. Should I connect to database and insert it every time for each information or there is any direct way we can do that by communicating to database only one time. This is my code where I am getting all the information for each server. And IndexData is the class where I insert values into Oracle Database.
public void fetchlog() {
InputStream is = null;
InputStream isUrl = null;
FileOutputStream fos = null;
try {
is = HttpUtil.getFile(monitorUrl);
if(monitorUrl.contains("stats.jsp") || monitorUrl.contains("monitor.jsp")) {
trimUrl = monitorUrl.replaceAll("(?<=/)(stats|monitor).jsp$", "ping");
}
isUrl = HttpUtil.getFile(trimUrl);
BufferedReader in = new BufferedReader (new InputStreamReader (is));
String line;
int i=0,j=0,k=0;
while ((line = in.readLine()) != null) {
if(line.contains("numDocs")) {
docs = in.readLine().trim();
//So should I keep on inserting into Database for each information, like this
//IndexData id = new IndexData(timeStamp, ServerName, InformationName, docs);
} else if(line.contains("indexSize")) {
indexSize = in.readLine().trim();
//For this information-- the same way?
//IndexData id = new IndexData(timeStamp, ServerName, InformationName, indexSize);
} else if(line.contains("cumulative_lookups")) {
cacheHits= in.readLine().trim();
//For this information too-- the same way?
//IndexData id = new IndexData(timeStamp, ServerName, InformationName, cacheHits);
} else if(line.contains("lastCommitTime")) {
lastCommitTime = in.readLine().trim();
//For this information too-- the same way?
//IndexData id = new IndexData(timeStamp, ServerName, InformationName, lastCommitTime );
}
BufferedReader inUrl = new BufferedReader (new InputStreamReader (isUrl));
String lineUrl;
Pattern regex = Pattern.compile("<str name=\"status\">(.*?)</str>");
while ((lineUrl = inUrl.readLine()) != null) {
System.out.println(lineUrl);
if(lineUrl.contains("str name=\"status\"")) {
Matcher regexMatcher = regex.matcher(lineUrl);
if (regexMatcher.find()) {
upDown= regexMatcher.group(1);
//For this information too-- the same way?
//IndexData id = new IndexData(timeStamp, ServerName, InformationName, upDown);
}
System.out.println("Status:- " + status);
}
}
//Or is there some other way we can insert directly into database by communicating with database only one time not multiple times for each information.
//IndexData id = new IndexData(timeStamp, ServerName, InformationName, Value);
fos = new FileOutputStream(buildTargetPath());
IOUtils.copy(is, fos);
} catch (FileNotFoundException e) {
log.error("File Exception in fetching monitor logs :" + e);
} catch (IOException e) {
log.error("Exception in fetching monitor logs :" + e);
}
}
I hope question is clear to everyone. Any suggestions will be appreciated.
There are two things I would suggest you look at. First, use a batch insert to perform all of the associated inserts in one JDBC transaction. For more information:
JDBC Batch Insert Example
I would also strongly recommend that you use a JDBC connection pooling library. We use c3p0 with our Postgres database. You can find more information here:
c3p0 Project Page
The basic idea would is to create a connection pool at startup time, then create JDBC batches for each set of related inserts.
Related
So i have database with value like this...
i'm trying to append the value by using insert into without replacing it,the data from this txt file...
but when i reload/refresh the database there is no new data being appended into the database...,
here is my code....
public static void importDatabase(String fileData){
try{
File database = new File(fileData);
FileReader fileInput = new FileReader(database);
BufferedReader in = new BufferedReader(fileInput);
String line = in.readLine();
line = in.readLine();
String[] data;
while (line != null){
data = line.split(",");
int ID = Integer.parseInt(data[0]);
String Nama = data[1];
int Gaji = Integer.parseInt(data[2]);
int Absensi = Integer.parseInt(data[3]);
int cuti = Integer.parseInt(data[4]);
String Status = data[5];
String query = "insert into list_karyawan values(?,?,?,?,?,?)";
ps = getConn().prepareStatement(query);
ps.setInt(1,ID);
ps.setString(2,Nama);
ps.setInt(3,Gaji);
ps.setInt(4,Absensi);
ps.setInt(5,cuti);
ps.setString(6,Status);
line = in.readLine();
}
ps.executeUpdate();
ps.close();
con.close();
System.out.println("Database Updated");
in.close();
}catch (Exception e){
System.out.println(e);
}
}
When i run it, it shows no error but the data never get into database, where did i go wrong?.,...
Auto-commit mode is enabled by default.
The JDBC driver throws a SQLException when a commit or rollback operation is performed on a connection that has auto-commit set to true.
Symptoms of the problem can be unexpected application behavior
update the JVM configuration for the ActiveMatrix BPM node to use the following Oracle connection property:
autoCommitSpecCompliant=false Try once
Note:I am not able to put as comment so i posted as a answer
I have csv of 32GB with almost 150million rows, i planned to use SStableloader to export data to cassandra on EC2, & to generate SStable i used java codes below.
Problem is, on server i am only getting 12k rows, also the filesize of generated SStable is just 28 m & process is not throwing any error.
Moreover, if i execute it on another .csv, one with 10 rows, no issues, i get all 10 rows.
if(args.length < 2){
System.out.println("Something wrong with parameters, heres pattern: <CSV_URL> <Default_Output_Dir>");
return;
}
CSV_URL = args[0];
DEFAULT_OUTPUT_DIR = args[1];
// magic!
Config.setClientMode(true);
// Create output directory that has keyspace and table name in the path
File outputDir = new File(DEFAULT_OUTPUT_DIR + File.separator + KEYSPACE + File.separator + TABLE);
if (!outputDir.exists() && !outputDir.mkdirs())
{
throw new RuntimeException("Cannot create output directory: " + outputDir);
}
// Prepare SSTable writer
CQLSSTableWriter.Builder builder = CQLSSTableWriter.builder();
// set output directory
builder.inDirectory(outputDir)
// set target schema
.forTable(SCHEMA)
// set CQL statement to put data
.using(INSERT_STMT)
// set partitioner if needed
// default is Murmur3Partitioner so set if you use different one.
.withPartitioner(new Murmur3Partitioner());
CQLSSTableWriter writer = builder.build();
try (
BufferedReader reader = new BufferedReader(new FileReader(CSV_URL));
CsvListReader csvReader = new CsvListReader(reader, CsvPreference.STANDARD_PREFERENCE)
){
//csvReader.getHeader(true);
// Write to SSTable while reading data
List<String> line;
while ((line = csvReader.read()) != null)
{
writer.addRow(
Integer.parseInt(line.get(0)),
..
new BigDecimal(line.get(22)),
new BigDecimal(line.get(23))
);
}
}
catch (Exception e)
{
e.printStackTrace();
}
try
{
writer.close();
}
catch (IOException ignore) {}
and here's schema:
CREATE KEYSPACE IF NOT EXISTS ma WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};
USE ma;
CREATE TABLE IF NOT EXISTS cassie (PKWID int,DX varchar,......, QS decimal,PRIMARY KEY (PKWID));
using Cassandra 22x.
Java driver for creating SSTable
I'm already taking some questions from database with JDBC, resultsets, etc... then putting this into arraylists.. and show all via Java GUI.
However now i want to extend this using sockets, threads etc.. to learn this staff.
This time i want to use three tier architecture approach, where the database access is
sorted in the application layer which interacts the same old database.
Application layer is going to act like a server which accept incoming
requests of multiple clients and provide the interaction with the data level.
I try something like this;
Server side: taking questions(20 tf questions, 8 multiple), writing to arraylist and sends to the client.
I send only tf questions. How can i send also multiples ?
Is this true way to do this ?
If threads are necessary ? How can i implement threads to this ?
public class trainingServer {
static ArrayList<TrueFalse> truefalseList = new ArrayList<TrueFalse>();
static ArrayList<Multiple> multipleList = new ArrayList<Multiple>();
static ArrayList<String> clientAnswers = new ArrayList<String>();
public static void main(String[] args) throws IOException, SQLException, ClassNotFoundException {
Connection con;
Statement st;
ResultSet resultSet, resultSet2;
ServerSocket serverSocket = null;
Socket clientSocket = null;
ObjectOutputStream out = null;
ObjectInputStream in = null;
PrintWriter outScore = null;
Class.forName("com.mysql.jdbc.Driver");
con = DriverManager.getConnection("jdbc:mysql://localhost/db", "root", "");
st = con.createStatement();
try {
serverSocket = new ServerSocket(9999); //listening on port 9999
} catch (Exception e) {
System.out.println("Port Error!");
}
System.out.println("Server is ready for connection..");
clientSocket = serverSocket.accept();
out = new ObjectOutputStream(clientSocket.getOutputStream());
in = new ObjectInputStream(clientSocket.getInputStream());
outScore = new PrintWriter(clientSocket.getOutputStream(), true);
resultSet = st.executeQuery("SELECT No, Question, Answer FROM truefalse");
//adding true/false questions and their answers to tfList from database
while (resultSet.next())
{
TrueFalse qa = new TrueFalse();
qa.number=resultSet.getInt(1);
qa.question=resultSet.getString(2);
qa.answer=resultSet.getString(3);
truefalseList.add(qa);
}
//writing true-false part to object output stream to send client
// for(TrueFalse tf : truefalseList)
// out.writeObject(tf);
resultSet2 = st.executeQuery("SELECT No, Question, Ans1, Ans2, Ans3, Ans4, Ans5, Explanation, Trueans FROM multiple");
//adding multiple questions and their answers to multList from database
while (resultSet2.next())
{
Multiple qm = new Multiple();
qm.num=resultSet2.getInt(1);
qm.question=resultSet2.getString(2);
qm.answer1=resultSet2.getString(3);
qm.answer2=resultSet2.getString(4);
qm.answer3=resultSet2.getString(5);
qm.answer4=resultSet2.getString(6);
qm.answer5=resultSet2.getString(7);
qm.explanation=resultSet2.getString(8);
qm.trueAns=resultSet2.getString(9);
multipleList.add(qm);
}
//writing mult. part to object output stream to send client
// for(Multiple mult : multipleList)
// out.writeObject(mult);
out.writeUTF("Server ready");
out.flush();
// If we are here, then connection was probably not a portscan
out.writeObject(truefalseList);
out.writeObject(multipleList);
out.flush();
clientAnswers = (ArrayList<String>) in.readObject();
// score = (Score)in.readObject();
// do something with score.
outScore.println(checkAnswers());
outScore.flush();
out.close();
in.close();
outScore.close();
clientSocket.close();
serverSocket.close();
}
static int checkAnswers(){
int score=0;
for(int i=0 ; i<clientAnswers.size() ; i++)
{
if(i<20)
{
if(clientAnswers.get(i).equalsIgnoreCase(truefalseList.get(i).answer))
{
score=score+3;
}
}
else
{
int count=0;
if(clientAnswers.get(i).equalsIgnoreCase(multipleList.get(count).trueAns))
{
score=score+5;
count++;
}
}
}
return score;
}
}
Client side: (except gui staff etc..)
Socket soket = null;
ObjectInputStream in = null;
ObjectOutputStream out = null;
BufferedReader inScore = null;
try {
soket = new Socket("localhost",9999);
in = new ObjectInputStream(soket.getInputStream());
out = new ObjectOutputStream(soket.getOutputStream());
inScore = new BufferedReader(new InputStreamReader(soket.getInputStream()));
String serverReady = in.readUTF();
tfList = (ArrayList<TrueFalse>) in.readObject();
multList = (ArrayList<Multiple>) in.readObject();
// score = new Score();
// prepare the score object
out.writeObject(userAnswers);
out.flush();
// wait for server to process score
returned_score = inScore.read();
System.out.println(returned_score);
// server has processed score and send "Bye".
} catch (ConnectException ce) {
System.out.println("Cannot connect to the server!");
} catch (IOException ie) {
System.out.println("I/O Error!");
}
in.close();
out.close();
inScore.close();
soket.close();
The true way of doing this depends on your non-functional requirements. If you expect non-Java clients to request the (status) data from the database, or clients are behind a proxy, then write your application server as a web-service.
Since you are writing a server that clients will depend on, also think about scalability (deploy the web-service on multiple web-servers behind a load-balancer) and availability (if one web-server goes down, the load-balancer should direct requests to other web-servers that are still up). And don't forget maintainability: ideally you should be able to upgrade/downgrade your application server without clients noticing.
If there are only Java clients (or the data on the line is only question-answer type text like with telnet), then you will need threads and also a database connection pool. I have used Yapool to create an "always on" application server like this. It contains a ServerSocket that you can use/look at to learn this stuff (see also the unit tests). A database pool implementation is also included.
I have also combined Yapool with a basic JSP web-service framework to create a war-file that can be deployed on Tomcat, exposing the application server as a (very simple) web-service. The advantage of a web-service like this is that you do not need to worry (much) about threads and connections: Tomcat does that for you. If you want to take it a step further, I suggest learning about REST.
As for receiving two types of lists: you are inventing your own protocol here and are free to send and receive data to/from the client as long as the client mirrors the server protocol exactly. You can send/receive any kind of object with the ObjectInputStream/ObjectOutputStream as long as it is serializable. Lists and Maps are already serializable.
For example server protocol:
out.writeUTF("Server ready");
out.flush();
// If we are here, then connection was probably not a portscan
out.writeObject(tfList);
out.writeObject(multipleList);
out.flush();
clientAnswers = (List<String>) in.readObject();
out.writeInt(checkAnswers());
out.writeUTF("Bye");
out.flush();
With the following client protocol:
String serverReady = in.readUTF();
tfList = (List<TrueFalse>) in.readObject();
multipleList = (List<Multiple>) in.readObject();
out.writeObject(userAnswers);
out.flush();
// wait for server to check answers and return score
int returned_score = in.readInt();
String serverBye = in.readUTF();
// server has processed score and send "Bye".
I'm making a whois in java for android to train about streams and tcp connections.
But I have a problem. I have a php script, I wrote some time ago and I´m trying to make the same in java.
this is my java code :
public String consultawhois(String domain,String tld)
{
String domquest = domain + "." + tld;
String resultado = "";
Socket theSocket;
String hostname = "whois.internic.net";
int port = 43;
try {
theSocket = new Socket(hostname, port, true);
Writer out = new OutputStreamWriter(theSocket.getOutputStream());
out.write(domquest + "\r\n");
out.flush();
DataInputStream theWhoisStream;
theWhoisStream = new DataInputStream(theSocket.getInputStream());
String s;
while ((s = theWhoisStream.readLine()) != null) {
resultado = resultado + s + "\n";
}
}
catch (IOException e) {
}
return resultado;
}
The answer of the server is not correct and I think the problem is that I'm sending a bad query. The query I send is "dominio.com\r\n" and in my php whois code, it works perfectly.
It seems that the DNS query matches multiple records. At least, that is how I interpret the response. In the returned reponse you should see the following line:
To single out one record, look it up with "xxx", where xxx is one of the
of the records displayed above. If the records are the same, look them up
with "=xxx" to receive a full display for each record.
So if you prepend the query with "=" it returns the data of that record only. The following worked for me.
public String consultawhois(String domain,String tld)
{
String domquest = domain + "." + tld;
String resultado = "";
Socket theSocket;
String hostname = "whois.internic.net";
int port = 43;
try {
theSocket = new Socket(hostname, port, true);
Writer out = new OutputStreamWriter(theSocket.getOutputStream());
out.write("="+domquest + "\r\n");
out.flush();
DataInputStream theWhoisStream;
theWhoisStream = new DataInputStream(theSocket.getInputStream());
String s;
while ((s = theWhoisStream.readLine()) != null) {
resultado = resultado + s + "\n";
}
}
catch (IOException e) {
}
return resultado;
}
One thing to consider: Use English for method names, variables, etc. instead of Spanish. It will make your code easier to read internationally. The programming language itself also uses English words. Try to avoid a strange mix of English and your native language.
The lookup for dominio.com results in three matches:
DOMINIO.COM.BR
DOMINIO.COM.ASCPROBIENESTARIDSS.COM
DOMINIO.COM
You should specify wich one you are interested in with the query.
=dominio.com<newline>
This will allways work, even cases where there are no multiple matches.
i'm new to Android and i'm developing an application who is supposed to show a list of places (the class extend activitylist) and when user choose a place it open a new activity with the place details (name, address, phone number, service..).
Actually, i'm looking for a simple way to store those predefined locations details + a simple way to show them later on the app.
Actually, the best solution that i found is:
Save a static file in your application at compile time, save the file in your project res/raw/ directory.
Open the file with openRawResource(), passing the R.raw. resource ID. This method returns an InputStream that you can use to read the file (but you cannot write to the original file).
InputStream dataIS =
getResources().openRawResource(R.raw.location);
Convert input stream to buffered reader then you can store your data in a Sqlite3 table and use them wherever you want.
public void fillDB(){
InputStreamReader in= new InputStreamReader(dataIS);
BufferedReader dataBR= new BufferedReader(in);
String dataLine;
try {
while ((dataLine = dataBR.readLine()) != null) {
// split the data line
dataLineTokenizer = new StringTokenizer(dataLine, ":");
//SQL query + save data to database
String sql = "INSERT INTO location ...";
//execute query
Log.v("Test Saving", sql);
clubberDB.execSQL(sql);
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}