I am new to JSON programming. I am trying to write a webservice to fetch data from a SQLite table and write the output to a JSON file. I have added the JSON simple jar to my eclipse project. When I add a line of code as shown below,
JSONObject obj = new JSONObject();
it compiles properly. But, when the webservice is invoked, I get an exception at this line as shown below,
Exception: java.lang.reflect.InvocationTargetException Message: java.lang.reflect.InvocationTargetException
This is my complete java file...
package wtp;
import java.io.FileWriter;
import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.Statement;
import org.json.simple.JSONObject;
import org.json.simple.JSONArray;
public class CollegeProcessor2 {
public static String fetchCollegeList2(String input)
{
Connection c = null;
Statement stmt = null;
String output = null;
java.util.Date date= new java.util.Date();
JSONObject collegeList = new JSONObject();
JSONObject college = new JSONObject();
JSONArray list = new JSONArray();
try {
Class.forName("org.sqlite.JDBC");
c = DriverManager.getConnection("jdbc:sqlite:C:\\Users\\Vevin.Kumar\\workspace\\JSONProject\\collegelist.db");
c.setAutoCommit(false);
System.out.println("Opened database successfully");
stmt = c.createStatement();
ResultSet rs = stmt.executeQuery( "SELECT * FROM COLLEGES WHERE DISTRICT LIKE \"" + input +"\" ;");
ResultSetMetaData metaData = rs.getMetaData();
int numberOfColumns=metaData.getColumnCount();
while ( rs.next() ) {
for(int i = 1; i <= numberOfColumns; i++)
{
college.put(String.valueOf(metaData.getColumnName(i)), String.valueOf(rs.getObject(i)));
}
list.add(college);
}
collegeList.put("collegeslist", list);
rs.close();
stmt.close();
c.close();
} catch ( Exception e ) {
System.err.println( e.getClass().getName() + ": " + e.getMessage() );
System.exit(0);
}
try{
FileWriter file = new FileWriter("c:\\json_outputs\\collegelist_" + date.getTime() +".json");
file.write(collegeList.toJSONString());
file.flush();
file.close();
} catch ( IOException e ){
System.err.println( e.getClass().getName() + ": " + e.getMessage() );
System.exit(0);
}
System.out.println("Operation done successfully");
output = "c:\\json_outputs\\collegelist_" + date.getTime() +".json";
return output;
}
/*public static void main (String[] args)
{
String output = fetchCollegeList2("CHENNAI");
System.out.println(output);
}*/
}
If I run this as a Java application, the JSON file is getting generated properly. If I create a bottom down web service from this and invoke the method from it, I am getting the exception mentioned above. The exception occurs right at the line where the JSONObject is initialized.
Can anyone please help me out on this?
Thanks & Regards,
Vevin.
Related
I am trying to connect to VSAM files in z/OS in Java.
I have tried using VSE Redirector Connector with below code:
package test;
import java.sql.DriverManager;
import java.sql.ResultSetMetaData;
public class TestVSAM {
public static void main(String[] args) {
String vsamCatalog = "VSESP.USER.CATALOG";
String flightsCluster = "USERNAME.TEST.KSDS1";
String flightsMapName = "FLIGHTS_MAP";
String ordersCluster = "FLIGHT.ORDERING.ORDERS";
String ordersMapName = "ORDERS_MAP";
String ipAddr = "XXX.XXX.XXX.XXX";
String userID = "XXXXXX";
String password = "XXXXXX";
Integer port = 123;
try {
System.out.println("VSE IP address------------->" + ipAddr);
System.out.println("Your VSE user ID----------->" + userID);
System.out.println("Password------------->" + password);
System.out.println("Port------------->" + port);
java.sql.Connection jdbcCon;
java.sql.Driver jdbcDriver =
(java.sql.Driver) Class.forName("com.ibm.vse.jdbc.VsamJdbcDriver").newInstance();
// Build the URL to use to connect
String url = "jdbc:vsam:" + ipAddr;
// Assign properties for the driver
java.util.Properties prop = new java.util.Properties();
prop.put("port", port);
prop.put("user", userID);
prop.put("password", password);
// Connect to the driver
jdbcCon = DriverManager.getConnection(url, prop);
// Get a statement
java.sql.Statement stmt = jdbcCon.createStatement();
// Execute the query ...
java.sql.ResultSet rs = stmt.executeQuery(
"SELECT * FROM " + vsamCatalog + "\\" + flightsCluster + "\\" + flightsMapName);
ResultSetMetaData rsmd = rs.getMetaData();
int columnCount = rsmd.getColumnCount();
for (int i = 1; i <= columnCount; i++) {
System.out.println("Key------------>" + rsmd.getColumnLabel(i));
}
rs.close();
stmt.close();
} catch (Exception e) {
System.out.println(e);
}
I'm getting this error:
My Questions:
Can we use VSE Redirector Connector to read VSAM files? If yes, then how can we resolve the above error and get connected?
Is there any other method to access VSAM files thru Java?
Have you considered something like: https://www.ibm.com/docs/en/sdk-java-technology/8?topic=jzos-zfile
This is a beanshell code , so a few things might look
odd for a java developer. The emailFileAttachment function is a sailpoint API, a tool I am using. My problem is the data I am putting in my map is being put in a single line in excel file. And the header ("Application, Num_entitlement") I am putting in the map is not being printed at the first line in the CSV file. Could anyone please help me. This is my code below:
import sailpoint.object.Application;
import sailpoint.object.Identity;
import java.sql.Connection;
import java.sql.PreparedStatement;
import javax.sql.DataSource;
import java.sql.SQLException;
import sailpoint.server.Environment;
import javax.sql.DataSource;
import java.sql.ResultSet;
import sailpoint.api.SailPointContext;
import java.io.BufferedWriter;
import java.io.IOExceptoin;
import java.io.FileWriter;
import sailpoint.object.EmailTemplate;
import sailpoint.object.EmailOptions;
import java.io.File;
import java.io.FileInputStream;
import sailpoint.object.EmailFileAttachment;
import java.util.HashMap;
import sailpoint.tools.Util;
String query = "SELECT app.name as application, count(*) as num_entitlements FROM spt_application as app, spt_identity_entitlement as ent WHERE app.id = ent.application GROUP BY app.name";
HashMap info = new HashMap();
info.put("Application ", "Num_Entitlement");
PreparedStatement getEntitlement_Num = null;
Connection conn = null;
/*
public static byte[] readFiletoByteArray(File file)
{
FileInputStream fileInputStream = null;
byte[] byteFile = new byte[(int) file.length()];
try
{
fileInputStream = new FileInputStream(file);
fileInputStream.read(byteFile);
fileInputStream.close();
}
catch (Exception e)
{
e.printStackTrace();
}
return byteFile;
}
*/
try{
// Prepared Statements
Environment e = Environment.getEnvironment();
DataSource src = e.getSpringDataSource();
//System.out.println("DataSource: " + src.toString());
conn = src.getConnection();
//System.out.println("Connection: " + conn);
getEntitlement_Num = conn.prepareStatement(query);
ResultSet rs = getEntitlement_Num.executeQuery();
//System.out.println("starting RS");
while(rs.next()) {
String appName = rs.getString("application");
int no_ent = rs.getInt("num_entitlements");
info.put(appName , no_ent);
}
System.out.println("finished RS");
}catch(SQLException e){
log.error( e.toString());
} finally {
if (getEntitlement_Num!= null) {
getEntitlement_Num.close();
}
if(conn != null) {
conn.close();
}
}
//I am using sailpoint APIs for the code below.
String emailDest = "//email address here";
EmailTemplate et = new EmailTemplate();
et.setFrom("//email address here");
et.setBody("Please find an attached CSV file that has the list of all applications in IIQ and their number of Entitlements");
et.setTo(emailDest);
et.setSubject("Entitlement count for each application in IIQ");
EmailOptions ops = new EmailOptions(emailDest,null);
String strInfo = Util.mapToString(info);
byte[] fileData = strInfo.getBytes();
EmailFileAttachment attachment = new EmailFileAttachment( "EntitlementCount.csv", EmailFileAttachment.MimeType.MIME_CSV, fileData );
ops.addAttachment(attachment);
context.sendEmailNotification(et, ops);
//System.out.println("email sent");
return "Success";
info is a HashMap which means there's no guarantee that you can extract data in the same order as you put it in. Therefore your header "Application" might not come first in the CSV file. Instead, use something that maintains the order, eg an ArrayList of Tuple objects (a class you write yourself that contain two String variables).
How does Util.mapToString(info) work? We need so see it so we can investigate the newline problem.
Util.mapToString() will just convert map to string.
Try changing your collection to list of list{app, count} and
iterate over the list to generate the string.
methods Util.listToCsv() or Util.listToQuotedCsv() will be helpful to prepare csv string.
Hope this helps.
You should use a StringBuilder in the same loop as the records iteration and then from the String builder formulate the attachment.
I think the Utility.MapToString with the chashma is the root cause.
I am using JTDS to connect to a MS SQL Server. Connection to the database is no problem, but when I try to execute a statement, I get a Database 'java' does not exist exception.
ConnectionString:
conn = DriverManager.getConnection("jdbc:jtds:sqlserver://localhost;DatabaseName=MyDatabase;user=testuser;password=testpassword");
Trying to execute the script:
private void runStatement(String scriptLocation) {
if(scriptLocation == null) {
return;
}
try {
InputStream is = getClass().getClassLoader().getResourceAsStream(scriptLocation);
String query = is.toString();
is.close();
Statement stmt = conn.createStatement();
stmt.executeQuery(query);
} catch(IOException | SQLException ex) {
log.warning(ex.getMessage());
}
}
Stacktrace:
WARNING: Database 'java' does not exist. Make sure that the name is entered correctly.
java.sql.SQLException: Database 'java' does not exist. Make sure that the name is entered correctly.
at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:372)
at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2988)
at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2421)
at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:671)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQLQuery(JtdsStatement.java:505)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeQuery(JtdsStatement.java:1427)
at com.exampe.MyJTDSConnection.runStatement(MyJTDSConnection.java:238)
at com.exampe.MyJTDSConnection.loadPageTitle(MyJTDSConnection.java:208)
at com.exampe.MyJTDSConnection.runTesting(MyJTDSConnection.java:69)
at com.exampe.SeleniumTesting.runTest(SeleniumTesting.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
As mentioned in the comment to your question, applying the .toString() method to the InputStream object does not read the InputStream. Instead it just returns a String representation of the object itself, not what the object contains.
For example, my Java project has a resource file named "script.sql" that contains:
SELECT ##VERSION
The following code compares the result of simply doing a .toString() on the object vs. using Apache Commons IO to actually read the InputStream into a String:
package resourceTest;
import java.io.InputStream;
import org.apache.commons.io.IOUtils;
public class ResourceTestMain {
public static void main(String[] args) {
try (InputStream is = ResourceTestMain.class.getClassLoader().getResourceAsStream("resources/script.sql")) {
String toStringValue = is.toString();
String contents = IOUtils.toString(is, "UTF-8");
is.close();
System.out.println("is.toString() returned:");
System.out.println(" " + toStringValue);
System.out.println();
System.out.println("IOUtils.toString(is, \"UTF-8\") returned:");
System.out.println(" " + contents);
} catch (Exception e) {
e.printStackTrace(System.out);
}
}
}
The results are:
is.toString() returned:
java.io.BufferedInputStream#804a77
IOUtils.toString(is, "UTF-8") returned:
SELECT ##VERSION
Excuse any wrong practices as I am very new to threading. I have a program that calls my api and gets data back in json format. Each request returns a row of data in json format. All together I need to retrieve about 2,000,000 rows a day which means 2,000,000 requests (I understand that this is bad design, but the system was not designed for this purpose it is just what I need to do for the next couple of weeks). When I tried running it on a single thread I was processing about 200 requests a minute which is much too slow. As a result I created 12 threads and I was processing 5500 rows a minutes which was a great improvement. The problem was only about on average 90% of the rows were inserted into the database as I ran it a few times to make sure. Before each insert printed to a file each URL which was sent and then I checked to see if each insert statement was successful (returned 1 when executed ) and it all seems fine. Every time I run it it inserts about 90% but it does varies and it has never been a consistent number. Am I doing something wrong inside my java code? Essentially the code starts in main by creating 12 threads. Each thread's creates a run method which calls a new instance of MySQLPopulateHistData and passes a start and end integer which are used in the insert statement for ranges. I have done many system.out.println type testing and can see all the threads do start and all the 12 instances (one instance for each thread) called are executing? Does anyone have any idea what it could be?
MAIN:
import java.io.IOException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class MainClass {
public static void main(String[] args) {
try {
//create a pool of threads
Thread[] threads = new Thread[12];
// submit jobs to be executing by the pool
for (int i = 0; i <12; i++) {
threads[i] = new Thread(new Runnable() {
public void run() {
try {
new MySQLPopulateHistData(RangeClass.IdStart, RangeClass.IdEnd);
} catch (Throwable e) {
//TODO Auto-generated catch block
e.printStackTrace();
}
}
});
threads[i].start();
Thread.sleep(1000);
RangeClass.IdStart = RangeClass.IdEnd + 1;
RangeClass.IdEnd = RangeClass.IdEnd + 170000;
}
} catch (Throwable e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
MyDataSourceFactory.class
import java.io.FileInputStream;
import java.io.IOException;
import java.util.Properties;
import javax.sql.DataSource;
import com.mysql.jdbc.jdbc2.optional.MysqlDataSource;
public class MyDataSourceFactory {
static String url = "jdbc:mysql://localhost:3306/my_schema";
static String userName = "root";
static String password = "password";
public synchronized static DataSource getMySQLDataSource() {
MysqlDataSource mysqlDS = null;
mysqlDS = new MysqlDataSource();
mysqlDS.setURL(url);
mysqlDS.setUser(userName);
mysqlDS.setPassword(password);
return mysqlDS;
}
}
MySQLPopulateHistData.class
public class MySQLPopulateHistData {
public MySQLPopulateHistData(int s, int e ) throws IOException, Throwable{
getHistory(s,e);
}
public synchronized void getHistory(int start, int end){
DataSource ds = MyDataSourceFactory.getMySQLDataSource();
Connection con = null;
Connection con2 = null;
Statement stmt = null;
Statement stmt2 = null;
ResultSet rs = null;
try {
con = ds.getConnection();
con2 = ds.getConnection();
stmt = con.createStatement();
stmt2 = con.createStatement();
rs = stmt.executeQuery("SELECT s FROM sp_t where s_id BETWEEN "+ start +" AND "+ end + " ORDER BY s;");
String s = "";
while(rs.next()){
s = rs.getString("s");
if( s == ""){
}
else{
try{
URL fullUrl = new URL(//My Url to my api with password with start and end range);
InputStream is = fullUrl.openStream();
String jsonStr = getStringFromInputStream(is);
JSONObject j = new JSONObject(jsonStr);
JSONArray arr = j.getJSONObject("query").getJSONObject("results").getJSONArray("quote");
for(int i=0; i<arr.length(); i++){
JSONObject obj = arr.getJSONObject(i);
String symbol = obj.getString("s");
stmt2.executeUpdate("INSERT into sp2_t(s) VALUES ('"+ s +"') BETWEEN "+start+" AND "+ end +";");
}
}
catch(Exception e){
}
}
s = "";
}
} catch (Exception e) {
e.printStackTrace();
}finally{
try {
if(rs != null) rs.close();
if(stmt != null) stmt.close();
if(con != null) con.close();
if(stmt2 != null) stmt.close();
if(con2 != null) con.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
UPDATE:
So I put:
(if s.equals("")){
System.out.println("EMPTY");
}
and it never printed out EMPTY. After the JSON requests gets converted to the JSONArray I added:
if(arr.length()>0){
StaticClassHolder.cntResponses++;
}
This is just a static variable in another class that gets incremented everytime there is a valid JSON response. It equalled to the exact right amount it was supposed to be. So it seems as if the URL gets all the responses properly, parses them properly, but is not INSERTING them properly into the database? I can't figure out why?
I also faced the similar issue while inserting records in Oracle. Since I didn't find any concrete solution. I tried with single thread and all went fine.
There are several reasons why this does not work:
A normal computer can only handle about 4-8 threads in total per cpu. As the system uses some of thise threads you would only be able to run some threads at the same time. The computer handles this by pausing some threads then running another thread.
If you try to send several queries through the socket to the mysql server at the same time chanses are that some of the requests will not work and you lose some of your data.
As for now I do not have any solution for faster updates of the table.
I have a cvs file which schema is, every field is surrounded with ", and seperated by , and every tuple is a newline with \n
So in my Java file, I wrote
String path = "o.csv";
String esquel = " LOAD DATA LOCAL INFILE " + path +
" INTO TABLE recommendations " +
" FIELDS TERMINATED BY \',\' ENCLOSED BY \'\"'" +
" LINES TERMINATED BY \'\\n\'";
And I execute the statement with the following statement
statement.executeUpdate(esquel);
But it throws an SQLException which says:
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an
error in your SQL syntax; check the manual that corresponds to your
MySQL server version for the right syntax to use near 'o.csv INTO
TABLE recommendations FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES'
at line 1
What is my error ?
I would be appreciate if you can help me.
Thanks
Here is working code which I tested:
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
public class LoadTRPLog2MySql {
public static void main(String[] args) {
Class driver_class = null;
try {
driver_class = Class.forName("com.mysql.jdbc.Driver");
} catch (ClassNotFoundException e) {
e.printStackTrace();
return;
}
System.out.println("found driver" + driver_class);
Connection connection = null;
try {
connection = DriverManager.getConnection("jdbc:mysql://mysqlserver.com:3306/dbname", "myid","pwd");
} catch (SQLException e) {
e.printStackTrace();
}
try {
System.out.println("Established connection to " + connection.getMetaData().getURL());
} catch (SQLException e1) {
e1.printStackTrace();
}
Statement statement = null;
try {
statement = connection.createStatement();
Statement statement1 = connection.createStatement();
//windows
//statement1.executeUpdate( "LOAD DATA LOCAL INFILE 'C:\\Users\\senthil_sivasamy\\Documents\\Projects\\messageprocessing\\log.txt' INTO TABLE trpwatchlog_tb FIELDS TERMINATED BY ' ' LINES TERMINATED BY '\\n'");
//linux ( " LOAD DATA LOCAL INFILE '/home/username/logname.log' INTO TABLE logname.log FIELDS TERMINATED BY ' ' LINES TERMINATED BY '\\n'");
statement.executeUpdate( "LOAD DATA LOCAL INFILE '/home/username/avail30trplog' INTO TABLE logname.log FIELDS TERMINATED BY ' ' LINES TERMINATED BY '\\n'");
statement1.execute("select * from dbname.tablelog_tb");
ResultSet rs = statement1.getResultSet();
System.out.println("Row hostname and timestamp");
while(rs.next()) {
System.out.println("Row hostname and timestamp");
System.out.println(rs.getRow());
System.out.println(""+rs.getString("hostname"));
System.out.println(""+rs.getString("timestamp"));
}
rs.close();
} catch(SQLException e) {
} finally {
if (statement != null) {
try {
statement.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
}
if (connection != null) {
try {
connection.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
}
}
Oh I got it ! I didn't surround my path file with '.
New sql statement should be:
String esquel = " LOAD DATA LOCAL INFILE '" + path +
"' INTO TABLE recommendations " +
" FIELDS TERMINATED BY \',\' ENCLOSED BY \'\"'" +
" LINES TERMINATED BY \'\\n\'";