I'm familiar with the java.sql.DatabaseMetaData interface, but I find it quite clunky to use. For example, in order to find out the table names, you have to call getTables and loop through the returned ResultSet, using well-known literals as the column names.
Is there an easier way to obtain database metadata?
It's easily done using DdlUtils:
import javax.sql.DataSource;
import org.apache.ddlutils.Platform;
import org.apache.ddlutils.PlatformFactory;
import org.apache.ddlutils.model.Database;
import org.apache.ddlutils.platform.hsqldb.HsqlDbPlatform;
public void readMetaData(final DataSource dataSource) {
final Platform platform = PlatformFactory.createNewPlatformInstance(dataSource);
final Database database = platform.readModelFromDatabase("someName");
// Inspect the database as required; has objects like Table/Column/etc.
}
Take a look at SchemaCrawler (free and open source), which is another API designed for this purpose. Some sample SchemaCrawler code:
// Create the options
final SchemaCrawlerOptions options = new SchemaCrawlerOptions();
// Set what details are required in the schema - this affects the
// time taken to crawl the schema
options.setSchemaInfoLevel(SchemaInfoLevel.standard());
options.setShowStoredProcedures(false);
// Sorting options
options.setAlphabeticalSortForTableColumns(true);
// Get the schema definition
// (the database connection is managed outside of this code snippet)
final Database database = SchemaCrawlerUtility.getDatabase(connection, options);
for (final Catalog catalog: database.getCatalogs())
{
for (final Schema schema: catalog.getSchemas())
{
System.out.println(schema);
for (final Table table: schema.getTables())
{
System.out.print("o--> " + table);
if (table instanceof View)
{
System.out.println(" (VIEW)");
}
else
{
System.out.println();
}
for (final Column column: table.getColumns())
{
System.out.println(" o--> " + column + " (" + column.getType()
+ ")");
}
}
}
}
http://schemacrawler.sourceforge.net/
Related
I have a collection with some documents in it. And in my application I am creating this collection first and then inserting documents. Also, based on the requirement I need to truncate (delete all documents) the collection as well. Using document db java api I have written the following code for my this purpose-
DocumentClient documentClient = getConnection(masterkey, server, portNo);
List<Database> databaseList = documentClient.queryDatabases("SELECT * FROM root r WHERE r.id='" + schemaName + "'", null).getQueryIterable().toList();
DocumentCollection collection = null;
Database databaseCache = (Database)databaseList.get(0);
List<DocumentCollection> collectionList = documentClient.queryCollections(databaseCache.getSelfLink(), "SELECT * FROM root r WHERE r.id='" + collectionName + "'", null).getQueryIterable().toList();
// truncate logic
if (collectionList.size() > 0) {
collection = ((DocumentCollection) collectionList.get(0));
if (truncate) {
try {
documentClient.deleteDocument(collection.getSelfLink(), null);
} catch (DocumentClientException e) {
e.printStackTrace();
}
}
} else { // create logic
RequestOptions requestOptions = new RequestOptions();
requestOptions.setOfferType("S1");
collection = new DocumentCollection();
collection.setId(collectionName);
try {
collection = documentClient.createCollection(databaseCache.getSelfLink(), collection, requestOptions).getResource();
} catch (DocumentClientException e) {
e.printStackTrace();
}
With the above code I am able to create a new collection successfully. Also, I am able to insert documents as well in this collection. But while truncating the collection I am getting below error-
com.microsoft.azure.documentdb.DocumentClientException: The input authorization token can't serve the request. Please check that the expected payload is built as per the protocol, and check the key being used. Server used the following payload to sign: 'delete
colls
eyckqjnw0ae=
I am using Azure Document DB Java API version 1.9.5.
It will be of great help if you can point out the error in my code or if there is any other better way of truncating collection. I would really appreciate any kind of help here.
According to your description & code, I think the issue was caused by the code below.
try {
documentClient.deleteDocument(collection.getSelfLink(), null);
} catch (DocumentClientException e) {
e.printStackTrace();
}
It seems that you want to delete a document via the code above, but pass the argument documentLink with a collection link.
So if your real intention is to delete a collection, please using the method DocumentClient.deleteCollection(collectionLink, options).
I'm trying to generate recommendations using Apache Mahout while using MongoDB to create the datamodel as per the MongoDBDataModel. My code is as follows :
import java.net.UnknownHostException;
import java.util.List;
import org.apache.mahout.cf.taste.common.TasteException;
import org.apache.mahout.cf.taste.impl.model.mongodb.MongoDBDataModel;
import org.apache.mahout.cf.taste.impl.neighborhood.ThresholdUserNeighborhood;
import org.apache.mahout.cf.taste.impl.recommender.GenericItemBasedRecommender;
import org.apache.mahout.cf.taste.impl.recommender.GenericUserBasedRecommender;
import org.apache.mahout.cf.taste.impl.similarity.PearsonCorrelationSimilarity;
import org.apache.mahout.cf.taste.neighborhood.UserNeighborhood;
import org.apache.mahout.cf.taste.recommender.RecommendedItem;
import org.apache.mahout.cf.taste.recommender.UserBasedRecommender;
import org.apache.mahout.cf.taste.similarity.ItemSimilarity;
import org.apache.mahout.cf.taste.similarity.UserSimilarity;
import com.mongodb.MongoException;
public class usingMongo {
public static void main(String[] args) throws UnknownHostException, Mong oException
,TasteException {
final long startTime = System.nanoTime();
MongoDBDataModel model = new MongoDBDataModel("AdamsLaptop", 27017,
"test", "ratings100k", false, false, null);
System.out.println("connected to mongo ");
UserSimilarity UserSim = new PearsonCorrelationSimilarity(model);
UserNeighborhood neighborhood = new ThresholdUserNeighborhood(0.5, UserSim, model);
UserBasedRecommender UserRecommender = new GenericUserBasedRecommender(model, neighborhood, UserSim);
List<RecommendedItem>UserRecommendations = UserRecommender.recommend(1, 3);
for (RecommendedItem recommendation : UserRecommendations) {
System.out.println("You may like movie " + recommendation.getItemID() + " as a user similar to you also rated it " + recommendation.getValue() + " USER");
}
ItemSimilarity ItemSim = new PearsonCorrelationSimilarity(model);//LogLikelihoodSimilarity(model);
GenericItemBasedRecommender ItemRecommender = new GenericItemBasedRecommender(model, ItemSim);
List<RecommendedItem>ItemRecommendations = ItemRecommender.recommend(1, 3);
for (RecommendedItem recommendation : ItemRecommendations) {
System.out.println("You may like movie " + recommendation.getItemID() + " as a user similar to you also rated it " + recommendation.getValue() + " ITEM");
}
final long duration = System.nanoTime() - startTime;
System.out.println(duration);
}
}
I cant see where I've gone wrong but with numerous changes and lots of trial and error the error message remains the same :
Exception in thread "main" java.lang.NullPointerException
at org.apache.mahout.cf.taste.impl.model.mongodb.MongoDBDataModel.getID(MongoDBDataModel.java:743)
at org.apache.mahout.cf.taste.impl.model.mongodb.MongoDBDataModel.buildModel(MongoDBDataModel.java:570)
at org.apache.mahout.cf.taste.impl.model.mongodb.MongoDBDataModel.<init>(MongoDBDataModel.java:245)
at recommender.usingMongo.main(usingMongo.java:24)
Any suggestions? Here's an example of my data within MongoDB :
{ "_id" : ObjectId("56ddf61f5960960c333f3dcb"),"userId" : 1, "movieId" : 292, "rating" : 4, "timestamp" : 847116936 }
I succesfully integrated MongoDB data to mahout.
The structure of the data in mongoDB depends on the kind of Similarity algorithm you use.for eg,
UserSimilarity
MongoDBDataModel datamodel = new MongoDBDataModel("127.0.0.1", 27017, "testing", "ratings", true, true, null);
where the user_id, item_id are integer values, preference are float values and created_at as timestamp
SVDRecommender
the user_id, item_id are MongoDB Objects and preference are float values and created_at as timestamp
The obvious troubleshooting you can do is whether the MongoDB server is running or not. As per the exception it's running. I think the problem lies in your structure of data..
Use user_id instead of userId, item_id instead of itemId, preference instead of rating. I don't know if this will make any difference. I used one of the tutorial online, but can't find it at the moment.
It's working but too slow when I have more than 10000 users with 1000 items.
I think that the problem is that mahout assumes some default values when it comes to some fields that need to reside in your mongoDB the item ID, User ID and preferences that are user_id, item_id and preference so The solution might lie on using another MongoDBDataModel constructor that will give you the possibility to pass as parameters the names of those fields in your mongoDB instance or redesign your Collections Schema.
I hope that makes sense.
I'm using Dynamicloud and something is wrong in my code.
This is my code:
DynamicProvider<ContactForm> provider = new DynamicProviderImpl<ContactForm>(new RecordCredential(csk, aci));
contactFormModel.setBoundClass(ContactForm.class);
Query<ContactForm> query = provider.createQuery(contactFormModel);
try {
RecordResults<ContactForm> list = query.add(Conditions.like("contact.namef", "ProBusiness%")).list();
System.out.println("list = " + list.getFastReturnedSize());
if (list.getFastReturnedSize() > 0) {
System.out.println("Contact Name = " + list.getRecords().get(0).getName());
System.out.println("Contact Comments = " + list.getRecords().get(0).getComments());
}
} catch (DynamicloudProviderException e) {
e.printStackTrace();
}
This code throws the following Exception:
org.dynamicloud.exception.DynamicloudProviderException: Invalid statement. Please check aliases, field identifiers, projections and query conditions.
What's wrong?
Thanks!
Hi
The problems is the contact word right before namef in your like condition.
Dynamicloud API provides a method to set the alias of a model. For example, the aliases are used in situations where you need to execute JOINS
The problem is solved by calling the method setAlias from the query object query.setAlias("contact")
query.setAlias("contact");
Dynamicloud resources about Query class
Hope this helps
I am attempting to write a simple java utility that extracts data from SAP into a MySQL database, using JCo. I have understood the JCo documentation and tried out the relevant examples mentioned in SAP help portal, I am able to retrieve data from Table and insert into MySQL DB.
What I would like to have is a facility to filter data in following two ways :
I would like to fetch only the required fields.
I would like to fetch rows only if the value of the a particular field matches certain pattern.
After doing some research I didn't find any way to specify query parameters so that it retrieves only the filtered data, it basically queries all the fields from a Table, I think I will have to filter out the data that I don't want in my java-client layer. Please let me know if I am missing out something here.
Here is a code example :
public static void readTables() throws JCoException, IOException {
final JCoDestination destination = JCoDestinationManager
.getDestination(DESTINATION_NAME2);
final JCoFunction function = destination.getRepository().getFunction(
"RFC_READ_TABLE");
function.getImportParameterList().setValue("QUERY_TABLE", "DD02L");
function.getImportParameterList().setValue("DELIMITER", ",");
if (function == null) {
throw new RuntimeException("BAPI RFC_READ_TABLE not found in SAP.");
}
try {
function.execute(destination);
} catch (final AbapException e) {
System.out.println(e.toString());
return;
}
final JCoTable codes = function.getTableParameterList().getTable(
"FIELDS");
String header = "SN";
for (int i = 0; i < codes.getNumRows(); i++) {
codes.setRow(i);
header += "," + codes.getString("FIELDNAME");
}
final FileWriter outFile = new FileWriter("out.csv");
outFile.write(header + "\n");
final JCoTable rows = function.getTableParameterList().getTable("DATA");
for (int i = 0; i < rows.getNumRows(); i++) {
rows.setRow(i);
outFile.write(i + "," + rows.getString("WA") + "\n");
outFile.flush();
}
outFile.close();
}
This method tries to read a table where SAP stores meta data or data dictionary and writes the output to a csv file. This works fine but takes 30-40 secs and returns around 4 hundred thousand records with 32 columns. My intention was to ask if there is a way I can restrict my query to return only a particular field, instead of reading all the fields and discarding them in the client layer.
Thanks.
This works fine :
JCoTable table = function.getTableParameterList().getTable("FIELDS");
table.appendRow();
table.setValue("FIELDNAME", "TABNAME");
table.appendRow();
table.setValue("FIELDNAME", "TABCLASS");
Please check this Thread
Thanks.
What is the fastest way to check if Hbase table exists? Looking at this api :
http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html
Which of these is the fastest :
tableExists
isTableEnabled
isTableAvailable
listTables
With #4 you get list of all tables and iterate trough it and compare if one of those tables matches your table name.
Or there is another, more smart way ?
Here is my sample code. (scala)
import org.apache.hadoop.hbase.HBaseConfiguration
var TableName = "sample"
val conf = HBaseConfiguration.create()
var hbaseAdmin = new HBaseAdmin(conf)
if (!hbaseAdmin.tableExists(TableName)) {
println(TableName + " Does Not Exist")
}
Here, you just need to use "tableExists" to check whether this TableName exists.
HBaseAdmin hba = new HBaseAdmin(hbaseTemplate.getConfiguration());
if (hba.tableExists(tableName) == false) {
HTableDescriptor tableDescriptor = new HTableDescriptor(tableName);
HColumnDescriptor columnDescriptor = new HColumnDescriptor(columnFamilyProfile);
tableDescriptor.addFamily(columnDescriptor);
hba.createTable(tableDescriptor);
}
Using HBaseAdmin.tableExists only takes about 500ms to check if the table exists. We only have two nodes in our cluster, so it might be dependent on the size of your cluster, but it doesn't seem unreasonably slow.
You could attempt to open an HTable to the table and (I think) it will throw an exception/error (not at work yet so can't do a quick test) if the table doesn't exist.
Not 100% this will work, just an off the top of the head idea. :)
I have to check if table exist every time i start my app. I have made this in a configuration class, with spring boot
Here is the code, hope it helps.
#Configuration
public class CustomHbaseConfiguration {
#Bean
public Connection hbaseConnection() throws IOException {
// Create connection
final org.apache.hadoop.conf.Configuration configuration = HBaseConfiguration.create();
// Validate that Hbase is available
HBaseAdmin.available(configuration);
// return the hbaseConnection Bean
return ConnectionFactory.createConnection(configuration);
}
#PostConstruct
public void hbaseTableLogic() throws IOException {
// With the hbaseConnection bean, get the HbaseAdmin instance
Admin admin = hbaseConnection().getAdmin();
// The name of my table
TableName YOUR_TABLE_NAME_HERE = TableName.valueOf("PUT_YOUR_TABLE_NAME_HERE");
// Check if the table already exists ? else : create table and colum family
if (!admin.tableExists(YOUR_TABLE_NAME_HERE)) {
HTableDescriptor hTableDescriptor = new HTableDescriptor(YOUR_TABLE_NAME_HERE);
hTableDescriptor.addFamily(new HColumnDescriptor("PUT_YOUR_COLUM_FAMILY_HERE"));
admin.createTable(hTableDescriptor);
}
}
}