i was trying to upload an RDF/OWL file to my Sparql endpoint (given by Fuseki). Right now i'm able to upload a single file, but if i try to repeat the action, the new dataset will override the old one. I'm searching a way to "merge" the content of the data in the dataset with the new ones of the rdf file just uploaded. Anyone can help me? thanks.
Following the code to upload/query the endpoint (i'm not the author)
// Written in 2015 by Thilo Planz
// To the extent possible under law, I have dedicated all copyright and related and neighboring rights
// to this software to the public domain worldwide. This software is distributed without any warranty.
// http://creativecommons.org/publicdomain/zero/1.0/
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.ByteArrayOutputStream;
import org.apache.jena.query.DatasetAccessor;
import org.apache.jena.query.DatasetAccessorFactory;
import org.apache.jena.query.QueryExecution;
import org.apache.jena.query.QueryExecutionFactory;
import org.apache.jena.query.QuerySolution;
import org.apache.jena.query.ResultSet;
import org.apache.jena.query.ResultSetFormatter;
import org.apache.jena.rdf.model.Model;
import org.apache.jena.rdf.model.ModelFactory;
import org.apache.jena.rdf.model.RDFNode;
class FusekiExample {
public static void uploadRDF(File rdf, String serviceURI)
throws IOException {
// parse the file
Model m = ModelFactory.createDefaultModel();
try (FileInputStream in = new FileInputStream(rdf)) {
m.read(in, null, "RDF/XML");
}
// upload the resulting model
DatasetAccessor accessor = DatasetAccessorFactory.createHTTP(serviceURI);
accessor.putModel(m);
}
public static void execSelectAndPrint(String serviceURI, String query) {
QueryExecution q = QueryExecutionFactory.sparqlService(serviceURI,
query);
ResultSet results = q.execSelect();
// write to a ByteArrayOutputStream
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
//convert to JSON format
ResultSetFormatter.outputAsJSON(outputStream, results);
//turn json to string
String json = new String(outputStream.toByteArray());
//print json string
System.out.println(json);
}
public static void execSelectAndProcess(String serviceURI, String query) {
QueryExecution q = QueryExecutionFactory.sparqlService(serviceURI,
query);
ResultSet results = q.execSelect();
while (results.hasNext()) {
QuerySolution soln = results.nextSolution();
// assumes that you have an "?x" in your query
RDFNode x = soln.get("x");
System.out.println(x);
}
}
public static void main(String argv[]) throws IOException {
// uploadRDF(new File("test.rdf"), );
uploadRDF(new File("test.rdf"), "http://localhost:3030/MyEndpoint/data");
}
}
Use accessor.add(m) instead of putModel(m). As you can see in the Javadoc, putModel replaces the existing data.
Related
When I tried to give the source URI of a folder inside my bucket (which has around 400 CSV files) in the Java program, it has not moved any files to BQ table. If I try with a single csv file , it moves.
package com.example.bigquerydatatransfer;
import com.google.api.gax.rpc.ApiException;
import com.google.cloud.bigquery.datatransfer.v1.CreateTransferConfigRequest;
import com.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient;
import com.google.cloud.bigquery.datatransfer.v1.ProjectName;
import com.google.cloud.bigquery.datatransfer.v1.TransferConfig;
import com.google.protobuf.Struct;
import com.google.protobuf.Value;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
// Sample to create google cloud storage transfer config
public class Cloud_to_BQ {
public static void main(String[] args) throws IOException {
final String projectId = "dfp-bq";
String datasetId = "mytest1";
String tableId = "PROG_DATA";
String sourceUri = "gs://dfp-bq/C:\\PROG_Reports";
String fileFormat = "CSV";
String fieldDelimiter = ",";
String skipLeadingRows = "1";
Map<String, Value> params = new HashMap<>();
params.put(
"destination_table_name_template", Value.newBuilder().setStringValue(tableId).build());
params.put("data_path_template", Value.newBuilder().setStringValue(sourceUri).build());
params.put("write_disposition", Value.newBuilder().setStringValue("APPEND").build());
params.put("file_format", Value.newBuilder().setStringValue(fileFormat).build());
params.put("field_delimiter", Value.newBuilder().setStringValue(fieldDelimiter).build());
params.put("skip_leading_rows", Value.newBuilder().setStringValue(skipLeadingRows).build());
TransferConfig transferConfig =
TransferConfig.newBuilder()
.setDestinationDatasetId(datasetId)
.setDisplayName("Trial_Run_PROG_DataTransfer")
.setDataSourceId("google_cloud_storage")
.setParams(Struct.newBuilder().putAllFields(params).build())
.setSchedule("every 24 hours")
.build();
createCloudStorageTransfer(projectId, transferConfig);
}
public static void createCloudStorageTransfer(String projectId, TransferConfig transferConfig)
throws IOException {
try (DataTransferServiceClient client = DataTransferServiceClient.create()) {
ProjectName parent = ProjectName.of(projectId);
CreateTransferConfigRequest request =
CreateTransferConfigRequest.newBuilder()
.setParent(parent.toString())
.setTransferConfig(transferConfig)
.build();
TransferConfig config = client.createTransferConfig(request);
System.out.println("Cloud storage transfer created successfully :" + config.getName());
} catch (ApiException ex) {
System.out.print("Cloud storage transfer was not created." + ex.toString());
}
}
}
Is there any way I can move all the files to the BQ table at a stretch?
2022-08-04T07:27:50.185847509ZNo files found matching: "gs://dfp-bq/C:\PROG_Reports" - This is the BQ logs for the Run.
How is arbitrary JSON converted to arbitrary XML using BaseX?
I'm looking at JsonParser from BaseX for this specific solution.
In this case, I have tweets using Twitter4J:
package twitterBaseX;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Properties;
import java.util.logging.Logger;
import main.LoadProps;
import org.basex.core.BaseXException;
import twitter4j.JSONException;
import twitter4j.JSONObject;
import twitter4j.Query;
import twitter4j.QueryResult;
import twitter4j.Status;
import twitter4j.Twitter;
import twitter4j.TwitterException;
import twitter4j.TwitterFactory;
import twitter4j.TwitterObjectFactory;
import twitter4j.conf.ConfigurationBuilder;
public class TwitterOps {
private static final Logger log = Logger.getLogger(TwitterOps.class.getName());
public TwitterOps() {
}
private TwitterFactory configTwitterFactory() throws IOException {
LoadProps loadTwitterProps = new LoadProps("twitter");
Properties properties = loadTwitterProps.loadProperties();
log.fine(properties.toString());
ConfigurationBuilder configurationBuilder = new ConfigurationBuilder();
configurationBuilder.setDebugEnabled(true)
.setJSONStoreEnabled(true)
.setOAuthConsumerKey(properties.getProperty("oAuthConsumerKey"))
.setOAuthConsumerSecret(properties.getProperty("oAuthConsumerSecret"))
.setOAuthAccessToken(properties.getProperty("oAuthAccessToken"))
.setOAuthAccessTokenSecret(properties.getProperty("oAuthAccessTokenSecret"));
return new TwitterFactory(configurationBuilder.build());
}
public List<JSONObject> getTweets() throws TwitterException, IOException, JSONException {
Twitter twitter = configTwitterFactory().getInstance();
Query query = new Query("lizardbill");
QueryResult result = twitter.search(query);
String string = null;
JSONObject tweet = null;
List<JSONObject> tweets = new ArrayList<>();
for (Status status : result.getTweets()) {
tweet = jsonOps(status);
tweets.add(tweet);
}
return tweets;
}
private JSONObject jsonOps(Status status) throws JSONException, BaseXException {
String string = TwitterObjectFactory.getRawJSON(status);
JSONObject json = new JSONObject(string);
String language = json.getString("lang");
log.fine(language);
return json;
}
}
The JSONObject from Twitter4J cannot just get jammed into XML?
There are a number of online converters which purport to accomplish this, and, which, at least at first glance, seem quite adequate.
see also:
Converting JSON to XML in Java
Java implementation of JSON to XML conversion
Use the (excellent) JSON-Java library from json.org then
JSONObject json = new JSONObject(str);
String xml = XML.toString(json);
toString can take a second argument to provide the name of the XML root node.
This library is also able to convert XML to JSON using XML.toJSONObject(java.lang.String string)
Check the Javadoc for more information
I am using the java2word library for generating a word document from IBM Notes Database data.
My problem is that the document I am getting is interpreted as containing errors by ms word and only recoverable by text.
When I click the "Go to" button on the Word Repair pop up window (once I opened my doc in recovery) nothing happens and from the dialog, I can't tell anything at all. (in German)
I have Successfully used 2 other libraries with my Agent Class so I am pretty sure it can't be the data Collection and parsing to my Document Writing class.
The following Code runs successfully.
DataRow Class used for temporarily storing data:
public class DataRow {
String Date;
String VorgangDesc;
String DayShort;
double Hours;
public DataRow(String Dayshort, String Vorgangdesc, double hours, String date1){
Date=date1;
VorgangDesc=Vorgangdesc;
DayShort=Dayshort;
Hours=hours;
}
}
BerichtsHeft Class used for implementing java2word:
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.ArrayList;
import lotus.domino.Session;
import word.api.interfaces.IDocument;
import word.api.interfaces.IElement;
import word.utils.TestUtils;
import word.utils.Utils;
import word.w2004.Document2004;
import word.w2004.Document2004.Encoding;
import word.w2004.elements.BreakLine;
import word.w2004.elements.Table;
import word.w2004.elements.tableElements.TableEle;
public class BerichtsHeft {
public String Name;
public String startD;
public Session CurrentS;
public int TableCount=1;
public String Abteilung;
public int AusbildungsJahr;
String[] ItemsLastRow = new String[] {"..." , "...", "..."};
String[] ItemsFirstRow = new String[] {"Ausbildungsnachweis", "Nr." + TableCount, "Woche vom" + startD + "bis" + "e end"};
PrintWriter writer = null;
Table CurrentTable;
IDocument myDoc;
public BerichtsHeft(String strName, String startDate, Session CurrentSes, String abteilung){
this.Name=strName;
this.startD=startDate;
this.CurrentS=CurrentSes;
this.Abteilung=abteilung;
this.myDoc = new Document2004();
myDoc.encoding(Encoding.UTF_8);
}
public void Spacer(){
myDoc.addEle(BreakLine.times(1).create());
}
public void createTable(ArrayList<DataRow> DataList){
Table tbl = new Table();
CurrentTable = tbl;
String[] ItemsFlexible = new String[3];
AddFirstRow(ItemsFirstRow);
for(int ij2=0; ij2<DataList.size(); ij2++){
ItemsFlexible[0]=DataList.get(ij2).DayShort.toString();
ItemsFlexible[1]=DataList.get(ij2).VorgangDesc.toString();
ItemsFlexible[2]=Double.toString(DataList.get(ij2).Hours);
AddRow(ItemsFlexible);
}
AddLastRow(ItemsLastRow);
myDoc.addEle(CurrentTable);
TableCount++;
Spacer();
}
public void AddFirstRow(String[] Items){
CurrentTable.addTableEle(TableEle.TH, Items);
}
public void AddRow(String[] items){
CurrentTable.addTableEle(TableEle.TD, items);
}
public void AddLastRow(String[] items){
CurrentTable.addTableEle(TableEle.TD, items);
}
public void logNext(){
}
public void SaveDoc(){
File fileObj = new File("C:\\temp\\test2.doc");
PrintWriter writer = null;
try {
writer = new PrintWriter(fileObj);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
String myWord = myDoc.getContent();
writer.println(myWord);
writer.close();
}
}
Pastebin link to the text formated word document
PasteBin link to text Word Doc
What is the best technique to find the origin of these errors?
I found the Error in the Text document, it was caused by a special character, because the encoding was UTF_8 and not ISO8859_1.
I am using the recent Solr 4.2.1 solrj libraries.
I am trying to execute an MLT Query from a java program. It works fine as long as I only provide small chunks in the stream.body, but that kind of defeats my purpose.
When I try to use the ContentStream I don't get a response back, when I do the solr.query, it makes another request.
It looks like the server is getting my solr.request() ok. Appreciate any pointers.
Oh, and I am talking to a solr 3.6.1
Here is what I have so far:
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.response.QueryResponse;
import org.apache.solr.common.SolrDocumentList;
import org.apache.solr.common.params.ModifiableSolrParams;
import org.apache.solr.common.util.ContentStream;
import org.apache.solr.common.util.ContentStreamBase;
import org.apache.solr.common.util.NamedList;
import org.apache.solr.client.solrj.*;
import org.apache.solr.client.solrj.impl.HttpSolrServer;
import org.apache.solr.common.*;
import java.io.DataInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.net.MalformedURLException;
import org.apache.solr.client.solrj.request.AbstractUpdateRequest;
import org.apache.solr.client.solrj.request.ContentStreamUpdateRequest;
import org.apache.solr.client.solrj.util.ClientUtils;
public class SolrJSearcher {
public static void main(String[] args) throws MalformedURLException, SolrServerException {
HttpSolrServer solr = new HttpSolrServer("http://localhost:8983/solr");
ModifiableSolrParams params = new ModifiableSolrParams();
String mltv[] = {"Big bunch of text for testing - redacted for brevity"};
String dvalues[] = {"mlt"};
String svalues[] = {"0"};
ContentStreamUpdateRequest up = new ContentStreamUpdateRequest("/mlt");
ContentStream cs = new ContentStreamBase.StringStream(mltv[0]);
up.addContentStream( cs);
SolrQuery theQuery = new SolrQuery();;
theQuery.set("qt", dvalues);
up.setParam("start", "0");
try {
solr.request(up);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
QueryResponse response = solr.query(theQuery);
SolrDocumentList results = response.getResults();
for (int i = 0; i < results.size(); ++i) {
System.out.println(results.get(i));
}
}
}
As far as I know, MoreLikeThis is meant to find documents similar to a document already in the index. If you're searching documents similar to an input string, then just insert a temporary item in your index before you do the query, and remove it afterwards.
I've been using the following successfully:
/*
* Build up a MoreLikeThis query to retrieve documents
* similar to the one with id originalId
*/
private SolrQuery buildUpMoreLikeThisQuery(String field3, String originalId) {
SolrQuery query = new SolrQuery();
query.setQueryType("/" + MoreLikeThisParams.MLT);
query.set(MoreLikeThisParams.MATCH_INCLUDE, true);
query.set(MoreLikeThisParams.MIN_DOC_FREQ, 1);
query.set(MoreLikeThisParams.MIN_TERM_FREQ, 1);
query.set(MoreLikeThisParams.MIN_WORD_LEN, 7);
query.set(MoreLikeThisParams.BOOST, false);
query.set(MoreLikeThisParams.MAX_QUERY_TERMS, 1000);
query.set(MoreLikeThisParams.SIMILARITY_FIELDS,
"field1,field2");
query.setQuery("id:" + originalId);
query.set("fl", "id,score");
query.addFilterQuery("field3:" + field3);
int maxResults = 20;
query.setRows(maxResults);
return query;
}
I have been playing with Amazon's Product Advertising API, and I cannot get a request to go through and give me data. I have been working off of this: http://docs.amazonwebservices.com/AWSECommerceService/2011-08-01/GSG/ and this: Amazon Product Advertising API signed request with Java
Here is my code. I generated the SOAP bindings using this: http://docs.amazonwebservices.com/AWSECommerceService/2011-08-01/GSG/YourDevelopmentEnvironment.html#Java
On the Classpath, I only have: commons-codec.1.5.jar
import com.ECS.client.jax.AWSECommerceService;
import com.ECS.client.jax.AWSECommerceServicePortType;
import com.ECS.client.jax.Item;
import com.ECS.client.jax.ItemLookup;
import com.ECS.client.jax.ItemLookupRequest;
import com.ECS.client.jax.ItemLookupResponse;
import com.ECS.client.jax.ItemSearchResponse;
import com.ECS.client.jax.Items;
public class Client {
public static void main(String[] args) {
String secretKey = <my-secret-key>;
String awsKey = <my-aws-key>;
System.out.println("API Test started");
AWSECommerceService service = new AWSECommerceService();
service.setHandlerResolver(new AwsHandlerResolver(
secretKey)); // important
AWSECommerceServicePortType port = service.getAWSECommerceServicePort();
// Get the operation object:
com.ECS.client.jax.ItemSearchRequest itemRequest = new com.ECS.client.jax.ItemSearchRequest();
// Fill in the request object:
itemRequest.setSearchIndex("Books");
itemRequest.setKeywords("Star Wars");
// itemRequest.setVersion("2011-08-01");
com.ECS.client.jax.ItemSearch ItemElement = new com.ECS.client.jax.ItemSearch();
ItemElement.setAWSAccessKeyId(awsKey);
ItemElement.getRequest().add(itemRequest);
// Call the Web service operation and store the response
// in the response object:
com.ECS.client.jax.ItemSearchResponse response = port
.itemSearch(ItemElement);
String r = response.toString();
System.out.println("response: " + r);
for (Items itemList : response.getItems()) {
System.out.println(itemList);
for (Item item : itemList.getItem()) {
System.out.println(item);
}
}
System.out.println("API Test stopped");
}
}
Here is what I get back.. I was hoping to see some Star Wars books available on Amazon dumped out to my console :-/:
API Test started
response: com.ECS.client.jax.ItemSearchResponse#7a6769ea
com.ECS.client.jax.Items#1b5ac06e
API Test stopped
What am I doing wrong (Note that no "item" in the second for loop is being printed out, because its empty)? How can I troubleshoot this or get relevant error information?
I don't use the SOAP API but your Bounty requirements didn't state that it had to use SOAP only that you wanted to call Amazon and get results. So, I'll post this working example using the REST API which will at least fulfill your stated requirements:
I would like some working example code that hits the amazon server and returns results
You'll need to download the following to fulfill the signature requirements:
http://associates-amazon.s3.amazonaws.com/signed-requests/samples/amazon-product-advt-api-sample-java-query.zip
Unzip it and grab the com.amazon.advertising.api.sample.SignedRequestsHelper.java file and put it directly into your project. This code is used to sign the request.
You'll also need to download Apache Commons Codec 1.3 from the following and add it to your classpath i.e. add it to your project's library. Note that this is the only version of Codec that will work with the above class (SignedRequestsHelper)
http://archive.apache.org/dist/commons/codec/binaries/commons-codec-1.3.zip
Now you can copy and paste the following making sure to replace your.pkg.here with the proper package name and replace the SECRET and the KEY properties:
package your.pkg.here;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.StringWriter;
import java.util.HashMap;
import java.util.Map;
import java.util.Properties;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.transform.OutputKeys;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import org.w3c.dom.Document;
import org.xml.sax.SAXException;
public class Main {
private static final String SECRET_KEY = "<YOUR_SECRET_KEY>";
private static final String AWS_KEY = "<YOUR_KEY>";
public static void main(String[] args) {
SignedRequestsHelper helper = SignedRequestsHelper.getInstance("ecs.amazonaws.com", AWS_KEY, SECRET_KEY);
Map<String, String> params = new HashMap<String, String>();
params.put("Service", "AWSECommerceService");
params.put("Version", "2009-03-31");
params.put("Operation", "ItemLookup");
params.put("ItemId", "1451648537");
params.put("ResponseGroup", "Large");
String url = helper.sign(params);
try {
Document response = getResponse(url);
printResponse(response);
} catch (Exception ex) {
Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex);
}
}
private static Document getResponse(String url) throws ParserConfigurationException, IOException, SAXException {
DocumentBuilder builder = DocumentBuilderFactory.newInstance().newDocumentBuilder();
Document doc = builder.parse(url);
return doc;
}
private static void printResponse(Document doc) throws TransformerException, FileNotFoundException {
Transformer trans = TransformerFactory.newInstance().newTransformer();
Properties props = new Properties();
props.put(OutputKeys.INDENT, "yes");
trans.setOutputProperties(props);
StreamResult res = new StreamResult(new StringWriter());
DOMSource src = new DOMSource(doc);
trans.transform(src, res);
String toString = res.getWriter().toString();
System.out.println(toString);
}
}
As you can see this is much simpler to setup and use than the SOAP API. If you don't have a specific requirement for using the SOAP API then I would highly recommend that you use the REST API instead.
One of the drawbacks of using the REST API is that the results aren't unmarshaled into objects for you. This could be remedied by creating the required classes based on the wsdl.
This ended up working (I had to add my associateTag to the request):
public class Client {
public static void main(String[] args) {
String secretKey = "<MY_SECRET_KEY>";
String awsKey = "<MY AWS KEY>";
System.out.println("API Test started");
AWSECommerceService service = new AWSECommerceService();
service.setHandlerResolver(new AwsHandlerResolver(secretKey)); // important
AWSECommerceServicePortType port = service.getAWSECommerceServicePort();
// Get the operation object:
com.ECS.client.jax.ItemSearchRequest itemRequest = new com.ECS.client.jax.ItemSearchRequest();
// Fill in the request object:
itemRequest.setSearchIndex("Books");
itemRequest.setKeywords("Star Wars");
itemRequest.getResponseGroup().add("Large");
// itemRequest.getResponseGroup().add("Images");
// itemRequest.setVersion("2011-08-01");
com.ECS.client.jax.ItemSearch ItemElement = new com.ECS.client.jax.ItemSearch();
ItemElement.setAWSAccessKeyId(awsKey);
ItemElement.setAssociateTag("th0426-20");
ItemElement.getRequest().add(itemRequest);
// Call the Web service operation and store the response
// in the response object:
com.ECS.client.jax.ItemSearchResponse response = port
.itemSearch(ItemElement);
String r = response.toString();
System.out.println("response: " + r);
for (Items itemList : response.getItems()) {
System.out.println(itemList);
for (Item itemObj : itemList.getItem()) {
System.out.println(itemObj.getItemAttributes().getTitle()); // Title
System.out.println(itemObj.getDetailPageURL()); // Amazon URL
}
}
System.out.println("API Test stopped");
}
}
It looks like the response object does not override toString(), so if it contains some sort of error response, simply printing it will not tell you what the error response is. You'll need to look at the api for what fields are returned in the response object and individually print those. Either you'll get an obvious error message or you'll have to go back to their documentation to try to figure out what is wrong.
You need to call the get methods on the Item object to retrieve its details, e.g.:
for (Item item : itemList.getItem()) {
System.out.println(item.getItemAttributes().getTitle()); //Title of item
System.out.println(item.getDetailPageURL()); // Amazon URL
//etc
}
If there are any errors you can get them by calling getErrors()
if (response.getOperationRequest().getErrors() != null) {
System.out.println(response.getOperationRequest().getErrors().getError().get(0).getMessage());
}