I want to load a gzipped rdf file into a org.eclipse.rdf4j.repository.Repository. During the upload, status messages must be logged to the console. The size of my rdf file is ~1GB of uncompressed or ~50MB of compressed data.
Actually an RDF4J repository will automatically process a compressed (zip/gzip) file correctly, already. So you can simply do this:
RepositoryConnection conn = ... ; // your store connection
conn.add(new File("file.zip"), null, RDFFormat.NTRIPLES):
If you want to include reporting, a different (somewhat simpler) approach is to use an org.eclipse.rdf4j.repository.util.RDFLoader class in combination with an RDFInserter:
RepositoryConnection conn = ... ; // your store connection
RDFInsert inserter = new RDFInserter(conn);
RDFLoader loader = new RDFLoader(conn.getParserConfig(), conn.getValueFactory());
loader.load(new File("file.zip"), RDFFormat.NTRIPLES, inserter));
The RDFLoader takes care of properly uncompressing the (zip or gzip) file.
To get intermediate reporting you can wrap your RDFInserter in your own custom AbstractRDFHandler that does the counting and reporting (before passing on to the wrapper inserter).
Variant 1
The following sample will load an InputStream with gzipped data into an in-memory rdf repository. The zipped format is supported directly by rdf4j.
Every 100000th statement will be printed to stdout using the RepositoryConnectionListenerAdapter.
import java.io.InputStream;
import org.eclipse.rdf4j.model.IRI;
import org.eclipse.rdf4j.model.Resource;
import org.eclipse.rdf4j.model.Value;
import org.eclipse.rdf4j.repository.Repository;
import org.eclipse.rdf4j.repository.RepositoryConnection;
import org.eclipse.rdf4j.repository.event.base.NotifyingRepositoryConnectionWrapper;
import org.eclipse.rdf4j.repository.event.base.RepositoryConnectionListenerAdapter;
import org.eclipse.rdf4j.repository.sail.SailRepository;
import org.eclipse.rdf4j.rio.RDFFormat;
import org.eclipse.rdf4j.sail.memory.MemoryStore;
public class MyTripleStore {
Repository repo;
/**
* Creates an inmemory triple store
*
*/
public MyTripleStore() {
repo = new SailRepository(new MemoryStore());
repo.initialize();
}
/**
* #param in gzip compressed data on an inputstream
* #param format the format of the streamed data
*/
public void loadZippedFile(InputStream in, RDFFormat format) {
System.out.println("Load zip file of format " + format);
try (NotifyingRepositoryConnectionWrapper con =
new NotifyingRepositoryConnectionWrapper(repo, repo.getConnection());) {
RepositoryConnectionListenerAdapter myListener =
new RepositoryConnectionListenerAdapter() {
private long count = 0;
#Override
public void add(RepositoryConnection arg0, Resource arg1, IRI arg2,
Value arg3, Resource... arg4) {
count++;
if (count % 100000 == 0)
System.out.println("Add statement number " + count + "\n"
+ arg1+ " " + arg2 + " " + arg3);
}
};
con.addRepositoryConnectionListener(myListener);
con.add(in, "", format);
} catch (Exception e) {
throw new RuntimeException(e);
}
}
}
Variant 2
This variant implements an AbstractRDFHandler to provide the reporting.
import java.io.InputStream;
import org.eclipse.rdf4j.model.Statement;
import org.eclipse.rdf4j.repository.Repository;
import org.eclipse.rdf4j.repository.RepositoryConnection;
import org.eclipse.rdf4j.repository.sail.SailRepository;
import org.eclipse.rdf4j.repository.util.RDFInserter;
import org.eclipse.rdf4j.repository.util.RDFLoader;
import org.eclipse.rdf4j.rio.RDFFormat;
import org.eclipse.rdf4j.rio.helpers.AbstractRDFHandler;
import org.eclipse.rdf4j.sail.memory.MemoryStore;
public class MyTripleStore {
Repository repo;
/**
* Creates an inmemory triple store
*
*/
public MyTripleStore() {
repo = new SailRepository(new MemoryStore());
repo.initialize();
}
/**
* #param in gzip compressed data on an inputstream
* #param format the format of the streamed data
*/
public void loadZippedFile1(InputStream in, RDFFormat format) {
try (RepositoryConnection con = repo.getConnection()) {
MyRdfInserter inserter = new MyRdfInserter(con);
RDFLoader loader =
new RDFLoader(con.getParserConfig(), con.getValueFactory());
loader.load(in, "", RDFFormat.NTRIPLES, inserter);
} catch (Exception e) {
throw new RuntimeException(e);
}
}
class MyRdfInserter extends AbstractRDFHandler {
RDFInserter rdfInserter;
int count = 0;
public MyRdfInserter(RepositoryConnection con) {
rdfInserter = new RDFInserter(con);
}
#Override
public void handleStatement(Statement st) {
count++;
if (count % 100000 == 0)
System.out.println("Add statement number " + count + "\n"
+ st.getSubject().stringValue() + " "
+ st.getPredicate().stringValue() + " "
+ st.getObject().stringValue());
rdfInserter.handleStatement(st);
}
}
}
Here is, how to call the code
MyTripleStore ts = new MyTripleStore();
ts.loadZippedFile(new FileInputStream("your-ntriples-zipped.gz"),
RDFFormat.NTRIPLES);
Related
#JasonPlutext,
Hi Jason! I tried the above code but it just replaces an totally the image deleting the whole template.
I would like to just replace/add a particular relationship of the image ,say
<Relationship Id="rId8" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/image" Target="../media/image10.png"/>
in place of rId8 i would like to replace rId7 image.
My Source Code:
public static void main(String[] args) throws Exception {
String inputfilepath = "C:\\Users\\saranyac\\QUERIES\\Estimation\\PPT-PSR\\PSR_Dev0ps\\PSRAutomationTemplate.pptx";
PresentationMLPackage presentationMLPackage = (PresentationMLPackage)OpcPackage.load(new java.io.File(inputfilepath));
MainPresentationPart pp = presentationMLPackage.getMainPresentationPart();
SlidePart slidePart = presentationMLPackage.getMainPresentationPart().getSlide(0);
SlideLayoutPart layoutPart = slidePart.getSlideLayoutPart();
System.out.println("SlidePart Name:::::"+slidePart.getPartName().getName());
String layoutName = layoutPart.getJaxbElement().getCSld().getName();
System.out.println("layout: " + layoutPart.getPartName().getName() + " with cSld/#name='" + layoutName + "'");
System.out.println("Master: " + layoutPart.getSlideMasterPart().getPartName().getName());
System.out.println("layoutPart.getContents()::::::::s: " + layoutPart.getContents());
//layoutPart.setContents( (SldLayout)XmlUtils.unmarshalString(SAMPLE_PICTURE, Context.jcPML));
// Add image part
File file = new File("C:\\Users\\saranyac\\PPT-PSR\\PSR_Dev0ps\\ppt\\media\\image10.png" );
BinaryPartAbstractImage imagePart
= BinaryPartAbstractImage.createImagePart(presentationMLPackage, slidePart, file);
Relationship rel = pp.getRelationshipsPart().getRelationshipByID("rId8");
System.out.println("Relationship:::::::s: " +imagePart.getSourceRelationship().getId());
// pp.removeSlide(rel);
java.util.HashMap<String, String>mappings = new java.util.HashMap<String, String>();
mappings.put("rId8", imagePart.getSourceRelationship().getId());
String outputfilepath = "C:\\Work\\24Jan2018_CheckOut\\PPT-TRAILS\\Success.pptx";
//presentationMLPackage.save(new java.io.File(outputfilepath));
SaveToZipFile saver = new SaveToZipFile(presentationMLPackage);
saver.save(outputfilepath);
System.out.println("\n\n done .. saved " + outputfilepath);
}
Please help me how to replace an image in the generated PPT.
With Regards,
Saranya
See https://github.com/plutext/docx4j/blob/master/src/samples/pptx4j/org/pptx4j/samples/TemplateReplaceSimple.java (just added):
package org.pptx4j.samples;
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import javax.xml.bind.JAXBException;
import org.apache.commons.io.FileUtils;
import org.docx4j.TraversalUtil;
import org.docx4j.TraversalUtil.CallbackImpl;
import org.docx4j.dml.CTBlip;
import org.docx4j.dml.CTBlipFillProperties;
import org.docx4j.openpackaging.exceptions.Docx4JException;
import org.docx4j.openpackaging.packages.OpcPackage;
import org.docx4j.openpackaging.packages.PresentationMLPackage;
import org.docx4j.openpackaging.parts.Part;
import org.docx4j.openpackaging.parts.PresentationML.SlidePart;
import org.docx4j.openpackaging.parts.WordprocessingML.BinaryPartAbstractImage;
import org.pptx4j.Pptx4jException;
/**
* Example of how to replace text and images in a Pptx.
*
* Text is replaced using the familiar VariableReplace approach.
*
* Images are replaced by replacing their byte content.
*
* #author jharrop
*
*/
public class TemplateReplaceSimple {
public static void main(String[] args) throws Docx4JException, Pptx4jException, JAXBException, IOException {
// Input file
String inputfilepath = System.getProperty("user.dir") + "/sample-docs/pptx/image.pptx";
// String replacements
HashMap<String, String> mappings = new HashMap<String, String>();
mappings.put("colour", "green");
// Image replacements
List<ImageReplacementDetails> imageReplacements = new ArrayList<ImageReplacementDetails>();
ImageReplacementDetails example1 = new ImageReplacementDetails();
example1.slideIndex = 0;
example1.imageRelId = "rId2";
example1.replacementImageBytes = FileUtils.readFileToByteArray(new File("test.png"));
imageReplacements.add(example1);
PresentationMLPackage presentationMLPackage =
(PresentationMLPackage)OpcPackage.load(new java.io.File(inputfilepath));
// First, the text replacements
List<SlidePart> slideParts=
presentationMLPackage.getMainPresentationPart().getSlideParts();
for (SlidePart slidePart : slideParts) {
slidePart.variableReplace(mappings);
}
// Second, the image replacements.
// We have a design choice here.
// Either we can replace text placeholders with images,
// or we can replace existing images with new images, but keep the XML specifying size etc
// Here I opt for the latter, so what we need is the relId and image bytes.
for( ImageReplacementDetails ird : imageReplacements) {
// its a bit inefficient to potentially traverse a single slide
// multiple times, but I've done it this way to keep this example simple
SlidePart slidePart=
presentationMLPackage.getMainPresentationPart().getSlide(ird.slideIndex);
SlidePicFinder traverser = new SlidePicFinder();
new TraversalUtil(slidePart.getJaxbElement().getCSld().getSpTree().getSpOrGrpSpOrGraphicFrame(), traverser);
for(org.pptx4j.pml.Pic pic : traverser.pics) {
CTBlipFillProperties blipFill = pic.getBlipFill();
if (blipFill!=null) {
CTBlip blip = blipFill.getBlip();
if (blip.getEmbed()!=null) {
String relId = blip.getEmbed();
// is this the one we want?
if (relId.equals(ird.imageRelId)) {
Part part = slidePart.getRelationshipsPart().getPart(relId);
try {
BinaryPartAbstractImage imagePart = (BinaryPartAbstractImage)part;
// you'll need to ensure that you replace like with like,
// ie png for png, not eg jpeg for png!
imagePart.setBinaryData(ird.replacementImageBytes);
} catch (ClassCastException cce) {
System.out.println(part.getClass().getName());
}
} else {
System.out.println(relId + " isn't a match for this replacement. ");
}
} else {
System.out.println("No a:blip/#r:embed");
}
}
}
}
System.out.println("\n\n saving .. \n\n");
String outputfilepath = System.getProperty("user.dir") + "/OUT_VariableReplace.pptx";
presentationMLPackage.save(new java.io.File(outputfilepath));
System.out.println("\n\n done .. \n\n");
}
static class ImageReplacementDetails {
int slideIndex;
String imageRelId;
byte[] replacementImageBytes;
}
static class SlidePicFinder extends CallbackImpl {
List<org.pptx4j.pml.Pic> pics = new ArrayList<org.pptx4j.pml.Pic>();
public List<Object> apply(Object o) {
if (o instanceof org.pptx4j.pml.Pic) {
pics.add((org.pptx4j.pml.Pic) o);
System.out.println("added pic");
}
return null;
}
}
}
I'm creating an app for schoolproject that reads data from firebase database, converts it to a .csv file and then I want to upload this file to firebase storage so that the user then can share it with just the downloadUrl.
Below is the class for creating a csvfile and then upload it to firebase storage.
see csvUploader.
import android.content.Context;
import android.net.Uri;
import android.support.annotation.NonNull;
import android.util.Log;
import android.widget.Toast;
import com.google.android.gms.tasks.OnFailureListener;
import com.google.android.gms.tasks.OnSuccessListener;
import com.google.firebase.storage.CancellableTask;
import com.google.firebase.storage.FirebaseStorage;
import com.google.firebase.storage.OnProgressListener;
import com.google.firebase.storage.StorageMetadata;
import com.google.firebase.storage.StorageReference;
import com.google.firebase.storage.UploadTask;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
public class CsvHandler {
private MainActivity mainActivity;
public CsvHandler(MainActivity mainActivity) {
this.mainActivity = mainActivity;
}
/**
* Method that writes a two-dimensional array with strings, to a .csv-file with a specified
* date as the filename.
*
* #param dataArray array to write to a .csv
* #param callDate specified date that gets passed to the filename
*/
public void writeFileFromArray(String callDate, String[][] dataArray) {
String filename = callDate + ".csv";
//Creates the String which will make up the text for the .csv
String csvText = "";
//Adds all elements in Array to the string
//TODO: Make sure this parses the text correctly to .csv-file format (dependent on Sara & Annies method)
for (int i = 0; i < dataArray.length; i++) {
for (int j = 0; j < dataArray[0].length; j++) {
csvText = csvText + dataArray[i][j];
}
}
//Creates a FileOutputStream for writing the file to internal storage
FileOutputStream outputStream;
try {
//Opens a FileOutputStream to a file with the specified filename.
//Creates file if it doesn't exist.
outputStream = mainActivity.openFileOutput(filename, Context.MODE_PRIVATE);
//Writes the string to the specified file
outputStream.write(csvText.getBytes());
//Closes the FileOutputStream to produce a file
outputStream.close();
} catch (FileNotFoundException e) {
Toast.makeText(mainActivity, "Internal Error: No such file found", Toast.LENGTH_SHORT).show();
} catch (IOException e) {
Toast.makeText(mainActivity, "Internal Error: IOException", Toast.LENGTH_SHORT).show();
}
}
/**
* TESTMETOD
* TODO: Ta bort innan merge med master. Låt stå till develop
*/
public void readCsvFile(String callDate) {
try {
String Message;
FileInputStream fileInputStream = mainActivity.openFileInput(callDate + ".csv");
InputStreamReader inputStreamReader = new InputStreamReader(fileInputStream);
BufferedReader bufferedReader = new BufferedReader(inputStreamReader);
StringBuffer stringBuffer = new StringBuffer();
while ((Message = bufferedReader.readLine()) != null) {
stringBuffer.append(Message + "\n");
}
Toast.makeText(mainActivity, stringBuffer.toString(), Toast.LENGTH_SHORT).show();
} catch (Exception e) {
e.printStackTrace();
}
}
/**
* Method to extract a filePath for a specified date.
*
* #param callDate a String with the date to return a filepath for
* #return the filepath for the specified date
*/
public String getFilePath(String callDate) {
String filePath = mainActivity.getFilesDir().getAbsolutePath() + callDate + ".csv";
Log.e("LOG", "Output from getFilePath " + filePath);
return filePath;
}
public void csvUploader(String filePath, final String callDate) {
StorageReference mStorageReference = FirebaseStorage.getInstance().getReference();
Log.e("LOG", "Entering CSVUPLOADER");
Uri file = Uri.fromFile(new File(filePath));
Log.e("csvUploader Uri File:", filePath.toString());
// Create the file metadata
StorageMetadata metadata = new StorageMetadata.Builder().setContentType("text/csv").build();
Log.e("LOG","Metadata: " + metadata.toString());
// Upload file and metadata to the path 'reports/date.csv'
CancellableTask uploadTask = mStorageReference.child("reports/" + file.getLastPathSegment()).putFile(file, metadata);
uploadTask.addOnProgressListener(new OnProgressListener<UploadTask.TaskSnapshot>() {
#Override
public void onProgress(UploadTask.TaskSnapshot taskSnapshot) {
double progress = (100.0 * taskSnapshot.getBytesTransferred()) / taskSnapshot.getTotalByteCount();
//System.out.println("Upload is " + progress + "% done");
}
}).addOnFailureListener(new OnFailureListener() {
#Override
public void onFailure(#NonNull Exception exception) {
// Handle unsuccessful uploads
Log.e("LOG", "Unsucessfull in CSVUPLOADER");
}
}).addOnSuccessListener(new OnSuccessListener<UploadTask.TaskSnapshot>() {
#Override
public void onSuccess(UploadTask.TaskSnapshot taskSnapshot) {
// Handle successful uploads on complete
//Uri downloadUrl = taskSnapshot.getMetadata().getDownloadUrl();
//mainActivity.setDownloadLink(downloadUrl);
Log.e("LOG", "Successfull in CSVUPLOADER");
mainActivity.getUrlAsync(callDate);
}
});
}
}
Most of this are the example code from firebase.google.com But i don't get it to work.
Logs:
E/LOG: Output from getFilePath: /data/user/0/com.example.eliasvensson.busify/files2016-05-18.csv
E/LOG: Entering CSVUPLOADER
E/LOG: Metadata: com.google.firebase.storage.StorageMetadata#7fd93a9
E/LOG: Unsucessfull in CSVUPLOADER
Whats wrong?
I gather that I "reserve" the path on my storage bucket, and then place the file in that place. Is that correct?
Any help would be appreciated.
It all actually almost worked.
the filePath was wrong, outputing /data/user/0/com.example.eliasvensson.busify/files2016-05-18.csv
when expecting
/data/user/0/com.example.eliasvensson.busify/files/2016-05-18.csv (notice the slash between files and date)
I've been pondering this for a fair amount of time now. I'm trying to download the data from Yahoo!'s Stock API. When you use the API, it gives you a .csv file. I've been looking at opencsv, which seems perfect, except I want to avoid downloading and saving the file, if at all possible.
OpenCSV, according to the examples, can only read from a FileReader. According to Oracle's docs on FileReader, the file needs to be local.
Is it possible to read from a remote file using OpenCSV without downloading?
CSVReader takes a Reader argument according to the documentation, so it isn't limited to a FileReader for the parameter.
To use a CSVReader without saving the file first, you could use a BufferedReader around a stream loading the data:
URL stockURL = new URL("http://example.com/stock.csv");
BufferedReader in = new BufferedReader(new InputStreamReader(stockURL.openStream()));
CSVReader reader = new CSVReader(in);
// use reader
Implementation of opencsv for reading csv file and saving to database.
import com.opencsv.CSVParser;
import com.opencsv.CSVParserBuilder;
import com.opencsv.CSVReader;
import com.opencsv.CSVReaderBuilder;
import com.opencsv.bean.CsvBindByPosition;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import javax.persistence.Column;
import java.io.*;
import java.lang.annotation.Annotation;
import java.lang.reflect.Field;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
#Service
#Slf4j
public class FileUploadService {
#Autowired
private InsertCSVContentToDB csvContentToDB;
/**
* #param csvFileName location of the physical file.
* #param type Employee.class
* #param delimiter can be , | # etc
* #param obj //new Employee();
*
* import com.opencsv.bean.CsvBindByPosition;
* import lombok.Data;
*
* import javax.persistence.Column;
*
* #Data
* public class Employee {
*
* #CsvBindByPosition(position = 0, required = true)
* #Column(name = "EMPLOYEE_NAME")
* private String employeeName;
*
* #CsvBindByPosition(position = 1)
* #Column(name = "Employee_ADDRESS_1")
* private String employeeAddress1;
* }
*
* #param sqlQuery query to save data to DB
* #param noOfLineSkip make it 0(Zero) so that it should not skip any line.
* #param auditId apart from regular column in csv we need to add more column for traking like file id or audit id
* #return
*/
public <T> void readCSVContentInArray(String csvFileName, Class<? extends T> type, char delimiter, Object obj,
String sqlQuery, int noOfLineSkip, Long auditId) {
List<T> lstCsvContent = new ArrayList<>();
Reader reader = null;
CSVReader csv = null;
try {
reader = new BufferedReader(new InputStreamReader(new FileInputStream(csvFileName), "utf-8"));
log.info("Buffer Reader : " + ((BufferedReader) reader).readLine().isEmpty());
CSVParser parser = new CSVParserBuilder().withSeparator(delimiter).withIgnoreQuotations(true).build();
csv = new CSVReaderBuilder(reader).withSkipLines(noOfLineSkip).withCSVParser(parser).build();
String[] nextLine;
int size = 0;
int chunkSize = 10000;
Class params[] = { Long.class };
Object paramsObj[] = { auditId };
long rowNumber = 0;
Field field[] = type.getDeclaredFields();
while ((nextLine = csv.readNext()) != null) {
rowNumber++;
try {
obj = type.newInstance();
for (Field f : field) {
if(!f.isSynthetic()){
f.setAccessible(true);
Annotation ann[] = f.getDeclaredAnnotations();
CsvBindByPosition csv1 = (CsvBindByPosition) ann[0];
Column c = (Column)ann[1];
try {
if (csv1.position() < nextLine.length) {
if (csv1.required() && (nextLine[csv1.position()] == null
|| nextLine[csv1.position()].trim().isEmpty())) {
String message = "Mandatory field is missing in row: " + rowNumber;
log.info("null value in " + rowNumber + ", " + csv1.position());
System.out.println(message);
}
if (f.getType().equals(String.class)) {
f.set(obj, nextLine[csv1.position()]);
}
if (f.getType().equals(Boolean.class)) {
f.set(obj, nextLine[csv1.position()]);
}
if (f.getType().equals(Integer.class)) {
f.set(obj, Integer.parseInt(nextLine[csv1.position()]));
}
if (f.getType().equals(Long.class)) {
f.set(obj, Long.parseLong(nextLine[csv1.position()]));
}
if (f.getType().equals(Double.class) && null!=nextLine[csv1.position()] && !nextLine[csv1.position()].trim().isEmpty() ) {
f.set(obj, Double.parseDouble(nextLine[csv1.position()]));
}if(f.getType().equals(Double.class) && ((nextLine[csv1.position()]==null) || nextLine[csv1.position()].isEmpty())){
f.set(obj, new Double("0.0"));
}
if (f.getType().equals(Date.class)) {
f.set(obj, nextLine[csv1.position()]);
}
}
} catch (Exception fttEx) {
log.info("Exception when parsing the file: " + fttEx.getMessage());
System.out.println(fttEx.getMessage());
}
}
}
lstCsvContent.add((T) obj);
if (lstCsvContent.size() > chunkSize) {
size = size + lstCsvContent.size();
//write code to save to data base of file system in chunk.
lstCsvContent = null;
lstCsvContent = new ArrayList<>();
}
} catch (Exception ex) {
log.info("Exception: " + ex.getMessage());
}
}
//write code to save list into DB or file system
System.out.println(lstCsvContent);
} catch (Exception ex) {
log.info("Exception:::::::: " + ex.getMessage());
} finally {
try {
if (csv != null) {
csv.close();
}
if (reader != null) {
reader.close();
}
} catch (IOException ioe) {
log.info("Exception when closing the file: " + ioe.getMessage());
}
}
log.info("File Processed successfully: ");
}
}
Is it possible to do file/directory sync in Java using JSch ? I need to sync directory from a remote linux machine to my local windows machine. Is this possible ?
-Tivakar
The easiest way to download files from SCP server is using Commons VFS along with JSch:
import java.io.*;
import org.apache.commons.io.FileUtils;
import org.apache.commons.vfs2.*;
public class CopyRemoteFile {
public static void copyRemoteFiles(String host, String user, String remotePath, String localPath) throws IOException {
FileSystemOptions fsOptions = new FileSystemOptions();
SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(fsOptions, "no");
SftpFileSystemConfigBuilder.getInstance().setIdentities(fsOptions,
new File[] { new File(FileUtils.getUserDirectoryPath() + "/.ssh/id_dsa") });
DefaultFileSystemManager fsManager = (DefaultFileSystemManager) VFS.getManager();
String uri = "sftp://" + user + "#" + host + "/" + remotePath;
FileObject fo = fsManager.resolveFile(uri, fsOptions);
FileObject[] files = fo.getChildren();
for (FileObject file : files) {
// We will be dealing with the files here only
if (file.getType() == FileType.FILE) {
FileUtils.copyInputStreamToFile(file.getContent().getInputStream(),
new File(localPath + "/" + file.getName().getBaseName()));
}
file.close();
}
fo.close();
fsManager.close();
}
}
It's just an example I got in my Wiki, so nothing fancy. But do keep in mind that if you'll close fsManager, you will not be able to open it again in the same VM. I got this issue while testing this solution...
Although the example above does not import any JSch classes, you need to put it in the classpath anyway.
The above example is using private key to authenticate with the remote host. You can easily change that by providing password and modifying the uri to include that.
If you need to sync files, you can compare dates of the files on the local file system (or DB, or any other source of the information) and the remote files:
import java.io.*;
import org.apache.commons.io.*;
import org.apache.commons.vfs2.*;
import org.apache.commons.vfs2.impl.*;
import org.apache.commons.vfs2.provider.sftp.*;
public class CopyRemoteFile {
public static void copyRemoteFiles(final String host, final String user, final String remotePath, final String localPath)
throws IOException {
FileSystemOptions fsOptions = new FileSystemOptions();
SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(fsOptions, "no");
SftpFileSystemConfigBuilder.getInstance().setIdentities(fsOptions,
new File[] { new File(FileUtils.getUserDirectoryPath() + "/.ssh/id_dsa") });
DefaultFileSystemManager fsManager = (DefaultFileSystemManager) VFS.getManager();
String uri = "sftp://" + user + "#" + host + "/" + remotePath;
FileObject fo = fsManager.resolveFile(uri, fsOptions);
FileObject[] files = fo.getChildren();
for (FileObject file : files) {
// We will be dealing with the files here only
File newFile = new File(localPath + "/" + file.getName().getBaseName());
if (file.getType() == FileType.FILE && newFile.lastModified() != file.getContent().getLastModifiedTime()) {
FileUtils.copyInputStreamToFile(file.getContent().getInputStream(), newFile);
newFile.setLastModified(file.getContent().getLastModifiedTime());
}
file.close();
}
fo.close();
fsManager.close();
}
}
Look at: http://the-project.net16.net/Projekte/projekte/Projekte/Programmieren/sftp-synchronisierung.html
There is a whole Programm uploadet.
Here is the sync Part:
import java.io.File;
import java.io.FileNotFoundException;
import java.util.ArrayList;
import java.util.Vector;
import com.jcraft.jsch.ChannelSftp.LsEntry;
import com.jcraft.jsch.SftpException;
/*
* This is the heart of the whole Program. I hope, the descriptions are precise enought.
*/
public class Sync{
public String ServerPath;
public File LocalFolder;
public sFTPclient client;
public ArrayList<String> serverContentList;
public ArrayList<String> pathList;
public Sync(File local, String to, sFTPclient client){
this.LocalFolder = local;
this.ServerPath = to;
this.client = client;
}
/*
* Executed once. Sets the Server Directory if it exists.
* If the local folder doesn't exist on the Server, it creates it.
*/
public void setServerDirectory() throws SftpException{
try{
client.sftpChannel.cd(ServerPath);
}catch(Exception e){
GUI.addToConsole(ServerPath + " don't exist on your server!");
}
String serverFolder = ServerPath.substring(ServerPath.lastIndexOf('/')+1, ServerPath.length());
if(!LocalFolder.getName().equals(serverFolder)){
try{
client.sftpChannel.mkdir(LocalFolder.getName());
client.sftpChannel.cd(LocalFolder.getName());
} catch (Exception e){
client.sftpChannel.cd(LocalFolder.getName());
}
this.ServerPath = ServerPath + "/" + LocalFolder.getName();
GUI.setNewServerFolder(ServerPath);
}
serverContentList = new ArrayList<String>();
pathList = new ArrayList<String>();
}
//The contentlist contains all Filenames, that should be synchronized
public void setToContentList(String ServerFolder) throws SftpException{
#SuppressWarnings("unchecked")
Vector<LsEntry> fileList = client.sftpChannel.ls(ServerFolder);
int size = fileList.size();
for(int i = 0; i < size; i++){
if(!fileList.get(i).getFilename().startsWith(".")){
serverContentList.add(fileList.get(i).getFilename());
pathList.add(ServerFolder);
}
}
}
/*
* Deletes the synchronized elements from the Lists
*/
public void deleteFromLists(String name){
int position = serverContentList.lastIndexOf(name);
if(position >= 0){
serverContentList.remove(position);
pathList.remove(position);
}
}
/*
* Main function for synchronizing. Works recursive for local folders.
*/
#SuppressWarnings("unchecked")
public void synchronize(File localFolder, String ServerDir) throws SftpException, FileNotFoundException{
if(client.sftpChannel.pwd() != ServerDir){
client.sftpChannel.cd(ServerDir);
}
setToContentList(ServerDir);
File[] localList = localFolder.listFiles();
Vector<LsEntry> ServerList = client.sftpChannel.ls(ServerDir);
ServerList.remove(0); ServerList.remove(0);
/*
* Upload missing Files/Folders
*/
int size = localList.length;
for(int i = 0; i < size; i++){
if(localList[i].isDirectory()){
if(checkFolder(localList[i], ServerDir)){
synchronize(localList[i], ServerDir + "/" + localList[i].getName());
deleteFromLists("SubFolder");
}else {
newFileMaster(true, localList[i], ServerDir);
}
} else {
checkFile(localList[i], ServerDir);
}
deleteFromLists(localList[i].getName());
}
}
/*
* Deletes all files on the server, which are not in the local Folder. Deletes also all missing folders
*/
public void deleteRest() throws SftpException, FileNotFoundException{
int size = serverContentList.size();
for(int i = 0; i < size; i++){
client.sftpChannel.cd(pathList.get(i));
newFileMaster(false, null, serverContentList.get(i));
}
}
/*
* Copy or delete Files/Folders
*/
public void newFileMaster(boolean copyOrNot, File sourcePath, String destPath) throws FileNotFoundException, SftpException{
FileMaster copy = new FileMaster(copyOrNot, sourcePath, destPath, client.sftpChannel);
copy.runMaster();
}
/*
*Useful to find errors - Prints out the content-List every time you call the method.
*If you have Problems, call it before and after every changes of the serverContentList!
*/
/*public void printServerContent(){
System.out.println("SERVER-Content: " + "\n");
for(int i = 0; i < serverContentList.size(); i++){
System.out.println(serverContentList.get(i) + " in " + pathList.get(i));
}
}*/
/*
* Looks ond the server, if the file is there. If not, or the local file has changed, it copies the file on the server.
*/
public void checkFile(File file, String path) throws SftpException, FileNotFoundException{
client.sftpChannel.cd(path);
if(!serverContentList.contains(file.getName())){
newFileMaster(true, file, ServerPath);
} else {
Long localTimeStamp = file.lastModified();
Long timeStamp = client.sftpChannel.stat(file.getName()).getATime()*1000L;
if(localTimeStamp > timeStamp){
newFileMaster(false, null, path + "/" + file.getName());
newFileMaster(true, file, path);
}
}
deleteFromLists(file.getName());
}
/*
* The same as the checkFile function. But it returns a boolean. (Easier to handle in the synchronized funtion)
* Don't check, if the folder has changed (I think this can't be the case)
*/
public boolean checkFolder(File folder, String path) throws SftpException{
client.sftpChannel.cd(path);
if(serverContentList.contains(folder.getName())){
return true;
}else { return false; }
}
}
I have a requirement, for example, there will be number of .txt files in one loation c:\onelocation. I want to write the content to another location in txt format. This part is pretty easy and straight forward. But there is speed breaker here.
There will be time interval take 120 seconds. Read the files from above location and write it to another files with formate txt till 120secs and save the file with name as timestamp.
After 120sec create one more files with that timestamp but we have to read the files were cursor left in previous file.
Please can you suggest any ideas, if code is provided that would be also appreciable.
Thanks Damu.
How about this? A writer that automatically changes where it is writing two every 120 seconds.
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.Writer;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;
public class TimeBoxedWriter extends Writer {
private static DateFormat FORMAT = new SimpleDateFormat("yyyyDDDHHmm");
/** Milliseconds to each time box */
private static final int TIME_BOX = 120000;
/** For testing only */
public static void main(String[] args) throws IOException {
Writer w = new TimeBoxedWriter(new File("."), "test");
// write one line per second for 500 seconds.
for(int i = 0;i < 500;i++) {
w.write("testing " + i + "\n");
try {
Thread.sleep(1000);
} catch (InterruptedException ie) {}
}
w.close();
}
/** Output folder */
private File dir_;
/** Timestamp for current file */
private long stamp_ = 0;
/** Stem for output files */
private String stem_;
/** Current output writer */
private Writer writer_ = null;
/**
* Create new output writer
*
* #param dir
* the output folder
* #param stem
* the stem used to generate file names
*/
public TimeBoxedWriter(File dir, String stem) {
dir_ = dir;
stem_ = stem;
}
#Override
public void close() throws IOException {
synchronized (lock) {
if (writer_ != null) {
writer_.close();
writer_ = null;
}
}
}
#Override
public void flush() throws IOException {
synchronized (lock) {
if (writer_ != null) writer_.flush();
}
}
private void rollover() throws IOException {
synchronized (lock) {
long now = System.currentTimeMillis();
if ((stamp_ + TIME_BOX) < now) {
if (writer_ != null) {
writer_.flush();
writer_.close();
}
stamp_ = TIME_BOX * (System.currentTimeMillis() / TIME_BOX);
String time = FORMAT.format(new Date(stamp_));
writer_ = new FileWriter(new File(dir_, stem_ + "." + time
+ ".txt"));
}
}
}
#Override
public void write(char[] cbuf, int off, int len) throws IOException {
synchronized (lock) {
rollover();
writer_.write(cbuf, off, len);
}
}
}
Use RamdomAccessFile in java to move the cursor within the file.
Before start copying check the file modification/creation(in case of new files) time, if less than 2 mins then only start copying or else skip it.
Keep a counter of no.of bytes/lines read for each file. move the cursor to that position and read it from there.
You can duplicate the file rather than using reading and writing operations.
sample code:
FileChannel ic = new FileInputStream("<source file location>")).getChannel();
FileChannel oc = new FileOutputStream("<destination location>").getChannel();
ic.transferTo(0, ic.size(), oc);
ic.close();
oc.close();
HTH
File io is simple in java, here is an example I found on the web of copying a file to another file.
import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
public class Copy {
public static void main(String[] args) throws IOException {
File inputFile = new File("farrago.txt");
File outputFile = new File("outagain.txt");
FileReader in = new FileReader(inputFile);
FileWriter out = new FileWriter(outputFile);
int c;
while ((c = in.read()) != -1)
out.write(c);
in.close();
out.close();
}
}