I need to transfer files to my web server for processing and I'd like to do it in a generic way if possible.
I need to be able to transfer files from the following protocols at a minimum (with more to follow eventually):
HTTP
FTP
SCP
I'd really like to be able to send files to SMTP also
So my question, is there a toolkit available that does this already? If so, it must be open source as this is part of an open source project.
If there isn't a toolkit that already does this, what is the best way to structure an interface that will handle most file transfers?
I've thought about something like this:
public interface FileTransfer {
public void connect(URL url, String userid, String password);
public void disconnect();
public void getFile(String sourceFile, File destFile);
public void putFile(File sourceFile, File destFile);
}
And then a Factory that takes the source URL or protocol and instantiates the correct file handler.
Apache commons VFS speaks to this problem, although a quick check didn't show that it will do SCP or SMTP. Commons NET does SMTP, but I don't know that you could get the common interface out of the box. For SCP, here are some possibilities.
The bottom line seems to be to check out the VFS implementation and see if it does something for you, perhaps you can extend it for different protocols. If it isn't appropriate, regarding your interface, you are probably going to want all remote file references to be Strings rather than File objects, and specifically a string representing a URI pointing to the remote location and telling you what protocol to use.
I'm working at a problem very similar to yours, I couldn't find any open source solution so I'm trying to sketch a solution myself. This is what I've come up with.
I think you should represent inputSources and outputSources as different things, like
public interface Input{
abstract InputStream getFileInputStream();
abstract String getStreamId();
}
//You can have differen implementation of this interface (1 for ftp, 1 for local files, 1 for Blob on db etc)
public interface Output{
abstract OutputStream getOutputStream();
abstract String getStreamId();
}
//You can have differen implementation of this interface (1 for ftp, 1 for local files, 1 for mailing the file etc)
Then you should have a Movement to describe which input should go to which output.
class Movement{
String inputId;
String outputId;
}
A class to describe the list of Movement to make.
class MovementDescriptor{
public addMovement(Movement a);
public Movement[] getAllMovements();
}
And then a class to perform the work itself.
class FileMover{
HashMap<String,Input> inputRegistry;
HashMap<String,Output> outputRegistry;
addInputToRegistry(Input a ){
inputRegistry.put(a.getId(),a);
}
addOutputToRegistry(Output a){
outputRegistry.put(a.getId(),a);
}
transferFiles(MovementDescriptor movementDescriptor){
Movement[] movements =movementDescriptor.getAllMovements();
foreach (Movement movement: movements){
//get the input Id
//find it in the registry and retrieve the associated InputStream
//get the output Id
//find it in the registry and retrieve the associated OutputStream
//copy the stream from the input to the output (you may want to use a temporary file in between)
}
}
}
The code that would use this would operate like this:
FileMover fm=new FileMover();
//Register your sources and your destinations
fm.addInputToRegistry(input);
fm.addOutputToRegistry(output)
// each time you have to make a movement create a MovementDescriptor and call
fm.transferFiles(movementDescriptor)
If you would like to exchange by mail our views on the subject, just send me an e mail at (my nickname)#gmail dot com.
NOTE: The code is just a sketch :-)
I think JSch implements SCP, so that covers that one.
please make use of JCraft . Open "sftp" channel and try that.
Related
I'm using Java 8. I want to extract the domain portion of a URL. Just in case I'm using the word "domain" incorrectly, what i want is if my server name is
test.javabits.com
I want to extract "javabits.com". Similarly, if my server name is
firstpart.secondpart.lastpart.org
I want to extract "lastpart.org". I tried the below
final String domain = request.getServerName().replaceAll(".*\\.(?=.*\\.)", "");
but its not extracting the domain properly. Then I tried what this guy has in his site -- https://www.mkyong.com/regular-expressions/domain-name-regular-expression-example/, e.g.
private static final String DOMAIN_NAME_PATTERN = "^((?!-)[A-Za-z0-9-]{1,63}(?<!-)\\.)+[A-Za-z]{2,6}$";
but that is also not extracting what I want. How can I extract the domain name portion properly?
Summary: Do not use regex for this. Use whois.
If I try to extrapolate from your question, to find out what you really want to do, I guess you want to find the domain belonging to some non-infrastructural owner from the host part of a URL. Additionally, from the tag of your question, you want to do it with the help of a regex.
The task you are undertaking is at best impractical, but probably impossible.
There are a number of corner cases that you would have to weed out. Apart from the list of infrastructural domains kindly provided by Lennart in https://publicsuffix.org/list/public_suffix_list.dat, you also have the cases of an empty host field in the URL or an IP-address forming the host part.
So, is there a better approach to this? Of course there is. What you do want to do is query a public database for the data you need. The protocol for such queries is called WHOIS.
Apache Commons provide an easy way to access WHOIS information in the WhoisClient. From there you can query the domain field, and find some more information that may be useful to you.
It shouldn't be harder than
import org.apache.commons.net.whois.WhoisClient;
import java.io.IOException;
public class CommonsTest {
public static void main(String args) {
WhoisClient c = new WhoisClient();
try {
c.connect(WhoisClient.DEFAULT_HOST);
System.out.println(c.query(URL));
c.disconnect();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Using this will get you the whois information aboutt he domain you are asking for. If the domain is uregistered, that is, is a private domain, as in the case of www.stackexchange.com you will get an error saying no domain is registered. Remove the first part of the address and try again. Once you found the registered domain, you will also find the registrar and the registrer.
Now, unfortunately, whois is not as simple as one would think. Read further on https://manpages.debian.org/jessie/whois/whois.1.en.html for an elaboration on how to use it and what information you can expect from different sources.
Also, check related questions here.
try it like this:
String parts[] = longDomain.split(".");
String domain = parts[parts.length-2] + "." + [parts.length -1];
Simple QUESTION : Are there ways to run or load java functions inside Lua?
I am trying to create a phone application that transfers files between server and client using Lua. The server uses Java while client uses Lua.
this is a lua function that receives file
function UDPClientModule.receiveFile()
local data, status
local chunks = {}
while true do
data, status = udp:receive()
print("status: ", status)
if data ~= nil then
table.insert(chunks, data)
--the filename is the last chunk to be received
if string.match(data, ".jpg") then
-- but strangely returns true
break
end
end
socket.sleep(0.5)
end
--combineAndOpenImage(t)
end
No problems so far. However, the chunks sent by the server are encapsulated in a class like this:
public class FileChunk {
private List<Data> dataList;
//functions below
}
public class Data{
private byte[] fileData;
// functions and adding file headers below
} // then UDPServer.java sends bytes of FileChunk
Because of this, packets received by the lua function are strange which also results in string.match(data, ".jpg") returning true. So I want to run java files (eg. UDPClient.java) in order to receive and decipher the chunks, instead of lua.
I don't want to change the server nor migrate the client language to java. I haven't found any resources about this so I need help.
You would need to create a wrapper library, such as the ones in C. I do not know how, but I hope this provides you a sense of direction.
I think I have an interesting question and PERHAPS there is already the answer which is still a secret for me, so I hope to get some helps from expers. :)
So here is the thing:
I work for the test/validation team to test our Java API and basically my job is to follow test plan and write the test code. After writing that for more than two months, I find the codes are really similar. For example:
To test function could return expected result or throw exception correctly, we may need write several .java to run.
1.java set up server connection, connect client and send request, initiate variables with correct values and pass them to the function A, catch the answer and analyse it
2.java set up server connection, connect client and send request, initiate all variables with correct values but one with bad value and pass them to function A, catch the answer and analyse it
3.java set up server connection, connect client and send request, initiate all variables with correct values but two with bad values and pass them to function A, catch the answer and analyse it
so you see in three java test files, the most part of them are the same or similar enough and even copy/paste make the job boring and possible to be wrong.
I wonder whether or not I could define test code corresponding different behavior, then for every test java file, I define a text including the behavior and then a mother class who is in charge of loading the text file and assembling the final test java file according to the text file?
Like this:
Text File:
1) set up server
2) connect client
3) send request
4) initiate variables with correct values
5) initiate variables with correct values but one with bad value
6) initiate variables with correct values but two with bad values
7) catch the result and analyse it
Mother.java
1) load Text file
2) create a son.java
3) find the code corresponding the Text file and write them to son.java
Then the coder open son.java at IDE to check syntax, or import or anything conflict then run it.
Is my idea realizable or not? Is there already something similar?
Any information would be appreciated, thanks a millions in advance!
Honestly, this does not sound like a good use case for code generation. Instead of generating a class for each test case, you should implement a more general testing utility which takes the required input as its data and executes the generic testing code based on this data.
From what you write, this would for example be something like a simple base class for a JUnit test:
abstract class AbstractServerDependantTest {
protected Server server;
protected Client client;
#Before
public void setUp() {
server = new Server();
server.start();
client = new Client();
client.connectTo(server);
}
#After
public void tearDpwm() {
client.disconnect();
server.shutDown();
}
}
Now you can write three test classes which inherit from this AbstractServerDependantTest without copy pasting your code.
I'm successfully using Play 1.2.4 to serve large binary file downloads to users using the renderBinary() method.
I'd like to have a hint of when the user actually completes the download. Generally speaking, I know this is somewhat possible as I've done it before. In an old version of my website, I wrote a simple servlet that served up binary file downloads. Once that servlet finished writing out the contents of the file, a notification was sent. Certainly not perfect, but useful nonetheless. In my testing, it did provide an indication of how long the user took to download a file.
Reviewing the Play source, I see that the play.mvc.results.RenderBinary class has a handy apply() method that I could use. I wrote my own version of RenderBinary so I could send the notification after the apply() method finished writing out the file contents.
The problem I found is that calls to response.out.write() obviously cache the outgoing bytes (via Netty?), so even though I am writing out several megabytes of data, the calls to play.mvc.Http.Response.out.write() complete in seconds, even though it takes the downloader a couple minutes to download the file.
I don't mind writing custom classes, although I'd prefer to use a stock Play 1.2.4 distribution.
Any ideas on how to get a notification of when the end of a file download is pushed out towards the user's browser?
It seems this may help you, as it tackles a somehow similar problem:
Detect when browser receives file download
I'm not sure you'll eb able to do it via renderBinary nor an #After annotation in the controller. Some browser-side detection of the download and then a notification to the server (pinging the download's end) would work.
There may be an alternative: using WebSockets (streaming the file via the socket and then having teh client to answer) but it may be overkill for this :)
you can use ArchivedEventStream.
first create a serializable ArcivedEventStream class..
public class Stream<String> extends ArchivedEventStream<String> implements Serializable{
public Stream(int arg0) {
super(arg0);
}
}
then on your controller...
public static void downloadPage(){
Stream<String> userStream = Cache.get(session.getId(),Stream.class);
if( userStream == null){
userStream = new Stream<String>(5);
Cache.add(session.getId(), userStream);
}
render();
}
public static void download(){
await(10000);// to provide some latency. actually no needed
renderBinary(Play.getFile("yourfile!"));
}
public static void isDownloadFinished(){
Stream<String> userStream = Cache.get(session.getId(),Stream.class);
List<IndexedEvent<String>> list = await(userStream.nextEvents(0));
renderJSON(list.get(0).data);
}
#After(only="download")
static void after(){
Stream<String> userStream = Cache.get(session.getId(),Stream.class);
userStream.publish("ok");
}
on your html...
#{extends 'main.html' /}
#{set title:'downloadPage' /}
download
<script>
$(document).ready(function(){
$.ajax('/application/isDownloadFinished',{success:function(data){
if(data){
console.log("downloadFinished");
}
}});
});
</script>
when your download finished, the original page will retrieve the notification.
This code is just a sample. You could read the api of ArchivedEventStream and make your own implementation..
I have a challenge:
Imagine you have a set of messages like this:
Code / Message
200567 = A new user was created
462001 = Unknown client number
...
I'm trying to find the neatest, lightest, and easiest to maintain way to use this messages in java.
The rules are:
You need to be able to access the message by its code
You need to be able to print the code
You need to be able to easily change the number of a code in the future
Other notes:
The messages can be in a proprieties file, in other file, in a class, or other place (?), whatever you find better.
The code can be a integer or a string (Like MSG_423456), whatever you find better.
So, anybody has ideas?
(Sorry about my lousy English)
Go for Properties File. Use ResourceBundle
200567 A new user was created
462001 Unknown client number
I would put this in an enumeration.
public enum MessageType {
NEW_USER("String 123", "A new user was created");
private String code, message;
private MessageType(String code, String message) {
this.code = code;
this.message = message
}
I would go for enumerations because they are to be checked into source control, typically in the environments where I work the properties file is meant to be configured by the individual. Such as an ant build properties file.
If I had to implemtn this I would use a HashMap. The code are the keys and the message the values.
Your best bet is probably to have a HashMap<string, string> or HashMap<int, string> that is a static member of your main class or some other relevant class. Write a properties file and a simple method that is called near the start of the program to populate the HashMap.
http://download.oracle.com/javase/6/docs/api/java/util/HashMap.html