I am using org.apache.commons.net.ftp.FTPClient in one of my applications to work with a FTP server. I am able to connect, login, pwd and cwd. However, when I try to list the files it doesn't return the list of files in that directory, where I know for sure that there are files. I am using the method FTPFile[] listFiles(), it returns an empty array of FTPFile.
Please find below the code snippet where I am trying this:
String hostname = properties.getProperty("FTP_SERVER");
String user = properties.getProperty("FTP_USER");
String passwd = properties.getProperty("FTP_PASSWD");
FTPClient client = new FTPClient();
client.connect(hostname);
client.login(user, passwd);
String reply = client.getStatus();
System.out.println(reply);
client.enterRemotePassiveMode();
client.changeWorkingDirectory("/uploads");
FTPFile[] files = client.listFiles();
System.out.println(files.length);
for (FTPFile file : files) {
System.out.println(file.getName());
}
String[] fileNames = client.listNames();
if (fileNames != null) {
for (String file : fileNames) {
System.out.println(file);
}
}
client.disconnect();
This seems like the same issue I had (and solved), see this answer:
Apache Commons Net FTPClient and listFiles()
After I set the mode as PASV it is working fine now!
Thanks for all your efforts and suggestions!
I added client.enterLocalPassiveMode() and it works:
client.connect("xxx.com");
boolean login = client.login("xxx", "xxx");
client.enterLocalPassiveMode();
Just a silly suggestion... can you do a listing on the /uploads folder using a normal FTP client. I ask this because some FTP servers are setup to not display the listing of an upload folder.
First, make sure the listing works in other programs. If so, one possibility is that the file listing isn't being parsed correctly. You can try explicitly specifying the parser to use with initiateListParsing.
I had to same problem and it turned out to be that it couldn't parse what the server was returning for a file listing. I this line after connecting to the ftp server ftpClient.setParserFactory(new MyFTPFileEntryParserFactory());
public class MyFTPFileEntryParserFactory implements FTPFileEntryParserFactory {
private final static FTPFileEntryParser parser = new UnixFTPEntryParser() {
#Override public FTPFile parseFTPEntry(String entry) {
FTPFile ftpFile = new FTPFile();
ftpFile.setTimestamp(getCalendar(entry));
ftpFile.setSize(get(entry));
ftpFile.setName(getName(entry));
return ftpFile;
}
};
#Override public FTPFileEntryParser createFileEntryParser(FTPClientConfig config) throws ParserInitializationException {
return parser;
}
#Override public FTPFileEntryParser createFileEntryParser(String key) throws ParserInitializationException {
return parser;
}
}
In my case, on top of applying enterLocalPassiveMode and indicating correct operation system, I also need to set UnparseableEntries to true to make the listFile method work.
FTPClientConfig conf = new FTPClientConfig(FTPClientConfig.SYST_UNIX);
conf.setUnparseableEntries(true);
f.configure(conf);
boolean isLoginSuccess = client.login(username, password);
Related
Currently I am unable to grab -> archive -> decrypt a file from an SFTP server. I have tested the logic using local directories but with no success using SFTP.
The connection appears to be established to the server as neglecting to pass the private key will result in a connection exception. When the key is being passed no exception is given from the route itself but no files are copied. What would be a potential solution or next steps to help in troubleshooting this issue?
I am using the absolute directory's in which the files would be stored from the sftp location.
CamelContext camelContext = new DefaultCamelContext();
camelContext.getRegistry().bind("SFTPPrivateKey",Byte[].class,privateKey.getBytes());
String sftpInput = buildURISFTP(input,inputOptions,connectionConfig);
String sfpOutput = buildURISFTP(output,outputOptions,connectionConfig);
String sfpArchive = buildURISFTP(archive,archiveOptions,connectionConfig);
camelContext.addRoutes(new RouteBuilder() {
public void configure() throws Exception {
PGPDataFormat pgpDataFormat = new PGPDataFormat();
pgpDataFormat.setKeyFileName(pPgpSecretKey);
pgpDataFormat.setKeyUserid(pgpUserId);
pgpDataFormat.setPassword(pgpPassword);
pgpDataFormat.setArmored(true);
from(sftpInput)
.to(sfpArchive);
//tested decryption local with file to file routing
.unmarshal(pgpDataFormat)
.to(sfpOutput);
}
});
camelContext.start();
Thread.sleep(timeout);
camelContext.stop();
public String buildURISFTP(String directory, String options, ConnectionConfig connectionConfig){
StringBuilder uri = new StringBuilder();
uri.append("sftp://");
uri.append(connectionConfig.getSftpHost());
uri.append(":");
uri.append(connectionConfig.getSftpPort());
uri.append(directory);
uri.append("?username=");
uri.append(connectionConfig.getSftpUser());
if(!StringUtils.isEmpty(connectionConfig.getSftpPassword())){
uri.append("&password=");
uri.append(connectionConfig.getSftpPassword());
}
uri.append("&privateKey=#SFTPPrivateKey");
if(!StringUtils.isEmpty(options)){
uri.append(options);
}
return uri.toString();
}
Issue was due to lack of knowledge around FTP component
https://camel.apache.org/components/3.18.x/ftp-component.html
Where it is specified that absolute paths are not supported, unfortunately I did not read this page and only referenced the SFTP component page where it is not specified.
https://camel.apache.org/components/3.18.x/sftp-component.html
Issue was resolved by backtracking directories with /../../ before giving the absolute path.
I am using Grizzly to serve my REST service which can have multiple "modules". I'd like to be able to use the same base URL for the service and for static content so I can access all these urls:
http://host:port/index.html
http://host:port/module1/index.html
http://host:port/module1/resource
http://host:port/module2/index.html
http://host:port/module2/resource
The code I'm trying to set this up with looks like this:
private HttpServer createServer(String host, int port, ResourceConfig config)
{
HttpServer server = GrizzlyHttpServerFactory.createHttpServer(URI.create("http://" + host + ":" + port + "/"), config, false);
HttpHandler httpHandler = new CLStaticHttpHandler(HttpServer.class.getClassLoader(), "docs/");
server.getServerConfiguration().addHttpHandler(httpHandler, "/");
return server;
}
With this code, I am only able to see the html pages and I get a "Resource identified by path does not exist" response when I try to get my resources.
When I comment out the code to add the HttpHandler, then I am able to access my resources (but don't have the docs of course).
What do I need to do to access both my resources and my static content?
I ended up writing a service to handle static resources myself. I decided to serve my files from the file system, but this approach would also work for serving them from a jar - you'd just have to get the file as a resource instead of creating the File directly.
#Path("/")
public class StaticService
{
#GET
#Path("/{docPath:.*}.{ext}")
public Response getHtml(#PathParam("docPath") String docPath, #PathParam("ext") String ext, #HeaderParam("accept") String accept)
{
File file = new File(cleanDocPath(docPath) + "." + ext);
return Response.ok(file).build();
}
#GET
#Path("{docPath:.*}")
public Response getFolder(#PathParam("docPath") String docPath)
{
File file = null;
if ("".equals(docPath) || "/".equals(docPath))
{
file = new File("index.html");
}
else
{
file = new File(cleanDocPath(docPath) + "/index.html");
}
return Response.ok(file).build();
}
private String cleanDocPath(String docPath)
{
if (docPath.startsWith("/"))
{
return docPath.substring(1);
}
else
{
return docPath;
}
}
}
One thing you can do is run Grizzly as a servlet container. That way you can run Jersey as servlet filter, and add a default servlet to handle the static content. For example
public class Main {
public static HttpServer createServer() {
WebappContext context = new WebappContext("GrizzlyContext", "");
createJerseyFilter(context);
createDefaultServlet(context);
HttpServer server = GrizzlyHttpServerFactory
.createHttpServer(URI.create("http://localhost:8080/"));
context.deploy(server);
return server;
}
private static void createJerseyFilter(WebappContext context) {
ResourceConfig rc = new ResourceConfig().packages("com.grizzly.test");
// This causes Jersey to forward 404s to default servlet
// which will catch all the static content requests.
rc.property(ServletProperties.FILTER_FORWARD_ON_404, true);
FilterRegistration reg = context.addFilter("JerseyApp", new ServletContainer(rc));
reg.addMappingForUrlPatterns(EnumSet.allOf(DispatcherType.class), "/*");
}
private static void createDefaultServlet(WebappContext context) {
ArraySet<File> baseDir = new ArraySet<>(File.class);
baseDir.add(new File("."));
ServletRegistration defaultServletReg
= context.addServlet("DefaultServlet", new DefaultServlet(baseDir) {});
defaultServletReg.addMapping("/*");
}
public static void main(String[] args) throws IOException {
HttpServer server = createServer();
System.in.read();
server.stop();
}
}
You will need to add the Jersey Grizzly servlet dependency
<dependency>
<groupId>org.glassfish.jersey.containers</groupId>
<artifactId>jersey-container-grizzly2-servlet</artifactId>
<version>${jersey2.version}</version>
</dependency>
The only problem with this approach is that the default servlet is meant to serve files from the file system, not from the classpath, as you are currently trying to do. You can see in the createDefaultServlet method I just set the base directory to the current working directory. So that's where all your files would need to be. You can change it to "docs" so all your files would be in the docs folder, which would be in the current working directory.
If you want to read files from the classpath, you may need to implement your own servlet. You can look at the source code for DefaultServlet and try to modify it to serve from the classpath. You can also check out Dropwizard's AssetServlet, which already does serve content from the classpath.
Or you can just say forget it, and just serve from the file system :-)
I am using the JSON API - Google API Client Library for Java to access the objects in Google Cloud Storage. I need to create (not upload) an empty folder in the bucket. Google Developer Web Console has that option to creating a directory, but neither the Java API nor the gsutil command has a create folder command. If anybody knows how to do so, please let me know. Thanks in advance...
You can emulate a folder by uploading a zero-sized object with a trailing slash.
As noted in the question comments, Google Cloud Storage is not a filesystem and emulating folders has serious limitations.
I think is better that you create the folder within the file name. For example if you need a folder called images and other one called docs, when you give the name of the object to upload do it in the following way images/name_of_file or docs/name_of_file.
If the name of the file is images/dogImage and you upload that file, you will find in your bucket a folder called images.
I hope to help you and others
This is my Java method to create an empty (emulated) folder:
public static void createFolder(String name) throws IOException {
Channels.newOutputStream(
createFile(name + "/")
).close();
}
public static GcsOutputChannel createFile(String name) throws IOException {
return getService().createOrReplace(
new GcsFilename(getName(), name),
GcsFileOptions.getDefaultInstance()
);
}
private static String name;
public static String getName() {
if (name == null) {
name = AppIdentityServiceFactory.getAppIdentityService().getDefaultGcsBucketName();
}
return name;
}
public static GcsService service;
public static GcsService getService() {
if (service == null) {
service = GcsServiceFactory.createGcsService(
new RetryParams.Builder()
.initialRetryDelayMillis(10)
.retryMaxAttempts(10)
.totalRetryPeriodMillis(15000)
.build());
}
return service;
}
I am trying to access a file via a URI using the FTP protocol. For obvious security reasons I had to make some changes but this is where the problems seem to be coming from.
My URI is as follows:
ftp://user:pasword#host.net/u/Bigpathname/XYZ/ABC/BigPathname/bigpathname/xyz/abc/MY_LOG.LOG
And I see this exception:
sun.net.ftp.FtpProtocolException: CWD Bigpathname:501 A qualifier in "Bigpathname" is more than 8 characters
This is really confusing as I can access the file from a Windows 7 command line with the CD command just fine. Both one directory at a time and as a full path.
I found one article mentioning that MVS file names must be 8 or fewer characters but this does not explain how I can get to these same files from my command line! They do exist there is data there that I can download manual but I can not get there via a URI in Java.
PS I use .toURL().openStream() to get files on my local machine just fine, it only fails when I try to get them from my server.
EDIT October 1st
I am able to access files on the MVS host using FileZilla and the basic FTP client from the Windows 7 command line - but I still cannot get them from a URI/URL. I downloaded a very basic Java built FTP client and tried accessing the same file in my program from there and the path works but because my file name has a dot in it "MY_LOG.LOG" I am getting File does not exist 501 Invalid data set name "MY_LOG.LOG". Use MVS Dsname conventions. I am utterly perplexed by this...
EDIT Ocotober 1st afternoon :)
OK I finally got it to work with a FTP client in my Java code - but I still want to use the URL class as I have logs on both local and remote machines. Is there a way to encode a URL string so that it can retrieve a file from a remote machine with the FTP protocol? I am not sure how it works in the Java URL class but in the FTP client I had to use the CWD and then the RETR command.
If I can do this then I have one solution for getting all my logs, otherwise I will have to detect if it is a file or ftp URL and then behave differently. Not the end of the world but not what I want...
The code that tries to get the file with just a URL is as follows: (sysc is a valid host)
void testFTP()
{
String ftp = "ftp://user:pword#sysc/u/Xxxxxxxxxx/ICS/YT7/XxxxxXxxxxxxx/xxxxxxxxx/logs/xxxxxxxx/XX_YT.LOG";
try
{
URI uri = new URI(ftp);
URL ftpFile = uri.toURL();
BufferedReader in = new BufferedReader(new InputStreamReader(ftpFile.openStream()));
String inputLine;
while ((inputLine = in.readLine()) != null)
System.out.println(inputLine);
in.close();
}
catch(Exception e)
{
e.printStackTrace();
}
}
In this case I think the problem is also Server Related, It all works fine for me with Filezilla Server except when the filename length(including directories) exceeds 255 chars but if you want to use the URL class with another FTP you must override or implement your own URLStreamHandlerFactory.
URL.setURLStreamHandlerFactory(...);
I haven't found any for my favorite java FTP Client witch is Apache one so I have developed one but may need a few touch ups.
package net.custom.streamhandler.apacheftp;
import java.io.IOException;
import java.io.InputStream;
import java.net.SocketException;
import java.net.URL;
import java.net.URLConnection;
import java.net.URLStreamHandler;
import java.net.URLStreamHandlerFactory;
import org.apache.commons.net.ftp.FTPClient;
import org.apache.commons.net.ftp.FTPReply;
public class ApacheURLStreamHandlerFactory implements URLStreamHandlerFactory {
public URLStreamHandler createURLStreamHandler(String protocol) {
//this will only override the chosen protocol
if ( protocol.equalsIgnoreCase("ftp") )
return new CustomHandler();
else
return null;
}
}
class CustomHandler extends URLStreamHandler {
protected URLConnection openConnection(URL url)
throws IOException {
return new CustomURLConnection(url);
}
}
class CustomURLConnection extends URLConnection {
int reply;
FTPClient ftp = new FTPClient();
InputStream in;
static int defaultPort = 21;
static String defaultPath = "/";
CustomURLConnection ( URL url)
throws IOException {
super( url );
}
synchronized public void connect() throws IOException {
try {
int port;
if ((port = url.getPort()) == -1 )
port = defaultPort;
ftp.connect(url.getHost(), port);
String login = "anonymous";
String password = "";
if(url.getAuthority().indexOf(':')>-1 &&
url.getAuthority().indexOf('#')>-1){
String []auxArray = url.getAuthority().replaceAll("#", ":").split(":");
login = auxArray[0];
password = auxArray[1];
}
ftp.login(login, password);
reply = ftp.getReplyCode();
if (FTPReply.isPositiveCompletion(reply)) {
System.out.println("Connected Apache Success");
} else {
System.out.println("Connection Apache Failed");
ftp.disconnect();
}
in = ftp.retrieveFileStream(url.getFile());
} catch (SocketException ex) {
ex.printStackTrace();
} catch (IOException ex) {
ex.printStackTrace();
}
connected = true;
}
synchronized public InputStream getInputStream()
throws IOException {
if (!connected)
connect();
return ( in );
}
}
*Keep in mind that you can implement new ways to handle different protocols for the java.net.URL this way.
Your code...
...
{
String ftp = "ftp://user:pword#sysc/u/Xxxxxxxxxx/ICS/YT7/XxxxxXxxxxxxx/xxxxxxxxx/logs/xxxxxxxx/XX_YT.LOG";
try
{
URL.setURLStreamHandlerFactory(new ApacheURLStreamHandlerFactory());
...
G'Bye
**(To err is human, to forgive is divine)
Try using the short name for the path. Something like /U/BIGPAT~1/XYZ/ABC/BIGPAT~1/BIGPAT~1/XYZ/ABC/MY_LOG.LOG
You can find the short name for any directory longer than 8 characters with dir /x.
FTP clients are notoriously difficult to write given the variation of (and bugs in) server implementations.
I'm betting that MVS is not completely supported by sun.net.ftp.FtpClient, which is the class used under the hood when you call URL.openStream on an FTP URL.
The Apache Commons Net library should support MVS, but it sounds like you already found a working client.
Have you considered using an RMI for transporting the files that way you can give a direct path to the file as a parameter without the use of ftp then have the file sent back in a byte array.
i wrote programmatically a startup code of the apache Server like this:
public void _start()
{
String Path = "C:\\Dokumente und Einstellungen\\andjock\\Desktop\\ab";
File ftpDirectory = new File(Path);
ftpDirectory.mkdirs();
FtpServerFactory serverFactory = new FtpServerFactory();
ListenerFactory factory = new ListenerFactory();
factory.setPort(2221);
try {
serverFactory.addListener("default", factory.createListener());
PropertiesUserManagerFactory userFactory = new PropertiesUserManagerFactory();
File userFile = new File("C:\\Dokumente und Einstellungen\\andjock\\Desktop\\ftpusers.properties");
userFactory.setFile(userFile);
UserManager um = userFactory.createUserManager();
BaseUser user = new BaseUser();
user.setName("myNewUser");
user.setPassword("secret");
user.setHomeDirectory(Path);
um.save(user);
serverFactory.setUserManager(um);
FtpServer ftpServer = serverFactory.createServer();
ftpServer.start();
} catch (Exception e) {
Logger LOGGER = Logger.getLogger(TestapacheFtpServer.class);
LOGGER.log(Level.FATAL, "Unable to start test ftpserver", e);
}
How do i know that the server is really working ?
how can i access this apache Server , from the "outside"?
i tried with a telnet and ftp (ftp 127.0.0.1) on my machine but i received:
FTP: connect : unknown error code
does someone got any idea ? i just don't want to rely on the jvm log, but rather test it , and accesing the started it
i figure it out!! i wrote a client using the FTP client library (the apache commons library) to test connectivity and list the files ; something like that
FTPClient ftp = new FTPClient();
ftp.connect(InetAddress.getLocalHost(), 2221);// or "localhost" in your case
String loging_success = ftp.login("myNewUser", "secret") == true ? "success" : "failed";
System.out.println("login: "+ loging_success);
FTPFile[] files = ftp.listFiles();
System.out.println("Listed "+files.length+" files.");
for(FTPFile file : files) {
System.out.println(file.getName());
}