I think I understand the difference between ASCII mode and Binary mode on regular FTP transfers -- in Binary mode the file is copied exactly, and in ASCII mode the client may modify line endings (stripping the Carriage Return from Windows -> UNIX or adding it in the other direction). However, I thought that the SFTP protocol only supported Binary mode style transfers; the source file would not be modified.
However, when using the JSch library to copy a file from Windows to UNIX, the Windows-style line endings are stripped. This is problematic because these files are retrieved by others, using various methods, for Windows machines, and I can't guarantee that their clients will re-add the Carriage Return before each Line Feed.
Properties properties = new Properties();
properties.setProperty("StrictHostKeyChecking", "no");
Session session = jsch.getSession(UserName, Address, Port);
session.setPassword(Password);
session.setConfig(properties);
session.connect();
ChannelSftp channel = (ChannelSftp)session.openChannel("sftp");
channel.connect();
channel.
channel.cd(SCPDir);
channel.put(new ByteArrayInputStream(WindowsStyleString.getBytes()), FileName);
channel.disconnect();
session.disconnect();
Is there something I can do to ensure that JSch transfers the files exactly? It is frustratingly lacking in documentation, so I'm not sure if there's some parameter of its or perhaps some additional SSH property I can specify to ensure verbatim transfer. And why is this even happening in the first place, when ASCII mode style modifications aren't part of the SFTP standard?
Another case of misdirection; I apologize for all of these unsatisfactory questions.
It turns out that on Windows, where I can debug, Java's ubiquitous toString returns line endings with the Carriage Return and Line Feed (\r\n). On the Linux production servers, toString returns only \n. However, every use I had of these toString objects, whether uploaded via FTP (even though the host is also a Linux machine), e-mailed, or used in SQL queries, either automatically appended \r or didn't require it. But SFTP didn't.
So I guess the lesson here is that toString probably returns a platform-appropriate line ending format, that org.apache.commons.net.ftp.FTPClient in ASCII mode will add \r before \n even if the FTP server is hosted on Linux (or perhaps that server is masquerading as Windows), and that e-mail and SQL don't particularly care which line endings they have.
SFTP supported only binary mode until version 4 and in version 4 and later (5, 6) binary mode is still default, though ASCII mode is available too. Whatever JSh does with your file is it's own initiative. Maybe it strips CRs itself, or maybe the server does this, I can't say for sure without seeing any kind of log that Jsh or server exposes.
Related
i try to connect to sftp server in my local machine i generate the knownHosts file with the command ssh
and i use it like jsch.setKnownHosts(knownHosts);
but i would to run my job in other machine wish i didn't have access to his knownHosts file
so i decided to disable the check of this rsa key and i wont to know if the action is safe
i will use this line to disable it
session.setConfig("StrictHostKeyChecking", "no");
The question itself has a comment about its safety, but I would like to add that if you don't want to have the StrictHostKeyChecking set as "no", and do not want to rely on a knownHosts file, I would recommend you to:
1) Generate the host fingerprint in a way that's compatible with Jsch, please refer to this question. You can output the generated fingerprint to another file or elsewhere as needed.
2) You can then get the generated value and store it as a variable (environment variable, config file, property, etc.) so that your application can use it. You can pass this fingerprint (not filepath) to Jsch with the setKnownHosts method.
We just upgraded to Java 8 on Amazon Linux. We are using Spring 4.3.8.RELEASE. It used to be that we could get our machine hostname by setting up beans in our application context file like so ...
<bean id="localhostInetAddress" class="java.net.InetAddress" factory-method="getLocalHost" />
<bean id="hostname" factory-bean="localhostInetAddress" factory-method="getHostName" />
But with Java 8, the bean "hostname" now contains the string
localhost
Before Java 8, it used to contain the "hostname" value as run on the command line, which is
[myuser#machine1 ~]$ hostname
machine1.mydomain.org
How can I reconfigure our bean so that it gets the hostname that the command line lists out? I don't want to hard-code anything anywhere.
From InetAddress java 8 is not getting the hostname :
There was similar bug fired in JDK.
What I understand is that they changed default resolution process.
They honor configuration in /etc/nsswitch.conf where hosts are configured for /etc/hosts that gives it main priority for name resolution.
Usually /etc/hosts has record for 127.0.0.1 localhost that provide name for host localhost
There are a similar question in the OpenJDK bugs
The newer calls respect the localhosts /etc/nsswitch.conf configuration files. In the case of this machine that file tells these
calls to look in files before referencing other naming services.
Since the /etc/hosts file contains an explicit mapping for this
hostname / IP combination, that is what is returned.
In the older JDK's the gethostbyname actually ignored the local
machines settings and immediately delegated to the naming service.
You can always use the Runtime class for that :
Process hostname = Runtime.getRuntime().exec("hostname");
BufferedReader stdInput = new BufferedReader(new
InputStreamReader(hostname.getInputStream()));
String s;
while ((s = stdInput.readLine()) != null) {
System.out.println(s);
}
But it's not so recommended as you can see
I believe Java still calls the native gethostname(), but couldn't say why this fails.
For Linux, you could try this alternative:
String hostname = new String(Files.readAllBytes(Paths.get("/proc/sys/kernel/hostname")), StandardCharsets.US_ASCII);
You could try InetAddress.getCanonicalHostName()
The JavaDocs for getCanonicalHostName() says
Gets the fully qualified domain name for this IP address. Best effort method, meaning we may not be able to return the FQDN depending on the underlying system configuration.
You really have three options of differing complexity and information value:
Actually run the hostname command using Runtime. It's fairly portable and should work pretty much everywhere. (Maybe not on IBM's Mainframes)
Grab all available network interfaces and get their hostnames. https://docs.oracle.com/javase/tutorial/networking/nifs/listing.html
Set an environment or system property as the value of hostname.
Like java -Dmy.hostname=hostname ...
Sad part is that it's not that straightforward in Spring XML.
All other options are known to give varying degrees of accuracy.
Like 127.0.0.1 will likely resolve to localhost or localhost.localdomain.
Alternatively, you can run the hostname linux command using Java Runtime, code should be something like below:
String cmdResult="";
Runtime r = Runtime.getRuntime();
Process p = r.exec("hostname");
BufferedReader in = new BufferedReader(new
InputStreamReader(p.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null) {
cmdResult += inputLine;
}
in.close();
Since my comment got lost in the stream, please try
<bean id="hostname" class="com.amazonaws.util.EC2MetadataUtils" factory-method="getLocalHostName()">
Have a look at Amazon's documemtation if the result does not suits your needs.
InetAddress in Java 8 had a bug regarding the resolution of hostname. The bug was reported in version 8u20 and has been fixed since then. Please check that you are not on this version. If not, then using InetAddress.getLocalHost().getHostName() will give you the hostname.
Also I suggest, setting up a static hostname on the AWS instance.
Update /etc/hosts
Update /etc/hostname
Run hostname to check the hostname
Reboot and check the persisted hostname
This should allow you to read the hostname from the file if required.
Also, if hostname command is giving the correct result, then use exec method in Runtime class to execute the command and the read the InputStream to get the response.
PS: You can use InetAddress.getLocalHost().isLoopbackAddress() to check whether the address is a LoopbackAddress and then use a fallback method.
I can be better to define a property in your .properties file for hostname like.
hostname=myserver.mydomain.com
And use it in your application. This way is not harcoded. It also avoid dependency to operating system config which can be changed without attention.
Is there any way to lock file, which i am reading with JSch sftp channel, to prevent other applications to modificate it while session is open?
The problem i have: while i am reading/writing file, some other application changes the file and its causes errors.
Currently i am reading file in such way:
ChannelSftp sftp = (ChannelSftp) session.openChannel("sftp");
InputStream stream = sftp.get("/some/file");
try {
BufferedReader br = new BufferedReader(new InputStreamReader(stream));
// read from br
} finally {
stream.close();
}
Rather than locking the file (which could adversely affect whatever application you suggest is changing the file), and presuming you have ssh access and permissions to do so you could copy the file into a temp file using an ssh command (if on a 'nix system: 'cp /some/file/ /some/temp/file/'), which in all likelihood will be much faster than transferring via scp. (See http://www.jcraft.com/jsch/examples/Exec.java.html for exec example). From there, scp-get the temp file. Lastly, if necessary, delete the temp file via an other JSch exec command.
Jsch at this time supports version 3 of the SFTP protocol. The wikipedia page contains links to different versions of the protocol. Version 3 is here, and the word "lock" doesn't appear anywhere in it. In other words, the protocol doesn't support locking, so there's nothing for Jsch to support.
SFTP versions 5 and 6 do contain support for locking remote files. However, Jsch doesn't support these protocol versions.
I'll add that OpenSSH, the most widely used SSH/SFTP server, only supports SFTP version 3. You'd have to use some other server software to have any hope of doing file locks.
I know there is a lot of thread about this problem but I dont found right: I follow this example: Want to invoke a linux shell command from Java to run command. Problem with ssh command is in authentication. When I run it I need to set password
$ ssh root#server 'fgrep Exception *.log*'
Enter passphrase for key '/././.ssh/id_rsa':
How can I pass here password ?
There are libraries are available to invoke ssh. The Java Secure Channel (JSCH) is a very popular library, used by maven, ant and eclipse. It is open source with a BSD style license.
If you need authentication for ssh, you can use through java.
If your still need to by pass password passing, there are two ways to do what you want. One involves a stored password, and one does not.
Both are non-interactive, meaning that they can work when you're not there to enter a password.
The way that does not require a password. You can use public/private
key authentication instead of passwords with SSH. I'm going to
assume that you're using OpenSSH, which comes with practically every
Linux distribution.
Steps :
Configure your SSH server to accept private key logins. In /etc/ssh/sshd_config make sure that there's a line that says PubkeyAuthentication yes (and that there is no # infront of it). If you change this file, you need to restart the sshd service.
On your local machine (not the server), create yourself a pair of keys with ssh-keygen -t rsa (you can use other options than rsa, but I'm keeping it simple). Do not specify a password. Save the keys in the locations prompted.
Open the contents of the id_rsa.pub file that you just created (it's one very long line of text), and copy the contents into the end of the file $HOME/.ssh/authorized_keys on the server machine. Create the file if it doesn't exist.
Further Detail refer here.
The basic idea is to use expect, which is an
administration automation tool, to type your password in to ssh when
prompted. It might not always work, and when it doesn't, it's hard
to figure out why not. I recommend the first method.
Anyway, here's a command that you can poke at until it does what you
want.
The script Code is:
expect -c 'spawn ssh user#remote.host ; expect assword ; send "passphrase\n" ; interact'
Expect might not be installed on your system. Make sure install that
You need to get the InputStream (which has the output) from the execution and wait for it to ask you for the password, then get the OutputStream (into which you give the command its input) and send it the password you want.
Have a read of this article
Im exporting a set of data to excel in java, the data has certain non ascii characters, while exporting in Windows machine the data are coming correctly in UTF-8 encoded format.But when i deploy my code in Unix machine it is not working properly.UTF-8 encoding is not working properly.
Im using Tomcat 5.5 server.I have also included URIencoding="UTF_8" parameter in server.xml. But still in unix machine it is not working properly
Running Tomcat with a
-Dfile.encoding=UTF8
option will force the VM to adopt UTF8 as it's default encoding regardless of your environment. I suspect that's your problem (and it's good practise nonetheless)
When you are working with UTF-8 data it can be very fragile. Every step in the chain needs to specify utf8.
(1) In the database, be sure the table has UTF8 encoding and collation. "show table status" will tell you this. Google for "mysql alter table utf8" for more info.
(2) The JDBC connection needs to specify the UTF8 encoding. For Connector/J this will be something similar to:
useUnicode=true&characterEncoding=UTF-8
(3) Every stream read/write, every string to byte conversion in the code needs to specify UTF-8 as the encoding. It's easy to miss one. If it's missed Java will use the system default which will vary server to server.
(4) Not applicable here, but for user submitted data from a form, you need to set the request character encoding. (I do this in a servlet filter)
request.setCharacterEncoding("UTF-8");
(5) Probably not applicable here, but if you output HTML/XML/text from your servlet, set the HTTP character encoding header. (This might apply if you are generating the Excel file as an XML file).
response.setContentType("text/html; charset=UTF-8");