Develop Reference

text encoding in ftp download app causing errors

I have created a script to download files from an ftp endpoint. I was assured that the files would be in utf-8 encoding but upon downloading and parsing the xml, we encounter bad formatting. The process is to download the file, convert the xml to json and parse and convert to a different format. What we see after converting to json is for example the following which appears instead of chinese/hindi/arabic characters:
"Size": 3227,
"Title": "??? ???? ????? ?? ???? ?? 5 ??? ?? ??? ?? ?? ???? ?? ????????? ?? ???? ???? ??????-Pakistan new army chief
The code snippet is the following:
ftp.connect("xx.xxx.xxx.xx");
ftp.login("xxxx", "xxxxx");
ftp.enterLocalPassiveMode();
ftp.setControlEncoding("UTF-8");
ftp.setFileType(FTP.BINARY_FILE_TYPE);
...
String remoteFile1 = ftp.printWorkingDirectory() + "/" + file.getName();
File downloadFile1 = new File(destFolder + "/" + "/" + file.getName());
OutputStream outputStream1 = new BufferedOutputStream(new FileOutputStream(downloadFile1));
boolean success = ftp.retrieveFile(remoteFile1, outputStream1);
outputStream1.flush();
outputStream1.close();
....
DocumentBuilderFactory docFactory =
DocumentBuilderFactory.newInstance();
DocumentBuilder docBuilder = docFactory.newDocumentBuilder();
Document doc = docBuilder.newDocument();
doc = docBuilder.parse(xmlFile);
doc.getDocumentElement().normalize();
TransformerFactory tf = TransformerFactory.newInstance();
Transformer trans = tf.newTransformer();
StringWriter sw = new StringWriter();
trans.transform(new DOMSource(doc), new StreamResult(sw));
String xml = sw.toString();
JSONObject xmlJSONObj = XML.toJSONObject(xml);
String jsonPrettyPrintString = xmlJSONObj.toString(4);
jsonMapper.configure(SerializationFeature.WRAP_ROOT_VALUE, false);...
Can someone advise how to ensure the encoding can be changed to output the correct format for foreign characters?

Remove SOAP envelope

I have an InputStream containing a SOAP message, including the envelope. I don't know the contents of the body beforehand and therefore cannot create a Jaxb annotated class for it.
I've tried many ways, inlcuding a custom SOAPWrapper JaxB Class with XmlAnyElement and other ways. Currently I have this:
private InputStream removeSoapEnvelope(final InputStream inputStream) throws IOException, TransformerException
{
final SoapBody body = messageFactory.createWebServiceMessage(inputStream)
.getSoapBody();
final Transformer transformer = TransformerFactory.newInstance()
.newTransformer();
final DOMResult domResult = new DOMResult();
transformer.transform(body.getPayloadSource(), domResult);
final StringWriter writer = new StringWriter();
transformer.transform(new DOMSource(domResult.getNode()), new StreamResult(writer));
byte[] barray = writer.toString()
.getBytes(StandardCharsets.UTF_8);
return new ByteArrayInputStream(barray);
}
It seems to work but is horribly inefficient. Is there no short and concise way of achieving this with standard libraries and without regex?
Thanks

Here's a solution using XPath to get the element (pure JaxB? not sure). Takes the document as a regular XML document so it should work for any I guess
FileInputStream fileIS;
fileIS = new FileInputStream(System.getProperty("user.home") + "/tmp/soap.xml");
DocumentBuilderFactory builderFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder builder = builderFactory.newDocumentBuilder();
Document xmlDocument;
xmlDocument = builder.parse(fileIS);
XPath xPath = XPathFactory.newInstance().newXPath();
String expression01 = "//*[local-name()='Body']";
Node currentNode = (Node) xPath.compile(expression01).evaluate(xmlDocument, XPathConstants.NODE);
StringWriter buf = new StringWriter();
Transformer xform = TransformerFactory.newInstance().newTransformer();
xform.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");
xform.setOutputProperty(OutputKeys.INDENT, "yes");
xform.transform(new DOMSource(currentNode), new StreamResult(buf));
System.out.println(buf.toString());
Result:
<soap:Body>
<incident xmlns="http://example.com">
<Company type="String">Test</Company>
</incident>
</soap:Body>

I ended up doing it with regex. All other options are too slow:
private InputStream removeSoapEnvelope(final InputStream inputStream) throws IOException
{
final String text = new String(inputStream.readAllBytes(), UTF_8);
final String replace = text.replaceAll("\\s*<\\/?(?:SOAP-ENV|soap):(?:.|\\s)*?>", "");
File file = File.createTempFile("temp", XML_NS_PREFIX);
Files.writeString(file.toPath(), replace);
return new FileInputStream(file);
}

creating space in end of xml tag in java

My xml tag is :
<Description/>
I want with space :
<Description />
How can I do this in Java?
I am signing xml document , in original file space has been used but when I used following code and print it, it printing without space.
String thisLine = "";
String xmlString = "";
BufferedReader br = new BufferedReader(new FileReader(originalXmlFilePath));
while ((thisLine = br.readLine()) != null) {
xmlString = xmlString + thisLine.trim();
}
br.close();
ByteArrayInputStream xmlStream = new ByteArrayInputStream(xmlString.getBytes());
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
dbf.setNamespaceAware(true);
dbf.setIgnoringElementContentWhitespace(true);
dbf.setValidating(false);
Document doc = dbf.newDocumentBuilder().parse
(xmlStream );
doc.setXmlStandalone(true);
DOMSignContext dsc = new DOMSignContext
(keyEntry.getPrivateKey(), doc.getDocumentElement());
javax.xml.crypto.dsig.XMLSignature signature = fac.newXMLSignature(si, ki);
signature.sign(dsc);
// Output the resulting document.
// OutputStream os = new FileOutputStream(new File(destnSignedXmlFilePath));
TransformerFactory tf = TransformerFactory.newInstance();
Transformer trans = tf.newTransformer();
trans.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");
trans.setOutputProperty(OutputKeys.INDENT, "yes");
StringWriter writer = new StringWriter();
trans.transform(new DOMSource(doc), new StreamResult(writer));
String output = writer.getBuffer().toString();//.replaceAll("\n|\r", "");
System.out.println("output== "+output);

What you are doing wrong is signing an arbitrary unprocessed text instead of submitting a canonical version of your document (without spaces in tags, but also with sorted attributes, with quotes of the same type, etc.) to the digital signature computation.
The Canonical XML and Exclusive Canonical XML W3C recommendations specify a standard and comprehensive way to eliminate arbitrary differences.

Convert DOM element encoding from CP1251 to UTF-8

I have a simple server-side code that takes request xml and inserts it as string into Oracle database Clob column. The problem is that client-side sends request xml with CP1251 encoded text, but I need to insert it into Oracle with UTF-8 encoding.
Now the code that I use for CP1251 is:
Element soapinElement = (Element) streams.getSoapin().getValue().getAny(); //retrieve request xml
Node node = (Node) soapinElement;
Document document = node.getOwnerDocument();
DOMImplementationLS domImplLS = (DOMImplementationLS) document.getImplementation();
LSSerializer serializer = domImplLS.createLSSerializer();
LSOutput output = domImplLS.createLSOutput();
output.setEncoding("CP1251");
Writer stringWriter = new StringWriter();
output.setCharacterStream(stringWriter);
serializer.write(document, output);
String soapinString = stringWriter.toString();
This code recognizes text encoded in CP1251.
The task is to make the same but with readable text encoded in UTF-8. Please suggest any ideas.
I tried this, but it produced unreadable characters instead of cyrillic:
Element soapinElement = (Element) streams.getSoapin().getValue().getAny();
Node node = (Node) soapinElement;
Document document = node.getOwnerDocument();
DOMImplementationLS domImplLS = (DOMImplementationLS) document.getImplementation();
LSSerializer serializer = domImplLS.createLSSerializer();
LSOutput output = domImplLS.createLSOutput();
output.setEncoding("CP1251");
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
output.setByteStream(byteArrayOutputStream);
serializer.write(document, output);
byte[] result = byteArrayOutputStream.toByteArray();
InputStream is = new ByteArrayInputStream(result);
Reader reader = new InputStreamReader(is, "CP1251");
OutputStream out = new ByteArrayOutputStream();
Writer writer = new OutputStreamWriter(out, "UTF-8");
char[] buffer = new char[10];
int read;
while ((read = reader.read(buffer)) != -1) {
writer.write(buffer, 0, read);
}
reader.close();
writer.close();
String soapinString = out.toString();

You can decode the CP1251 characterset Data like below
Charset utf8charset = Charset.forName("UTF-8");
Charset cp1251charset = Charset.forName("CP1251");
// decode CP1251
CharBuffer data = cp1251charset.decode(ByteBuffer.wrap(result));
and encode to UTF-8 character set
// encode UTF-8
ByteBuffer outputBuffer = utf8charset.encode(data);
and convert the ByteBuffer to byte[]
// UTF-8 Value
byte[] outputData = outputBuffer.array();
This should probably solve your issue.

Transformer not reading Special Character from Document Object

I am trying to read xml data from Document Object, and then using transformer to render the data inside the document object to pdf,using XSL,
My code is :
Document doc = toXML(arg1,arg2);
doc contains data like :
İlkyönetmeliği
with in tags
InputStream inputStream = new FileInputStream(xslFilePath);
transformer = factory.newTransformer(new StreamSource(inputStream));
transformer.setParameter("encoding", "UTF-8");
transformer.setOutputProperty(OutputKeys.ENCODING, "UTF-8");
transformer.transform(new DOMSource(doc.getDocumentElement()), res);
Special characters present in xml are not getting rendered accordingly and displaying like
#lk yard#m.
I have also set encoding to UTF-8 ,but still it is displaying like above.

It is not clear what causes your encoding problem because I cannot see how your document is read/constructed and how your transformation result res is set up. Try the following standalone example code which handles encoding with XSLT. Maybe you can even modify it gradually to use your actual data in order to see what goes wrong.
public static void main(String[] args) {
try {
String inputEncoding = "UTF-16";
String xsltEncoding = "ASCII";
String outputEncoding = "UTF-8";
ByteArrayOutputStream bos = new ByteArrayOutputStream();
OutputStreamWriter osw = new OutputStreamWriter(bos, inputEncoding);
osw.write("<?xml version='1.0' encoding='" + inputEncoding + "'?>");
osw.write("<root>İlkyönetmeliği</root>"); osw.close();
byte[] inputBytes = bos.toByteArray();
bos.reset();
osw = new OutputStreamWriter(bos, xsltEncoding);
osw.write("<?xml version='1.0' encoding='" + xsltEncoding + "'?>");
osw.write("<xsl:stylesheet xmlns:xsl='http://www.w3.org/1999/XSL/Transform' version='1.0'>");
osw.write("<xsl:template match='#*|node()'><xsl:copy><xsl:apply-templates select='#*|node()'/></xsl:copy></xsl:template>");
osw.write("</xsl:stylesheet>"); osw.close();
byte[] xsltBytes = bos.toByteArray();
bos.reset();
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
DocumentBuilder db = dbf.newDocumentBuilder();
Document d = db.parse(new InputSource(new InputStreamReader(new ByteArrayInputStream(inputBytes), inputEncoding)));
// if encoding declaration correct, use: Document d = db.parse(new InputSource(new ByteArrayInputStream(inputBytes)));
System.out.println(XPathFactory.newInstance().newXPath().evaluate("/root[1]", d));
TransformerFactory tf = TransformerFactory.newInstance();
Transformer t = tf.newTransformer(new StreamSource(new InputStreamReader(new ByteArrayInputStream(xsltBytes), xsltEncoding)));
// if encoding declaration correct, use: Transformer t = tf.newTransformer(new StreamSource(new ByteArrayInputStream(xsltBytes)));
StreamResult sr = new StreamResult(new OutputStreamWriter(bos, outputEncoding));
t.setOutputProperty(OutputKeys.ENCODING, outputEncoding);
t.transform(new DOMSource(d.getDocumentElement()), sr);
byte[] outputBytes = bos.toByteArray();
Scanner s = new Scanner(new InputStreamReader(new ByteArrayInputStream(outputBytes), outputEncoding));
String output = s.useDelimiter("</>").next(); // read all
s.close();
System.out.println(output);
} catch (Exception ex) {
ex.printStackTrace(System.err);
}
The example code applies the XSLT identity template to a minimal input containing the non-ASCII characters.
I output the string to check if it has been parsed correctly in the document using XPath. You may want to check your (intermediate) document if you know how to locate it with XPath.
Note that, if present, the parser tries to pick up the encoding declared in the XML processing instruction (PI) by default when reading an XML file. It assumes that actual and declared encoding are the same. If they differ or the PI is missing, then you can enforce the actual encoding e.g. by using an InputStreamReader as in the code above.