As part of a larger task we need to extract information from fixed field length text files. The data files were original developed for EDI but are in widespread use through the industry so we can't ask for a more modern way of encoding data.
I wrote a Java program, implemented as a user defined function, to do the file parsing. It works properly when run locally within jDeveloper. One of the things it does is remove all the newline characters so I can count record lengths accurately. When I try to run it from a simple composite application on the SOA server I find a strange problem: the newlines are gone but I get a new whitespace character in their place.
I do not know what can cause this or how to reliably deal with it. My composite is very simple; I just paste the file content into the "input" field using the test console on the SOA server and this sends the string to the user defined function which parses the file and outputs an XML fragment I can then read.
If I manually strip all the newlines and paste that in, all is well and it works fine but if I send a the data with newlines I get extra whitespace.
Is the composite trying to normalize newlines for me? If so; I would like to know how to make it quit.
Related
I am trying to modify a pbit archive in java in my application.
The point is to update the data source of a pbit without using any PowerBI application so I have to modify the DataModelSchema entry.
My problem is : when I read the file with an InputStream and display it in the console, there are blank spaces added between each letter so I am not able to search and replace the right string. Even if I add artificially blank spaces in my string.
For example if i search "content" or even "c o n t e n t", it never finds it.
That problem never appears when I read "normal" .zip archives.
An overview of my ouput when i read the file with additional blank spaces :
https://community.powerbi.com/t5/Service/modify-pbit-data-source-in-a-program/m-p/1744747#M124339
So I would like to get to know if there is a special encoding in pbit templates which might add special spaces or whatever if there is a way to read it properly.
Thanks for your help,
Regards
J.MARQUE
Just to clarify if someone get the same problem: that appears that it is not possible (for now) to update directly the content of the data schema of a pbit template, the only way seems to use Power BI REST API.
I am making a java program where I input answers for a friendship survey. It spits out the student's top ten friends. However I need to print out the results and give them to the students. The old of doing it was to have the java program write to write html then we would open each file one at a time and print out the page. However, having 400+ students to do it for takes a while.
So since I am re making the program I would like to make it so I can just have it on word files and print them all out at once. However, I don't know how to write to a word file and notepad isn't stylish enough. Anyone know how to make this possible or another way that is easier?
I did a similar thing some years ago, using Rich Text Format. Its advantage is that it's a plain text format that can easily be manipulated.
I created the form document in Word with some unique placeholder strings where I'd later fill in the actual data and saved it as RTF.
With a text editor, I made sure that Word didn't split the placeholders by inserting some junk formatting directives, and corrected that manually where necessary.
Then, filling in the actual data just meant to do some simple text replacement (in my case, there was no risk to interfere with the formatting directives), and saving the resulting RTF file.
As Word typically opens RTF files just as easy as DOC or DOCX ones, this was an easy working solution for me.
I've done a lot of internet searching to find some information to no avail.. Hopefully you can help me..
I want to be able to use a flat file, with normal content (i.e. full english sentences, paragraphs etc), extract each word and store each word individually, one word per row, in a SQL database (doesn't matter if there are spaces but characters such as apostrophes can be kept in)
I then want to have a HTML page with code to access this DB and output the text to the user one word at a time, essentially 'writing' the inputted files text word-by-word on the web page.
This is just a coding exercise but I am frustrated as I know the what but not the how.. I am not sure where to start. Please note some of these files can be quite big ~ 20,000 words so there may be a performance element to consider to any solution.
TL;DR: I want to extract individual words from a text file with normal everyday sentences into a SQL DB that I can retrieve from a HTML page.
Simple read & split exercise
with open(<filename>) as f:
dd = {}
for ln in f:
wds = ln.strip().split()
for word in wds:
dd[word] = 1 # need something for value
for wkey in dd:
<insert into db>
Well, before you start you should choose just one programming language. Since you seem like you are a beginner I would highly recommend Python over Java, but it depends on if you're required to use any particular language by an employer/professor/etc.
Also just to point out, this is also a very BIG task that you've chosen. I'll try to break it down into parts for you, but I recommend starting with just one of these parts before you move on, and make sure it works on your local machine before you try putting it on the web.
First you need to use something read in your file, preferably line by line. A method similar to FileReader/BufferedReader in Java or the open(), readlines() functions in Python will do these. I would also check out the tutorials online on file handling for whichever of these two languages you're going to use. The Python one is here. Practice this with a test file or a small section of your real file before you start working on your real input files.
When you start processing the lines from the file, I would recommend splitting them into individual words using a string split function on spaces or on any punctuation, such as ,.!?". This way you'll pull out the individual words from the each line in the file.
Next, you'll want to choose a database API for the appropriate programming language. I used PyMySQL but there is also MySQLDB for Python. In Java there is JDBC.
You'll need to then build your database on a server somewhere, preferably on the same server as your HTML page for ease of connection. You'll want to practice connecting to your database and adding sample rows before you start trying to process your real input files.
You can't have normal HTML access the database directly - you'll need to use a coding language like Python for that. I've never used Java for webpages, but with Python you'll simply output text and tell the server to display it as the webpage. This will do the trick:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import otherstuffhere
## Must have this header to tell browser how to handle this output
## and must be printed first
print ("Content-Type: text/html\n\n")
## Connect to database here
## Your code to display words from the database goes below here
print (myfield1)
Also remember that when you output your text, you'll need to add all the HTML tags to the normal text output. For example, when printing each word, you'll need to add <p> or <br> to end each line, because although the Python print() function will automatically add a line break, this doesn't translate to a line break in HTML. For example:
print ("My word list is: <br>")
for word in dbOutputList:
print (word)
print ("<br>")
After that the REAL fun/crying begins, but you should work on the above before you move on.
I've built a content management tool that allows a product team to create and manage product that gets exported to a website and for a different team of designers to create print ads for newspapers displaying the same product data.
My problem is with the InDesign graphic designers and the macros that they use within InDesign. The macros have the ability to take copy/pasted text/data and auto format the text inside InDesign based on the presence of certain characters. In particular the design team uses tab, "soft line break" (shift return), and regular line breaks (hard returns) in their macros.
Right now I generate a block of text with the records and the desired formatting characters in a java Class and then that's sent via DWR to the client side. When there is a requirement for a tab character I send \t, return is \r and I was hoping that a soft line break would be \n however InDesign seems to regard both \r and \n as a regular line break.
I had given up on being able to pass a soft-return until yesterday when I cam across Unicode \u2028 (soft line break) and \u2029 (regular line break). I've tried outputting both of these characters instead of \r and \n in the hopes that InDesign may regard these characters differently. In the box that the designers copy the output from it looks like there is no character there. There's no line break at all in the places where I've specific \u2028 to appear. When I copy/paste the output into a text editor it shows me that there is an unrecognized character there (it displays as a box with a question mark around it).
Platform is Java/MySQL running on Tomcat.
To date, I haven't had to deal too much with character encoding in this application. Header has <meta charset="utf-8" /> set but that's about it so far. I've tried setting this to utf-16 but it doesn't change the output. All of the tables in the MySQL database are set to utf8/utf8_general_ci.
Thoughts? How can I force InDesign to take copy/pasted text and recognize all of its macro capable characters? Actually, it's just the soft line breaks that it's not recognizing. HELP! :)
Thank you. Sorry this is so long!
Ryan V
I've been playing around with ID CS6 (OS X) for a while and I can't for the life of me get it to recognize a pasted LF as a forced line break. LF and CR and CRLF all go to paragraph breaks. U+2028 and U+2029 are display as empty glyphs, not breaks.
I'm a little wary of posting this as an answer, but I'll give it a go:
You might consider providing the text as a downloaded .txt file. CS5 introduced "Tagged Text" (a sort of XML-ish text document with full support for InDesign characters, attributes, etc.,) so this means your designers will be able to place the text file and InDesign will treat everything as intended.
To turn your existing text into CS5+'s Tagged Text (see the reference here), plop a <ASCII-MAC> or <ASCII-WIN> (as appropriate) as the first line and escape any '<' or '>'s with a backslash, then you're free to use <0x000A> as a forced line break. (literally those 8 characters)
That's probably mega-overkill, but it's certainly the most stupidly reliable way I can think of. I'll edit if I get anything else working.
NB. "forced line break" is the term InDesign itself uses for the character produced by Shift+Enter, your "soft line break;" contrast with "paragraph break" for a standard carriage return. InDesign apparently represents forced breaks with LF (U+000A) and paragraph breaks with CR (U+000D).
I'm not sure how you were trying to transfer and print out your characters (if you post your DWR and javascript code I might be able to help more), but one thing I would try is to ensure that your java output is actual UTF-8 using something like:
String yourRecordString = "Some line 1. \u2028Some line 2.";
ByteBuffer bb = Charset.forName("UTF-8").encode(yourRecordString);
Then, you can write out the bytes in bb into an output stream/file and check them. (Make sure to write them as bytes and not as a String nor as chars.) For example, the UTF-8 encoding of \u2028 is E2 80 A8, so you should see that sequence at the appropriate place in your output. (I use hexmode in vim for things like this.)
Then, make sure that these bytes get received back on the javascript side. (While I'm not an expert with DWR, I might prefer to make your java function return something other than a String.)
This should at least help you diagnose where the problem lies. If you do see that sequence and if InDesign still isn't recognizing the soft line breaks, then you at least know the problem is with InDesign and that you will have to find some other solution (such as modifying the designer's macros to recognize other characters).
(Also, note that you can see the default encoding for your JVM using Charset.defaultCharset(). My guess is that your default is not UTF-8 and that InDesign may have also had a problem with the UTF-16 you tried due to endianess or something like that.)
I have a text file that was provided to me and no one knows the encoding on it. Looking at it in a text editor, everything looks fine, aligned properly into neat columns.
However, I'm seeing some anomalies when I read the data. Even though, visually, the field "Foo" appears in the same columns in the text file (for instance, in columns 15-20), when I try to pull it out using substring(15,20) my data varies wildly. Sometimes I'll pull bytes 11-16, sometimes 18-23, sometimes 15-20...there's no consistency between records.
I suspect that there are some special chartacters, invisible to my text editor, but readable by (and counted in the index of) the String methods. Is there any way in Java to dump the contents of the file with any special characters visible so I can see what I need to Strings I need replace with regex?
If not in Java, can anyone recommed a tool that may be able to help me out?
I would start with having a look at the file directly. Any code adds a layer of doubt. Take a Total Commander (or equivalent on your platform), view the file (F3) and switch to hex mode. You suggest that the special characters behavior is not even consistent between lines, so you should get some visual clue about the format before you even attempt to fix it algorithmically.
Have you tried printing the contents of the file as individual integers or bytes? That way you can see if there are any hidden characters.