I want to check if all the files in a given folder
have portable names or if they have some unfortunate names that may make impossible to represent the same file structure on various file systems; I want to at least support the most common cases.
For example, on Windows, you can not have a file called
aux.txt, and file names are not case sensitive.
This is my best attempt, but I'm not an expert in operative systems and file systems design.
Looking on wikipedia, I've found 'incomplete' lists of possible problems... but... how can I catch all the issues?
Please, look to my code below and see if I've forgotten any subtle unfortunate case. In particular, I've found a lot of 'Windows issues'. Is there any Linux/Mac issue that I should check for?
class CheckFileSystemPortable {
Path top;
List<Path> okPaths=new ArrayList<>();
List<Path> badPaths=new ArrayList<>();
List<Path> repeatedPaths=new ArrayList<>();
CheckFileSystemPortable(Path top){
assert Files.isDirectory(top);
this.top=top;
try (Stream<Path> walk = Files.walk(top)) {//the first one is guaranteed to be the root
walk.skip(1).forEach(this::checkSystemIndependentPath);
} catch (IOException e) {
throw new Error(e);
}
for(var p:okPaths) {
checkRepeatedPaths(p);
}
okPaths.removeAll(repeatedPaths);
}
private void checkRepeatedPaths(Path p) {
var s=p.toString();
for(var pi:okPaths){
if (pi!=p && pi.toString().equalsIgnoreCase(s)) {
repeatedPaths.add(pi);
}
}
}
//incomplete list from wikipedia below:
//https://en.wikipedia.org/wiki/Filename#Reserved_characters_and_words
private static final List<String>forbiddenWin=List.of(
"CON", "PRN", "AUX", "CLOCK$", "NUL",
"COM0", "COM1", "COM2", "COM3", "COM4", "COM5", "COM6", "COM7", "COM8", "COM9",
"LPT0", "LPT1", "LPT2", "LPT3", "LPT4", "LPT5", "LPT6", "LPT7", "LPT8", "LPT9",
"LST", "KEYBD$", "SCREEN$", "$IDLE$", "CONFIG$",
"$Mft", "$MftMirr", "$LogFile", "$Volume", "$AttrDef", "$Bitmap", "$Boot",
"$BadClus", "$Secure", "$Upcase", "$Extend", "$Quota", "$ObjId", "$Reparse"
);
private void checkSystemIndependentPath(Path path) {
String lastName=path.getName(path.getNameCount()-1).toString();
String[] parts=lastName.split("\\.");
var ko = forbiddenWin.stream()
.filter(f -> Stream.of(parts).anyMatch(p->p.equalsIgnoreCase(f)))
.count();
if(ko!=0) {
badPaths.add(path);
} else {
okPaths.add(path);
}
}
}
If I understand your question correctly and by reading the Filename wikipedia page, portable file names must:
Be posix compliant. Eg. alpha numeric ascii characters and _, -
Avoid windows and DOS device names.
Avoid NTFS special names.
Avoid special characters. Eg. \, |, /, $ etc
Avoid trailing space or dot.
Avoid filenames begining with a -.
Must meet max length. Eg. 8-bit Fat has max 9 characters length.
Some systems expect an extension with a . and followed by a 3 letter extension.
With all that in mind checkSystemIndependentPath could be simplified a bit, to cover most of those cases using a regex.
For example, POSIX file name, excluding special devices, NTFS, special characters and trailing space or dot:
private void checkSystemIndependentPath(Path path){
String reserved = "^(CON|PRN|AUX|NUL|COM[1-9]|LPT[1-9])(\\..*)*$";
String posix = "^[a-zA-Z\\._-]+$";
String trailing = ".*[\s|\\.]$";
int nameLimit = 9;
String fileName = path.getFileName().toString();
if (fileName.matches(posix) &&
!fileName.matches(reserved) &&
!fileName.matches(trailing) &&
fileName.length() <= nameLimit) {
okPaths.add(path);
} else {
badPaths.add(path);
}
}
Note that the example is not tested and doesn't cover edge conditions.
For example some systems ban dots in a directory names.
Some system will complain about multiple dots in a filename.
Assuming your windows forbidden list is correct, and adding ":" (mac) and nul (everywhere), use regex!
private static final List<String> FORBIDDEN_WINDOWS_NAMES = List.of(
"CON", "PRN", "AUX", "CLOCK$", "NUL",
"COM0", "COM1", "COM2", "COM3", "COM4", "COM5", "COM6", "COM7", "COM8", "COM9",
"LPT0", "LPT1", "LPT2", "LPT3", "LPT4", "LPT5", "LPT6", "LPT7", "LPT8", "LPT9",
"LST", "KEYBD$", "SCREEN$", "$IDLE$", "CONFIG$",
"$Mft", "$MftMirr", "$LogFile", "$Volume", "$AttrDef", "$Bitmap", "$Boot",
"$BadClus", "$Secure", "$Upcase", "$Extend", "$Quota", "$ObjId", "$Reparse"
); // you can add more
private static final String FORBIDDEN_CHARACTERS = "\0:"; // you can add more
private static final String REGEX = "^(?i)(?!.*[" + FORBIDDEN_CHARACTERS + "])(.*/)?(?!(\\Q" +
String.join("\\E|\\Q", FORBIDDEN_WINDOWS_NAMES) + "\\E)(\\.[^/]*)?$).*";
private static Pattern ALLOWED_PATTERN = Pattern.compile(REGEX);
public static boolean isAllowed(String path) {
return ALLOWED_PATTERN.matcher(path).matches();
}
fyi, the regex generated from the lists/chars as defined here is:
^(?i)(?!.*[<nul>:])(.*/)?(?!(\QCON\E|\QPRN\E|\QAUX\E|\QCLOCK$\E|\QNUL\E|\QCOM0\E|\QCOM1\E|\QCOM2\E|\QCOM3\E|\QCOM4\E|\QCOM5\E|\QCOM6\E|\QCOM7\E|\QCOM8\E|\QCOM9\E|\QLPT0\E|\QLPT1\E|\QLPT2\E|\QLPT3\E|\QLPT4\E|\QLPT5\E|\QLPT6\E|\QLPT7\E|\QLPT8\E|\QLPT9\E|\QLST\E|\QKEYBD$\E|\QSCREEN$\E|\Q$IDLE$\E|\QCONFIG$\E|\Q$Mft\E|\Q$MftMirr\E|\Q$LogFile\E|\Q$Volume\E|\Q$AttrDef\E|\Q$Bitmap\E|\Q$Boot\E|\Q$BadClus\E|\Q$Secure\E|\Q$Upcase\E|\Q$Extend\E|\Q$Quota\E|\Q$ObjId\E|\Q$Reparse\E)(\.[^/]*)?$).*
Each forbidden filename has been wrapped in \Q and \E, which is how you quote an expression in regex so all chars are treated as literal chars. For example, the dollar sign in \Q$Boot\E does't mean end of input, it's just a plain dollar sign.
Thanks everyone.
I have now made the complete code for this,
I'm sharing it as a potential answer, since I think the balances I had to walk are likelly quite common.
Main points:
I had to chose 248 as a max size
I had to accept '$' in file names.
I had to completelly skip any file/folder/subtree that is either labelled as hidden (win) or startin with '.'; those files are hidden and likelly to be autogenerated, out of my
control, and anyway not used by my application.
Of course if your application relies on ".**" files/folders, you may have to check for those.
Another point of friction is multiple dots: not only some system may be upset, but it is not clear where the extension starts and the main name end.
For example, I had a usecase with the file derby-10.15.2.0.jar inside.
Is the extension .jar or .15.2.0.jar? does some system disagree on this?
For now, I'm forcing to rename those files as, for example, derby-10_15_2_0.jar
public class CheckFileSystemPortable{
Path top;
List<Path> okPaths = new ArrayList<>();
List<Path> badPaths = new ArrayList<>();
List<Path> repeatedPaths = new ArrayList<>();
public void makeError(..) {..anything you need for a good message..}
public boolean isDirectory(Path top){ return Files.isDirectory(top); }
//I override the above when I do mocks for testing
public CheckFileSystemPortable(Path top){
assert isDirectory(top);
this.top = top;
walkIn1(top);
for(var p:okPaths){ checkRepeatedPaths(p); }
okPaths.removeAll(repeatedPaths);
}
public void walkIn1(Path path) {
try(Stream<Path> walk = Files.walk(path,1)){
//the first one is guaranteed to be the root
walk.skip(1).forEach(this::checkSystemIndependentPath);
}
catch(IOException e){ throw /*unreachable*/; }
}
private void checkRepeatedPaths(Path p){
var s = p.toString();
for(var pi:okPaths){
if (pi!=p && pi.toString().equalsIgnoreCase(s)) {repeatedPaths.add(pi);}
}
}
private static final List<String>forbiddenWin = List.of(
"CON", "PRN", "AUX", "CLOCK$", "NUL",
"COM0", "COM1", "COM2", "COM3", "COM4", "COM5", "COM6", "COM7", "COM8", "COM9",
"LPT0", "LPT1", "LPT2", "LPT3", "LPT4", "LPT5", "LPT6", "LPT7", "LPT8", "LPT9",
"LST", "KEYBD$", "SCREEN$", "$IDLE$", "CONFIG$",
"$Mft", "$MftMirr", "$LogFile", "$Volume", "$AttrDef", "$Bitmap", "$Boot",
"$BadClus", "$Secure", "$Upcase", "$Extend", "$Quota", "$ObjId", "$Reparse",
""
);
static final Pattern regex = Pattern.compile(//POSIX + $,
"^[a-zA-Z0-9\\_\\-\\$]+$");// but . is handled separately
public void checkSystemIndependentPath(Path path){
String lastName=path.getFileName().toString();
//too dangerous even for ignored ones
if(lastName.equals(".") || lastName.equals("..")) { badPaths.add(path); return; }
boolean skip = path.toFile().isHidden() || lastName.startsWith(".");
if(skip){ return; }
var badSizeEndStart = lastName.length()>248
||lastName.endsWith(".")
||lastName.endsWith("-")
|| lastName.startsWith("-");
if(badSizeEndStart){ badPaths.add(path); return; }
var i=lastName.indexOf(".");
var fileName = i==-1?lastName:lastName.substring(0,i);
var extension = i==-1?"":lastName.substring(i+1);
var extensionDots = extension.contains(".");
if(extensionDots){ badPaths.add(path); return; }
var badDir = isDirectory(path) && i!=-1;
if(badDir){ badPaths.add(path); return; }
var badFileName = !regex.matcher(fileName).matches();
var badExtension = !extension.isEmpty() && !regex.matcher(extension).matches();
if(badFileName||badExtension){ badPaths.add(path); return; }
var ko = forbiddenWin.stream()
.filter(f->fileName.equalsIgnoreCase(f)).count();
if(ko!=0){ badPaths.add(path); return; }
okPaths.add(path);
walkIn1(path);//recursive exploration
}
}
I've written this code to replace object property value:
public void changeObjectPropertyValue(String ind, String propertyFragment, String newValueFragment) {
OWLNamedIndividual individualToReplaceValueOn = factory.getOWLNamedIndividual(prefix + ind);
OWLNamedIndividual newValueInd = factory.getOWLNamedIndividual(prefix + newValueFragment);
OWLObjectProperty theObjectProperty = factory.getOWLObjectProperty(prefix + propertyFragment);
OWLIndividual theOldValue = EntitySearcher.getObjectPropertyValues(individualToReplaceValueOn, theObjectProperty, ont).findFirst().get();
OWLAxiom oldAxiom = factory.getOWLObjectPropertyAssertionAxiom(
theObjectProperty,
individualToReplaceValueOn,
theOldValue);
OWLAxiom newAxiom = factory.getOWLObjectPropertyAssertionAxiom(
theObjectProperty,
individualToReplaceValueOn,
newValueInd);
List<OWLOntologyChange> changes = new Vector<OWLOntologyChange>();
changes.add(new RemoveAxiom(ont, oldAxiom));
changes.add(new AddAxiom(ont, newAxiom));
manager.applyChanges(changes);
}
I want to know if this is a correct way to replace value and if there is a method in OWLAPI library to do this?
This is correct - and the only way to do this sort of changes in OWL API. Axioms are immutable objects, so there is no other way to modify an axiom than recreating it and changing the parts that need modifying in the process.
I encode two EPC tags through "NiceLabel Pro" with data:
First tag: EPC: 555555555, UserData: 9876543210123456789
Second tag: EPC: 444444444, UserData: 123456789123456789
Now I'm trying to get that data through LLRP (in my Java application):
My LLRPClient (one function):
public void PrepareInventoryRequest() {
AccessCommand accessCommand = new AccessCommand();
// A list to hold the op specs for this access command.
accessCommand.setAccessCommandOpSpecList(GenerateOpSpecList());
// Create a new tag spec.
C1G2TagSpec tagSpec = new C1G2TagSpec();
C1G2TargetTag targetTag = new C1G2TargetTag();
targetTag.setMatch(new Bit(1));
// We want to check memory bank 1 (the EPC memory bank).
TwoBitField memBank = new TwoBitField("2");
targetTag.setMB(memBank);
// The EPC data starts at offset 0x20.
// Start reading or writing from there.
targetTag.setPointer(new UnsignedShort(0));
// This is the mask we'll use to compare the EPC.
// We want to match all bits of the EPC, so all mask bits are set.
BitArray_HEX tagMask = new BitArray_HEX("00");
targetTag.setTagMask(tagMask);
// We only only to operate on tags with this EPC.
BitArray_HEX tagData = new BitArray_HEX("00");
targetTag.setTagData(tagData);
// Add a list of target tags to the tag spec.
List <C1G2TargetTag> targetTagList =
new ArrayList<>();
targetTagList.add(targetTag);
tagSpec.setC1G2TargetTagList(targetTagList);
// Add the tag spec to the access command.
accessCommand.setAirProtocolTagSpec(tagSpec);
accessSpec.setAccessCommand(accessCommand);
...
private List<AccessCommandOpSpec> GenerateOpSpecList() {
// A list to hold the op specs for this access command.
List <AccessCommandOpSpec> opSpecList =
new ArrayList<>();
// Set default opspec which for eventcycle of accessspec 3.
C1G2Read opSpec1 = new C1G2Read();
// Set the OpSpecID to a unique number.
opSpec1.setOpSpecID(new UnsignedShort(1));
opSpec1.setAccessPassword(new UnsignedInteger(0));
// We'll read from user memory (bank 3).
TwoBitField opMemBank = new TwoBitField("3");
opSpec1.setMB(opMemBank);
// We'll read from the base of this memory bank (0x00).
opSpec1.setWordPointer(new UnsignedShort(0));
// Read two words.
opSpec1.setWordCount(new UnsignedShort(0));
opSpecList.add(opSpec1);
return opSpecList;
}
My tag handler function:
private void updateTable(TagReportData tag) {
if (tag != null) {
EPCParameter epcParam = tag.getEPCParameter();
String EPCStr;
List<AccessCommandOpSpecResult> accessResultList = tag.getAccessCommandOpSpecResultList();
for (AccessCommandOpSpecResult accessResult : accessResultList) {
if (accessResult instanceof C1G2ReadOpSpecResult) {
C1G2ReadOpSpecResult op = (C1G2ReadOpSpecResult) accessResult;
if ((op.getResult().intValue() == C1G2ReadResultType.Success) &&
(op.getOpSpecID().intValue() < 1000)) {
UnsignedShortArray_HEX userMemoryHex = op.getReadData();
System.out.println("User Memory read from the tag is = " + userMemoryHex.toString());
}
}
}
...
For the first tag, "userMemoryHex.toString()" = "3938 3736"
For the second tag, "userMemoryHex.toString()" = "3132 3334"
Why? How do I get all user data?
This is my rfid tag.
The values that you get seem to be the first 4 characters of the number (interpreted as an ASCII string):
39383736 = "9876" (when interpreting those 4 bytes as ASCII characters)
31323334 = "1234" (when interpreting those 4 bytes as ASCII characters)
Since the specification of your tag says
Memory: EPC 128 bits, User 32 bits
your tag can only contain 32 bits (= 4 bytes) of user data. Hence, your tag simply can't contain the full value (i.e. 9876543210123456789 or 123456789123456789) that you tried to write as UserData (regardless of whether this was interpreted as a decimal number or a string).
Instead, your writer application seems to have taken the first 4 characters of those values, encoded them in ASCII, and wrote them to the tag.
Given a grammar (simplified version below) where I can enter arbitrary text in a section of the grammar, is it possible to format the content of the arbitrary text? I understand how to format the position of the arbitrary text in relation to the rest of the grammar, but not whether it is possible to format the content string itself?
Sample grammar
Model:
'content' content=RT
terminal RT: // (returns ecore::EString:)
'RT>>' -> '<<RT';
Sample content
content RT>>
# Some sample arbitrary text
which I would like to format
<<RT
you can add custom ITextReplacer to the region of the string.
assuming you have a grammar like
Model:
greetings+=Greeting*;
Greeting:
'Hello' name=STRING '!';
you can do something like the follow in the formatter
def dispatch void format(Greeting model, extension IFormattableDocument document) {
model.prepend[newLine]
val region = model.regionFor.feature(MyDslPackage.Literals.GREETING__NAME)
val r = new AbstractTextReplacer(document, region) {
override createReplacements(ITextReplacerContext it) {
val text = region.text
var int index = text.indexOf(SPACE);
val offset = region.offset
while (index >=0){
it.addReplacement(region.textRegionAccess.rewriter.createReplacement(offset+index, SPACE.length, "\n"))
index = text.indexOf(SPACE, index+SPACE.length()) ;
}
it
}
}
addReplacer(r)
}
this will turn this model
Hello "A B C"!
into
Hello "A
B
C"!
of course you need to come up with a more sophisticated formatter logic.
see How to define different indentation levels in the same document with Xtext formatter too
I have a routine that I repeatedly doing for many projects and I want to generalized it. I used iText for PDF manipulation.
Let say that I have 2000 PDFs inside a folder, and I need to zip these together. Let say the limit is 1000 PDFs per zip. So the name of the zip would follow this rule: job name + job sequence. For example, the zip name of the first 1000 PDF would be XNKXMN + AA and the second zip name would be XNKXMN + AB. Before zipping these PDFs, I need to add some text to each PDF. Text look something like this job name + job sequence + pdf sequence. So the first PDF inside the first zip will have this text XNKXMN + AA + 000001, and the one after that is XNKXMN + AA + 000002. Here is my attempt
First I have abstract clas GenericText that represent my text.
public abstract class GenericText {
private float x;
private float y;
private float rotation;
/**
* Since the text that the user want to insert onto the Pdf might vary
* from page to page, or from logical document to logical document, we allow
* the user to write their own implementation of the text. To give the user enough
* flexibility, we give them the reference to the physical page index, the logical page index.
* #param physcialPage The actual page number that the user current looking at
* #param logicalPage A Pdf might contain multiples sub-documents, <code>logicalPage</code>
* tell the user which logical sub-document the system currently looking at
*/
public abstract String generateText(int physicalPage, int logicalPage);
GenericText(float x, float y, float rotation){
this.x = x;
...
}
}
JobGenerator.java: my generic API to do what I describe above
public String generatePrintJob(List<File> pdfList, String outputPath,
String printName, String seq, List<GenericText> textList, int maxSize)
for (int currentPdfDocument = 0; currentPdfDocument < pdfList.size(); currentPdfDocument++) {
File pdf = pdfList.get(currentPdfDocument);
if (currentPdfDocument % maxSize != 0) {
if(textList != null && !textList.isEmpty()){
for(GenericText gt : textList){
String text = gt.generateText(currentPdfDocument, currentPdfDocument)
//Add the text content to the PDF using PdfReader and PdfWriter
}
}
...
}else{
//Close the current output stream and zip output stream
seq = Utils.getNextSeq(seq);
jobPath = outputPath + File.separator + printName + File.separator + seq + ".zip"
//Open new zip output stream with the new <code>jobPath</code>
}
}
}
So now in my main class I would just do this
final String printName = printNameLookup.get(baseOutputName);
String jobSeq = config.getPrintJobSeq();
final String seq = jobSeq;
GenericText keyline = new GenericText(90, 640, 0){
#Override
public String generateText(int physicalPage, int logicalPage) {
//if logicalPage = 1, Utils.right(String.valueOf(logicalPage), 6, '0') -> 000001
return printName + seq + " " + Utils.right(String.valueOf(logicalPage), 6, '0');
}
};
textList.add(keyline);
JobGenerator pjg = new JobGenerator();
pjg.generatePrintJob(...,..., printName, jobSeq, textList, 1000);
The problem that I am having with this design is that, even though I process archive the PDF into two zip correctly, the text is not correctly reflect. The print and the sequence does not change accordingly, it stay XNKXMN + AA for 2000 PDF instead of XNKXMN + AA for the first 1000 and change to XNKXMN + AB for the later 1000. There seems to be flawed in my design, please help
EDIT:
After looking at toto2 code, I see my problem. I create GenericText with the hope of adding text anywhere on the pdf page without affecting the basic logic of the process. However, the job sequence is by definition depending on the logic,as it need to increment if there are too many PDFs for one ZIP to handle (> maxSize). I need to rethink this.
When you create an anonymous GenerateText, the final seq which you use in the overridden generateText method is truly final and will always remain the value given at creation time. The update you carry on seq inside the else in generatePrintJob does nothing.
On a more general note, your code looks very complex and you should probably take a step back and do some major refactoring.
EDIT:
I would instead try something different, with no template method pattern:
int numberOfZipFiles =
(int) Math.ceil((double) pdfList.size() / maxSize);
for (int iZip = 0; iZip < numberOfZipFiles; iZip++) {
String batchSubName = generateBatchSubName(iZip); // gives AA, AB,...
for (int iFile = 0; iFile < maxSize; iFile++) {
int fileNumber = iZip * maxSize + iFile;
if (fileNumber >= pdfList.size()) // can happen for last batch
return;
String text = jobName + batchSubName + iFile;
... add "text" to pdfList.get(fileNumber)
}
}
However, you might also want to maintain the template pattern. In that case, I would keep the for-loops I wrote above, but I would change the generating method to genericText.generateText(iZip, iFile) where iZip = 0 gives AA and iZip = 1 gives AB, etc:
for (int iZip = 0; iZip < numberOfZipFiles; iZip++) {
for (int iFile = 0; iFile < maxSize; iFile++) {
int fileNumber = iZip * maxSize + iFile;
if (fileNumber >= pdfList.size()) // can happen for last batch
return;
String text = genericText.generateText(iZip, iFile);
... add "text" to pdfList.get(fileNumber)
}
}
It would be possible also to have genericText.generateText(fileNumber) which could itself decompose the fileNumber in AA000001, etc. But that would be somewhat dangerous because maxSize would be used in two different places and it might be bug prone to have duplicate data like that.