Develop Reference

Seemingly random tika performance issues

I use RxJava to pipe documents from a source to tika and from tika to ElasticSearch.
At some point in time tika takes about 5 minutes to index a document and continues normally afterwards.
I am unable to properly point out the cause: If I restart the application, everything is still the same, let's say it took 5mins at the 301st document last time, it will take 5mins at the 301st document again. But if I change up the order of the documents, it happens neither at the same index (301), nor with the same document (the previous 301st).
Here are the relevant parts of the application:
public Indexable analyze(Indexable indexable) {
Timestamp from = new Timestamp(System.currentTimeMillis());
if (indexable instanceof NCFile) {
// there is some code here that has no effect
Metadata md = this.generateMetadata(((NCFile) indexable).getPath());
((NCFile) indexable).setType(md.get("Content-Type"));
if (((NCFile) indexable).getType().startsWith("text/")) {
((NCFile) indexable).setContent(this.parseContent(((NCFile) indexable).getPath())); //todo . hier könnte man noch viel mehr machen
} else {
((NCFile) indexable).setContent("");
}
((NCFile) indexable).setType(this.detectMediaType(((NCFile) indexable).getPath()));
}
Timestamp to = new Timestamp(System.currentTimeMillis());
System.out.println("the file " + ((NCFile) indexable).getPath() + " took " + (to.getTime()-from.getTime()) + " ms to parse");
return indexable;
}
and the pipeline that is feeding the code above:
nc.filter(action -> action.getOperation() == Operation.INSERT)
.map(IndexingAction::getIndexable)
.subscribeOn(Schedulers.computation())
.map(indexableAction -> metadataAnalyzer.analyze(indexableAction))
.map(indexable -> {
indexer.insert(indexable);
return indexable;
})
.subscribeOn(Schedulers.io())
.map(indexable -> "The indexable " + indexable.getIdentifier() + " of class " + indexable.getClass().getName() + " has been inserted.")
.subscribe(message -> this.logger.log(Level.INFO, message));
My guess would be that the problem is memory- or thread-related; but as far as I can see, the code should work perfectly fine.
the file Workspace/xx/xx-server/.test-documents/testRFC822_base64 took 5 ms to index
the file Workspace/xx/xx-server/.test-documents/testPagesHeadersFootersAlphaLower.pages took 306889 ms to index
the file Workspace/xx/xx-server/.test-documents/testFontAfterBufferedText.rtf took 2 ms to index
the file Workspace/xx/xx-server/.test-documents/testOPUS.opus took 7 ms to index
Funny thing is, these are the tika testfiles provided in their repo.
EDIT:
After a request I looked into it using FR, but I am not sure what exactly I have to look at:
Right after the plateau it stops working, even though neither the RAM nor the CPU limit is met
EDIT 2:
Is it the PipedReader that is blocking alle of these threads? Do I understand that correctly?
EDIT 3:
Here is a 1min flight recording:
Note: The flight recording seems wrong. In my system monitor application I do not see such a big memory consumption (apparent 16GB?!)...
What am I doing wrong?

Best way to validate and read files from a date wise folder in Java

I have to validate and read .txt files from a date wise folder. Need suggestions on best possible ways to achieve this
I will have a property file which will have following information
value active channel
5092 Y 11
5092 Y 12
5092 Y 13
5093 N 10
5093 N 11
from this property file first i need get active value(i.e. 5092) and their channel information i.e 11,12,13
based on the above information need to iterate files from a date wise folder.
Input(folder)
10JUN2017
HW_5092_ABC_11.txt
HW_5092_ABC_12.txt
HW_5092_ABC_13.txt
11JUN2017
HW_5092_ABC_11.txt
HW_5092_ABC_12.txt
Based on the property file information (i.e. 5092 is active file and it has channel 11,12,13) look for the current date folder i.e. 11JUN2017 then look for files of 5092. If 11JUN2017 has files related to 5092 (i.e. all 3 files 11,12 and 13) then need to read files from 11JUN2017 folder and process the files. Else need to go back to previous date then look for the files.
In the above example, 11JUN2017 does not have all the files so, i need to go back to previous date i.e. 10JUN2017 and look for the files if found then process it. If 10JUN2017 also does not have all files then go back to previous date (max no of days to traverse back is 43 days).
Update
There is some change in my requirement. There will not be date wise folder instead file name itself contains date in YYYYMMdd format and all files will be in single folder, for example, filename will be like as follows BIG_ABCHINE_MATERIAL_2092_11_20170614-150136-243.txt.
So below is what I am trying
public void fileLoadingProcess() {
//Reading text file which contains all the information
in = new BufferedReader(new FileReader("C:\\OrgDetails.txt"));
in.readLine();
String str;
while ((str = in.readLine()) != null) {
values = str.split("\t");
//Adding config property values to Map
if (values[5].equalsIgnoreCase("Y")) {
System.out.println("Active sales org " + values[0]);
if (distChannelMap.containsKey(values[0])) {
list = distChannelMap.get(values[0]);
list.add(Integer.parseInt(values[6]));
} else {
list = new ArrayList<Integer>();
list.add(Integer.parseInt(values[6]));
distChannelMap.put(values[0], list);
}
} else {
System.out.println("Inactive Sales org " + values[0]);
}
}
Set set = distChannelMap.entrySet();
Iterator iterator = set.iterator();
while (iterator.hasNext()) {
Map.Entry mentry = (Map.Entry) iterator.next();
System.out.println("Key --> " + mentry.getKey() + " Value(s) -->" + mentry.getValue());
//Calling this method to check file needs to be processed based on active and inactive states
isFileNeedsToBeProcessed(mentry.getKey().toString(), (ArrayList<Integer>) mentry.getValue());
}
//Check whether this file is exists or not
//
}
public void isFileNeedsToBeProcessed(String salesOrg, ArrayList<Integer> distChannel) {
System.out.println("Previous Day " + new TxtToXMLCommon().previousDate(1));
//String previousDate = new TxtToXMLCommon().getCurrentDate();
for (int i = 0; i < distChannel.size(); i++) {
int x =1;
previousDate = new TxtToXMLCommon().previousDate(x);
System.out.println("Distribution channel " + distChannel.get(i));
File folder = new File(Constants.INPUT_FOLDER);
File[] files = folder.listFiles();
if (files.length == 0) {
_logger.info("***No files present to process***");
return;
} else {
for (int k = 0; k < files.length; k++) {
if (files[k].getName().contains("MATERIAL")) {
if (salesOrg.equalsIgnoreCase(files[k].getName().substring(21, 25))
&& (distChannel.get(i) == Integer.parseInt(files[k].getName().substring(26, 28)))&&
previousDate.equalsIgnoreCase(files[k].getName().substring(29, 37))) {
System.out.println("File present to process " + files[k].getName());
break;
// if () {
// //processingFilesMap.put("MATERIAL", files[k].getName());
// System.out.println("File present to process " + files[k].getName());
// }else{
//
// }
}else{
//previousDate = new TxtToXMLCommon().previousDate(x+1);
}
//previousDate = new TxtToXMLCommon().previousDate(2);
}//end of if(MATERIAL)
}// end of files length
}
}
}
But I am little bit stuck here while finding for the file in a folder(single source folder). Say, as you see, I have put the property file information in the distChannelMap and then for each value I am iterating source folder and trying to find the file which contains org value values[0] and key values[6] which I have stored in arraylist. because for one value 5092 there are 3 channel.
Now if I do not find file matching org value (5092), channel (11) and date (20170615), I need to iterate the folder again with date-1 means value 5092, channel 11 and date 20170614 like this till 43 days. once i found the file I am thinking to put it into map so that I have all files which are ready to process. But I am little bit stuck in going to previous date when i do not the file for particular channel. once this channel is find, I need to do the same process for other channels like 12 and 13 for the org 5092.

I think it’s simplest to take one channel at a time.
Construct the file name from active value and channel number, for example
String.format("HW_%d_ABC_%d.txt", activeValue, channel)
Use LocalDate for the folder names.
DateTimeFormatter dateFormatter = DateTimeFormatter.ofPattern("ddMMMuuuu", Locale.ENGLISH);
LocalDate date = LocalDate.now(ZoneId.systemDefault());
String folderName = date.format(dateFormatter).toUpperCase(Locale.ROOT);
Think twice about the time zone to use. ZoneId.systemDefault() will give you the JVM’s current setting, it may not be what you need. I am typing this on June 10th and understand that in your time zone it’s already June 11th. Would be a pity to miss today’s files because of incorrect time zone.
In a loop, look for the file in the folder. If found, exit the loop. If not found, do
date = date.minusDays(1);
folderName = date.format(dateFormatter).toUpperCase(Locale.ROOT);
This will give you the folder name of the previous day’s folder. Then look there.
This will work across the beginning of the month and the beginning of the year. For the limit of looking 43 days back, either use a counter or set
LocalDate limit = date.minusDays(43);
and then give up when date.isBefore(limit).
Edit If you want to start from the folders and files, look into Files.newDirectoryStream(Path, String). You may use it first to find the folders of this month and the previous two months (that should cover 43 days and more), maybe filter and sort them. Next for each folder you can use the same method to determine which files are present. You will have a challenge keeping track of which files are present in more than one folder and which one is the newest, it can be solved. I am not immediately convinced about the advantage of this approach, but I trust you to make a good decision for yourself.

BasicFileAttributes confusion: what's the real creation time?

I'm automating processing of flat files received from a mainframe and am confused on creating and modified times. The files are created on the mainframe then emailed to the required individuals. The individuals then save the file from email (Outlook) and do whatever it is they do with them.
For that automation, the file will be processed automatically. I need to let the user know when the last file was loaded and give them the option to load a newer file.
My issue is with the times: I'm getting a creation date/time that is later than the lastmodified time. Using this code:
try {
DateFormat utcFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'");
DateFormat cstFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
utcFormat.setTimeZone(TimeZone.getTimeZone("UTC"));
cstFormat.setTimeZone(TimeZone.getTimeZone("America/Chicago"));
Path filePath = Paths.get("C:/data/KWJFLTD.XLS");
BasicFileAttributes basicAttr = Files.readAttributes(filePath, BasicFileAttributes.class);
FileTime creationTime = basicAttr.creationTime();
FileTime modifiedTime = basicAttr.lastModifiedTime();
String cTime = creationTime.toString();
String mTime = modifiedTime.toString();
Date dc = utcFormat.parse(cTime);
Date dm = utcFormat.parse(mTime);
cTime = cstFormat.format(dc);
mTime = cstFormat.format(dm);
System.out.println("Creation Time: " + cTime);
System.out.println("Modified Time: " + mTime);
} catch (IOException ex) {
Logger.getLogger(FAULoad.class.getName()).log(Level.SEVERE, null, ex);
} catch (ParseException ex){
Logger.getLogger(FAULoad.class.getName()).log(Level.SEVERE, null, ex);
}
I get this result:
Creation Time: 2015-06-24 15:25:12
Modified Time: 2015-06-24 02:28:05
I end up with a creation time after the last modified time. Is this because the creation time is changed when it's saved from email? In this situation, given that there's (theoretically) no modification going on since it was generated, would the last modified time be the same as the creation date?

According to Microsoft Documentation (the link relates to Win XP but most like also applies up to Win 10) the modification time is the time the file's content was last written while the creation time is the time a specific file has been created.
Thus if you copy a file the modification date would not change since no content has been written/changed but a new file (the copy) has been created and thus the creation time would be after the modification time.

Returning time of last modification of file to Stata

I'm a Stata user trying to make use of Stata's Java API. I want to use Java to return the time of last modification of a file to Stata. However, I think I'm running into issues with time zones.
Quick background on times in Stata for non-Stata users: Times are represented by the number of milliseconds since January 1, 1960 00:00:00.000. Stata does not have time zones, so I want the return value from the Java function to be the number of milliseconds since midnight on January 1, 1960 in my time zone (America/New York).
Here is my attempt at the Java function (I can post it in its entirety if that would be helpful):
import com.stata.sfi.*;
// Other imports
public class SFileAttribs
{
// Syntax: lastModified(String filename, String scalar_name)
// Stores in the Stata scalar scalar_name the time of last
// modification of filename.
public static int lastModified(String[] args)
{
// Code here...
// Get the time.
FileTime time;
try {
time = (FileTime) Files.getAttribute(path, "lastModifiedTime");
}
catch (IOException e) {
SFIToolkit.errorln("file I/O error on read");
return(692);
/*NOTREACHED*/
}
// Store the time in the scalar.
Calendar cal1960 = Calendar.getInstance();
cal1960.set(1960, 0, 1, 0, 0, 0);
cal1960.set(Calendar.MILLISECOND, 0);
Scalar.setValue(args[1], time.toMillis() - cal1960.getTimeInMillis());
return(0);
}
}
Back in Stata, I run this command:
// Calls SFileAttribs.lastModified("Test.txt", "filetime").
javacall SFileAttribs lastModified, args("Test.txt" "filetime")
However, the result is an hour earlier than it should be:
. display %tc filetime
24oct2013 12:54:36
The time of last modification in my time zone is 13:54, not 12:54.
I am really struggling on this one and would appreciate any suggestions.
EDIT:
Perhaps predictably given that I'm coming from a language with no time zones, I think I messed up daylight savings. In my time zone, it is now DST, but in the same time zone in January 1, 1960, it was not DST. I need to account for that:
// Store the time in the scalar.
Calendar now = Calendar.getInstance();
Calendar cal1960 = Calendar.getInstance();
cal1960.set(1960, 0, 1, 0, 0, 0);
cal1960.set(Calendar.MILLISECOND, 0);
int dstHour = (cal1960.getTimeZone().inDaylightTime(cal1960.getTime()) ? 1 : 0) -
(now.getTimeZone().inDaylightTime(now.getTime()) ? 1 : 0);
cal1960.set(Calendar.HOUR, dstHour);
Scalar.setValue(args[1], time.toMillis() - cal1960.getTimeInMillis());
Is there an easier way to do this?

The community-contributed command filesys is the easiest way to do exactly what you want:
. findfile auto.dta
. filesys `r(fn)', attributes
. return list
macros:
(some output omitted)
r(accessednum) : "1871843910000"
r(modifiednum) : "1745362526000"
r(creatednum) : "1745362526000"
r(accessed) : "25apr2019 20:38:30"
r(modified) : "22apr2015 22:55:26"
r(created) : "22apr2015 22:55:26"
. display %tcHH:MM `r(modifiednum)'
22:55
You can install the filesys command by using the
community-contributed command github:
. net install github, from("https://haghish.github.io/github/")
. github install wbuchanan/StataFileSystem

Calculate client-server time difference in Borland Starteam server 8

Problem. I need a way to find Starteam server time through Starteam Java SDK 8.0. Version of server is 8.0.172 so method Server.getCurrentTime() is not available since it was added only in server version 9.0.
Motivation. My application needs to use views at specific dates. So if there's some difference in system time between client (where the app is running) and server then obtained views are not accurate. In the worst case the client's requested date is in the future for server so the operation results in exception.

After some investigation I haven't found any cleaner solution than using a temporary item. My app requests the item's time of creation and compares it with local time. Here's the method I use to get server time:
public Date getCurrentServerTime() {
Folder rootFolder = project.getDefaultView().getRootFolder();
Topic newItem = (Topic) Item.createItem(project.getTypeNames().TOPIC, rootFolder);
newItem.update();
newItem.remove();
newItem.update();
return newItem.getCreatedTime().createDate();
}

If your StarTeam server is on a Windows box and your code will be executing on a Windows box, you could shell out and execute the NET time command to fetch the time on that machine and then compare it to the local time.
net time \\my_starteam_server_machine_name
which should return:
"Current time at \\my_starteam_server_machine_name is 10/28/2008 2:19 PM"
"The command completed successfully."

We needed to come up with a way of finding the server time for use with CodeCollab. Here is a (longish) C# code sample of how to do it without creating a temporary file. Resolution is 1 second.
static void Main(string[] args)
{
// ServerTime replacement for pre-2006 StarTeam servers.
// Picks a date in the future.
// Gets a view, sets the configuration to the date, and tries to get a property from the root folder.
// If it cannot retrieve the property, the date is too far in the future. Roll back the date to an earlier time.
DateTime StartTime = DateTime.Now;
Server s = new Server("serverAddress", 49201);
s.LogOn("User", "Password");
// Getting a view - doesn't matter which, as long as it is not deleted.
Project p = s.Projects[0];
View v = p.AccessibleViews[0]; // AccessibleViews saves checking permissions.
// Timestep to use when searching. One hour is fairly quick for resolution.
TimeSpan deltaTime = new TimeSpan(1, 0, 0);
deltaTime = new TimeSpan(24 * 365, 0, 0);
// Invalid calls return faster - start a ways in the future.
TimeSpan offset = new TimeSpan(24, 0, 0);
// Times before the view was created are invalid.
DateTime minTime = v.CreatedTime;
DateTime localTime = DateTime.Now;
if (localTime < minTime)
{
System.Console.WriteLine("Current time is older than view creation time: " + minTime);
// If the dates are so dissimilar that the current date is before the creation date,
// it is probably a good idea to use a bigger delta.
deltaTime = new TimeSpan(24 * 365, 0, 0);
// Set the offset to the minimum time and work up from there.
offset = minTime - localTime;
}
// Storage for calculated date.
DateTime testTime;
// Larger divisors converge quicker, but might take longer depending on offset.
const float stepDivisor = 10.0f;
bool foundValid = false;
while (true)
{
localTime = DateTime.Now;
testTime = localTime.Add(offset);
ViewConfiguration vc = ViewConfiguration.CreateFromTime(testTime);
View tempView = new View(v, vc);
System.Console.Write("Testing " + testTime + " (Offset " + (int)offset.TotalSeconds + ") (Delta " + deltaTime.TotalSeconds + "): ");
// Unfortunately, there is no isValid operation. Attempting to
// read a property from an invalid date configuration will
// throw an exception.
// An alternate to this would be proferred.
bool valid = true;
try
{
string testname = tempView.RootFolder.Name;
}
catch (ServerException)
{
System.Console.WriteLine(" InValid");
valid = false;
}
if (valid)
{
System.Console.WriteLine(" Valid");
// If the last check was invalid, the current check is valid, and
// If the change is this small, the time is very close to the server time.
if (foundValid == false && deltaTime.TotalSeconds <= 1)
{
break;
}
foundValid = true;
offset = offset.Add(deltaTime);
}
else
{
offset = offset.Subtract(deltaTime);
// Once a valid time is found, start reducing the timestep.
if (foundValid)
{
foundValid = false;
deltaTime = new TimeSpan(0,0,Math.Max((int)(deltaTime.TotalSeconds / stepDivisor), 1));
}
}
}
System.Console.WriteLine("Run time: " + (DateTime.Now - StartTime).TotalSeconds + " seconds.");
System.Console.WriteLine("The local time is " + localTime);
System.Console.WriteLine("The server time is " + testTime);
System.Console.WriteLine("The server time is offset from the local time by " + offset.TotalSeconds + " seconds.");
}
Output:
Testing 4/9/2009 3:05:40 PM (Offset 86400) (Delta 31536000): InValid
Testing 4/9/2008 3:05:40 PM (Offset -31449600) (Delta 31536000): Valid
...
Testing 4/8/2009 10:05:41 PM (Offset 25200) (Delta 3): InValid
Testing 4/8/2009 10:05:38 PM (Offset 25197) (Delta 1): Valid
Run time: 9.0933426 seconds.
The local time is 4/8/2009 3:05:41 PM
The server time is 4/8/2009 10:05:38 PM
The server time is offset from the local time by 25197 seconds.

<stab_in_the_dark>
I'm not familiar with that SDK but from looking at the API if the server is in a known timezone why not create and an OLEDate object whose date is going to be the client's time rolled appropriately according to the server's timezone?
</stab_in_the_dark>