I need to create a program that sets an int to 01 at the beginning of each day. Every time the file is run, the int increments up until the next day. This int will be inserted into a file name, e.g. FileName(insertdatehere)01.txt, FileName(insertdatehere02.txt, FileName(insertdatehere)03.txt, etc...
I was wondering if this is possible:
-Checking if the file already exists, and if it does, then the int value will increment. This will work since the file name has the date on it, so that each day, a new file name will be created anyways.
Am I going in the right direction, or should I completely rethink this question?
Sorry if this isn't clear, if you need me to clarify, I will.
Your ideas seem correct and doing it in this way would probably work well.
Something to watch out for would be if two of the same process exist and both try to create a file assuming it doesn't exist.
As long as you consider this case, and your process runs reliably throughout the day (and you don't fall into timezone traps), you should be good to go.
Did you try using java.util.Date class for setting time stamp , date etc.
You can set date whenever the file is opened in some other file or you can set the same value at some specific place in the same file.
Then whenever you open the file again you can compare and check the earlier date which is already set.
This would surely help you.
Firstly try it yourself and then if you remain unable to do the same post issues whichever you face.
Your Idea is good .
But for the case when there does not exists any file and two processes are trying to create it simultaniously with same name then the problem with occur .
Above problem you can solve by using Synchronization in Java, so that code block(which contains logic for check file if it exists and creates new file) cannot be accessed simultaneously .
Related
I have a Talend job that I need to pull down an XML file from an sFTP server to then be processed into an Oracle database. The date of the XML extraction is in the file name, for example "FileNameHere_Outbound_201407092215.xml", which I believe is yyyyMMddhhmm formatting. The beginning portion where "FileNameHere" is the same for all the files. I need to be able to read the date from the end of the file name and only pull that one down from the server to be processed.
I am not sure how to do this with FTP. I've previously used tFilelist to order the items by date descending, but that is not an option with FTP. I know it probably has some Java involved in how to pull the portion of the File Name out, but I'm not very Java-literate. I can manage though with a bit of assistance.
Does anyone have any insight on how to only download the most recent file from an FTP?
There's a tFTPFileList component on the palette. That should give you a list of all the files on the FTP location. From here you then want to parse out the time stamp which could be done with a regular expression or alternatively by substringing it depending on which you feel more comfortable with.
Then it's just a case of sorting by the extracted time stamp and then that gives you the newest file name so you can then go fetch that specific file.
Here's an outline of an overly laborious way to get this done but it works. You should be able to tweak this easily yourself too:
In the above job design I've gone for a tFileList rather than a tFTPFileList because I don't have an example FTP location to play with for testing here. The premise stays the same although this would be pointless with a real tFileList due to the ability to sort by modified date (among other options).
We start off by running the tFileList/tFTPFileList component to iterate through all the files (it's possible to file mask these too to limit what you return here) in the location. We then read this in iteratively to a tFixedFlowInput component which allows us to retrieve the values from the globalMap as the tFileList/tFTPFileList iterates through each file:
I've listed everything that the tFileList provides (you can see the options by pressing ctrl+space) but you only really need the file name and potentially the file path or file directory. From here we then throw everything into a buffer with a tBufferOutput component so that we can gather every iteration of the location.
Once the tFileList/tFTPFileList has iterated through every file in the directory it then triggers the next sub job with an OnSubjobOk link where we start by reading the completed buffer back in with a tBufferInput component. At this point I've started scattering tLogRow components throughout the flow so I can better visualise the data at each step.
After this we then use a tExtractRegexFields component to extract the date time stamp from the file name:
Here, I am using the following regex "^.+?_Outbound_([0-9]{12})\\.xml$" to capture the date time stamp. It relies on the file name being a combination of any characters, followed by the string literal _Outbound_, then followed by the date time stamp that we want to capture (which is represented by 12 numeric characters) and then finished with .xml.
We also add a column to our schema to accommodate the captured date time stamp like so:
As the extra column is a date time stamp of the form yyyyMMddhhmm we can specify this directly here and use it as a date object from then on.
From here we simply sort by date descending on the extracted date time stamp column and then use a tSampleRow to take only the first row of the flow of data as per the guidelines on the component configuration.
To finish this job you would then output the target file path to the globalMap (either in a tJavaRow or using a tFlowToIterate that will automatically do this for you) and then use the globalMap stored file path in the tFTPFileGet's file mask setting:
I built a camera app. My problem is that I want to continue the logical naming the default camera app uses.
EG: The camera app produces files such as:
IMG_20130104_033852
IMG_20130104_033853
If these are the only contents of my /camera folder, I should theoretically name my photo IMG_20130104_033854. Should I look for the image with the "biggest" name and add a number or is there a better solution for this problem?
I would suggest building the name up until the point you get to the last sequence of numbers, and then you could look for the biggest number and add one to it if you feel like it.
So for instance, based off of today's date, you could build the IMG_20130121_ part. You could do File [] fList = f.listFiles();, where f is pointing to the /camera/ directory. From there, you could check the files by doing fileName.startsWith("IMG_20130121_") in an IF statement.
From there, do a substring to grab the string-numbers after the last underscore (_), convert to Integer, and loop through until you finish comparing all the Integers.
I did this for folder names. A standalone cd burning device creates folders that start with ezb_, followed by a number. I can paste my looping code tomorrow if this info wasn't enough.
As a sidenote, if you were to start your own naming algorithm after IMG_20130121_, like adding an s or something, you'd still have to either keep track or look for the biggest number. You might as well just implement something like what I tried to explain. Let me know if I need to clarify!
The Problem:
I have numerous files that contain Apache web server log entries. Those entries are not in date time order and are scattered across the files. I am trying to use Pig to read a day's worth of files, group and order the log entries by date time, then write them to files named for the day and hour of the entries it contains.
Setup:
Once I have imported my files, I am using Regex to get the date field, then I am truncating it to hour. This produces a set that has the record in one field, and the date truncated to hour in another. From here I am grouping on the date-hour field.
First Attempt:
My first thought was to use the STORE command while iterating through my groups using a FOREACH and quickly found out that is not cool with Pig.
Second Attempt:
My second try was to use the MultiStorage() method in the piggybank which worked great until I looked at the file. The problem is that MulitStorage wants to write all fields to the file, including the field I used to group on. What I really want is just the original record written to the file.
The Question:
So...am I using Pig for something it is not intended for, or is there a better way for me to approach this problem using Pig? Now that I have this question out there, I will work on a simple code example to further explain my problem. Once I have it, I will post it here. Thanks in advance.
Out of the box, Pig doesn't have a lot of functionality. It does the basic stuff, but more times than not I find myself having to write custom UDFs or load/store funcs to get form 95% of the way there to 100% of the way there. I usually find it worth it since just writing a small store function is a lot less Java than a whole MapReduce program.
Your second attempt is really close to what I would do. You should either copy/paste the source code for MultiStorage or use inheritance as a starting point. Then, modify the putNext method to strip out the group value, but still write to that file. Unfortunately, Tuple doesn't have a remove or delete method, so you'll have to rewrite the entire tuple. Or, if all you have is the original string, just pull that out and output that wrapped in a Tuple.
Some general documentation on writing Load/Store functions in case you need a bit more help: http://pig.apache.org/docs/r0.10.0/udf.html#load-store-functions
I currently have an Android application that displays a schedule for a ferry boat. The application can display the full schedule (just a giant list), but the selling point in the application is it will display when the next two ferries are departing and how long from the current time that departure is.
I am relatively new to Java and currently use large Switch() statements in my code. Basically it gets the current phone time and compares it to all of the times in the schedule at which point it displays the next two departure times and then calculates the difference between current time and the departure times.
I am sure that a switch statement is not the best idea for speed purposes as well as code changing purposes. For example if one time changes its a bunch of lines of code to go in and fix for that one time change. Also if the entire schedule changes everyone has to update their app for the time change to take effect. My ideal situation would be to store a file somewhere on my webserver that could be downloaded and inserted into a hashmap (I think is the correct term) that would load the new schedule if there was a time change.
Not sure how confusing this is, but it would be greatly appreciated if someone could explain how I might use a hashmap or something else you might recommend to get this task accomplished. Currently the variables are the two ferry terminals as well as the day of the week since the schedule changes per day (monday, tues-friday, saturday, sunday).
Below is a screenshot of the application so you can understand it if my post wasn't clear. Thank you in advance.
Screenshot:
Store the schedule objects in a sorted array. You can then binary search the array for the first value greater than the current time. You'll probably use some parent array consisting of the location and applicable day of the week.
You can easily write that kind of data structure to a file that is read & parsed by the application for updates instead of being compiled into the code.
Details of this? First, understand resources in Android. If no updated schedule exists, fall back to the default resource.
Second, use an HTTP head request to check if a newer file exists. If it does, parse, download & save state. Saving Android Activity state using Save Instance State.
Finally, XML is handy for data distribution, even if it's not fast. Everybody understands it and it's easy to update or hand off.
<ferry location=0 time=2045>
<day>1</day>
<day>2</day>
<day>3</day>
<day>4</day>
<day>5</day>
</ferry>
<ferry location=0 time=0800>
<day>6</day>
</ferry>
You will need something like a database to hold the schedule data. That will help you to seperate code from data. I'm not familiar with Android but i think there is a interface to sqlite database on the device.
Further, as this is an application on a small device you may connect to the schedule database on a server thru the internet connection. That way you have to maintain schedule data only in one place (on the server) and clients will use always up to date data.
java.io.File doesn't provide a way to get a file's creation date. You can get file.lastModified, but not anything like dateCreated.
Java 7 adds the excellent java.nio.file package with access to date created, but it's not out yet.
My question: what's the best way to get a file's date created from Java on a UNIX/OSX system? I suppose it's executing a shell script, but my command line skills are pretty weak. So if shell scripting's the way to go, if you could provide a full example I'd be very grateful.
Thanks!
From the command line, the easiest way to get the creation date is with mdls. I think you'd want to do /usr/bin/mdls -name kMDItemFSCreationDate $filename (where $filename is the file you're asking about). You can specify multiple filenames, but that might make it harder to parse the output.
Unix doesn't have a "creation date". The only dates stored on files are modification date, which stores the time the file contents where changed; the access date, which stores the last time a file was read; and the "change date", which stores the last time the file's metadata was changed. (The metadata contains things like permission bits, ownership, etc.)
If you review the structure supported by the stat(2) API, you can see the three timespec's.
You can't - it isn't stored anywhere
It actually depends on the file system, and most file systems don't provide such feature as storing a file "birth time". You need to check the filesystem.