Multiple clients send request to write a file and expecting a response either success or fail. I would like to describe concisly the work done at server side.
handle the request by servlet class and invoke another class to proceed the further.
FileWriter class is invoked and this class performs follwing the file writing process.
a). create directory under context and write a *.txt file inside directory
b). copy some files from context's existing directory to newly created directory.
c). compress (*.zip) this directory
class FileWriter {
public synchronized writeFile(String contextPath) {
creates a directory & new file under context
copyFiles(path_to_directory);
}
private void copyFiles(String path_to_directory){
copies files to /contextPath/directory/... from existingDirectory;
compressDir( Directory_path ); // to compress the file
}
private void compressDir(String Directory_path) {
compress the newly created directory
}
}
As you can see above in the class that there is one method is synchronized and two methods are private. only synchrnized method is invoking from servlet class others method are invoking inside from the method.
so is this a good / standard way to handling the multiple client request ?
or should i invoke each method directly from servlet class. please correct me and suggest a better way to implement the class.
#Edit : req1 comes and create directory & file e.g.
context/directory_1/file_1.txt
in the mean time req2 comes and checks that directory_1 is existing already so it creates directory_2 e.g. context/directory_2/file_1.txt.
now the second step is to copy the file from context to newly created directory. Let me tell you directory_1 has nothing to do with directory_2
all the newly created directory copies the file from a common_directory e.g. `context/common_directory/... to context/directory_1, context/directory_2'
and third step is to compress the directory : e.g. directory_1.zip, directory_2.zip
Two advices:
Do not name the class same to already existing class in JDK.
Do not chain method calls this way, create one-purpose methods and then
put them together in one method clearly showing your intension.
class FileProcessor /*FileUtil whatever, but not FileWriter */ {
public synchronized writeFile(String contextPath) {
// create a directory & new file under context
copyFiles(contextPath);
compressDir(contextPath); // to compress the file
}
// copies files to /contextPath/directory/... from existingDirectory;
private void copyFiles(String path_to_directory){ }
// compress the newly created directory
private void compressDir(String Directory_path) { }
Looking at the above code, if you calling writeFile from the servlet, your servlet ends up as a single threaded application.
If two are working on two separate directories and separate files and you guaranty that there is no overlap, you should call both methods directly and ditch synchronized. Looks like this is what your situation is. So you can use below approach:
Servlet Code
{
....
String uniqDir = createUniqDir();
copyFiles(uniqDir);
compressDir(uniqDir);
}
Now the whole idea is to create uniq dir name. Now there are many approaches to create uniq dir name. I ll use one which is based on time-stamp.
String createUniqDir() {
// Use SimpleDateFormat or just millis from Date
// We just trying to be as uniq as possible.
String timeStampStr;
Date now = new Date();
timeStampStr = "" + now.getTime(); // If using EPOC
// This soln if you wana use SimpleDateFormat
// SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMdd_HHmmssSSS");
// timeStampStr = sdf.format(dt);
int counter = 1;
String dirToCreateStr = "some_prefix-" + timeStampStr;
File dirToCreate = new File(dirToCreate);
while(!dirToCreate.mkdir()) {
dirToCreateStr = "some_prefix-" + timeStampStr + "-" + counter;
file = new File(dirToCreate);
counter++;
}
return dirToCreateStr;
}
Since we are using mkdir and it is atomic and only return true if it is able to create a uniq dir. This soln is optimized as requesting colliding during a millisecond are way less and we dont need any synchronization overhead.
You can use some counter too for creating uniq name. But if your counter always starts from the beginning (i.e. you are not maintaining its state that too in a thread safe fashion) then you have performance/accuracy issues.
Related
I am a new bee to flink and facing some challenges to solve the below use case
Use Case description:
I will receive a csv file with a timestamp on every single day in some folder say input. The file format would be file_name_dd-mm-yy-hh-mm-ss.csv.
Now my flink pipeline will read this csv file in a row by row fashion and it will be written to my Kafka topic.
Immediately after completion of data reading this file needs to be moved to another folder historic folder.
Why i need this is because : suppose that your ververica server stops either abruptly or manually and if you have all the processed files lying at the same location then after the ververica restart flink will re read all the files that it had processed earlier. So to prevent this scenario those files needs to be immediately move already read files to another location.
I googled a lot but did not find anything so can you guide me to achieve this.
Let me know if anything else is required.
Out of the box Flink provides the facility to monitor directory for new files and read them - via StreamExecutionEnvironment.getExecutionEnvironment.readFile (see similar stack overflow threads for examples - How to read newly added file in a directory in Flink / Monitoring directory for new files with Flink for data streams , etc.)
Looking into the source code of the readFile function, it calls for createFileInput() method, which simply instantiates ContinuousFileMonitoringFunction, ContinuousFileReaderOperatorFactory and configures the source -
addSource(monitoringFunction, sourceName, null, boundedness)
.transform("Split Reader: " + sourceName, typeInfo, factory);
ContinuousFileMonitoringFunction is actually a place where most of the logic happens.
So, if I were to implement your requirement, I would extend the functionality of ContinuousFileMonitoringFunction with my own logic of moving the processed file into the history folder and constructed the source from this function.
Given that the run method performs the read and forwarding inside the checkpointLock -
synchronized (checkpointLock) {
monitorDirAndForwardSplits(fileSystem, context);
}
I would say it's safe to move to historic folder on checkpoint completion files which have the modification day older then globalModificationTime, which is updated in monitorDirAndForwardSplits on splits collecting.
That said, I would extend the ContinuousFileMonitoringFunction class and implement the CheckpointListener interface, and in notifyCheckpointComplete would move the already processed files to historic folder:
public class ArchivingContinuousFileMonitoringFunction<OUT> extends ContinuousFileMonitoringFunction<OUT> implements CheckpointListener {
...
#Override
public void notifyCheckpointComplete(long checkpointId) throws Exception {
Map<Path, FileStatus> eligibleFiles = listEligibleForArchiveFiles(fs, new Path(path));
// do move logic
}
/**
* Returns the paths of the files already processed.
*
* #param fileSystem The filesystem where the monitored directory resides.
*/
private Map<Path, FileStatus> listEligibleForArchiveFiles(FileSystem fileSystem, Path path) {
final FileStatus[] statuses;
try {
statuses = fileSystem.listStatus(path);
} catch (IOException e) {
// we may run into an IOException if files are moved while listing their status
// delay the check for eligible files in this case
return Collections.emptyMap();
}
if (statuses == null) {
LOG.warn("Path does not exist: {}", path);
return Collections.emptyMap();
} else {
Map<Path, FileStatus> files = new HashMap<>();
// handle the new files
for (FileStatus status : statuses) {
if (!status.isDir()) {
Path filePath = status.getPath();
long modificationTime = status.getModificationTime();
if (shouldIgnore(filePath, modificationTime)) {
files.put(filePath, status);
}
} else if (format.getNestedFileEnumeration() && format.acceptFile(status)) {
files.putAll(listEligibleForArchiveFiles(fileSystem, status.getPath()));
}
}
return files;
}
}
}
and then define the data stream manually with the custom function:
ContinuousFileMonitoringFunction<OUT> monitoringFunction =
new ArchivingContinuousFileMonitoringFunction <>(
inputFormat, monitoringMode, getParallelism(), interval);
ContinuousFileReaderOperatorFactory<OUT, TimestampedFileInputSplit> factory = new ContinuousFileReaderOperatorFactory<>(inputFormat);
final Boundedness boundedness = Boundedness.CONTINUOUS_UNBOUNDED;
env.addSource(monitoringFunction, sourceName, null, boundedness)
.transform("Split Reader: " + sourceName, typeInfo, factory);
Flink itself does not provide a solution for doing this. You might need to build something yourself, or find a workflow tool that can be configured to handle this.
You can ask about this on the flink user mailing list. I know others have written scripts to do this; perhaps someone can share a solution.
I have a requirement of implementing a Watch Service on a folder. This is straight forward approach of using Java7's watch service. I have successfully done it, I am able to capture events whenever a file is created/updated/deleted on the folder where I have been watching. The problem here is it is not applicable for contents of sub folders and it is clearly written in the documentation. My requirement is to watch over contents of sub folder as well. This is not possible using the above approach unless I write a loop over all the sub folders manually and listen to each and every folder, this I think leads to some memory leak if not programmed well. Hence I am going with what spring suggested in the newer release explained here This is very clear approach which I have seen for WatchService. The problem here is this will listen to only ENTRY_CREATE events i.e., only the events where we have created the file and this can be at any level. This is not working when I change the file or delete the file. How should we go ahead in this case.
public static void watchFolderTree(String pathStr)
throws Exception
{
long waitTime = 10000;
WatchServiceDirectoryScanner scanner = new WatchServiceDirectoryScanner(pathStr);
scanner.start();
List<File> changedFiles = null;
while(true)
{
changedFiles = scanner.listFiles(new File(pathStr));
if(changedFiles.size() > 0)
{
System.out.println("There is a file ");
}
Thread.sleep(waitTime);
}
}
References :
Monitor subfolders with a Java watch service
JAVA 7 watch service
In my application i have a set of file which contains some information.Now what i have to do i have to process the files and among those files which are duplicate i have to skip them so to do that i have used file CRC to check which files are processed and which are not,so now in the case of duplicate checking i have to store the file CRC to some where because when today's processing is over then i will have to process the file again tomorrow.Then if some file are duplicate of today those should be skipped. so now what i have done in my code is..
filesSize += fileInf.srcFileSize;
// if file if already polled and under processing or in execution queue
if (filesInQueue.get(fileInf.srcFileCRC) != null) {
System.out.println("Skipping queued file: " + fileInf.srcFileName);
ifCRCExist.add(fileInf.srcFileCRC);
fileInf.srcFileCRC, is the required fileCRC which i have added to the list now i have store the list non persistently some where which i can use to check the record later..I think i am clear now..Please any one help me..
Each instance in Java is transient by nature. Are you in search of a Singleton? You may provide the ArrayList as a static field of some Class. For example:
public class InMemoryStore {
public static final List<String> MY_SINGLETON_LIST;
static {
MY_SINGLETON_LIST = ...
}
}
Or you may do it by creating a bean with Spring, for example.
If you want a non persistent storage means not in DB or XML then make a class
and create a static list member. Now you can use it as many times by initializing only once. Just like Singleton pattern.
Part of a program I am working on requires looking up preprocessor macros by name, and then getting their values. I opted to use the CDT Indexer API. In order to make sure I am on the right track, I wrote a test method that does nothing but create a simple C file and confirm that it can find certain symbols in the index. However, I failed to get that test to run properly. Attempting to use IIndex.findBindings(char[], IndexFilter, IProgressMonitor) returns empty arrays for symbols that I know exist in the AST because they are part of the example file in the test method.
I can't post the exact test method because I use some custom classes and it would be overkill to post all of them, so I will just post the important code. First, my example file:
final String exampleCode =
"#define HEAVY 20\n" +
"#define TEST 5\n" +
"void function() { }\n" +
"int main() { return 0; }\n";
IFile exampleFile = testProject.getFile("findCodeFromIndex.c");
exampleFile.create(new ByteArrayInputStream(exampleCode.getBytes("UTF-8") ), true, null);
I have a custom class that automatically gets the IASTTranslationUnit from that file. The translation unit is fine (I can see the nodes making up everything except the macros). I get the index from that AST, and the code I use to look up in the index is
try {
index.acquireReadLock();
returnBinding = index.findBindings(name.toCharArray(), IndexFilter.ALL, null);
... catch stuff...
} finally {
index.releaseReadLock();
}
Where 'name' is going to be either "HEAVY", "TEST", or "function". None of them are found, despite existing in the example test c file.
I am guessing that the issue is the index is not rebuilt, which causes findBindings to return an empty array even if I know the given variable name exists in the AST.
My current attempt to start up the indexer looks like this:
final ICProject cProject = CoreModel.getDefault().getCModel().getCProject(testProject.getName());
CCorePlugin.getIndexManager().reindex(cProject);
CCorePlugin.getIndexManager().joinIndexer(IIndexManager.FOREVER, new NullProgressMonitor() );
Question Breakdown:
1) Is my method for searching the index sound?
2) If the issue is the index needing to be rebuilt, how should I properly force the index to be up to date for my test methods? Otherwise, what exactly is the reason I am not resolving the bindings for macros/functions I know exist?
I solved my own issue so I will post it here. I was correct in my comment that the lack of the project being a proper C project hindered the Indexer from working properly, however I also discovered I had to use a different method in the indexer to get the macros I needed.
Setting up the test enviornment:
Here is the code I have that creates a basic C project. The only purpose it serves is to allow the indexer to work for test methods. Still, it is large:
public static IProject createBareCProject(String name) throws Exception {
IProject bareProjectHandle = ResourcesPlugin.getWorkspace().getRoot().getProject(name);
IProjectDescription description =
bareProjectHandle.getWorkspace().newProjectDescription("TestProject");
description.setLocationURI(bareProjectHandle.getLocationURI() );
IProject bareProject =
CCorePlugin.getDefault().createCDTProject(description, bareProjectHandle, new NullProgressMonitor() );
IManagedBuildInfo buildInfo = ManagedBuildManager.createBuildInfo(bareProject);
IManagedProject projectManaged =
ManagedBuildManager
.createManagedProject(bareProject,
ManagedBuildManager.getExtensionProjectType("cdt.managedbuild.target.gnu.mingw.exe") );
List<IConfiguration> configs = getValidConfigsForPlatform();
IConfiguration config =
projectManaged.createConfiguration(
configs.get(0),
ManagedBuildManager.calculateChildId(configs.get(0).getId(), null));
ICProjectDescription cDescription =
CoreModel.getDefault().getProjectDescriptionManager().createProjectDescription(bareProject, false);
ICConfigurationDescription cConfigDescription =
cDescription.createConfiguration(ManagedBuildManager.CFG_DATA_PROVIDER_ID, config.getConfigurationData() );
cDescription.setActiveConfiguration(cConfigDescription);
cConfigDescription.setSourceEntries(null);
IFolder srcFolder = bareProject.getFolder("src");
srcFolder.create(true, true, null);
ICSourceEntry srcFolderEntry = new CSourceEntry(srcFolder, null, ICSettingEntry.RESOLVED);
cConfigDescription.setSourceEntries(new ICSourceEntry[] { srcFolderEntry });
buildInfo.setManagedProject(projectManaged);
cDescription.setCdtProjectCreated();
IIndexManager indexMgr = CCorePlugin.getIndexManager();
ICProject cProject = CoreModel.getDefault().getCModel().getCProject(bareProject.getName() );
indexMgr.setIndexerId(cProject, IPDOMManager.ID_FAST_INDEXER);
CoreModel.getDefault().setProjectDescription(bareProject, cDescription);
ManagedBuildManager.setDefaultConfiguration(bareProject, config );
ManagedBuildManager.setSelectedConfiguration(bareProject, config );
ManagedBuildManager.setNewProjectVersion(bareProject);
ManagedBuildManager.saveBuildInfo(bareProject, true);
return bareProject;
}
As I discovered when debugging, it is indeed important to set proper configurations and descriptions as the indexer was postponed so long as the project didn't have those features set. To get the configurations for the platform as a starting point for an initial configuration:
public static List<IConfiguration> getValidConfigsForPlatform() {
List<IConfiguration> configurations =
new ArrayList<IConfiguration>();
for (IConfiguration cfg : ManagedBuildManager.getExtensionConfigurations() ) {
IToolChain currentToolChain =
cfg.getToolChain();
if ( (currentToolChain != null ) &&
(ManagedBuildManager.isPlatformOk(currentToolChain) ) &&
(currentToolChain.isSupported() ) ) {
configurations.add(cfg);
}
}
return configurations;
}
This basically answers the second part of the question, and thus I can create a c project for the purposes of testing code using the index. The testing code still needs to do some work.
Testing Code
I create files in the the "src" folder in the project (created in the above code), and I either have to name them .c, or if I want to name them .h have them included by some .c file (otherwise the indexer won't see them). Finally, I can populate the files with some test code. To answer number 1,
I need to block on both auto refresh jobs in Eclipse and then the index:
public static void forceIndexUpdate(IProject project) throws Exception {
ICProject cProject = CoreModel.getDefault().create(project);
Job.getJobManager().join(ResourcesPlugin.FAMILY_AUTO_REFRESH, null);
CCorePlugin.getIndexManager().reindex(cProject);
CCorePlugin.getIndexManager().joinIndexer(IIndexManager.FOREVER, new NullProgressMonitor() );
assertTrue(CCorePlugin.getIndexManager().isIndexerIdle() );
assertFalse(CCorePlugin.getIndexManager().isIndexerSetupPostponed(cProject));
}
After I change the files in the project. This makes sure Eclipse is refreshed, and then makes sure the indexer completes without being postponed. Finally, I can run tests depending on the indexer.
And the last point, I was wrong about using IBinding. The correct way in which I was able to get the macros was using the method IIndex.findMacros(char[] name, IndexFilter filter, IProgressMonitor monitor)
I hope this helps at least someone out there. I would also appreciate it if there was some feedback regarding the validity of this solution, as this is simply the first solution I managed to create that worked. Just to confirm, I am not testing the indexer itself, but rather code I wrote that uses the indexer and I want to test it under as realistic conditions as I can given how critical it is.
I'm using dcevm + run-jetty-run + livereload , try to develop a web app without restarting jetty when modifing java sources.
Everything works fine. When I modified a java class, livereload monitored the change, and triggered the browser refreshing opened pages to see the modified result.
But I found it still not that convenient: When browser reloads, dcevm and jetty may have not reloaded that modified classes yet. I have to manually refresh the page again, but I'm not sure if it shows the modified result this time, without checking the content carefully.
So I wonder is there any way to let jetty blocks the request when I modified some classes and dcevm is reloading. It will make sure the pages displayed are always modified.
It's maybe too hacky for your palate, but you could insert a static initialization snippet in your Java sources to update a known, separate file after reloading. Than livereload can watch that separate file instead of letting it work directly on .java sources.
Something along the lines of:
public class ReloadUtils {
public static void notifyUpdate(String className) {
String baseDir = System.getProperty("DEV_MODE_BASEDIR") + "/";
File file = new File(baseDir + className + ".updated");
FileWriter fw = new FileWriter(file.getAbsoluteFile(), false); // overwrite instead of append
BufferedWriter bw = new BufferedWriter(fw);
bw.write(Long.toString(System.currentTimeMillis()));
bw.close();
}
}
public class Reloadable {
private final static boolean DEV_MODE = System.getProperty("DEV_MODE").equals("true");
static {
// static finals trigger most compilers to remove the statements in this case
if (DEV_MODE) {
ReloadUtils.notifyUpdate(Reloadable.class.getName());
}
}
/* lots of useful stuff */
}