My company processes data that comes from our clients. Our clients submit data by pushing it to an sftp server we maintain. Our code (mostly in Java, Scala and Node.js with a little of Python/shell) then hits this servers in read-only mode and does bunch of data processing after downloading these files.
We are frustrated by this system because not only Java libraries aren't that great for doing sftp but even the node one does not work in Windows which some developers use so we have some hacky shell scripts to sync files from sftp.
What are some good architectural patterns to solve this problem? We want a secure internal file server that can be accessed easily by good OS agnostic libraries and that has good tools for browsing through a file-viewer?
Expanding on Brad's question, "why make you internal interface SFTP"? ...
I'd run rsync against your SFTP server to keep a working copy of the client files in your preferred environment. From there go with whatever tools make your crew most comfortable.
Related
I have developed an android application and i need to set a server for sending and receiving data from clients. The server must also be able to run python script for data mining.I want to code the server in java(for analysing the data obtained from mining and handling the client requests).
what will be the way to set the server for my purpose. I have read that servers can be set up in eclipse or installing apache in linux in a virtual machine. will anyone of them fit my need or is there any other better alternative.
Well, it depends on your needs. If you want to run Python scripts from Java, I don't really see why you would code your server-side logic in Java and then use Python data-mining scripts (you might have some problems with execution of Python scripts through Java and server setup). I know that Python is much better in data-mining, so stick with it. Consider Python for server-side logic. In case you want to run data-mining script manually that doesn't really mater, so you can separate the languages.
In case you want to separate it into two languages, then you must install an application server for Java. There are plenty of options for that, even open-sourced (Wildfly, Glassfish, ...):
https://blog.idrsolutions.com/2015/04/top-10-open-source-java-and-javaee-application-servers/
You can set most of them in Eclipse for development, for production releases they can be installed either on Unix or Windows. The Apache option you mentioned is probably Apache Tomcat application server. You can read more about Java application server set-up in Eclipse:
http://www.eclipse.org/webtools/community/education/web/t320/Configuring_an_Application_Server_in_Eclipse.pdf
Depending upon the requirement, you can setup the server in many ways in any operating system like linux (fedora, ubuntu), windows, mac osx etc. There are many java web frameworks to follow to make your own server and provide API for mobile.
These are some java web frameworks:
playframework
Spring MVC
Grails
You can setup above and other frameworks and can setup in any enviorment. For initial, you can setup in your local system, then later you can upload into your remote server.
For setup, please see the respective framework's documentation.
It would be wise to choose json for communication between mobile and server. You can consume the API in android using default HttpURLConnection or you can follow other opensource http connection library like retrofit, asynchttp etc.
I am relatively new to web development, but I have some C++/Java experience. I have got the following conversion to do:
Current:
Desktop Application (Automation Software) developed in C# that communicates with remote PLC (Controller that overlooks different sensors in realtime) using TCP Sockets over the Web.
My Idea:
Convert the application into a server side software that will still communicate with the PLC over TCP/Socket. And use a browser to operate it, so the remote site can be monitored and controlled from any computer in our Intranet (possibly Tablets in the future).
Motive for doing it:
We had a computer fault which left the operators without control.
The new app:
I am planning on writing the server app using Java and OOP (so far no problem). And use HTML/CSS/Javascript for the WebApp and AJAX to update the page.
But I am still lost at how can I transport all this data between them in a proper and decent manner. I have read about SOAP and JSON in this Post. Although, I am not sure if I need to use them at all, is it a good solution to use either JSON or SOAP? Or is there any other solution that you may recommend?
Cheers,
Leo
If you consider skipping the development work to convert your app into a server-side software and just go for a third party solution, I suggest you take a look to Thinfinity VirtualUI.
"...offers a GUI remoting solution for in-house Windows desktop
developments, allowing them to be delivered as Windows/HTML5 dual-platform applications
simply by adding one line of code.
These Windows applications can keep their standard desktop environment behavior and,
alternatively, be accessed remotely from any modern web browser in a multi-user,
multi-instance fashion when hosted on a Thinfinity VirtualUI Server environment."
https://www.cybelesoft.com/docs/thinfinity_virtualui_whitepaper.pdf
SOAP is for defining public APIs that are published on the internet for other people to use, which does not seem like your use case. It is not particularly awesome to have to deal with it from inside a browser either, although there are javascript SOAP-client libraries. There is also going to be a fair bit more overhead on the server side parsing and validating XML than de/serializing between JSON and POJOs.
JSON is much easier to deal with in a browser, being natively understood and all that. Everything you need is built into the core of jQuery, no dependence on plugins that may have unknown levels of future support.
We have an enterprise application which saves some files on a network share (\\physicalfileserver\files\). We need to use a network share because the application is installed on multiple application server instances distributed over multiple physical server connected via LAN. Sometimes, we need to create a copy of certain files on the network share itself and save it in same the directory (on the same network share) with a different name or in a new directory (Again on the same network share). What is the most efficient way to create a copy of the file? I do not want to create any IO traffic on the network as the source and destination files reside on the same server. It also increases the time taken to create a copy of the file. This however, is difficult from What I have seen so far.
Reading files in input stream and writing them back on the server through an output stream with a different path/name.
FileUtils.copyFile() from apache commons.
NIO Channels library of JAVA (java.nio.channels.FileChannel.transferFrom())
However, all of them seem to copy the file to the server executing the program and then copy the data back to network share which isn't really required.
What can the gurus suggest to be the best possible approach to handle this situation? I can only use JAVA 1.5/1.6 due to some restrictions but am open to suggestions using higher java version as well.
The application is installed on a Windows 2008 server and the network share is also present on a Windows 2008 server.
Thanks
If you want to avoid network traffic you could write a small server in Java that would allow you to copy files locally (this would run on file server itself). Securing this would be quite important as this could pose huge security risk if somebody could use it to overwrite system files.
How do you decide which protocol you've to use to transfer/move/copy files across servers?
I'm writing a service to copy files from a Windows server file system to a oracle-based database.
How about the File Transfer Protocol for transferring files.
Since Oracle doesn't store files, I suspect you need more than just transferring files. I suspect you will need to use JDBC in any case.
I am not sure it's clear what you are building and why is it it a java question?
Do you need to implement a custom protocol in Java to transfer files for a web service you are building? Or can you use an off the shelf product and a bunch of shell scripts to take care of the file transfer?
If it's a custom thing you need to build in Java (the question is tagged with java) then more questions:
Is it a web app that provides end-users with GUI allowing them to submit files?
- you need to take care of file transfer using http post. Apache httpclient library can help you there.
Is it a SOAP Web Service that saves incoming data into oracle?
- again then SOAP implementations provide answer to that.
Is it a command-line tool running on Windows machine that talk to an Oracle database ?
-then protocol is irrelevant, as you can just read the files using file.io libs and use jdbc to save them into oracle.
Is it a custom server with a yet-to-be built protocol and subsequently a custom client that will use that protocol?
- then I would use binary over https for my implementation of the custom server/service.
4.If you are doing everything using COTS, FTP is simple and fast, but it's not secure and it's not reliable. SFTP will almost always going to be better.
http://geekswithblogs.net/bvamsi/archive/2006/03/23/73147.aspx
5.If your question is really about how to save files into Oracle database using jdbc, then you need to find a way to stream them (or you will run out of memory).
E.g. this may be helpful http://www.coderanch.com/t/415625/JDBC/databases/save-files-oracle-database-java
We are planning to develop remote desktop sharing feature in out web application (written in Java) where one user will be able to connect other users desktop with full access.
I don't want to use applications like team-viewer. It has to be some kind of web browser plug-in or feature.
I found following couple of solutions like Adobe Connect & Java.awt.Robot class.
Can anyone please let me know best option available?
It will be great if there is any open source project which can satisfy these requirements.
Try RealVNC out of the box, it has a built in Java web-client that allows full control. Most of the standard flavors of Linux run it, this, of course, totally depends on your user base and what you're trying to do but for an internal tool it is perfect.
http://www.realvnc.com/products/free/4.1/man/vncserver.html
The above link is the documentation for VNC server. It can be configured to one-session per port, it is just a matter of making sure the ports are correct. It sounds like you're reinventing the wheel on this one. There's a horde of good free products out there to handle this. For an infinite user-base this probably wouldn't be the best solution. For an internal tool it works wonders.