Loading Files Locally Using GSUtil - java

Does anyone know a good way to load a set of files locally into the Java dev_appserver's emulated Cloud Storage.
This didn't work:
$ gsutil rsync gs://mybucket http://localhost:8888/mybucket
InvalidUrlError: Unrecognized scheme "http".
I'm open to suggestions on either:
How to load a bunch of files locally (preferably through gsutil)
How to point my local dev_appserver to a non-emulated bucket at Google
This is painful to test things out locally without proper data. I'm trying to write some transformations to load data into BigQuery (from Datastore backups) and it won't be possible without some real data.

"How to point my local dev_appserver to a non-emulated bucket at Google": it's not documented all that clearly, but it is implemented in the dev_appserver and cloudstorage.
To verify what I'm saying, first svn checkout http://appengine-gcs-client.googlecode.com/svn/trunk/python gcs-client to get cloudstorage's source code onto your machine (you'll need to install subversion if you don't have it already, but, that's free too:-).
Then, cd gcs-client/src/cloudstorage/ and look at storage_api.py. In the very first function _get_storage_api, the docstring says:
On dev appserver, this instance by default will talk to a local stub
unless common.ACCESS_TOKEN is set. That token will be used to talk
to the real GCS.
So, look at common.py, and again in the first function, set_access_token, you'll see:
Args:
access_token: you can get one by run 'gsutil -d ls' and copy the
str after 'Bearer'.
So there you are -- in every entry to your app (best in appengine_config.py in your root directory), import cloudstorage's common module, then **if and only if you're on dev_appserver[*] call
common.set_access_token('whatever_the_token')
using as argument string the one you get by run 'gsutil -d ls', right after Bearer i.e among much else you'll spot something like (faking and way shortening the actual value...:-):
Bearer xy15.WKXJQEzXPQQy2dt7qK9\r\n
in which case you'd be calling
common.set_access_token('xy15.WKXJQEzXPQQy2dt7qK9')
[*] many ways to find out if you're on dev_appserver, e.g see GAE: python code to check if i'm on dev_appserver or deployed to appspot .

Related

Prevent jar from running on network drive

An executable jar is distributed to users per network drive. I want to prevent users from running it directly on the network because it causes performance and other problems.
One solution can be to check the execution path and compare it with the network drive path, but what if the network drive path is not known in advance? Is there any other possibility?
I don't think you can do this in pure Java. I'm pretty sure you will need to either run an external command or make a call into native code to detect the whether the JAR file is on a network share.
The first part you need to do is to obtain the absolute path for the JAR file:
Class mainClass = MyMainClass.class;
File jarFile = new File(mainClass.getProtectionDomain()
.getCodeSource()
.getLocation()
.toURI());
I found a "gist" on github written by Aaron Digulla that does the second rest of the task:
https://gist.github.com/digulla/31eed31c7ead29ffc7a30aaf87131def
Aaron's code takes a Windows absolute path, extracts the drive letter and then runs the Windows net use <drive-letter>: command to test whether the drive is a share drive.
An alternative would be to implement the logic a batch file does the following:
Extract the drive name for the JAR file
Use net use ... to test for a share drive
Use the same technique to check that the Java installation is not on a share drive either1.
Use the java command to start the application
Finally, is probably a bad idea to refuse to run on a share drive. The user may have reasons to do this that you are not aware of. I would suggest prompting the user and giving the user the option of continuing anyway. Or something like that.
Alternatively, document in the application's installation instructions that it is inadvisable to install the JRE / JDK and the application on a share drive (or equivalent).
1 - A Java install on a share drive will give as bad if not worse performance than an application JAR on a share drive.

copy files to a machine in local network in java with authentification

I've used Commons IO to write a program that copy files and other things. But I want to copy a file to a local ip address \\10.x.x.x, but the user doesn't have rights to do the copy, so I need to put an ID and password to access it. However I cannot find a way I can do that.
To move file I use :
FileUtils.moveFileToDirectory(fichier, destDir,true);
But my directory is something like \\10.x.x.x\files and only a few users can write in that directory so I have an ID & password that let you move files there. I want that even if the users don't have rights to move files to that directory my program can do it.
It is not really the way Windows security works. If you really want to do it that way, you will have to use Java Native Interface or Java Native Access, and manage to call the WNetAddConnection function from Mpr.dll (and do not forget to call WNetCancelConnection when done).
But you would have to store a password in your program, which is poor security practice.
The standard way to do that would be to start a service that would run under a user that has access to the desired directory, and have your program to communicate with it using whatever you want, the simplest way being probably TCP/IP. But unless you have special requirement for that I would not recommend to use Jave for those kinds of program.
A more Java alternative would be to start a Tomcat service on server machine running under a user having access to the directory. That way you just have to develop a standard Java Web Application able to upload files that would save the files to the proper directory. But it would be a traditionnal and portable Java application with no need for JNI nor JNA.
If cannot use a Tomcat and do not want to invest to much on it, you could split the program in pieces :
one client program that copies files on a directory (on server machine) with File creation rights for everybody - can decays to the copy utility if nothing more has to be done or can easily written in Java
one server program that will run on server machine under a user that has full write permissions on target directory. This one can also be easily written in Java
you can easily install the server program as a service on the server machine with sc and srvany according to this answer on ServerFault
If you use a client program, you could easily add a digital signature file with each copied file, but as I said above, it is poor security practice and add little if any security. At least the program should be executable and not readable, and the sources should be kept hidden. It is better to log the users that copied the file and ask them what happened is you find a problem.

Revision number with SVN [duplicate]

I'm using Visual SVN on my Windows Box.
I have Repository Application, which has Framework as an svn:external. All well and good.
When I make a checkout of Application, I'd like to have the version of Application and Framework for inclusion in a footer file. This way I could have something like:
Application Version $ApplicationVersion$, Framework Version $FrameworkVersion$
Ordinarily, I understand I could use svn:keywords and add the revision - but as I understand it, svn:keywords apply on a per-file basis. A few sites have suggested using svnversion to produce the output for each variable, but I'm not entirely sure how to go about this.
Once again, on a Windows Box, using VisualSVN. I also develop on a Mac using Versions.app if it provides a more familiar interface for people to answer :)
Edit - my application is a PHP web application. As such, there is no compiling.
Thanks!
To use svnversion, you need to integrate it into the build process. If you run it on a subversion checkout, it will output a string like 73597:73598, indicating what version your tree has (notice that different files may have different versions, plus files may have also local modifications). You put something like
CFLAGS+=-DSVNVERSION="\"`svnversion`\""
into your Makefile, and then put
#define VERSION_STRING "Application version" SVNVERSION ", Framework version" FRAMEWORK_VERSION
into the code. If you don't use Make, or cannot readily have your build process run a command whose output produces a compiler command line option, then you can also use the subwcrev utility that comes with TortoiseSVN. You use that as a pre-build step, and have it transform some file with placeholders into a copy of the file with the placeholders replaced with the actual version; then your compilation will compile and link the new file.
Edit: For the PHP case, it is not possible to have the revision written automatically into a file on checkout or update. Instead, you could run svnversion on every PHP access, putting its output into the HTML response. If that gets too expensive, you can cache the svnversion result in a file and only regenerate the file if it is older than one hour (say), leaving it up to the user to remember to delete the file after an update to make it recompute the cache right away.

Java, Tomcat 6: saving files to local HD

I've searching for this for a while now online (Google, and StackOverflow), but haven't yet come across this question. Maybe my query is not correct (please redirect then!)
I've developed and set up a WebApp on TomCat 6 under Linux. Tomcat isn't running in a virtual host environment yet, I have full control over server. Therefore, .war file is saved to Tomcat's standard deploy dir. The issue I have is with images: different web services provide differently-sized images which need to be presented in uniform sizes.
I download them and resize them without problems, but have to store a local copy of the image as this takes some time if done real-time, plus a lot of bandwidth waste. I don't save them under the .war's temp dir under Tomcat, due to case where a server shutdown would force me to reload all images.
I have created a different directory under /home/username/images, which I then serve under a different subdomain through regular Apache, and the HTML code generated in the .jsp is simply a correct URL to the file. Works great if the image doesn't exist. However, due to permission issues, the Tomcat instance cannot remove or overwrite files already created, even though I've marked the folder where images are stored with 777 permissions. As an aside, I don't see need to give it 777 perms, but with 755 (for example), I had permission issues even when trying to save a new file.
So: is there a better solution (I considered database, but the images dir is now 250mb, and I see no need to overload the db so much)?
Don't store the images in the database. Your /home/user/www.example.com/resources approach is in the correct direction, just sort out the privileges issue. Make sure the path is in a group where the user who runs tomcat (tomcat?) belongs in and reduce the 777 because it's too broad.
It is a little unclear as to whether tomcat and another user are writing, or only tomcat.
If it is only tomcat, the directory in question should be owned by tomcat, and the permissions can be either 750 (setting the group to match apache's group) or 755 (the group doesn't matter, anyone on the machine can read the files).
Note that if some other user is writing files, you have to be trickier. In that case, you need 775, user tomcat, group tomcat, and put the additional user(s) in the tomcat group. If those users are in multiple groups, you can use the group sticky bit on the directory to force the group to remain tomcat. In this case, anyone on the machine can read those files.
The documentation for chown and chmod should be of great help.
(Yes, I do realize that this question is very old, but the same principles apply, and no one seemed to give a clear answer).
What specifically is happening with the permissions? Tomcat will be running as some userid (say tomcat1) and will be creating files presumably owned by the same user (tomcat1). If your files are being created with a different owner, then that will explain why you can't overwrite existing downloads and you'll need to work out why the ownership is differen. If it's not ownership, then are the files being created without write permission to the user. In this case, consider explicitly setting the permissions on each file saved to allow the owner to write to it, or change the umask of the user account.

How to upload bulk data to Google Servers using Google App Engine running on Java?

I am not able to figure out how to upload bulk data to the Google's servers bypassing the 10mb upload limit and 30 sec session timeout. I want to design an application that takes my standard SQL data and pushes it to the Google's servers.
I might sound naive but your help is most valuable for my project.
There's not currently a native Java bulkloader, so what you need to do is use the Python one. The process goes like this:
First, you'll need to download the Python SDK and extract it. Then, create an empty directory, and in it create a file called app.yaml, containing the following:
application: yourappid
version: bulkload
runtime: python
api_version: 1
handlers:
- url: /remote_api
script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
login: admin
Now, run "appcfg.py update yourdir" from the Python SDK, and enter your credentials when prompted. appcfg will upload a new version of your app, which will run side-by-side with your main version, and allow you to bulkload.
Now, to do the actual bulkloading, you need to use the Python Bulkloader. Follow the instructions here. You'll need to know a (very) little bit of Python, but it's mostly copy-and-paste. When you're done, you can run the bulkloader as described in the article, but add the "-s bulkload.latest.yourapp.appspot.com" argument to the command line, like this:
appcfg.py upload_data --config_file=album_loader.py --filename=album_data.csv --kind=Album -s bulkload.latest.yourapp.appspot.com <app-directory>
Finally, to load data directly from an SQL database instead of from a CSV file, follow the instructions in my blog post here.
I wanna do the same thing also. So, here's my naivest concept to achieve the goal.
Web Server Preparation
Create a servlet that will receive the uploaded data (e.g. for data type
XML, JSON)
(optional) store it as Blobstore
Parse the data using JAXB/JSoup and/or GSON
Dynamically interpret the data structure
Store it using Datastore/
Client Uploader Preparation
Using a local computer, create a Java/C++/PHP script that generates XML/JSON files and store it locally
Create a shell script (linux) or batch file (windows) to programatically upload the files using cURL.
Please drop a comment to this one if you have better idea guys.

Categories

Resources