Develop Reference

Google Cloud Storage with a service account in Java - 403 Caller does not have storage.objects.list access to bucket

We want to download files from Google Storage in our application server. It is important to have read-only restricted access to a single bucket and nothing else.
At first I used a regular user account (not a service account) which have permissions to access all buckets in our Google Cloud project, and everything worked fine - my Java code opened buckets and downloaded files without problems.
Storage storage = StorageOptions.getDefaultInstance().getService();
Bucket b = storage.get( "mybucketname" );
Then I wanted to switch to use a specially created service account which has access to a single bucket only. So I created a service account, gave permissions to read a single bucket, and downloaded its key file. The permissions in Google Cloud Console are named as:
Storage Object Viewer (3 members) Read access to GCS objects.
gsutil command line utility works fine with this account - from the command line it allows accessing this bucket but not the others.
The initialization from the command line is done using the following command:
gcloud --project myprojectname auth activate-service-account files-viewer2#myprojectname.iam.gserviceaccount.com --key-file=/.../keyfilename.json
I even tried two different service accounts which have access to different buckets, and from the command line I can switch between them and gsutil gives access to a relevant bucket only, and for any other it returns the error:
"AccessDeniedException: 403 Caller does not have storage.objects.list access to bucket xxxxxxxxxx."
So, from the command line everything worked fine.
But in Java there is some problem with the authentication.
The default authentication I previously used with a regular user account stopped working - it reports the error:
com.google.cloud.storage.StorageException: Anonymous users does not have storage.buckets.get access to bucket xxxxxxxxxx.
Then I've tried the following code (this is the simplest variant because it relies on the key json file, but I've already tried a number of other variants found in various forums, with no success):
FileInputStream fis = new FileInputStream( "/path/to/the/key-file.json" );
ServiceAccountCredentials credentials = ServiceAccountCredentials.fromStream( fis );
Storage storage = StorageOptions.newBuilder().setCredentials( credentials )
.setProjectId( "myprojectid" ).build().getService();
Bucket b = storage.get( "mybucketname" );
And all I receive is this error:
com.google.cloud.storage.StorageException: Caller does not have storage.buckets.get access to bucket mybucketname.
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
The same error is returned no matter to what buckets I'm trying to access (even non-existing).
What confuses me is that the same service account, initialized with the same JSON key file, works fine from the command line.
So I think something is missing in Java code that ensures correct authentication.

TL;DR - If you're using Application Default Credentials (which BTW you are when you do StorageOptions.getDefaultInstance().getService();), and if you need to use the credentials from a service account, you can do so without changing your code. All you need to do is set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the full path of your service account json file and you are all set.
Longer version of the solution using Application Default Credentials
Use your original code as-is
Storage storage = StorageOptions.getDefaultInstance().getService();
Bucket b = storage.get( "mybucketname" );
Set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the full path of your json file containing the service account credentials.
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service_account_credentials.json
Run your java application once again to verify that it is working as expected.
Alternate solution using hard-coded Service Account Credentials
The code example you posted for initializing ServiceAccountCredentials looks valid to me on a quick glance. I tried the following code snippet and it is working for me as expected.
String SERVICE_ACCOUNT_JSON_PATH = "/path/to/service_account_credentials.json";
Storage storage =
StorageOptions.newBuilder()
.setCredentials(
ServiceAccountCredentials.fromStream(
new FileInputStream(SERVICE_ACCOUNT_JSON_PATH)))
.build()
.getService();
Bucket b = storage.get("mybucketname");
When specifying a service account credential, the project ID is automatically picked up from the information present in the json file. So you do not have to specify it once again. I'm not entirely sure though if this is related to the issue you're observing.
Application Default Credentials
Here is the full documentation regarding Application Default Credentials explaining which credentials are picked up based on your environment.
How the Application Default Credentials work
You can get Application Default Credentials by making a single client
library call. The credentials returned are determined by the
environment the code is running in. Conditions are checked in the
following order:
The environment variable GOOGLE_APPLICATION_CREDENTIALS is checked. If this variable is specified it should point to a file that
defines the credentials. The simplest way to get a credential for this
purpose is to create a Service account key in the Google API Console:
a. Go to the API Console Credentials page.
b. From the project drop-down, select your project.
c. On the Credentials page, select the Create credentials drop-down,
then select Service account key.
d.From the Service account drop-down, select an existing service
account or create a new one.
e. For Key type, select the JSON key option, then select Create. The
file automatically downloads to your computer.
f. Put the *.json file you just downloaded in a directory of your
choosing. This directory must be private (you can't let anyone get
access to this), but accessible to your web server code.
g. Set the environment variable GOOGLE_APPLICATION_CREDENTIALS to
the path of the JSON file downloaded.
If you have installed the Google Cloud SDK on your machine and have run the command gcloud auth application-default login, your
identity can be used as a proxy to test code calling APIs from that
machine.
If you are running in Google App Engine production, the built-in service account associated with the application will be used.
If you are running in Google Compute Engine production, the built-in service account associated with the virtual machine instance
will be used.
If none of these conditions is true, an error will occur.
IAM roles
I would recommend going over the IAM permissions and the IAM roles available for Cloud Storage. These provide control at project and bucket level. In addition, you can use ACLs to control permissions at the object level within the bucket.
If your use case involves just invoking storage.get(bucketName). This operation will require just storage.buckets.get permission and the best IAM role for just this permission is roles/storage.legacyObjectReader.
If you also want to grant the service account permissions to get (storage.objects.get) and list (storage.objects.list) individual objects, then also add the role roles/storage.objectViewer to the service account.

Thanks to #Taxdude's long explanation, I understood that my Java code should be all right, and started looking at other possible reasons for the problem.
One of additional things I've tried were the permissions set to the service account, and there I've found the solution – it was unexpected, actually.
When a service account is created, it must not be given permissions to read from Google Storage, because then it will have read permissions to ALL buckets, and it is impossible to change that (not sure why), because the system marks these permissions as "inherited".
Therefore, you have to:
Create a "blank" service account with no permissions, and
Configure permissions from the bucket configuration
To do so:
Open Google Cloud Web console
Open Storage Browser
Select your bucket
Open the INFO PANEL with Permissions
Add the service account with the Storage Object Viewer permission, but there are also permissions named Storage Legacy Object Reader and Storage Legacy Bucket Reader
Because of the word "Legacy" I thought those should not be used – they look like something kept for backward compatibility. And after experimenting and adding these "legacy" permissions, all of a sudden the same code I was trying all the time started working properly.
I'm still not entirely sure what is the minimal set of permissions I should assign to a service account, but at least now it works with all three "read" permissions on the bucket – two "legacy" and one "normal".

Maven automated deploy on google app engine

I know that to deploy onto app-engine you can run the maven command:
mvn appengine:update
Which then prompts for the auth code and launches a browser to confirm you want to publish onto the app engine. I'm looking to automate this fully so it can just be run as a command from maven. I have the appengine-maven-plugin setup but there doesn't seem to be any functionality to skip the manual step.
Has anyone managed to solve this? I know you can deploy right from Jenkins but we're looking for a solution that could be automates if our Jenkins server was down
Thanks

The authentication flow which redirects you to the browser for login and consent is call three-legged Oauth because it involves three steps: 1) request for a token 2) login and consent 3) API call with token.
An authentication flow which does not involve user login and consent is called two-legged Oauth (server to server): 1) request for a token, signed with a private key 2) API call with token. Server to server auth requires that you setup a service account with appropriate permissions (scopes). You must also safely distribute the private key for that service account so that you can sign your requests. See the OAuth2ServiceAccount instructions
The Bitnami Jenkins image actually does quite a bit for you out of the box. When you setup Push-to-Deploy you launch an instance with gcloud. That launch command specifies the scopes you want that images to have access too with the --scopes parameter, including "https://www.googleapis.com/auth/appengine.admin". Permission to these scopes is given to the default project service account. When Jenkins deploys your code, the service account credentials are made available through the Jenkins Google Oauth Plugin.
You can build another tool that will push code programatically. You will need to create a service account which generates a JSON Web Token (JWT) Again see the Outh2ServiceAccount instructions. The JWT is your private credential and you need to find a secure place to store it. The App Engine SDK knows how to use JWTs to get auth tokens.
During runtime you can pass the path to the JWT file as the --service_account_json_key_file parameter to appcfg.py. To plumb that parameter through the Maven plugin you'll have to use -DadditionalParams="--service_account_json_key_file " which is available on the tip of github.com/GoogleCloudPlatform/appengine-maven-plugin/master. (Will be in App Engine release 1.9.27, but you can pull from Github until then.)

How the architecture should be for a Java command line app integrating with shopify

In my company we have a custom point of sale which was built in house, and I am assigned to integrate the POS with the shopify. For the mvp, we have decided to go with the simple java app, which will run from the command line. I have done the following
1) Went to www.shopify.com/app-developers and then got my private store bla.myshopify.com. 2) Then went to Apps -> private apps. Got my API key, password and shared secret. As I have mentioned, we are doing a simple java app which can run from command line. This app will read all the keys from the local file. Now should I be doing OAUTH? As they have mentioned to redirect to a web page in here https://docs.shopify.com/api/authentication/oauth. Since we have don't have a webpage, where should I redirect the user to?
And is there anything wrong I am doing? Kindly correct me if there is a better architecture

Basic Authentication works with Shopify: Whatever package you are using for your http connection should have tools to let you set the Authorization header. Look up pre-emptive authorization for your package and use the API Key:Password (from the private app) for the Base 64 encrypted credential token

JBoss Form based authentication automatically

A customer wants to reach a section (where a process with a wizard take place) of our app from their portal. It means, from their app (.Net) they want to have a link in order to reach a particular section of our app BUT! without having to go through the login process. Our app is a java app (GWT) and it uses FORM based authentication. I did an implementation with oAuth but they don't want login. I have not idea how to deal with this problem, I don't know if there is an alternative to get logged in by following the way they are looking for.

You probalbly need Single Sign On.
If you choose to go the SSO way, there are some libraries that might help like JCIFS (useful for auth using NTLM) , Jespa. There is also Waffle, but this one works just on Windows server.
This article explore many options for JBoss:
https://dzone.com/articles/choosing-sso-your-jboss

GlassFish v3 Prelude - admin questions

I've been mucking around with GlassFish v3 prelude this evening. A couple of things I'm not sure about is how the security works.
It installed like a dream, so the first thing I did was remove the anonymous account and setup an "admin" account with "mysecretpassword" and specified the asadmin group for this user in the admin realm.
At that point the stop-domain command stopped working from the console (and/or start menu shortcut). Fair enough, but the stop-domain command accepts a --user argument but then wants a --passwordfile path supplied which I didn't know where it was.
After some googling I chanced upon asaadmin login -p 4848 which created a password file in my documents and then stop-domain started working!
So the questions:
1) Is it possible to get stop-domain to prompt for a user/password?
2) This whole business of having a file with a (hashed) password doesn't sound great to me - why did they do it this way, and is it transferable, can it be disabled and if it can, how then do I stop the domain?
3) How do security realms work - as in what are they, and how do they differ?
4) What's the best resource (book, website, whatever) on GF v3?

Glassfish relies upon a "master-password" file which is used to control access to the core admin commands (notably asadmin).
This master password is used to encrypt assorted bits within the GF system and configuration. By default, it's store in the file $domain-dir/master-password.
If that file exists, then asadmin will not prompt for a password to make changes. If it does not exist, then it will prompt you. If you ever change the password, the system will automatically recreate this file.
Note, this isn't a role based thing, it's simply a password to the entire system.
So, that's the core password.
Next is the actual admin password.
When you create a domain, it asks you for an admin name and password. This information is stored in ~/.asadminpass file.
So, in the simple case of a local install, you can use asadmin without passwords or user information.
If you delete that file, then you will be prompted for a user and password.
The admin passwords are managed in the "admin-realm" within the app server.
Realms are a Java EE construct used to represent authentication and authorization stores.
Realms provide user name, password, and role information.
By default GF uses file realms for its security information. Out of the box you have a "admin-realm" which is for users of the admin console. There is also an empty "file" realm for use by applications. Out of the box, Glassfish also supports and LDAP realm and a JDBC realm where credentials are stored in an LDAP and RDBMS respectively.
The Glassfish website and forums are currently the best resources available for this. It's a bit of a muddy time as v3 is still being worked on (v3 Prelude is just that, a prelude to the final release later this year), but there is still a lot of overlap with v2.

GlassFish v3 Prelude Domain Start Failed
first think stop any one server which is running in system --
Exam:- suppose you using glass fish server and netbeansIDE both are working on your system so when you want compile your application so that time you can stop your glassfish server then compile and run your application within netbeans IDE it's working