Develop Reference

Manipulate with cache as with collection in Spring

I looked a lot of stuff on on internet but I don't found any solution for my needs.
Here is a sample code which doesn't work but show my requirements for better understanding.
#Service
public class FooCachedService {
#Autowired
private MyDataRepository dataRepository;
private static ConcurrentHashMap<Long, Object> cache = new ConcurrentHashMap<>();
public void save(Data data) {
Data savedData = dataRepository.save(data);
if (savedData.getId() != null) {
cache.put(data.getRecipient(), null);
}
}
public Data load(Long recipient) {
Data result = null;
if (!cache.containsKey(recipient)) {
result = dataRepository.findDataByRecipient(recipient);
if (result != null) {
cache.remove(recipient);
return result;
}
}
while (true) {
try {
if (cache.containsKey(recipient)) {
result = dataRepository.findDataByRecipient(recipient);
break;
}
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
return result;
}
}
and data object:
public class Data {
private Long id;
private Long recipient;
private String payload;
// getters and setters
}
As you can see in code above I need implement service which will be stored new data into database and into cache as well.
Whole algorithm should looks something like that:
Some userA create POST request to my controller to store data and it fire save method of my service.
Another userB logged in system send request GET to my controller which fire method load of my service. In this method is compared logged user's id which sent request with recipients' ids in map. If map contains data for this user they are fetched with repository else algorithm check every second if there are some new data for that user (this checking will be some timeout, for example 30s, and after 30s request return empty data, and user create new GET request and so on...)
Can you tell me if it possible do it with some elegant way and how? How to use cache for that or what is the best practice for that? I am new in this area so I will be grateful for any advice.

Java - How to delete an entity from Google Cloud Datastore

Architecture: I have a web application from where I'm interacting with the Datastore and a client (raspberry pi) which is calling methods from the web application using Google Cloud Endpoints.
I have to add that I'm not very familiar with web applications and I assume that something's wrong with the setConsumed() method because I can see the call of /create in the app engine dashboard but there's no entry for /setConsumed.
I'm able to add entities to the Datastore using objectify:
//client method
private static void sendSensorData(long index, String serialNumber) throws IOException {
SensorData data = new SensorData();
data.setId(index+1);
data.setSerialNumber(serialNumber);
sensor.create(data).execute();
}
//api method in the web application
#ApiMethod(name = "create", httpMethod = "post")
public SensorData create(SensorData data, User user) {
// check if user is authenticated and authorized
if (user == null) {
log.warning("User is not authenticated");
System.out.println("Trying to authenticate user...");
createUser(user);
// throw new RuntimeException("Authentication required!");
} else if (!Constants.EMAIL_ADDRESS.equals(user.getEmail())) {
log.warning("User is not authorised, email: " + user.getEmail());
throw new RuntimeException("Not authorised!");
}
data.save();
return data;
}
//method in entity class SensorData
public Key<SensorData> save() {
return ofy().save().entity(this).now();
}
However, I'm not able to delete an entity from the datastore using the following code.
EDIT: There are many logs of the create-request in Stackdriver Logging, but none of setConsumed(). So it seems like the calls don't even reach the API although both methods are in the same class.
EDIT 2: The entity gets removed when I invoke the method from the Powershell so the problem is most likely on client side.
//client method
private static void removeSensorData(long index) throws IOException {
sensor.setConsumed(index+1);
}
//api method in the web application
#ApiMethod(name = "setConsumed", httpMethod = "put")
public void setConsumed(#Named("id") Long id, User user) {
// check if user is authenticated and authorized
if (user == null) {
log.warning("User is not authenticated");
System.out.println("Trying to authenticate user...");
createUser(user);
// throw new RuntimeException("Authentication required!");
} else if (!Constants.EMAIL_ADDRESS.equals(user.getEmail())) {
log.warning("User is not authorised, email: " + user.getEmail());
throw new RuntimeException("Not authorised!");
}
Key serialKey = KeyFactory.createKey("SensorData", id);
datastore.delete(serialKey);
}

This is what I follow to delete an entity from datastore.
public boolean deleteEntity(String propertyValue) {
String entityName = "YOUR_ENTITY_NAME";
String gql = "SELECT * FROM "+entityName +" WHERE property= "+propertyValue+"";
Query<Entity> query = Query.newGqlQueryBuilder(Query.ResultType.ENTITY, gql)
.setAllowLiteral(true).build();
try{
QueryResults<Entity> results = ds.run(query);
if (results.hasNext()) {
Entity rs = results.next();
ds.delete(rs.getKey());
return true;
}
return false;
}catch(Exception e){
logger.error(e.getMessage());
return false;
}
}
If you don't want to use literals, you can also use binding as follows:
String gql = "SELECT * FROM "+entityName+" WHERE property1= #prop1 AND property2= #prop2";
Query<Entity> query = Query.newGqlQueryBuilder(Query.ResultType.ENTITY, gql)
.setBinding("prop1", propertyValue1)
.setBinding("prop2", propertyValue2)
.build();
Hope this helps.

I was able to solve it by myself finally!
The problem was just related to the data type of the index used for removeSensorData(long index) which came out of a for-loop and therefore was an Integer instead of a long.

Why I am seeing lot of TimeoutException if any one server goes down?

Here is my DataClientFactory class.
public class DataClientFactory {
public static IClient getInstance() {
return ClientHolder.INSTANCE;
}
private static class ClientHolder {
private static final DataClient INSTANCE = new DataClient();
static {
new DataScheduler().startScheduleTask();
}
}
}
Here is my DataClient class.
public class DataClient implements IClient {
private ExecutorService service = Executors.newFixedThreadPool(15);
private RestTemplate restTemplate = new RestTemplate();
// for initialization purpose
public DataClient() {
try {
new DataScheduler().callDataService();
} catch (Exception ex) { // swallow the exception
// log exception
}
}
#Override
public DataResponse getDataSync(DataKey dataKeys) {
DataResponse response = null;
try {
Future<DataResponse> handle = getDataAsync(dataKeys);
response = handle.get(dataKeys.getTimeout(), TimeUnit.MILLISECONDS);
} catch (TimeoutException e) {
// log error
response = new DataResponse(null, DataErrorEnum.CLIENT_TIMEOUT, DataStatusEnum.ERROR);
} catch (Exception e) {
// log error
response = new DataResponse(null, DataErrorEnum.ERROR_CLIENT, DataStatusEnum.ERROR);
}
return response;
}
#Override
public Future<DataResponse> getDataAsync(DataKey dataKeys) {
Future<DataResponse> future = null;
try {
DataTask dataTask = new DataTask(dataKeys, restTemplate);
future = service.submit(dataTask);
} catch (Exception ex) {
// log error
}
return future;
}
}
I get my client instance from the above factory as shown below and then make a call to getDataSync method by passing DataKey object. DataKey object has userId and Timeout values in it. Now after this, call goes to my DataTask class to call method as soon as handle.get is called.
IClient dataClient = DataClientFactory.getInstance();
long userid = 1234l;
long timeout_ms = 500;
DataKey keys = new DataKey.Builder().setUserId(userid).setTimeout(timeout_ms)
.remoteFlag(false).secondaryFlag(true).build();
// call getDataSync method
DataResponse dataResponse = dataClient.getDataSync(keys);
System.out.println(dataResponse);
Here is my DataTask class which has all the logic -
public class DataTask implements Callable<DataResponse> {
private DataKey dataKeys;
private RestTemplate restTemplate;
public DataTask(DataKey dataKeys, RestTemplate restTemplate) {
this.restTemplate = restTemplate;
this.dataKeys = dataKeys;
}
#Override
public DataResponse call() {
DataResponse dataResponse = null;
ResponseEntity<String> response = null;
int serialId = getSerialIdFromUserId();
boolean remoteFlag = dataKeys.isRemoteFlag();
boolean secondaryFlag = dataKeys.isSecondaryFlag();
List<String> hostnames = new LinkedList<String>();
Mappings mappings = ClientData.getMappings(dataKeys.whichFlow());
String localPrimaryAdress = null;
String remotePrimaryAdress = null;
String localSecondaryAdress = null;
String remoteSecondaryAdress = null;
// use mappings object to get above Address by using serialId and basis on
// remoteFlag and secondaryFlag populate the hostnames linked list
if (remoteFlag && secondaryFlag) {
hostnames.add(localPrimaryHostIPAdress);
hostnames.add(localSecondaryHostIPAdress);
hostnames.add(remotePrimaryHostIPAdress);
hostnames.add(remoteSecondaryHostIPAdress);
} else if (remoteFlag && !secondaryFlag) {
hostnames.add(localPrimaryHostIPAdress);
hostnames.add(remotePrimaryHostIPAdress);
} else if (!remoteFlag && !secondaryFlag) {
hostnames.add(localPrimaryHostIPAdress);
} else if (!remoteFlag && secondaryFlag) {
hostnames.add(localPrimaryHostIPAdress);
hostnames.add(localSecondaryHostIPAdress);
}
for (String hostname : hostnames) {
// If host name is null or host name is in local block host list, skip sending request to this host
if (hostname == null || ClientData.isHostBlocked(hostname)) {
continue;
}
try {
String url = generateURL(hostname);
response = restTemplate.exchange(url, HttpMethod.GET, dataKeys.getEntity(), String.class);
// make DataResponse
break;
} catch (HttpClientErrorException ex) {
// make DataResponse
return dataResponse;
} catch (HttpServerErrorException ex) {
// make DataResponse
return dataResponse;
} catch (RestClientException ex) {
// If it comes here, then it means some of the servers are down.
// Add this server to block host list
ClientData.blockHost(hostname);
// log an error
} catch (Exception ex) {
// If it comes here, then it means some weird things has happened.
// log an error
// make DataResponse
}
}
return dataResponse;
}
private String generateURL(final String hostIPAdress) {
// make an url
}
private int getSerialIdFromUserId() {
// get the id
}
}
Now basis on userId, I will get the serialId and then get the list of hostnames, I am suppose to make a call depending on what flag is passed. Then I iterate the hostnames list and make a call to the servers. Let's say, if I have four hostnames (A, B, C, D) in the linked list, then I will make call to A first and if I get the data back, then return the DataResponse back. But suppose if A is down, then I need to add A to block list instantly so that no other threads can make a call to A hostname. And then make a call to hostname B and get the data back and return the response (or repeat the same thing if B is also down).
I have a background thread as well which runs every 10 minutes and it gets started as soon we get the client instance from the factory and it parses my another service URL to get the list of block hostnames that we are not supposed to make a call. Since it runs every 10 minutes so any servers which are down, it will get the list after 10 minutes only, In general suppose if A is down, then my service will provide A as the block list of hostnames and as soon as A becomes up, then that list will be updated as well after 10 minutes.
Here is my background thread code DataScheduler-
public class DataScheduler {
private RestTemplate restTemplate = new RestTemplate();
private static final Gson gson = new Gson();
private final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
public void startScheduleTask() {
scheduler.scheduleAtFixedRate(new Runnable() {
public void run() {
try {
callDataService();
} catch (Exception ex) {
// log an error
}
}
}, 0, 10L, TimeUnit.MINUTES);
}
public void callDataService() throws Exception {
String url = null;
// execute the url and get the responseMap from it as a string
parseResponse(responseMap);
}
private void parseResponse(Map<FlowsEnum, String> responses) throws Exception {
// .. some code here to calculate partitionMappings
// block list of hostnames
Map<String, List<String>> coloExceptionList = gson.fromJson(response.split("blocklist=")[1], Map.class);
for (Map.Entry<String, List<String>> entry : coloExceptionList.entrySet()) {
for (String hosts : entry.getValue()) {
blockList.add(hosts);
}
}
if (update) {
ClientData.setAllMappings(partitionMappings);
}
// update the block list of hostnames
if (!DataUtils.isEmpty(responses)) {
ClientData.replaceBlockedHosts(blockList);
}
}
}
And here is my ClientData class which holds all the information for block list of hostnames and partitionMappings details (which is use to get the list of valid hostnames).
public class ClientData {
private static final AtomicReference<ConcurrentHashMap<String, String>> blockedHosts = new AtomicReference<ConcurrentHashMap<String, String>>(
new ConcurrentHashMap<String, String>());
// some code here to set the partitionMappings by using CountDownLatch
// so that read is blocked for first time reads
public static boolean isHostBlocked(String hostName) {
return blockedHosts.get().contains(hostName);
}
public static void blockHost(String hostName) {
blockedHosts.get().put(hostName, hostName);
}
public static void replaceBlockedHosts(List<String> blockList) {
ConcurrentHashMap<String, String> newBlockedHosts = new ConcurrentHashMap<>();
for (String hostName : blockList) {
newBlockedHosts.put(hostName, hostName);
}
blockedHosts.set(newBlockedHosts);
}
}
Problem Statement:-
When all the servers are up (A,B,C,D as an example) above code works fine and I don't see any TimeoutException happening at all from the handle.get but if let's say one server (A) went down which I was supposed to make a call from the main thread then I start seeing lot of TimeoutException, by lot I mean, huge number of client timeouts happening.
And I am not sure why this is happening? In general this won't be happening right since as soon as the server goes down, it will get added to blockList and then no thread will be making a call to that server, instead it will try another server in the list? So it should be smooth process and then as soon as those servers are up, blockList will get updated from the background thread and then you can start making a call.
Is there any problem in my above code which can cause this problem? Any suggestions will be of great help.
In general, what I am trying to do is - make a hostnames list depending on what user id being passed by using the mappings object. And then make a call to the first hostname and get the response back. But if that hostname is down, then add to the block list and make a call to the second hostname in the list.
Here is the Stacktrace which I am seeing -
java.util.concurrent.TimeoutException\n\tat java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:258)
java.util.concurrent.FutureTask.get(FutureTask.java:119)\n\tat com.host.client.DataClient.getDataSync(DataClient.java:20)\n\tat
NOTE: For multiple userId's, we can have same server, meaning server A can get resolve to multiple userId's.

In DataClient class, at the below line:
public class DataClient implements IClient {
----code code---
Future<DataResponse> handle = getDataAsync(dataKeys);
//BELOW LINE IS PROBLEM
response = handle.get(dataKeys.getTimeout(), TimeUnit.MILLISECONDS); // <--- HERE
} catch (TimeoutException e) {
// log error
response = new DataResponse(null, DataErrorEnum.CLIENT_TIMEOUT, DataStatusEnum.ERROR);
} catch (Exception e) {
// log error
response = new DataResponse(null, DataErrorEnum.ERROR_CLIENT, DataStatusEnum.ERROR);
----code code-----
You have assigned a timeout to handle.get(...), which is timing out before your REST connections can respond. The rest connections themselves may or may not be timing out, but since you are timing out of get method of future before the completion of the execution of the thread, the blocking of hosts has no visible effect, while the code inside the call method of DataTask may be performing as expected. Hope this helps.

You asked about suggestions, so here are some suggestions:
1.) Unexpected return value
Method returns unexpectedly FALSE
if (ClientData.isHostBlocked(hostname)) //this may return always false! please check
2.) Exception-Handling
Are you really sure, that a RestClientException occurs?
Only when this exception occured, the host will be added to blocked list!
Your posted code seems to ignore logging (it is commented out!)
...catch (HttpClientErrorException ex) {
// make DataResponse
return dataResponse;
} catch (HttpServerErrorException ex) {
// make DataResponse
return dataResponse;
} catch (RestClientException ex) {
// If it comes here, then it means some of the servers are down.
// Add this server to block host list
ClientData.blockHost(hostname);
// log an error
} catch (Exception ex) {
// If it comes here, then it means some weird things has happened.
// log an error
// make DataResponse
}

how to get payer_id programmatically in my java code?

In my code, I am trying to send to Paypal REST API the needed information and Paypal is giving me an "APPROVED" status but in order to finalize the payment, I need to execute the payment.
payment.execute(accesstoken,paymentExecution)... but I couldn't get the payer_id from the response.
Here is my code for further information and thanks for your help in advance!
public class PaypalPaymentWithCreditCardServlet {
// private static final long serialVersionUID = 734220724945040319L;
// #Override
public void init () throws Exception {
InputStream is = PaypalPaymentWithCreditCardServlet.class.getResourceAsStream("/sdk_config.properties");
try {
PayPalResource.initConfig(is);
} catch (PayPalRESTException e) {
// goto failure page. can't do anything without configuration file
e.printStackTrace();
}
}
public boolean doPost (PaymentPojo paymentpojo) throws Exception {
try {
String accessToken = GenerateAccessToken.getAccessToken();
APIContext apiContext = new APIContext(accessToken);
final Payment payment = new PaymentBuilder().setBillingAddress(paymentpojo.getBillingAddress())
.setCreditCard(paymentpojo.getCreditCard())
.setPaymentDetail(paymentpojo.getDetails())
.setTransactionAmount(paymentpojo.getAmount())
.build();
Payment createdPayment = payment.create(apiContext);
System.out.println(Payment.getLastResponse());
System.out.println(String.format("created payment with id [%s] and status=[%s]", createdPayment.getId(), createdPayment.getState()));
if(!createdPayment.getState().toLowerCase().equalsIgnoreCase(PaypalState.APPROVED.getStatus())) {
// payment is not created. throw an exception
System.out.println("Payment handshake did not go through!!!");
return false;
}
// if it is not created throw exception
// payment.execute(accessToken, paymentExecution);
return true;
} catch (PayPalRESTException e) {
}
return false;
}
}

For credit card payments, payer_id is the unique attribute (e.g email address) that you use to identify the user in your system and you would provide that for payment each time in payerID.
If you are not aware of the user's information, such as a guest checkout, you should not need to provide a payerID.
For paypal payments, payer_id always needs to be provided and is appended to redirect_url once the user approves the payment. Full docs are at https://developer.paypal.com/webapps/developer/docs/integration/web/accept-paypal-payment/

UnboundID : How to encapsulate processBindRequest return value?

I'm building an LDAP interface for my database. When a client request bind(), it will search in the database and check if it is valid or not.
public class Main {
LDAPListener listener ;
Main() {}
public static void main(String[] args) {
Main main = new Main();
int port = main.StartServer();
try {
LDAPConnection cn = new LDAPConnection("localhost",port);
System.out.println("."+cn.isConnected()+" "+cn.getConnectedPort());
cn.bind("uid=user,ou=People,dc=example,dc=com", "pass");
cn.close();
main.StopServer();
} catch (Exception e){e.printStackTrace();
main.StopServer();}
}
public int StartServer() {
int listenPort = 0;
RequestHandler requestHandler = new RequestHandler();
LDAPListenerConfig config = new LDAPListenerConfig(listenPort, requestHandler);
listener = new LDAPListener(config);
try {
listener.startListening();
System.out.println(">port "+listener.getListenPort());
} catch (Exception e){System.out.println("e1> "+e.getMessage());}
return listener.getListenPort();
}
public void StopServer(){
System.out.println(">shutdown");
listener.shutDown(true);
}
}
Then, i modify LDAPListenerRequestHandler to communicate with the database, get the record as return value:
class RequestHandler extends LDAPListenerRequestHandler {
#Override
public LDAPMessage processBindRequest(int arg0, BindRequestProtocolOp arg1,
List<Control> arg2) {
String uid = arg1.getBindDN();
String pass = arg1.getSimplePassword();
System.out.println(">bind: "+ uid);
// Database query: SELECT * FROM user WHERE username='uid' AND password='pass'
// Get the record as return value
return null;
}
}
When i run it, i got error message from the bind line:
LDAPException(resultCode=80 (other), errorMessage='An unexpected exception was thrown while attempting to process the requested operation: NullPointerException(trace='run(LDAPListenerClientConnection.java:461)', revision=15579)', diagnosticMessage='An unexpected exception was thrown while attempting to process the requested operation: NullPointerException(trace='run(LDAPListenerClientConnection.java:461)', revision=15579)')
at com.unboundid.ldap.sdk.LDAPConnection.bind(LDAPConnection.java:1881)
at com.unboundid.ldap.sdk.LDAPConnection.bind(LDAPConnection.java:1799)
I think, it is caused by processBindRequest() that return null. How to encapsulate my database record as LDAPMessage in that process?

You are correct that the processBindRequest method must return a non-null response.
If the bind is successful (the user exists, is allowed to authenticate, and has provided the correct credentials) then you can create a successful response with code like:
#Override()
public LDAPMessage processBindRequest(final int messageID,
final BindRequestProtocolOp request,
final List<Control> controls)
{
return new LDAPMessage(messageID,
new BindResponseProtocolOp(ResultCode.SUCCESS_INT_VALUE,
null, // No matched DN is needed
null, // No diagnostic message is needed
null, // No referral URLs are needed
null), // No server SASL credentials are needed
Collections.<Control>emptyList()); // Add empty list to return
}
If the authentication is not successful, then you should probably return a response with a result code of INVALID_CREDENTIALS rather than SUCCESS, and if you want to provide a message to the client with information about why the bind failed, you can put that in the diagnostic message element.