There are many similar questions related to my question. But I didnt find any satisfactory ans for my question. So I am putting this question in this forum.
I have a question in concurrency control in inventory management system.
Say I have products A, B, C with quantity 2,3,4. And my application is multi user.
I have product page where user see the list of products and available quantity.
and I have check out and payment page which may take some time to reach after product page.
Now if it is multi user web application and say user 1 has ordered 2 quantity product A but order is not yet placed, user 2 can still see A with 2 quantities.
Should I temporarily(configurable time) lock the 2 quantities of product A until order is placed? Is it a good design. If yes, should I lock in java or in the database?
No, it's not a good design because it lend itself for abuse and problems. What if a user (competitor?) locks all your products during the whole month? What if another person fills up his/her shopping cart with product and then decide it's too much money and just turns off his/her device?
Best alternatives are:
1) Tell the user how much availability is of each product, but also tell him/her it could be gone if an order is not placed soon. This should also incentive sales. Do lock/unlock your products after the payment page, i.e. when there is a business commit to the operation. If available quantities are not enough anymore, send the user back a previous page, updating the amounts to whetever is available.
2) Similar to 1), but you could also update availability from time to time. Or send warnings like "the stock of some items in your cart is running low". Again, this also could prompt sales.
3) Reserve ("lock") items as they are moved to a shopping cart, but not forever. Release the items after being locked for certain amount of time. Keep informed the user. The time-out can be per the whole shopping cart or per item.
It is very important to notice that any "lock" mentioned above is a "business lock/reservation". It doesn't need to be implemented in the form of locks or any other particular technical solution. For example, 3) above can be implemented by adding fields locked_by and locked_until. While you check/update/manipulate these fields you will probably need to do it within a "technical lock". After the checks/updates are done, the technical locks are released. However, a "business lock" could still in place because locked_until has not elapsed yet and any other code will check this field to consider the product available or not. Why? Because business rules mandate so (not because there is a technical lock in place, which in fact is not).
"Technical locks" should be very quick. "Business locks" can be much longer (but never forever; always define a time limit for them).
It's hard to tell if you should lock "in Java" on "in the database" with the very little information you give. Are you using Entity Beans, for example? What are your fault tolerance requirements? Etc.
That said, in the general case it's probably better to keep locks (as long as they are "business locks") in the database, for these main reasons:
Persistence (in case of a power failure). You should also provide a mechanism to recover shopping carts. Perhaps also storing them in the DB?
Ability to interface with other environments (i.e. a corporate ERP or fullfillment system).
Should I temporarily(configurable time) lock the 2 quantities of product A until order is placed? Is it a good design.
It depends on the conversation rate of your site, i.e. the number of checkouts/the number of payments. If you have a high conversation rate, you can pre lock the quantities to get a better user experience.
If yes, should I lock in java or in the database?
You need a global lock to guarantee the correctness. If you have multiple application servers, you have to put the lock in the database.
Inventory Management should be handled by a Custom System Built from ground up. You cannot rely simply on ACID compliance for performance reasons. Implementing Transactions on Inventory during the order is a very bad idea and is not scalable. I propose the following solution.
An Inventory Management Backend App that updates the inventory as new items come in. Use row lock to update the inventory.
An Inventory Management Micro Service App to give the inventory to Order Management System as it needs the inventory to finish the order and keep track of time out.
a. If the order finished with Acknowledgement, The given inventory is already deducted by management system. We are good.
b. If the order is not finished and no Acknowledgement is received, the given inventory is added back to the inventory system after the timeout (Typically 2 - 3 minutes).
c. If the order has failed and acknowledgement is received that the order is failed, the given inventory is added back to the inventory system.
So, you don't lock the product rows until the order is finished. Instead, you do a soft lock on inventory and release if the order was not successful or failed due to an exception/App Failure.
Related
I am learning microservices and trying to design an e-commerce website. I can't figure out how big shopping sites take care of the last item in the cart problem.
For example, I selected an item from Amazon which had just a single item available in stock. I logged in from two different accounts and placed the item in cart. I even reached the payment page from both the account and the site didn't restrict me anywhere saying that the item is not available. I am not sure after the payment page when payment from both the account is in progress, how Amazon handles it.
Few solutions which come to my mind are like:
Accept payment from both the accounts and later cancel transaction for one of them which paid later than the first. This will not be a good practice though as it will result in bas customer experience.
Keep few items in reserve and use them in case of overbooking.
I forget what Amazon is doing and implement quantity checks in Order service from Item service via REST calls, at every stage of the order. But these checks sometimes can fail when a lot of people are ordering the same item. for e.g. in flash sales
Please share if you guys have worked on similar problem and solved it even with few limitations. If I need to put any in more details, let me know.
I cannot answer how Amaozon does it, nor I think anyone could on a public forum I can tell you how I think this could be managed.
So you have to take lock on your inventory if you want to make sure you precisely map inventory to an order. If you intend to do that, question will be where you take lock. When an item gets added to the cart, when user goes for payment or when payment is done. But the problem with lock is that it will make you system slow.
So that is something you should avoid.
Rest all the options you have already covered in your question and it boils down to tradeoffs.
First point, user experience will suffer and you also need to incur the cost of the transaction.
Second option ask you to be ready to undersell or oversell.
When you keep reserves, you are basically saying that I will be underselling. This can also backfire because say you decide to reserve 5 items but you get 20 concurrent request foir checkout and payment, you will be back to the square one. But it can help in most scenarios, given you are willing to take a hit.
Doing inventory check at checkout can help you get better resolution on inventory but it will not help when you literally have last item in inventory and 10 people doing a checkout on it. Read calls even for two such request coincides you will give them inventory and back to square one.
So what I do in such scenarios, is
1. My inventory goes not as number but enum i.e critical, low, med, high, very high
Depending on some analytics we configure inventory check. For high and very high we will not do any check and book the item. for critical and we take the lock. (not exactly a db lock but we reserve the inventory for them), for low and medium we check the inventory and proceed if we have enough. All these values are configurable and help us mitigate the scenarios we have.
Another thing that we are trying is to distribute inventory to inventory brokers and assign inventory broker to some set of services to see this inventory. Even if we reserve the inventory on one broker others can continue selling freely. And there brokers regularly update the inventory master about the status of inventory. Its like Inventory master has 50 items, it distributes 5 each to all ten. After 10 mins they come back and if they need more inventory they ask for it, if they have left over (in case of failure) they drop back inventory to the master for it to be assigned to others.
The above approach will not help you resolve the issue precisely but it gives you certain degree of freedom as to how you can manage the inventory.
Consider doing:
On the payment page, you should re-check if the product is still available. This can be a simple HTTP GET.
If the GET call is slow for you, consider caching recent product added by user to some in-memory databases (eg. REDIS). Now if first users successfully processes the payment, decrease counter for that product-id in redis. And before proceeding payment for second user, check the counter of that product-id in redis.
(BONUS: Redis offers atomic operations, so you can successfully handle the race condition in ordering the product as well.)
I'm maintaining a system (in Java, with Tomcat, Spring MVC, and Hibernate) where I have to set access rules for user groups. These rules are saved in a database (PostgreSQL) as records / rows. The logic is very simple. Each user of a company's team belongs (is connected) to a group, and each group has a set of rules.
I have to allow administrators to configure (through a web application) rules for groups, so that each rule has a logic and this is recognized and reproduced on the server side.
I need to define rules with parameters, such as:
Authentications only weekends.
Authentications only on weekdays.
Authentications only at a certain time (from time X to time Y).
X authentications per day.
Account expiration from date X
And so on...
My intention is that the company team can organize itself dynamically, just setting up any rules they want at any time, without the need for maintenance every time their policies change.
I've been searching on google and found nothing about it. I know I can do this in Java code, I would have to tie Java code with values of rules names present in database, something that could change in the future (or between companies), and this does not seem right to me. I'm not sure if this is correct, or preferable (maintainable). I appreciate any suggestions, ideas, or corrections (for real).
Note: Team/Groups names may change, but its rules should remain the same (if desired).
EDIT
The database is already modeled and ready. Groups and rules represent values from two different tables, with no logic at all. Querying these values works trivial. However, as I'm maintaining a web application, I'm in charge of creating a code or procedure that applies logic to the choice of rule values.
I was very clear in my question, but I will add more things:
Imagine that my clients (companies) want a website (a web application) that can manage their employees. Every company has teams of employees (groups), each with its function. Otherwise, some employees are sometimes hired as temporary employees.
My duty is to restrict access to the accounts of users who are part of company teams. This will allow business leaders to restrict things according to their policies.
For any company, the process works something like this:
The person in charge defines groups (with names and descriptions).
The same person defines restrictions rules for each group.
User accounts are created and linked to groups with rules.
The accounts are given (assigned) to each person part of the company
team, each according to their function.
Why should this be done?
Management
Control
Security
Speaking more technically now, I do not know where or how I should implement this properly. I know of a way to accomplish this, which is in programming code (Java, in my case), but again, I do not know if this is appropriate.
I also know that it is possible to define users and groups on the database side. But creating and deleting such definitions for each time an employee is hired or his or her length of service is expired can not become practical. My intention is to avoid to the maximum that companies have to spend more money on maintenance (Although sometimes this is obviously impossible).
My question based on a real case can be answered indicating to me an ideal way / approach for this type of scenario, either the solution being something that should be implemented in the database, or something done in the application layer, or both, or something else (I do not have experience to solve this kind of situation properly, so I'm here).
For practical purposes, I have decided to describe what technologies I am using in this system. If you want more information, I'll be happy to show you here.
Also, as this is a question that covers a larger context, not specifically databases, and also not specifically web applications, I have decided to put it here (instead of other StackExchange communities).
Thank you.
I would like to get some advice on designing a count based access control. For example I want to restrict the number of users that a customer can create in my system based on their account. So by default a customer can create 2 users but if the upgrade their account they get to create 5 users and so on.
There are a few more features that I need to restrict on a similar basis.
The application follows a generic model so every feature exposed has a backing table and we have a class which handles the CRUD operation on that table. Also the application runs on multiple nodes and has a distributed cache.
The approach that I am taking to implement this is as follows
- I have a new table which captures the functionality to control and the allowed limit (stored per customer).
- I intercept the create method for all tables and check if the table in question needs to have access control applied. If so I fetch the count of created entities and compare against the limit to decide if I should allow the creation or not.
- I am using the database to handle synchronization in case of concurrent requests. So after the create method is called I update the table using the following where clause
where ( count_column + 1 ) = #countInMemory#
. i.e. the update will succeed only if the value stored in the DB + 1 = value in memory. This will ensure that even if two threads attempt a create at the same time, only one of them will be able to successfully update. The thread that successfully updates wins and the other one is rolled back. This way I do not need to synchronize any code in the application.
I would like to know if there is any other / better way of doing this. My application runs on Oracle and MySQL DB.
Thanks for the help.
When you roll back, do you retry (after fetching the new user count) or do you fail? I recommend the former, assuming that the new fetched user count would permit another user.
I've dealt with a similar system recently, and a few things to consider: do you want CustomerA to be able to transfer their users to CustomerB? (This assumes that customers are not independent, for example in our system CustomerA might be an IT manager and CustomerB might be an accounting manager working for the same company, and when one of CustomerA's employees moves to accounting he wants this to be reflected by CustomerB's account.) What happens to a customer's users when the customer is deleted? (In our case another customer/manager would need to adopt them, or else they would be deleted.) How are you storing the customer's user limit - in a separate table (e.g. a customer has type "Level2," and the customer-type table says that "Level2" customers can create 5 users), or in the customer's row (which is more error prone, but would also allow a per-customer override on their max user count), or a combination (a customer has a type column that says they can have 5 users, and an override column that says they can have an additional 3 users)?
But that's beside the point. Your DB synchronization is fine.
I'm currently ddeveloping a hospital managment system.
My application will have multi cashiers at the hospital who working 24/7, they collecting money from the patients & issuing a receipt voucher or returning money back to patient in case of refunding.
currently i'm storing the employee user account no for each transaction, at the end of the day the employee print a list of the transaction that he made with total of money he have to send to the accountant department.
but i need some one to tell me what is the polices needed to prevent any msitake with total amount that employee have in his/her cash box.
Do i need to prevent others from deleting or revesing the record even the head of the accounting department?
I thing i need something as bank tellers, but i can't find something to help me or explain to me what exactly i need to put in my polices for this part.
I'm using: Java as GUI & programming language, MySQL (Percona) as my database, Crystal Reports As my reporting & Printing solution.
I think you need to think about this a bit more atomically. Rather than there being transactions which can be edited or removed, just have actions which occur.
So sure have your people run a transaction in, and if someone wants to edit that transaction let them but do it as changeset, a bit like how version control systems work. So you still retain the information on the original transaction but then have the change that should be applied to that transaction.
If you have some base Id for the transaction/action and then each change set points to that then at least you can tie all changes together by date order and then run them to work out the total, which you can easily cache for performance purposes later on.
Whenever money is involved its VERY risky to allow editing or removal of data, which is why most places tend to retain all information and just either archive it as out of date and put a more up to date record in its place, or have some data trail of the original record and what has changed on it.
Ultimately it all comes down to tracability...
I have a requirement in which I need to capture data changes (not auditing) and life cycle states on inventory.
Technology:
Jave, Oracle, Hibernate + JPA
For the data changes, we have been given a list of data elements that are to be monitored. If the element changes we are to notify a given 3rd party vendor. What I want to do is make this a generic service that we can provide to any of our current and future 3rd party vendors.
We don't care who made the change or what the new value is just that it changed.
The thought is that the data layer of our application would use annotation on each of the data elements. If that data element changed, then it would place a message into a queue. The message bean would then read the queue and make an entry in a table.
Table to look something like the following:
Table Name: ATL_CHANGE_TRACKER
Key columns
INVENTORY_ID Inventory Id of the vehicle
SALEEVENT_ITEM_ID SaleEvent item of the vehicle
FIELD_CHANGED_ID Id of the field that got changed or action. Link to subscription
UPDATE_DTM Indicates the date time when change occured.
For a given inventory, we could have up to 200 entries in this table (monitoring 200 fields across many tables).
Then a daemon for the given 3rd party would then read from this table based on the fields that it has subscribed to (could be all the fields). It would then read what every table it is required to to create the message to be sent to the 3rd party. Decouple the provider of the data and the user of the data.
Identify the list of fields/actions that are available
Table Name: ATL_FIELD_ACTION
Key columns
ID
NAME Name of the field/action - Example Color,Make
REC_CRE_TIME_STAMP
REC_CRE_USER_ID
LAST_UPDATE_USER_ID
LAST_UPDATE_TIME_STAMP
Subscription table, if 3rd Party company xyz is interested in 60 fields, the 60 fields will be mapped to this table.
ATL_FIELD_ACTION_SUBSCRIPTION
Key columns
ATL_FIELD_ACTION_ ID ID of the atl_field_action table
CONSUMER 3rd Party Name
FUNCTION Name of the 3rd Party Transmission that it is used for
STATUS
REC_CRE_TIME_STAMP
REC_CRE_USER_ID
LAST_UPDATE_USER_ID
LAST_UPDATE_TIME_STAMP
The second part is that there will be actions on the life cycle of the inventory which will need to be recored also. In this case, when the state of the inventory changes a message will be placed on the same queue and that entry will be entered in the same table.
Again, the daemon will have subscribed to these states and will collect the ones it is interested in.
The goal here is to not have the business tier/data tier care who wants the data - just that it needs to provide it so those interested can get it.
Wonder if anyone has done something like this - any gotchas - off the shelf - open source solutions to do this.
For a high-level discussion on the topic, I would suggest reading this article by Martin Fowler.
Its sounds like you have write-once, read-many type of data, it might produce large volumes of data, and the data is different for different clients. If you ask me, it sounds like this may be a good place to make use of either a NOSQL database or hack your Oracle database to act as a NOSQL database. See here for a discussion on how someone did this with MySQL.
Otherwise, you may look at creating an "immutable" database table and have Hibernate write new records every time it does an update as described here.
Couple things.
First, you get to do all of this work yourself. The JPA/Hibernate lifecycle listeners, while they have an event for when an update occurs, you aren't passed the "old" object and the "new" object. So, you're going to have to keep track of what fields change using some other method.
Second, again with lifecycle listeners, be careful inside of them, as the transaction state is a bit murky. At least on Glassfish/EclipseLink, I've had "strange" problems using either the JPA or JMS from a lifecycle listener. Just weird behavior. We went to a non-transactional queue to capture all of our information that we track from the lifecycle events.
If having the change data committed on its own transaction is acceptable, then there is value is pushing the data on to a faster, internal queue (which can feed a listener that posts it to an MDB). This just gets the auditing "out of band" with your transaction, give you better transaction throughput. But if you need to have the change information committed with the same transaction, this won't work. For example, you could put something on the queue and then the transaction may be rolled back (for whatever) reason, leaving the change on the queue showing it happened, when it in fact failed. That's a potential issue with this.
But if you're posting a lot of audit information, then this can be a concern.
If the auditing information has a short life span (with respect to the rest of the data), then you should probably make an effort to cull the audit tables, they can get pretty large.
Also, if practical, don't disregard the use of DB triggers for this. They can be quite efficient and effective at this process.