Situation :
We have a product with approx 30 attributes (String, Enum, Double)
values
We have iMap with indexes for all attributes IndexType.HASH for
string value and IndexType.SORTED for double values. (900MB together)
We have 300k products in map.(aprox 500MB )
We use local Datagrid with one member
JVM config: -Xms6G -Xmx8G
For HZ 5: we enabled JetConfig
config.getJetConfig().setEnabled(true);
Use Java AdoptOpenJDK 11.0.8
When invoking SQL query with pagination in HZ4 we got a response approx in 20-50ms, but the same query in Hazelcast 5 we got results in 2000-2500 ms
...ORDER BY param1 ASC LIMIT 20 OFFSET 0...
SqlResult sqlRows = hazelcastInstance.getSql().execute(sqlBuilder.toString());
When we tried to use predicates on the same map and in HZ4 and HZ5 we got the same results about 2000-2500 ms to get predicated page
PagingPredicate<Long, Product> pagingPredicate = Predicates.pagingPredicate(predicate, ProductComparatorFactory.getEntryComparator(sortName), max);
pagingPredicate.setPage(from / max);
///get final list of products
List<Product> selectedPageA = new ArrayList<>(productMap.getAll(productMap.keySet(pagingPredicate)).values());
For HZ 5 we add Mapping
hazelcastInstance.getSql().execute("CREATE MAPPING "ProductScreenerRepositoryProductMap" EXTERNAL NAME "ProductScreenerRepositoryProductMap"
TYPE IMap
OPTIONS (
'keyFormat' = 'java',
'keyJavaClass' = 'java.lang.Long',
'valueFormat' = 'java',
'valueJavaClass' = 'com.finmason.finriver.product.Product'
)");
}
There is used SQL
SELECT * FROM ProductScreenerRepositoryProductMap
WHERE doubleValue1 >= -0.9624378795139998
AND doubleValue1 <= 0.9727269574354098
AND doubleValue2 >= -0.9
AND doubleValue2 <= 0.9
ORDER BY doubleValue3 ASC LIMIT 20 OFFSET 0
And Product use simple serialization
Please upgrade to Hazelcast 5.1 (planned for February 23 right now).
It should be fixed with https://github.com/hazelcast/hazelcast/pull/20681
Actually this case will speed up by 3 separate PRs from 5.1:
https://github.com/hazelcast/hazelcast/pull/20681 - this one make your query use index
https://github.com/hazelcast/hazelcast/pull/20402 - this one will do less deserialization on cluster side
https://github.com/hazelcast/hazelcast/pull/20398 - this one makes deserialization on client side faster for multi-column queries
There are two cases not resolved in 5.1, they are described in https://github.com/hazelcast/hazelcast/pull/20796 - it should not be
a problem in your case, but if someone else see this post, it may be his/her. I hope that fix will be delivered in 5.1.1.
If you have a possibility to upgrade to full 5.1 after the release then I strongly recommend you to do it.
I am trying to find out the time spent on each tab/website by the user.
For example if I visited youtube and watched it for 10 minutes then I should be able to see something like this
www.youtube.com ---> 10 minutes
I already made a connection with sqlite database i.e. History file present in chrome directory and was able to run the following sql command to fetch the data:
SELECT urls.id, urls.url, urls.title, urls.visit_count, urls.typed_count, urls.last_visit_time, urls.hidden, urls.favicon_id, visits.visit_time, visits.from_visit, visits.visit_duration, visits.transition, visit_source.source FROM urls JOIN visits ON urls.id = visits.url LEFT JOIN visit_source ON visits.id = visit_source.id
So can anyone tell me which combination of column can i use to get the time spent on each website.
Please note that: visit_duration is not giving me appropriate data.
visit_duration Stores duration in microseconds, you need to convert and format that number. Here is one way to show a human-readable visit duration:
SELECT urls.url AS URL, (visits.visit_duration / 3600 / 1000000) || ' hours ' || strftime('%M minutes %S seconds', visits.visit_duration / 1000000 / 86400.0) AS Duration
FROM urls LEFT JOIN visits ON urls.id = visits.url
Here is a sample output:
URL
Duration
http://www.stackoverflow.com/
3 hours 14 minutes 15 seconds
You can also use strftime if you want more format options
I have a large text file which contains many abstracts (7k of them). I want to separate them. They have the following properties:
a number at the begining with a period right after
123.
and it always ends in:
[PubMed - indexed for MEDLINE]
It would be even better if I can get the title and abstract out of the separated string. I am fine if I have to split the articles first then split the texts.
In the example the title is the third line:
Effects of propofol and isoflurane on haemodynamics and the inflammatory response in cardiopulmonary bypass surgery.
The abstract is on the 8th line:
Cardiopulmonary bypass (CPB) causes reperfusion injury...
I have tried to use the following code for this text
Regex:
[0-9\.]*\s*(((?![0-9\.]*|MEDLINE).)+)\s*MEDLINE
Text:
1. Br J Biomed Sci. 2015;72(3):93-101.
Effects of propofol and isoflurane on haemodynamics and the inflammatory response
in cardiopulmonary bypass surgery.
Sayed S, Idriss NK, Sayyedf HG, Ashry AA, Rafatt DM, Mohamed AO, Blann AD.
Cardiopulmonary bypass (CPB) causes reperfusion injury that when most severe is
clinically manifested as a systemic inflammatory response syndrome. The
anaesthetic propofol may have anti-inflammatory properties that may reduce such a
response. We hypothesised differing effects of propofol and isoflurane on
inflammatory markers in patients having CBR Forty patients undergoing elective
CPB were randomised to receive either propofol or isoflurane for maintenance of
anaesthesia. CRP, IL-6, IL-8, HIF-1α (ELISA), CD11 and CD18 expression (flow
cytometry), and haemoxygenase (HO-1) promoter polymorphisms (PCR/electrophoresis)
were measured before anaesthetic induction, 4 hours post-CPB, and 24 hours later.
There were no differences in the 4 hours changes in CRP, IL-6, IL-8 or CD18
between the two groups, but those in the propofol group had higher HIF-1α (P =
0.016) and lower CD11 expression (P = 0.026). After 24 hours, compared to the
isoflurane group, the propofol group had significantly lower levels of CRP (P <
0.001), IL-6 (P < 0.001) and IL-8 (P < 0.001), with higher levels CD11 (P =
0.009) and CD18 (P = 0.002) expression. After 24 hours, patients on propofol had
increased expression of shorter HO-1 GT(n) repeats than patients on isoflurane (P
= 0.001). Use of propofol in CPB is associated with a less adverse inflammatory
profile than is isofluorane, and an increased up-regulation of HO-1. This
supports the hypothesis that propofol has anti-inflammatory activity.
PMID: 26510263 [PubMed - indexed for MEDLINE]
Two useful solutions have been proposed by Mariano and stribizhev:
Mariano's solution: Use the split method with the typical end
(?m)\[PubMed - indexed for MEDLINE\]$
DEMO : http://ideone.com/Qw5ss2
Java 4+
stribizhev's solution: Fully extract data from the text
(?m)^\s*\d+\..*\R{2} # Get to the title
(?<title>[^\n]*(?:\n(?!\n)[^\n]*)*) # Get title
\R{2} # Get to the authors
[^\n]*(?:\n(?!\R)[^\R]*)* # Consume authors
(?<abstract>[^\[]*(?:\[(?!PubMed[ ]-[ ]indexed[ ]for[ ]MEDLINE\])[^\[]*)*) #Grab abstract
DEMO: https://regex101.com/r/sG2yQ2/2
Java 8+
Try this:
"^[0-9]+\..*\s+(.*)\s+.*\s+((?:\s|.)*?)\[PubMed - indexed for MEDLINE\]"
First group would be title. Second would be abstract.
Using Microsoft Azure's billing and usage API, I see that we can get the usage quantity of a resource for a project and the rate for the resource consumption from the rate card API. However, the rate card API, in some cases, has a list of key-value pairs.
For ex, consider the following MeterRates:
"MeterRates": {
"0": 0.0832,
"1024": 0.0819,
"51200": 0.0806,
"512000": 0.0794,
"1024000": 0.0775,
"5120000": 0.0775
},
Here, if the usage quantity is, say, 102400; does it mean the cost of using the resource will be 102400 * 0.0784 or (102400 - 1023) * 0.0832 + (102400 - 1023 - 51200) * 0.0819?
N.B.: I have an issue on the github repository for the billing and usage API regarding the same.
Here, if the usage quantity is, say, 102400; does it mean the cost of
using the resource will be 102400 * 0.0784 or (102400 - 1023) * 0.0832
+ (102400 - 1023 - 51200) * 0.0819?
It will actually be more like:
1023 * 0.0832 + 50176 * 0.0819 + 460800 * 0.0794 + 512000 * 0.0775
Essentially these are pricing tiers so the way you read them is from 0 to 1023, it would be 0.0832 / unit. From 1024 - 51200, it would be 0.0819 units and so on.
+1 to Gaurav's response!
RateCard follows tiered pricing, which means that if there are multiple tiers for rating a particular resource, each subset of usage in each tier is rated differently.
Let’s take another example for consumed quantity = 250
Let’s assume the rates:
0: 1
200: 0.9
500: 0.8
This means that usage for units 0 – 199 should be rated at 1 per unit, 200 – 499 at 0.9 per unit and 500+ at 0.8 per unit.
Consumed quantity of 250 would be billed at: 51*0.9 + 199 * 1 = 244.9
Messy, complicated question, but here goes. I'm working on an integration project with Google Checkout, and there is a Google Checkout GWT service that returns the currency conversion rates used by the Checkout web interface to convert USD into local currencies. This endpoint is hosted at https://market.android.com/publish/gwt/, and staring at Firebug I see this going to the server:
7|0|6|https://market.android.com/publish/gwt/|FCCA4108CB89BFC2FEC78BA7363D4AF6|com.google.wireless.android.vending.developer.
shared.MerchantService|getCurrencyExchangeRates|com.google.common.money.CurrencyCode/112449834|java.util.ArrayList/4159755760
|1|2|3|4|2|5|6|5|235|6|13|5|18|5|81|5|53|5|72|5|102|5|121|5|177|5|175|5|205|5|204|5|55|5|86|-1|
and this being returned
//OK[235,3,'D0JA',2,86,3,'CXXg',2,55,3,'DW2A',2,204,3,'X9NA',2,205,3,'EuvA',2,175,3,'VIig',2,177,3,'E2Dw',2,121,3,'E4ziA',2,1
02,3,'do$Q',2,72,3,'T82w',2,53,3,'Ds0Q',2,81,3,'Cq5g',2,18,3,'Dlfg',2,13,1,["com.google.common.collect.RegularImmutableList/4
40499227","com.google.common.money.SimpleMoney/627983206","com.google.common.money.CurrencyCode/112449834"],0,7]
Forgive the odd formatting: can't quite get the code block to format right.
Wandering the web for hours on end I was able to determine that the RegularImmutableList class is in the Guava libraries (at http://code.google.com/p/guava-libraries/). What I'm looking for is:
I can't find the com.google.common.money.SimpleMoney or com.google.common.money.CurrencyCode classes anywhere: anyone seen them?
The GWT wire format appears to be an odd JSON string. I see various references to Google Groups messages talking about descriptions of the wire format, but can't find the underlying messages or any coherent reference that would let me reverse this: anyone have a handle on a handy reference? If I can at least understand WHAT the encoding is I might be able to get away without the class files from question 1 above.
I started wandering through the Android Market api library at http://code.google.com/p/android-market-api/, figuring they have to have done SOME of the Android Market communication integration, and they appear to have done so using protobufs. Is there any decent reference for the GWT/protobufs communication bits?
The underlying reason for this craziness is that I need to be able to take regular exchange rate values from Google Checkout so when I'm importing sales transactions in foreign currencies I can do the conversion at the prevailing rate at the time of the transaction. The current Checkout reporting formats do NOT provide this, so most folks end up using alternative sources of exchange rates that don't match what Google uses. It is clearly a shortcoming on the part of Google Checkout's integration interface, but if we got started on shortcomings of Google Checkout's interface we'd be here all week. My intention is to poll the Checkout interface for newly fulfilled orders and then request the appropriate exchange rate table so I can figure out in near real-time what the incoming payments are. I've got the polling bit down pat but can't quite get past the exchange rate bit.
While trying to create a script to bulk upload in-app products for my application (CSV upload constantly failed with obscure error messages), I have managed to understand the GWT AJAX protocol.
It's actually pretty simple, except it requires you to know structure of all used classes. Or guess it, as is the case with internal classes used by Google. :)
I'll use examples from the question to explain the protocol in detail.
Request format
7|0|6|https://market.android.com/publish/gwt/|FCCA4108CB89BFC2FEC78BA7363D4AF6|com.google.wireless.android.vending.developer.shared.MerchantService|getCurrencyExchangeRates|com.google.common.money.CurrencyCode/112449834|java.util.ArrayList/4159755760|1|2|3|4|2|5|6|5|235|6|13|5|18|5|81|5|53|5|72|5|102|5|121|5|177|5|175|5|205|5|204|5|55|5|86|-1|
The request is pipe-delimited list of tokens with the following meaning:
7 - protocol version
0 - flags. 1 is FLAG_ELIDE_TYPE_NAMES, 2 is FLAG_RPC_TOKEN_INCLUDED
6 - string token count
6 string tokens:
https://market.android.com/publish/gwt/
FCCA4108CB89BFC2FEC78BA7363D4AF6
com.google.wireless.android.vending.developer.shared.MerchantService
getCurrencyExchangeRates
com.google.common.money.CurrencyCode/112449834
java.util.ArrayList/4159755760
The actual encoded request, which references strings from the list above using 1-based indices:
1 - https://market.android.com/publish/gwt/ - base URL
2 - FCCA4108CB89BFC2FEC78BA7363D4AF6 - some hash, which is references as serializationPolicyStrongName in GWT sources.
3 - com.google.wireless.android.vending.developer.shared.MerchantService - service name
4 - getCurrencyExchangeRates - method name
2 - parameter count. Parameter types follow:
5 - com.google.common.money.CurrencyCode/112449834
6 - java.util.ArrayList/4159755760
Serialized parameters. Each object is represented either by its classname and list of serialized fields or by negative integer back-reference to previously encountered object. In our case we have two objects:
5 - com.google.common.money.CurrencyCode/112449834, which only has one integer field: 235
6 - java.util.ArrayList/4159755760, which has one integer length field 13, followed by 13 serialized list items. Note that 12 of them are CurrencyCode objects serialized just as the above one, and the last one is a backreference (-1) to the very first object we encountered while (de-)serializing this request, i.e. CurrencyCode(235)
Response format
//OK[235,3,'D0JA',2,86,3,'CXXg',2,55,3,'DW2A',2,204,3,'X9NA',2,205,3,'EuvA',2,175,3,'VIig',2,177,3,'E2Dw',2,121,3,'E4ziA',2,102,3,'do$Q',2,72,3,'T82w',2,53,3,'Ds0Q',2,81,3,'Cq5g',2,18,3,'Dlfg',2,13,1,["com.google.common.collect.RegularImmutableList/440499227","com.google.common.money.SimpleMoney/627983206","com.google.common.money.CurrencyCode/112449834"],0,7]
The response is very similar in format to the request except it's JS-formatted array (though not JSON, as it uses invalid single quotes), and it's in reverse order.
The field meaning is as follows:
7 - protocol version
0 - flags, same as for request
Array of string tokens:
com.google.common.collect.RegularImmutableList/440499227
com.google.common.money.SimpleMoney/627983206
com.google.common.money.CurrencyCode/112449834
And then goes one serialized object of type 1 - com.google.common.collect.RegularImmutableList/440499227 with one integer length field 13, followed by 13 serialized objects of class 2 - com.google.common.money.SimpleMoney/627983206. Each SimpleMoney object has two fields, for example:
'Dlfg' - long integer field encoded as base64 number. This particular one is 940000
3, 18 - CurrencyCode object with integer field 18
What you are looking at is GWT-RPC serialization format. Unfortunatelly it is not publicly documented. Fortunatelly GWT is open-source so you could look at the source to see how it is produced.
Note: This format might change between GWT versions (I known it did in 2.2). This is most likelly also a reason why Google does not document it - if they did they'd need to keep it backward compatible.
Class names that you see are Java classes that Google Checkout uses internally. When GWT is compiled to JS the names get mangled so you don't see them any more.
As noted this is GWT-RPC.
What you are trying to do is reverse-engineer Google internal APIs. I wouldn't do that because, a. It might change without notice, breaking your app and, b. I'm sure Goog wouldn't like it and it probably violates the service agreement (have you read it?).
I have some code made in VB that may be useful for you to realize how to parse GWT Serialized strings. "Datos" contains the string you received.
aAux = Split(Datos, ",[")
aAux(1) = Replace(aAux(1), "],0,7]", "")
aAux(0) = Replace(aAux(0), "//OK[", "")
aAux(0) = Replace(aAux(0), "'", "")
aDescripcion = Split(aAux(1), """,""")
aValor = Split(aAux(0), ",")
InvertirArray aValor
For X = 0 To UBound(aValor)
If Not IsNumeric(aValor(X)) Then
Exit For
End If
If adescripcion(Int(aValor(X))-1) = "gov.senasa.embalajemadera.shared.domain.Pais/3238585366" Then
For Y = X + 1 To UBound(aValor)
If Int(aValor(Y)) = "" Then '- Do what you want
end if
If adescripcion(Int(aValor(Y))) = "java.lang.Integer/3438268394" Then
'- Do what you want
Next Y
End If
Next X
Of course you have to adapt it to your needs and you will have to play a little bit with the arrays...
InvertirArray:
Public Sub InvertirArray(ByRef Arr() As String)
'- el array va tiene que empezar en 0
Dim X As Long
Dim Hasta As Long
Dim Tmp As String
If UBound(Arr) Mod 2 = 0 Then
'- Es impar
Hasta = UBound(Arr) + 1
Else
Hasta = UBound(Arr)
End If
For X = LBound(Arr) To UBound(Arr) \ 2
Tmp = Arr(X)
Arr(X) = Arr(UBound(Arr) - X)
Arr(UBound(Arr) - X) = Tmp
Next X
end sub
And of course you need to decode and encode Long Numbers and dates. So:
Public Function EncodeDateGwt(Numero As Double, Optional isDate As Boolean = False) As String
Dim s As String
Dim a As Double
Dim i As Integer
Dim u As Integer
Dim Base As String
Numero = IIf(isDate, Numero * 1000, Numero)
Base = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_$"
Do While Val(Numero) <> 0
a = Numero
i = 0
Do While a >= 64
i = i + 1
a = a / 64
Loop
If i <> u - 1 And u <> 0 Then EncodeDateGwt = EncodeDateGwt & String(u - i - 1, Left(Base, 1))
a = Int(a)
EncodeDateGwt = EncodeDateGwt + Mid(Base, a + 1, 1)
Numero = Numero - a * (64 ^ i)
u = i
Loop
EncodeDateGwt = EncodeDateGwt & String(i, Left(Base, 1))
End Function
Public Function DecodeDateGwt(Texto As String, Optional isDate As Boolean = False) As Long
Dim Base As String
Dim a As Integer
Base = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_$"
For a = 1 To Len(Texto)
DecodeDateGwt = DecodeDateGwt + (InStr(Base, Mid(Texto, a, 1)) - 1) * (Len(Base) ^ ((Len(Texto) - (a))))
Next
DecodeDateGwt = IIf(isDate, DecodeDateGwt / 1000, DecodeDateGwt)
'devuelve timestamp
End Function
If what you need to encode/decode is a date, then you need to do this before:
Call encodegwtdate(date2unix("20/02/2016"),true)
Public Function Date2Unix(ByVal vDate As Date) As Long
Date2Unix = DateDiff("s", Unix1970, vDate)
End Function
Public Function Unix2Date(vUnixDate As Long) As Date
Unix2Date = DateAdd("s", vUnixDate, Unix1970)
End Function
Hope you solve it. By the way, does anyone knows what negative numbers means?????