Situation :
We have a product with approx 30 attributes (String, Enum, Double)
values
We have iMap with indexes for all attributes IndexType.HASH for
string value and IndexType.SORTED for double values. (900MB together)
We have 300k products in map.(aprox 500MB )
We use local Datagrid with one member
JVM config: -Xms6G -Xmx8G
For HZ 5: we enabled JetConfig
config.getJetConfig().setEnabled(true);
Use Java AdoptOpenJDK 11.0.8
When invoking SQL query with pagination in HZ4 we got a response approx in 20-50ms, but the same query in Hazelcast 5 we got results in 2000-2500 ms
...ORDER BY param1 ASC LIMIT 20 OFFSET 0...
SqlResult sqlRows = hazelcastInstance.getSql().execute(sqlBuilder.toString());
When we tried to use predicates on the same map and in HZ4 and HZ5 we got the same results about 2000-2500 ms to get predicated page
PagingPredicate<Long, Product> pagingPredicate = Predicates.pagingPredicate(predicate, ProductComparatorFactory.getEntryComparator(sortName), max);
pagingPredicate.setPage(from / max);
///get final list of products
List<Product> selectedPageA = new ArrayList<>(productMap.getAll(productMap.keySet(pagingPredicate)).values());
For HZ 5 we add Mapping
hazelcastInstance.getSql().execute("CREATE MAPPING "ProductScreenerRepositoryProductMap" EXTERNAL NAME "ProductScreenerRepositoryProductMap"
TYPE IMap
OPTIONS (
'keyFormat' = 'java',
'keyJavaClass' = 'java.lang.Long',
'valueFormat' = 'java',
'valueJavaClass' = 'com.finmason.finriver.product.Product'
)");
}
There is used SQL
SELECT * FROM ProductScreenerRepositoryProductMap
WHERE doubleValue1 >= -0.9624378795139998
AND doubleValue1 <= 0.9727269574354098
AND doubleValue2 >= -0.9
AND doubleValue2 <= 0.9
ORDER BY doubleValue3 ASC LIMIT 20 OFFSET 0
And Product use simple serialization
Please upgrade to Hazelcast 5.1 (planned for February 23 right now).
It should be fixed with https://github.com/hazelcast/hazelcast/pull/20681
Actually this case will speed up by 3 separate PRs from 5.1:
https://github.com/hazelcast/hazelcast/pull/20681 - this one make your query use index
https://github.com/hazelcast/hazelcast/pull/20402 - this one will do less deserialization on cluster side
https://github.com/hazelcast/hazelcast/pull/20398 - this one makes deserialization on client side faster for multi-column queries
There are two cases not resolved in 5.1, they are described in https://github.com/hazelcast/hazelcast/pull/20796 - it should not be
a problem in your case, but if someone else see this post, it may be his/her. I hope that fix will be delivered in 5.1.1.
If you have a possibility to upgrade to full 5.1 after the release then I strongly recommend you to do it.
I am writing a program that gets From IP address and To IP address from the user and displays the list of IP addresses between them. For example, if the user gives 10.0.0.1 and 10.0.0.5 then I will display the five IP addresses between these two. The current solution that is coming in my mind are:
To have a list of all IP addresses and then look for the resultant IP address list
Use a nested loop
What solution should I adopt between these (or suggest a better solution)? For the first solution, what is the link for IP address table/list?
What is the solution in terms of JavaScript or Java?
First split the IP addresses with .. From the first IP address, start increasing the fourth part up to 255 and then add 1 to the third part and set the fourth one to 1. Until you reach the to IP address.
IP address bytes -> bits -> Int32
From: 10.0.10.10 -> 00001010 00000000 00001010 00001010 -> 167774730
To: 10.1.45.1 -> 00001010 00000001 00101101 00000001 -> 167849217
Start count from From to To and just check the unwanted bytes which is 11111111 and 00000000.
That's all.
The "dot"-writing is for humans. For computers, it is one 4-byte-number. So parse it to a number. Then you will get all addresses in the range by simply increasing a number until the bound is reached and format them back for output.
I was experimenting in an updated jsFiddle, and finally I came to the solution below. The following code should work for all IP addresses. You have to provide a start and end IP address in hex (since it is easy, I did not write code for it).
var startIp = 0x0A000001,
endIp = 0x0A000F05;
var temp, list = [],str;
for(var i=startIp ; i <= endIp ; i++){
temp = (i).toString(16);
str ='';
if(temp.length == 7){
temp = "0"+temp;
}
for(var k=temp.length-1; k >= 0 ; k-=2){
str = parseInt(temp[k-1] + "" + temp[k], 16) +"." + str ;
}
document.write(temp + " " + str+ "<br>");
list.push(str.substring(0, str.length-1));
}
?
If you know enough to get the addresses (from the network or from the file system or user input), you can test the address itself with subtraction and get the number of IP addresses right there.
This is simplified, but you will get it if you know about addresses: 000044-000002 = 000042.
Messy, complicated question, but here goes. I'm working on an integration project with Google Checkout, and there is a Google Checkout GWT service that returns the currency conversion rates used by the Checkout web interface to convert USD into local currencies. This endpoint is hosted at https://market.android.com/publish/gwt/, and staring at Firebug I see this going to the server:
7|0|6|https://market.android.com/publish/gwt/|FCCA4108CB89BFC2FEC78BA7363D4AF6|com.google.wireless.android.vending.developer.
shared.MerchantService|getCurrencyExchangeRates|com.google.common.money.CurrencyCode/112449834|java.util.ArrayList/4159755760
|1|2|3|4|2|5|6|5|235|6|13|5|18|5|81|5|53|5|72|5|102|5|121|5|177|5|175|5|205|5|204|5|55|5|86|-1|
and this being returned
//OK[235,3,'D0JA',2,86,3,'CXXg',2,55,3,'DW2A',2,204,3,'X9NA',2,205,3,'EuvA',2,175,3,'VIig',2,177,3,'E2Dw',2,121,3,'E4ziA',2,1
02,3,'do$Q',2,72,3,'T82w',2,53,3,'Ds0Q',2,81,3,'Cq5g',2,18,3,'Dlfg',2,13,1,["com.google.common.collect.RegularImmutableList/4
40499227","com.google.common.money.SimpleMoney/627983206","com.google.common.money.CurrencyCode/112449834"],0,7]
Forgive the odd formatting: can't quite get the code block to format right.
Wandering the web for hours on end I was able to determine that the RegularImmutableList class is in the Guava libraries (at http://code.google.com/p/guava-libraries/). What I'm looking for is:
I can't find the com.google.common.money.SimpleMoney or com.google.common.money.CurrencyCode classes anywhere: anyone seen them?
The GWT wire format appears to be an odd JSON string. I see various references to Google Groups messages talking about descriptions of the wire format, but can't find the underlying messages or any coherent reference that would let me reverse this: anyone have a handle on a handy reference? If I can at least understand WHAT the encoding is I might be able to get away without the class files from question 1 above.
I started wandering through the Android Market api library at http://code.google.com/p/android-market-api/, figuring they have to have done SOME of the Android Market communication integration, and they appear to have done so using protobufs. Is there any decent reference for the GWT/protobufs communication bits?
The underlying reason for this craziness is that I need to be able to take regular exchange rate values from Google Checkout so when I'm importing sales transactions in foreign currencies I can do the conversion at the prevailing rate at the time of the transaction. The current Checkout reporting formats do NOT provide this, so most folks end up using alternative sources of exchange rates that don't match what Google uses. It is clearly a shortcoming on the part of Google Checkout's integration interface, but if we got started on shortcomings of Google Checkout's interface we'd be here all week. My intention is to poll the Checkout interface for newly fulfilled orders and then request the appropriate exchange rate table so I can figure out in near real-time what the incoming payments are. I've got the polling bit down pat but can't quite get past the exchange rate bit.
While trying to create a script to bulk upload in-app products for my application (CSV upload constantly failed with obscure error messages), I have managed to understand the GWT AJAX protocol.
It's actually pretty simple, except it requires you to know structure of all used classes. Or guess it, as is the case with internal classes used by Google. :)
I'll use examples from the question to explain the protocol in detail.
Request format
7|0|6|https://market.android.com/publish/gwt/|FCCA4108CB89BFC2FEC78BA7363D4AF6|com.google.wireless.android.vending.developer.shared.MerchantService|getCurrencyExchangeRates|com.google.common.money.CurrencyCode/112449834|java.util.ArrayList/4159755760|1|2|3|4|2|5|6|5|235|6|13|5|18|5|81|5|53|5|72|5|102|5|121|5|177|5|175|5|205|5|204|5|55|5|86|-1|
The request is pipe-delimited list of tokens with the following meaning:
7 - protocol version
0 - flags. 1 is FLAG_ELIDE_TYPE_NAMES, 2 is FLAG_RPC_TOKEN_INCLUDED
6 - string token count
6 string tokens:
https://market.android.com/publish/gwt/
FCCA4108CB89BFC2FEC78BA7363D4AF6
com.google.wireless.android.vending.developer.shared.MerchantService
getCurrencyExchangeRates
com.google.common.money.CurrencyCode/112449834
java.util.ArrayList/4159755760
The actual encoded request, which references strings from the list above using 1-based indices:
1 - https://market.android.com/publish/gwt/ - base URL
2 - FCCA4108CB89BFC2FEC78BA7363D4AF6 - some hash, which is references as serializationPolicyStrongName in GWT sources.
3 - com.google.wireless.android.vending.developer.shared.MerchantService - service name
4 - getCurrencyExchangeRates - method name
2 - parameter count. Parameter types follow:
5 - com.google.common.money.CurrencyCode/112449834
6 - java.util.ArrayList/4159755760
Serialized parameters. Each object is represented either by its classname and list of serialized fields or by negative integer back-reference to previously encountered object. In our case we have two objects:
5 - com.google.common.money.CurrencyCode/112449834, which only has one integer field: 235
6 - java.util.ArrayList/4159755760, which has one integer length field 13, followed by 13 serialized list items. Note that 12 of them are CurrencyCode objects serialized just as the above one, and the last one is a backreference (-1) to the very first object we encountered while (de-)serializing this request, i.e. CurrencyCode(235)
Response format
//OK[235,3,'D0JA',2,86,3,'CXXg',2,55,3,'DW2A',2,204,3,'X9NA',2,205,3,'EuvA',2,175,3,'VIig',2,177,3,'E2Dw',2,121,3,'E4ziA',2,102,3,'do$Q',2,72,3,'T82w',2,53,3,'Ds0Q',2,81,3,'Cq5g',2,18,3,'Dlfg',2,13,1,["com.google.common.collect.RegularImmutableList/440499227","com.google.common.money.SimpleMoney/627983206","com.google.common.money.CurrencyCode/112449834"],0,7]
The response is very similar in format to the request except it's JS-formatted array (though not JSON, as it uses invalid single quotes), and it's in reverse order.
The field meaning is as follows:
7 - protocol version
0 - flags, same as for request
Array of string tokens:
com.google.common.collect.RegularImmutableList/440499227
com.google.common.money.SimpleMoney/627983206
com.google.common.money.CurrencyCode/112449834
And then goes one serialized object of type 1 - com.google.common.collect.RegularImmutableList/440499227 with one integer length field 13, followed by 13 serialized objects of class 2 - com.google.common.money.SimpleMoney/627983206. Each SimpleMoney object has two fields, for example:
'Dlfg' - long integer field encoded as base64 number. This particular one is 940000
3, 18 - CurrencyCode object with integer field 18
What you are looking at is GWT-RPC serialization format. Unfortunatelly it is not publicly documented. Fortunatelly GWT is open-source so you could look at the source to see how it is produced.
Note: This format might change between GWT versions (I known it did in 2.2). This is most likelly also a reason why Google does not document it - if they did they'd need to keep it backward compatible.
Class names that you see are Java classes that Google Checkout uses internally. When GWT is compiled to JS the names get mangled so you don't see them any more.
As noted this is GWT-RPC.
What you are trying to do is reverse-engineer Google internal APIs. I wouldn't do that because, a. It might change without notice, breaking your app and, b. I'm sure Goog wouldn't like it and it probably violates the service agreement (have you read it?).
I have some code made in VB that may be useful for you to realize how to parse GWT Serialized strings. "Datos" contains the string you received.
aAux = Split(Datos, ",[")
aAux(1) = Replace(aAux(1), "],0,7]", "")
aAux(0) = Replace(aAux(0), "//OK[", "")
aAux(0) = Replace(aAux(0), "'", "")
aDescripcion = Split(aAux(1), """,""")
aValor = Split(aAux(0), ",")
InvertirArray aValor
For X = 0 To UBound(aValor)
If Not IsNumeric(aValor(X)) Then
Exit For
End If
If adescripcion(Int(aValor(X))-1) = "gov.senasa.embalajemadera.shared.domain.Pais/3238585366" Then
For Y = X + 1 To UBound(aValor)
If Int(aValor(Y)) = "" Then '- Do what you want
end if
If adescripcion(Int(aValor(Y))) = "java.lang.Integer/3438268394" Then
'- Do what you want
Next Y
End If
Next X
Of course you have to adapt it to your needs and you will have to play a little bit with the arrays...
InvertirArray:
Public Sub InvertirArray(ByRef Arr() As String)
'- el array va tiene que empezar en 0
Dim X As Long
Dim Hasta As Long
Dim Tmp As String
If UBound(Arr) Mod 2 = 0 Then
'- Es impar
Hasta = UBound(Arr) + 1
Else
Hasta = UBound(Arr)
End If
For X = LBound(Arr) To UBound(Arr) \ 2
Tmp = Arr(X)
Arr(X) = Arr(UBound(Arr) - X)
Arr(UBound(Arr) - X) = Tmp
Next X
end sub
And of course you need to decode and encode Long Numbers and dates. So:
Public Function EncodeDateGwt(Numero As Double, Optional isDate As Boolean = False) As String
Dim s As String
Dim a As Double
Dim i As Integer
Dim u As Integer
Dim Base As String
Numero = IIf(isDate, Numero * 1000, Numero)
Base = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_$"
Do While Val(Numero) <> 0
a = Numero
i = 0
Do While a >= 64
i = i + 1
a = a / 64
Loop
If i <> u - 1 And u <> 0 Then EncodeDateGwt = EncodeDateGwt & String(u - i - 1, Left(Base, 1))
a = Int(a)
EncodeDateGwt = EncodeDateGwt + Mid(Base, a + 1, 1)
Numero = Numero - a * (64 ^ i)
u = i
Loop
EncodeDateGwt = EncodeDateGwt & String(i, Left(Base, 1))
End Function
Public Function DecodeDateGwt(Texto As String, Optional isDate As Boolean = False) As Long
Dim Base As String
Dim a As Integer
Base = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_$"
For a = 1 To Len(Texto)
DecodeDateGwt = DecodeDateGwt + (InStr(Base, Mid(Texto, a, 1)) - 1) * (Len(Base) ^ ((Len(Texto) - (a))))
Next
DecodeDateGwt = IIf(isDate, DecodeDateGwt / 1000, DecodeDateGwt)
'devuelve timestamp
End Function
If what you need to encode/decode is a date, then you need to do this before:
Call encodegwtdate(date2unix("20/02/2016"),true)
Public Function Date2Unix(ByVal vDate As Date) As Long
Date2Unix = DateDiff("s", Unix1970, vDate)
End Function
Public Function Unix2Date(vUnixDate As Long) As Date
Unix2Date = DateAdd("s", vUnixDate, Unix1970)
End Function
Hope you solve it. By the way, does anyone knows what negative numbers means?????