My scenario is like this.
Solr Indexing happens for a product and then product approval status is made unapproved from backoffice. After then, when you search the related words that is placed in description of the product or directly product code from website, you get a server error since the product that is made unapproved is still placed in solr.
If you perform any type of indexing from backoffice manually, it works again. But it is not a good solution since there might be lots of products whose status is changed or that is not a solution which happens instantly. If you use cronjob for indexing, that is not a fast solution again.You get server error until cronjob starts to work.
I would like to update solr index instantly for the attributes which changes frequently like price, status, etc. For instance, when an attribute changes, Is it a good way to start partial index immediately in java code? If it is, how? (by IndexerService?). For another solution, Is it a better idea to make http request to solr for the attribute?
In summary, I am looking for the best solution to perform partial index.
Any ideas?
For this case you need to write two new important SOLR-Configuration parts:
1) A new SOLR-Cronjob that trigger the indexing
2) A new SOLR-IndexerQuery for indexing with your special requirements.
When you have a look at the default stuff from hybris you see:
INSERT_UPDATE CronJob;code[unique=true];job(code);singleExecutable;sessionLanguage(isocode);active;
;backofficeSolrIndexerUpdateCronJob;backofficeSolrIndexerUpdateJob;false;en;false;
INSERT Trigger;cronJob(code);active;activationTime;year;month;day;hour;minute;second;relative;weekInterval;daysOfWeek;
;backofficeSolrIndexerUpdateCronJob;true;;-1;-1;-1;-1;-1;05;false;0;;
This part above is to configure when the job should run. You can modify him, that he should run ever 5 seconds for example.
INSERT_UPDATE SolrIndexerQuery; solrIndexedType(identifier)[unique = true]; identifier[unique = true]; type(code); injectCurrentDate[default = true]; injectCurrentTime[default = true]; injectLastIndexTime[default = true]; query; user(uid)
; $solrIndexedType ; $solrIndexedType-updateQuery ; update ; false ; false ; false ; "SELECT DISTINCT {PK} FROM {Product AS p JOIN VariantProduct AS vp ON {p.PK}={vp.baseProduct} } WHERE {p.modifiedtime} >= ?lastStartTimeWithSuccess OR {vp.modifiedtime} >= ?lastStartTimeWithSuccess" ; admin
The second part here is the more important. Here you define which products should be indexed. Here you can see that the UPDATE-Job is looking for every Product that was modified. Here you could write a new FlexibleSearch with your special requirements.
tl;tr Answear: You have to write a new performant solrIndexerQuery that could be trigger every 5 seconds
Java, hibernate + oracle. Users are stored in datababase and have national characters in their names (ü, ß etc). Now I need to create a convenient search function. Example: when user type 'do', 'dö' or 'doe' then entry 'Fidörner' should be found. Currently there are only 3 rules like above one defined by business but I expect there will more.
what are the recommended solutions? google search appliance? lucene? hibernate search? solr? custom text normalization (can it be done quickly)? any other tools?
Take a look at Oracle Text. It's a free option installed by default on your database.
For example:
create table users(name varchar2(100));
insert into users values ('Fidörner');
insert into users values ('Fido');
insert into users values ('Smith');
commit;
begin
ctx_ddl.drop_preference('mylex');
ctx_ddl.create_preference('mylex', 'basic_lexer');
ctx_ddl.set_attribute('mylex', 'base_letter', 'YES');
ctx_ddl.set_attribute('mylex','alternate_spelling','german');
end;
/
create index users_index on users(name)
indextype is ctxsys.ctxcat parameters ('LEXER mylex');
select * from users where catsearch(name, '**do*', null) > 0;
select * from users where catsearch(name, '**dö*', null) > 0;
select * from users where catsearch(name, '**doe*', null) > 0;
All return:
Name
----
Fidörner
Fido
I'd like iterate over every document in a (probably big) Lotus Domino database and be able to continue it from the last one if the processing breaks (network connection error, application restart etc.). I don't have write access to the database.
I'm looking for a way where I don't have to download those documents from the server which were already processed. So, I have to pass some starting information to the server which document should be the first in the (possibly restarted) processing.
I've checked the AllDocuments property and the DocumentColletion.getNthDocument method but this property is unsorted so I guess the order can change between two calls.
Another idea was using a formula query but it does not seem that ordering is possible with these queries.
The third idea was the Database.getModifiedDocuments method with a corresponding Document.getLastModified one. It seemed good but
it looks to me that the ordering of the returned collection is not documented and based on creation time instead of last modification time.
Here is a sample code based on the official example:
System.out.println("startDate: " + startDate);
final DocumentCollection documentCollection =
database.getModifiedDocuments(startDate, Database.DBMOD_DOC_DATA);
Document doc = documentCollection.getFirstDocument();
while (doc != null) {
System.out.println("#lastmod: " + doc.getLastModified() +
" #created: " + doc.getCreated());
doc = documentCollection.getNextDocument(doc);
}
It prints the following:
startDate: 2012.07.03 08:51:11 CEDT
#lastmod: 2012.07.03 08:51:11 CEDT #created: 2012.02.23 10:35:31 CET
#lastmod: 2012.08.03 12:20:33 CEDT #created: 2012.06.01 16:26:35 CEDT
#lastmod: 2012.07.03 09:20:53 CEDT #created: 2012.07.03 09:20:03 CEDT
#lastmod: 2012.07.21 23:17:35 CEDT #created: 2012.07.03 09:24:44 CEDT
#lastmod: 2012.07.03 10:10:53 CEDT #created: 2012.07.03 10:10:41 CEDT
#lastmod: 2012.07.23 16:26:22 CEDT #created: 2012.07.23 16:26:22 CEDT
(I don't use any AgentContext here to access the database. The database object comes from a session.getDatabase(null, databaseName) call.)
Is there any way to reliably do this with the Lotus Domino Java API?
If you have access to change the database, or could ask someone to do so, then you should create a view that is sorted on a unique key, or modified date, and then just store the "pointer" to the last document processed.
Barring that, you'll have to maintain a list of previously processed documents yourself. In that case you can use the AllDocuments property and just iterate through them. Use the GetFirstDocument and GetNextDocument as they are reportedly faster than GetNthDocument.
Alternatively you could make two passes, one to gather a list of UNIDs for all documents, which you'll store, and then make a second pass to process each document from the list of UNIDs you have (using GetDocumentByUNID method).
I don't use the Java API, but in Lotusscript, I would do something like this:
Locate a view displaying all documents in the database. If you want the agent to be really fast, create a new view. The first column should be sorted and could contain the Universal ID of the document. The other columns contains all the values you want to read in your agent, in your example that would be the created date and last modified date.
Your code could then simply loop through the view like this:
lastSuccessful = FunctionToReadValuesSomewhere() ' Returns 0 if empty
Set view = thisdb.GetView("MyLookupView")
Set col = view.AllEntries
Set entry = col.GetFirstEntry
cnt = 0
Do Until entry is Nothing
cnt = cnt + 1
If cnt > lastSuccessful Then
universalID = entry.ColumnValues(0)
createDate = entry.ColumnValues(1)
lastmodifiedDate = entry.ColumnValues(2)
Call YourFunctionToDoStuff(universalID, createDate, lastmodifiedDate)
Call FunctionToStoreValuesSomeWhere(cnt, universalID)
End If
Set entry = col.GetFirstEntry
Loop
Call FunctionToClearValuesSomeWhere()
Simply store the last successful value and Universal ID in say a text file or environment variable or even profile document in the database.
When you restart the agent, have some code that check if the values are blank (then return 0), otherwise return the last successful value.
Agents already keep a field to describe documents that they have not yet processed, and these are automatically updated via normal processing.
A better way of doing what you're attempting to do might be to store the results of a search in a profile document. However, if you're trying to relate to documents in a database you do not have write permission to, the only thing you can do is keep a list of the doclinks you've already processed (and any information you need to keep about those documents), or a sister database holding one document for each doclink plus multiple fields related to the processing you've done on them. Then, transfer the lists of IDs and perform the matching on the client to do per-document lookups.
Lotus Notes/Domino databases are designed to be distributed across clients and servers in a replicated environment. In the general case, you do not have a guarantee that starting at a given creation or mod time will bring you consistent results.
If you are 100% certain that no replicas of your target database are ever made, then you can use getModifiedDocuments and then write a sort routine to place (modDateTime,UNID) pairs into a SortedSet or other suitable data structure. Then you can process through the Set, and if you run into an error you can save the modDateTime of the element that you were attempting to process as your restart point. There may be a few additional details for you to work out to avoid duplicates, however, if there are multiple documents with the exact same modDateTime stamp.
I want to make one final remark. I understand that you are asking about Java, but if you are working on a backup or archiving system for compliance purposes, the Lotus C API has special functions that you really should look at.
Messy, complicated question, but here goes. I'm working on an integration project with Google Checkout, and there is a Google Checkout GWT service that returns the currency conversion rates used by the Checkout web interface to convert USD into local currencies. This endpoint is hosted at https://market.android.com/publish/gwt/, and staring at Firebug I see this going to the server:
7|0|6|https://market.android.com/publish/gwt/|FCCA4108CB89BFC2FEC78BA7363D4AF6|com.google.wireless.android.vending.developer.
shared.MerchantService|getCurrencyExchangeRates|com.google.common.money.CurrencyCode/112449834|java.util.ArrayList/4159755760
|1|2|3|4|2|5|6|5|235|6|13|5|18|5|81|5|53|5|72|5|102|5|121|5|177|5|175|5|205|5|204|5|55|5|86|-1|
and this being returned
//OK[235,3,'D0JA',2,86,3,'CXXg',2,55,3,'DW2A',2,204,3,'X9NA',2,205,3,'EuvA',2,175,3,'VIig',2,177,3,'E2Dw',2,121,3,'E4ziA',2,1
02,3,'do$Q',2,72,3,'T82w',2,53,3,'Ds0Q',2,81,3,'Cq5g',2,18,3,'Dlfg',2,13,1,["com.google.common.collect.RegularImmutableList/4
40499227","com.google.common.money.SimpleMoney/627983206","com.google.common.money.CurrencyCode/112449834"],0,7]
Forgive the odd formatting: can't quite get the code block to format right.
Wandering the web for hours on end I was able to determine that the RegularImmutableList class is in the Guava libraries (at http://code.google.com/p/guava-libraries/). What I'm looking for is:
I can't find the com.google.common.money.SimpleMoney or com.google.common.money.CurrencyCode classes anywhere: anyone seen them?
The GWT wire format appears to be an odd JSON string. I see various references to Google Groups messages talking about descriptions of the wire format, but can't find the underlying messages or any coherent reference that would let me reverse this: anyone have a handle on a handy reference? If I can at least understand WHAT the encoding is I might be able to get away without the class files from question 1 above.
I started wandering through the Android Market api library at http://code.google.com/p/android-market-api/, figuring they have to have done SOME of the Android Market communication integration, and they appear to have done so using protobufs. Is there any decent reference for the GWT/protobufs communication bits?
The underlying reason for this craziness is that I need to be able to take regular exchange rate values from Google Checkout so when I'm importing sales transactions in foreign currencies I can do the conversion at the prevailing rate at the time of the transaction. The current Checkout reporting formats do NOT provide this, so most folks end up using alternative sources of exchange rates that don't match what Google uses. It is clearly a shortcoming on the part of Google Checkout's integration interface, but if we got started on shortcomings of Google Checkout's interface we'd be here all week. My intention is to poll the Checkout interface for newly fulfilled orders and then request the appropriate exchange rate table so I can figure out in near real-time what the incoming payments are. I've got the polling bit down pat but can't quite get past the exchange rate bit.
While trying to create a script to bulk upload in-app products for my application (CSV upload constantly failed with obscure error messages), I have managed to understand the GWT AJAX protocol.
It's actually pretty simple, except it requires you to know structure of all used classes. Or guess it, as is the case with internal classes used by Google. :)
I'll use examples from the question to explain the protocol in detail.
Request format
7|0|6|https://market.android.com/publish/gwt/|FCCA4108CB89BFC2FEC78BA7363D4AF6|com.google.wireless.android.vending.developer.shared.MerchantService|getCurrencyExchangeRates|com.google.common.money.CurrencyCode/112449834|java.util.ArrayList/4159755760|1|2|3|4|2|5|6|5|235|6|13|5|18|5|81|5|53|5|72|5|102|5|121|5|177|5|175|5|205|5|204|5|55|5|86|-1|
The request is pipe-delimited list of tokens with the following meaning:
7 - protocol version
0 - flags. 1 is FLAG_ELIDE_TYPE_NAMES, 2 is FLAG_RPC_TOKEN_INCLUDED
6 - string token count
6 string tokens:
https://market.android.com/publish/gwt/
FCCA4108CB89BFC2FEC78BA7363D4AF6
com.google.wireless.android.vending.developer.shared.MerchantService
getCurrencyExchangeRates
com.google.common.money.CurrencyCode/112449834
java.util.ArrayList/4159755760
The actual encoded request, which references strings from the list above using 1-based indices:
1 - https://market.android.com/publish/gwt/ - base URL
2 - FCCA4108CB89BFC2FEC78BA7363D4AF6 - some hash, which is references as serializationPolicyStrongName in GWT sources.
3 - com.google.wireless.android.vending.developer.shared.MerchantService - service name
4 - getCurrencyExchangeRates - method name
2 - parameter count. Parameter types follow:
5 - com.google.common.money.CurrencyCode/112449834
6 - java.util.ArrayList/4159755760
Serialized parameters. Each object is represented either by its classname and list of serialized fields or by negative integer back-reference to previously encountered object. In our case we have two objects:
5 - com.google.common.money.CurrencyCode/112449834, which only has one integer field: 235
6 - java.util.ArrayList/4159755760, which has one integer length field 13, followed by 13 serialized list items. Note that 12 of them are CurrencyCode objects serialized just as the above one, and the last one is a backreference (-1) to the very first object we encountered while (de-)serializing this request, i.e. CurrencyCode(235)
Response format
//OK[235,3,'D0JA',2,86,3,'CXXg',2,55,3,'DW2A',2,204,3,'X9NA',2,205,3,'EuvA',2,175,3,'VIig',2,177,3,'E2Dw',2,121,3,'E4ziA',2,102,3,'do$Q',2,72,3,'T82w',2,53,3,'Ds0Q',2,81,3,'Cq5g',2,18,3,'Dlfg',2,13,1,["com.google.common.collect.RegularImmutableList/440499227","com.google.common.money.SimpleMoney/627983206","com.google.common.money.CurrencyCode/112449834"],0,7]
The response is very similar in format to the request except it's JS-formatted array (though not JSON, as it uses invalid single quotes), and it's in reverse order.
The field meaning is as follows:
7 - protocol version
0 - flags, same as for request
Array of string tokens:
com.google.common.collect.RegularImmutableList/440499227
com.google.common.money.SimpleMoney/627983206
com.google.common.money.CurrencyCode/112449834
And then goes one serialized object of type 1 - com.google.common.collect.RegularImmutableList/440499227 with one integer length field 13, followed by 13 serialized objects of class 2 - com.google.common.money.SimpleMoney/627983206. Each SimpleMoney object has two fields, for example:
'Dlfg' - long integer field encoded as base64 number. This particular one is 940000
3, 18 - CurrencyCode object with integer field 18
What you are looking at is GWT-RPC serialization format. Unfortunatelly it is not publicly documented. Fortunatelly GWT is open-source so you could look at the source to see how it is produced.
Note: This format might change between GWT versions (I known it did in 2.2). This is most likelly also a reason why Google does not document it - if they did they'd need to keep it backward compatible.
Class names that you see are Java classes that Google Checkout uses internally. When GWT is compiled to JS the names get mangled so you don't see them any more.
As noted this is GWT-RPC.
What you are trying to do is reverse-engineer Google internal APIs. I wouldn't do that because, a. It might change without notice, breaking your app and, b. I'm sure Goog wouldn't like it and it probably violates the service agreement (have you read it?).
I have some code made in VB that may be useful for you to realize how to parse GWT Serialized strings. "Datos" contains the string you received.
aAux = Split(Datos, ",[")
aAux(1) = Replace(aAux(1), "],0,7]", "")
aAux(0) = Replace(aAux(0), "//OK[", "")
aAux(0) = Replace(aAux(0), "'", "")
aDescripcion = Split(aAux(1), """,""")
aValor = Split(aAux(0), ",")
InvertirArray aValor
For X = 0 To UBound(aValor)
If Not IsNumeric(aValor(X)) Then
Exit For
End If
If adescripcion(Int(aValor(X))-1) = "gov.senasa.embalajemadera.shared.domain.Pais/3238585366" Then
For Y = X + 1 To UBound(aValor)
If Int(aValor(Y)) = "" Then '- Do what you want
end if
If adescripcion(Int(aValor(Y))) = "java.lang.Integer/3438268394" Then
'- Do what you want
Next Y
End If
Next X
Of course you have to adapt it to your needs and you will have to play a little bit with the arrays...
InvertirArray:
Public Sub InvertirArray(ByRef Arr() As String)
'- el array va tiene que empezar en 0
Dim X As Long
Dim Hasta As Long
Dim Tmp As String
If UBound(Arr) Mod 2 = 0 Then
'- Es impar
Hasta = UBound(Arr) + 1
Else
Hasta = UBound(Arr)
End If
For X = LBound(Arr) To UBound(Arr) \ 2
Tmp = Arr(X)
Arr(X) = Arr(UBound(Arr) - X)
Arr(UBound(Arr) - X) = Tmp
Next X
end sub
And of course you need to decode and encode Long Numbers and dates. So:
Public Function EncodeDateGwt(Numero As Double, Optional isDate As Boolean = False) As String
Dim s As String
Dim a As Double
Dim i As Integer
Dim u As Integer
Dim Base As String
Numero = IIf(isDate, Numero * 1000, Numero)
Base = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_$"
Do While Val(Numero) <> 0
a = Numero
i = 0
Do While a >= 64
i = i + 1
a = a / 64
Loop
If i <> u - 1 And u <> 0 Then EncodeDateGwt = EncodeDateGwt & String(u - i - 1, Left(Base, 1))
a = Int(a)
EncodeDateGwt = EncodeDateGwt + Mid(Base, a + 1, 1)
Numero = Numero - a * (64 ^ i)
u = i
Loop
EncodeDateGwt = EncodeDateGwt & String(i, Left(Base, 1))
End Function
Public Function DecodeDateGwt(Texto As String, Optional isDate As Boolean = False) As Long
Dim Base As String
Dim a As Integer
Base = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_$"
For a = 1 To Len(Texto)
DecodeDateGwt = DecodeDateGwt + (InStr(Base, Mid(Texto, a, 1)) - 1) * (Len(Base) ^ ((Len(Texto) - (a))))
Next
DecodeDateGwt = IIf(isDate, DecodeDateGwt / 1000, DecodeDateGwt)
'devuelve timestamp
End Function
If what you need to encode/decode is a date, then you need to do this before:
Call encodegwtdate(date2unix("20/02/2016"),true)
Public Function Date2Unix(ByVal vDate As Date) As Long
Date2Unix = DateDiff("s", Unix1970, vDate)
End Function
Public Function Unix2Date(vUnixDate As Long) As Date
Unix2Date = DateAdd("s", vUnixDate, Unix1970)
End Function
Hope you solve it. By the way, does anyone knows what negative numbers means?????