Develop Reference

Going to next activity once the list is complete

So I am working on a project and I don't know much about the android studio. So what I am doing is, I am using jsoup as my web parser and I am creating a list of products from e-commerce websites. So I want that once I get all the details from the parser, then my next activity starts. Till now I am using handler and setting some delay time in it. But that is not helping because sometimes it takes more than 3-4 seconds.
Below is the code by which I am sending the allproduct ArrayList and producturl ArrayList
new Handler().postDelayed(new Runnable() {
#Override
public void run() {
Intent intent = new Intent(MainActivity.this, Main2Activity.class);
Bundle args = new Bundle();
args.putSerializable("ARRAYLIST", (Serializable) allproducts);
args.putSerializable("URLLINKS", (Serializable) producturl);
intent.putExtra("BUNDLE", args);
intent.putExtra(EXTRA_TEXT, searchtext);
startActivity(intent);
}
}, 6500);
Sometimes the listview shows only 2 websites result and sometimes it shows every website.
private class Snapdeal extends AsyncTask<String, Void, ArrayList<String>> {
ArrayList<String> tempurlstore = new ArrayList<>();
String link;
#Override
protected void onPostExecute(ArrayList<String> s) {
String product;
String urlstore;
super.onPostExecute(s);
for (int j = 0; j < 6; j++) {
product = s.get(j);
urlstore = tempurlstore.get(j);
allproducts.add(product);
producturl.add(urlstore);
}
String seemore = "See more products on website....";
allproducts.add(seemore);
producturl.add(link);
}
#Override
protected ArrayList<String> doInBackground(String... strings) {
try {
Document doc = Jsoup.connect(strings[0]).get();
Elements links = doc.getElementsByClass("col-xs-6 favDp product-tuple-listing js-tuple ");
ArrayList<String> mainlist = new ArrayList<String>();
mainlist.add("SNAPDEAL");
link = strings[0];
tempurlstore.add("https://www.snapdeal.com");
for (Element link : links) {
String temp1 = null, temp2 = null, temp3 = null, temp4 = null, temp5 = null;
String permanent1 = null;
Elements elLink = link.getElementsByTag("a");
Elements eltitle = link.getElementsByClass("product-title"); //for product title
Elements elpricebefore = link.getElementsByClass("lfloat product-desc-price strike ");
Elements elpriceafter = link.getElementsByClass("lfloat product-price");
Elements discount = link.getElementsByClass("product-discount");
//product title loop
for (Element titleOfProduct : eltitle) {
temp1 = "Title: " + titleOfProduct.text();
}
//product original price loop
for (Element priceOfProductBefore : elpricebefore) {
temp2 = "Price before: " + priceOfProductBefore.text();
}
//product discounted price loop
for (Element priceOfProductAfter : elpriceafter) {
temp3 = "Discounted price: " + priceOfProductAfter.text();
}
//discount in number loop
for (Element productdiscount : discount) {
temp4 = "Discount: " + productdiscount.text();
}
ArrayList<String> linkArray = new ArrayList<String>();
for (Element elementLink : elLink) {
String MainLink = elementLink.attr("href");
linkArray.add(MainLink);
}
for (int j = 0; j < 1; j++) {
temp5 = linkArray.get(0);
}
if (elpricebefore.text()==null)
{
permanent1 = "\n" + temp1 + "\n" + "Price :" + elpriceafter.text() + "\n" + temp4 + "\n";
}
else
{
permanent1 ="\n" + temp1 + "\n" + temp2 + "\n" + temp3 + "\n" + temp4 + "\n";
}
mainlist.add(permanent1);
tempurlstore.add(temp5);
}
return mainlist;
} catch (Exception e) {
ArrayList<String> exception = new ArrayList<String>();
String ex = e.toString();
exception.add(ex);
return exception;
}
}
}
Above is some part of my project. So this is how I am taking all the products and all from the parser.
So how can I do this that once all the products are ready in the Arraylist then only intent gets started?
Please help. Thank you.

Why are you using a timer instead of a callback from your parser? This is creating a race condition which is at the heart of your issue. The only way around it is to fire your intent only after your parser is complete.
Without seeing the parser code it is tough to know the exact place to add your hook.

You shouldn't use a handler for that. What you could do is use an Observer or use Android's Live Data so the one you choose can let you know when the list is ready so you can start the activity.

what in kafka DefaultRecord value field

Recently, I review the kafka code and test. I found a strange case:
I print the bytebuffer on the entry of SocketServer processCompletedReceives, as well as print the value on the point of Log sotre as follows:
the entry of SocketServer
private def processCompletedReceives() {
selector.completedReceives.asScala.foreach { receive =>
try {
openOrClosingChannel(receive.source) match {
case Some(channel) =>
val header = RequestHeader.parse(receive.payload)
val connectionId = receive.source
val context = new RequestContext(header, connectionId, channel.socketAddress,
channel.principal, listenerName, securityProtocol)
val req = new RequestChannel.Request(processor = id, context = context,
startTimeNanos = time.nanoseconds, memoryPool, receive.payload, requestChannel.metrics)
if(header.apiKey() == ApiKeys.PRODUCE){
LogHelper.log("produce request: %v" + java.util.Arrays.toString(receive.payload.array()))
}
...
the point of Log
validRecords.records().asScala.foreach { record =>
LogHelper.log("buffer info: value " + java.util.Arrays.toString(record.value().array()))
}
but, the result of print is different. and record.value() is not what I passed in client value like this:
public void run() {
int messageNo = 1;
while (true) {
String messageStr = "Message_" + messageNo;
long startTime = System.currentTimeMillis();
if (isAsync) { // Send asynchronously
producer.send(new ProducerRecord<>(topic,
messageNo,
messageStr), new DemoCallBack(startTime, messageNo, messageStr));
} else { // Send synchronously
try {
producer.send(new ProducerRecord<>(topic,
messageNo,
messageStr)).get();
System.out.println("Sent message: (" + messageNo + ", " + messageStr + ")");
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
}
++messageNo;
}
}
the print result is not the not String messageStr = "Message_" + messageNo;
so what happend in the case.

done. I write the code as follows:
public class KVExtractor {
private static final Logger logger = LoggerFactory.getLogger(KVExtractor.class);
public static Map.Entry<byte[], byte[]> extract(Record record) {
if (record.hasKey() && record.hasValue()) {
byte[] key = new byte[record.key().limit()];
record.key().get(key);
byte[] value = new byte[record.value().limit()];
record.value().get(value);
System.out.println("key : " + new String(key) + " value: " + new String(value));
return new AbstractMap.SimpleEntry<byte[], byte[]>(key, value);
}else if(record.hasValue()){
// illegal impl
byte[] data = new byte[record.value().limit()];
record.value().get(data);
System.out.println("no key but with value : " + new String(data));
}
return null;
}
}

JavaFX thread is hanging when using ExecutorService

I'm trying to write a program that uses Imgur's API to download images based on an account name.
private volatile int threadCount;
private volatile double totalFileSize;
private volatile List<String> albums = new ArrayList<>();
private volatile Map<JSONObject, String> images = new HashMap<>();
private final ExecutorService executorService = Executors.newFixedThreadPool(100, (Runnable r) -> {
Thread t = Executors.defaultThreadFactory().newThread(r);
t.setDaemon(true);
return t;
});
private void downloadAlbums(List<String> albums) {
threadCount = 0;
albums.forEach((albumHash) -> {
if (hasRemainingRequests()) {
incThreadCount();
executorService.execute(() -> {
try {
String responseString;
String dirTitle;
String albumUrl = URL_ALBUM + albumHash;
String query = String.format("client_id=%s", URLEncoder.encode(CLIENT_ID, CHARSET));
URLConnection connection = new URL(albumUrl + "?" + query).openConnection();
connection.setRequestProperty("Accept-Charset", CHARSET);
InputStream response = connection.getInputStream();
try (Scanner scanner = new Scanner(response)) {
responseString = scanner.useDelimiter("\\A").next();
JSONObject obj = new JSONObject(responseString).getJSONObject("data");
dirTitle = obj.getString("title");
String temp = "";
// Get save path from a TextField somewhere else on the GUI
ObservableList<Node> nodes = primaryStage.getScene().getRoot().getChildrenUnmodifiable();
for (Node node : nodes) {
if (node instanceof VBox) {
ObservableList<Node> vNodes = ((VBox) node).getChildrenUnmodifiable();
for (Node vNode : vNodes) {
if (vNode instanceof DestinationBrowser) {
temp = ((DestinationBrowser) vNode).getDestination().trim();
}
}
}
}
final String path = temp + "\\" + formatPath(accountName) + "\\" + formatPath(dirTitle);
JSONArray arr = obj.getJSONArray("images");
arr.forEach((jsonObject) -> {
totalFileSize += ((JSONObject) jsonObject).getDouble("size");
images.put((JSONObject) jsonObject, path);
});
}
} catch (IOException ex) {
//
} catch (Exception ex) {
//
} finally {
decThreadCount();
if (threadCount == 0) {
Platform.runLater(() -> {
DecimalFormat df = new DecimalFormat("#.#");
Alert alert = new Alert(Alert.AlertType.CONFIRMATION);// 714833218
alert.setHeaderText("Found " + images.size() + " images (" + (totalFileSize < 1000000000 ? df.format(totalFileSize / 1000000) + " MB)" : df.format(totalFileSize / 1000000000) + " GB)"));
alert.setContentText("Proceed with download and save images?");
Optional<ButtonType> alertResponse = alert.showAndWait();
if (alertResponse.get() == ButtonType.OK) {
progressBar.setTotalWork(images.size());
executorService.execute(() -> {
for (JSONObject obj : images.keySet()) {
(new File(images.get(obj))).mkdirs();
downloadImage(obj, images.get(obj));
}
});
}
});
}
}
});
}
});
}
albums is a List of codes that are required to send a GET request to Imgur in order to receive that album's images. The data returned is then used in another method which downloads the images themselves.
Now, all of this works fine, but when the program is making all the GET requests the JavaFX thread hangs (GUI becomes unresponsive). And after all the GET requests have been executed, the JavaFX thread stops hanging and the alert shows up with the correct information.
I just don't understand why the GUI becomes unresponsive when I'm not (I believe I'm not) blocking it's thread and I'm using an ExecutorService to do all the network requests.

Get All User timeline tweets using twitter4j and java

I have a problem if anyone can help,
I'm trying to get tweets done by a specific user, here's my code:
Paging pg = new Paging();
String userName = "Obama";
pg.setCount(200);
ConfigurationBuilder cb = new ConfigurationBuilder();
cb.setOAuthConsumerKey("");
cb.setOAuthConsumerSecret("");
cb.setOAuthAccessToken("");
cb.setOAuthAccessTokenSecret("");
Twitter twitter = new TwitterFactory(cb.build()).getInstance();
int numberOfTweets = 1000000;
long lastID = Long.MAX_VALUE;
ArrayList<Status> tweets = new ArrayList<Status>();
while (tweets.size () < numberOfTweets) {
tweets.addAll(twitter.getUserTimeline(userName,pg));
//System.out.println("Gathered " + tweets.size() + " tweets");
for (Status t: tweets) {
System.out.println(t.getUser().getName() + ": " + t.getText()+ " " );
};
pg.setMaxId(lastID-1);
}
System.out.println(tweets.size());
}
The problem is that the result is only the same results, the algorithm takes only the first few tweets from the timeline and makes them X time, and the profile has a million of tweets.
Can someone tell me how can I solve this problem please?
Thanks

Here is a way to do :
ArrayList<Status> statuses = new ArrayList<>();
int pageno = 1;
while(true) {
try {
System.out.println("getting tweets");
int size = statuses.size(); // actual tweets count we got
Paging page = new Paging(pageno, 200);
statuses.addAll(twitter.getUserTimeline(screenName, page));
System.out.println("total got : " + statuses.size());
if (statuses.size() == size) { break; } // we did not get new tweets so we have done the job
pageno++;
sleep(1000); // 900 rqt / 15 mn <=> 1 rqt/s
}
catch (TwitterException e) {
System.out.println(e.getErrorMessage());
}
} // while(true)
And you need a sleep function to respect rate limit :
static void sleep(long ms) {
try { Thread.sleep(ms); }
catch(InterruptedException ex) { Thread.currentThread().interrupt(); }
}
Reference : https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html

100 records at a time to udf

I have to pass record to an UDF which calls an API but as we want to do it parallely,we are using spark and thats why UDF is being developed, the problem here is that that UDF needs to take only 100 records at a time not more than that, it can't handle more than 100 records parallely, so how to ensure that only 100 record pass to it in one go please note we don't want to use count() function on whole record.
I am attaching the UDF code here,it's a generic UDF which returns array of struct.moreover if we pass 100 records in batchsize variable each time then,if suppose there are 198 records then if as we dont want to use count() we will not be knowing that its last batchsize is going to be 98.so how to handle that thing.
Guys... I have a generic UDF in which call is made for an API but before calling it creates batch of 100 firstly then only call restapi.. So the argument UDF takes are x1:string, x2:string, batchsize:integer(currently the batchsize is 100)..so in UDF until and unless the batchsize is not 100 the call will not happen.. And for each record it will return null.
So till 99th record it will return. Null but at 100th record the call will happen
[So, now the problem part:as we are taking batchsize 100 and call will take place only at 100th record. So, in condition like if we have suppose 198 record in file then 100 record will get the output but, other 98 will only return null as they will not get processed..
So please help a way around, and UDF take one record at a time, but it keep on collecting till 100th record.. I hope this clears up
public class Standardize_Address extends GenericUDF {
private static final Logger logger = LoggerFactory.getLogger(Standardize_Address.class);
private int counter = 0;
Client client = null;
private Batch batch = new Batch();
public Standardize_Address() {
client = new ClientBuilder().withUrl("https://ss-staging-public.beringmedia.com/street-address").build();
}
// StringObjectInspector streeti;
PrimitiveObjectInspector streeti;
PrimitiveObjectInspector cityi;
PrimitiveObjectInspector zipi;
PrimitiveObjectInspector statei;
PrimitiveObjectInspector batchsizei;
private ArrayList ret;
#Override
public String getDisplayString(String[] argument) {
return "My display string";
}
#Override
public ObjectInspector initialize(ObjectInspector[] args) throws UDFArgumentException {
System.out.println("under initialize");
if (args[0] == null) {
throw new UDFArgumentTypeException(0, "NO Street is mentioned");
}
if (args[1] == null) {
throw new UDFArgumentTypeException(0, "No Zip is mentioned");
}
if (args[2] == null) {
throw new UDFArgumentTypeException(0, "No city is mentioned");
}
if (args[3] == null) {
throw new UDFArgumentTypeException(0, "No State is mentioned");
}
if (args[4] == null) {
throw new UDFArgumentTypeException(0, "No batch size is mentioned");
}
/// streeti =args[0];
streeti = (PrimitiveObjectInspector)args[0];
// this.streetvalue = (StringObjectInspector) streeti;
cityi = (PrimitiveObjectInspector)args[1];
zipi = (PrimitiveObjectInspector)args[2];
statei = (PrimitiveObjectInspector)args[3];
batchsizei = (PrimitiveObjectInspector)args[4];
ret = new ArrayList();
ArrayList structFieldNames = new ArrayList();
ArrayList structFieldObjectInspectors = new ArrayList();
structFieldNames.add("Street");
structFieldNames.add("city");
structFieldNames.add("zip");
structFieldNames.add("state");
structFieldObjectInspectors.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
structFieldObjectInspectors.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
structFieldObjectInspectors.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
structFieldObjectInspectors.add(PrimitiveObjectInspectorFactory.writableStringObjectInspector);
StructObjectInspector si2 = ObjectInspectorFactory.getStandardStructObjectInspector(structFieldNames,
structFieldObjectInspectors);
ListObjectInspector li2;
li2 = ObjectInspectorFactory.getStandardListObjectInspector(si2);
return li2;
}
#Override
public Object evaluate(DeferredObject[] args) throws HiveException {
ret.clear();
System.out.println("under evaluate");
// String street1 = streetvalue.getPrimitiveJavaObject(args[0].get());
Object oin = args[4].get();
System.out.println("under typecasting");
int batchsize = (Integer) batchsizei.getPrimitiveJavaObject(oin);
System.out.println("batchsize");
Object oin1 = args[0].get();
String street1 = (String) streeti.getPrimitiveJavaObject(oin1);
Object oin2 = args[1].get();
String zip1 = (String) zipi.getPrimitiveJavaObject(oin2);
Object oin3 = args[2].get();
String city1 = (String) cityi.getPrimitiveJavaObject(oin3);
Object oin4 = args[3].get();
String state1 = (String) statei.getPrimitiveJavaObject(oin4);
logger.info("address passed, street=" + street1 + ",zip=" + zip1 + ",city=" + city1 + ",state=" + state1);
counter++;
try {
System.out.println("under try");
Lookup lookup = new Lookup();
lookup.setStreet(street1);
lookup.setCity(city1);
lookup.setState(state1);
lookup.setZipCode(zip1);
lookup.setMaxCandidates(1);
batch.add(lookup);
} catch (BatchFullException ex) {
logger.error(ex.getMessage(), ex);
} catch (Exception e) {
logger.error(e.getMessage(), e);
}
/* batch.add(lookup); */
if (counter == batchsize) {
System.out.println("under if");
try {
logger.info("batch input street " + batch.get(0).getStreet());
try {
client.send(batch);
} catch (Exception e) {
logger.error(e.getMessage(), e);
logger.warn("skipping current batch, continuing with the next batch");
batch.clear();
counter = 0;
return null;
}
Vector<Lookup> lookups = batch.getAllLookups();
for (int i = 0; i < batch.size(); i++) {
// ListObjectInspector candidates;
ArrayList<Candidate> candidates = lookups.get(i).getResult();
if (candidates.isEmpty()) {
logger.warn("Address " + i + " is invalid.\n");
continue;
}
logger.info("Address " + i + " is valid. (There is at least one candidate)");
for (Candidate candidate : candidates) {
final Components components = candidate.getComponents();
final Metadata metadata = candidate.getMetadata();
logger.info("\nCandidate " + candidate.getCandidateIndex() + ":");
logger.info("Delivery line 1: " + candidate.getDeliveryLine1());
logger.info("Last line: " + candidate.getLastLine());
logger.info("ZIP Code: " + components.getZipCode() + "-" + components.getPlus4Code());
logger.info("County: " + metadata.getCountyName());
logger.info("Latitude: " + metadata.getLatitude());
logger.info("Longitude: " + metadata.getLongitude());
}
Object[] e;
e = new Object[4];
e[0] = new Text(candidates.get(i).getComponents().getStreetName());
e[1] = new Text(candidates.get(i).getComponents().getCityName());
e[2] = new Text(candidates.get(i).getComponents().getZipCode());
e[3] = new Text(candidates.get(i).getComponents().getState());
ret.add(e);
}
counter = 0;
batch.clear();
} catch (Exception e) {
logger.error(e.getMessage(), e);
}
return ret;
} else {
return null;
}
}
}