Develop Reference

java.nio.ByteBuffer wrap method partly working with sbt run

I have an issue where I read a bytestream from a big file ~ (100MB) and after some integers I get the value 0 (but only with sbt run ). When I hit the play button on IntelliJ I get the value I expected > 0.
My guess was that the environment is somehow different. But I could not spot the difference.
// DemoApp.scala
import java.nio.{ByteBuffer, ByteOrder}
object DemoApp extends App {
val inputStream = getClass.getResourceAsStream("/HandRanks.dat")
val handRanks = new Array[Byte](inputStream.available)
inputStream.read(handRanks)
inputStream.close()
def evalCard(value: Int) = {
val offset = value * 4
println("value: " + value)
println("offset: " + offset)
ByteBuffer.wrap(handRanks, offset, handRanks.length - offset).order(ByteOrder.LITTLE_ENDIAN).getInt
}
val cards: List[Int] = List(51, 45, 14, 2, 12, 28, 46)
def eval(cards: List[Int]): Unit = {
var p = 53
cards.foreach(card => {
println("p = " + evalCard(p))
p = evalCard(p + card)
})
println("result p: " + p);
}
eval(cards)
}
The HandRanks.dat can be found here: (I put it inside a directory called resources)
https://github.com/Robert-Nickel/scala-texas-holdem/blob/master/src/main/resources/HandRanks.dat
build.sbt is:
name := "LoadInts"
version := "0.1"
scalaVersion := "2.13.4"
On my windows machine I use sbt 1.4.6 with Oracle Java 11
You will see that the evalCard call will work 4 times but after the fifth time the return value is 0. It should be higher than 0, which it is when using IntelliJ's play button.

You are not reading a whole content. This
val handRanks = new Array[Byte](inputStream.available)
allocates only as much as InputStream buffer and then you read the amount in buffer with
inputStream.read(handRanks)
Depending of defaults you will process different amount but they will never be 100MB of data. For that you would have to read data into some structure in the loop (bad idea) or process it in chunks (with iterators, stream, etc).
import scala.util.Using
// Using will close the resource whether error happens or not
Using(getClass.getResourceAsStream("/HandRanks.dat")) { inputStream =>
def readChunk(): Option[Array[Byte]] = {
// can be done better, but that's not the point here
val buffer = new Array[Byte](inputStream.available)
val bytesRead = inputStream.read(buffer)
if (bytesRead >= 0) Some(buffer.take(bytesRead))
else None
}
#tailrec def process(): Unit = {
readChunk() match {
case Some(chunk) =>
// do something
process()
case None =>
// nothing to do - EOF reached
}
}
process()
}

Is there an alternative for GroupReduceFunction running apache-flink java in parallel?

The code below is running locally but not on the cluster. It hangs on GroupReduceFunction and do not terminates even after hours (it takes for large data ~ 9 minutes to compute locally). The last message in the log:
GroupReduce (GroupReduce at main(MyClass.java:80)) (1/1) (...) switched from DEPLOYING to RUNNING.
The code fragment:
DataSet<MyData1> myData1 = env.createInput(new UserDefinedFunctions.MyData1Set());
DataSet<MyData2> myData2 = DataSetUtils.sampleWithSize(myData1, false, 8, Long.MAX_VALUE)
.reduceGroup(new GroupReduceFunction<MyData1, MyData2>() {
#Override
public void reduce(Iterable<MyData1> itrbl, Collector<MyData2> clctr) throws Exception {
int id = 0;
for (MyData1 myData1 : itrbl) {
clctr.collect(new MyData2(id++, myData1));
}
}
});
Any ideas how I could run this segment in parallel? Thanks in advance!

Can produce to Kafka but cannot consume

I'm using the Kafka JDK client ver 0.10.2.1 . I am able to produce simple messages to Kafka for a "heartbeat" test, but I cannot consume a message from that same topic using the sdk. I am able to consume that message when I go into the Kafka CLI, so I have confirmed the message is there. Here's the function I'm using to consume from my Kafka server, with the props - I pass the message I produced to the topic only after I have indeed confirmed the produce() was succesful, I can post that function later if requested:
private def consumeFromKafka(topic: String, expectedMessage: String): Boolean = {
val props: Properties = initProps("consumer")
val consumer = new KafkaConsumer[String, String](props)
consumer.subscribe(List(topic).asJava)
var readExpectedRecord = false
try {
val records = {
val firstPollRecs = consumer.poll(MAX_POLLTIME_MS)
// increase timeout and try again if nothing comes back the first time in case system is busy
if (firstPollRecs.count() == 0) firstPollRecs else {
logger.info("KafkaHeartBeat: First poll had 0 records- trying again - doubling timeout to "
+ (MAX_POLLTIME_MS * 2)/1000 + " sec.")
consumer.poll(MAX_POLLTIME_MS * 2)
}
}
records.forEach(rec => {
if (rec.value() == expectedMessage) readExpectedRecord = true
})
} catch {
case e: Throwable => //log error
} finally {
consumer.close()
}
readExpectedRecord
}
private def initProps(propsType: String): Properties = {
val prop = new Properties()
prop.put("bootstrap.servers", kafkaServer + ":" + kafkaPort)
propsType match {
case "producer" => {
prop.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer")
prop.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer")
prop.put("acks", "1")
prop.put("producer.type", "sync")
prop.put("retries", "3")
prop.put("linger.ms", "5")
}
case "consumer" => {
prop.put("group.id", groupId)
prop.put("enable.auto.commit", "false")
prop.put("auto.commit.interval.ms", "1000")
prop.put("session.timeout.ms", "30000")
prop.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
prop.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
prop.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
// poll just once, should only be one record for the heartbeat
prop.put("max.poll.records", "1")
}
}
prop
}
Now when I run the code, here's what it outputs in the console:
13:04:21 - Discovered coordinator serverName:9092 (id: 2147483647
rack: null) for group 0b8947e1-eb68-4af3-ac7b-be3f7c02e76e. 13:04:23
INFO o.a.k.c.c.i.ConsumerCoordinator - Revoking previously assigned
partitions [] for group 0b8947e1-eb68-4af3-ac7b-be3f7c02e76e 13:04:24
INFO o.a.k.c.c.i.AbstractCoordinator - (Re-)joining group
0b8947e1-eb68-4af3-ac7b-be3f7c02e76e 13:04:25 INFO
o.a.k.c.c.i.AbstractCoordinator - Successfully joined group
0b8947e1-eb68-4af3-ac7b-be3f7c02e76e with generation 1 13:04:26 INFO
o.a.k.c.c.i.ConsumerCoordinator - Setting newly assigned partitions
[HeartBeat_Topic.Service_5.2018-08-03.13_04_10.377-0] for group
0b8947e1-eb68-4af3-ac7b-be3f7c02e76e 13:04:27 INFO
c.p.p.l.util.KafkaHeartBeatUtil - KafkaHeartBeat: First poll had 0
records- trying again - doubling timeout to 60 sec.
And then nothing else, no errors thrown -so no records are polled. Does anyone have any idea what's preventing the 'consume' from happening? The subscriber seems to be successful, as I'm able to successfully call the listTopics and list partions no problem.

Your code has a bug. It seems your line:
if (firstPollRecs.count() == 0)
Should say this instead
if (firstPollRecs.count() > 0)
Otherwise, you're passing in an empty firstPollRecs, and then iterating over that, which obviously returns nothing.

setText not working in while loop

My app gets traffic updates from an API (this works) and returns a JSON array, which i'm then taking each element of in a while loop (JSONobject) and attempting to update a TextView with each result every 5 seconds.
However, my script is waiting 15 seconds and then updating to the last value. I've done some research and it says to use asynctask, which I have done, but it has not made a difference.
I've added System.out.println(thestring_to_update_to), and this is working as I would like my app to do (changing every 5 seconds).
The following is in a try/catch block :
JSONArray TrafficInformation = new JSONArray(response);
int TrafficEvents = TrafficInformation.length();
int TrafficEvent = 0;
JSONObject CurrentEvent = new JSONObject();
do{
CurrentEvent = new JSONObject(TrafficInformation.getString(TrafficEvent));
TextView affected_route = (TextView)findViewById(R.id.disrupted_route);
try {
Object[] passTo = new Object[1];
passTo[0] = CurrentEvent.getString("9");
System.out.println(passTo[0]);
new tasker().doInBackground(passTo);
TrafficEvent++;
Thread.sleep(5000);
} catch (Exception e){
Toast.makeText(LiftShare.this, "There was an error with getting traffic info.", Toast.LENGTH_LONG).show();
}
} while (TrafficEvent < TrafficEvents);
I also have this public class
public class tasker extends AsyncTask {
#Override
protected Object[] doInBackground(Object[] Objects) {
TextView affected_route = (TextView)findViewById(R.id.disrupted_route);
affected_route.setText(Objects[0].toString());
return null;
};
}
this is the JSONArray that goes in to the code (It is formatted correctly)
Array
(
[0] => {"1":"Congestion","2":"Minor Disruption - up to 15 minutes delay","3":"Location : The M3 eastbound exit slip at junction J9 . \nReason : Congestion. \nStatus : Currently Active. \nReturn To Normal : Normal traffic conditions are expected between 11:30 and 11:45 on 25 January 2018. \nDelay : There are currently delays of 10 minutes against expected traffic. \n","7":"M3 J9 eastbound exit | Eastbound | Congestion","9":"M3","10":"South East","11":"Hampshire","14":"2018-01-25T11:22:38+00:00"}
[1] => {"1":"Overturned Vehicle","2":"Severe Disruption - in excess of 3 hours delay or road closure","3":"Location : The M3 westbound between junctions J8 and J9 . \nReason : Clearing the scene of an overturned vehicle. \nStatus : Currently Active. \nTime To Clear : The event is expected to clear between 14:45 and 15:00 on 25 January 2018. \nReturn To Normal : Normal traffic conditions are expected between 14:45 and 15:00 on 25 January 2018. \nLanes Closed : All lanes are closed. \nPrevious Reason : Following an earlier accident. \n","7":"M3 westbound between J8 and J9 | Westbound | Overturned Vehicle","9":"M3","10":"South East","11":"Hampshire","14":"2018-01-25T06:51:12+00:00"}
[2] => {"1":"Congestion","2":"Moderate Disruption - between 15 minutes and 3 hours delay","3":"Location : The A34 southbound between the A272 and the junction with the M3 . \nReason : Congestion. \nStatus : Currently Active. \nReturn To Normal : Normal traffic conditions are expected between 12:45 and 13:00 on 25 January 2018. \nDelay : There are currently delays of 40 minutes against expected traffic. \n","7":"A34 southbound within the A272 junction | Southbound | Congestion","9":"A34","10":"South East","11":"Hampshire","14":"2018-01-25T07:48:23+00:00"}
)
How can I get the textview to update to the new value every 5 seconds?

You have to use
new tasker().execute(passTo);
to start asynctask as a thread otherwise, with current implementation, it will just act as a normal method call
Note: you cannot update UI from background thread i.e. inside doInBackground, instead override onPostExecute which runs on UI thread
#Override
protected Object[] doInBackground(Object[] Objects) {
TextView affected_route = (TextView)findViewById(R.id.disrupted_route);
//affected_route.setText(Objects[0].toString()); crash, instead do this in onPostExecute
return null;
};
Update : you can use postDelayed with delay to update UI after some interval
int i = 0;
affected_route.postDelayed(new Runnable() {
public void run() {
textView.setText(yourText);
}
},i+=5000);

AsyncTask seems like a overkill for your requirement as you are not really doing any work in the background. You could schedule the text to be updated after a time period using a Handler (from android.os) like this:
Handler handler = new Handler(Looper.getMainLooper());
Runnable textUpdater = new Runnable() {
#Override
public void run() {
// this needs to execute in the UI thread
affected_route.setText(lastUpdate);
}
};
String lastUpdate = "Store your last update here";
void updateText(){
handler.postDelayed(textUpdater, 5000);
}

localized duration format for french

What is the Python analog of Time4J's code example:
// duration in seconds normalized to hours, minutes and seconds
Duration<?> dur = Duration.of(337540, ClockUnit.SECONDS).with(Duration.STD_CLOCK_PERIOD);
// custom duration format => hh:mm:ss
String s1 = Duration.Formatter.ofPattern("hh:mm:ss").format(dur);
System.out.println(s1); // output: 93:45:40
// localized duration format for french
String s2 = PrettyTime.of(Locale.FRANCE).print(dur, TextWidth.WIDE);
System.out.println(s2); // output: 93 heures, 45 minutes et 40 secondes
It is easy to get 93:45:40:
#!/usr/bin/env python3
from datetime import timedelta
dur = timedelta(seconds=337540)
print(dur) # -> 3 days, 21:45:40
fields = {}
fields['hours'], seconds = divmod(dur // timedelta(seconds=1), 3600)
fields['minutes'], fields['seconds'] = divmod(seconds, 60)
print("%(hours)02d:%(minutes)02d:%(seconds)02d" % fields) # -> 93:45:40
but how do I emulate PrettyTime.of(Locale.FRANCE).print(dur, TextWidth.WIDE) Java code in Python (without hardcoding the units)?

babel module allows to get close to desired output:
from babel.dates import format_timedelta # $ pip install babel
print(", ".join(format_timedelta(timedelta(**{unit: fields[unit]}),
granularity=unit.rstrip('s'),
threshold=fields[unit] + 1,
locale='fr')
for unit in "hours minutes seconds".split()))
# -> 93 heures, 45 minutes, 40 secondes
It handles locale and plural forms automatically e.g., for dur = timedelta(seconds=1) it produces:
0 heure, 0 minute, 1 seconde
Perhaps a better solution would be to translate the format string manually using standard tools such as gettext.

If you're using Kotlin, I just came across a similar problem with the Kotlin Duration type with localized formatting and because I couldn't find a good solution, I wrote one myself. It is based on APIs provided starting in Android 9 (for localized units), but with a fallback to English units for lower Android versions so it can be used with lower targeting apps.
Here's how it looks like on the usage side (see Kotlin Duration type to understand 1st line):
val duration = 5.days.plus(3.hours).plus(2.minutes).plus(214.milliseconds)
DurationFormat().format(duration) // "5day 3hour 2min"
DurationFormat(Locale.GERMANY).format(duration) // "5T 3Std. 2Min."
DurationFormat(Locale.forLanguageTag("ar").format(duration) // "٥يوم ٣ساعة ٢د"
DurationFormat().format(duration, smallestUnit = DurationFormat.Unit.HOUR) // "5day 3hour"
DurationFormat().format(15.minutes) // "15min"
DurationFormat().format(0.hours) // "0sec"
As you can see, you can specify a custom locale to the DurationFormat type. By default it uses Locale.getDefault(). Languages that have different symbols for number than romanic are also supported (via NumberFormat). Also, you can specify a custom smallestUnit, by default it is set to SECOND, so milliseconds will not be shown. Note that any unit with a value of 0 will be ignored and if the entire number is 0, the smallest unit will be used with the value 0.
This is the full DurationFormat type, feel free to copy (also available as a GitHub gist incl. unit tests):
import android.icu.text.MeasureFormat
import android.icu.text.NumberFormat
import android.icu.util.MeasureUnit
import android.os.Build
import java.util.Locale
import kotlin.time.Duration
import kotlin.time.ExperimentalTime
import kotlin.time.days
import kotlin.time.hours
import kotlin.time.milliseconds
import kotlin.time.minutes
import kotlin.time.seconds
#ExperimentalTime
data class DurationFormat(val locale: Locale = Locale.getDefault()) {
enum class Unit {
DAY, HOUR, MINUTE, SECOND, MILLISECOND
}
fun format(duration: kotlin.time.Duration, smallestUnit: Unit = Unit.SECOND): String {
var formattedStringComponents = mutableListOf<String>()
var remainder = duration
for (unit in Unit.values()) {
val component = calculateComponent(unit, remainder)
remainder = when (unit) {
Unit.DAY -> remainder - component.days
Unit.HOUR -> remainder - component.hours
Unit.MINUTE -> remainder - component.minutes
Unit.SECOND -> remainder - component.seconds
Unit.MILLISECOND -> remainder - component.milliseconds
}
val unitDisplayName = unitDisplayName(unit)
if (component > 0) {
val formattedComponent = NumberFormat.getInstance(locale).format(component)
formattedStringComponents.add("$formattedComponent$unitDisplayName")
}
if (unit == smallestUnit) {
val formattedZero = NumberFormat.getInstance(locale).format(0)
if (formattedStringComponents.isEmpty()) formattedStringComponents.add("$formattedZero$unitDisplayName")
break
}
}
return formattedStringComponents.joinToString(" ")
}
private fun calculateComponent(unit: Unit, remainder: Duration) = when (unit) {
Unit.DAY -> remainder.inDays.toLong()
Unit.HOUR -> remainder.inHours.toLong()
Unit.MINUTE -> remainder.inMinutes.toLong()
Unit.SECOND -> remainder.inSeconds.toLong()
Unit.MILLISECOND -> remainder.inMilliseconds.toLong()
}
private fun unitDisplayName(unit: Unit) = if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.P) {
val measureFormat = MeasureFormat.getInstance(locale, MeasureFormat.FormatWidth.NARROW)
when (unit) {
DurationFormat.Unit.DAY -> measureFormat.getUnitDisplayName(MeasureUnit.DAY)
DurationFormat.Unit.HOUR -> measureFormat.getUnitDisplayName(MeasureUnit.HOUR)
DurationFormat.Unit.MINUTE -> measureFormat.getUnitDisplayName(MeasureUnit.MINUTE)
DurationFormat.Unit.SECOND -> measureFormat.getUnitDisplayName(MeasureUnit.SECOND)
DurationFormat.Unit.MILLISECOND -> measureFormat.getUnitDisplayName(MeasureUnit.MILLISECOND)
}
} else {
when (unit) {
Unit.DAY -> "day"
Unit.HOUR -> "hour"
Unit.MINUTE -> "min"
Unit.SECOND -> "sec"
Unit.MILLISECOND -> "msec"
}
}
}

This humanize package may help. It has a french localization, or you can add your own. For python 2.7 and 3.3.

Using pendulum module:
>>> import pendulum
>>> it = pendulum.interval(seconds=337540)
>>> it.in_words(locale='fr_FR')
'3 jours 21 heures 45 minutes 40 secondes'