I have written a piece of code in Java that connect to S3 and read list of files in the bucket. I tested the code and is working properly (in both local and EC2). However, when I run the code in EMR I face this error:
"java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
I get suspected it can be because of ClientConfiguration and I increased all of the Timeout, RequestTimeout and SocketTimeOut to 5 minutes, but still not change and again I get the same error. What can be reason of this error? The content of bucket is big, but in client mode in EC2, I can read all of them in 3 minutes! something should be wrong!
and this is my code:
try {
ListVersionsRequest request = new ListVersionsRequest().withBucketName(bucketname).withPrefix(prefix)
.withMaxResults(999999);
VersionListing versionListing;
do {
versionListing = s3.listVersions(request);
for (S3VersionSummary versionSummary : versionListing.getVersionSummaries()) {
if (conditions.length != 0)
if (contain_str(versionSummary.getKey(), conditions))
collect.add("s3://" + bucketname + "/" + versionSummary.getKey());
else
continue;
else
collect.add("s3://" + bucketname + "/" + versionSummary.getKey());
}
request.setKeyMarker(versionListing.getNextKeyMarker());
request.setVersionIdMarker(versionListing.getNextVersionIdMarker());
} while (versionListing.isTruncated()); //check remaining pagination
}
catch (AmazonServiceException ase) {
System.out.println("Caught an AmazonServiceException, " +
"which means your request made it " +
"to Amazon S3, but was rejected with an error response " +
"for some reason.");
System.out.println("Error Message: " + ase.getMessage());
System.out.println("HTTP Status Code: " + ase.getStatusCode());
System.out.println("AWS Error Code: " + ase.getErrorCode());
System.out.println("Error Type: " + ase.getErrorType());
System.out.println("Request ID: " + ase.getRequestId());
} catch (AmazonClientException ace) {
System.out.println("Caught an AmazonClientException, " +
"which means the client encountered " +
"an internal error while trying to communicate" +
" with S3, " +
"such as not being able to access the network.");
System.out.println("Error Message: " + ace.getMessage());
}
finally {
Collections.shuffle(collect); //randomizing the file orders
if (collect.size() * percentage < 1) // if sample number is less than 1 take the whole data
percentage = 1;
collect2 = collect.subList(0, (int) (collect.size() * percentage)); //take the samples
return String.join(",", collect2);
}
And the error:
18/01/04 10:36:31 ERROR ApplicationMaster: Uncaught exception:
java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:201)
at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:401)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:254)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:764)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:67)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:66)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:66)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:762)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Thanks
Related
I'm implementing a Redis client (standalone mode) on Quarkus.
I get this intermittent error when I query if a key already exists in Redis :
Caused by: MOVED 3271 xxxx-memorydb-redis-cluster-0001-002.XXXX-memorydb-redis-cluster.xxxx.memorydb.us-east-1.amazonaws.com:6379
java.util.concurrent.CompletionException: MOVED 3814 xxxx-memorydb-redis-cluster-XXXX-002.xxx-memorydb-redis-cluster.xxxx.memorydb.us-east-1.amazonaws.com:6379
at io.smallrye.mutiny.operators.uni.UniBlockingAwait.await(UniBlockingAwait.java:79)
at io.smallrye.mutiny.groups.UniAwait.atMost(UniAwait.java:65)
at io.quarkus.redis.client.runtime.RedisClientImpl.await(RedisClientImpl.java:1046)
at io.quarkus.redis.client.runtime.RedisClientImpl.exists(RedisClientImpl.java:172)
at io.quarkus.redis.client.RedisClient_761b9a6e5f634178e3291b09c1921f229025da0c_Synthetic_ClientProxy.exists(Unknown Source)
PART OF CODE WHERE THE ERROR OCCURS:
public ObjectNode generateMobileDeviceToken(long idClient) {
Integer mobileDeviceToken = null;
while (mobileDeviceToken == null) {
mobileDeviceToken = ThreadLocalRandom.current().nextInt(100000, 1000000);
// ************************************
// ***the error occurs on this line ***
// Verify if token already exists
int keysFound = this.redisClient.exists(List.of(REDIS_ENV + "." + MOBILE_DEVICE_TOKEN_GROUP_NAME + "." + mobileDeviceToken)).toInteger();
// ***the error occurs on this line ***
// ************************************
if (keysFound != 0) {
mobileDeviceToken = null;
}
}
this.redisClient.setex(REDIS_ENV + "." + MOBILE_DEVICE_TOKEN_GROUP_NAME + "." + mobileDeviceToken, TOKEN_TTL_S.toString(), String.valueOf(idClient));
ObjectNode mobileDeviceTokenJSON = OBJECT_MAPPER.createObjectNode();
mobileDeviceTokenJSON.put("mobileDeviceToken", mobileDeviceToken);
mobileDeviceTokenJSON.put("mobileDeviceTokenExpiration", LocalDateTime.now()
.plus(TOKEN_TTL_S, ChronoUnit.SECONDS).toEpochSecond(ZoneOffset.UTC));
return mobileDeviceTokenJSON;
}
APPLICATION PROPERTIES
# Redis
quarkus.redis.hosts=rediss://<MY-REDIS-ADDRESS>.amazonaws.com:6379
quarkus.redis.ssl.enabled=true
quarkus.redis.max-pool-size=25
quarkus.redis.max-pool-waiting=100
quarkus.redis.client-type=standalone
quarkus.redis.devservices.enabled=false
I am using Docx4j in an internship project and when I try to generate a ToC with page numbers it does not work.
Executing the code with
if (template.isGenerateToc()) {
Toc.setTocHeadingText("Inhaltsverzeichnis");
TocGenerator tocGenerator = new TocGenerator(wordPackage);
tocGenerator.generateToc(2, " TOC \\o \"1-3\" \\h \\z \\u ", true);
}
Works perfectly fine, but results in no page numbers in the .docx (when exporting to PDF via the Gotenberg API (https://gotenberg.dev), the page numbers are existant and correct).
When I try to pass "false" for the third parameter, the application crashes with the following exception/stacktrace:
org.docx4j.toc.TocException: Exception exporting package
at org.docx4j.toc.TocGenerator.getPageNumbersMapViaFOP(TocGenerator.java:871)
at org.docx4j.toc.TocGenerator.getPageNumbersMap(TocGenerator.java:694)
at org.docx4j.toc.TocGenerator.populateToc(TocGenerator.java:441)
at org.docx4j.toc.TocGenerator.generateToc(TocGenerator.java:306)
at org.docx4j.toc.TocGenerator.generateToc(TocGenerator.java:234)
at org.docx4j.toc.TocGenerator.generateToc(TocGenerator.java:194)
If usefull: This is my TocHeading:
private static String XML_TOCHeading_BasedOn_Nothing = "<w:style w:styleId=\"TOCHeading\" w:type=\"paragraph\" xmlns:w=\"http://schemas.openxmlformats.org/wordprocessingml/2006/main\">"
+ "<w:name w:val=\"TOC Heading\"/>"
// + "<w:basedOn w:val=\"Heading1\"/>" // would be ok if provided already present, since
+ "<w:next w:val=\"Normal\"/>"
+ "<w:uiPriority w:val=\"39\"/>"
+ "<w:semiHidden/>"
+ "<w:unhideWhenUsed/>"
+ "<w:qFormat/>"
+ "<w:pPr>"
+ "<w:keepNext/>"
+ "<w:keepLines/>"
+ "<w:spacing w:after=\"150\" w:before=\"480\"/>"
+ "<w:outlineLvl w:val=\"9\"/>"
+ "</w:pPr>"
+ "<w:rPr>"
+ "<w:rFonts w:ascii=\"Arial\" w:hAnsi=\"Arial\" w:cs=\"Arial\"/>"
+ "<w:b/>"
+ "<w:bCs/>"
+ "<w:sz w:val=\"36\"/>"
+ "<w:szCs w:val=\"36\"/>"
+ "</w:rPr>"
+ "</w:style>";
Any ideas on how to fix this exception and generate page numbers in my toc?
EDIT:
I updated the dependencies and now I am still getting the TocException but now with another cause:
Caused by: org.docx4j.openpackaging.exceptions.Docx4JException: Exception exporting package
at org.docx4j.convert.out.common.AbstractExporter.export(AbstractExporter.java:106)
at org.docx4j.Docx4J.toFO(Docx4J.java:734)
at org.docx4j.toc.TocGenerator.getPageNumbersMapViaFOP(TocGenerator.java:843)
... 97 more
Caused by: java.lang.NullPointerException
at org.docx4j.convert.out.common.preprocess.Containerization.borderChanged(Containerization.java:390)
at org.docx4j.convert.out.common.preprocess.Containerization.bordersChanged(Containerization.java:377)
at org.docx4j.convert.out.common.preprocess.Containerization.groupBodyContent(Containerization.java:204)
at org.docx4j.convert.out.common.preprocess.Containerization.groupAdjacentBorders(Containerization.java:129)
at org.docx4j.convert.out.common.preprocess.Containerization.process(Containerization.java:120)
at org.docx4j.convert.out.common.Preprocess.process(Preprocess.java:165)
at org.docx4j.convert.out.common.AbstractWmlExporter.preprocess(AbstractWmlExporter.java:51)
at org.docx4j.convert.out.common.AbstractWmlExporter.preprocess(AbstractWmlExporter.java:32)
at org.docx4j.convert.out.common.AbstractExporter.export(AbstractExporter.java:63)
Not sure which version of docx4j you are using, but generally, to update page numbers, you need FOP on your classpath.
We broke this in 8.3.3; it is fixed at https://github.com/plutext/docx4j/commit/50cfa5cabe08ab133f28c54b59f73d082d3e80a4#diff-29cfd63b63bfe585aa3468870ca54f1afb6e2246342c396155801b0d0aa9ae43L763
So you could try 8.3.2 with FOP on your classpath.
Alternatively, you could try the approach described at https://www.docx4java.org/blog/2020/03/documents4j-for-toc-update/
See further:
documents4j-local: https://github.com/plutext/docx4j/blob/master/docx4j-samples-documents4j-local/src/main/java/org/docx4j/samples/documents4j/local/TocOperations.java
documents4j-remote: https://github.com/plutext/docx4j/tree/master/docx4j-samples-documents4j-remote
As per documentation, max.poll.interval.ms is the time before which processor thread must complete processing. Otherwise, rebalancing will trigger.
Time interval mentioned in consumer.poll() has a separate purpose i.e. waiting for data before time-out is triggered.
Now, I am having a weird problem. In my code, consumer.poll() is set with value of 5000ms. And max.poll.interval.ms has a value of 20000. As per above details, my processor thread should wait for 5 seconds during polling, but it is waiting for 20 seconds. Similary, when max.poll.interval.ms value is not set explicitly, I am getting delays of 5 minutes i.e. code gets blocked during poll for 5 minutes (verified from debugger and logging).
Code:
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(kafkaProps);
consumer.subscribe(Arrays.asList(topic));
LGR.info(consumer.partitionsFor(topic));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Integer.parseInt(kafkaProps.getProperty("consume.poll.timeout")));
if(records.count() == 0) {
LGR.info("No records found for kafka topic: " + kafkaProps.getProperty("topic") + ", and bootstrap servers: " + kafkaProps.getProperty("bootstrap.servers"));
break;
}
for (ConsumerRecord<String, String> record : records) {
LGR.info("Topic: " + record.topic() + ", Timestamp: " + record.timestamp() + ", Partition: " + record.partition() + ", Offset: " + record.offset() + ", Key: " + record.key() + ", Value: " + record.value());
}
}
Configurations:
topic=****
bootstrap.servers=****
group.id=orderstatus_61
enable.auto.commit=false
auto.commit.interval.ms=10000
auto.offset.reset=earliest
session.timeout.ms=30000
client.id=orderstatus_61
max.poll.interval.ms=20000
max.pool.records=500
consume.poll.timeout=5000
ssl=no
value.deserializer=org.apache.kafka.common.serialization.StringDeserializer
key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
Kafka Version: kafka-clients-1.0.0
I've been trying to add a reaction roles command to my discord bot. But strangely it isn't working properly.
But the problem is that it worked six months ago. I haven't changed anything in my code.
The code stops here: Emote emote = emotes.get(0);
My class:
package de.nameddaniel.bot.commands;
import java.util.List;
import de.nameddaniel.bot.main.LiteSQL;
import de.nameddaniel.bot.main.Utils;
import de.nameddaniel.bot.types.ServerCommand;
import net.dv8tion.jda.api.Permission;
import net.dv8tion.jda.api.entities.Emote;
import net.dv8tion.jda.api.entities.Member;
import net.dv8tion.jda.api.entities.Message;
import net.dv8tion.jda.api.entities.Role;
import net.dv8tion.jda.api.entities.TextChannel;
public class ReactionRolesCommand implements ServerCommand {
#Override
public void performCommand(Member m, TextChannel channel, Message msg) {
//!rr channel messageid :ok: ...
String[] args = msg.getContentDisplay().split(" ");
if(m.hasPermission(Permission.MANAGE_ROLES)) {
if(args.length >= 5) {
List<TextChannel> channels = msg.getMentionedChannels();
List<Role> roles = msg.getMentionedRoles();
List<Emote> emotes = msg.getEmotes();
System.out.println("3");
TextChannel tc = msg.getMentionedChannels().get(0);
String messageIDString = args[1];
Role role = roles.get(0);
System.out.println("1");
try {
long messageID = Long.parseLong(messageIDString);
Emote emote = emotes.get(0);
tc.addReactionById(messageID, emote).queue();
System.out.println("2");
LiteSQL.onUpdate("INSERT INTO reactroles(guildid, channelid, messageid, emote, roleid) "
+ "VALUES(" + channel.getGuild().getIdLong() + ", " + tc.getIdLong() + ", "
+ messageID + ", '" + emote.getId() + "', " + role.getIdLong() + ")");
Utils.Embed("Deiner Nachricht in " + msg.getMentionedChannels() + " wurde die Reaktion " + msg.getEmotes()
+ " hinzugefügt.", channel, "Erfolgreich!");
}
catch (NumberFormatException e) {
e.printStackTrace();
System.out.println("ReactionCommand");
}
}
else
Utils.Embed("Bitte benutze: !rr [MessageID] #channel [Emote] #Rolle", channel, "Falscher Syntax");
System.out.println(args.length);
}
else
msg.delete().queue();
}
}
The error Code:
3
1
[JDA [0 / 1] MainWS-ReadThread] ERROR JDA - One of the EventListeners had an uncaught exception
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(Unknown Source)
at java.util.ArrayList.get(Unknown Source)
at java.util.Collections$UnmodifiableList.get(Unknown Source)
at de.nameddaniel.bot.commands.ReactionRolesCommand.performCommand(ReactionRolesCommand.java:40)
at de.nameddaniel.bot.main.CommandManager.perform(CommandManager.java:43)
at de.nameddaniel.bot.listener.CommandListener.onMessageReceived(CommandListener.java:25)
at net.dv8tion.jda.api.hooks.ListenerAdapter.onEvent(ListenerAdapter.java:430)
at net.dv8tion.jda.api.hooks.InterfacedEventManager.handle(InterfacedEventManager.java:96)
at net.dv8tion.jda.internal.hooks.EventManagerProxy.handleInternally(EventManagerProxy.java:82)
at net.dv8tion.jda.internal.hooks.EventManagerProxy.handle(EventManagerProxy.java:69)
at net.dv8tion.jda.internal.JDAImpl.handleEvent(JDAImpl.java:147)
at net.dv8tion.jda.internal.handle.MessageCreateHandler.handleInternally(MessageCreateHandler.java:122)
at net.dv8tion.jda.internal.handle.SocketHandler.handle(SocketHandler.java:36)
at net.dv8tion.jda.internal.requests.WebSocketClient.onDispatch(WebSocketClient.java:948)
at net.dv8tion.jda.internal.requests.WebSocketClient.onEvent(WebSocketClient.java:835)
at net.dv8tion.jda.internal.requests.WebSocketClient.handleEvent(WebSocketClient.java:813)
at net.dv8tion.jda.internal.requests.WebSocketClient.onBinaryMessage(WebSocketClient.java:986)
at com.neovisionaries.ws.client.ListenerManager.callOnBinaryMessage(ListenerManager.java:385)
at com.neovisionaries.ws.client.ReadingThread.callOnBinaryMessage(ReadingThread.java:276)
at com.neovisionaries.ws.client.ReadingThread.handleBinaryFrame(ReadingThread.java:996)
at com.neovisionaries.ws.client.ReadingThread.handleFrame(ReadingThread.java:755)
at com.neovisionaries.ws.client.ReadingThread.main(ReadingThread.java:108)
at com.neovisionaries.ws.client.ReadingThread.runMain(ReadingThread.java:64)
at com.neovisionaries.ws.client.WebSocketThread.run(WebSocketThread.java:45)
What I could figure out is, that my List emotes is empty. But I have no idea why..
Most of the code is in German. So ask me if you need a translation.
If there is anything missing also please tell me.
Hope you can help me, Daniel :D
The mistake was, that I confounded Emote with Emoji. So it has to be an Emoji instead of an emote.
package modifiedlines;
import edu.nyu.cs.javagit.api.DotGit;
import edu.nyu.cs.javagit.api.JavaGitConfiguration;
import edu.nyu.cs.javagit.api.JavaGitException;
import edu.nyu.cs.javagit.api.WorkingTree;
import edu.nyu.cs.javagit.api.commands.GitLogResponse;
import edu.nyu.cs.javagit.api.commands.GitStatus;
import edu.nyu.cs.javagit.api.commands.GitStatusOptions;
import edu.nyu.cs.javagit.api.commands.GitStatusResponse;
import java.io.File;
import java.io.IOException;
import java.util.List;
/**
*
* #
author aryan000
*/
public class UseGit {
private static File repositoryDirectory;
private static DotGit dotGit;
public static void main(String s[]) throws JavaGitException, IOException
{
File f = new File("C:\\Program Files\\Git\\bin\\git.exe");
if(!f.exists())
{
System.out.println("does not exist");
}
else
System.out.println("exists at " + f.getPath());
JavaGitConfiguration.setGitPath("C:\\Program Files\\Git\\bin");
System.out.println("Git version : " + JavaGitConfiguration.getGitVersion());
// repositoryDirectory = new File("/home/aryan000/Desktop/retrofit");
repositoryDirectory = new File("/home/aryan000/Desktop/changeprone/changeprone");
System.out.println("Git Repository Location : " + repositoryDirectory.getAbsolutePath());
//get the instance of the dotGit Object
dotGit = DotGit.getInstance(repositoryDirectory);
// System.out.println("checking what i have got ");
// GitLogResponse.CommitFile com ;
// com = (GitLogResponse.CommitFile) dotGit.getLog();
//
// System.out.println(com);
WorkingTree wt = dotGit.getWorkingTree();
File workingTreePath = wt.getPath();
GitStatus gitStatus = new GitStatus();
GitStatusResponse status = gitStatus.status(workingTreePath);
System.out.println("status is : " + status);
File anotherFileDir = new File("/home/aryan000/Desktop/retrofit/test.txt");
GitStatusOptions options = new GitStatusOptions();
options.setOptOnly(true);
status = gitStatus.status(workingTreePath);
System.out.println("status is : " + status);
System.out.println("----- Print log to see our commit -----");
for (GitLogResponse.Commit c : dotGit.getLog()) {
System.out.println("commit id is : " + c.getSha());
System.out.println(" commit message is : " + c.getMessage());
System.out.println(" author of the commit is : " + c.getAuthor());
System.out.println(" date modified is : " + c.getDateString());
System.out.println(" number of files changed is : " + c.getFiles());
List<GitLogResponse.CommitFile> store = c.getFiles();
if(store!=null)
System.out.println("the number of files changed is : " + store.size());
System.out.println("list of files changed is : " + c.getFilesChanged());
System.out.println("total number of additions : " + c.getLinesDeleted());
System.out.println("total number of merger : " + c.getMergeDetails());
}
// for(GitLogResponse.CommitFile c : dotGit.getLog())
}
}
Output is shown as :
Exception in thread "main" edu.nyu.cs.javagit.api.JavaGitException: 100002: Invalid path to git specified. { path=[C:\Program Files\Git\bin] }
at edu.nyu.cs.javagit.api.JavaGitConfiguration.setGitPath(JavaGitConfiguration.java:220)
at edu.nyu.cs.javagit.api.JavaGitConfiguration.setGitPath(JavaGitConfiguration.java:247)
at modifiedlines.UseGit.main(UseGit.java:40)
Caused by: edu.nyu.cs.javagit.api.JavaGitException: 100002: Invalid path to git specified.
at edu.nyu.cs.javagit.api.JavaGitConfiguration.determineGitVersion(JavaGitConfiguration.java:81)
at edu.nyu.cs.javagit.api.JavaGitConfiguration.setGitPath(JavaGitConfiguration.java:217)
... 2 more
Java Result: 1
My query is how to find git logs and the files changed due to a particular commit using a Java Program.
Can any 1 help me in this.
Please See : This code is working fine in Ubuntu i.e. no Path problem still I am unable to get the files changed during a commit. It is give me a List as a null.
This bug was opened a long time ago and just fixed it. Please give it a try , use master branch please, and let me know if it works for you.