I wrote this r function that uses the xlsx package to write one or more data frames out to a .xlsx file. When given the same input (3 data frames: 6185 obs of 23 variables, 4 of 17 and 2 of 3) it throws an error most of the time, but not all of the time.
Can anyone tell me how to optimize my code, get the same outcome more elegantly or continue in spite of the error?
Here is the console output:
Running: WriteToFile()
WriteToFile:1
WriteToFile:4
Error in: Example Report
java.lang.OutOfMemoryError: GC overhead limit exceededReturning from: Example Report
and here is the function:
WriteToFile <- function() {
# Write a data frame(s) out to .xlsx file
if(debug > 2) {message("WriteToFile:1")}
# If the file already exists today, then delete it
if(file.exists(paste0(report.name, '(', today(), ')', ".xlsx"))) {
if(debug > 2) {message("WriteToFile:2")}
writeLines(paste0("File '", report.name, '(', today(), ')', ".xlsx", "' already exists and will be replaced."))
flush.console()
file.remove(paste0(report.name, '(', today(), ')', ".xlsx"))
}
# Get tab names that were generated in ProcessOutputs()
tabs <- ls(pattern=paste0("\\.data$"), name=.GlobalEnv)
# If no tabs, send fail-mail
if(length(tabs) == 0) {
if(debug > 2) {message("WriteToFile:3")}
SendPerlMail(fail.mail=TRUE, fail.msg="No data for tabs in WriteToFile()")
return(-1)
}
# Write first tab to first sheet, then write any remaining tabs to additional sheets
else {
if(debug > 2) {message("WriteToFile:4")}
write.xlsx2(x=get(tabs[1]), file=paste0(report.name, ' (', today(), ')', ".xlsx"),
sheetName=substr(x=tabs[1], start=0, stop=nchar(tabs[1])-5), col.names=TRUE, row.names=FALSE, append=FALSE)
if(length(tabs) > 1) {
if(debug > 2) {message("WriteToFile:5")}
for(t in mget(tabs[2:length(tabs)])) {
write.xlsx2(x=t, file=paste0(report.name, ' (', today(), ')', ".xlsx"),
sheetName=substr(x=substitute(t), start=0, stop=nchar(t)-5), col.names=TRUE, row.names=FALSE, append=TRUE)
}
}
}
email.file <<- paste0(getwd(), "/", report.name, ' (', today(), ')', ".xlsx")
return(1)
}
The outside variables referenced in the function are:
options(java.parameters = "-Xms2048m -Xmx4096m", "-XX:-UseGCOverheadLimit") (I set these Java parameters because they worked for someone in another post with a the same GC error)
debug = 3
report.name = "Example Report"
Example Query 1.data = data frame of 6185x23
Example Query 2.data = data frame of 4x17
Example Query 3.data = data frame of 2x3
Sometimes it outputs WriteToFile:5 to the console and sometimes it succeeds without throwing the error. Any help is greatly appreciated--I've been trying to figure out why this doesn't work reliably for a few hours now.
Related
I am trying to read the clob which is basically XML from Oracle DB and populate in AngularJS UI Grid.
I am doing the same with JSON and is working perfectly fine.
JSON response from backend
{"events":{"ORDER_NO":"BBY01-100000709660","ORDER_HEADER_KEY":"2020040811522311790606 ","CREATETS":"2020-04-08 11:52:47","TMPLT_NM":"EOMS_0194 ","EMAIL_XML":"<email CommunicationType=\"Email\" SourceSystem=\"OMS\" TemplatePageZone=\"\" brand=\"BESTBUY\" channel=\"BESTBUY\" emailAddr=\"test.tester#bestbuy.com\" template=\"EOMS_0178_TEST\">"" <name firstName=\"Test\" lastName=\"\" middleInitial=\"\"/>"" <order ATGID=\"ATG28268080246\" IsSuppressRequired=\"Y\" LoggedInFlag=\"Y\" LoyaltyID=\"0160140134\" OrderName=\"MSFTAllAccess\" PartyID=\"123456\" PriorityNumber=\"160140134\" customerPhoneNo=\"6515554321\" hasActivatedDevice=\"N\" orderDate=\"01/28/2020\" orderHeaderKey=\"2020012813423582265743\" orderIdATG=\"BBY01-1MT2010012802\" orderStatusLinkDisplayFlag=\"Y\" orderTotal=\"0.00\" orderTotalMinusCoupons=\"0.00\" partnerID=\"\" partnerOrderNo=\"MAV513281qweq1\" salesSource=\"BBYC\" shippingTotal=\"0.00\" taxTotal=\"0.00\">"" <creditCard cardType=\"\" number=\"\"/>"" <digitalCoupons digitalCouponTotal=\"0.00\"/>"" <lineItems>"" <lineItem CustPromiseDate=\"02/26/2020\" CustPromiseType=\"InHandDate\" availabilityMsg=\"\" beginEstArrivalDate=\"02/24/2020\" conditionVariableOne=\"\" conditionVariableTwo=\"\" description=\"Microsoft Surface Pro 3 12 Intel Core i7 256GB Silver\" endEstArrivalDate=\"02/26/2020\" expectedShipDays=\"\" format=\"\" giftPackaging=\"N\" inHandDate=\"02/26/2020\" itemID=\"\" itemShortDesc=\"Microsoft Surface Pro 3 12 Intel Core i7 256GB Silver\" lineItemProductTotal=\"0.00\" lineItemShippingCost=\"0.00\" merchClass=\"\" modelNo=\"1000186097\" orderLineKey=\"2020021911334791500160\" oversizeFlag=\"\" pickupDate=\"\" preOrder=\"\" primeLine=\"1\" productLine=\"6.403.635\" quantity=\"1\" releaseDate=\"\" reshipReasonCode=\"RESHIP_DAMAGED_ITEM\" shipDate=\"\" shippingMethod=\"\" signatureRequiredFlag=\"N\" sku=\"9248206\" status=\"\" subLine=\"1\" tax=\"0.00\" total=\"0.00\" unitPrice=\"0.00\" unitShippingCost=\"0.00\">"" <shippingAddr city=\"RICHFIELD\" line1=\"1000 W 78TH ST\" line2=\"\" state=\"MN\" zip=\"55423\">"" <name firstName=\"Test\" lastName=\"Tester\" middleInitial=\"\"/>"" </shippingAddr>"" <allowance allowanceAmt=\"0.00\" reason=\"\"/>"" <return date=\"\" lineQty=\"\" lineTotal=\"0.00\" productCredit=\"0.00\" reason=\"\" restockingFee=\"0.00\" shippingCredit=\"0.00\" taxCredit=\"0.00\"/>"" <cancel backOrderExtendedXNumDays=\"\" reason=\"\"/>"" <ros actualDeliveryDate=\"\" pickupDate=\"\"/>"" <store storeName=\"\" storeNum=\"\"/>"" <psp plan=\"\"/>"" <carriers>"" <carrier los=\"\" name=\"\" quantity=\"\" trackingNum=\"\"/>"" </carriers>"" </lineItem>"" </lineItems>"" <makeGood makeGoodFlag=\"N\"/>"" </order>"" <account atgProfileId=\"\" cirisID=\"\" info=\"\" password=\"\"/>"" <comments/>""</email>"}}
Whenever i am trying to read the values it is throwing exception
Unexpected string in JSON at position 372
at JSON.parse (<anonymous>)
Below is the AJAX response code:
$http.get(url).then(function(response) {
if(response.data.events == null || response.data.events == undefined ||
response.data.events == "undefined"){
$("#loader1").hide();
$scope.close = true;
$scope.responseMessage = "";
$scope.gridOptions1.data.length=0;
$scope.errorMessage = "Order not found!!!!";
}else{
console.log("1");
$("#loader1").hide();
var responseNew = JSON.stringify(response.data.events);
$scope.gridOptions1.data = responseNew;
$scope.mySelectedRows = $scope.gridApi.selection.getSelectedRows();
$scope.close = true;
$scope.errorMessage = "";
$scope.responseMessage = "Order details fetched successfully";
}
}, function(response) {
$("#loader1").hide();
$scope.close = true;
$scope.responseMessage = "";
$scope.gridOptions.data.length=0;
$scope.gridOptions1.data.length=0;
});
There's one double quote extra here:
Parse error on line 1:
...\"EOMS_0178_TEST\">"" <name firstName...
-----------------------^
Expecting 'EOF', '}', ':', ',', ']', got 'STRING'
use JSON.parse instead of JSON.stringify. The response you're getting from back-end (the one you mentioned above) is already a stringified JSON, you have to parse it out to read the values.
The above issue waswhile storing the xml in DB. since the new elements had spaces in between. it was considering that as a string and was getting appended with double quotes in JSON.
I am trying to export a lot of data (2 TB, 30kkk rows) from Cassandra to BigQuery. All my infrastructure is on GCP. My Cassandra cluster have 4 nodes (4 vCPUs, 26 GB memory, 2000 GB PD (HDD) each). There is one seed node in the cluster. I need to transform my data before writing to BQ, so I am using Dataflow. Worker type is n1-highmem-2. Workers and Cassandra instances are at the same zone europe-west1-c. My limits for Cassandra:
Part of my pipeline code responsible for reading transform is located here.
Autoscaling
The problem is that when I don't set --numWorkers, the autoscaling set number of workers in such manner (2 workers average):
Load balancing
When I set --numWorkers=15 the rate of reading doesn't increase and only 2 workers communicate with Cassandra (I can tell it from iftop and only these workers have CPU load ~60%).
At the same time Cassandra nodes don't have a lot of load (CPU usage 20-30%). Network and disk usage of the seed node is about 2 times higher than others, but not too high, I think:
And for the not seed node here:
Pipeline launch warnings
I have some warnings when pipeline is launching:
WARNING: Size estimation of the source failed:
org.apache.beam.sdk.io.cassandra.CassandraIO$CassandraSource#7569ea63
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.132.9.101:9042 (com.datastax.driver.core.exceptions.TransportException: [/10.132.9.101:9042] Cannot connect), /10.132.9.102:9042 (com.datastax.driver.core.exceptions.TransportException: [/10.132.9.102:9042] Cannot connect), /10.132.9.103:9042 (com.datastax.driver.core.exceptions.TransportException: [/10.132.9.103:9042] Cannot connect), /10.132.9.104:9042 [only showing errors of first 3 hosts, use getErrors() for more details])
My Cassandra cluster is in GCE local network and it seams that some queries are made from my local machine and cannot reach the cluster (I am launching pipeline with Dataflow Eclipse plugin as described here). These queries are about size estimation of tables. Can I specify size estimation by hand or launch pipline from GCE instance? Or can I ignore these warnings? Does it have effect on rate of read?
I'v tried to launch pipeline from GCE VM. There is no more problem with connectivity. I don't have varchar columns in my tables but I get such warnings (no codec in datastax driver [varchar <-> java.lang.Long]). :
WARNING: Can't estimate the size
com.datastax.driver.core.exceptions.CodecNotFoundException: Codec not found for requested operation: [varchar <-> java.lang.Long]
at com.datastax.driver.core.CodecRegistry.notFound(CodecRegistry.java:741)
at com.datastax.driver.core.CodecRegistry.createCodec(CodecRegistry.java:588)
at com.datastax.driver.core.CodecRegistry.access$500(CodecRegistry.java:137)
at com.datastax.driver.core.CodecRegistry$TypeCodecCacheLoader.load(CodecRegistry.java:246)
at com.datastax.driver.core.CodecRegistry$TypeCodecCacheLoader.load(CodecRegistry.java:232)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3628)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2336)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2295)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2208)
at com.google.common.cache.LocalCache.get(LocalCache.java:4053)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4986)
at com.datastax.driver.core.CodecRegistry.lookupCodec(CodecRegistry.java:522)
at com.datastax.driver.core.CodecRegistry.codecFor(CodecRegistry.java:485)
at com.datastax.driver.core.CodecRegistry.codecFor(CodecRegistry.java:467)
at com.datastax.driver.core.AbstractGettableByIndexData.codecFor(AbstractGettableByIndexData.java:69)
at com.datastax.driver.core.AbstractGettableByIndexData.getLong(AbstractGettableByIndexData.java:152)
at com.datastax.driver.core.AbstractGettableData.getLong(AbstractGettableData.java:26)
at com.datastax.driver.core.AbstractGettableData.getLong(AbstractGettableData.java:95)
at org.apache.beam.sdk.io.cassandra.CassandraServiceImpl.getTokenRanges(CassandraServiceImpl.java:279)
at org.apache.beam.sdk.io.cassandra.CassandraServiceImpl.getEstimatedSizeBytes(CassandraServiceImpl.java:135)
at org.apache.beam.sdk.io.cassandra.CassandraIO$CassandraSource.getEstimatedSizeBytes(CassandraIO.java:308)
at org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.startDynamicSplitThread(BoundedReadEvaluatorFactory.java:166)
at org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:142)
at org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:146)
at org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Pipeline read code
// Read data from Cassandra table
PCollection<Model> pcollection = p.apply(CassandraIO.<Model>read()
.withHosts(Arrays.asList("10.10.10.101", "10.10.10.102", "10.10.10.103", "10.10.10.104")).withPort(9042)
.withKeyspace(keyspaceName).withTable(tableName)
.withEntity(Model.class).withCoder(SerializableCoder.of(Model.class))
.withConsistencyLevel(CASSA_CONSISTENCY_LEVEL));
// Transform pcollection to KV PCollection by rowName
PCollection<KV<Long, Model>> pcollection_by_rowName = pcollection
.apply(ParDo.of(new DoFn<Model, KV<Long, Model>>() {
#ProcessElement
public void processElement(ProcessContext c) {
c.output(KV.of(c.element().rowName, c.element()));
}
}));
Number of splits (Stackdriver log)
W Number of splits is less than 0 (0), fallback to 1
I Number of splits is 1
W Number of splits is less than 0 (0), fallback to 1
I Number of splits is 1
W Number of splits is less than 0 (0), fallback to 1
I Number of splits is 1
What I'v tried
No effect:
set read consistency level to ONE
nodetool setstreamthroughput 1000, nodetool setinterdcstreamthroughput 1000
increase Cassandra read concurrency (in cassandra.yaml): concurrent_reads: 32
setting different number of workers 1-40.
Some effect:
1. I'v set numSplits = 10 as #jkff proposed. Now I can see in logs:
I Murmur3Partitioner detected, splitting
W Can't estimate the size
W Can't estimate the size
W Number of splits is less than 0 (0), fallback to 10
I Number of splits is 10
W Number of splits is less than 0 (0), fallback to 10
I Number of splits is 10
I Splitting source org.apache.beam.sdk.io.cassandra.CassandraIO$CassandraSource#6d83ee93 produced 10 bundles with total serialized response size 20799
I Splitting source org.apache.beam.sdk.io.cassandra.CassandraIO$CassandraSource#25d02f5c produced 10 bundles with total serialized response size 19359
I Splitting source [0, 1) produced 1 bundles with total serialized response size 1091
I Murmur3Partitioner detected, splitting
W Can't estimate the size
I Splitting source [0, 0) produced 0 bundles with total serialized response size 76
W Number of splits is less than 0 (0), fallback to 10
I Number of splits is 10
I Splitting source org.apache.beam.sdk.io.cassandra.CassandraIO$CassandraSource#2661dcf3 produced 10 bundles with total serialized response size 18527
But I'v got another exception:
java.io.IOException: Failed to start reading from source: org.apache.beam.sdk.io.cassandra.Cassandra...
(5d6339652002918d): java.io.IOException: Failed to start reading from source: org.apache.beam.sdk.io.cassandra.CassandraIO$CassandraSource#5f18c296
at com.google.cloud.dataflow.worker.WorkerCustomSources$BoundedReaderIterator.start(WorkerCustomSources.java:582)
at com.google.cloud.dataflow.worker.util.common.worker.ReadOperation$SynchronizedReaderIterator.start(ReadOperation.java:347)
at com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:183)
at com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:148)
at com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:68)
at com.google.cloud.dataflow.worker.DataflowWorker.executeWork(DataflowWorker.java:336)
at com.google.cloud.dataflow.worker.DataflowWorker.doWork(DataflowWorker.java:294)
at com.google.cloud.dataflow.worker.DataflowWorker.getAndPerformWork(DataflowWorker.java:244)
at com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:135)
at com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:115)
at com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:102)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.datastax.driver.core.exceptions.SyntaxError: line 1:53 mismatched character 'p' expecting '$'
at com.datastax.driver.core.exceptions.SyntaxError.copy(SyntaxError.java:58)
at com.datastax.driver.core.exceptions.SyntaxError.copy(SyntaxError.java:24)
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:68)
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:43)
at org.apache.beam.sdk.io.cassandra.CassandraServiceImpl$CassandraReaderImpl.start(CassandraServiceImpl.java:80)
at com.google.cloud.dataflow.worker.WorkerCustomSources$BoundedReaderIterator.start(WorkerCustomSources.java:579)
... 14 more
Caused by: com.datastax.driver.core.exceptions.SyntaxError: line 1:53 mismatched character 'p' expecting '$'
at com.datastax.driver.core.Responses$Error.asException(Responses.java:144)
at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:179)
at com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:186)
at com.datastax.driver.core.RequestHandler.access$2500(RequestHandler.java:50)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.setFinalResult(RequestHandler.java:817)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:651)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1077)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1000)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:565)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:479)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
... 1 more
Maybe there is a mistake: CassandraServiceImpl.java#L220
And this statement looks like mistype: CassandraServiceImpl.java#L207
Changes I'v done to CassandraIO code
As #jkff proposed, I've change CassandraIO in the way I needed:
#VisibleForTesting
protected List<BoundedSource<T>> split(CassandraIO.Read<T> spec,
long desiredBundleSizeBytes,
long estimatedSizeBytes) {
long numSplits = 1;
List<BoundedSource<T>> sourceList = new ArrayList<>();
if (desiredBundleSizeBytes > 0) {
numSplits = estimatedSizeBytes / desiredBundleSizeBytes;
}
if (numSplits <= 0) {
LOG.warn("Number of splits is less than 0 ({}), fallback to 10", numSplits);
numSplits = 10;
}
LOG.info("Number of splits is {}", numSplits);
Long startRange = MIN_TOKEN;
Long endRange = MAX_TOKEN;
Long startToken, endToken;
String pk = "$pk";
switch (spec.table()) {
case "table1":
pk = "table1_pk";
break;
case "table2":
case "table3":
pk = "table23_pk";
break;
}
endToken = startRange;
Long incrementValue = endRange / numSplits - startRange / numSplits;
String splitQuery;
if (numSplits == 1) {
// we have an unique split
splitQuery = QueryBuilder.select().from(spec.keyspace(), spec.table()).toString();
sourceList.add(new CassandraIO.CassandraSource<T>(spec, splitQuery));
} else {
// we have more than one split
for (int i = 0; i < numSplits; i++) {
startToken = endToken;
endToken = startToken + incrementValue;
Select.Where builder = QueryBuilder.select().from(spec.keyspace(), spec.table()).where();
if (i > 0) {
builder = builder.and(QueryBuilder.gte("token(" + pk + ")", startToken));
}
if (i < (numSplits - 1)) {
builder = builder.and(QueryBuilder.lt("token(" + pk + ")", endToken));
}
sourceList.add(new CassandraIO.CassandraSource(spec, builder.toString()));
}
}
return sourceList;
}
I think this should be classified as a bug in CassandraIO. I filed BEAM-3424. You can try building your own version of Beam with that default of 1 changed to 100 or something like that, while this issue is being fixed.
I also filed BEAM-3425 for the bug during size estimation.
I am trying to gather data from jenkins using groovy script and getting an error:
unable to resolve class com.cloudbees.hudson.plugins.folder.Folder
Below is the code:
import jenkins.model.*
import hudson.model.*
import groovy.time.TimeCategory
use ( TimeCategory ) {
// e.g. find jobs not run in last 1 year
sometimeago = (new Date() - 1.year)
}
jobs = Jenkins.instance.getAllItems()
lastabort = null
jobs.each { j ->
if (j instanceof com.cloudbees.hudson.plugins.folder.Folder) { return }
numbuilds = j.builds.size()
if (numbuilds == 0) {
println 'JOB: ' + j.fullName
println ' -> no build'
return
}
lastbuild = j.builds[numbuilds - 1]
if (lastbuild.timestamp.getTime() < sometimeago) {
println 'JOB: ' + j.fullName
println ' -> lastbuild: ' + lastbuild.displayName + ' = ' + lastbuild.result + ', time: ' + lastbuild.timestampString2
}
}
The error is:
rg.codehaus.groovy.control.MultipleCompilationErrorsException:
startup failed: Script1.groovy: 12: unable to resolve class
com.cloudbees.hudson.plugins.folder.Folder # line 12, column 20. if (j
instanceof com.cloudbees.hudson.plugins.folder.Folder) { return } ^ 1
error at
org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:302)
I see Folder.java in jenkinsci/cloudbees-folder-plugin.
That means you need to:
check if you do have JENKINS/CloudBees Folders Plugin installed, or your groovy script would not be able to resolve that dependency.
Add "import com.cloudbees.hudson.plugins.folder.*" to be sure the script is able to make the instanceOf work.
When running groovy scripts that import libraries in Jenkins, check that your Jenkins build step is an "Execute system Groovy script", not a plain old "Execute Groovy script".
The 'system' scripts run on the existing JVM, as opposed to spawning a new one and therefore losing access to the shared libraries available to the original Jenkins JVM instance.
Groovy Script vs System Groovy Script - https://plugins.jenkins.io/groovy/
I am building a dynamic web project in eclipse Java EE and I added the external files of CodeMirror editor. I tried to use the external files of CodeMirror and it works good, but when I add the CodeMirror editor under WebContent 3 errors appear in index.html.
The code :
#types
immutable Color{T<:Number}
r::T
g::T
b::T
end
The error at the second line:
> Multiple annotations found at this line :
> - Start tag (<:Number>) not closed properly, expected '>'.
> - Invalid location of text (}) in tag (<:Number>).
> - Unknown tag (:Number).
The external files of CodeMirror work well but only when I add them to eclipse, it gives these errors inside the external files. How could I fix them?
edit:
this is all code :
<!doctype html>
<title>CodeMirror: Julia mode</title>
<meta charset="utf-8"/>
<link rel=stylesheet href="../../doc/docs.css">
<link rel="stylesheet" href="../../lib/codemirror.css">
<script src="../../lib/codemirror.js"></script>
<script src="julia.js"></script>
<style type="text/css">.CodeMirror {border-top: 1px solid black; border-bottom: 1px solid black;}</style>
<div id=nav>
<h1>CodeMirror</h1><img id=logo src="../../doc/logo.png">
<ul>
<li>Home
<li>Manual
<li>Code
</ul>
<ul>
<li>Language modes
<li><a class=active href="#">Julia</a>
</ul>
</div>
<article>
<h2>Julia mode</h2>
<div><textarea id="code" name="code">
#numbers
1234
1234im
.234
.234im
2.23im
2.3f3
23e2
0x234
#strings
'a'
"asdf"
r"regex"
b"bytestring"
"""
multiline string
"""
#identifiers
a
as123
function_name!
#unicode identifiers
# a = x\ddot
a⃗ = ẍ
# a = v\dot
a⃗ = v̇
#F\vec = m \cdotp a\vec
F⃗ = m·a⃗
#literal identifier multiples
3x
4[1, 2, 3]
#dicts and indexing
x=[1, 2, 3]
x[end-1]
x={"julia"=>"language of technical computing"}
#exception handling
try
f()
catch
#printf "Error"
finally
g()
end
#types
immutable Color{T<:Number}
r::T
g::T
b::T
end
#functions
function change!(x::Vector{Float64})
for i = 1:length(x)
x[i] *= 2
end
end
#function invocation
f('b', (2, 3)...)
#operators
|=
&=
^=
\-
%=
*=
+=
-=
<=
>=
!=
==
%
*
+
-
<
>
!
=
|
&
^
\
?
~
:
$
<:
.<
.>
<<
<<=
>>
>>>>
>>=
>>>=
<<=
<<<=
.<=
.>=
.==
->
//
in
...
//
:=
.//=
.*=
./=
.^=
.%=
.+=
.-=
\=
\\=
||
===
&&
|=
.|=
<:
>:
|>
<|
::
x ? y : z
#macros
#spawnat 2 1+1
#eval(:x)
#keywords and operators
if else elseif while for
begin let end do
try catch finally return break continue
global local const
export import importall using
function macro module baremodule
type immutable quote
true false enumerate
</textarea></div>
<script>
var editor = CodeMirror.fromTextArea(document.getElementById("code"), {
mode: {name: "julia",
},
lineNumbers: true,
indentUnit: 4,
matchBrackets: true
});
</script>
<p><strong>MIME types defined:</strong> <code>text/x-julia</code>.</p>
</article>
EDIT :
the first error he said processing instruction not closed
the second error
the all instruction in error 2:
cancast:{nw:x$"";if[not x in"BXCS";nw:(min 0#;max 0#;::)#\:nw];$[not any nw in x$(11&count y)#y;$[11<count y;not any nw in x$y;1b];0b]}
Some characters such as '<' or '>' are used to define HTML tags, etc, and therefore will be treated as code.
However, if you want to write '<', '>' as text, you must use the HTML codes for those characters. For exemple, '<' will become <,
'>' will become >. By replacing those characters by their corresponding code, they will not be recognized as HTML pure code.
At http://www.ascii.cl/htmlcodes.htm you will find a more complete list of those codes. This principle is called "HTML escape".
I'm debugging J2ME (written in eclipse) code on nokia N95 (wasn't written by me) trying to find error and printing out stacktrace gave me this:
03:08:479 TSKR. Error:
java.lang.NullPointerException: 0
- java.lang.String.<init>(), bci=6
- v.b(), bci=9
- v.e(), bci=805
- v.e(), bci=3
Could anyone please help me to understand that? Which line , where to look for and is there a way to understand it or at least get some valuable and usefull information.
Thank you
UPDATE
Sorry.. Here's the function that throws error:
public void bluetoothFileProcessBytes(){//--------tracing out of memory error
try{
partCurrentLoop++;
fileCurrentLoop++;
debug("Loop " + fileCurrentLoop + " of " + fileTotalLoops);
bluetoothUpdateBytes(fileDataString.length());
guiUpdateProgressBar(true, partCurrentLoop, partTotalLoops);
// LOOP Step 2: If there is no byte[] created for storing the bytes, create it.
if (fileBytesIsEmpty) {
if (partCurrentNumber == partTotalNumber)
{
fileBytes = new byte[fileSize % (loopsPerHttpComm * BYTES_PER_LOOP)];
}
else fileBytes = new byte[loopsPerHttpComm * BYTES_PER_LOOP];
}
fileBytesIsEmpty = false;
//LOOP Step 3: fill in the byte array with data from StringBuffer
for (int i = 0; i < fileDataString.length(); i++)
{
j = i + (partCurrentLoop - 1) * BYTES_PER_LOOP;
c = fileDataString.charAt(i);
fileBytes[j] = (byte) c;
}
c=0;
j=0;
i=0;
//LOOP Step 4: Send the email if the byte array is full with a new HttpComm Thread
if ((fileCurrentLoop % loopsPerHttpComm == 0
|| fileCurrentLoop == fileTotalLoops) && checkHttpCommStatus()) {
// update partName and httpCommStatus
String partName = fileName + " .part " + partCurrentNumber;
httpCommStatus = HTTP_RUNNING;
if (fileCurrentLoop == fileTotalLoops) { // FILE_END
debug("New HttpComm Thread: FILE END");
httpCommUpdateBytes(fileBytes.length);//<===================remove
httpCommSucceeded();//<====================================remove
/*new Thread(new HttpFileEnd(this, fileBytes, toAddress, fromAddress, fromName, digidownMAC, partName, fileName, fileSize, digidownSoftwareVersion, partCurrentNumber, //<===============uncomment
partTotalNumber, DigidownApp.textObject.getActiveLanguage())).start();*/
} else { // FILE_PART
//debug("Step 5");//<--------------------------------------------------------------------------------<-remove
debug("New HttpComm Thread: FILE PART: " + partName);
debug(">>>>>SEEEENDIIIING!<<<<<<<<");//<===========remove
httpCommUpdateBytes(fileBytes.length);//<================remove
httpCommSucceeded();//<==============================remove
/*new Thread(new HttpFilePart(this, fileBytes, toAddress, fromAddress, fromName,//<===============uncomment
digidownMAC, partName, fileSize, digidownSoftwareVersion,
partCurrentNumber, DigidownApp.textObject.getActiveLanguage())).start();*/
// Updating the new partTotalLoops
if (partCurrentNumber == partTotalNumber){
partTotalLoops = fileTotalLoops - (loopsPerHttpComm * (partCurrentNumber - 1));
} else partTotalLoops = loopsPerHttpComm;
partCurrentNumber++;
partCurrentLoop = 0;
}
fileBytesIsEmpty = true;
}
// Leave the loop if failed
if (!errorHandlerActivated) {
if (fileCurrentLoop < fileTotalLoops)
{
try{
bluetoothIOStream.getFileBytes();//-throws null pointer exception !
}catch(RuntimeException ea){debug("Wammaaa!!! " + ea.toString());
ea.printStackTrace();}
}
else if(fileCurrentLoop == fileTotalLoops && checkHttpCommStatus())
{ bluetoothIOStream.getFileEnd(); }
}
}
catch(RuntimeException e1)
{
Alert alert = new Alert("Fckn error!", e1.toString(), null, null);
alert.setTimeout(Alert.FOREVER);
debug("Error:");
System.err.println();
e1.printStackTrace();
//throw e1;
}
}
..and the stack trace of unobfuscated app; this time it was null pointer exception after
// Leave the loop if failed:
02:57:382 TSKR. Loop 972 of 1349
02:57:383 BIOS. Rec 978
02:57:588 TSKR. Loop 973 of 1349
02:57:590 BIOS. Rec 979
02:57:815 TSKR. Wammaaa!!! java.lang.NullPointerException: 0
java.lang.NullPointerException: 0
- java.lang.String.<init>(), bci=6
- net.digidown.m.digidown.bluetooth.BluetoothIOStream.readLineAsString(), bci=9
- net.digidown.m.digidown.bluetooth.BluetoothIOStream.getVariable(), bci=844
- net.digidown.m.digidown.bluetooth.BluetoothIOStream.getFileBytes(), bci=3
- net.digidown.m.digidown.TaskRunner.bluetoothFileProcessBytes(), bci=430
- net.digidown.m.digidown.bluetooth.BluetoothIOStream.getVariable(), bci=857
- net.digidown.m.digidown.bluetooth.BluetoothIOStream.getFileBytes(), bci=3
- net.digidown.m.digidown.TaskRunner.bluetoothFileProcessBytes(), bci=430
- net.digidown.m.digidown.bluetooth.BluetoothIOStream.getVariable(), bci=857
- net.digidown.m.digidown.bluetooth.BluetoothIOStream.getFileBytes(), bci=3
- net.digidown.m.digidown.TaskRunner.bluetoothFileProcessBytes(), bci=430
- net.digidown.m.digidown.bluetooth.BluetoothIOStream.getVariable(), bci=857
- net.digidown.m.digidown.bluetooth.BluetoothIOStream.getFileBytes(), bci=3
- net.digidown.m.digidown.TaskRunner.bluetoothFileProcessBytes(), bci=430
- net.digidown.m.digidown.bluetooth.BluetoothIOStream.getVariable(), bci=857
Goes like that for a while and then..:
- net.digidown.m.digidown.TaskRunner.bluetoothDoCommand_file(), bci=227
- net.digidown.m.digidown.bluetooth.BluetoothIOStream.getVariable(), bci=829
- net.digidown.m.digidown.bluetooth.BluetoothIOStream.getFileSettings(), bci=9
- net.digidown.m.digidown.TaskRunner.bluetoothTask(), bci=90
- net.digidown.m.digidown.bluetooth.BluetoothIOStream.getVariable(), bci=513
- net.digidown.m.digidown.bluetooth.BluetoothIOStream.getTask(), bci=2
- net.digidown.m.digidown.TaskRunner.bluetoothTask(), bci=172
- net.digidown.m.digidown.bluetooth.BluetoothIOStream.getVariable(), bci=513
- net.digidown.m.digidown.bluetooth.BluetoothIOStream.getTask(), bci=2
- net.digidown.m.digidown.TaskRunner.bluetoothConnected(), bci=89
- net.digidown.m.digidown.bluetooth.BluetoothIOStream.getVariable(), bci=444
- net.digidown.m.digidown.bluetooth.BluetoothIOStream.getVariable(), bci=426
- net.digidown.m.digidown.bluetooth.BluetoothIOStream.run(), bci=366
- java.lang.Thread.run(), bci=11
03:01:211 TSKR. Finished Task
03:01:212 BIOS. Rec 6
03:01:356 BIOS. task = >
03:01:357 TSKR. :: Got Task: (0x3e)
03:01:358 TSKR.
03:01:359 TSKR. ERROR HANDLER: 116 - Bluetooth communication error
03:01:389 BIOS. Initiated
03:01:389 BIOS. About to read
03:01:394 BIOS. Phone(InitiateDigidownError) received: '0x3e 0xea Data CK'
03:01:397 BIOS. About to writeLine 'error'
03:01:468 BIOS. quitStatus = QUIT_STATUS
Your problem seems to be obfuscation.
The build process of your eclipse application probably uses a tool like proguard to increase the performance and decrease the size of the application .jar file that will be deployed in the field.
One of the thing obfuscation does is rewrite class and method names into much smaller names, hence the completely unintelligible last 3 lines of your stack trace.
You need to produce an unobfuscated .jar file to use when you want to debug your application.
(at least until you find an issue that only happens on the obfuscated version of your application. it happens)
Eclipse should allow you to turn obfuscation off (or reduce it to its lowest level) by modifying your project properties. Failing that, manually and temporarily hacking the ant .xml files used to build your .jar file will do the trick.
Obfuscation parameters should include the name of your MIDlet class so that its startApp() method is not renamed.
Your problem may be solve in two ways:
Run your application without obfuscation. So the output shows exact method names.
Put System.out.println('method_name::sample_tag') in your method and lines that the problem may be from them. Then in the output you can trace your execution and you can find the point that the problem occurs.