Develop Reference

Why am I getting StringIndexOutOfBoundsException error in this substring method?

protected Void doInBackground(Void... voids) {
...
ABV = elem.select("td > span.muted").text();
Log.d("myLOG_ABV", ABV);
Log.d("myLOG_ABVlength", String.valueOf(ABV.length()));
/*String temp_ABV = ABV.substring(ABV.length()-6, ABV.length());*/
... }
Result
D/myLOG_ABV: Russian River Brewing Company American Wild Ale | 7.50%
D/myLOG_ABVlength: 55
and then, I cancled the annotation code.
...
ABV = elem.select("td > span.muted").text();
Log.d("myLOG_ABV", ABV);
Log.d("myLOG_ABVlength", String.valueOf(ABV.length()));
***String temp_ABV = ABV.substring(ABV.length()-6, ABV.length());***
...
Result
Caused by: java.lang.StringIndexOutOfBoundsException: length=0;
index=-6
Why am I getting StringIndexOutOfBoundsException error in this substring method?
I got the result that 'ABVlength : 55' in my code with annotation.
But after cancellation of annotation, I got StringIndexOutOfBoundsException.
Seriously, I am fighting with this code for 7hours 30minutes.

So we have:
***String temp_ABV = ABV.substring(ABV.length()-6, ABV.length());***
Caused by: java.lang.StringIndexOutOfBoundsException: length=0; index=-6
How can you get an index of -6 in this situation? ABV must have zero length, i.e. "".
It may be the case that ABV should have at least six characters, but that indicates a bug elsewhere.
What to do?
First and foremost, validate your inputs.
Somewhere you would normally want something like:
if (abv.length() < 6) {
throw new IllegalArgumentException("...");
}
Next find where that value is coming from and fix the bug.
Looks like you're parsing some kind of external document. In this case you may want the handling to be more lenient:
int abvLen = abv.length();
int abvEndIndex = abvLen - 6;
if (abvEndIndex < 0) {
log.error("ABV length less than 6", abv);
abvEndIndex = 0;
}
String abvEnd = abv.substring(abvEndIndex, abvLen);

Java Spark : Spark Bug Workaround for Datasets Joining with unknow Join Column Names

I am using Spark 2.3.1 with Java.
I have encountered what (I think), is this known bug of Spark.
Here is my code :
public Dataset<Row> compute(Dataset<Row> df1, Dataset<Row> df2, List<String> columns){
Seq<String> columns_seq = JavaConverters.asScalaIteratorConverter(columns.iterator()).asScala().toSeq();
final Dataset<Row> join = df1.join(df2, columns_seq);
join.show()
join.withColumn("newColumn", abs(col("value1").minus(col("value2")))).show();
return join;
}
I call my code like this :
Dataset<Row> myNewDF = compute(MyDataset1, MyDataset2, Arrays.asList("field1","field2","field3","field4"));
Note : MyDataset1 and MyDataset2 are two datasets that come from the same Dataset MyDataset0 with multiple different transformations.
On the join.show() line, I get the following error :
2018-08-03 18:48:43 - ERROR main Logging$class - - - failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 235, Column 21: Expression "project_isNull_2" is not an rvalue
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 235, Column 21: Expression "project_isNull_2" is not an rvalue
at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11821)
at org.codehaus.janino.UnitCompiler.toRvalueOrCompileException(UnitCompiler.java:7170)
at org.codehaus.janino.UnitCompiler.getConstantValue2(UnitCompiler.java:5332)
at org.codehaus.janino.UnitCompiler.access$9400(UnitCompiler.java:212)
at org.codehaus.janino.UnitCompiler$13$1.visitAmbiguousName(UnitCompiler.java:5287)
at org.codehaus.janino.Java$AmbiguousName.accept(Java.java:4053)
...
2018-08-03 18:48:47 - WARN main Logging$class - - - Whole-stage codegen disabled for plan (id=7):
But it does not stop the execution and still displays the content of the dataset.
Then, on the line join.withColumn("newColumn", abs(col("value1").minus(col("value2")))).show();
I get the error :
Exception in thread "main" org.apache.spark.sql.AnalysisException: Resolved attribute(s) 'value2,'value1 missing from field6#16,field7#3,field8#108,field5#0,field9#4,field10#28,field11#323,value1#298,field12#131,day#52,field3#119,value2#22,field2#35,field1#43,field4#144 in operator 'Project [field1#43, field2#35, field3#119, field4#144, field5#0, field6#16, value2#22, field7#3, field9#4, field10#28, day#52, field8#108, field12#131, value1#298, field11#323, abs(('value1 - 'value2)) AS newColumn#2579]. Attribute(s) with the same name appear in the operation: value2,value1. Please check if the right attribute(s) are used.;;
'Project [field1#43, field2#35, field3#119, field4#144, field5#0, field6#16, value2#22, field7#3, field9#4, field10#28, day#52, field8#108, field12#131, value1#298, field11#323, abs(('value1 - 'value2)) AS newColumn#2579]
+- AnalysisBarrier
...
This error end the program.
The workaround proposed Mijung Kim on the Jira Issue is to create a Dataset clone thanks to toDF(Columns). But in my case, where the column names used for the join are not known in advance (I only have a List), I can't use this workaround.
Is there another way to get around this very annoying bug ?

Try to call this method:
private static Dataset<Row> cloneDataset(Dataset<Row> ds) {
List<Column> filterColumns = new ArrayList<>();
List<String> filterColumnsNames = new ArrayList<>();
scala.collection.Iterator<StructField> it = ds.exprEnc().schema().toIterator();
while (it.hasNext()) {
String columnName = it.next().name();
filterColumns.add(ds.col(columnName));
filterColumnsNames.add(columnName);
}
ds = ds.select(JavaConversions.asScalaBuffer(filterColumns).seq()).toDF(scala.collection.JavaConverters.asScalaIteratorConverter(filterColumnsNames.iterator()).asScala().toSeq());
return ds;
}
on both datasets just before the join like this :
df1 = cloneDataset(df1);
df2 = cloneDataset(df2);
final Dataset<Row> join = df1.join(df2, columns_seq);
// or ( based on Nakeuh comment )
final Dataset<Row> join = cloneDataset(df1.join(df2, columns_seq));

How can I remove the output layer of pre-trained model with TensorFlow java api?

I have the pre-trained model like Inception-v3. I want to remove the output layer and use it in image cognition. Here is the example given by tensorflow:
Just like the python framework Keras, it has a method like model.layers.pop(). I tried do it with tensorflow java api. First I tried to use dl4j, but when I imported the keras model, I got an error like this:
2017-06-15 21:15:43 INFO KerasInceptionV3Net:52 - Importing Inception model from data/inception-model.json
2017-06-15 21:15:43 INFO KerasInceptionV3Net:53 - Importing Weights model from data/inception_v3_complete
Exception in thread "main" java.lang.RuntimeException: Unknown exception.
at org.bytedeco.javacpp.hdf5$H5File.allocate(Native Method)
at org.bytedeco.javacpp.hdf5$H5File.<init>(hdf5.java:12713)
at org.deeplearning4j.nn.modelimport.keras.Hdf5Archive.<init>(Hdf5Archive.java:61)
at org.deeplearning4j.nn.modelimport.keras.KerasModel$ModelBuilder.weightsHdf5Filename(KerasModel.java:603)
at org.deeplearning4j.nn.modelimport.keras.KerasModelImport.importKerasModelAndWeights(KerasModelImport.java:176)
at edu.usc.irds.dl.dl4j.examples.KerasInceptionV3Net.<init>(KerasInceptionV3Net.java:55)
at edu.usc.irds.dl.dl4j.examples.KerasInceptionV3Net.main(KerasInceptionV3Net.java:108)
HDF5-DIAG: Error detected in HDF5 (1.10.0-patch1) thread 0:
#000: C:\autotest\HDF5110ReleaseRWDITAR\src\H5F.c line 579 in H5Fopen(): unable to open file
major: File accessibilty
minor: Unable to open file
#001: C:\autotest\HDF5110ReleaseRWDITAR\src\H5Fint.c line 1100 in H5F_open(): unable to open file: time = Thu Jun 15 21:15:44 2017,name = 'data/inception_v3_complete', tent_flags = 0
major: File accessibilty
minor: Unable to open file
#002: C:\autotest\HDF5110ReleaseRWDITAR\src\H5FD.c line 812 in H5FD_open(): open failed
major: Virtual File Layer
minor: Unable to initialize object
#003: C:\autotest\HDF5110ReleaseRWDITAR\src\H5FDsec2.c line 348 in H5FD_sec2_open(): unable to open file: name = 'data/inception_v3_complete', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0
major: File accessibilty
minor: Unable to open file
So I went back to tensorflow. I'm going to modify the model in keras and convert the model to tensor. Here is my conversion script:
input_fld = './'
output_node_names_of_input_network = ["pred0"]
write_graph_def_ascii_flag = True
output_node_names_of_final_network = 'output_node'
output_graph_name = 'test2.pb'
from keras.models import load_model
import tensorflow as tf
import os
import os.path as osp
from keras.applications.inception_v3 import InceptionV3
from keras.applications.vgg16 import VGG16
from keras.models import Sequential
from keras.layers.core import Flatten, Dense, Dropout
from keras.layers.convolutional import Convolution2D, MaxPooling2D, ZeroPadding2D
from keras.optimizers import SGD
output_fld = input_fld + 'tensorflow_model/'
if not os.path.isdir(output_fld):
os.mkdir(output_fld)
net_model = InceptionV3(weights='imagenet', include_top=True)
num_output = len(output_node_names_of_input_network)
pred = [None]*num_output
pred_node_names = [None]*num_output
for i in range(num_output):
pred_node_names[i] = output_node_names_of_final_network+str(i)
pred[i] = tf.identity(net_model.output[i], name=pred_node_names[i])
print('output nodes names are: ', pred_node_names)
from keras import backend as K
sess = K.get_session()
if write_graph_def_ascii_flag:
f = 'only_the_graph_def.pb.ascii'
tf.train.write_graph(sess.graph.as_graph_def(), output_fld, f, as_text=True)
print('saved the graph definition in ascii format at: ', osp.join(output_fld, f))
from tensorflow.python.framework import graph_util
from tensorflow.python.framework import graph_io
constant_graph = graph_util.convert_variables_to_constants(sess, sess.graph.as_graph_def(), pred_node_names)
graph_io.write_graph(constant_graph, output_fld, output_graph_name, as_t ext=False)
print('saved the constant graph (ready for inference) at: ', osp.join(output_fld, output_graph_name))
I got the model as .pb file, but when I put it into the tensor example, The LabelImage example, I got this error:
Exception in thread "main" java.lang.IllegalArgumentException: You must feed a value for placeholder tensor 'batch_normalization_1/keras_learning_phase' with dtype bool
[[Node: batch_normalization_1/keras_learning_phase = Placeholder[dtype=DT_BOOL, shape=<unknown>, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
at org.tensorflow.Session.run(Native Method)
at org.tensorflow.Session.access$100(Session.java:48)
at org.tensorflow.Session$Runner.runHelper(Session.java:285)
at org.tensorflow.Session$Runner.run(Session.java:235)
at com.dlut.cmh.sheng.LabelImage.executeInceptionGraph(LabelImage.java:98)
at com.dlut.cmh.sheng.LabelImage.main(LabelImage.java:51)
I don't know how to solve this. Can anyone help me? Or you have another way to do this?

The error message you get from the TensorFlow Java API:
Exception in thread "main" java.lang.IllegalArgumentException: You must feed a value for placeholder tensor 'batch_normalization_1/keras_learning_phase' with dtype bool
[[Node: batch_normalization_1/keras_learning_phase = Placeholder[dtype=DT_BOOL, shape=<unknown>, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
suggests that the model is constructed in a way that requires you to feed a boolean value for the tensor named batch_normalization_1/keras_learning_phase.
So, you'd have to include that in your call to run by changing:
try (Session s = new Session(g);
Tensor result = s.runner().feed("input",image).fetch("output").run().get(0)) {
to something like:
try (Session s = new Session(g);
Tensor learning_phase = Tensor.create(false);
Tensor result = s.runner().feed("input", image).feed("batch_normalization_1/keras_learning_phase", learning_phase).fetch("output").run().get(0)) {
The names of nodes you feed and fetch depend on the model, so it's possible that the names of the 'input' and 'output' nodes are different as well.
You might also want to consider using the TensorFlow SavedModel format (see also https://github.com/tensorflow/serving/issues/310#issuecomment-297015251)
Hope that helps

Selenium testing with lists for load performance

I'm attempting to use a unit test of mine for "load" testing on our browser. For various reasons, we have seen performance degradation on the browser side, because we heavily rely on the print dialog.
I have the following unit test working via ScalaTest:
class LoadPrePaidSpec extends FlatSpec with Matchers with Chrome with Eventually {
implicit override val patienceConfig =
PatienceConfig(timeout = scaled(Span(40, Seconds)), interval = scaled(Span(100, Millis)))
def build(csvLine:String):TestCSVHolder ={
val split = csvLine.split(",")
TestCSVHolder(memberId = split(0), preSaleCode = split(1),
prePaidCode = split(2), lastName = split(3), firstName = split(4), badgeName = split(5))
}
def memberHelper(member: TestCSVHolder): Unit = {
//insert member id via prepaid code
textField("member_id").value = member.prePaidCode
//fire keyup event
executeScript("var eventToFire=jQuery.Event(\"keyup\");eventToFire.keyCode=221;eventToFire.which=221;" +
"$(\"#member_id\").trigger(eventToFire)")
eventually {
val eles = webDriver.findElements(By.xpath(s"//*[contains(#id, '${member.memberId}')]"))
eles.get(0).getTagName
//We remove the head element because it just says Prep For Print
val tdEles = (eles.get(0).findElements(By.tagName("td")).toList.tail)
tdEles(0).getText() should be(member.lastName)
tdEles(1).getText() should be(member.firstName)
tdEles(2).getText() should be(member.badgeName)
}
}
"Scanning an ID" should "look up the member" in {
val member = new TestCSVHolder("100001", "ABCD", "[-100001-ABCD]", "John", "Doe", "JohnDoe")
go to (url)
//login
textField("user_name").value = "mrkaiser"
webDriver.findElementById("credentials").sendKeys("somepassword")
click on ("btnLogin")
//click to pre-paid
click on linkText("Pre-Paid")
memberHelper())
webDriver.quit()
}
}
However when I try to iterate through a list of elements using a foreach and passing in memberHelper, after a list of about 5 elements, I get the following stack trace:
The code passed to eventually never returned normally. Attempted 369 times over 40.110734904 seconds. Last failure message: Index: 0, Size: 0.
ScalaTestFailureLocation: LoadPrePaidSpec at (LoadPrePaidSpec.scala:43)
org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed to eventually never returned normally. Attempted 369 times over 40.110734904 seconds. Last failure message: Index: 0, Size: 0.
at org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:420)
at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:438)
at LoadPrePaidSpec.eventually(LoadPrePaidSpec.scala:17)
at LoadPrePaidSpec.memberHelper(LoadPrePaidSpec.scala:43)
at LoadPrePaidSpec$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(LoadPrePaidSpec.scala:70)
at LoadPrePaidSpec$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(LoadPrePaidSpec.scala:70)
at scala.collection.immutable.List.foreach(List.scala:383)
at LoadPrePaidSpec$$anonfun$1.apply$mcV$sp(LoadPrePaidSpec.scala:70)
at LoadPrePaidSpec$$anonfun$1.apply(LoadPrePaidSpec.scala:54)
at LoadPrePaidSpec$$anonfun$1.apply(LoadPrePaidSpec.scala:54)
at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1647)
at org.scalatest.Suite$class.withFixture(Suite.scala:1122)
at org.scalatest.FlatSpec.withFixture(FlatSpec.scala:1683)
at org.scalatest.FlatSpecLike$class.invokeWithFixture$1(FlatSpecLike.scala:1644)
at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1656)
at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1656)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
at org.scalatest.FlatSpecLike$class.runTest(FlatSpecLike.scala:1656)
at org.scalatest.FlatSpec.runTest(FlatSpec.scala:1683)
at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1714)
at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1714)
at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
at scala.collection.immutable.List.foreach(List.scala:383)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:390)
at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:427)
at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
at scala.collection.immutable.List.foreach(List.scala:383)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
at org.scalatest.FlatSpecLike$class.runTests(FlatSpecLike.scala:1714)
at org.scalatest.FlatSpec.runTests(FlatSpec.scala:1683)
at org.scalatest.Suite$class.run(Suite.scala:1424)
at org.scalatest.FlatSpec.org$scalatest$FlatSpecLike$$super$run(FlatSpec.scala:1683)
at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1760)
at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1760)
at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
at org.scalatest.FlatSpecLike$class.run(FlatSpecLike.scala:1760)
at org.scalatest.FlatSpec.run(FlatSpec.scala:1683)
at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:55)
at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2563)
at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2557)
at scala.collection.immutable.List.foreach(List.scala:383)
at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2557)
at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1044)
at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1043)
at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:2722)
at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1043)
at org.scalatest.tools.Runner$.run(Runner.scala:883)
at org.scalatest.tools.Runner.run(Runner.scala)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:138)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:653)
at java.util.ArrayList.get(ArrayList.java:429)
at LoadPrePaidSpec$$anonfun$memberHelper$1.apply$mcV$sp(LoadPrePaidSpec.scala:45)
at LoadPrePaidSpec$$anonfun$memberHelper$1.apply(LoadPrePaidSpec.scala:43)
at LoadPrePaidSpec$$anonfun$memberHelper$1.apply(LoadPrePaidSpec.scala:43)
at org.scalatest.concurrent.Eventually$class.makeAValiantAttempt$1(Eventually.scala:394)
at org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:408)
... 63 more
My end goal is to actually test something in the 20K range of elements from a file, but until I can get a small list like this working, I'm up a creek.
I'm using the chromedriver and am on Scala 2.11.6, scala test 2.2.0, selenium 2.35.0.

How can i extract components from fastICA

I have problems to get ICA component fastICA in R. when i try to extract 6 component from fastICA function it give only one component but there should be 6 components . upto 5 it was worked perfectly but after the 5 it gives different different number of components.can anyone tell me what is the reason for that
Function and Parameters:
ICA6 <- fastICA(X, 6, alg.typ = "parallel", fun = "logcosh", alpha = 1,
method = "R", row.norm = FALSE, maxit = 200, tol = 0.0001, verbose = TRUE)

here is the last few lines of my output for ICA6 and ICA5
hpc-admin#aiken:~/gayan$ tail ICA6
[1992,] -1.755614e-01
[1993,] -1.931838e-01
[1994,] -1.403488e-01
[1995,] 4.952370e-01
[1996,] 3.798545e-02
[1997,] -8.870945e-02
[1998,] -1.847535e-01
[1999,] 2.084906e-01
[2000,] 2.235841e-01
hpc-admin#aiken:~/gayan$ tail ICA5
[1992,] -4.449966e-02 2.348224e-02 -0.1296879740 4.220189e-02 -0.1751827781
[1993,] -7.690094e-02 1.725353e-02 -0.1153838819 1.694351e-01 -0.1308105118
[1994,] -4.777415e-02 2.299214e-02 -0.1259907838 -6.011591e-03 -0.1605316621
[1995,] 4.354237e-02 2.295694e-02 -0.2499377363 -2.227481e-01 0.4414782035
[1996,] -3.848286e-02 2.121986e-02 -0.1361600020 -8.448882e-02 -0.0005046113
[1997,] -3.030994e-03 2.285310e-02 -0.1407370888 -1.215308e-02 -0.1062227838
[1998,] -3.988264e-03 2.335983e-02 -0.1497881709 1.787074e-02 -0.1982725941
[1999,] -7.483824e-02 1.096696e-02 -0.0672348301 -1.665848e-01 0.1489732404
[2000,] -7.123032e-01 1.123832e-02 0.5153474842 -1.785166e-01 0.1632015019