Why R calculations are inconsistent in my virtual machine? - java

I'm trying to build a new virtual machine with R and the follow packages below running as a R server to my calculations.
#this is how I install my R-packages
function install_packages(){
folder='dir.create(Sys.getenv("R_LIBS_USER"), showWarnings = FALSE, recursive = TRUE)'
packages='install.packages(c("Rserve","fArma","fGarch","tseries","MASS","lattice","gtools","gmodels","gplots","HiddenMarkov", "xts", "PerformanceAnalytics"), Sys.getenv("R_LIBS_USER"), repos = "http://cran.rstudio.com")'
echo "$folder" >> ./install_packages.R
echo "$packages" >> ./install_packages.R
sudo /usr/bin/R CMD BATCH install_packages.R
rm -f ./install_packages.R
}
If I make a call (using mvn clean package) from my host machine to this new virtual machine, it gives me a strange error in my calculations:
Running com.company.documentengine.statistics.JensensAlphaTest
Oct 28, 2015 2:17:45 PM com.company.documentengine.toolbox.util.DatabaseConnection connectToDB
INFO: PostgreSQL JDBC Driver Registered
Oct 28, 2015 2:17:45 PM com.company.documentengine.toolbox.util.DatabaseConnection connectToDB
INFO: test Database connection confirmed for user postgres
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 17.971 sec <<< FAILURE! - in com.company.documentengine.statistics.JensensAlphaTest
testCalculate(com.company.documentengine.statistics.JensensAlphaTest) Time elapsed: 8.821 sec <<< FAILURE!
java.lang.AssertionError: Calculation wrong. expected:<0.039801296645998546> but was:<NaN>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:553)
at com.company.documentengine.statistics.JensensAlphaTest.testCalculate(JensensAlphaTest.java:40)
Now, if I make the same call but from new virtual machine to my host machine (which also has all these packages installed), everything works.
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running com.company.documentengine.statistics.JensensAlphaTest
Oct 28, 2015 1:23:13 PM com.company.documentengine.toolbox.util.DatabaseConnection connectToDB
INFO: PostgreSQL JDBC Driver Registered
Oct 28, 2015 1:23:13 PM com.company.documentengine.toolbox.util.DatabaseConnection connectToDB
INFO: test Database connection confirmed for user postgres
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.465 sec - in com.company.documentengine.statistics.JensensAlphaTest
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 21.423s
[INFO] Finished at: Wed Oct 28 13:23:20 UTC 2015
[INFO] Final Memory: 18M/362M
[INFO] ------------------------------------------------------------------------
I'm really confuse about this, can anyone please give me some suggestion/idea, please!
EDIT
I tried to debug my test to see where I'm making the mistake, but still no clue. Now I know at least that my problem is with ... look my debug comparison please. And this is the comparison to all my packages used in both cases.
Java Code
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(classes = {TestContext.class})
#ActiveProfiles(profiles = {"test"})
public class JensensAlphaTest {
#Autowired
private TestSeriesManager testSeriesManager;
#Test
public void testCalculate() throws Exception {
PriceSeries<PriceSeriesDatum> dax = testSeriesManager.getDax();
PriceSeries<PriceSeriesDatum> sDax = testSeriesManager.getSDax();
InterestRateSeries<InterestRateDatum> euribor = testSeriesManager.getEuribor();
LocalDate asOfDate = LocalDate.of(2014, 10, 1);
JensensAlpha jensensAlpha = new JensensAlpha(dax, sDax, euribor, asOfDate);
double eps = 1e-15;
/* here is the inconsistent part */
double actualValue = jensensAlpha.calculate(Period.SINCE_INCEPTION, ReturnsType.DAILY_DISCRETE);
double expectedValue = 0.039801296645998546;
assertEquals("Calculation wrong.", expectedValue, actualValue, eps);
}
}
This is the method called:
public double calculate(Period period, ReturnsType returnsType) {
NavigableMap<LocalDate, Double> returnSeries = returnsType.getReturnSeries(series);
NavigableMap<LocalDate, Double> returnBenchmark = returnsType.getReturnSeries(benchmark);
NavigableMap<LocalDate, Double> returnRiskFree = returnsType.getReturnSeries(riskFree);
LocalDate startDate = period.getStartDate(returnSeries);
NavigableMap<LocalDate, Double> cutReturnSeries = StatisticsUtils.getMapSince(startDate, returnSeries);
NavigableMap<LocalDate, Double> cutBenchmarkReturnSeries;
NavigableMap<LocalDate, Double> cutRiskFreeReturnSeries;
try {
cutBenchmarkReturnSeries = StatisticsUtils.getMapSince(startDate, returnBenchmark);
cutRiskFreeReturnSeries = StatisticsUtils.getMapSince(startDate, returnRiskFree);
} catch (IllegalArgumentException e) {
throw new NotEnoughDataException(
"This error can occur when the price series is short (only a few returns), so the benchmark is not"
+ " updated for the taken first date of the series.", e);
}
REXPS4[] inputClasses =
{RexpParser.createREXPS4Class(cutReturnSeries), RexpParser.createREXPS4Class(cutBenchmarkReturnSeries),
RexpParser.createREXPS4Class(cutRiskFreeReturnSeries)};
RScript script = RScript.fromFileName("JensensAlpha.R");
REXPS4 resultClass = script.execute(inputClasses);
try {
return resultClass.getAttribute("value").asDouble();
} catch (REXPMismatchException e) {
throw new RScriptException("Exception while getting results from the R script.", e);
}
}
And the execute method:
#Override
public REXPS4 execute(REXPS4[] inputClasses) {
RConnection c = RConnectionSingleton.INSTANCE.getRConnection();
try {
int inputClassNumber = 1;
for (REXPS4 inputClass : inputClasses) {
c.assign("inputClass" + inputClassNumber, inputClass);
inputClassNumber++;
}
c.eval(code);
/* the resultClass is wrong only when I connect to my vm */
return (REXPS4) c.get("resultClass", null, true);
} catch (REngineException e) {
throw new ScriptExecutionException("Exception while trying to execute the RScript.", e);
}
}

Just to let you know what happened with my issue.
The problem was the TIMEZONE. I don't know exactly why, but R or some of the packages that we use for our calculations requires the timezone to be the same.
I'm located in Germany (timezone CET which is +1 UTC) and I was setting my virtual machine to use UTC and thus the problem. Oh man, I'm really glad to solve this issue (3 days straight working on this!) but now everything is fine! A huge thank you to my colleague #Ralf for the tip!
Other issue related:1, 2, 3.
I hope this helps someone! :)

Related

JAVA maven clean install test not finishing iterator

I have simple script that reads a file:
/**
* Takes a txt file with all songs listed in music directory
* and saves this to the songs database
*/
public void rebuildFromFilelist() {
final List<Song> songs = new ArrayList<>();
try {
final Scanner sc = new Scanner(new File(songsfile));
sc.useDelimiter("\r?\n|\r");
sc.forEachRemaining(l -> getSongFromLine(songs, l)); //function constructs a Song object
} catch (final Exception e) {
log.error("Error in processing file: {}", songsfile);
e.printStackTrace();
}
songs.forEach(song -> log.debug(song.toString()));
songRepository.saveAll(songs);
}
Then i have a test:
#RunWith(SpringRunner.class)
#SpringBootTest
class SongRepositoryServiceTest {
#Autowired
private SongRepositoryService songRepositoryService;
#Test
#DisplayName("Rebuild library from file")
void RebuildLibraryWithFile() {
songRepositoryService.rebuildLibraryWithFile();
final List<Song> songs = songRepositoryService.findByCategory(Song.SongCategory.SONG);
final List<Song> others = songRepositoryService.findByCategory(Song.SongCategory.OTHER);
assertFalse(songs.isEmpty());
assertFalse(others.isEmpty());
}
}
The file that I'm processing is about 620 items long. The first 400 are of category SONG, the rest is category OTHER.
When I run the test from Intellij, the test succeeds. However, if I run it during mvn clean install, it only processes the first 155 items and stops (thus failing the second assertion).
Why is this? How can I fix this, or even debug this?
Regards,
Barbet
EDIT:
The failing test only reports this, no errors or stacktraces:
-------------------------------------------------------------------------------
Test set: com.gsm.GsmWeb.service.SongRepositoryServiceTest
-------------------------------------------------------------------------------
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.161 s <<< FAILURE! - in com.gsm.GsmWeb.service.SongRepositoryServiceTest
com.gsm.GsmWeb.service.SongRepositoryServiceTest.RebuildLibraryWithFile Time elapsed: 0.532 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <false> but was: <true>
at com.gsm.GsmWeb.service.SongRepositoryServiceTest.RebuildLibraryWithFile(SongRepositoryServiceTest.java:88)
Seems some problem with the maven build in relation to the java Scanner iterator. I could not debug it as it was very flaky. I solved the problem by replacing the
Files.lines(new File(songsfile).toPath()).forEach(l -> getSongFromLine(songs, l));
Now everything works perfectly.

Cucumber and Jenkins: False "duplicate step definition"

I'm getting this error message in a Jenkins build for a Java, Selenium and Cucumber project:
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 1.466 sec <<< FAILURE! - in
e2e.CucumberTest
e2e.CucumberTest Time elapsed: 1.466 sec <<< ERROR!
cucumber.runtime.DuplicateStepDefinitionException: Duplicate step definitions in void
e2e.sak.OpprettSakSteps.TestCaseDSLTester(String) in file:/tmp/workspace/n_DSL-og-
TestsakBuilder_combined/e2e/cucumber/target/test-classes/ and
e2e.sak.OpprettSakSteps.TestCaseDSLTester$default(OpprettSakSteps,String,int,Object) in file:/tmp
/workspace/n_DSL-og-TestsakBuilder_combined/e2e/cucumber/target/test-classes/
I don't see where the supposed duplicate step definition is? When looking through the Java/Kotlin file, there are absolutely no duplicates, and all step definitions have a trailing "$" (which has been the cause of earlier erroneous duplicate messages). Also, I don't understand what Jenkins is comparing, even though it seems that it tries to show me exactly where the duplicate is:
e2e.sak.OpprettSakSteps.TestCaseDSLTester(String)
and
e2e.sak.OpprettSakSteps.TestCaseDSLTester$default(OpprettSakSteps,String,int,Object)
It points to the same method, but with different parameters? Neither the method name nor the Cucumber step definition name is the same, so what's it complaining about?
Here's the step definition kode (Kotlin):
#Gitt("^jeg prøver meg på DSL for \"([^\"]*)\"$")
fun TestsakDSLTester(saksRef: String = "saken") {
TestsakDSL.create {
opprettet = LocalDateTime.now().minusDays(1)
kommunenr = "5035"
hovedsoker = "Kåre Kotlin"
fnr = "22097930922"
sokere = 2
arbeidlisteHovedsoker { mutableListOf(mutableListOf("Knus og Knask AS", 100, true),
mutableListOf("Del og Hel", 20, false)) }
arbeidlisteMedsoker { mutableListOf(mutableListOf("Kiosken på hjørnet", 50, false)) }
barn = 2
biler = 1
verger = mutableListOf(VergeUtils.Companion.VergeFor.HOVEDSOKER,
VergeUtils.Companion.VergeFor.MEDSOKER)
}
System.out.println("");
}
#Gitt("en sak med (\\d+) søkere med hver sin verge$")
fun enSakMedNSokereMedHverSinVerge(): Testsak {
return TestsakBuilder("saken", "5035")
.setSoknadtype(Bakgrunn.Hva.values().toList().shuffled().first())
.setVerger(vergeFor =
mutableListOf(VergeUtils.Companion.VergeFor.HOVEDSOKER)).createSak().build();
}

Mesuring Derby speed with hibernate. Strange values

I'm writing application to mesure speed of CRUD with hibernate for Derby.
This is my function :
#Override
public BulkTestResult testBulkInsertScenario(Long volume, Integer repeat) {
StopWatch sw = new StopWatch();
BulkTestResult bulkTestResult = new BulkTestResult();
bulkTestResult.setStartDate(Instant.now());
bulkTestResult.setCountTest(volume);
bulkTestResult.setTestRepeat(repeat);
familyService.clear();
for(int i =0; i < repeat; i++) {
List<ProjectEntity> projects = dataAnonymization.generateProjectEntityList(volume);
runBulkTest(sw, bulkTestResult, projects, true);
}
bulkTestResult.setEndDate(Instant.now());
return bulkTestResult;
}
private void runBulkTest(StopWatch sw, BulkTestResult bulkTestResult, List<ProjectEntity> projects, boolean resetAfter) {
sw.reset();
sw.start();
familyService.save(projects);
sw.stop();
bulkTestResult.addMsSpeedResult(sw.getTime());
if (resetAfter) familyService.clear();
sw.reset();
}
clear method remove all record from DB.
The problem that I have is values that I recieved as output of application.
Testing data : 1000 record, and 10 repeats
Example speed values recieved running this test few times:
311, 116, 87, (...)38
32, 27, 30, (...) 24
22, 19, 18, (...) 21
19, 18, 18, (...) 19
Why there are so many difference and why for first time insert is always slower ?
It could be any hardware acceleration ?
I found solution.
This issue is related to optimalization. After Disable JIT, recieved values are correct.
-Djava.compiler=NONE -Xint

Maven fail with varargs parameter

I'm trying to create the unit test for my application but unfortunately I got one strange problem.
I have one method with two parameters and the last it's a varargs classes:
public JsonApi convertStringToJsonApi(String json, Class<?>... classes)
My test is extremely simple:
jsonApiString = "{ 'data': [ { 'type': 'TEST', 'id': '1', 'attributes': { 'attr1': 'Attribute', 'attr2': 200, 'attr3': { 'attr': { 'Model': 'XPTO', 'License': '85599' } } }, 'relationships': { 'name': 'Miguel', 'age': '30' } } ] }";
// jsonApiObjectResult = jsonMaker.convertStringToJsonApi(jsonApiString. EntityTestAttr1.class, EntityTestRels.class);
jsonApiObjectResult = jsonMaker.convertStringToJsonApi(jsonApiString, new Class<?>[] { EntityTestAttr1.class, EntityTestRels.class });
Assert.assertNotNull(jsonApiObjectResult);
Assert.assertNotNull(jsonApiObjectResult.getData().get(0));
Assert.assertEquals("1", jsonApiObjectResult.getData().get(0).getId());
Assert.assertEquals("TEST", jsonApiObjectResult.getData().get(0).getType());
Assert.assertEquals("Attribute", ((EntityTestAttr1) jsonApiObjectResult.getData().get(0).getAttr()).getAttr1());
Assert.assertEquals(new BigDecimal(200), ((EntityTestAttr1) jsonApiObjectResult.getData().get(0).getAttr()).getAttr2());
Assert.assertEquals("XPTO", ((EntityTestAttr1) jsonApiObjectResult.getData().get(0).getAttr()).getAttr3().getAttr().get("Model"));
Assert.assertEquals("85599", ((EntityTestAttr1) jsonApiObjectResult.getData().get(0).getAttr()).getAttr3().getAttr().get("License"));
Assert.assertEquals("Miguel", ((EntityTestRels) jsonApiObjectResult.getData().get(0).getRels()).getName());
Assert.assertEquals("30", ((EntityTestRels) jsonApiObjectResult.getData().get(0).getRels()).getAge());
When I run the unit test with eclipse all work perfect and the test get success; however, with maven I get an exception "java.lang.NullPointerException".
Maven stack:
Tests run: 12, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.075 sec <<< FAILURE! - in org.test.TestAux
convertAllResourcesStringToJsonApiTest(org.test.Maker) Time elapsed: 0.063 sec <<< ERROR!
java.lang.NullPointerException
at org.test.Maker.convertAllResourcesStringToJsonApiTest(TestAux.java:208)
I believe it's because the varargs for when I use only one class in the second parameter (varargs) then works fine and maven gets success.
Why? I don't understand! :(

Getting "junit.framework.AssertionFailedError: Forked Java VM exited abnormally" Exception

I have a java program for JUnit test which I have written in netbeans IDE where I am testing a class with single thread.
When I am going to compile this its working fine and showing results 100% success, but
when I am trying to run this program I am getting an exception that is -
Testsuite: glb.chatmeter.crawler.yahoolocal.YahooBusinessDataTest
Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 0.057 sec
Testcase: warning(junit.framework.TestSuite$1): FAILED
No tests found in glb.chatmeter.crawler.yahoolocal.YahooBusinessDataTest
junit.framework.AssertionFailedError: No tests found in glb.chatmeter.crawler.yahoolocal.YahooBusinessDataTest
Testsuite: glb.chatmeter.crawler.yahoolocal.YahooBusinessDataTest
Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
Testcase: glb.chatmeter.crawler.yahoolocal.YahooBusinessDataTest:null: Caused an ERROR
Forked Java VM exited abnormally. Please note the time in the report does not reflect the time until the VM exit.
junit.framework.AssertionFailedError: Forked Java VM exited abnormally. Please note the time in the report does not reflect the time until the VM exit.
at org.netbeans.core.execution.RunClassThread.run(RunClassThread.java:154)
is its netbean problem or some thing else? is any one help me thanks.
My code is like this-
public class BusinessDataTest {
String url = "http://local.com/info-66122628-andy-richards-house-painting-mesa";
Integer[] locId = {1354};
String customerDetail ="Business Name";
public BusinessDataTest() {
}
#BeforeClass
public static void setUpClass() throws Exception {
}
#AfterClass
public static void tearDownClass() throws Exception {
}
#Test
public void testGetFullDataOfBusiness() {
System.out.println("getFullDataOfBusiness");
BusinessData instance = new BusinessData();
Integer expResult = 4;
Integer result = instance.getFullDataOfBusiness(url, customerDetail);
System.out.println("Result : " + result);
assertEquals(expResult, result);
}
}
This looks like a bug in Netbeans - See Bug 204729

Categories

Resources