I am running nl4py (a python module for NetLogo) in Jupyter notebook. I am trying to get import a list from netlogo into python, but the import is in a Java format. However, when I try to convert the JavaObject to a python format using py4j I get an error of: JavaObject has no len(). Is there a better way to convert JavaObject in python? Thanks.
python 3.8, ipython 7.10.0, nl4py 0.5.0, jdk 15.0.2, Netlogo 6.0, MacOS Catalina 10.15.7
#start of code for nl4py
import nl4py
nl4py.startServer("/Applications/NetLogo 6.0/")
n = nl4py.NetLogoApp()
n.openModel('/Users/tracykuper/Desktop/Netlogo models/Mucin project/1_21_20/PA_metabolite_model_1_21.nlogo')
n.command("setup")
#run abm model for n number of times
#change patch variable under a specific turtle
for i in range(1):
n.command("repeat 10 [go]")
#A = np.array([1,2,3,4],[3,2,-1,-6])) #turtle number, metabolite diff.
#run simulation of metabolic network to get biomass and metabolite values
#change patch variable under a specific turtle
names = ["1", "2", "3"] #turtle names
patch_values = ["-0.5", "50", "-0.5"] #metabolite values
for i in range(len(names)):
x = ('ask turtle {} [ask patch-here [set succinate succinate + {}]]'.format(names[i],patch_values[i]))
n.command(x)
#set new bacteria mass values
values = ["5", "30", "5"] #biomass values
y = ('ask turtle {} [set m m + {}]'.format(names[i],values[i]))
n.command(y)
n.command("ask turtle {} [set color red]".format(names[i]))
import py4j
mass = n.report("mass-list")
print(mass)
self = n.report("self-list")
type(mass)
s = py4j.protocol.get_return_value(mass, object)
[[0.69], [0.8], [0.73], [0.71], [0.5], [0.51], [0.54], [0.82], [0.72], [0.88]]
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-31-0b04d0127b47> in <module>
11 #map(mass + mass,mass)
12
---> 13 s = py4j.protocol.get_return_value(mass, object)
~/opt/anaconda3/envs/netlogo4/lib/python3.6/site-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
319 (e.g., *hello* in `object1.hello()`). Optional.
320 """
--> 321 if is_error(answer)[0]:
322 if len(answer) > 1:
323 type = answer[1]
~/opt/anaconda3/envs/netlogo4/lib/python3.6/site-packages/py4j/protocol.py in is_error(answer)
372
373 def is_error(answer):
--> 374 if len(answer) == 0 or answer[0] != SUCCESS:
375 return (True, None)
376 else:
TypeError: object of type 'JavaObject' has no len()
Related
I'd like to run some PySpark script on JupyterLab, and create custom UDF from JAR packages. To do so I need to broadcast these JAR packages to executor nodes. This answer has showed the command line interface approach (invoking --jars option in spark-submit). But I'd like to know the SparkConf() approach. On my JupyterLab sc.version=3.3.0-SNAPSHOT.
I'm very new to Spark.. your help will be highly appreciated!
Code:
import findspark
findspark.init()
findspark.find()
import pyspark
from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession
import os
# ------------ create spark session ------------
app_name = 'PySpark_Example'
path = os.getcwd()
conf = SparkConf().setAppName(os.environ.get('JUPYTERHUB_USER').replace(" ", "") + "_" + app_name).setMaster(
'spark://spark-master-svc.spark:7077')
command = os.popen("hostname -i")
hostname = command.read().split("\n")[0]
command.close()
conf.set("spark.scheduler.mode","FAIR")
conf.set("spark.deployMode","client")
conf.set("spark.driver.host",hostname)
conf.set('spark.extraListeners','sparkmonitor.listener.JupyterSparkMonitorListener')
conf.set("spark.jars", "{path}/my_func.jar,{path}/javabuilder.jar".format(path=path))
conf.set("spark.executor.extraClassPath", "{path}/".format(path=path))
sc = pyspark.SparkContext(conf=conf)
spark = SparkSession(sc)
spark._jsc.addJar("{}/my_func.jar".format(path))
spark._jsc.addJar("{}/javabuilder.jar".format(path))
# ------------- create sample dataframe ---------
sdf = spark.createDataFrame(
[
(1, 2.),
(2, 3.),
(3, 5.),
],
["col1", "col2"]
)
sdf.createOrReplaceTempView("temp_table")
# -------------- create UDF ----------------------
create_udf_from_jar = "CREATE OR REPLACE FUNCTION my_func AS 'my_func.Class1' " + \
"USING JAR '{}/my_func.jar'".format(path)
spark.sql(create_udf_from_jar)
spark.sql("SHOW USER FUNCTIONS").show()
# -------------- test ----------------------------
spark.sql("SELECT my_func(col1) FROM temp_table").show()
Error:
---------------------------------------------------------------------------
Py4JJavaError Traceback (most recent call last)
~tmp/ipykernel_4398/644670379.py in <cell line: 1>()
----> 1 spark.sql("SELECT my_func(col1) FROM temp_table").show()
~opt/spark/python/pyspark/sql/session.py in sql(self, sqlQuery, **kwargs)
1033 sqlQuery = formatter.format(sqlQuery, **kwargs)
1034 try:
-> 1035 return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
1036 finally:
1037 if len(kwargs) > 0:
~opt/spark/python/lib/py4j-0.10.9.3-src.zip/py4j/java_gateway.py in __call__(self, *args)
1319
1320 answer = self.gateway_client.send_command(command)
-> 1321 return_value = get_return_value(
1322 answer, self.gateway_client, self.target_id, self.name)
1323
~opt/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
188 def deco(*a: Any, **kw: Any) -> Any:
189 try:
--> 190 return f(*a, **kw)
191 except Py4JJavaError as e:
192 converted = convert_exception(e.java_exception)
~opt/spark/python/lib/py4j-0.10.9.3-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
325 if answer[1] == REFERENCE_TYPE:
--> 326 raise Py4JJavaError(
327 "An error occurred while calling {0}{1}{2}.\n".
328 format(target_id, ".", name), value)
Py4JJavaError: An error occurred while calling o3813.sql.
: java.lang.NoClassDefFoundError: com/mathworks/toolbox/javabuilder/internal/MWComponentInstance
spark.jars is the one you're looking for (Doc)
I'm using Python to visualize a graph through a tool named wot using jupyter notebook. It utilizes Gephi, a java-based graph utility. I try to run function to return coordinate output files as below:
run_gephi(input_graph_file, output_coord_file, n_steps):
layout = 'fa'
import psutil
memory = int(0.5 * psutil.virtual_memory()[0] * 1e-9)
classpath = os.path.dirname(
pkg_resources.resource_filename('wot', 'commands/resources/graph_layout/GraphLayout.class')) + ':' + \
pkg_resources.resource_filename('wot', 'commands/resources/graph_layout/gephi-toolkit-0.9.2-all.jar')
subprocess.check_call(['java', '-Djava.awt.headless=true', '-Xmx{memory}g'.format(memory=memory), '-cp', classpath, \
'GraphLayout', input_graph_file, output_coord_file, layout, str(n_steps), str(os.cpu_count())])
Then it returns following error in my jupyter notebook:
CalledProcessError Traceback (most recent call last)
<ipython-input-18-5fc832689b87> in <module>
----> 1 df, adata = compute_force_layout(ds)
<ipython-input-7-6cb84b9e0fa0> in compute_force_layout(ds, n_neighbors, n_comps, neighbors_diff, n_steps)
24 writer.write("{u} {v} {w:.6g}\n".format(u=i + 1, v=j + 1, w=W[i, j]))
25
---> 26 run_gephi(input_graph_file, output_coord_file, n_steps)
27 # replace numbers with cids
28 df = pd.read_table(output_coord_file, header=0, index_col='id')
<ipython-input-16-28772d0d10cc> in run_gephi(input_graph_file, output_coord_file, n_steps)
7 pkg_resources.resource_filename('wot', 'commands/resources/graph_layout/gephi-toolkit-0.9.2-all.jar')
8 subprocess.check_call(['java', '-Djava.awt.headless=true', '-Xmx{memory}g'.format(memory=memory), '-cp', classpath, \
----> 9 'GraphLayout', input_graph_file, output_coord_file, layout, str(n_steps), str(os.cpu_count())])
~/anaconda3/lib/python3.7/subprocess.py in check_call(*popenargs, **kwargs)
339 if cmd is None:
340 cmd = popenargs[0]
--> 341 raise CalledProcessError(retcode, cmd)
342 return 0
343
CalledProcessError: Command '['java', '-Djava.awt.headless=true', '-Xmx25g', '-cp', '/home/iik/.local/lib/python3.7/site-packages/wot/commands/resources/graph_layout:/home/iik/.local/lib/python3.7/site-packages/wot/commands/resources/graph_layout/gephi-toolkit-0.9.2-all.jar', 'GraphLayout', '/tmp/gephiznxedn32.net', '/tmp/coordsd64x05ww.txt', 'fa', '10000', '8']' returned non-zero exit status 1.
and following message was found in terminal
Error: Could not find or load main class GraphLayout
I can found "GraphLayout.java" and "gephi-toolkit-0.9.2-all.jar" files in the path, so I really don't know why it can't be loaded.
Do you have any suggestions?
Add *
The class GraphLayout is not contained in Gephi but defined by GraphLayout.java.
I currently have a ipynb file (ipython notebook) that contains a linear regression code / model(im not entirely sure if it's a model) that I've created earlier.
How do i implement this model in an android application such that if I were to input a value of 'x' in a text box, it'll output in a textview the predicted value of 'y'. Function: Y = mx + b.
I've tried looking at different tutorials, but they were mostly not "step-by-step" guides, which made it really hard to understand, I'm a beginner at coding.
Here's my code for the model:
import tensorflow as tf
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
rng = np.random
from numpy import genfromtxt
from sklearn.datasets import load_boston
# Parameters
learning_rate = 0.0001
training_epochs = 1000
display_step = 50
n_samples = 222
X = tf.placeholder("float") # create symbolic variables
Y = tf.placeholder("float")
filename_queue = tf.train.string_input_producer(["C://Users//Shiina//battdata.csv"],shuffle=False)
reader = tf.TextLineReader() # skip_header_lines=1 if csv has headers
key, value = reader.read(filename_queue)
# Default values, in case of empty columns. Also specifies the type of the
# decoded result.
record_defaults = [[1.], [1.]]
height, soc= tf.decode_csv(
value, record_defaults=record_defaults)
features = tf.stack([height])
# Set model weights
W = tf.Variable(rng.randn(), name="weight")
b = tf.Variable(rng.randn(), name="bias")
# Construct a linear model
pred_soc = tf.add(tf.multiply(height, W), b) # XW + b <- y = mx + b where W is gradient, b is intercept
# Mean squared error
cost = tf.reduce_sum(tf.pow(pred_soc-soc, 2))/(2*n_samples)
# Gradient descent
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
# Initializing the variables
init = tf.global_variables_initializer()
with tf.Session() as sess:
# Start populating the filename queue.
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(coord=coord)
sess.run(init)
# Fit all training data
for epoch in range(training_epochs):
_, cost_value = sess.run([optimizer,cost])
#Display logs per epoch step
if (epoch+1) % display_step == 0:
c = sess.run(cost)
print( "Epoch:", '%04d' % (epoch+1), "cost=", "{:.9f}".format(c), \
"W=", sess.run(W), "b=", sess.run(b))
print("Optimization Finished!")
training_cost = sess.run(cost)
print ("Training cost=", training_cost, "W=", sess.run(W), "b=", sess.run(b), '\n')
#Plot data after completing training
train_X = []
train_Y = []
for i in range(n_samples): #Your input data size to loop through once
X, Y = sess.run([height, pred_soc]) # Call pred, to get the prediction with the updated weights
train_X.append(X)
train_Y.append(Y)
#Graphic display
plt.plot(train_X, train_Y, 'ro', label='Original data')
plt.ylabel("SoC")
plt.xlabel("Height")
plt.axis([0, 1, 0, 100])
plt.plot(train_X, train_Y, linewidth=2.0)
plt.legend()
plt.show()
coord.request_stop()
coord.join(threads)
I've downloaded some grib data files from here: ftp://data-portal.ecmwf.int/20160721000000/ (file type is .bin) and want to extract the data from this file in my Java application (I want to load the extracted data into a database later). I'm just trying with the file ftp://wmo:essential#data-portal.ecmwf.int/20160721000000/A_HWXE85ECEM210000_C_ECMF_20160721000000_24h_em_ws_850hPa_global_0p5deg_grib2.bin.
Therefore I've created a new Java project and added the two libraries grib-8.0.29.jar and netcdfAll-4.6.6.jar. Documentation for the grib API can be found here: http://www.unidata.ucar.edu/software/decoders/grib/javadoc/. I need to open the downloaded files to get the data. Retrieving some metadata via Grib2Dump seems to work (see below). Also the Grib2Input instance sais, that I have a valid GRIB file of version 2.
Here my working code for retrieving some metadata:
public static void main(String[] args) throws IOException, InterruptedException {
File srcDir = new File("C://test//");
File[] localFiles = srcDir.listFiles();
for (File tempFile : localFiles) {
RandomAccessFile raf = new RandomAccessFile(tempFile.getAbsolutePath(), "r");
System.out.println("======= Grib2GDSVariables ==========");
Grib2GDSVariables gdsVariables = new Grib2GDSVariables(raf.readBytes(raf.read()));
System.out.println("Gds key : " + gdsVariables.getGdsKey());
System.out.println("======= Grib2Input ==========");
Grib2Input input = new Grib2Input(raf);
System.out.println(Grib2Input.isValidFile(raf));
System.out.println("scan : " + input.scan(true, true));
System.out.println("getGDSs.size: " + input.getGDSs().size());
System.out.println("getProducts.size: " + input.getProducts().size());
System.out.println("getRecords.size: " + input.getRecords().size());
System.out.println("edition: " + input.getEdition());
System.out.println("======= Grib2Dump ==========");
Grib2Dump dump = new Grib2Dump();
dump.gribDump(new String[] {tempFile.getAbsolutePath()});
System.out.println("======= Grib2ExtractRawData ==========");
Grib2ExtractRawData extractRawData = new
Grib2ExtractRawData(raf); extractRawData.main(new String[] {tempFile.getAbsolutePath()});
}
System.out.println("finished");
}
This produces the following output:
======= Grib2GDSVariables ==========
Gds key : -1732955898
======= Grib2Input ==========
true
scan : true
getGDSs.size: 0
getProducts.size: 0
getRecords.size: 0
edition: 2
======= Grib2Dump ==========
--------------------------------------------------------------------
Header : GRIB2
Discipline : 0 Meteorological products
GRIB Edition : 2
GRIB length : 113296
Originating Center : 98 European Center for Medium-Range Weather Forecasts (RSMC)
Originating Sub-Center : 0
Significance of Reference Time : 1 Start of forecast
Reference Time : 2016-07-21T00:00:00Z
Product Status : 0 Operational products
Product Type : 1 Forecast products
Number of data points : 259920
Grid Name : 0 Latitude_Longitude
Grid Shape: 6 Earth spherical with radius of 6,371,229.0 m
Number of points along parallel: 720
Number of points along meridian: 361
Basic angle : 0
Subdivisions of basic angle: -9999
Latitude of first grid point : 90.0
Longitude of first grid point : 0.0
Resolution & Component flags : 48
Winds : True
Latitude of last grid point : -90.0
Longitude of last grid point : 359.5
i direction increment : 0.5
j direction increment : 0.5
Grid Units : degrees
Scanning mode : 0
Product Definition : 2 Derived forecast on all ensemble members at a point in time
Parameter Category : 2 Momentum
Parameter Name : 1 Wind_speed
Parameter Units : m s-1
Generating Process Type : 4 Ensemble Forecast
ForecastTime : 24
First Surface Type : 100 Isobaric surface
First Surface value : 85000.0
Second Surface Type : 255 Missing
Second Surface value : -9.999E-252
======= Grib2ExtractRawData ==========
finished
I tried around for two days now but couldn't get it to work! I can't obtain the content data (lat, lon, value) from the file...
Can someone give an example in Java?
You shouldn't use the GRIB classes in netCDF-java directly. Instead, use
NetcdfFile.open()
That will give you access through the CDM, giving you a straightforward interface with variables and attributes. There's a tutorial here: https://www.unidata.ucar.edu/software/thredds/current/netcdf-java/tutorial/NetcdfFile.html
There are lots of good examples out there on how to read Microsoft Excel files into R with the XLConnect package, but I can't find any examples of how to read in an Excel file directly from a URL. The reproducible example below returns a "FileNotFoundException (Java)". But, I know the file exists because I can pull it up directly by pasting the URL into a browser.
fname <- "https://www.misoenergy.org/Library/Repository/Market%20Reports/20140610_sr_nd_is.xls"
sheet <- c("Sheet1")
data <- readWorksheetFromFile(fname, sheet, header=TRUE, startRow=11, startCol=2, endCol=13)
Although, the URL is prefixed with "https:" it is a public file that does not require a username or password.
I have tried to download the file first using download.file(fname, destfile="test.xls") and got a message that says it was downloaded but when I try to open it in Excel to check to see if it was successful i get a Excel popup box that says "..found unreadable content in 'test.xls'.
Below are the specifics of my system:
Computer: 64-bit Dell running
Operating System: Windows 7 Professional
R version: R-3.1.0
Any assistance would be greatly appreciated.
You can use RCurl to download the file:
library(RCurl)
library(XLConnect)
appURL <- "https://www.misoenergy.org/Library/Repository/Market%20Reports/20140610_sr_nd_is.xls"
f = CFILE("exfile.xls", mode="wb")
curlPerform(url = appURL, writedata = f#ref, ssl.verifypeer = FALSE)
close(f)
out <- readWorksheetFromFile(file = "exfile.xls", sheet = "Sheet1", header = TRUE
, startRow = 11, startCol = 2, endCol = 15, endRow = 35)
> head(out)
Col1 EEI Col3 IESO MHEB Col6 PJM SOCO SWPP TVA WAUE Col12 Other Total
1 Hour 1 272 NA 768 1671 NA 148 200 -52 198 280 NA 700 4185
2 Hour 2 272 NA 769 1743 NA 598 200 -29 190 267 NA 706 4716
3 Hour 3 272 NA 769 1752 NA 598 200 -28 194 267 NA 710 4734
4 Hour 4 272 NA 769 1740 NA 598 200 -26 189 266 NA 714 4722
5 Hour 5 272 NA 769 1753 NA 554 200 -27 189 270 NA 713 4693
6 Hour 6 602 NA 769 1682 NA 218 200 -32 223 286 NA 714 4662
Two things:
Try using a different package--I know the gdata package's read.xls function has support for URLs
Try loading in a publicly-available xls file to make sure it's not an issue with the particular website.
For instance, you can try:
library("gdata")
site <- "http://www.econ.yale.edu/~shiller/data/chapt26.xls"
data <- read.xls(site, header=FALSE, skip=8)
head(data)
XLConnect does not support importing directly from URLs. You have to use e.g. download.file first to download the file to your local machine:
require(XLConnect)
tmp = tempfile(fileext = ".xls")
download.file(url = "http://www.econ.yale.edu/~shiller/data/chapt26.xls", destfile = tmp)
readWorksheetFromFile(file = tmp, sheet = "Data", header = FALSE, startRow = 9, endRow = 151)
or with your originally proposed URL:
require(XLConnect)
tmp = tempfile(fileext = ".xls")
download.file(url = "https://www.misoenergy.org/Library/Repository/Market%20Reports/20140610_sr_nd_is.xls", destfile = tmp, method = "curl")
readWorksheetFromFile(file = tmp, sheet = "Sheet1", header = TRUE, startRow = 11, startCol = 2, endCol = 13)
library(relenium)
library(XML)
library(RCurl)
firefox=firefoxClass$new()
url="https://www.misoenergy.org/Library/Repository/Market%20Reports/20140610_sr_nd_is.xls"
url=sprintf(url)
firefox$get(url)
This will open a Firefox instance within R and ask you to download the file, which you could then open in the next line of code. I don't know of any R utilities that will open an excel spreadsheet from HTTPS.
You could then set a delay while you're saving the file and then read the sheet from your downloads folder:
Sys.sleep(10)
sheet <- c("Sheet1")
data <- readWorksheetFromFile(path, sheet, header=TRUE, startRow=11, startCol=2, endCol=13)