Develop Reference

Why can't Nextflow handle this awk phrase?

Background:
Using a csv as input, I want to combine the first two columns into a new one (separated by an underscore) and add that new column to the end of a new csv.
Input:
column1,column2,column3
1,2,3
a,b,c
Desired output:
column1,column2,column3,column1_column2
1,2,3,1_2
a,b,c,a_b
The below awk phrase works from the command line:
awk 'BEGIN{FS=OFS=","} {print \$0, (NR>1 ? \$1"_"\$2 : "column1_column2")}' file.csv > full_template.csv
However, when placed within a nextflow script (below) it gives an error.
#!/usr/bin/env nextflow
params.input = '/file/location/here/file.csv'
process unique {
input:
path input from params.input
output:
path 'full_template.csv' into template
"""
awk 'BEGIN{FS=OFS=","} {print \$0, (NR>1 ? \$1"_"\$2 : "combined_header")}' $input > full_template.csv
"""
}
Here is the error:
N E X T F L O W ~ version 21.10.0
Launching `file.nf` [awesome_pike] - revision: 1b63d4b438
class groovyx.gpars.dataflow.expression.DataflowInvocationExpression cannot be cast to class java.nio.file.FileSystem (groovyx.gpars.dataflow.expression.Dclass groovyx.gpars.dataflow.expression.DataflowInvocationExpression cannot be cast to class java.nio.file.FileSystem (groovyx.gpars.dataflow.expression.DataflowInvocationExpression is in unnamed module of loader 'app'; java.nio.file.FileSystem is in module java.base of loader 'bootstrap')
I'm not sure what is causing this, and any help would be appreciated.
Thanks!
Edit:
Yes it seems this was not the source of the error (sorry!). I'm trying to use splitCsv on the resulting csv and this appears to be what's causing the error. Like so:
Channel
.fromPath(template)
.splitCsv(header:true, sep:',')
.map{ row -> tuple(row.column1, file(row.column2), file(row.column3)) }
.set { split }
I expect my issue is it's not acceptable to use .fromPath on a channel, but I can't figure out how else to do it.
Edit 2:
So this was a stupid mistake. I simply needed to add the .splitCsv option directly after the input line where I invoked the channel. Hardly elegant, but appears to be working great now.
process blah {
input:
what_you_want from template.splitCsv(header:true, sep:',').map{ row -> tuple(row.column1, file(row.column2), file(row.column3)) }

I was unable to reproduce the error you're seeing with your example code and Nextflow version. In fact, I get the expected output. This shouldn't be much of a surprise though, because you have correctly escaped the special dollar variables in your AWK command. The cause of the error is likely somewhere else in your code.
If escaping the special characters gets tedious, another way is to use a shell block instead:
It is an alternative to the Script definition with an important
difference, it uses the exclamation mark ! character as the variable
placeholder for Nextflow variables in place of the usual dollar
character.
The example becomes:
params.input_csv = '/file/location/here/file.csv'
input_csv = file( params.input_csv)
process unique {
input:
path input_csv
output:
path 'full_template.csv' into template
shell:
'''
awk 'BEGIN { FS=OFS="," } { print $0, (NR>1 ? $1 "_" $2 : "combined_header") }' \\
"!{input_csv}" > "full_template.csv"
'''
}
template.view { it.text }
Results:
$ nextflow run file.nf
N E X T F L O W ~ version 20.10.0
Launching `file.nf` [wise_hamilton] - revision: b71ff1eb03
executor > local (1)
[76/ddbb87] process > unique [100%] 1 of 1 ✔
column1,column2,column3,combined_header
1,2,3,1_2
a,b,c,a_b

"ValueError: Incompatible Language version 13. Must not be between 9 and 12" with Google Colab

I am trying to build a deep learning model with transformer model architecture. In that case when I am trying to cleaning the dataset following error occurred.
I am using Pytorch and google colab for that case & trying to clean Java methods and comment dataset.
Tested Code
import re
from fast_trees.core import FastParser
parser = FastParser('java')
def get_cmt_params(cmt: str) -> List[str]:
'''
Grabs the parameter identifier names from a JavaDoc comment
:param cmt: the comment to extract the parameter identifier names from
:returns: an array of the parameter identifier names found in the given comment
'''
params = re.findall('#param+\s+\w+', cmt)
param_names = []
for param in params:
param_names.append(param.split()[1])
return param_name
Occured Error
Downloading repo https://github.com/tree-sitter/tree-sitter-java to /usr/local/lib/python3.7/dist-packages/fast_trees/tree-sitter-java.
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-31-64f6fa6ed39b> in <module>()
3 from fast_trees.core import FastParser
4
----> 5 parser.set_language = FastParser('java')
6
7 def get_cmt_params(cmt: str) -> List[str]:
3 frames
/usr/local/lib/python3.7/dist-packages/fast_trees/core.py in FastParser(lang)
96 }
97
---> 98 return PARSERS[lang]()
/usr/local/lib/python3.7/dist-packages/fast_trees/core.py in __init__(self)
46
47 def __init__(self):
---> 48 super().__init__()
49
50 def get_method_parameters(self, mthd: str) -> List[str]:
/usr/local/lib/python3.7/dist-packages/fast_trees/core.py in __init__(self)
15 class BaseParser:
16 def __init__(self):
---> 17 self.build_parser()
18
19 def build_parser(self):
/usr/local/lib/python3.7/dist-packages/fast_trees/core.py in build_parser(self)
35 self.language = Language(build_dir, self.LANG)
36 self.parser = Parser()
---> 37 self.parser.set_language(self.language)
38
39 # Cell
ValueError: Incompatible Language version 13. Must not be between 9 and 12
an anybody help me to solve this issue?

fast_trees uses tree_sitter and according to tree_sitter repo it is an incomatibility issue. If you know the owner of fast_trees ask them to upgrade their tree_sitter version.
Or you can fork it and upgrade it yourself, but keep in mind it may not be backwards compatible if you take it upon yourself and it may not be just a simple new version install.

The fast-trees library uses the tree-sitter library and since they recommended using the 0.2.0 version of tree-sitter in order to use fast-trees. Although downgrade the tree-sitter to the 0.2.0 version will not be resolved your problem. I also tried out it by downgrading it.
So, without investing time to figure out the bug in tree-sitter it is better to move to another stable library that satisfies your requirements. So, as your requirement, you need to extract features from a given java code. So, you can use javalang library to extract features from a given java code.
javalang is a pure Python library for working with Java source code.
javalang provides a lexer and parser targeting Java 8. The
implementation is based on the Java language spec available at
http://docs.oracle.com/javase/specs/jls/se8/html/.
you can refer it from - https://pypi.org/project/javalang/0.13.0/
Since javalang is a pure library it will help go forward on your research without any bugs

getting syntax error from nifi ExecuteScript processor inspite of correct python code

I am getting below error inspite of correct python code don't know how to resolve this error. Any help is much appreciated
org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: SyntaxError: no viable alternative at input '*' in <script> at line number 35 at column number 26
python code
def get_match_list(regEx, line):
match = re.search(regEx, line)
print(match)
if match:
match_list = [*match.groups()] # this is the line exception is pointed
return match_list
else:
return []

It looks like jython use python 2.7 and as Unpacking Generalizations is a feature that introduced in python 3.5 you can not use this syntax in jython, so an alternative way to convert a tuple to a list is that use list ( match.groups) it works fine in older versions of python and current version of jython (2.7.2)

hprof profiler output doesn't include line numbers regardless of `lineno` value

I'm running
java -cp some:jars:out \
-agentlib:hprof=cpu=times,format=a,file=java.hprof.txt,lineno=y,doe=y com.foo.Benchmark \
< /dev/null
and in the output I get stack frames without line numbers
THREAD START (obj=50000150, id = 200002, name="HPROF gc_finish watcher", group="system")
THREAD START (obj=50000151, id = 200001, name="main", group="main")
THREAD START (obj=50000281, id = 200003, name="Thread-1", group="main")
THREAD END (id = 200003)
THREAD END (id = 200001)
THREAD START (obj=500002a5, id = 200004, name="DestroyJavaVM", group="main")
THREAD END (id = 200004)
TRACE 307081:
com.foo.Benchmark.methodName(Benchmark.java:Unknown line)
com.foo.Benchmark.anotherMethodName(Benchmark.java:Unknown line)
...
If I change lineno=y to lineno=n I still get Unknown line.
I compiled the classes with -g. My javac looks like
javac -g -Xlint -encoding UTF-8 -source 1.5 -d out -classpath ... src/main/com/foo/*.java
I checked the .class files to make sure they have line numbers:
javap -classpath out -c -l com.foo.Benchmark
shows plenty of things like
LineNumberTable:
line 1077: 0
line 1078: 8
line 1079: 14
line 1080: 21
line 1082: 23
line 1083: 31
line 1084: 43
Am I using some flag combination that prevents line number output?

I faced exactly the same problem, but compiling the source with -g helped. After compiling with -g, I see the line numbers like this (which I don't see otherwise, without -g option) -
LineNumberTable:
line 16: 0
line 17: 8
line 18: 12
line 19: 20
line 18: 29
line 21: 35
Now, if I run this -
java -cp "./build/classes" -agentlib:hprof=heap=sites,depth=20 org.sample.welcome.Main a b c
I do get the line numbers for the user-defined classes. Even though I am not sure what's going wrong in your case, these are my observations -
Using javac without -g: If I have -lineno (default y) set to y, I still don't see the line numbers for most of the classes except for the user-defined classes (Main in the above case). If I've set -lineno to n, then I won't see line numbers for any of the classes anyway.
Using javac with -g: If I have -lineno set to y, I can see the line numbers for all the classes (not sure what's going wrong in your case).
The only documentation I could find for HPROF doesn't say anything beyond this. I think one option would be to try with fewer optional arguments and see the results.
Note: I'm using JDK 1.6 in the above example

Failed to install WWW::HtmlUnit in Windows box

I'm trying to install WWW::HTMLUnit on Windows 7. There're step that I run through:
Install Inline::Java 0.53
Install WWW::HTMLUnit 0.15
At step 2, after nmake, I type nmake test to test module but it failed. Here's output:
C:\nmake test
Microsoft (R) Program Maintenance Utility Version 9.00.30729.01
Copyright (C) Microsoft Corporation. All rights reserved.
C:\Perl\bin\perl.exe "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib\lib', 'blib\arch')" t/*.t
t/00_basic...........
t/00_basic...........NOK 1/1# Failed test 'use WWW::HtmlUnit;'
# at t/00_basic.t line 9.
# Tried to use 'WWW::HtmlUnit'.
# Error: Class com.gargoylesoftware.htmlunit.WebClient not found at C:/Perl/site/lib/Inline/Java.pm line 619
# BEGIN failed--compilation aborted at (eval 4) line 2, <GEN7> line 4.
# Looks like you failed 1 test of 1.
t/00_basic...........dubious
Test returned status 1 (wstat 256, 0x100)
DIED. FAILED test 1
Failed 1/1 tests, 0.00% okay
t/01_hello...........Class com.gargoylesoftware.htmlunit.WebClient not found at C:/Perl/site/lib/Inline/Java.pm line 619
BEGIN failed--compilation aborted at t/01_hello.t line 4, <GEN7> line 4.
t/01_hello...........dubious
Test returned status 26 (wstat 6656, 0x1a00)
t/02_hello_sweet.....dubious
Test returned status 19 (wstat 4864, 0x1300)
t/03_clickhandler....Class com.gargoylesoftware.htmlunit.WebClient not found at C:/Perl/site/lib/Inline/Java.pm line 619
BEGIN failed--compilation aborted at t/03_clickhandler.t line 6, <GEN7> line 4.
t/03_clickhandler....dubious
Test returned status 29 (wstat 7424, 0x1d00)
DIED. FAILED tests 1-8
Failed 8/8 tests, 0.00% okay
Failed Test Stat Wstat Total Fail List of Failed
-------------------------------------------------------------------------------
t/00_basic.t 1 256 1 1 1
t/01_hello.t 26 6656 ?? ?? ??
t/02_hello_sweet.t 19 4864 ?? ?? ??
t/03_clickhandler.t 29 7424 8 16 1-8
Failed 4/4 test scripts. 9/9 subtests failed.
Files=4, Tests=9, 3 wallclock secs ( 0.00 cusr + 0.00 csys = 0.00 CPU)
Failed 4/4 test programs. 9/9 subtests failed.
NMAKE : fatal error U1077: 'C:\Perl\bin\perl.exe' : return code '0x1d'
Stop.
From above log, I could see that:
class Error: com.gargoylesoftware.htmlunit.WebClient could not be found.
I have no idea that I missed anything.
Any help would be appreciated.
Thanks.
Minh.

I found it.
There's different between path in Unix and Windows system. Unix uses ':' for a delimiter but Windows uses ';'. So what I've done is that open HTMLUnit.pm and change all of ':' to ';'.
With HTMLUnit version 0.15 I made changes at these lines below:
Line 78:
return join ';', map { "$jar_path/$_" } qw( # return join ':', map { "$jar_path/$_" } qw(
Line 127:
$custom_jars = join(';', #{$parameters{'jars'}}); # $custom_jars = join(':', #{$parameters{'jars'}});
Line 148:
CLASSPATH => collect_default_jars() . ";" . $custom_jars, # CLASSPATH => collect_default_jars() . ":" . $custom_jars,
And it works like a magic.

(it wouldn't let me comment on an existing answer)
I see your answer about ':' vs ';'. I'll try to include a fix in the next WWW::HtmlUnit release (I am the author of the perl bindings).