Develop Reference

Why can't Nextflow handle this awk phrase?

Background:
Using a csv as input, I want to combine the first two columns into a new one (separated by an underscore) and add that new column to the end of a new csv.
Input:
column1,column2,column3
1,2,3
a,b,c
Desired output:
column1,column2,column3,column1_column2
1,2,3,1_2
a,b,c,a_b
The below awk phrase works from the command line:
awk 'BEGIN{FS=OFS=","} {print \$0, (NR>1 ? \$1"_"\$2 : "column1_column2")}' file.csv > full_template.csv
However, when placed within a nextflow script (below) it gives an error.
#!/usr/bin/env nextflow
params.input = '/file/location/here/file.csv'
process unique {
input:
path input from params.input
output:
path 'full_template.csv' into template
"""
awk 'BEGIN{FS=OFS=","} {print \$0, (NR>1 ? \$1"_"\$2 : "combined_header")}' $input > full_template.csv
"""
}
Here is the error:
N E X T F L O W ~ version 21.10.0
Launching `file.nf` [awesome_pike] - revision: 1b63d4b438
class groovyx.gpars.dataflow.expression.DataflowInvocationExpression cannot be cast to class java.nio.file.FileSystem (groovyx.gpars.dataflow.expression.Dclass groovyx.gpars.dataflow.expression.DataflowInvocationExpression cannot be cast to class java.nio.file.FileSystem (groovyx.gpars.dataflow.expression.DataflowInvocationExpression is in unnamed module of loader 'app'; java.nio.file.FileSystem is in module java.base of loader 'bootstrap')
I'm not sure what is causing this, and any help would be appreciated.
Thanks!
Edit:
Yes it seems this was not the source of the error (sorry!). I'm trying to use splitCsv on the resulting csv and this appears to be what's causing the error. Like so:
Channel
.fromPath(template)
.splitCsv(header:true, sep:',')
.map{ row -> tuple(row.column1, file(row.column2), file(row.column3)) }
.set { split }
I expect my issue is it's not acceptable to use .fromPath on a channel, but I can't figure out how else to do it.
Edit 2:
So this was a stupid mistake. I simply needed to add the .splitCsv option directly after the input line where I invoked the channel. Hardly elegant, but appears to be working great now.
process blah {
input:
what_you_want from template.splitCsv(header:true, sep:',').map{ row -> tuple(row.column1, file(row.column2), file(row.column3)) }

I was unable to reproduce the error you're seeing with your example code and Nextflow version. In fact, I get the expected output. This shouldn't be much of a surprise though, because you have correctly escaped the special dollar variables in your AWK command. The cause of the error is likely somewhere else in your code.
If escaping the special characters gets tedious, another way is to use a shell block instead:
It is an alternative to the Script definition with an important
difference, it uses the exclamation mark ! character as the variable
placeholder for Nextflow variables in place of the usual dollar
character.
The example becomes:
params.input_csv = '/file/location/here/file.csv'
input_csv = file( params.input_csv)
process unique {
input:
path input_csv
output:
path 'full_template.csv' into template
shell:
'''
awk 'BEGIN { FS=OFS="," } { print $0, (NR>1 ? $1 "_" $2 : "combined_header") }' \\
"!{input_csv}" > "full_template.csv"
'''
}
template.view { it.text }
Results:
$ nextflow run file.nf
N E X T F L O W ~ version 20.10.0
Launching `file.nf` [wise_hamilton] - revision: b71ff1eb03
executor > local (1)
[76/ddbb87] process > unique [100%] 1 of 1 ✔
column1,column2,column3,combined_header
1,2,3,1_2
a,b,c,a_b

Get specific java version with powershell

I have some issues with getting the java version out as a string.
In a batch script I have done it like this:
for /f tokens^=2-5^ delims^=.-_^" %%j in ('%EXTRACTPATH%\Java\jdk_extract\bin\java -fullversion 2^>^&1') do set "JAVAVER=%%j.%%k.%%l_%%m"
The output is: 1.8.0_121
Now I want to do this for PowerShell, but my output is: 1.8.0_12, I miss one "1" in the end Now I have tried it with trim and split but nothing gives me the right output can someone help me out?
This is what I've got so var with PowerShell
$javaVersion = (& $extractPath\Java\jdk_extract\bin\java.exe -fullversion 2>&1)
$javaVersion = "$javaVersion".Trim("java full version """).TrimEnd("-b13")
The full output is: java full version "1.8.0_121-b13"

TrimEnd() works a little different, than you might expect:
'1.8.0_191-b12'.TrimEnd('-b12')
results in: 1.8.0_19 and so does:
'1.8.0_191-b12'.TrimEnd('1-b2')
The reason is, that TrimEnd() removes a trailing set of characters, not a substring. So .TrimEnd('-b12') means: remove all occurrences of any character of the set '-b12' from the end of the string. And that includes the last '1' before the '-'.
A better solution in your case would be -replace:
'java full version "1.8.0_191-b12"' -replace 'java full version "(.+)-b\d+"','$1'

Use a regular expression for matching and extracting the version number:
$javaVersion = if (& java -fullversion 2>&1) -match '\d+\.\d+\.\d+_\d+') {
$matches[0]
}
or
$javaVersion = (& java -fullversion 2>&1 | Select-String '\d+\.\d+\.\d+_\d+').Matches[0].Groups[0].Value

Getting no such file error when trying to run Maven wrapper? [duplicate]

I am trying to format a variable in linux
str="Initial Value = 168"
echo "New Value=$(echo $str| cut -d '=' -f2);">>test.txt
I am expecting the following output
Value = 168;
But instead get
Value = 168 ^M;

Don't edit your bash script on DOS or Windows. You can run dos2unix on the bash script. The issue is that Windows uses "\r\n" as a line separator, Linux uses "\n". You can also manually remove the "\r" characters in an editor on Linux.

str="Initial Value = 168"
newstr="${str##* }"
echo "$newstr" # 168
pattern matching is the way to go.

Try this:
#! /bin/bash
str="Initial Value = 168"
awk '{print $2"="$4}' <<< $str > test.txt
Output:
cat test.txt
Value=168
I've got comment saying that it doesn't address ^M, I actually does:
echo -e 'Initial Value = 168 \r' | cat -A
Initial Value = 168 ^M$
After awk:
echo -e 'Initial Value = 168 \r' | awk '{print $2"="$4}' | cat -A
Value=168$

First off, always quote your variables.
#!/bin/bash
str="Initial Value = 168"
echo "New Value=$(echo "$str" | cut -d '=' -f2);"
For me, this results in the output:
New Value= 168;
If you're getting a carriage return between the digits and the semicolon, then something may be wrong with your echo, or perhaps your input data is not what you think it is. Perhaps you're editing your script on a Windows machine and copying it back, and your variable assignment is getting DOS-style newlines. From the information you've provided in your question, I can't tell.
At any rate I wouldn't do things this way. I'd use printf.
#!/bin/bash
str="Initial Value = 168"
value=${str##*=}
printf "New Value=%d;\n" "$value"
The output of printf is predictable, and it handily strips off gunk like whitespace when you don't want it.
Note the replacement of your cut. The functionality of bash built-ins is documented in the Bash man page under "Parameter Expansion", if you want to look it up. The replacement I've included here is not precisely the same functionality as what you've got in your question, but is functionally equivalent for the sample data you've provided.

shell script error : [: : integer expression expected

i am trying to trigger an email alert based on the java script output but i am getting an error like below in the shell script,
script.sh: line 22: [: : integer expression expected
Below is my shell script format,
out="$(java -jar /waitrose/scripts/OF/BOBIErrorAutomation/BOBIAutomation.jar
2>&1)"
if [ "$out" -gt 0 ]
then
mail -s "script did not completed successfully" $mailid_list
exit 0
fi
Below is the sample error returned by the java program :
The error was: com.ibm.db2.jcc.c.a.<init>(a.java:174) 174
com.ibm.db2.jcc.c.b.a(b.java:1745) 1745
com.ibm.db2.jcc.b.p.<init>(p.java:934) 934
Can anyone tell me how to capture the output from the java program and trigger the mail in shell ?

are you not comparing the output of your java program which in the case where it crashes, most likely going to be a string of some sort, against (-gt greater than) an integer? the comparator expects a whole number, not a crash message from java.
EDIT: ill elaborate, perhaps it would be better to test if $out is an integer first, if your jar only outputs integers if it runs correctly:
if ! [[ "$out" =~ ^[0-9]+$ ]]
then use this to trigger your mail, and else to normal functionality.

ksh command subsitution with quotes into array

I have a Java class with the following output:
"Roses are red" "Violets are blue" "Daisies are white"
I am trying to create a KornShell (ksh) script that executes this Java class, stores the result into an array, and then outputs
Roses are red
Violets are blue
Daises are white
This is my first attempt at the script:
1 #!/bin/ksh
2
3 set -A colors $(java Colors)
4
5 for i in "${colors[#]}"
6 do
7 echo "$i"
8 done
However, the output looks like this:
"Roses
are
Red"
"Violets
are
blue"
"Daisies
are
white"
When I throw:
echo set -A colors $(java Colors)
into the script to see what is being called, it returns:
set -A colors "Roses are Red" "Violets are blue" "Daisies are white"
If I replace that exact output with line 3 from above, I get the desired output:
Roses are red
Violets are blue
Daises are white
Why does the command substitution throw the whole thing off?
Below is the Java class if that helps:
public class Colors {
public static void main(String[] args) {
System.out.println("\"Roses are Red\" \"Violets are blue\" \"Daisies are white\"");
}
}

So I ended up making it work by throwing an 'eval' in front of the set command. It looked like this:
1 #!/bin/ksh
2
3 eval set -A colors $(java Colors)
4
5 for i in "${colors[#]}"
6 do
7 echo "$i"
8 done
I'm not sure why that worked exactly, but it did.

You may use quotes on $( ) :
set -A colors "$(java Colors)"
to avoid word splitting

pass the output of your java command as arguments to another shell script.
Then it will give $1=>"Roses are red",$2=>"Violets are blue", and so on.
you can get hold of all the input arguments using $#

This sort of thing is distressingly hard to do. This seems to work:
$ cat roses
echo '"Roses are red" "Violets are blue" "Daisies are white"'
$ IFS='
> '
$ set -A colors $(eval printf '%s\\n' $(./roses))
$ printf "%s\n" "${colors[#]}"
Roses are red
Violets are blue
Daisies are white
$
The script ./roses does the same job as your Java class. The variable IFS is the inter-field separator string; it is set to a newline (only). The set line creates an array colors. The value is what you get from executing and capturing eval printf '%s\\n' $(./roses). The second printf statement prints the elements of the array one per line.
Tested with ksh on Mac OS X 10.8.4.
An equivalent in bash is:
$ IFS='
> '
$ colors=($(eval printf '%s\\n' $(./roses)))
$ printf "%s\n" "${colors[#]}"
Roses are red
Violets are blue
Daisies are white
$
Depending a bit on the version of bash, there are probably other ways to do it too.
You should save and restore the value of $IFS before trampling it.
$ old_ifs="$IFS"
$ IFS='
> '
$ ...use the modified value...
$ IFS="$old_ifs"
$