Develop Reference

SCons: compiling generated .java files doesn't check if they are part of a package

I'm pretty sure this is a bug, and I've already sent it in to the SCons user mailing list; I'm posting to ask for possible workarounds while I wait for it to be fixed, or clarification that this isn't a bug and how I'm supposed to work around it.
I've got a minimal case to reproduce the bug, though, with 2 files, SConstruct and src/Hello/World/HelloWorld.java.
I'm using SCons 3.0.0, with Python 2.7, on Windows 10.
src/Hello/World/HelloWorld.java:
package Hello.World;
public class HelloWorld {
public static void main(String[] args) {
System.out.println("Hello, World!");
}
}
SConstruct:
env = Environment( tools = ['javac'] )
# java
env.Append( JAVAVERSION = '1.8' )
env.Append( JAVA_HOME = 'C:/Program Files/Java/jdk1.8.0_152' )
env.AppendENVPath( 'PATH', 'C:/Program Files/Java/jdk1.8.0_152/bin' )
##### directories
src = Dir( 'src' )
build = Dir( 'build' )
classDir = build.Dir( 'classes' )
##### functions
srcJava = src.File( 'Hello/World/HelloWorld.java' )
copiedJava = build.File( 'Hello/World/HelloWorld.java' )
builtJava = env.Command( target = copiedJava, source = srcJava,
action = Copy( '$TARGET', '$SOURCE' ) )
classes = env.Java( target = classDir, source = builtJava )
HelloWorld.java gets copied into another build directory with env.Command() running Copy(), then the copied .java file gets put in env.Java(). The build works, and build/classes/Hello/World/HelloWorld.class gets created properly, but running SCons again yields an issue.
C:/example> scons --debug=explain -Q
scons: building `build\classes\HelloWorld.class' because it doesn't exist
javac -d build\classes -sourcepath build\Hello\World build\Hello\World\HelloWorld.java
The SCons node is pointing in the wrong location. The reason this happens is because of the Javac emitter.
On line 96/97 of SCons/Tool/javac.py:
if not f.is_derived():
pkg_dir, classes = parse_java_file(f.rfile().get_abspath(), version)
These two lines essentially mean that the java file only gets parsed for things like import statements if the source file is considered 'not derived.' I simulate this in my example by copying the file, but it's a huge issue in practice if you have any generated .java files in a project.
As for workarounds I've looked into so far, trying to access the is_derived() attribute of a file is very difficult, with a lot of read-only functions that essentially check if a node has builders. The Java builder doesn't seem to take .class files as target arguments, so I can't fix it by being more explicit.
Has anyone else run into this problem? Are there clever workarounds I haven't thought of? For now, I'm just changing that if statement in Tool/javac.py to 'if True:' until I can excise the package structure from this part of the project.
——— Update ———
As bdbaddog pointed out, my 'if True:' fix didn't work; as soon as the generated file was gone, SCons would try to parse the file before it had been built. Errors ensue.
Since then, I've created my own version of Tool/javac.py, which I called JavaFix.py, which resets the Java() builder with a slightly improved emitter. Most of the file is the same, so I'll just post the updated emitter and helper function:
def findComDir(entry):
"""Return a node representing the base of the source tree.
Assumes that the first module of the package is named 'com.'
"""
node = entry
while node.name != 'com':
node = node.dir
if ':' in node.name:
return entry
return node
def emit_java_classes(target, source, env):
"""Create and return lists of source java files
and their corresponding target class files.
"""
java_suffix = env.get('JAVASUFFIX', '.java')
class_suffix = env.get('JAVACLASSSUFFIX', '.class')
target[0].must_be_same(SCons.Node.FS.Dir)
classdir = target[0]
s = source[0].rentry().disambiguate()
if isinstance(s, SCons.Node.FS.File):
comDir = findComDir(s)
sourcedir = comDir.dir.rdir()
elif isinstance(s, SCons.Node.FS.Dir):
sourcedir = s.rdir()
else:
raise SCons.Errors.UserError("Java source must be File or Dir, not '%s'" % s.__class__)
slist = []
js = _my_normcase(java_suffix)
for entry in source:
entry = entry.rentry().disambiguate()
if isinstance(entry, SCons.Node.FS.File):
slist.append(entry)
elif isinstance(entry, SCons.Node.FS.Dir):
result = SCons.Util.OrderedDict()
dirnode = entry.rdir()
def find_java_files(arg, dirpath, filenames):
java_files = sorted([n for n in filenames
if _my_normcase(n).endswith(js)])
mydir = dirnode.Dir(dirpath)
java_paths = [mydir.File(f) for f in java_files]
for jp in java_paths:
arg[jp] = True
for dirpath, dirnames, filenames in os.walk(dirnode.get_abspath()):
find_java_files(result, dirpath, filenames)
entry.walk(find_java_files, result)
slist.extend(list(result.keys()))
else:
raise SCons.Errors.UserError("Java source must be File or Dir, not '%s'" % entry.__class__)
version = env.get('JAVAVERSION', '1.4')
full_tlist = []
for f in slist:
tlist = []
source_file_based = True
pkg_dir = None
if not f.is_derived():
pkg_dir, classes = parse_java_file(f.rfile().get_abspath(), version)
if classes:
source_file_based = False
if pkg_dir:
d = target[0].Dir(pkg_dir)
p = pkg_dir + os.sep
else:
d = target[0]
p = ''
for c in classes:
t = d.File(c + class_suffix)
t.attributes.java_classdir = classdir
t.attributes.java_sourcedir = sourcedir
t.attributes.java_classname = classname(p + c)
tlist.append(t)
if source_file_based:
base = f.name[:-len(java_suffix)]
if pkg_dir:
t = target[0].Dir(pkg_dir).File(base + class_suffix)
else:
pkg_dir = sourcedir.rel_path( f.dir ).replace('.', os.sep)
if pkg_dir == os.sep:
pkg_dir = None
if pkg_dir:
p = pkg_dir + os.sep
else:
p = ''
t = target[0].File(p + base + class_suffix)
t.attributes.java_classdir = classdir
t.attributes.java_sourcedir = sourcedir
t.attributes.java_classname = classname(p + base)
tlist.append(t)
for t in tlist:
t.set_specific_source([f])
full_tlist.extend(tlist)
return full_tlist, slist
The main changes come after the line if source_file_based:. This is just a first draft as of right now, but it works. I'll just post the comment I put at the top of JavaFix.py to explain:
The original Tools/javac.py couldn't really handle generated
.java files, so this slightly enhanced copy provides a fix.
Simply import the module and call JavaFix.generate(env),
and the Java builder will be remade with an improved emitter
that is better at dealing with generated files.
--Improvements--
When calling env.Java( target, source ), source can now handle
generated files.
--If source is a File node, the output directory structure is
deduced by copying the directory structure source is in,
starting from 'com.' If 'com' is not in the directory path,
SCons will not correctly locate the output node
--If source is a Dir node, SCons will correctly build all .java
files in the directory tree so long as source is the directory
containing the start of the package structure
(e.g. source contains com, and the package structure is com.Foo)
--Functionality concerning non-generated .java files is preserved
There are still a few downsides; SCons doesn't pick up on generated inner classes (.class files with a $ in them), and it relies heavily on the directory structure to figure out where the output files go, but at least it's not getting confused and trying to rebuild files it doesn't need to all the time.
I hope this helps bdbaddog or whoever working at SCons will take this bug up to fix. I admit some of my code is a little messy, but it should be understandable.
-Matthew

Generation of C# files with Google Protocol Fails

I am working on a project that uses Java, C# and also C++ applications. To communicate between them I am trying to use Google protocol buffer. I am using following .proto file, which was taken from examples:
package tutorial;
message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
required string number = 1;
optional PhoneType type = 2 [default = HOME];
}
repeated PhoneNumber phone = 4;
}
message AddressBook {
repeated Person person = 1;
}
I am referring to following tutorial:
https://developers.google.com/protocol-buffers/docs/csharptutorial
Tutorials for other languages are also there.
I tried following command line arguments for each language:
Java:
C:\ProtoBuf\protoc -I=C:\trash --java_out=C:\trash C:\trash/addressbook.proto
C++:
C:\ProtoBuf\protoc -I=C:\trash --cpp_out=C:\trash C:\trash/addressbook.proto
C#:
C:\ProtoBuf\protoc -I=C:\trash --csharp_out=C:\trash C:\trash/addressbook.proto
Java and C++ compilations work properly even with some warning in case of Java. But I get following output with C# :
--csharp_out: protoc-gen-csharp: The system cannot find the file specified.
I am using this compiler: https://github.com/google/protobuf/releases/download/v2.6.1/protoc-2.6.1-win32.zip
What am I doing wrong here? do I need any other files for C# compilation?

You are trying to generate C# files using the old version of the protoc
protoc-2.6.1-win32.zip
C# code generator for both proto2 and proto3 was introduced only in Version 3.0.0-alpha-3
Introduced two new language implementations (Objective-C, C#) to
proto3.
So, download protoc Version 3.0.0-alpha-3, install it and call:
protoc -I=$SRC_DIR --csharp_out=$DST_DIR $SRC_DIR/your.proto
Beware that started from Version 3.0.0-beta-1 C# code generator only supports generating proto3:
Proto3 semantics supported; proto2 files are prohibited for C# codegen

I know how to gen proto files in c#
open visual studio, open nuget command line, type : Install-Package Google.ProtocolBuffers , link : Google.ProtocolBuffers 2.4.1.555
find Package/Google.ProtocolBuffers.2.4.1.555/tools/ProtoGen.exe
use command line, type : ProtoGen.exe addressbook.proto -output_directory=C:\trash
I write a python script to gen proto files, gen.py
import os, subprocess, threading
def main():
with open("conf.txt") as file:
exe = os.path.join(os.getcwd(), "..\\Package\\Google.ProtocolBuffers.2.4.1.555\\tools\\ProtoGen.exe")
out = "-output_directory=%s" % (os.path.join(os.getcwd(), "..\\Common\\libs\\protos"))
def gen(proto):
subprocess.check_call([exe, os.path.join("protos", proto), out])
list = []
for proto in file.read().split(','):
t = threading.Thread(target = gen, args = (proto, ))
t.start()
list.append(t)
for t in list:
t.join()
if __name__ == '__main__':
main()
conf.txt
base.proto,test.proto,addressbook.proto

I solved all my problems using / (slash) intead of \ (anti-slash) in all the path.
protoc.exe -I=D:/dev/xyz/projects --csharp_out=d:/temp D:/dev/xyz/projects/messages.proto

The type initializer for 'weka.core.WekaPackageManager' threw an exception - weka c#

I am using weka api through c#. I have converted weka jar file to c# dll by using ikvm. Then I have added the converted dll (wekacsharp.dll) in my reference.
I have also added ikvm.gnu.classpath.dll, IKVM.OpenJDK.Core.dll, IKVM.OpenJDK.Util.dll, IKVM.OpenJDK.Text.dll, IKVM.OpenJDK.Core in my reference.
I am trying to use j48 algorithm but i am getting the error. Screen shot of code error is attached. Kindly check it and suggest me something to fix it.
Code:
public static void classifyTest()
{
try
{
weka.core.Instances insts = new weka.core.Instances(new java.io.FileReader("iris.arff"));
insts.setClassIndex(insts.numAttributes() - 1);
weka.classifiers.Classifier cl = new weka.classifiers.trees.J48();
//Console.WriteLine("Performing " + percentSplit + "% split evaluation.");
//randomize the order of the instances in the dataset.
// weka.filters.Filter myRandom = new weka.filters.unsupervised.instance.Randomize();
// myRandom.setInputFormat(insts);
// insts = weka.filters.Filter.useFilter(insts, myRandom);
int trainSize = insts.numInstances() * percentSplit / 100;
int testSize = insts.numInstances() - trainSize;
weka.core.Instances train = new weka.core.Instances(insts, 0, trainSize);
cl.buildClassifier(train);
int numCorrect = 0;
for (int i = trainSize; i < insts.numInstances(); i++)
{
weka.core.Instance currentInst = insts.instance(i);
double predictedClass = cl.classifyInstance(currentInst);
if (predictedClass == insts.instance(i).classValue())
numCorrect++;
}
//java.io.Console.WriteLine(numCorrect + " out of " + testSize + " correct (" +(double)((double)numCorrect / (double)testSize * 100.0) + "%)");
}
catch (java.lang.Exception ex)
{
ex.printStackTrace();
}
}

Make sure that you did not skip the conversion of the weka.jar to weka.dll using Ikvm before adding it as a reference to your C# project.
Conversion from Java to a .NET dll
With that out of the way, the first thing you will want to do is to convert the Weka .jar file into a .NET dll. To do this, we will use ikvmc, which is the IKVM static compiler.
On the console, go to the directory which contains weka.jar, and type:
ikvmc -target:library weka.jar
The -target:library call causes ikvmc to create a .dll library instead of an executable.
Note that the IKVM tutorial tells you that you should add
-reference:/usr/lib/IKVM.GNU.Classpath.dll
(or appropriate path) to the above command, it tells IKVM where to find the GNU Classpath library. However, IKVM.GNU.Classpath.dll is no longer included in the download package, and is from very old versions of IKVM. When Sun open sourced Java, it got replaced by the IKVM.OpenJDK.*.dll files.
You should now have a file called "weka.dll", which is a .NET version of the entire weka API. That's exactly what we want!
Use the dll in a .NET application
To try it out, let's use a small C# program that I wrote. The program simply runs the J48 classifier on the Iris dataset with a 66% test/data split, and prints out the correctness percentage. It also uses a few Java classes, and is already about 95% legal Java code.

Android protobuf nano usage

I am trying to generate java files from below proto file using protobuf nano. I got some basic instruction on how to proceed in this SO thread.
I have this proto file, personal.proto:
package tutorial;
option java_package = "com.example.tutorial";
option java_outer_classname = "AddressBookProtos";
message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
required string number = 1;
optional PhoneType type = 2 [default = HOME];
}
repeated PhoneNumber phone = 4;
}
message AddressBook {
repeated Person person = 1;
}
I tried to follow the instruction from here, more specifically NANO version:
Downloaded protobuf-2.5.0.zip and compiler protoc-2.5.0-win32.zip from here.
Unzipped protobuf-2.5.0.zip to a folder and in there in src subfolder I unzipped protoc.exe.
Changed to java folder and in there issued: mvn clean package -P nano. That command ran fine and in target folder I have protobuf-java-2.5.0.jar
From here I am not sure how to proceed since in the initial documentation I have this statement:
- Link with the generated jar file
<protobuf-root>java/target/protobuf-java-2.3.0-nano.jar.
I am not sure what that means, how to link? Is there some parameter for protoc.exe that specifies the jar file to use?
I tried to issue this command: protoc --javanano_out=enum_style=java --java_out=generated personal.proto
but I get this error: --javanano_out: protoc-gen-javanano: The system cannot find the file specified.
The question would be: what am I missing/doing wrong above? I am trying to generate java files from above proto file.

I think this protoc is not compiled with javanano support.
The pre-compiled windows version 2.5.0 does not include nano support, take a look at the source code, in the "src\google\protobuf\compiler" path, includes the java generator but not the javanano generator. The latest source code at google repositories includes javanano.
You can download the latest source code and try to compile it using MinGW and msys or CygWin, take a look at this post How to build google protocol buffers in Windows for mingw?
(I will post details for the building process later)
UPDATE:
The final command line after building protoc.exe
For one proto file
protoc --javanano_out=store_unknown_fields=true:target/generated-sources personal.proto, target/generated-sources
For multiple proto files
protoc --javanano_out=store_unknown_fields=true:target/generated-sources --proto_path=inputpath input/*.proto
EDIT Nano generator replaces enum members with public static final int fields. This is a problem if a class has an optional enum member because that member will be compiled to a primitive int value and will take the default value of zero, which will be the first element from enum. To distinguish the cases when an enum value was not set, one can take advantage of optional_field_style parameter that will generate java.lang.Integer instead of a primitive int. When the proto is parsed, the caller can check if the value is null before using the value. Null means the value was not set.
The above call script can become:
protoc --javanano_out=store_unknown_fields=true,optional_field_style=reftypes:target/generated-sources --proto_path=input input/*.proto

Alloy API resulting in java.lang.UnsatisfiedLinkError

I'm currently using the Alloy Analyzer API to build a program, and getting some peculiar behavior. Specifically, if I open a file and parse it (using CompUtil.parseEverything), then make a new Command and call TranslateAlloyToKodkod.execute_command on the parsed file and newly created command using MiniSat with UNSAT core, it runs fine. However, later in execution, my program parses a second input file (also using CompUtil.parseEverything), gets another world, makes a new command, and then I try to call TranslateAlloyToKodkod.execute_command again, it throws the following error:
ERROR: class edu.mit.csail.sdg.alloy4.ErrorFatal: The required JNI library cannot be found:
java.lang.UnsatisfiedLinkError: no minisatproverx5 in java.library.path
edu.mit.csail.sdg.alloy4compiler.translator.TranslateAlloyToKodkod.execute_command(TranslateAlloyToKodkod.java:390)
Does anyone have any idea why this is thrown the second time, but not the first?
To summarize, I have something similar to the following:
Module someWorld = CompUtil.parseEverything_fromFile(rep, null, "someFile.als");
//For the following, "sig" is a sig in someWorld.getAllReachableSigs();
Command command = sig.not();
A4Options options = new A4Options();
options.solver = A4Options.SatSolver.MiniSatProverJNI;
A4Solution ans =
TranslateAlloyToKodkod.execute_command(rep, someWorld, command, options);
//No thrown error
Module someOtherWorld = CompUtil.parseEverything_fromFile(rep, null, "someOtherFile.als");
//For the following, "sig" is a sig in someOtherWorld.getAllReachableSigs();
Command commandTwo = sig.not();
A4Solution ansTwo =
TranslateAlloyToKodkod.execute_command(rep, someOtherWorld, commandTwo, options);
//Thrown error above. Why?

I tried to reproduce this behavior, but I couldn't. If I don't add MiniSat binaries to the LD_LIBRARY_PATH environment variable, I get the exception you mentioned the very first time I invoke execute_command. After configuring LD_LIBRARY_PATH, the exception doesn't happen.
To configure LD_LIBRARY_PATH:
(1) if using Eclipse, you can right-click on one of your source folders, choose Build Path -> Configure Build Path, then on the "Source" tab make sure that "Native library location" points to a folder in which MiniSat binaries reside.
(2) if running from the shell, just add the path to a folder with MiniSat binaries to LD_LIBRARY_PATH, e.g., something like export LD_LIBRARY_PATH=alloy/extra/x86-linux:$LD_LIBRARY_PATH.
Here is the exact code that I was running, and everything worked
public static void main(String[] args) throws Exception {
A4Reporter rep = new A4Reporter();
A4Options options = new A4Options();
options.solver = A4Options.SatSolver.MiniSatProverJNI;
Module someWorld = CompUtil.parseEverything_fromFile(rep, null, "someFile.als");
Command command = someWorld.getAllCommands().get(0);
A4Solution ans = TranslateAlloyToKodkod.execute_command(rep, someWorld.getAllReachableSigs(), command, options);
System.out.println(ans);
Module someOtherWorld = CompUtil.parseEverything_fromFile(rep, null, "someOtherFile.als");
Command commandTwo = someOtherWorld.getAllCommands().get(0);
A4Solution ansTwo = TranslateAlloyToKodkod.execute_command(rep, someOtherWorld.getAllReachableSigs(), commandTwo, options);
System.out.println(ansTwo);
}
with "someFile.als" being
sig A {}
run { some A } for 4
and "someOtherFile.als"
sig A {}
run { no A } for 4

I use alloy4.2.jar as a library in my eclipse plugin project.
A4Reporter rep = new A4Reporter();
Module world = CompUtil.parseEverything_fromFile(rep, null, "civi.als");
A4Options options = new A4Options();
options.solver = A4Options.SatSolver.SAT4J;
options.skolemDepth = 1;
When I use SAT4J, the default solver, the problem mentioned here will not show up. But another exception comes out. The reason is that my civi.als file need Integer model, which located in alloy4.2.jar under the folder /models/util/. But when I run the application, it tries to find the file util/Integer.als directly. That causes the exception. Is it possible to fix that problem?
Besides, I also tried to put the alloy4.2.jar in eclipse plugin project and run my application as an eclipse application (running my application as a plugin). With the default solver, the application has no problem at all. But when I switch to MiniSatProverJNI, the problem mentioned here comes out (I have set the alloy4.2.jar as classpath).