I am having a problem when trying to use the Z3Str3 from z3-4.8.9-x64-ubuntu-16.04 notably if I substitute the com.microsoft.z3.jar to the one in z3-4.8.8-x64-ubuntu-16.04 I no longer have that issue. The problem is that the Z3 process never comes back with a result, despite the simplicity of the query. I noticed though that it returns the valid answer when I kill my program. I am not noticing that behavior when I am trying to run the same query on the executable, so I am guessing there is something about using the jar file that I might need to tweak one way or the other.
Here is my code. I am using Ubuntu 16.04 LTS, and IntelliJ version ultimate 2020.3.
Many thanks!
import com.microsoft.z3.*;
public class Z3String3Processor_reduced {
public static void main(String[] args) {
StringBuilder currentQuery = new StringBuilder("\n" +
"(declare-const string0 String)\n" +
"(assert (= (str.indexof string0 \"a\" 1) 6))\n" +
"(check-sat)\n" +
"(get-model)\n" +
"\n");
Context context1 = new Context();
Solver solver1 = context1.mkSolver();
Params params = context1.mkParams();
params.add("smt.string_solver", "z3str3");
solver1.setParameters(params);
StringBuilder finalQuery = new StringBuilder(currentQuery.toString());
// attempt to parse the query, if successful continue with checking satisfiability
try {
// throws z3 exception if malformed or unknown constant/operation
BoolExpr[] assertions = context1.parseSMTLIB2String(finalQuery.toString(), null, null, null, null);
solver1.add(assertions);
// check sat, if so we can go ahead and get the model....
if (solver1.check() == Status.SATISFIABLE) {
System.out.println("sat");
} else
System.out.println("not sat");
context1.close();
} catch (Z3Exception e) {
System.out.println("Z3 exception: " + e.getMessage());
}
}
}
I don't think this has anything to do with Java. Let's extract your query and put it in a file named a.smt2:
$ cat a.smt2
(declare-const string0 String)
(assert (= (str.indexof string0 "a" 1) 6))
(check-sat)
(get-model)
Now, if I run:
$ z3 a.smt2
sat
(
(define-fun string0 () String
"FBCADEaGaa")
)
That's good. But if I run:
$ z3 smt.string_solver=z3str3 a.smt2
... does not terminate ..
So, bottom line, your query (as simple as it looks), gives hard time to the z3str3 solver.
I see that you already reported this as a bug at https://github.com/Z3Prover/z3/issues/5673
Given that the default string-solver can handle the query just fine, why not just use that one? If you have to use z3str3 for some other reason, then you've found a case where it doesn't handle this query well; I'm not sure how inclined the z3 folks will be to fix this given the query is handled by the default solver rather quickly. Please report what you find out!
Related
I have a setup where I execute jython scripts from a Java application. The java application feed the jython script with variables, coming from the command line, so that a user can write the following code in it's jython script:
print("Hello, %s" % foobar)
And will call the java program with this:
$ java -jar myengine.jar script.py --foobar=baz
Hello, baz
My java application parse the command-line, and create a variable of that name with the given value to give to the jython scripting environment to consume. All is well so far.
My issue is that when the user does not provide the foobar command-line parameter, I'd like to be able to easily provide a fallback in my script. For now, the user needs to write that sort of code to handle the situation where the foobar parameter is missing from the command-line:
try: foobar
except NameError: foobar = "some default value"
But this is cumbersome, especially if the number of parameters is growing. Is there a way to handle that better from the script user point of view?
I was thinking of catching the jython NameError in the Java code, initializing the variable causing the exception to a default value if the variable causing the exception "looks like" a parameter (adding a naming convention is OK), and restarting where the exception occurred. Alternatively, I can require the script user to write code such as this:
parameter(foobar, "some default value")
Or something equivalent.
Well, this is one ugly workaround I found so far. Be careful, as this will call the script in loop many times, and is O(n^2).
private void callScriptLoop(String scriptfile) {
PythonInterpreter pi = new PythonInterpreter();
pi.set("env", someEnv);
int nloop = 0;
boolean shouldRestart;
do {
shouldRestart = false;
try {
pi.execfile(scriptfile);
} catch (Throwable e) {
if (e instanceof PyException) {
PyException pe = (PyException) e;
String typ = pe.type.toString();
String val = pe.value.toString();
Matcher m = Pattern.compile("^name '(.*)' is not defined")
.matcher(val);
if (typ.equals("<type 'exceptions.NameError'>")
&& m.find()) {
String varname = m.group(1);
pi.set(varname, Py.None);
System.out.println(
"Initializing missing parameter '"
+ varname + "' to default value (None).");
shouldRestart = true;
nloop++;
if (nloop > 100)
throw new RuntimeException(
"NameError handler infinite loop detected: bailing-out.");
}
}
if (!shouldRestart)
throw e;
}
} while (shouldRestart);
}
I need to raise a warning during one of my scenario but i don't stop to have this error appearing : "Cannot infer type arguments for Result.Warning<>"
I actually tried to raise the Warning the same way i was raising Failure until now :
new Result.Warning<>(targetKey, Messages.format(TaroMessages.WARNING_RESOURCES_VALUE_DIFFERENCE_AFTER_REAFFECTATION, existing_value, new_value), true, oscarAccesClientPage.getCallBack());
The custom step i am using it inside is the following : I'm trying to go over a list of Element and checking that the existing value of them is the same or not as the one saved before.
protected void checkXyResourcesValue(Integer xyIterator, List<WebElement> elements, String keyParameter) throws TechnicalException, FailureException {
try {
Integer resIterator = 1;
for(WebElement element : elements) {
String targetKey = "XY" + xyIterator + "RES" + resIterator + keyParameter;
String new_value = element.getAttribute(VALUE) != null ? element.getAttribute(VALUE) : element.getText();
String existing_value = Context.getValue(targetKey) != null ? Context.getValue(targetKey) : targetKey;
if (new_value != existing_value) {
new Result.Warning<>(targetKey, Messages.format(TaroMessages.WARNING_RESOURCES_VALUE_DIFFERENCE_AFTER_REAFFECTATION, existing_value, new_value), true, oscarAccesClientPage.getCallBack());
}
resIterator++;
}
} catch (Exception e) {
new Result.Failure<>(e.getMessage(), Messages.format(TaroMessages.FAIL_MESSAGE_ACCES_CLIENT_XY_CHECK_RESOURCES_VALUE, keyParameter, xyIterator), true, oscarAccesClientPage.getCallBack());
}
}
For the method to check and saved value I actually inspired myself for the piece of code from NoraUI to save a value on Context or read it from.
I'm using Eclipse Luna 4.4.2 and i try to compile using JDK1.8.0_131.
It may be more related to me not knowing how this work in Java than a real problem so thank you in advance for your help or insights. Don't hesitate to ask if you need more information on the piece of code or the context.
new Result.Warning<>(targetKey, Messages.format(TaroMessages.WARNING_RESOURCES_VALUE_DIFFERENCE_AFTER_REAFFECTATION, existing_value, new_value), true, 0);
use 0 if you do not use any Model (data serialized) or use id of your Object in the serial.
I'm having a hard time getting HBase's FuzzyRowFilter to work.
I have the following test table:
hbase(main):014:0> scan 'test'
ROW COLUMN+CELL
row-01 column=colfam1:col1, timestamp=1481193793338, value=value1
row-02 column=colfam1:col1, timestamp=1481193799186, value=value2
row-03 column=colfam1:col1, timestamp=1481193803941, value=value3
row-04 column=colfam1:col1, timestamp=1481193808209, value=value4
row-05 column=colfam1:col1, timestamp=1481193812737, value=value5
5 row(s) in 0.0200 seconds
Here is my Java code (I started with Scala, but the results are the same - none):
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.zookeeper.quorum", "localhost:2182");
conf.set("hbase.master", "localhost:60000");
conf.set("hbase.rootdir", "/hbase");
try {
Scan scan = new Scan();
scan.setCaching(5);
byte[] rowKeys = Bytes.toBytesBinary("???-01");
byte[] fuzzyInfo = {0x01,0x01,0x01,0x00,0x00,0x00};
FuzzyRowFilter fuzzyFilter = new FuzzyRowFilter(
Arrays.asList(
new Pair<byte[], byte[]>(
rowKeys,
fuzzyInfo)));
System.out.println("### fuzzyFilter: " + fuzzyFilter.toString());
scan.addFamily(Bytes.toBytesBinary("colfam1"));
scan.setStartRow(Bytes.toBytesBinary("row-01"));
scan.setStopRow(Bytes.toBytesBinary("row-05"));
scan.setFilter(fuzzyFilter);
Connection conn = ConnectionFactory.createConnection(conf);
Table table = conn.getTable(TableName.valueOf("test"));
ResultScanner results = table.getScanner(scan);
int count = 0;
int limit = 100;
for ( Result r : results ) {
System.out.println("" + r.toString());
if (count++ >= limit) break;
}
} catch (Exception e) {
e.printStackTrace();
}
I simply do not get any results back from the server. If I comment out the line scan.setFilter(fuzzyFilter);, I get the exepcted results:
keyvalues={row-01/colfam1:col1/1481193793338/Put/vlen=6/seqid=0}
keyvalues={row-02/colfam1:col1/1481193799186/Put/vlen=6/seqid=0}
keyvalues={row-03/colfam1:col1/1481193803941/Put/vlen=6/seqid=0}
keyvalues={row-04/colfam1:col1/1481193808209/Put/vlen=6/seqid=0}
Am I doing something wrong? Is there a bug in HBase (version 1.2.2)? I am using the version installed through Homebrew on latest Mac OS Sierra.
Update
On a Cloudera Hadoop cluster running CDH 5.7 with HBase 1.2.0-cdh5.7.0, I get the desired output for rowkey row-01. The error must somehow be related to my local setup.
Solution
Indeed, the problem was that HBase server installation and client JAR versions did not match. In my case, I was using the artifacts
hbase-common
hbase-client
hbase-server
with version 1.2.0-cdh5.7.0 instead of 1.2.2.
My mistake was assuming that minor version differences would not have a large impact, but apparently Cloudera has applied some major changes in their versions with respect to the official code base. Changing to the official version 1.2.2 made the FuzzyRowFilter work as expected.
It should print only rowkey of row-01 as can be perceived from the filter condition.
There is no such bug and it will work as expected as I have been using same for some time now.
Check your configurations,dependencies,etc.
Due to versioning,many times libraries and their clients becom incompatible.
Lets take a simple example:
class ServerVersionA {
public static void getData() {
return DataOject(data with headerVersionA);
}
}
class ClientVersionB {
public void showData() {
DataObject dataObject = makeRequest(params);
//Check whether data recieved is of version B after veryfying header boolean status=validate(dataObject);
if (status) {
doIO(dataObject);
}
}
}
In this case,if the header does not match,client does simply sit idle.
These kind of issues are mostly taken care of but sometimes they creep in.
If we look at the sources of installation and client version,we can find out why data is not being returned and no exception is propagated.
I am having trouble fixing a Log Forging issue in Fortify. The issue, "writes unvalidated user input to the log", is being raised from both of the logging calls in the getLongFromTimestamp() method.
public long getLongFromTimestamp(final String value) {
LOGGER.info("getLongFromTimestamp(" + cleanLogString(value) + ")");
long longVal = 0;
Date tempDate = null;
try {
tempDate = new SimpleDateFormat(FORMAT_YYYYMMDDHHMMSS, Locale.US).parse(value);
} catch (ParseException e) {
LOGGER.warn("Failed to convert to Date: " + cleanLogString(value) + " Exception: " + cleanLogString(e.getMessage()));
throw new Exception(e);
}
if (tempDate != null) {
longVal = tempDate.getTime();
}
return longVal;
}
private cleanLogString(String logString) {
String clean = logString.replaceAll("[^A-Za-z0-9]", "");
if(!logString.equals(clean)) {
clean += " (CLEANED)";
}
return clean;
}
The cleanLogString() method has fixed other Log Forging Fortify issues in my project, however it has no effect on the 2 above.
Any help would be appreciated!
It is possible to use fortify Java annotations to tell Fortify that the data returned from a sanitizing function is now safe.
When looking at my log forging problems I had strings coming in through a web API and thus had the flags XSS and WEB on my strings. I tried to find annotations that would only remove these flags, but couldn't find any way to remove the WEB flag. The only documentation I've found is the Samples/advanced/javaAnnotation directory.
Since my sanitation method does sanitize strings, I choose to remove all flags. This could be a problem though, as it could hide privacy violations.
#FortifyValidate("return")
private String sanitizeString(String taintedString) {
return doSomethingWithTheString(taintedString);
}
Originally when this question was written our team was using log4j v1.2.8, however we noticed that all the log forging issues disappeared after upgrading to log4j v2.6.2.
Once the log4j version is upgraded the Fortify log forging issues should go away. The cleanLogString() method form the question above is also unnecessary. For example:
LOGGER.info("getLongFromTimestamp(" + value + ")");
I know I have run into situations where the complexity of my application would stop any malicious input from working as intended; Fortify does not consider this to be secure. I bet you are running into the same thing.
You are stripping any really useful characters out of the log message, but see what happens if you do some encoding on the output prior to writing to the log.
http://www.jtmelton.com/2010/09/21/preventing-log-forging-in-java/
// ensure no CRLF injection into logs for forging records
String clean = message.replace( '\n', '_' ).replace( '\r', '_' );
if ( ESAPI.securityConfiguration().getLogEncodingRequired() ) {
clean = ESAPI.encoder().encodeForHTML(message);
if (!message.equals(clean)) {
clean += " (Encoded)";
}
}
Use reflect or try-catch.
Its easy to cheat fortify.
What I've tried so far is:
## $NAME is something like 'my_controller_c'
#set($NAME = $NAME.removeAndHump($NAME))
#set($NAME = $NAME.underscoresToCamelCase(String)
But that does not work. The first one does nothing, the second one throws an java error.
I also tried using regular expressions and to loop through the string, but my java knowledge is very basic.
The following works in PhpStorm 9 (and probably all of the other JetBrains IDEs, I would guess):
#set($new_name = ${StringUtils.removeAndHump(${NAME}, "-")})
class $new_name {
}
This is, what I ended up doing:
#set($ctrlName = $NAME.replaceAll("-c$", ""))
#set($ctrlNewName = "")
#foreach($str in $ctrlName.split("-"))
#set($str = $str.substring(0,1).toUpperCase()+$str.substring(1))
#set($ctrlNewName = $ctrlNewName + $str)
#end
#set ( $ctrlNewName = $ctrlNewName + "Ctrl" )