i would like to know whether exist a way to parallelize queries in Java (or there is framework or a library) like in C# and Linq:
var query = from item in source.AsParallel().WithDegreeOfParallelism(2)
where Compute(item) > 42
select item;
and if i can't parallelize queries if i can do something like this in c# (a for each parallelized) :
Parallel.ForEach(files, currentFile =>
{
// The more computational work you do here, the greater
// the speedup compared to a sequential foreach loop.
string filename = System.IO.Path.GetFileName(currentFile);
System.Drawing.Bitmap bitmap = new System.Drawing.Bitmap(currentFile);
bitmap.RotateFlip(System.Drawing.RotateFlipType.Rotate180FlipNone);
bitmap.Save(System.IO.Path.Combine(newDir, filename));
// Peek behind the scenes to see how work is parallelized.
// But be aware: Thread contention for the Console slows down parallel loops!!!
Console.WriteLine("Processing {0} on thread {1}", filename,
Thread.CurrentThread.ManagedThreadId);
}
please if you post any framework or library, can you tell me the experience that you had with it ?
thx for your time.
about c# and linq you can find here the documentation : http://msdn.microsoft.com/en-us/library/dd997425.aspx
There isn't a direct translation. Firstly Java doesn't have LINQ nor does it have any standard parallel collection classes.
GPars is perhaps closest fit.
Note that while it is targeted at groovy it's API is perfectly usable from java
Maybe Fork/Join Framework can help you ? Here is java tutorial
Related
I'm a C# Developer and recently starting get into Java development and here I have a question. Did Java have any build in method that doing the same thing with C# .Take()?
C# example:
int diffNo = 1;
someNumber.OrderBy(x => x.someNumber).Take(diffNo).ToList();
Java example:
someNumber.stream().sorted(Comparator.comparing(Object::getSomeNumber)).collect(Collectors.toList());
So now for Java part I only able to do sorting but don't know is there method can use to replace .Take()
Streams have a limit method, used to truncate a stream to up to the number of elements you provide as an argument.
So, assuming diffNo is a number, you can call it like this
someNumber.stream()
.sorted(Comparator.comparing(SomeClass::getSomeNumber))
.limit(diffNo)
.collect(Collectors.toList());
I am learning Java through an introductory course using the textbook, Java Programming 9th Edition by Joyce Farrel. The examples and exercises are written for Java 9e, however, I am using Java SE 14.
I've successfully navigated Java API and found updates as well as useful explanations as to what errors I have been encountering between the two versions and what is the best way to make corrections to get the examples and exercises to work.
However, in this case, I have been having a really hard time. I'm pretty sure it's due to the lack of experience, but I can't find anything I can understand using the Java API that has given me an idea on how to resolve this issue. Google and Stackoverflow posts have not been that much more successful as I am assuming people are using a much more streamlined method or approach.
Code with the comment on the line in question:
...
Path rafTest = Paths.get("raf.txt");
String addIn = "abc";
byte[] data = addIn.getBytes();
ByteBuffer out = ByteBuffer.wrap(data);
FileChannel fc = null;
try {
fc = (FileChannel)Files.newByteChannel(file, READ, WRITE); // Error READ and Write is ambiguous?
...
} catch (Exception e){
System.out.println("Error message: " + e);
}
...
What is the best way to go about finding an approach to figuring out what exactly is going here?
#Bradley: Found the answer by trying to rewrite my question. The compiler returned 3 specific errors dealing with StandardOpenOption. Using that and Java API, I found the solution. Thank you.
#NomadMaker: First thought was that I did not include the package correctly for newByteChannel. The second options were that the arguments needed a more specific reference.
Answer: newByteChannel(...); requires the open options argument to reference the StandardOpenOption.READ and WRITE. So that:
...newByteChannel(raf, StandardOpenOption.READ, StandardOpenOption.WRITE);
This change was implemented in Java SE 11. The program now works correctly.
I am exploring the possibilities of the Java Stream API in order to find out if one could possibly replace all loop-based constructs with stream-based constructs.
As an example that would probably hint to the assumption that this is actually possible, consider this:
Is it possible to use the Stream API to split a string containing words delimited by spaces into an array of strings like the following call of String.split(String) would do ?
String[] tokens = "Peter Paul Mary".split(" ");
I am aware of the fact that a stream-based solution could make itself use of the String.split(String) method like so:
Stream.of("Peter Paul Mary".split(" "))
.collect(Collectors.toList());
or make use of Pattern.splitAsStream(CharSequence) (the implementation of which certainly uses a loop-based approach) but I am looking for a Stream-only solution, meaning something along the lines of this Haskell snippet:
words :: String -> [String]
words s = case dropWhile Char.isSpace s of
"" -> []
s' -> w : words s''
where (w, s'') = break Char.isSpace s'
I am asking this because I am still wondering if the introduction of the Stream API will lead to a profound change in the way we handle object collections in Java or just add another option to it, thus making it more challenging to maintain a larger codebase rather than to simplify this task in the long run.
EDIT: Although there is an accepted answer (see below) it only shows that its possible in this special case. I am still interested in any hints to the general case as required in the question.
A distinct non answer here: you are asking the wrong question!
It doesn't matter if all "loop related" lines of Java code can be converted into something streamish.
Because: good programming is the ability to write code that humans can easily read and understand.
So when somebody puts up a rule that says "we only use streams from hereon for everything we do" then that rule significantly reduces your options when writing code. Instead of being able to carefully decide "should I use streams" versus "should I go with a simple old-school loop construct" you are down to "how do I get this working with streams"!
From that point of view, you should focus on coming up with "rules" that work for all people in your development team. That could mean to empasize the use of stream constructs. But you definitely want to avoid the absolutism, and leave it to each developer to write up that code that implements a given requirement in the "most readable" way. If that is possible with streams, fine. If not, don't force people to do it.
And beyond that: depending on what exactly you are doing, using streams comes also with performance cost. So even when you can find a stream solution for a problem - you have to understand its runtime cost. You surely want to avoid using streams in places where they cost too much (especially when that code is already on your "critical path" regarding performance, like: called zillions of times per second).
Finally: to a certain degree, this is a question of skills. Meaning: when you are trained in using streams, it is much easier for you to A) read "streamish" code that others wrote and B) coming up with "streamish" solutions that are in fact easy to read. In other words: this again depends on the context you are working. The other week I was educating another team on "clean code", and my last foil was "Clean Code, Java8 streams/lambdas". One guy asked me "what are streams?" No point in forcing such a community to do anything with streams tomorrow.
Just for fun (this is one horrible way to do it), neither do I know if this fits your needs:
List<String> result = ",,,abc,def".codePoints()
.boxed()
// .parallel()
.collect(Collector.of(
() -> {
List<StringBuilder> inner = new ArrayList<>();
inner.add(new StringBuilder());
return inner;
},
(List<StringBuilder> list, Integer character) -> {
StringBuilder last = list.get(list.size() - 1);
if (character == ',') {
list.add(new StringBuilder());
} else {
last.appendCodePoint(character);
}
},
(left, right) -> {
left.get(left.size() - 1).append(right.remove(0));
left.addAll(right);
return left;
},
list -> list.stream()
.map(StringBuilder::toString)
.filter(x -> !x.equals(""))
.collect(Collectors.toList())
));
Specifically I am converting a python script into a java helper method. Here is a snippet (slightly modified for simplicity).
# hash of values
vals = {}
vals['a'] = 'a'
vals['b'] = 'b'
vals['1'] = 1
output = sys.stdout
file = open(filename).read()
print >>output, file % vals,
So in the file there are %(a), %(b), %(1) etc that I want substituted with the hash keys. I perused the API but couldn't find anything. Did I miss it or does something like this not exist in the Java API?
You can't do this directly without some additional templating library. I recommend StringTemplate. Very lightweight, easy to use, and very optimized and robust.
I doubt you'll find a pure Java solution that'll do exactly what you want out of the box.
With this in mind, the best answer depends on the complexity and variety of Python formatting strings that appear in your file:
If they're simple and not varied, the easiest way might be to code something up yourself.
If the opposite is true, one way to get the result you want with little work is by embedding Jython into your Java program. This will enable you to use Python's string formatting operator (%) directly. What's more, you'll be able to give it a Java Map as if it were a Python dictionary (vals in your code).
Are there any production-ready libraries for streaming XPath expressions evaluation against provided xml-document? My investigations show that most of existing solutions load entire DOM-tree into memory before evaluating xpath expression.
XSLT 3.0 provides streaming mode of processing and this will become a standard with the XSLT 3.0 W3C specification becoming a W3C Recommendation.
At the time of writing this answer (May, 2011) Saxon provides some support for XSLT 3.0 streaming .
Would this be practical for a complete XPath implementation, given that XPath syntax allows for:
/AAA/XXX/following::*
and
/AAA/BBB/following-sibling::*
which implies look-ahead requirements ? i.e. from a particular node you're going to have to load the rest of the document anyway.
The doc for the Nux library (specifically StreamingPathFilter) makes this point, and references some implementations that rely on a subset of XPath. Nux claims to perform some streaming query capability, but given the above there will be some limitations in terms of XPath implementation.
There are several options:
DataDirect Technologies sells an XQuery implementation that employs projection and streaming, where possible. It can handle files into the multi-gigabyte range - e.g. larger than available memory. It's a thread-safe library, so it's easy to integrate. Java-only.
Saxon is an open-source version, with a modestly-priced more expensive cousin, which will do streaming in some contexts. Java, but with a .net port also.
MarkLogic and eXist are XML databases that, if your XML is loaded into them, will process XPaths in a fairly intelligent fashion.
Try Joost.
Though I have no practical experience with it, I thought it is worth mentioning QuiXProc ( http://code.google.com/p/quixproc/ ). It is a streaming approach to XProc, and uses libraries that provide streaming support for XPath amongst others..
FWIW, I've used Nux streaming filter xpath queries against very large (>3GB) files, and it's both worked flawlessly and used very little memory. My use case is been slightly different (not validation centric), but I'd highly encourage you to give it a shot with Nux.
I think I'll go for custom code. .NET library gets us quite close to the target, if one just wants to read some paths of the xml document.
Since all the solutions I see so far respect only XPath subset, this is also this kind of solution. The subset is really small though. :)
This C# code reads xml file and counts nodes given an explicit path. You can also operate on attributes easily, using xr["attrName"] syntax.
int c = 0;
var r = new System.IO.StreamReader(asArgs[1]);
var se = new System.Xml.XmlReaderSettings();
var xr = System.Xml.XmlReader.Create(r, se);
var lstPath = new System.Collections.Generic.List<String>();
var sbPath = new System.Text.StringBuilder();
while (xr.Read()) {
//Console.WriteLine("type " + xr.NodeType);
if (xr.NodeType == System.Xml.XmlNodeType.Element) {
lstPath.Add(xr.Name);
}
// It takes some time. If 1 unit is time needed for parsing the file,
// then this takes about 1.0.
sbPath.Clear();
foreach(object n in lstPath) {
sbPath.Append('/');
sbPath.Append(n);
}
// This takes about 0.6 time units.
string sPath = sbPath.ToString();
if (xr.NodeType == System.Xml.XmlNodeType.EndElement
|| xr.IsEmptyElement) {
if (xr.Name == "someElement" && lstPath[0] == "main")
c++;
// And test simple XPath explicitly:
// if (sPath == "/main/someElement")
}
if (xr.NodeType == System.Xml.XmlNodeType.EndElement
|| xr.IsEmptyElement) {
lstPath.RemoveAt(lstPath.Count - 1);
}
}
xr.Close();