Develop Reference

Dealing with .CSV files and sorting a listarrays into multiple listarrays? - JAVA

So i am trying to sort through a large amount of data in an CSV. file. The file includes a set amount of information for companies, but there are 1000s of companies. For example, I might need to go through 1000 companies, be able to acquire their annual earnings, current stock value, CEO, ect.. each company will have the same information provided (same number of commas but different char lengths), but as the file is a CSV. the company name and information is all separated by commas.
currently i am splitting the csv file into an array via the commas between information. But i want to be able to keep the information together with companies and be able to specify, call and, sort by the given information and company names. But because i have already separated the information via the commas its all listed out already in a listarray.
So is it possible to specify, on a mass scale, that every 15 commas (or splits in the listarray) should be joined back together?? This way each part of the listarray is a separate company. Or is there another way to separate the data so that the information doesnt get split up?
note: there is no similarities in the csv file that would allow me to split information so that it splits after each companies information.
here is a sample of what one of the csv files may look like.
"Tiffany & Co. Com",964270,"+0.81","1/14/2014",88.97,93.64,"87.795 - 88.97""Asia Pacific Fund",20700,"+0.04","1/14/2014",10.23,11.37,"10.19 - 10.23""Anadarko Petroleu",4236380,"+2.47","1/14/2014",80.99,98.47,"78.40 - 80.99""Proto Labs, Inc. ",451984,"-0.18","1/14/2014",73.83,89.97,"71.00 - 73.83""Zuoan Fashion Lim",201560,"-0.02","1/14/2014",1.79,3.62,"1.71 - 1.79"

I would agree with converting each row of CSV into Java object.
But traditional parsing mechanism is too verbose for me and I might need to handle too many conditions like comma in between quotes, new line character in a column with multiline description. So I suggest you use an existing awesome solution like supercsv.
I also have written a wrapper around it to make developer life easy.
QuickOCM will let you do this way.
Create a Company class
public class Company {
/* this specifies that
* it is a mandatory field in the csv,
* of header name "Company Name" and
* of type string.
* Header is the first line of the csv.
*/
#ImportField(mandatory = true, name = "Company Name", type = "String")
public String name;
#ImportField(mandatory = true, name = "Name of the CEO", type = "String")
public String ceoName;
}
You need public getter-setter or public fields, anything works
Create csv parser, a handler to handle each row, probably add to a list to sort, and then call parse
final List<Company> companies = new ArrayList<Company>();
csvParser.process(inputStream, Company.class, new RecordHandler<Company>() {
#Override
public void execute(Company imported, int rowNumber, Map supplementaryInfo) {
companies.add(imported);
}
});
Now you can sort the list with by using a sorted list implementation or use a comparator for the same.
For detailed info, you can look into QuickOCM page.

public class Read{
String original = "";
String company = "";
String otherValue = "";
public Read(String read){
//here Split the original string into the values
}
//public void getters and setters
}
Then make an array of Read Objects and sort them as you want

One idea would be to parse the CSV into objects and then sort those objects. The object would "know" how many fields it was made up of in the CSV and how to parse each field. Using the StringTokenizer to parse and a TreeMap to sort would look something like:
...
BufferedReader reader = new BufferedReader(new FileReader("somedata.csv"));
TreeMap<String, MyObject> map = new TreeMap<>();
String line = reader.readLine();
StringTokenizer tokens = new StringTokenizer(line,",");
while(tokens.hasMoreTokens()) {
MyObject obj = new MyObject(tokens);
//add the objects to the sorted map, where field1 is what we sort on
map.put(obj.field1, obj);
}
...
}
static class MyObject {
//would need the same number of fields as you want to group
String field1;
String field2;
//... so with 2 fields, input is field1,field2,field1,field2,...
MyObject (StringTokenizer input) {
this.field1 = input.nextToken();
this.field2 = input.nextToken();
}
}

Java regex and/or string magic to extract IDs from String

I have a Java app that is hitting a 3rd party RESTful web service that is returning the following JSON:
{"fizz":
{"widgets":
[
{
"widget_id":"295874"
},
{
"widget_id":"295873"
},
{
"widget_id":"295872"
}
],
"otime":1361993756
},
"resp":"ok"
}
Normally I would use GSON or Genson to map this back to a Java POJO, but this is the only area of the code where I have to do this and I want to be lazy here ;-).
I'm trying to come up with a nifty method that extracts the 3 widget_id values (, and `) and returns them as aList`:
public List<Long> extractIdsFromJson(String json) {
// Can I solve this with a regex perhaps?
}
Not sure what the right approach is - regex, replaceAll, something else? Thanks in advance.

Being lazy here will just bite you in the long run. Parse the JSON and extract the values that way; the 'effort' involved will be less, the code will be more understandable, and future code maintainers will not curse your name.

// untested
public List<Long> extractIdsFromJson(String json) {
List<Long> list = new ArrayList<Long>();
Matcher matcher = Pattern.compile("\"widget_id\":\"?(\\d+)\"?").matcher(json);
while (matcher.find())
list.add(Long.valueOf(matcher.group(1)));
return list;
}

If you like being lazy. Here is the solution. I hope you know whatever entails your choice of solving the problem with regex:
It doesn't check for the structure of the JSON. You ignore the fact that the JSON may be malformed and just blindly extract the data.
It works here since you want a property whose value is not an Object or Array.
RAW regex:
"widget_id"\s*:\s*"(\d+)"
In literal string:
"\"widget_id\"\\s*:\\s*\"(\\d+)\""
Use the regex above with Matcher loop:
Pattern p = Pattern.compile("\"widget_id\"\\s*:\\s*\"(\\d+)\"");
Matcher m = p.matcher(inputString);
while (m.find()) {
System.out.println(m.group(1));
}

Java CLI arguments syntax for object initialization

I'm looking for a tool which will allow me use command-line-style (preferably POSIX) strings to initialize an object' properties and attributes.
For example, you'd provide it with String input formatted like so:
String input = "--firstName=John --MiddleName=\"Louis Victor\" --lastName=Smith";
... and it would setFirstName("John"), setMiddleName("Louis Victor") and setLastName("Smith") on a given object. (which could be a JavaBean)
Please note that the input is a single String, not an array String[] as is the case with many popular CLI argument "parsers".
This is all similar to args4j but I couldn't get that to work... and I'm hoping to avoid using #annotations.
Does anyone have code/libraries/tools which could accomplish this?

For your use case, forget regular CLI parsers, you need a custom-tailored solution. If you really have such a simple argument syntax (parameters always begin with --, no occurrences of -- in the parameter values), you can use a simple Guava-based solution like this class:
Parse the String Arguments
public class ArgParser{
// split on (optional whitespace) + "--"
private final Splitter paramSplitter = Splitter.on(
Pattern.compile("\\s*\\-{2}")).omitEmptyStrings();
// find key=value (with optional double quotes around value)
private final Pattern keyValuePattern = Pattern
.compile("(.+?)=\"?(.*?)\"?$");
public Map<String, String> getParamValues(final String posixString){
final Map<String, String> paramValues = Maps.newLinkedHashMap();
Matcher matcher;
for(final String param : paramSplitter.split(posixString)){
matcher = keyValuePattern.matcher(param);
if(!matcher.find()){
throw new IllegalArgumentException("Bad parameter: " + param);
}
paramValues.put(matcher.group(1), matcher.group(2));
}
return paramValues;
}
}
Usage
final String input =
"--firstName=John --middleName=\"Louis Victor\" --lastName=Smith";
System.out.println(new ArgParser().getParamValues(input));
Output
{firstName=John, middleName=Louis Victor, lastName=Smith}
Now you can take the map and use it with a Bean library like commons-beanutils (I prefer the Spring BeanWrapper personally, but that only makes sense if you use Spring anyway)
Define the Bean Class
Any way, I'll use this value holder class:
public class Name{
private String firstName;
private String middleName;
private String lastName;
#Override
public String toString(){
return Objects
.toStringHelper(this)
.add("first name", firstName)
.add("middle name", middleName)
.add("last name", lastName)
.toString();
}
// + getters & setters
}
Set the Bean Properties
Now we'll use BeanUtils.populate(Object, Map) to apply the parameter values, like this:
final String input =
"--firstName=John --middleName=\"Louis Victor\" --lastName=Smith";
final Map<String, String> paramValues =
new ArgParser().getParamValues(input);
final Name name = new Name();
BeanUtils.populate(name, paramValues);
System.out.println(name);
Output:
Name{first name=John, middle name=Louis Victor, last name=Smith}
Caveat: Supported Property Types
BeanUtils.populate() supports setting the following property types:
... String, boolean, int, long, float, and double.
In addition, array setters for these
types (or the corresponding primitive
types) can also be identified.
Source: BeanUtilsBean.populate(Object, Map)
If you need parameter conversion beyond that, you should probably look into using the Spring BeanWrapper after all, it's extremely powerful, has many built-in property editors and you can add custom property editors. Just change the code like this:
final Name name = new Name();
final BeanWrapper wrapper = new BeanWrapperImpl(name);
wrapper.setPropertyValues(paramValues);
Reference:
BeanWrapper
PropertyAccessor.setPropertyValues(Map)

If I understand correctly, you are looking for a Java library to parse POSIX-style command line parameters. I used JSAP some time ago and it was really cool (it was using XML configuration back then).

This
-firstName John -lastName Smith
is no POSIX, you mean
--firstName John --lastName Smith
This may be the reason, why you can't get it working.
Update:
As I look at the example, it doesn't look like it could be the reason.

Custom conversion specifiers in java

I want my data structures to be custom formatted.
e.g. I have a DS
Address {
string house_number,
string street,
string city,
long pin_code,
}
Now, I want to associate certain conversion specifiers with each of these fields.
e.g. house_number -> H
street -> S,
city -> C,
pin_code -> P
...
So that something like
myPrintWriter.printf("Mr A lives in %C", address_instance)
yields "Mr A lives in boston" (if address_instance.city = boston) etc..
It seems there is no easy way to do this. java.util.Formatter seems to be final. The only customization it provides is via the interface Formattable, but that helps in customizing the 's' conversion specifier only.
Is there a way to add our custom conversion specifiers? Any help will be much appreciated.
Thanks,

It seems there is no easy way to do this. java.util.Formatter seems to be final.
That's true, but you can still use composition. I would do something like the following:
class ExtendedFormatter {
private Formatter defaultFormatter;
// provide the same methods like the normal Formatter and pipe them through
// ...
// then provide your custom method, or hijack one of the existing ones
// to extend it with the functionality you want
// ...
public Formatter format(String format, Object... args) {
// extract format specifiers from string
// loop through and get value from param array
ExtendedFormattable eft = (ExtendedFormattable)args1;
String specifierResult = eft.toFormat(formatSpecifier); // %C would return city
// use specifierResult for the just queried formatSpecifier in your result string
}
}
The hard part is to know how to attach the different format specifiers to the fields you want to output. The first way I can think of, is to provide your own ExtendedFormattable interface that each class that should be used with the ExtendedFormatter can implement, and return the according values for your custom format specifiers. That could be:
class Address implements ExtendedFormattable {
public String toFormat(String formatSpecifier) { // just an very simple signature example
// your custom return values here ...
}
}
There's also annotations, but I think that's not a very viable way.
A sample call would look like:
ExtendedFormatter ef = new ExtendedFormatter();
ef.format("Mr A lives in %C", address_instance);

I believe you will need to write your own formatter which works the way you want.

String replacement in java, similar to a velocity template

Is there any String replacement mechanism in Java, where I can pass objects with a text, and it replaces the string as it occurs?
For example, the text is:
Hello ${user.name},
Welcome to ${site.name}.
The objects I have are user and site. I want to replace the strings given inside ${} with its equivalent values from the objects. This is same as we replace objects in a velocity template.

Use StringSubstitutor from Apache Commons Text.
Dependency import
Import the Apache commons text dependency using maven as bellow:
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-text</artifactId>
<version>1.10.0</version>
</dependency>
Example
Map<String, String> valuesMap = new HashMap<String, String>();
valuesMap.put("animal", "quick brown fox");
valuesMap.put("target", "lazy dog");
String templateString = "The ${animal} jumped over the ${target}.";
StringSubstitutor sub = new StringSubstitutor(valuesMap);
String resolvedString = sub.replace(templateString);

Take a look at the java.text.MessageFormat class, MessageFormat takes a set of objects, formats them, then inserts the formatted strings into the pattern at the appropriate places.
Object[] params = new Object[]{"hello", "!"};
String msg = MessageFormat.format("{0} world {1}", params);

My preferred way is String.format() because its a oneliner and doesn't require third party libraries:
String message = String.format("Hello! My name is %s, I'm %s.", name, age);
I use this regularly, e.g. in exception messages like:
throw new Exception(String.format("Unable to login with email: %s", email));
Hint: You can put in as many variables as you like because format() uses Varargs

I threw together a small test implementation of this. The basic idea is to call format and pass in the format string, and a map of objects, and the names that they have locally.
The output of the following is:
My dog is named fido, and Jane Doe owns him.
public class StringFormatter {
private static final String fieldStart = "\\$\\{";
private static final String fieldEnd = "\\}";
private static final String regex = fieldStart + "([^}]+)" + fieldEnd;
private static final Pattern pattern = Pattern.compile(regex);
public static String format(String format, Map<String, Object> objects) {
Matcher m = pattern.matcher(format);
String result = format;
while (m.find()) {
String[] found = m.group(1).split("\\.");
Object o = objects.get(found[0]);
Field f = o.getClass().getField(found[1]);
String newVal = f.get(o).toString();
result = result.replaceFirst(regex, newVal);
}
return result;
}
static class Dog {
public String name;
public String owner;
public String gender;
}
public static void main(String[] args) {
Dog d = new Dog();
d.name = "fido";
d.owner = "Jane Doe";
d.gender = "him";
Map<String, Object> map = new HashMap<String, Object>();
map.put("d", d);
System.out.println(
StringFormatter.format(
"My dog is named ${d.name}, and ${d.owner} owns ${d.gender}.",
map));
}
}
Note: This doesn't compile due to unhandled exceptions. But it makes the code much easier to read.
Also, I don't like that you have to construct the map yourself in the code, but I don't know how to get the names of the local variables programatically. The best way to do it, is to remember to put the object in the map as soon as you create it.
The following example produces the results that you want from your example:
public static void main(String[] args) {
Map<String, Object> map = new HashMap<String, Object>();
Site site = new Site();
map.put("site", site);
site.name = "StackOverflow.com";
User user = new User();
map.put("user", user);
user.name = "jjnguy";
System.out.println(
format("Hello ${user.name},\n\tWelcome to ${site.name}. ", map));
}
I should also mention that I have no idea what Velocity is, so I hope this answer is relevant.

Here's an outline of how you could go about doing this. It should be relatively straightforward to implement it as actual code.
Create a map of all the objects that will be referenced in the template.
Use a regular expression to find variable references in the template and replace them with their values (see step 3). The Matcher class will come in handy for find-and-replace.
Split the variable name at the dot. user.name would become user and name. Look up user in your map to get the object and use reflection to obtain the value of name from the object. Assuming your objects have standard getters, you will look for a method getName and invoke it.

There are a couple of Expression Language implementations out there that does this for you, could be preferable to using your own implementation as or if your requirments grow, see for example JUEL and MVEL
I like and have successfully used MVEL in at least one project.
Also see the Stackflow post JSTL/JSP EL (Expression Language) in a non JSP (standalone) context

Handlebars.java might be a better option in terms of a Velocity-like syntax with other server-side templating features.
http://jknack.github.io/handlebars.java/
Handlebars handlebars = new Handlebars();
Template template = handlebars.compileInline("Hello {{this}}!");
System.out.println(template.apply("Handlebars.java"));

I use GroovyShell in java to parse template with Groovy GString:
Binding binding = new Binding();
GroovyShell gs = new GroovyShell(binding);
// this JSONObject can also be replaced by any Java Object
JSONObject obj = new JSONObject();
obj.put("key", "value");
binding.setProperty("obj", obj)
String str = "${obj.key}";
String exp = String.format("\"%s\".toString()", str);
String res = (String) gs.evaluate(exp);
// value
System.out.println(str);

I created this utility that uses vanilla Java. It combines two formats... {} and %s style from String.format.... into one method call. Please note it only replaces empty {} brackets, not {someWord}.
public class LogUtils {
public static String populate(String log, Object... objects) {
log = log.replaceAll("\\{\\}", "%s");
return String.format(log, objects);
}
public static void main(String[] args) {
System.out.println(populate("x = %s, y ={}", 5, 4));;
}
}

Since Java 15 you have the method String.formatted() (see documentation).
str.formatted(args) is the equivalent of String.format(str, args) with less ceremony.
For the example mentioned in the question, the method could be used as follows:
"Hello %s, Welcome to %s.".formatted(user.getName(), site.getName())

Good news. Java is most likely going to have string templates (probably from version 21).
See the string templates proposal (JEP 430) here.
It will be something along the lines of this:
String name = "John";
String info = STR."I am \{name}";
System.out.println(info); // I am John
P.S. Kotlin is 100% interoperable with Java. It supports cleaner string templates out of the box:
val name = "John"
val info = "I am $name"
println(info) // I am John
Combined with extension functions, you can achieve the same thing the Java template processors (e.g. STR) will do.

There is nothing out of the box that is comparable to velocity since velocity was written to solve exactly that problem. The closest thing you can try is looking into the Formatter
http://cupi2.uniandes.edu.co/site/images/recursos/javadoc/j2se/1.5.0/docs/api/java/util/Formatter.html
However the formatter as far as I know was created to provide C like formatting options in Java so it may not scratch exactly your itch but you are welcome to try :).