Develop Reference

get text from pdf - groovy

I need to extract text from a pdf using Groovy.
This is my code:
def PDDocumentClass = Class.forName("org.apache.pdfbox.pdmodel.PDDocument", true, urlLoader);
def PDPageClass = Class.forName("org.apache.pdfbox.pdmodel.PDPage", true, urlLoader);
def PDFTextStripper = Class.forName("org.apache.pdfbox.text.PDFTextStripper", true, urlLoader);
def pdfTextStripperArea = Class.forName("org.apache.pdfbox.text.PDFTextStripperByArea", true, urlLoader);
def pdfTextRandom = Class.forName("org.apache.pdfbox.io.RandomAccessFile", true, urlLoader);
def d = docman.getNodeByName(docman.getNode(346462 as long), "cv1.pdf").content
File f1 = new File("${d.content}")
def pdfstrip = PDFTextStripper.getText(PDDocumentClass.load(f1))
but the execution of this code gives me the following error:
No signature of method: static org.apache.pdfbox.text.PDFTextStripper.getText() is applicable for argument types: (org.apache.pdfbox.pdmodel.PDDocument) values: [org.apache.pdfbox.pdmodel.PDDocument#1590d92b]
Possible solutions: getText(org.apache.pdfbox.pdmodel.PDDocument), endText(), beginText(), getAt(java.lang.String), getLevel(), writeText(org.apache.pdfbox.pdmodel.PDDocument, java.io.Writer)
I'm sure the argument passed to the getText() method is a PDDocument, the required type!
When I try to create a new instace of PdfTextStripper I receive this error:
error instance

Spark listeners for database loading [duplicate]

I'm trying to run a couple of spark SQL statements and want to calculate their running time.
One of the solution is to resort to log. I’m wondering is there any other simpler methods to do it. Something like the following:
import time
startTimeQuery = time.clock()
df = sqlContext.sql(query)
df.show()
endTimeQuery = time.clock()
runTimeQuery = endTimeQuery - startTimeQuery

If you're using spark-shell (scala) you could try defining a timing function like this:
def show_timing[T](proc: => T): T = {
val start=System.nanoTime()
val res = proc // call the code
val end = System.nanoTime()
println("Time elapsed: " + (end-start)/1000 + " microsecs")
res
}
Then you can try:
val df = show_timing{sqlContext.sql(query)}

Strange scala java interoperability behavior

I ran into the following Java compiler warning issue (using IntelliJ 14):
I have the following class in my Scala library:
object ScalatoJava {
def asJava[A,B](map: Map[A,B]): java.util.Map[A,B] = { map.asJava }
def asJava[A](list: List[A]): java.util.Collection[A] = {list.asJavaCollection}
}
I use it to convert few collections coming from Scala in my Java program.
Now in my Java program I have the following code for which IntelliJ is underlying in red stating that it can't resolve things:
private final java.util.Map<String, FieldSettings> poolpartyfieldsettings =
ScalatoJava.asJava(poolpartyConnectorSettings.fieldsSettingsMap());
It says it can not resolve the method asJava(scala.collection.immutable.Map<String, org.....FieldSettings>).
While if I write either
private final java.util.Map <String, FieldSettings> poolpartyfieldsettings =
ScalatoJava.asJava((scala.collection.immutable.Map<String, FieldSettings>)poolpartyConnectorSettings.fieldsSettingsMap());
or
scala.collection.immutable.Map<String, FieldSettings> poolpartyfieldsettings =
poolpartyConnectorSettings.fieldsSettingsMap();
ScalatoJava.asJava(poolpartyfieldsettings)
the compiler does not complain.
Here is the PoolpartyConnectorSettings Scala class:
trait DspacePoolPartyConnectorSettings {
def poolpartyServerSettings : PoolpartySettings
def fieldsSettingsList : List[FieldSettings]
def fieldsSettingsMap : Map[String, FieldSettings]
}
case class DspaceDspacePoolPartyConnectorSettingImpl (configUri: String) extends DspacePoolPartyConnectorSettings {
private val config = ConfigFactory.parseURL(new URL(configUri)).resolve()
private val configFieldSettinglist = config.getConfigList("PoolPartyConnectorSettings.FieldSettings").asScala.toList
private val configPoolPartySetting = config.getConfig("PoolPartyConnectorSettings.PoolPartySettings")
val poolpartyServerSettings = PoolpartySettings( configPoolPartySetting.getString("apirootEndpoint"),
configPoolPartySetting.getString("thesaurusapiEndpoint"),
configPoolPartySetting.getString("extratorapiEndpoint"),
configPoolPartySetting.getString("coreProjectId"),
configPoolPartySetting.getString("coreThesaurusUri"),
configPoolPartySetting.getString("jelProjectId"),
configPoolPartySetting.getString("jelThesaurusUri"))
val fieldsSettingsList = configFieldSettinglist map { e =>
FieldSettings(e.getString("fieldname"), e.getBoolean("treeBrowser"),
e.getBoolean("multilanguage"), e.getStringList("languages").asScala.toList,
e.getBoolean("closed"), e.getString("scheme"), e.getString("poolpartyProjectId"))
}
val fieldsSettingsMap = Map(fieldsSettingsList map {e => (e.fieldName, e)}: _*)
}
Any idea what the problem might be?

Prohibit resolving during loading in typesafe config

I want to prohibit resolving of a.b. I want to substitute param from another config. Like this:
val d = ConfigFactory.load(ConfigFactory.parseString(
"""
|param = x
|a.b = ${param}
""".stripMargin))
val a = ConfigFactory.parseString("param = 1")
val result = a.withFallback(d).resolve()
In this case param gets value 1, but a.b remains x
I've tried to set ConfigResolveOptions.defaults().setAllowUnresolved(true) when loading config d, but that doesn't work.
How can I overcome this?

The problem is that Config.load is resolving the substitution immediately. If you take that out it resolves like you want it to:
val p = ConfigFactory.parseString(
"""
|param = x
|a.b = ${param}
""".stripMargin)
val a = ConfigFactory.parseString("param = 1")
val result = a.withFallback(p).resolve()
println(result.getString("a.b"))
This prints 1.
You don't need to use Config.load unless you want to use reference.conf, etc. If you do want to use Config.load then you should do it after you have composed all the configs together using withFallback.
For example, this also prints 1:
val p = ConfigFactory.parseString(
"""
|param = x
|a.b = ${param}
""".stripMargin)
val d = ConfigFactory.load(p)
val a = ConfigFactory.parseString("param = 1")
val result = a.withFallback(p)
val loaded = ConfigFactory.load(result)
println(loaded.getString("a.b"))
Or, say you have an application.conf with include that you want to use with ConfigFactory.load() (per your comment).
If application.conf looks like
include "foo"
and foo.conf looks like
a.b = ${param}
then this prints 1 also:
val a = ConfigFactory.parseString("param = 1")
val app = ConfigFactory.load("application", ConfigParseOptions.defaults,
ConfigResolveOptions.defaults.setAllowUnresolved(true))
val result = a.withFallback(app).resolve
println(result.getString("a.b"))
In general, if you want A to override B to override C then you should use A.withFallback(B).withFallback(C).

I struggled a bit with the same thing: trying to use "fallbacks" to override values, when it was designed for layering/merging configs
Assuming I understand your use case, I recommend instead using file includes.
In my application.conf I have the default value
a.b = "placeholder"
And at the bottom I have the following include
# Local overrides - for development use
include "local.conf"
And finally in local.conf
param = 1
a.b = ${param}
The end result is that a.b will be overridden with 1

Found a workaround for my problem:
So If I have config file application.conf which uses include to include config files which contain substitution syntax and files which contain declaration of the config values which are going to be substituted.
val a = ConfigFactory.parseString(s"""param = 1""")
val z = ConfigFactory.parseResources("application.conf") //this doesn't resolve substitutions
val result = a.withFallback(z).resolve().withFallback(ConfigFactory.load("application.conf"))

Scala opening write to stdout or file

Let's say I have a function
writeToFileOrStdout(fname: String = Nil) = { ... }
If the user passes a string value for fname, then I'd like to open a file with that name and write to it; otherwise, I'd like to print to stdout. I could always just write an if statement to take care of this, but how would I write a case statement on fname and open the correct corresponding outputStream?
val outStream = fname match {
case Nil => ???
case _ => new java.io.FileOutputStream(new java.io.File(fname))
}
outStream.write( ... )
Thanks!

Why not rewrite the function as:
def writeToFileOrStdout(fname: Option[String] = None) = {
val outStream = fname match{
case Some(name) => new java.io.FileOutputStream(new java.io.File(name))
case None => System.out
}
...
}
It's always a good idea to use Option for an optional input as opposed to using null. That's basically what it's there for. In good scala code, you will not see explicit references to null.
In fact, your code doesn't even compile for me. Nil is used to represent an empty list, not a null or non supplied String.

To augment cmbaxter's response...
Mapping a String with a possible null value to Option[String] is trivial: Option(stringValue) will return None where stringValue is null, and Some(stringValue) where non-null.
Thus, you can either:
writeToFileOrStdout(Option(stringValue)), or
If you're stuck on String (and possibly a null value) as the parameter to writeToFileOrStdout, then internally use Option(fname) and match to what it returns::
def writeToFileOrStdout(fname: String = null) = {
val outStream = Option(fname) match{
case Some(name) => new java.io.FileOutputStream(new java.io.File(name))
case None => System.out
}
...
}

To further augment cmbaxter's response, you might consider writing this:
def filenameToOutputStream(name: String) =
new java.io.FileOutputStream(new java.io.File(name))
def writeToFileOrStdout(fname: Option[String] = None) = {
val outStream = fname map filenameToOutputStream getOrElse System.out
...
}
As the post Idiomatic Scala: Your Options Do Not Match suggests, this might be more idiomatic Scala.