I'm using metadata-extractor to write a Java application that organizes images and finds duplicates. The API is great, but there's something I cannot figure out.
Suppose I have two JPG images. These images, visually, are exactly the same (i.e. same pixel-wise). However, maybe something within the metadata encapsulated in the file differs.
If I calculate MD5 hashes on each complete file, I will get two different hashes. However, I want to calculate a hash of only the image/pixel data, which would yield the same hash for both files.
So - Is there a way to pull out the raw image/pixel data from the JPG using metadata-extractor so that I can calculate my hash on this?
Also, is Javadoc available for this API? I cannot seem to find it.
You can achieve this using the library's JpegSegmentReader class. It'll let you pull out the JPEG segments that contain image data and ignore metadata segments.
I discussed this technique in another answer and the asker indicated they had success with the approach.
This would actually make a nice sample application for the library. If you come up with something and feel like sharing, please do.
my main activity has over 400 Lines Of Code and contains numerous methods which deals with pdf generations, making the page dynamic, and other condition checks..
i was wondering if i should make it all into different class files.. each specific for its task.. thereby creating at least 3-4 different classes..
so my query is
1) will this approach make the app faster..
2) will this increase the app's size drastically??
also is there anyway to reduce the app's size?
i have deleted all unnecessary pics, xml's & assests..
i just want the size to be below 5mb..
Thanks in advance..
1) No. This won't noticably affect the app's speed.
2) No. The compiled code is actually the smallest part of an APK. Most of the size is caused by resources.
A way to reduce your app's size would be to optimize your compression on the images that are contained in your app. Also, you might be able to draw some of the images in code, as primitives such as lines, circles, squares, depending on what's on the images.
These points may help you :
Java class files increase the app size which you can ignore because it is very very less. So that point you no need to be worried.
As you have mentioned you deleted images and all. Those things actually increase the app size. So delete those as many as possible is not being used.
Creating more classes which will be specific for their purposes, that is OOPs concept and it is very much recommended. So if you want to modify something in future iy will be easy for you find the code as well your code will be modified in that particular class made for that sole pupose.
Hope it will help.
1) will this approach make the app faster?
Faster, can't comment. However, That approach will provide you an opportunity to learn how to make good design and will help you in future maintainability and extension of your project.
2) will this increase the app's size drastically?
Not at all.
I have a application that reads a PDF, transform the content to collection of TIF files, and send them to Glass Fish Server for saving.
Usually there are 1-5 pages and it works nice, but when I got a input file with 100+ pages...
it throws error on the transfer.
Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
Putting more resources is not a good option in my case, so I m looking for a way to optimize it somehow.
I store the data in:
HashMap<TifProfile, List<byte[]>
Is there a better way to store or send them ?
EDIT
I did some tests and the final collections for PDF with 80 pages
has size over 280mb (240 tiffs with different settings inside)
Well you don't give us much to go on, but it seems clear to me that storing 100+ high resolution TIFF-encoded images in memory will very quickly exhaust any resources you have available.
It might be better to load it into the database in batches, e.g. just handle 5 pages at a time.
Alternatively depending on your JDBC driver, you might be able to stream the image data into a JDBC BLOB so you wont have to cache it in memory. Heres some food for thought...
http://artofsystems.blogspot.co.uk/2008/07/mysql-postgresql-and-blob-streaming.html
I need the functionality to show large images on a Java Applet without increasing the default heap space. I'm getting the "Out of memory" exception when reading the large image using ImageIO.read() into a BufferedImage (what was expected to happen).
To solve this issue, the first thing that comes to mind is to divide the large image into lesser parts.
My question is:
Does anyone have some tips on which direction to take to implement this functionality (which includes showing the correct part when scrolling the image)? Does Java have anything built-in to support this? Any examples?
You must work with parts. The simplest is to have the server provider smallish tiles you Can request As needed.
If not you must dó it yourself in the applet. Load the image - split it in tiles and have each tile reencode itself to a bytestream e.g as jpeg. Attach the rendered tile as a soft reference so it Will be garbage collected as late as possible. If the rendered tile has been gc'ed then decode the jpeg bytestream.
If you have a LOT then even consider serializong tiles to local storage.
So I have a "large" number of "very large" ASCII files of numerical data (gigabytes altogether), and my program will need to process the entirety of it sequentially at least once.
Any advice on storing/loading the data? I've thought of converting the files to binary to make them smaller and for faster loading.
Should I load everything into memory all at once?
If not, is opening what's a good way of loading the data partially?
What are some Java-relevant efficiency tips?
So then what if the processing requires jumping around in the data for multiple files and multiple buffers? Is constant opening and closing of binary files going to become expensive?
I'm a big fan of 'memory mapped i/o', aka 'direct byte buffers'. In Java they are called Mapped Byte Buffers are are part of java.nio. (Basically, this mechanism uses the OS's virtual memory paging system to 'map' your files and present them programmatically as byte buffers. The OS will manage moving the bytes to/from disk and memory auto-magically and very quickly.
I suggest this approach because a) it works for me, and b) it will let you focus on your algorithm and let the JVM, OS and hardware deal with the performance optimization. All to frequently, they know what is best more so than us lowly programmers. ;)
How would you use MBBs in your context? Just create an MBB for each of your files and read them as you see fit. You will only need to store your results. .
BTW: How much data are you dealing with, in GB? If it is more than 3-4GB, then this won't work for you on a 32-bit machine as the MBB implementation is defendant on the addressable memory space by the platform architecture. A 64-bit machine & OS will take you to 1TB or 128TB of mappable data.
If you are thinking about performance, then know Kirk Pepperdine (a somewhat famous Java performance guru.) He is involved with a website, www.JavaPerformanceTuning.com, that has some more MBB details: NIO Performance Tips and other Java performance related things.
You might want to have a look at the entries in the Wide Finder Project (do a google search for "wide finder" java).
The Wide finder involves reading over lots of lines in log files, so look at the Java implementations and see what worked and didn't work there.
You could convert to binary, but then you have 1+ something copies of the data, if you need to keep the original around.
It may be practical to build some kind of index on top of your original ascii data, so that if you need to go through the data again you can do it faster in subsequent times.
To answer your questions in order:
Should I load everything into memory all at once?
Not if don't have to. for some files, you may be able to, but if you're just processing sequentially, just do some kind of buffered read through the things one by one, storing whatever you need along the way.
If not, is opening what's a good way of loading the data partially?
BufferedReaders/etc is simplest, although you could look deeper into FileChannel/etc to use memorymapped I/O to go through windows of the data at a time.
What are some Java-relevant efficiency tips?
That really depends on what you're doing with the data itself!
Without any additional insight into what kind of processing is going on, here are some general thoughts from when I have done similar work.
Write a prototype of your application (maybe even "one to throw away") that performs some arbitrary operation on your data set. See how fast it goes. If the simplest, most naive thing you can think of is acceptably fast, no worries!
If the naive approach does not work, consider pre-processing the data so that subsequent runs will run in an acceptable length of time. You mention having to "jump around" in the data set quite a bit. Is there any way to pre-process that out? Or, one pre-processing step can be to generate even more data - index data - that provides byte-accurate location information about critical, necessary sections of your data set. Then, your main processing run can utilize this information to jump straight to the necessary data.
So, to summarize, my approach would be to try something simple right now and see what the performance looks like. Maybe it will be fine. Otherwise, look into processing the data in multiple steps, saving the most expensive operations for infrequent pre-processing.
Don't "load everything into memory". Just perform file accesses and let the operating system's disk page cache decide when you get to actually pull things directly out of memory.
This depends a lot on the data in the file. Big mainframes have been doing sequential data processing for a long time but they don't normally use random access for the data. They just pull it in a line at a time and process that much before continuing.
For random access it is often best to build objects with caching wrappers which know where in the file the data they need to construct is. When needed they read that data in and construct themselves. This way when memory is tight you can just start killing stuff off without worrying too much about not being able to get it back later.
You really haven't given us enough info to help you. Do you need to load each file in its entiretly in order to process it? Or can you process it line by line?
Loading an entire file at a time is likely to result in poor performance even for files that aren't terribly large. Your best bet is to define a buffer size that works for you and read/process the data a buffer at a time.
I've found Informatica to be an exceptionally useful data processing tool. The good news is that the more recent versions even allow Java transformations. If you're dealing with terabytes of data, it might be time to pony up for the best-of-breed ETL tools.
I'm assuming you want to do something with the results of the processing here, like store it somewhere.
If your numerical data is regularly sampled and you need to do random access consider to store them in a quadtree.
I recommend strongly leveraging Regular Expressions and looking into the "new" IO nio package for faster input. Then it should go as quickly as you can realistically expect Gigabytes of data to go.
If at all possible, get the data into a database. Then you can leverage all the indexing, caching, memory pinning, and other functionality available to you there.
If you need to access the data more than once, load it into a database. Most databases have some sort of bulk loading utility. If the data can all fit in memory, and you don't need to keep it around or access it that often, you can probably write something simple in Perl or your favorite scripting language.