I built a classic Hoffman code, with encoder and decoder. I noticed that I had a problem, I use code in "bitset", to compress the input file. But the "bitset" - does not decode all the files I send to, for example when I send a txt file, it works great, but when I send other files like BMP. It doesn't work.
Before I used bitset - the code worked - but without any compression - so I'm afraid the problem is with bitset.
The decoder I built is:
public void Decompress(String[] input_names, String[] output_names) {
HuffmanVerticle tree = new HuffmanVerticle();
tree = readTreeFile(output_names);
restoreInput(tree, output_names, input_names);
}
public static void restoreInput(HuffmanVerticle tree, String[] binary_names, String[] original_names) {
BitSet huffmanCodeBit;
try {
FileOutputStream to_original = new FileOutputStream(original_names[0]);
FileInputStream binary = new FileInputStream(binary_names[0]);
ObjectInputStream s = new ObjectInputStream(binary);
huffmanCodeBit = (BitSet) s.readObject();
System.out.println(huffmanCodeBit.toString());
int index = 0;
while(huffmanCodeBit.length() > index)
{
HuffmanVerticle tmp = tree;
while (!tmp.isNullTree())
{
boolean bit = huffmanCodeBit.get(index);
index++;
System.out.println(bit);
if (!bit)
tmp = tmp.left;
else
tmp = tmp.right;
}
to_original.write(tmp.character);
}
binary.close();
to_original.close();
} catch (Exception e) {
e.printStackTrace();
}
}
What am I missing here? Why doesn't the code work for certain files? I'm trying to run the code on some files but it doesn't work, the files that come back don't work.
The code does not work for bmp files at all, even after half an hour, for example txt files, it runs very fast.
Thank for your help.
Related
Am trying to encode pdf documents to base64, If it is less in number ( like 2000 documents) its working nicely. But am having 100k plus doucments to be encode.
Its take more time to encode all those files. Is there any better approach to encode large data set.?
Please find my current approach
String filepath=doc.getPath().concat(doc.getFilename());
file = new File(filepath);
if(file.exists() && !file.isDirectory()) {
try {
FileInputStream fileInputStreamReader = new FileInputStream(file);
byte[] bytes = new byte[(int) file.length()];
fileInputStreamReader.read(bytes);
encodedfile = new String(Base64.getEncoder().encodeToString(bytes));
fileInputStreamReader.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}
Try this:
Figure out how many files you need to encode.
int files = Files.list(Paths.get(directory)).count();
Split them up into a reasonable amount that a thread can handle in java. I.E) If you have 100k files to encode. Split it into 1000 lists of 1000, something like that.
int currentIndex = 0;
for (File file : filesInDir) {
if (fileMap.get(currentIndex).size() >= cap)
currentIndex++;
fileMap.get(currentIndex).add(file);
}
/** Its going to take a little more effort than this, but its the idea im trying to show you*/
Execute each worker thread one after another if the computers resources are available.
for (Integer key : fileMap.keySet()) {
new WorkerThread(fileMap.get(key)).start();
}
You can check the current resources available with:
public boolean areResourcesAvailable() {
return imNotThatNice();
}
/**
* Gets the resource utility instance
*
* #return the current instance of the resource utility
*/
private static OperatingSystemMXBean getInstance() {
if (ResourceUtil.instance == null) {
ResourceUtil.instance = ManagementFactory.getOperatingSystemMXBean();
}
return ResourceUtil.instance;
}
I want find out, if two audio files are same or one contains the other.
For this I use Fingerprint of musicg
byte[] firstAudio = readAudioFileData("first.mp3");
byte[] secondAudio = readAudioFileData("second.mp3");
FingerprintSimilarityComputer fingerprint =
new FingerprintSimilarityComputer(firstAudio, secondAudio);
FingerprintSimilarity fingerprintSimilarity = fingerprint.getFingerprintsSimilarity();
System.out.println("clip is found at " + fingerprintSimilarity.getScore());
to convert audio to byte array I use sound API
public static byte[] readAudioFileData(final String filePath) {
byte[] data = null;
try {
final ByteArrayOutputStream baout = new ByteArrayOutputStream();
final File file = new File(filePath);
final AudioInputStream audioInputStream = AudioSystem.getAudioInputStream(file);
byte[] buffer = new byte[4096];
int c;
while ((c = audioInputStream.read(buffer, 0, buffer.length)) != -1) {
baout.write(buffer, 0, c);
}
audioInputStream.close();
baout.close();
data = baout.toByteArray();
} catch (Exception e) {
e.printStackTrace();
}
return data;
}
but when I execute it, I became at fingerprint.getFingerprintsSimilarity() an Exception.
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 15999
at com.musicg.fingerprint.PairManager.getPairPositionList(PairManager.java:133)
at com.musicg.fingerprint.PairManager.getPair_PositionList_Table(PairManager.java:80)
at com.musicg.fingerprint.FingerprintSimilarityComputer.getFingerprintsSimilarity(FingerprintSimilarityComputer.java:71)
at Main.main(Main.java:42)
How can I compare 2 mp3 files with fingerprint in Java?
I never did any audio stuff in Java before, but I looked into your code briefly. I think that musicg only works for WAV files, not for MP3. Thus, you need to convert the files first. A web search reveals that you can e.g. use JLayer for that purpose. The corresponding code looks like this:
package de.scrum_master.so;
import com.musicg.fingerprint.FingerprintManager;
import com.musicg.fingerprint.FingerprintSimilarity;
import com.musicg.fingerprint.FingerprintSimilarityComputer;
import com.musicg.wave.Wave;
import javazoom.jl.converter.Converter;
import javazoom.jl.decoder.JavaLayerException;
public class Application {
public static void main(String[] args) throws JavaLayerException {
// MP3 to WAV
new Converter().convert("White Wedding.mp3", "White Wedding.wav");
new Converter().convert("Poison.mp3", "Poison.wav");
// Fingerprint from WAV
byte[] firstFingerPrint = new FingerprintManager().extractFingerprint(new Wave("White Wedding.wav"));
byte[] secondFingerPrint = new FingerprintManager().extractFingerprint(new Wave("Poison.wav"));
// Compare fingerprints
FingerprintSimilarity fingerprintSimilarity = new FingerprintSimilarityComputer(firstFingerPrint, secondFingerPrint).getFingerprintsSimilarity();
System.out.println("Similarity score = " + fingerprintSimilarity.getScore());
}
}
Of course you should make sure that you do not convert each file again whenever the program starts, i.e. you should check if the WAV files already exist. I skipped this step and reduced the sample code to a minimal working version.
For FingerprintSimilarityComputer(input1, input2), it suppose to take in the fingerprint of the loaded audio data and not the loaded audio data itself.
In your case, it should be:
// Convert your audio to wav using FFMpeg
Wave w1 = new Wave("first.wav");
Wave w2 = new Wave("second.wav");
FingerprintSimilarityComputer fingerprint =
new FingerprintSimilarityComputer(w1.getFingerprint(), w2.getFingerprint());
// print fingerprint.getFingerprintSimilarity()
Maybe I am missing a point, but if I understood you right, this should do:
byte[] firstAudio = readAudioFileData("first.mp3");
byte[] secondAudio = readAudioFileData("second.mp3");
byte[] smaller = firstAudio.length <= secondAudio.lenght ? firstAudio : secondAudio;
byte[] bigger = firstAudio.length > secondAudio.length ? firstAudio : secondAudio;
int ixS = 0;
int ixB = 0;
boolean contians = false;
for (; ixB<bigger.length; ixB++) {
if (smaller[ixS] == bigger[ixB]) {
ixS++;
if (ixS == smaller.lenght) {
contains = true;
break;
}
}
else {
ixS = 0;
}
}
if (contains) {
if (smaller.length == bigger.length) {
System.out.println("Both tracks are equal");
}
else {
System.out.println("The bigger track, fully contains the smaller track starting at byte: "+(ixB-smaller.lenght));
}
}
else {
System.out.println("No track completely contains the other track");
}
Stackoverflowers,
I am doing a simple project using Android smartphones to create 3D forms. I am using Android Processing to make a simple App.
My code makes a 3D shape and saves it as an .STL file. It works on my laptop and saves the .STL file, but in the App. version, I need it to save to the External storage/SD Card of my phone (HTC Sensation). It does not, because of the way the “save” function (writeSTL) in the Processing library I am using has been written.
I have posted for help here (my code more complete code is here too):
http://forum.processing.org/two/discussion/4809/exporting-geometry-stl-obj-dfx-modelbuilder-and-android
...and Marius Watz who wrote the library says that the writeSTL() code is pretty much standalone and the only thing missing is (or should be) replacing the code creating the output stream, which needs to be modified to work with Android. Basically, this line:
FileOutputStream out=(FileOutputStream)UIO.getOutputStream(p.sketchPath(filename));
I am not a great programmer in that I can usually get Processing to do what I need to do but no more; this problem has me beaten. I am looking for ideas for the correct code to replace the line:...
FileOutputStream out=(FileOutputStream)UIO.getOutputStream(p.sketchPath(filename));
...with something “Android-friendly”. Calling getExternalStorageDirectory() should work but I am at a loss to find the correct structure.
The code for the writeSTL function is below.
import java.io.*;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
/**
* Output binary STL file of mesh geometry.
* #param p Reference to PApplet instance
* #param filename Name of file to save to
*/
public void customWriteSTL(UGeometry geo, PApplet p, String filename) {
byte [] header;
ByteBuffer buf;
UFace f;
try {
if (!filename.toLowerCase().endsWith("stl")) filename+=".stl";
FileOutputStream out=(FileOutputStream)UIO.getOutputStream(p.sketchPath(filename));
buf = ByteBuffer.allocate(200);
header=new byte[80];
buf.get(header, 0, 80);
out.write(header);
buf.rewind();
buf.order(ByteOrder.LITTLE_ENDIAN);
buf.putInt(geo.faceNum);
buf.rewind();
buf.get(header, 0, 4);
out.write(header, 0, 4);
buf.rewind();
UUtil.logDivider("Writing STL '"+filename+"' "+geo.faceNum);
buf.clear();
header=new byte[50];
if (geo.bb!=null) UUtil.log(geo.bb.toString());
for (int i=0; i<geo.faceNum; i++) {
f=geo.face[i];
if (f.n==null) f.calcNormal();
buf.rewind();
buf.putFloat(f.n.x);
buf.putFloat(f.n.y);
buf.putFloat(f.n.z);
for (int j=0; j<3; j++) {
buf.putFloat(f.v[j].x);
buf.putFloat(f.v[j].y);
buf.putFloat(f.v[j].z);
}
buf.rewind();
buf.get(header);
out.write(header);
}
out.flush();
out.close();
UUtil.log("Closing '"+filename+"'. "+geo.faceNum+" triangles written.\n");
}
catch (Exception e) {
e.printStackTrace();
}
}
Any suggestions are gratefully received.
Thank you in advance.
There are a few ways of doing this - some that will just work and some that are proper ... as with all things Processing/Java. It's really not that different from regular Java though - the only quirk is the root SD path, and checking if it exists or not (note that some phones have "internal" rather than "external" storage (i.e. not removable/swappable), but Android should interpret these the same AFAIK.
In classic Java fashion, you should really be checking IF the SD Card is present beforehand... I use the following structure, taken from this answer by #kaolick
String state = Environment.getExternalStorageState();
if (state.equals(Environment.MEDIA_MOUNTED)) {
// Storage is available and writeable - ALL GOOD
} else if (state.equals(Environment.MEDIA_MOUNTED_READ_ONLY)) {
// Storage is only readable - RUH ROH
} else {
// Storage is neither readable nor writeable - ABORT
}
Note that he provides a full class for you to use, which is great, and has a few convenience functions.
The second thing you might want to look at is creating a custom directory on the SD Card of the device, probably in setup() - something like this:
try{
String dirName = "//sdcard//MyAppName";
File newFile = new File(dirName);
if(newFile.exists() && newFile.isDirectory()) {
println("Directory Exists... All Good");
}
else {
println("Directory Doesn't Exist... We're Making It");
newFile.mkdirs();
}
}
catch(Exception e) {
e.printStacktrace();
}
Of course, instead of HardCoding the Path name, you should do something like
String dirName = Environment.getExternalStorageDirectory().getAbsolutePath() + "/MyAppName";
instead...
Also, note that the above try/catch should go INSIDE the case statement of "if (state.equals(Environment.MEDIA_MOUNTED))" ... or should be wrapped in a separate function anc called from there.
Then, finally, saving it. If you wanted to use a BufferedWriter, it would look like this:
BufferedWriter writer = new BufferedWriter(new FileWriter(dirName, true));
writer.write(STL_STUFF);
writer.flush();
writer.close();
I've only use a FileOutputStream within a BufferedOutput Stream, and it looked like this:
try {
String fileName = "SOME_UNIQUE_NAME_PER_FILE";
String localFile = dirName + "/" +filename;
OutputStream output = new BufferedOutputStream(newFileOutputStream(localFile));
}
catch(Exception e) {
e.printStackTrace();
}
Finally, give my regards to Marius if you talk to him! ;-)
If I am continually parsing a log file and I detect that it has rotated, what is the best practice for handling this?
Update my internal hashmap to reflect a new filePointer (end of
rotated file)
Or should I update the hashmap to 0 so it can read the
rotated file from the start
My concern is that in the case of an anomaly in situation no. 2 mentioned above, I may indirectly parse a large file from the start and put significant load on the host.
However if I use no. 1 I may miss something critical that I was parsing the log file for.
This is the code that I've put together.
currentFilePointer = util.FileManagement.getLastFilePointerFromFile(file.getName());
lastFilePointer = Long.parseLong(lastReadFiles_.get(file.getName()).toString());
if (currentFilePointer < lastFilePointer) // file has grown
{
processLineByLine(file.getName(), currentFilePointer, lastFilePointer);
} else if (currentFilePointer > lastFilePointer) // file has been rotated
{
lastReadFiles_.put(file.getName(), currentFilePointer); // Option 1
}
From the code you posted, I can't see any Hashmap... Hard to tell which solution is better :)
Here's a snippet of code I used to "tail" a file, which also supports rotates (to be run in a separate Thread):
#Override
public void start() {
File file = ...
long filePointer = 0;
try {
while (running) {
Thread.sleep(updateInterval);
long len = file.length();
if (len < filePointer) {
// File was rotated
filePointer = 0;
} else if (len > filePointer) {
// File must have had something added to it!
RandomAccessFile raf = new RandomAccessFile(file, "r");
raf.seek(filePointer);
// Here you can read the next lines using for example a BufferedReader
filePointer = raf.getFilePointer();
raf.close();
}
}
} catch (Exception e) {
// ...
}
}
I think that your issues is how to manage log files in general. Such operations can be done using script language like bash or some general script language like python. An extensive tutorial for using java is managing Logs for the Java Subsystems.
I am beginning with Java and testng test cases.
I need to write a class, which reads data from a file and makes an in-memory data structure and uses this data structure for further processing. I would like to test, if this DS is being populated correctly. This would call for dumping the DS into a file and then comparing the input file with the dumped file. Is there any testNG assert available for file matching? Is this a common practice?
I think it would be better to compare the data itself not the written out data.
So I would write a method in the class to return this data structure (let's call it getDataStructure()) and then write a unit test to compare with the correct data.
This only needs a correct equals() method in your data structure class and do:
Assert.assertEquals(yourClass.getDataStructure(), correctData);
Of course if you need to write out the data structure to a file, then you can test the serialization and deserialization separately.
File compare/matching can be extracted to a utility method or something like that.
If you need it only for testing there are addons for jUnit
http://junit-addons.sourceforge.net/junitx/framework/FileAssert.html
If you need file compare outside the testing environment you can use this simple function
public static boolean fileContentEquals(String filePathA, String filePathB) throws Exception {
if (!compareFilesLength(filePathA, filePathB)) return false;
BufferedInputStream streamA = null;
BufferedInputStream streamB = null;
try {
File fileA = new File(filePathA);
File fileB = new File(filePathB);
streamA = new BufferedInputStream(new FileInputStream(fileA));
streamB = new BufferedInputStream(new FileInputStream(fileB));
int chunkSizeInBytes = 16384;
byte[] bufferA = new byte[chunkSizeInBytes];
byte[] bufferB = new byte[chunkSizeInBytes];
int totalReadBytes = 0;
while (totalReadBytes < fileA.length()) {
int readBytes = streamA.read(bufferA);
streamB.read(bufferB);
if (readBytes == 0) break;
MessageDigest digestA = MessageDigest.getInstance(CHECKSUM_ALGORITHM);
MessageDigest digestB = MessageDigest.getInstance(CHECKSUM_ALGORITHM);
digestA.update(bufferA, 0, readBytes);
digestB.update(bufferB, 0, readBytes);
if (!MessageDigest.isEqual(digestA.digest(), digestB.digest()))
{
closeStreams(streamA, streamB);
return false;
}
totalReadBytes += readBytes;
}
closeStreams(streamA, streamB);
return true;
} finally {
closeStreams(streamA, streamB);
}
}
public static void closeStreams(Closeable ...streams) {
for (int i = 0; i < streams.length; i++) {
Closeable stream = streams[i];
closeStream(stream);
}
}
public static boolean compareFilesLength(String filePathA, String filePathB) {
File fileA = new File(filePathA);
File fileB = new File(filePathB);
return fileA.length() == fileB.length();
}
private static void closeStream(Closeable stream) {
try {
stream.close();
} catch (IOException e) {
// ignore exception
}
}
Your choice, but having an utility class with that functionality that can be reused is better imho.
Good luck and have fun.
Personally I would do the opposite. Surely you need a way to compare two of these data structure in the Java world - so the test would read from the file, build the DS, do its processing, and then assert it's equal to an "expected" DS you set up in your test.
(using JUnit4)
#Test
public void testProcessingDoesWhatItShould() {
final DataStructure original = readFromFile(filename);
final DataStructure actual = doTheProcessingYouNeedToDo(original);
final DataStructure expected = generateMyExpectedResult();
Assert.assertEquals("data structure", expected, actual);
}
If this DS is a simple Java Bean. then you can use EqualsBuilder from Apache Commons to compare 2 objects.
compare bytes loaded from file system and bytes you are going to write file system
pseudo code
byte[] loadedBytes = loadFileContentFromFile(file) // maybe apache commons IOUtils.toByteArray(InputStream input)
byte[] writeBytes = constructBytesFromDataStructure(dataStructure)
Assert.assertTrue(java.util.Arrays.equals(writeBytes ,loadedBytes));