I m trying to generate SHA-1 in java of a file's content then read it back. one hex digit becomes changed, and I don't know why?
codes:
writing in file
//writer:
import java.util.*;
import java.io.*;
import java.security.*;
public class Writer{
public static void main(String []args)throws Exception{
InputStream is = new FileInputStream("input.txt");
PrintStream os=new PrintStream(new File("out.txt"));
byte[] buffer = new byte[1024];
String str;
MessageDigest complete = MessageDigest.getInstance("SHA-1");
MessageDigest partial = MessageDigest.getInstance("SHA-1");
int numRead;
do {
numRead = is.read(buffer);
if (numRead > 0) {
complete.update(buffer, 0, numRead);
partial.update(buffer,0,numRead);
byte []digest=partial.digest();
StringBuilder sb = new StringBuilder();
for (int i = 0; i < digest.length; i++) {
sb.append(String.format("%x", digest[i]));
}
System.out.println(sb.toString());
str=new String(digest);
os.println(str);
partial.reset();
}
} while (numRead != -1);
byte []digest=complete.digest();
StringBuilder sb = new StringBuilder();
for (int i = 0; i < digest.length; i++) {
sb.append(String.format("%x", digest[i]));
}
System.out.println(sb.toString());
str=new String(digest);
os.println(str);
is.close();
os.close();
}
}
code to readfile
//Reader:
import java.util.*;
import java.io.*;
import java.security.*;
public class Reader{
public static void main(String []args)throws Exception{
BufferedReader is = new BufferedReader(new FileReader("out.txt"));
String str;
while((str=is.readLine())!=null)
{
//System.out.println(str);
byte []digest=str.getBytes();
StringBuilder sb = new StringBuilder();
for (int i = 0; i < digest.length; i++) {
sb.append(String.format("%x", digest[i]));
}
System.out.println(sb.toString());
}
is.close();
}
}
I used another code as input.txt. it generates 5 outputlines. first 4 lines reading and converting is okay. last line came with a changed hex digit
inputfile:input.txt
/*
* To change this template, choose Tools | Templates
* and open the template in the editor.
*/
/**
*
* #author biborno
*/
import java.io.*;
import java.security.*;
import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;
import java.util.SortedMap;
import java.util.TreeMap;
public class Torrent {
private static void encodeObject(Object o, OutputStream out) throws IOException {
if (o instanceof String)
encodeString((String)o, out);
else if (o instanceof Map)
encodeMap((Map)o, out);
else if (o instanceof byte[])
encodeBytes((byte[])o, out);
else if (o instanceof Number)
encodeLong(((Number) o).longValue(), out);
else
throw new Error("Unencodable type");
}
private static void encodeLong(long value, OutputStream out) throws IOException {
out.write('i');
out.write(Long.toString(value).getBytes("US-ASCII"));
out.write('e');
}
private static void encodeBytes(byte[] bytes, OutputStream out) throws IOException {
out.write(Integer.toString(bytes.length).getBytes("US-ASCII"));
out.write(':');
out.write(bytes);
}
private static void encodeString(String str, OutputStream out) throws IOException {
encodeBytes(str.getBytes("UTF-8"), out);
}
private static void encodeMap(Map<String,Object> map, OutputStream out) throws IOException{
// Sort the map. A generic encoder should sort by key bytes
SortedMap<String,Object> sortedMap = new TreeMap<String, Object>(map);
out.write('d');
for(Entry<String, Object> e : sortedMap.entrySet()) {
encodeString(e.getKey(), out);
encodeObject(e.getValue(), out);
}
out.write('e');
}
private static byte[] hashPieces(File file, int pieceLength) throws IOException {
MessageDigest sha1;
try {
sha1 = MessageDigest.getInstance("SHA");
} catch (NoSuchAlgorithmException e) {
throw new Error("SHA1 not supported");
}
InputStream in = new FileInputStream(file);
ByteArrayOutputStream pieces = new ByteArrayOutputStream();
byte[] bytes = new byte[pieceLength];
int pieceByteCount = 0, readCount = in.read(bytes, 0, pieceLength);
while (readCount != -1) {
pieceByteCount += readCount;
sha1.update(bytes, 0, readCount);
if (pieceByteCount == pieceLength) {
pieceByteCount = 0;
pieces.write(sha1.digest());
}
readCount = in.read(bytes, 0, pieceLength-pieceByteCount);
}
in.close();
if (pieceByteCount > 0)
pieces.write(sha1.digest());
return pieces.toByteArray();
}
public static void createTorrent(File file, File sharedFile, String announceURL) throws IOException {
final int pieceLength = 512*1024;
Map<String,Object> info = new HashMap<String,Object>();
info.put("name", sharedFile.getName());
info.put("length", sharedFile.length());
info.put("piece length", pieceLength);
info.put("pieces", hashPieces(sharedFile, pieceLength));
Map<String,Object> metainfo = new HashMap<String,Object>();
metainfo.put("announce", announceURL);
metainfo.put("info", info);
OutputStream out = new FileOutputStream(file);
encodeMap(metainfo, out);
out.close();
}
public static void main(String[] args) throws Exception {
createTorrent(new File("C:\\Documents and Settings\\biborno\\Desktop\\Test.text.torrent"), new File("C:\\Documents and Settings\\biborno\\Desktop\\Test.text"), "http://example.com/announce");
}
}
output of writer:
f1556b25a651c3dc4831871ab3fce2d239c7f0d4
4ed5c741dbf78059fef3733340efcf9b16b4594f
e2b66127e6f25648e9e2d21cb18e26d4a541a17
26c213a35b3f2ccae28ad5e3b98b5a02386c368
e7d152cd8f5f22949abbc11eb049451c511df07b
out.txt:
ñUk%¦QÃÜH1‡³üâÒ9ÇðÔ
NÕÇAÛ÷€Yþós3#ïÏ›´YO
â¶a'æòVHéâÒ±Ž&Ô¥
&£[òÌ®(^;˜µ #†Ãh
çÑRÍ?_"”š»Á°IEQð{
Reader's Output
f1556b25a651c3dc4831871ab3fce2d239c7f0d4
4ed5c741dbf78059fef3733340efcf9b16b4594f
e2b66127e6f25648e9e2d21cb18e26d4a541a17
26c213a35b3f2ccae28ad5e3b98b5a02386c368
e7d152cd3f5f22949abbc11eb049451c511df07b
Related
I have two text files that are more than 600MB and I want to compare the content of them if they are the same (Ignoring any space at the end or the start of any line in it i.e. trim() each line).
I am thinking of reading each line of them as a string and then trim it and compare it.
Is there is a better idea and if not what is the fastest implementation to this idea?
Thanks in advance.
If you want to compare whether the files are consistent, please calculate the file md5 value to compare：
import java.io.FileInputStream;
import java.io.InputStream;
import java.math.BigInteger;
import java.security.MessageDigest;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.Date;
public class MainServer {
public static void main(String[] args) {
String filePath1 = "D:\\Download\\a.mp3";
String filePath2 = "D:\\Download\\b.mp3";
String file1_md5 = md5HashCode(filePath1);
String file2_md5 = md5HashCode(filePath2);
System.out.println(file1_md5);
System.out.println(file2_md5);
if(file1_md5.equals(file2_md5)){
System.out.println("Two files are the same ");
}
}
/**
* get file md5 value
*/
public static String md5HashCode(String filePath) {
try {
InputStream fis =new FileInputStream(filePath);
MessageDigest md = MessageDigest.getInstance("MD5");
byte[] buffer = new byte[1024];
int length = -1;
while ((length = fis.read(buffer, 0, 1024)) != -1) {
md.update(buffer, 0, length);
}
fis.close();
byte[] md5Bytes = md.digest();
BigInteger bigInt = new BigInteger(1, md5Bytes);
return bigInt.toString(16);
} catch (Exception e) {
e.printStackTrace();
return "";
}
}
}
If you need to read each line of the file for comparison：
List<String> file1_lines = null;
List<String> file2_lines = null;
try {
file1_lines = Files.readAllLines(Paths.get("D:/a.txt"), StandardCharsets.UTF_8);
file2_lines = Files.readAllLines(Paths.get("D:/b.txt"), StandardCharsets.UTF_8);
} catch (IOException e) {
e.printStackTrace();
}
for (int i = 0; i < file1_lines.size(); i++) {
String file1_line = file1_lines.get(i).trim();
String file2_line = file2_lines.get(i).trim();
if (file1_line.equals(file2_line)) {
//do some
}
}
i have following code for comparing the md5 hash values for two folder but i need to show the list of files and the hash value of each file. can anyone please help me out with this. i just need to get hash value for one folder only.
package com.example;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.security.MessageDigest;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Set;
public class Compare
{
//This can be any folder locations which you want to compare
File dir1 = new File("/Users/Samip/Desktop/crypto");
File dir2 = new File("/Users/Samip/Desktop/crypto1");
public static void main(String ...args)
{
Compare compare = new Compare();
try
{
compare.getDiff(compare.dir1,compare.dir2);
}
catch(IOException ie)
{
ie.printStackTrace();
}
}
public void getDiff(File dirA, File dirB) throws IOException
{
File[] fileList1 = dirA.listFiles();
File[] fileList2 = dirB.listFiles();
Arrays.sort(fileList1);
Arrays.sort(fileList2);
HashMap<String, File> map1;
if(fileList1.length < fileList2.length)
{
map1 = new HashMap<String, File>();
for(int i=0;i<fileList1.length;i++)
{
map1.put(fileList1[i].getName(),fileList1[i]);
}
compareNow(fileList2, map1);
}
else
{
map1 = new HashMap<String, File>();
for(int i=0;i<fileList2.length;i++)
{
map1.put(fileList2[i].getName(),fileList2[i]);
}
compareNow(fileList1, map1);
}
}
public void compareNow(File[] fileArr, HashMap<String, File> map) throws IOException
{
for(int i=0;i<fileArr.length;i++)
{
String fName = fileArr[i].getName();
File fComp = map.get(fName);
map.remove(fName);
if(fComp!=null)
{
if(fComp.isDirectory())
{
getDiff(fileArr[i], fComp);
}
else
{
String cSum1 = checksum(fileArr[i]);
String cSum2 = checksum(fComp);
if(!cSum1.equals(cSum2))
{
System.out.println(fileArr[i].getName()+"\t\t"+ "different");
}
else
{
System.out.println(fileArr[i].getName()+"\t\t"+"identical");
}
}
}
else
{
if(fileArr[i].isDirectory())
{
traverseDirectory(fileArr[i]);
}
else
{
System.out.println(fileArr[i].getName()+"\t\t"+"only in "+fileArr[i].getParent());
}
}
}
Set<String> set = map.keySet();
Iterator<String> it = set.iterator();
while(it.hasNext())
{
String n = it.next();
File fileFrmMap = map.get(n);
map.remove(n);
if(fileFrmMap.isDirectory())
{
traverseDirectory(fileFrmMap);
}
else
{
System.out.println(fileFrmMap.getName() +"\t\t"+"only in "+ fileFrmMap.getParent());
}
}
}
public void traverseDirectory(File dir)
{
File[] list = dir.listFiles();
for(int k=0;k<list.length;k++)
{
if(list[k].isDirectory())
{
traverseDirectory(list[k]);
}
else
{
System.out.println(list[k].getName() +"\t\t"+"only in "+ list[k].getParent());
}
}
}
public String checksum(File file)
{
try
{
InputStream fin = new FileInputStream(file);
java.security.MessageDigest md5er = MessageDigest.getInstance("MD5");
byte[] buffer = new byte[1024];
int read;
do
{
read = fin.read(buffer);
if (read > 0)
md5er.update(buffer, 0, read);
} while (read != -1);
fin.close();
byte[] digest = md5er.digest();
if (digest == null)
return null;
String strDigest = "0x";
for (int i = 0; i < digest.length; i++)
{
strDigest += Integer.toString((digest[i] & 0xff) + 0x100, 16).substring(1).toUpperCase();
}
return strDigest;
}
catch (Exception e)
{
return null;
}
}
}
In you main method, instead using Compare.getDiff(dir1, dir2) you want to
Get a file listing of your target directory
Invoke Compare.checksum(file) on each file and print the result
Looks like you have all the code, you just need to reshape it a little.
Consider this example. The hash-generating code has been taken from your previous question - same goes for the file-iteration code. You just replace that folder to match your.
import java.io.*;
import java.security.MessageDigest;
public class PrintChecksums {
public static void main(String[] args) {
String sourceDir = "/Users/Jan/Desktop/Folder1";
try {
new PrintChecksums().printHashs(new File(sourceDir));
} catch (Exception e) {
e.printStackTrace();
}
}
private void printHashs(File sourceDir) throws Exception {
for (File f : sourceDir.listFiles()) {
String hash = createHash(f); // That you almost have
System.out.println(f.getAbsolutePath() + " / Hashvalue: " + hash);
}
}
public String createHash(File datafile) throws Exception {
// SNIP - YOUR CODE BEGINS
MessageDigest md = MessageDigest.getInstance("SHA1");
FileInputStream fis = new FileInputStream(datafile);
byte[] dataBytes = new byte[1024];
int nread = 0;
while ((nread = fis.read(dataBytes)) != -1) {
md.update(dataBytes, 0, nread);
}
byte[] mdbytes = md.digest();
// convert the byte to hex format
StringBuffer sb = new StringBuffer("");
for (int i = 0; i < mdbytes.length; i++) {
sb.append(Integer.toString((mdbytes[i] & 0xff) + 0x100, 16).substring(1));
}
// SNAP - YOUR CODE ENDS
return sb.toString();
}
}
Please have a look at the below code. I have added a function printCheckSum() which iterates though directory, scans each file and prints its hash value.
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.security.MessageDigest;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Set;
public class Compare
{
//This can be any folder locations which you want to compare
File dir1 = new File("D:\\dir1");
File dir2 = new File("D:\\dir2");
public static void main(String ...args)
{
Compare compare = new Compare();
try
{
compare.printCheckSum(compare.dir1);
}
catch(IOException ie)
{
ie.printStackTrace();
}
}
public void getDiff(File dirA, File dirB) throws IOException
{
File[] fileList1 = dirA.listFiles();
File[] fileList2 = dirB.listFiles();
Arrays.sort(fileList1);
Arrays.sort(fileList2);
HashMap<String, File> map1;
if(fileList1.length < fileList2.length)
{
map1 = new HashMap<String, File>();
for(int i=0;i<fileList1.length;i++)
{
map1.put(fileList1[i].getName(),fileList1[i]);
}
compareNow(fileList2, map1);
}
else
{
map1 = new HashMap<String, File>();
for(int i=0;i<fileList2.length;i++)
{
map1.put(fileList2[i].getName(),fileList2[i]);
}
compareNow(fileList1, map1);
}
}
public void compareNow(File[] fileArr, HashMap<String, File> map) throws IOException
{
for(int i=0;i<fileArr.length;i++)
{
String fName = fileArr[i].getName();
File fComp = map.get(fName);
map.remove(fName);
if(fComp!=null)
{
if(fComp.isDirectory())
{
getDiff(fileArr[i], fComp);
}
else
{
String cSum1 = checksum(fileArr[i]);
String cSum2 = checksum(fComp);
if(!cSum1.equals(cSum2))
{
System.out.println(fileArr[i].getName()+"\t\t"+ "different");
}
else
{
System.out.println(fileArr[i].getName()+"\t\t"+"identical");
}
}
}
else
{
if(fileArr[i].isDirectory())
{
traverseDirectory(fileArr[i]);
}
else
{
System.out.println(fileArr[i].getName()+"\t\t"+"only in "+fileArr[i].getParent());
}
}
}
Set<String> set = map.keySet();
Iterator<String> it = set.iterator();
while(it.hasNext())
{
String n = it.next();
File fileFrmMap = map.get(n);
map.remove(n);
if(fileFrmMap.isDirectory())
{
traverseDirectory(fileFrmMap);
}
else
{
System.out.println(fileFrmMap.getName() +"\t\t"+"only in "+ fileFrmMap.getParent());
}
}
}
public void traverseDirectory(File dir)
{
File[] list = dir.listFiles();
for(int k=0;k<list.length;k++)
{
if(list[k].isDirectory())
{
traverseDirectory(list[k]);
}
else
{
System.out.println(list[k].getName() +"\t\t"+"only in "+ list[k].getParent());
}
}
}
public String checksum(File file)
{
try
{
InputStream fin = new FileInputStream(file);
java.security.MessageDigest md5er = MessageDigest.getInstance("MD5");
byte[] buffer = new byte[1024];
int read;
do
{
read = fin.read(buffer);
if (read > 0)
md5er.update(buffer, 0, read);
} while (read != -1);
fin.close();
byte[] digest = md5er.digest();
if (digest == null)
return null;
String strDigest = "0x";
for (int i = 0; i < digest.length; i++)
{
strDigest += Integer.toString((digest[i] & 0xff) + 0x100, 16).substring(1).toUpperCase();
}
return strDigest;
}
catch (Exception e)
{
return null;
}
}
public void printCheckSum(File dir) throws IOException{
File[] fileList = dir.listFiles();
for(File file : fileList){
if(file.isDirectory()){
printCheckSum(file);
}else
System.out.println(file.getName() +"\t :: \t" + checksum(file));
}
}
}
Hope this helps. Cheers!
this code couldn't find the files that the buffered reader is supposed to read from it and i have the files in the src folder in eclipse project and it still doesn't read from file so does anybody have any idea about what the problem is.
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.math.*;
import java.util.ArrayList;
public class Encrypt {
public static ArrayList<String> data = new ArrayList<String>();
public static BigInteger [] keys = new BigInteger[3];
public static BigInteger n;
public static double e;
public static BigInteger d;
public static String line;
public static String result;
public static String [] temp;
public static BigInteger tempVar;
public static BigInteger tempResult;
public static int tempVar2;
public static void encryption(ArrayList<String> data) throws IOException{
for (int i = 0; i<data.size(); i++){
if(data.get(i)!= null){
temp = new String[data.get(i).split(" ").length];
temp = data.get(i).split(" ");
for(int j = 0; j<temp.length;j++){
for (int k = 0; k< temp[j].length(); k++){
tempVar2 = (int)temp[j].charAt(k);
tempVar=BigInteger.valueOf((long)Math.pow(tempVar2,e));
tempResult = (tempVar.remainder(n));
result =""+ tempResult;
LogEncrypt(result);
}
}
}
}
}
public static void read() throws IOException{
try {
BufferedReader br = new BufferedReader(new FileReader("plainText.txt"));
System.out.println(br.ready());
while ((line = br.readLine()) != null) {
data.add(br.readLine());
}
System.out.println("done with text");
} catch (FileNotFoundException e) {
System.out.println("please add the text file");
e.printStackTrace();
}
try {
BufferedReader ba = new BufferedReader(new FileReader("Key.txt"));
System.out.println(ba.ready());
int i =0;
while ((line = ba.readLine()) != null) {
keys[i] = new BigInteger(ba.readLine());
i++;
}
n = keys[0];
e = keys[1].doubleValue();
d = keys[2];
System.out.println("done with key");
} catch (FileNotFoundException e) {
System.out.println("please add the key file");
e.printStackTrace();
}
}
public static void LogEncrypt(String result) throws IOException {
BufferedWriter out = new BufferedWriter(new FileWriter("output.txt"));
try {
out.write(result);
out.newLine();
} catch(IOException e1) {
System.out.println("Error during reading/writing");
} finally {
out.close();
}
}
public static void main(String[]args) throws IOException{
read();
encryption(data);
}
}
Put the file outside of the src, or at least add "src/" to the file location
Merged with non-static method encode(byte[]) cannot be referenced from a static context [closed].
package com.cordys.report;
import java.io.FileInputStream;
import org.apache.commons.codec.binary.Base64;
public class Encode
{
public static String encodeFileStream(String filePath) //file path ex : C:\Program Files\Cordys\Web\reports\I0001180.pdf
{
try
{
FileInputStream fin = new FileInputStream("E:/CSS Document/Test.pdf");
StringBuffer sb = new StringBuffer();
int lineLength = 72;
byte[] buf = new byte[lineLength/4*3];
while (true)
{
int len = fin.read(buf);
if (len <= 0) {
break;
}
//new Base64().encode(byte);
sb.append(Base64.encode(buf));
//sb.append(Base64.encodeBase64(buf));
return sb.toString();
}
}
catch(Exception e)
{
return e.getMessage();
}
}
}
this time someone should please relpy
i am struggling with running my code using distributed cahe. i have already the files on hdfs but when i run this code :
import java.awt.image.BufferedImage;
import java.awt.image.DataBufferByte;
import java.awt.image.Raster;
import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.DataInputStream;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URISyntaxException;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.imageio.ImageIO;
import org.apache.hadoop.filecache.*;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import java.lang.String;
import java.lang.Runtime;
import java.net.URI;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
public class blur2 {
public static class BlurMapper extends MapReduceBase implements Mapper<Text, BytesWritable, LongWritable, BytesWritable>
{
OutputCollector<LongWritable, BytesWritable> goutput;
int IMAGE_HEIGHT = 240;
int IMAGE_WIDTH = 320;
public BytesWritable Gmiu;
public BytesWritable Gsigma;
public BytesWritable w;
byte[] bytes = new byte[IMAGE_HEIGHT*IMAGE_WIDTH*3];
public BytesWritable emit = new BytesWritable(bytes);
int count = 0;
int initVar = 125;
public LongWritable l = new LongWritable(1);
byte[] byte1 = new byte[IMAGE_HEIGHT*IMAGE_WIDTH];
byte[] byte2 = new byte[IMAGE_HEIGHT*IMAGE_WIDTH];
byte[] byte3 = new byte[IMAGE_HEIGHT*IMAGE_WIDTH];
public void map(Text key, BytesWritable file,OutputCollector<LongWritable, BytesWritable> output, Reporter reporter) throws IOException {
goutput = output;
BufferedImage img = ImageIO.read(new ByteArrayInputStream(file.getBytes()));
Raster ras=img.getData();
DataBufferByte db= (DataBufferByte)ras.getDataBuffer();
byte[] data = db.getData();
if(count==0){
for(int i=0;i<IMAGE_HEIGHT*IMAGE_WIDTH;i++)
{
byte1[i]=20;
byte2[i]=125;
}
Gmiu = new BytesWritable(data);
Gsigma = new BytesWritable(byte1);
w = new BytesWritable(byte2);
count++;
}
else{
byte1 = Gmiu.getBytes();
byte2 = Gsigma.getBytes();
byte3 = w.getBytes();
for(int i=0;i<IMAGE_HEIGHT*IMAGE_WIDTH;i++)
{
byte pixel = data[i];
Double tempmiu=new Double(0.0);
Double tempsig=new Double(0.0);
double temp1=0.0; double alpha = 0.05;
tempmiu = (1-alpha)*byte1[i] + alpha*pixel;
temp1=temp1+(pixel-byte1[i])*(pixel-byte1[i]);
tempsig=(1-alpha)*byte2[i]+ alpha*temp1;
byte1[i] = tempmiu.byteValue();
byte2[i]= tempsig.byteValue();
Double w1=new Double((1-alpha)*byte3[i]+alpha*100);
byte3[i] = w1.byteValue();
}
Gmiu.set(byte1,0,IMAGE_HEIGHT*IMAGE_WIDTH);
Gsigma.set(byte2,0,IMAGE_HEIGHT*IMAGE_WIDTH);
w.set(byte3,0,IMAGE_HEIGHT*IMAGE_WIDTH);
}
byte1 = Gsigma.getBytes();
for(int i=0;i<IMAGE_HEIGHT*IMAGE_WIDTH;i++)
{
bytes[i]=byte1[i];
}
byte1 = Gsigma.getBytes();
for(int i=0;i<IMAGE_HEIGHT*IMAGE_WIDTH;i++)
{
bytes[IMAGE_HEIGHT*IMAGE_WIDTH+i]=byte1[i];
}
byte1 = w.getBytes();
for(int i=0;i<IMAGE_HEIGHT*IMAGE_WIDTH;i++)
{
bytes[2*IMAGE_HEIGHT*IMAGE_WIDTH+i]=byte1[i];
}
emit.set(bytes,0,3*IMAGE_HEIGHT*IMAGE_WIDTH);
}
#Override
public void close(){
try{
goutput.collect(l, emit);
}
catch(Exception e){
e.printStackTrace();
System.exit(-1);
}
}
}
//end of first job , this is running perfectly
public static void main(String[] args) throws URISyntaxException {
if(args.length!=3) {
System.err.println("Usage: blurvideo input output");
System.exit(-1);
}
JobClient client = new JobClient();
JobConf conf = new JobConf(blur2.class);
conf.setOutputValueClass(BytesWritable.class);
conf.setInputFormat(SequenceFileInputFormat.class);
//conf.setNumMapTasks(n)
SequenceFileInputFormat.addInputPath(conf, new Path(args[0]));
TextOutputFormat.setOutputPath(conf, new Path(args[1]));
conf.setMapperClass(BlurMapper.class);
conf.setNumReduceTasks(0);
//conf.setReducerClass(org.apache.hadoop.mapred.lib.IdentityReducer.class);
client.setConf(conf);
try {
JobClient.runJob(conf);
} catch (Exception e) {
e.printStackTrace();
}
// exec("jar cf /home/hmobile/hadoop-0.19.2/imag /home/hmobile/hadoop-0.19.2/output");
JobClient client2 = new JobClient();
JobConf conf2 = new JobConf(blur2.class);
conf2.setOutputValueClass(BytesWritable.class);
conf2.setInputFormat(SequenceFileInputFormat.class);
//conf.setNumMapTasks(n)
SequenceFileInputFormat.addInputPath(conf2, new Path(args[0]));
SequenceFileOutputFormat.setOutputPath(conf2, new Path(args[2]));
conf2.setMapperClass(BlurMapper2.class);
conf2.setNumReduceTasks(0);
DistributedCache.addCacheFile(new URI("~/ayush/output/part-00000"), conf2);// these files are already on the hdfs
DistributedCache.addCacheFile(new URI("~/ayush/output/part-00001"), conf2);
client2.setConf(conf2);
try {
JobClient.runJob(conf2);
} catch (Exception e) {
e.printStackTrace();
}
}
public static class BlurMapper2 extends MapReduceBase implements Mapper<Text, BytesWritable, LongWritable, BytesWritable>
{
int IMAGE_HEIGHT = 240;
int T =60;
int IMAGE_WIDTH = 320;
public BytesWritable Gmiu;
public BytesWritable Gsigma;
public BytesWritable w;
byte[] bytes = new byte[IMAGE_HEIGHT*IMAGE_WIDTH];
public BytesWritable emit = new BytesWritable(bytes);
int initVar = 125;int gg=0;
int K=64;int k=0,k1=0,k2=0;
public LongWritable l = new LongWritable(1);
byte[] Gmiu1 = new byte[IMAGE_HEIGHT*IMAGE_WIDTH*K];
byte[] Gsigma1 = new byte[IMAGE_HEIGHT*IMAGE_WIDTH*K];
byte[] w1 = new byte[IMAGE_HEIGHT*IMAGE_WIDTH*K];
public Path[] localFiles=new Path[2];
private FileSystem fs;
#Override
public void configure(JobConf conf2)
{
try {
fs = FileSystem.getLocal(new Configuration());
localFiles = DistributedCache.getLocalCacheFiles(conf2);
//System.out.println(localFiles[0].getName());
} catch (IOException ex) {
Logger.getLogger(blur2.class.getName()).log(Level.SEVERE, null, ex);
}
}
public void map(Text key, BytesWritable file,OutputCollector<LongWritable, BytesWritable> output, Reporter reporter) throws IOException
{
if(gg==0){
//System.out.println(localFiles[0].getName());
String wrd; String line;
for(Path f:localFiles)
{
if(!f.getName().endsWith("crc"))
{
// FSDataInputStream localFile = fs.open(f);
BufferedReader br = null;
try {
br = new BufferedReader(new InputStreamReader(fs.open(f)));
int c = 0;
try {
while ((line = br.readLine()) != null) {
StringTokenizer itr = new StringTokenizer(line, " ");
while (itr.hasMoreTokens()) {
wrd = itr.nextToken();
c++;
int i = Integer.parseInt(wrd, 16);
Integer I = new Integer(i);
byte b = I.byteValue();
if (c < IMAGE_HEIGHT * IMAGE_WIDTH) {
Gmiu1[k] = b;k++;
} else {
if ((c >= IMAGE_HEIGHT * IMAGE_WIDTH) && (c < 2 * IMAGE_HEIGHT * IMAGE_WIDTH)) {
Gsigma1[k] = b;k1++;
} else {
w1[k] = b;k2++;
}
}
}
}
} catch (IOException ex) {
Logger.getLogger(blur2.class.getName()).log(Level.SEVERE, null, ex);
}
} catch (FileNotFoundException ex) {
Logger.getLogger(blur2.class.getName()).log(Level.SEVERE, null, ex);
} finally {
try {
br.close();
} catch (IOException ex) {
Logger.getLogger(blur2.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
}
gg++;
}
}
}
}
tackled a lot with this, can anyone please tell why i am getting this error:
java.io.FileNotFoundException: File does not exist: ~/ayush/output/part-00000
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
at org.apache.hadoop.filecache.DistributedCache.getTimestamp(DistributedCache.java:475)
at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:676)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:774)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1127)
at blur2.main(blur2.java:175)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
The problem is with the filename you are using "~/ayush/output/part-00000" relies on Unix shell (sh, bash, ksh) tilde expansion to replace the "~" with the pathname of your home directory.
Java (and C, and C++, and most other programming languages) don't do tilde expansion. You need to provide the pathname as "/home/ayush/output/part-00000" ... or whatever absolute pathname it is that the tilded form expands to.
Strictly speaking, the URI should be created as follows:
new File("/home/ayush/output/part-00000").toURI()
not as
new URI("/home/ayush/output/part-00000")
The latter creates a URI without a "protocol", and that could be problematic.