Needed help with Java program that takes avro.avsc schema file and avrofile as input and converts them to text file in java.
This java code worked for me hope is helpful for others.
import java.io.;
import java.util.;
import org.apache.avro.*;
import org.apache.avro.generic.*;
import org.apache.avro.file.*;
import org.apache.avro.io.*;
public class AvrotoTextFormatter
{
public static void main ( String args[]) throws Exception
{
InputStream in = null;
in = new FileInputStream(args[0]);
BufferedReader br;
BufferedInputStream inStream = new BufferedInputStream(in);
PrintWriter pr1 = new PrintWriter(args[1], "UTF-8");
PrintWriter pr = new PrintWriter(args[2], "UTF-8");
StringTokenizer st;
StringTokenizer st1;
int row_counter = 0;
String header_fields = "";
String content_records = "";
String sCurrentLine = "";
GenericDatumReader<Object> reader = new GenericDatumReader<Object>();
DataFileStream<Object> fileReader = new DataFileStream<Object>(inStream, reader);
pr1.println(fileReader.getSchema().getFields());
pr1.close();
br = new BufferedReader(new java.io.FileReader(args[1]));
while ((sCurrentLine = br.readLine()) != null)
{
st = new StringTokenizer(sCurrentLine," ");
while (st.hasMoreTokens())
{
header_fields = header_fields + st.nextToken() + "|";
st.nextToken();
st.nextToken();
}
}
header_fields = header_fields.substring(1,header_fields.length()-1);
pr.println(header_fields);
File file = new File(args[0]);
DatumReader<GenericRecord> datumReader = new GenericDatumReader<GenericRecord>(fileReader.getSchema());
DataFileReader<GenericRecord> dataFileReader = new DataFileReader<GenericRecord>(file, datumReader);
GenericRecord user = null;
while (dataFileReader.hasNext())
{
content_records = "";
user = dataFileReader.next(user);
st1 = new StringTokenizer(header_fields,"|");
while (st1.hasMoreTokens())
{
content_records = content_records + user.get(st1.nextToken()) + "|";
}
content_records = content_records.substring(0,content_records.length()-1);
pr.println(content_records);
}
fileReader.close();
br.close();
pr.close();
}
}
Found snippet with spark to do the same is faster and easier.
Sharing so that will be helpful to others
import org.apache.avro.mapreduce.AvroKeyInputFormat
import org.apache.avro.mapred.AvroKey
import org.apache.hadoop.io.NullWritable
val avroRdd = sc.newAPIHadoopFile("/sit/data/presentation/bbsbi/alayer/test/000000_0", classOf[AvroKeyInputFormat[String]], classOf[AvroKey[String]], classOf[NullWritable]).keys.map(_.toString)
val n=avroRdd.map(_.split(",").map(_.split(":")(1).trim).map(l=>l.substring(l.indexOf("\"")+1,l.lastIndexOf("\""))).mkString("|"))
n.collect.foreach(println)
Below code working for me
private static JSONArray readJsonFromAvro(String absFilePath)throws IOException,
InterruptedException
{
JSONArray jsonarray = new JSONArray();
InputStream in = new FileInputStream(absFilePath);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
BufferedInputStream inStream = new BufferedInputStream(in);
GenericDatumReader<Object> reader = new GenericDatumReader<Object>();
DataFileStream<Object> fileReader = new DataFileStream<Object>(
inStream, reader);
try {
final Schema schema = fileReader.getSchema();
final JsonEncoder encoder = EncoderFactory.get().jsonEncoder(
schema, baos);
for (final Object datum : fileReader) {
//writer.write(datum, encoder);
JSONObject jsonObj = new JSONObject(datum.toString());
jsonarray.put(jsonObj);
}
encoder.flush();
System.out.println();
} finally {
fileReader.close();
}
return jsonarray;
}
How can I run below program? I mean what are 3 command line input I can provide?
import org.apache.avro.*;
import org.apache.avro.generic.*;
import org.apache.avro.file.*;
import org.apache.avro.io.*;
public class AvrotoTextFormatter
{
public static void main ( String args[]) throws Exception
{
InputStream in = null;
in = new FileInputStream(args[0]);
BufferedReader br;
BufferedInputStream inStream = new BufferedInputStream(in);
PrintWriter pr1 = new PrintWriter(args[1], "UTF-8");
PrintWriter pr = new PrintWriter(args[2], "UTF-8");
StringTokenizer st;
StringTokenizer st1;
int row_counter = 0;
String header_fields = "";
String content_records = "";
String sCurrentLine = "";
GenericDatumReader<Object> reader = new GenericDatumReader<Object>();
DataFileStream<Object> fileReader = new DataFileStream<Object>(inStream, reader);
pr1.println(fileReader.getSchema().getFields());
pr1.close();
br = new BufferedReader(new java.io.FileReader(args[1]));
while ((sCurrentLine = br.readLine()) != null)
{
st = new StringTokenizer(sCurrentLine," ");
while (st.hasMoreTokens())
{
header_fields = header_fields + st.nextToken() + "|";
st.nextToken();
st.nextToken();
}
}
header_fields = header_fields.substring(1,header_fields.length()-1);
pr.println(header_fields);
File file = new File(args[0]);
DatumReader<GenericRecord> datumReader = new GenericDatumReader<GenericRecord>(fileReader.getSchema());
DataFileReader<GenericRecord> dataFileReader = new DataFileReader<GenericRecord>(file, datumReader);
GenericRecord user = null;
while (dataFileReader.hasNext())
{
content_records = "";
user = dataFileReader.next(user);
st1 = new StringTokenizer(header_fields,"|");
while (st1.hasMoreTokens())
{
content_records = content_records + user.get(st1.nextToken()) + "|";
}
content_records = content_records.substring(0,content_records.length()-1);
pr.println(content_records);
}
fileReader.close();
br.close();
pr.close();
}
}
Related
Just wondering if my bufferedWrites will work outside this try clause as it should as. i atm have no way of checking the outcome atm but. As in i only need the stream to be open for the instantiation of the BufferedWriter correct.
So this code should work:
BufferedWriter bufferedWriter;
// ServletOutputStream responseOutputStream = null;
FileOutputStream fos = new FileOutputStream(file);
if (zip) {
try (ZipOutputStream zOut = new ZipOutputStream(fos);) {
String fileName = file.getName().replace(".zip", "");
zOut.putNextEntry(new ZipEntry(fileName));
bufferedWriter = new BufferedWriter(new OutputStreamWriter(zOut, Charset.forName(enc)));
}
} else {
try (OutputStreamWriter opsw = new OutputStreamWriter(fos, Charset.forName(enc))) {
bufferedWriter = new BufferedWriter(opsw);
}
}
File fin = new File(tempFile);
FileInputStream fis = new FileInputStream(fin);
BufferedReader bufferedReader;
try(InputStreamReader ipsr = new InputStreamReader(fis)){
bufferedReader = new BufferedReader(ipsr);
}
String aLine = null;
int rowCount = 1;
logger.info("Copy file for output");
while ((aLine = bufferedReader.readLine()) != null) {
if (rowCount == 1) {
// Ful fix wax klarar nog inte att s�tta encoding.
if (encoding.equals(ProductFeedOutputController.ENCODING_88591)) {
bufferedWriter.write("<?xml version=\"1.0\" encoding=\"ISO-8859-1\"?>");
} else {
bufferedWriter.write("<?xml version=\"1.0\" encoding=\"UTF-8\"?>");
}
......
I am able to read in a file right now, but I am confused on how to read then the strings line by line to run through a parser I created. Any suggestions would be helpful.
public void ReadBtn() {
char[] inputBuffer = new char[READ_BLOCK_SIZE];
int charRead;
String s = "";
int READ_BLOCK_SIZE = 100;
//reading text from file
try {
FileInputStream fileIn = openFileInput("mytextfile.txt");
InputStreamReader InputRead = new InputStreamReader(fileIn);
BufferedReader BR = new BufferedReader(InputRead);
while((charRead = InputRead.read(inputBuffer)) > 0) {
// char to string conversion
String readstring = String.copyValueOf(inputBuffer, 0, charRead);
s += readstring;
getContactInfo(s);
}
InputRead.close();
} catch(Exception e) {
e.printStackTrace();
}
}
-Try this code. Replace sdCard path to your file path where mytextfile.txt exists.
String sdCard = Environment.getExternalStorageDirectory().getAbsolutePath();
String fileName = "mytextfile.txt";
String path = sdCard + "/" + MarketPath + "/";
File directory = new File(path);
if (directory.exists()) {
File file = new File(path + fileName);
if (file.exists()) {
String myData = ""; // this variable will store your file text
try {
FileInputStream fis = new FileInputStream(file);
DataInputStream in = new DataInputStream(fis);
BufferedReader br =new BufferedReader(new InputStreamReader(in));
String strLine;
while ((strLine = br.readLine()) != null) {
myData = myData + strLine;
}
in.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
You can read all lines in an ArrayList:
public void ReadBtn() {
int READ_BLOCK_SIZE = 100;
ArrayList<String> linesList = new ArrayList<>();
// reading text from file
try {
FileInputStream fileIn=openFileInput("mytextfile.txt");
InputStreamReader InputRead= new InputStreamReader(fileIn);
BufferedReader br = new BufferedReader(InputRead);
String line = br.readLine();
while (line != null) {
linesList.add(line);
line = br.readLine();
}
InputRead.close();
// here linesList contains an array of strings
for (String s: linesList) {
// do something for each line
}
} catch (Exception e) {
e.printStackTrace();
}
}
I am trying to replace multiple strings in a file from source as ArrayList. But the application is erasing the old string before replacing a new one. Please help.
public static void writeNewFile(File template, ArrayList<String> data) {
File file = template;
String nameToReplace = "((name))";
String productToReplace = "((product))";
String giftToReplace = "((gift))";
String giftValueToReplace = "((gift-value))";
String outputFileName = data.get(0);
String workingDirectory = System.getProperty("user.dir");
Scanner scanner = null;
try {
scanner = new Scanner(file);
PrintWriter writer = new PrintWriter(workingDirectory + "\\Output\\" + outputFileName);
while (scanner.hasNextLine()) {
String line1 = scanner.nextLine();
writer.println(line1.replace(nameToReplace, data.get(1)));
writer.println(line1.replace(productToReplace, data.get(2)));
}
} catch (Exception e) {
System.out.println("Destination folder not found");
}
}
This worked for me
try {
BufferedReader reader = new BufferedReader(new FileReader(file));
String line = "", oldtext = "";
while ((line = reader.readLine()) != null) {
oldtext += line + "\r\n";
}
reader.close();
String result = oldtext.replace(nameToReplace, data.get(1))
.replace(productToReplace, data.get(2))
.replace(giftToReplace, data.get(3));
// Write updated record to a file
FileWriter writer = new FileWriter(workingDirectory + "\\Output\\" + outputFileName);
writer.write(result);
writer.close();
} catch (IOException ioe) {
System.out.println("Write error");
}
i have 2 #RequestMapping; the first uploades a video to my server and the second execute a transcoding linux command.
every #RequestMapping is working but i am tryig from a week to get the filename of the uploaded file from the first #RequestMapping and send it to the second one.
please help me !
this is the first One:
#RequestMapping(value = "/uploadajax", method = RequestMethod.POST)
#ResponseBody
public List<ObjectNode> uploadMultipleFiles(Model model, MultipartHttpServletRequest request, HttpServletResponse response,
Principal principal) throws IOException {
CommonsMultipartFile multipartFile = null;
Iterator<String> iterator = request.getFileNames();
ObjectMapper objectMapper = new ObjectMapper();
FileMeta fileMeta = null;
while (iterator.hasNext()) {
String key = iterator.next();
multipartFile = (CommonsMultipartFile) request.getFile(key);
String uploadedFileName = multipartFile.getOriginalFilename();
try {
List<ObjectNode> listFileNode = new ArrayList<ObjectNode>();
byte[] bytes = multipartFile.getBytes();
String phyPath = request.getSession().getServletContext().getRealPath("/");
String repoUserPath=phyPath+"resources/"+user.getId();
System.out.println("-------------"+user.getId()+"------------------");
System.out.println("####"+repoUserPath);
File repoUSer = new File(repoUserPath);
if (!repoUSer.exists()) {
repoUSer.mkdir();
String filepath =repoUserPath+"/"+multipartFile.getOriginalFilename();
System.out.println("####"+filepath);
BufferedOutputStream stream = new BufferedOutputStream(new FileOutputStream(filepath));
stream.write(bytes);
stream.close();
fileMeta = new FileMeta();
fileMeta.setFileName(multipartFile.getOriginalFilename());
fileMeta.setFileSize((multipartFile.getSize()/1024)/1024+" Mb");
fileMeta.setFileType(multipartFile.getContentType());
fileMeta.setPathfile("http://89.40.113.84:8080/ProjectTranscode/resources/"+user.getId()+"/"+multipartFile.getOriginalFilename());
fileMeta.setLinkdelete("/var/lib/tomcat7/webapps/ProjectTranscode/resources/"+user.getId()+"/"+multipartFile.getOriginalFilename());
serviceMangerTranscode.save(fileMeta);
fileMeta.setUser(user);
serviceMangerTranscode.updateFile(fileMeta);
file= fileMeta;
model.addAttribute("Nomf", file.getFileName());
System.out.println("created!!");
String[] cmdArray = {"ffprobe","-show_streams","-i",filepath };
model.addAttribute("filepath",filepath);
System.out.println("ffprobe2........!!");
//String[] cmdArray = {"ping","www.google.com" };
Process process = Runtime.getRuntime().exec(cmdArray);
InputStream processInputStream =process.getInputStream();
InputStreamReader inputStreamReader = new InputStreamReader(processInputStream);
BufferedReader bufferedReader = new BufferedReader(inputStreamReader);
String line = null;
System.out.println("<OUTPUT>");
while ( (line = bufferedReader.readLine()) != null){
ObjectNode node = objectMapper.createObjectNode();
System.out.println(line);
node.put("line",line );
listFileNode.add(node);
}
line ="aaa";
ObjectNode node = objectMapper.createObjectNode();
node.put("line",line );
System.out.println("</OUTPUT>");
return listFileNode;
}else {
long size = FileUtils.sizeOfDirectory(repoUSer);
System.out.println("size 1"+(size/1024)/1024);
if ((size/1024)/1024 >100) {
ObjectNode node = objectMapper.createObjectNode();
node.put("error","Size Not Supported for Your repository !");
listFileNode.add(node);
}else{
System.out.println("choix2!!++");
String filepath =repoUserPath+"//"+multipartFile.getOriginalFilename();
System.out.println("####"+filepath);
BufferedOutputStream stream = new BufferedOutputStream(new FileOutputStream(filepath));
stream.write(bytes);
stream.close();
// fileMeta = new FileMeta();
fileMeta.setFileName(multipartFile.getOriginalFilename());
fileMeta.setFileSize((multipartFile.getSize()/1024)/1024+" Mb");
fileMeta.setFileType(multipartFile.getContentType());
fileMeta.setPathfile(filepath);
fileMeta.setLinkdelete("/var/lib/tomcat7/webapps/ProjectTranscode/resources/"+user.getId()+"/"+multipartFile.getOriginalFilename());
serviceMangerTranscode.save(fileMeta);
fileMeta.setUser(user);
serviceMangerTranscode.updateFile(fileMeta);
file= fileMeta;
model.addAttribute("filedetails", fileMeta);
System.out.println("fichier créé!!");
String[] cmdArray = {"ffprobe","-show_streams","-i",filepath };
System.out.println("ffprobe executee........!!");
Process process = Runtime.getRuntime().exec(cmdArray);
InputStream processInputStream =process.getInputStream();
InputStreamReader inputStreamReader = new InputStreamReader(processInputStream);
BufferedReader bufferedReader = new BufferedReader(inputStreamReader);
String line = null;
System.out.println("<OUTPUT>");
while ( (line = bufferedReader.readLine()) != null){
ObjectNode node = objectMapper.createObjectNode();
// if (line.contains("codec_type") || line.contains("duration_ts")){
System.out.println(line);
node.put("line",line );
listFileNode.add(node);
}//}
line ="bbb";
ObjectNode node = objectMapper.createObjectNode();
node.put("line",line );
System.out.println("</OUTPUT>");
System.out.println("size 2"+(size/1024)/1024);
return listFileNode;
}
}
} catch (Exception e) {
return null;
}
}
return null;
}
and this is the second one:
#RequestMapping(value ="/Transcodage", method = RequestMethod.POST)
public String Trascoood(Model model, HttpServletRequest request, HttpServletResponse response
) throws IOException {
//Recuperation des valeurs utilisateur:
String userIDD =request.getParameter("useridd");
String newFileName =request.getParameter("newFileName");
String encolist=request.getParameter("encolist");
String codeclist=request.getParameter("codeclist");
String bitrate=request.getParameter("bitratevideo");
String Nomf= request.getParameter("mapping1Form");
System.out.println("NOMFFFFFFFFFFF: " + Nomf);
int userIDDD = Integer.parseInt(userIDD);
User user = userv.getUserById(userIDDD);
ObjectMapper objectMapper = new ObjectMapper();
org.codehaus.jackson.node.ObjectNode node = objectMapper.createObjectNode();
//file= serviceMangerTranscode.getFileById(idfile);
ModelAndView modelUserHome = new ModelAndView("userhome");
//System.out.println("file path : " + file.getPathfile() +"*************************************************");
System.out.println("userIDD: " + userIDD + "...............................................");
System.out.println("newFileName: " + newFileName + "...............................................");
System.out.println("encolist: " + encolist + "...............................................");
String newFileN = new StringBuilder().append(".").append(newFileName).append(".").append(encolist).toString();
System.out.println("FILE NAME + EXTENTION: " + newFileN + "...............................................");
//node.put("success", "true");
String[] cmdArray = {"sudo", "ssh", "-tt", "root#89.40.113.84", "/root/dved", "-l", "89.40.112.120,89.40.112.248", "you.mp4", "-s",newFileN};
//String[] cmdArray = {"sudo", "/root/dve", "-l", "89.40.112.120,89.40.112.248", "/root/greece.mkv", "-s",".teeeeeeeeeeeeest.avi" };
//String[] cmdArray = {"sudo","/root/dve","-s",".encoded.avi","-l","89.40.112.248","/root/greece.mkv" };
List<ObjectNode> listFileNode = new ArrayList<ObjectNode>();
try{
Runtime rt = Runtime.getRuntime();
ProcessBuilder pb = new ProcessBuilder(cmdArray);
Process proc = pb.start();
//Process proc = rt.exec(cmdArray);
int rc =0;
rc = proc.waitFor(); // Wait for the process to finish.
System.out.printf("Script executed successfully in ", rc);
InputStream stderr = proc.getErrorStream();
InputStreamReader isr = new InputStreamReader(stderr);
BufferedReader br = new BufferedReader(isr);
String line = null;
System.out.println("<ERROR___EXEC>");
while ( (line = br.readLine()) != null){
System.out.println(line);
node.put("line",line );
listFileNode.add(node);
}
System.out.println("</ERROR___EXEC>");
int exitVal = proc.waitFor();
System.out.println("Process exitValue: " + exitVal);
InputStream processInputStream =proc.getInputStream();
InputStreamReader inputStreamReader = new InputStreamReader(processInputStream);
BufferedReader bufferedReader = new BufferedReader(inputStreamReader);
System.out.println("<RESULTAT___EXEC>");
while ( (line = bufferedReader.readLine()) != null){
System.out.println(line);
node.put("lineR",line );
listFileNode.add(node);
}
System.out.println("</RESULTAT___EXEC>");
}catch (Throwable t)
{
t.printStackTrace();
}
model.addAttribute("usersize",user.getUserfiles().size());
model.addAttribute("listfiles",user.getUserfiles());
model.addAttribute("user",user);
model.addAttribute("MsgTranscode","Votre video a bien ete encodee, vous pouvez desormais la telecharger de votre portfolio!");
return "userhome";
}
Thank you for your reply, i think it cannot be done so i put the data in the database than recover it in the second RequestMapping
I have a .txt file with the following content:
1 1111 47
2 2222 92
3 3333 81
I would like to read line-by-line and store each word into different variables.
For example: When I read the first line "1 1111 47", I would like store the first word "1" into var_1, "1111" into var_2, and "47" into var_3. Then, when it goes to the next line, the values should be stored into the same var_1, var_2 and var_3 variables respectively.
My initial approach is as follows:
import java.io.*;
class ReadFromFile
{
public static void main(String[] args) throws IOException
{
int i;
FileInputStream fin;
try
{
fin = new FileInputStream(args[0]);
}
catch(FileNotFoundException fex)
{
System.out.println("File not found");
return;
}
do
{
i = fin.read();
if(i != -1)
System.out.print((char) i);
} while(i != -1);
fin.close();
}
}
Kindly give me your suggestions. Thank You
public static void main(String[] args) throws IOException {
File file = new File("/path/to/InputFile");
BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream(file)));
String line = null;
while( (line = br.readLine())!= null ){
// \\s+ means any number of whitespaces between tokens
String [] tokens = line.split("\\s+");
String var_1 = tokens[0];
String var_2 = tokens[1];
String var_3 = tokens[2];
}
}
try {
BufferedReader fr = new BufferedReader(new InputStreamReader(new FileInputStream(file), "ASCII"));
while(true)
{
String line = fr.readLine();
if(line==null)
break;
String[] words = line.split(" ");//those are your words
}
} catch (FileNotFoundException e) {
e.printStackTrace();
}
catch(IOException e)
{
e.printStackTrace();
}
Hope this Helps!
Check out BufferedReader for reading lines. You'll have to explode the line afterwards using something like StringTokenizer or String's split.
import java.io.*;
import java.util.Scanner;
class Example {
public static void main(String[] args) throws Exception {
File f = new File("main.txt");
StringBuffer txt = new StringBuffer();
FileOutputStream fos = new FileOutputStream(f);
for (int i = 0; i < args.length; i++) {
txt.append(args[i] + " ");
}
fos.write(txt.toString().getBytes());
fos.close();
FileInputStream fis = new FileInputStream("main.txt");
InputStreamReader input = new InputStreamReader(fis);
BufferedReader br = new BufferedReader(input);
String data;
String result = new String();
StringBuffer txt1 = new StringBuffer();
StringBuffer txt2 = new StringBuffer();
File f1 = new File("even.txt");
FileOutputStream fos1 = new FileOutputStream(f1);
File f2 = new File("odd.txt");
FileOutputStream fos2 = new FileOutputStream(f2);
while ((data = br.readLine()) != null) {
result = result.concat(data);
String[] words = data.split(" ");
for (int j = 0; j < words.length; j++) {
if (j % 2 == 0) {
System.out.println(words[j]);
txt1.append(words[j] + " ");
} else {
System.out.println(words[j]);
txt2.append(words[j] + " ");
}
}
}
fos1.write(txt1.toString().getBytes());
fos1.close();
fos2.write(txt2.toString().getBytes());
fos2.close();
br.close();
}
}