How to get cpu usage using java? - java

I would like to get cpu usage to update my database using java continuously.
At the first loop, this code is readable the correct cpu usage.
After then, it returns wrong values less than 0.
So, I stucked.
I used jdk 1.8.124.
plz, let me know how to get cpu usage continuously.
lib
import oshi.SystemInfo;
import oshi.hardware.CentralProcessor;
import oshi.hardware.HardwareAbstractionLayer;
src
public static void main(String[] args) {
OperatingSystemMXBean bean = (com.sun.management.OperatingSystemMXBean) ManagementFactory
.getOperatingSystemMXBean();
while (true) {
System.out.println(bean.getProcessCpuLoad());
System.out.println(bean.getSystemCpuLoad());
try {
Thread.sleep(3000);
}
catch (Exception e){
System.out.println(e.toString());
break;
}
}
}

It's done by using Oshi lib.
I can get cpu usage every 20 seconds
lib
import oshi.SystemInfo;
import oshi.hardware.CentralProcessor;
import oshi.hardware.HardwareAbstractionLayer;
src
private SystemInfo si = new SystemInfo();
private HardwareAbstractionLayer hal = si.getHardware();
private CentralProcessor cpu = hal.getProcessor();
long[] prevTicks = new long[TickType.values().length];
public static double getCPU()
{
double cpuLoad = cpu.getSystemCpuLoadBetweenTicks( prevTicks ) * 100;
prevTicks = cpu.getSystemCpuLoadTicks();
System.out.println("cpuLoad : " + cpuLoad);
return cpuLoad;
}
public static void main(String[] args) throws Exception{
while(true) {
// Sleep 20 seconds
tCPU = (int)getCPU();
}
}

I use the following code to get the CPU load, which works without invoking hidden methods in com.sun.* classes:
public static Double getProcessCpuLoad() {
try {
MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
ObjectName name = ObjectName.getInstance("java.lang:type=OperatingSystem");
AttributeList list = mbs.getAttributes(name, new String[]{"ProcessCpuLoad"});
return Optional.ofNullable(list)
.map(l -> l.isEmpty() ? null : l)
.map(List::iterator)
.map(Iterator::next)
.map(Attribute.class::cast)
.map(Attribute::getValue)
.map(Double.class::cast)
.orElse(null);
} catch (Exception ex) {
return null;
}
}
Works fine, tested with JRE 8/11/13.

Related

Why is this not creating a properties file?

I am trying to make a properties file in Java. Sadly, when I startup Minecraft (As this is a mod in Forge) the file is not created. I will be so thankful to anyone who helps me. Here is the code:
package mymod.properties;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.Properties;
public class WriteToProperties {
public static void main(String[] args) {
Properties prop = new Properties();
try {
prop.setProperty("Test", "Yay");
prop.store(new FileOutputStream("Test.properties"), null);
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
Basically what you want is initialize(event.getSuggestedConfigurationFile());
Forge basically wishes to hand you the default settings to you. I also suggest you structure things logically, so it's nice, clean and accessible later on, and easier to manage.
I use this as a config loader
package tschallacka.magiccookies.init;
import java.io.File;
import net.minecraftforge.common.config.Configuration;
import net.minecraftforge.common.config.Property;
import net.minecraftforge.fml.common.event.FMLPreInitializationEvent;
import tschallacka.magiccookies.MagicCookies;
import tschallacka.magiccookies.api.ModHooks;
import tschallacka.magiccookies.api.Preferences;
public class ConfigLoader {
public static Configuration config;
public static final String CATEGORY_GENERAL = "GeneralSettings";
public static final String CATEGORY_SERVER = "ServerPerformance";
public static void init(FMLPreInitializationEvent event) {
ModHooks.MODID = MagicCookies.MODID;
ModHooks.MODNAME = MagicCookies.MODNAME;
ModHooks.VERSION = MagicCookies.VERSION;
try {
initialize(event.getSuggestedConfigurationFile());
}
catch (Exception e) {
MagicCookies.log.error("MagicCookie failed to load preferences. Reverting to default");
}
finally {
if (config != null) {
save();
}
}
}
private static void initialize(final File file) {
config = new Configuration(file);
config.addCustomCategoryComment(CATEGORY_GENERAL, "General Settings");
config.addCustomCategoryComment(CATEGORY_SERVER, "Server performance settings");
Preferences.magicCookieIsLoaded = true;
config.load();
syncConfigurable();
}
private static void save() {
config.save();
}
private static void syncConfigurable() {
final Property awesomeMod = config.get(Configuration.CATEGORY_GENERAL, "awesome_mod", Preferences.awesomeMod);
awesomeMod.comment = "Set this to yes if you think this mod is awesome, maybe it will at some time unlock a special thing.... or not... but that's only for people who think this is an wesome Mod ;-)";
Preferences.awesomeMod = awesomeMod.getString();
final Property numberOfBlocksPlacingPerTickByStripper = config.get(CATEGORY_SERVER,"number_of_blocks_placing_per_tick_by_stripper",Preferences.numberOfBlocksPlacingPerTickByStripper);
numberOfBlocksPlacingPerTickByStripper.comment = "This affects how many blocks will be placed per tick by the creative stripper tool per tick. The stripper tool will only place blocks if ticks are take less than 50ms. If you experience lag lower this number, if you don't experience lag and want faster copy pasting, make this number higher. For an awesome slowmo build of your caste set this to 1 ;-). Set to 0 to render everything in one go per chunk";
Preferences.numberOfBlocksPlacingPerTickByStripper = numberOfBlocksPlacingPerTickByStripper.getInt();
final Property averageTickTimeCalculationSpan = config.get(CATEGORY_SERVER,"number_of_ticks_used_for_average_time_per_tick_calculation",Preferences.averageTickTimeCalculationSpan);
averageTickTimeCalculationSpan.comment = "This number is the number of tick execution times are added together to calculation an average. The higher number means less lag by some things like the strippers, but can also lead to longer execution times for the strippers. Basically if your server is always running behind on server ticks set this value to 1, to at least get some work done when your server is running under 50ms tickspeed";
Preferences.averageTickTimeCalculationSpan = averageTickTimeCalculationSpan.getInt();
final Property acceptableTickduration = config.get(CATEGORY_SERVER,"acceptable_tick_duration",Preferences.acceptableTickduration);
acceptableTickduration.comment = "Define here what you see as acceptable tick speed where MagicCookies can do some of its magic. If average tick speed or current tick speed is higher than this value it won't perform some tasks to help manage server load.";
Preferences.acceptableTickduration = (long)acceptableTickduration.getDouble();
}
}
The value holder Preferences class.
This is purely so I can do Preferences.valuename everywhere.
package tschallacka.magiccookies.api;
/**
* Here the preferences of MagicCookies will be stored
* and kept after the config's are loaded.
* #author Tschallacka
*
*/
public class Preferences {
public static boolean magicCookieIsLoaded = false;
public static String awesomeMod = "Well?";
public static boolean isLoadedThaumcraft = false;
public static int numberOfBlocksPlacingPerTickByStripper = 0;
public static int averageTickTimeCalculationSpan = 60;
public static long acceptableTickduration = 50l;
public static int darkShrineFrequency = 0;
public static boolean darkShrineSpawnLogging = true;
public static boolean entropyTempleSpawnLogging = true;
public static int entropySize = 30;
}
In your main mod file:
#EventHandler
public void preInit(FMLPreInitializationEvent e) {
MagicCookies.log.warn("Preinit starting");
MinecraftForge.EVENT_BUS.register((Object)MagicCookies.instance);
ConfigLoader.init(e);
}
And after that you can just fetch all your preferences from the Preferences class
This file was created in root of your project. If you want some specific path to save, create folder in root of your project like props and change path in FileOutputStream constructor to "props\\Test.properties"

JNA: Missing some specifics methods

I'm wanting create a dll injector in Java ( and only in Java ) for educational proporses for myself and found a basic example in a website especialized in online game.
The autor only said that was made using JNA interface.
So, i'm studyng this piece of code and trying compile with success using NetBeans IDE and JNA, but seem that JNA interface that i have here ( 4.2.2 ) not have all methods and functions used on piece of code left by autor.
Are they:
GetProcAddress
VirtualAllocEx
VirtualFreeEx
So, i'm wanting some help here if possible, for try solved this trouble of missing of methods in JNA.
I had fixed big part these erros but still missing some methods in JNA like i will show following point to point with comments.
package inject;
//////////////////// JNA-4.2.2 /////////////////////
import com.sun.jna.Memory;
import com.sun.jna.Native;
import com.sun.jna.Pointer;
import com.sun.jna.platform.win32.Kernel32;
import com.sun.jna.platform.win32.Tlhelp32;
import com.sun.jna.platform.win32.WinDef;
import com.sun.jna.platform.win32.WinDef.HMODULE;
import com.sun.jna.platform.win32.WinNT;
import com.sun.jna.platform.win32.WinNT.HANDLE;
import com.sun.jna.ptr.IntByReference;
import com.sun.jna.win32.W32APIOptions;
import java.io.File;
//////////////////////////////////////////////////
// Extracted from: https://github.com/warmuuh/AndroidCtx/tree/master/HotContext/src/luz/winapi
import inject.luz.winapi.constants.DwDesiredAccess;
import inject.luz.winapi.tools.Advapi32Tools;
import inject.luz.winapi.tools.Kernel32Tools;
import luz.winapi.api.exception.Kernel32Exception;
//////////////////////////////////////////////////////////////////////////////////////////////
public class Inject {
private static int GetPid(String proc){
int id = 0;
Kernel32 kernel32 = (Kernel32) Native.loadLibrary(Kernel32.class, W32APIOptions.UNICODE_OPTIONS);
Tlhelp32.PROCESSENTRY32.ByReference processEntry = new Tlhelp32.PROCESSENTRY32.ByReference();
WinNT.HANDLE snapshot = kernel32.CreateToolhelp32Snapshot(Tlhelp32.TH32CS_SNAPPROCESS, new WinDef.DWORD(0));
try {
while (kernel32.Process32Next(snapshot, processEntry)) {
if (Native.toString(processEntry.szExeFile).equalsIgnoreCase(proc)) {
id = processEntry.th32ProcessID.intValue();
}
}
}
finally {
kernel32.CloseHandle(snapshot);
}
return id;
}
private static String findProcessByPID(int pid){
String name = "";
Kernel32 kernel32 = (Kernel32) Native.loadLibrary(Kernel32.class, W32APIOptions.UNICODE_OPTIONS);
Tlhelp32.PROCESSENTRY32.ByReference processEntry = new Tlhelp32.PROCESSENTRY32.ByReference();
WinNT.HANDLE snapshot = kernel32.CreateToolhelp32Snapshot(Tlhelp32.TH32CS_SNAPPROCESS, new WinDef.DWORD(0));
try {
while (kernel32.Process32Next(snapshot, processEntry)) {
if (pid == processEntry.th32ProcessID.intValue()) {
name = processEntry.szExeFile.toString();
}
}
}
finally {
kernel32.CloseHandle(snapshot);
}
return name;
}
public static void inject(File dll, Integer pId) throws Kernel32Exception {
if(null == dll || !dll.exists() || !dll.isFile() || !dll.getName().endsWith(".dll"))
return;
String p = findProcessByPID(pId);
if(null == p) return;
Kernel32 kernel = Kernel32.INSTANCE;
HMODULE kernel32Pointer = kernel.GetModuleHandle("Kernel32");
// Cannot find "GetProcAddress"
Pointer loadLibraryAddress = kernel.GetProcAddress(kernel32Pointer, "LoadLibraryA");
HANDLE process = null;
DwDesiredAccess access = new DwDesiredAccess();
access.setPROCESS_ALL_ACCESS();
try {
Advapi32Tools.getInstance().enableDebugPrivilege(Kernel32Tools.getInstance().GetCurrentProcess());
} catch (Exception e) {
}
// Incompatible types "Pointer" and "HANDLE"
process = Kernel32Tools.getInstance().OpenProcess(access, false, pId);
String path = dll.getPath() + '\0';
byte[] bytes = path.getBytes();
int pathLength = bytes.length;
// Cannot find "VirtualAllocEx"
Pointer memoryDllPath = kernel.VirtualAllocEx(process, null, pathLength, Kernel32Tools.MEM_COMMIT, Kernel32Tools.PAGE_READWRITE);
Memory dllPathContent = new Memory(pathLength);
for(int i=0;i<pathLength;i++)
dllPathContent.setByte(i, bytes[i]);
IntByReference writeResult = new IntByReference();
boolean successWritting = kernel.WriteProcessMemory(process, memoryDllPath, dllPathContent, pathLength, writeResult);
if(!successWritting) {
kernel.CloseHandle(process);
return;
}
IntByReference threadId = new IntByReference();
// Pointer cannot be converted to "FOREIGN_THREAD_START_ROUTINE"
Pointer thread = kernel.CreateRemoteThread(process, null, 0, loadLibraryAddress, memoryDllPath, 0, threadId);
boolean res = false;
// Incompatible types "Pointer" and "HANDLE" //Cannot find "WAIT_TIMEOUT"
res = kernel.WaitForSingleObject(thread, Integer.MAX_VALUE) != Kernel32Tools.WAIT_TIMEOUT;
// Cannot find "VirtualFreeEx" method // Cannot find "MEM_RELEASE"
kernel.VirtualFreeEx(process, memoryDllPath, pathLength, Kernel32Tools.MEM_RELEASE);
kernel.CloseHandle(process);
}
/**
* #param args the command line arguments
*/
public static void main(String[] args) {
System.out.println(GetPid("notepad.exe"));
}
}
Thank in advance by any suggestion or help :-)
JNA missing methods? It ain't so!
You just need to extend the library and add your own (and, ideally, also contribute the "missing" methods back to the JNA library so others can benefit.
Here is an example of how someone has mapped GetProcAddress.
Someone has mapped VirtualAllocEx here (although they should properly have extended Kernel32 rather than copied it entirely and edited portions)
I couldn't find an example of VirtualFreeEx within the same 15 seconds I found the others... doesn't mean it's not out there but after writing the others you shouldn't have much trouble writing it as well.

Determine platform under JNA for setsockopt

I'm writing an implementation of setsockopt under JNA. Java itself supports setsockopt, but it doesn't support all the platform specific socket options. For instance, it doesn't support [TCP_KEEPIDLE][2] under Linux. Clearly, many of these options are not very portable, and using JNA is a route to poor portability; I am aware of this. Please don't bother to tell me the idea is deeply horrible.
What I'd like to do, however, is make my code a little more reuseable than something that just works under Linux. I'd like it to work (as far as is possible) on several target platforms. If the socket option is not available, it can throw an exception.
My challenge is this. The JNA works fine, but the values of the socket options are different across platforms. For instance, SO_RCVBUF is 0x1002 under OS-X and 8 under Linux (I realise SO_RCVBUF is controllable by the normal Java setSockOpt - it's an example that's easy to test with lsof). SO_DONTROUTE is 5 under Linux, and 0x0010 under OS-X (and that isn't controllable via Java setSockOpt).
So what I'd like it to do is to take an enum value representing the socket option (SO_SNDBUF, SO_RCVBUF or whatever), and look that up in a platform dependent map, so I get 0x1002 / 0x010 under OS-X and 8 / 5 under Linux.
That's easy enough, but how do I tell what the platform is under JNA so I know which map to use? JNA must somehow have a sense of its own platform, or it would not (I presume) know how to call the native libraries.
package sockettest;
import java.io.FileDescriptor;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.lang.reflect.Field;
import java.net.Socket;
import com.sun.jna.LastErrorException;
import com.sun.jna.Native;
import com.sun.jna.Pointer;
import com.sun.jna.ptr.IntByReference;
public class JNASockOpt {
private static Field fdField;
static {
Native.register("c");
try {
fdField = FileDescriptor.class.getDeclaredField("fd");
fdField.setAccessible(true);
} catch (Exception ex) {
fdField = null;
}
}
public static int getInputFd(Socket s) {
try {
FileInputStream in = (FileInputStream)s.getInputStream();
FileDescriptor fd = in.getFD();
return fdField.getInt(fd);
} catch (Exception e) { }
return -1;
}
public static int getOutputFd(Socket s) {
try {
FileOutputStream in = (FileOutputStream)s.getOutputStream();
FileDescriptor fd = in.getFD();
return fdField.getInt(fd);
} catch (Exception e) { }
return -1;
}
public static int getFd(Socket s) {
int fd = getInputFd(s);
if (fd != -1)
return fd;
return getOutputFd(s);
}
// The list of SOL_ and SO_ options is platform dependent
public static final int SOL_SOCKET = 0xffff; // that's under OS-X, but it's 1 under Linux
public static final int SO_RCVBUF = 0x1002; // that's under OS-X, but it's 8 under Linux
public static final int SO_DONTROUTE = 0x0010; // that's under OS-X, but it's 5 under Linux
private static native int setsockopt(int fd, int level, int option_name, Pointer option_value, int option_len) throws LastErrorException;
public static void setSockOpt (Socket socket, int level, int option_name, int option_value) throws IOException {
if (socket == null)
throw new IOException("Null socket");
int fd = getFd(socket);
if (fd == -1)
throw new IOException("Bad socket FD");
IntByReference val = new IntByReference(option_value);
try {
setsockopt(fd, level, option_name, val.getPointer(), 4);
} catch (LastErrorException ex) {
throw new IOException("setsockopt: " + strerror(ex.getErrorCode()));
}
}
public static native String strerror(int errnum);
private JNASockOpt() {
}
}
The class com.sun.jna.Platform provided by JNA is used by JNA itself and has functions for querying the OS family and CPU architecture.
There are static methods for isMac() and isLinux().
jnaplatform does this by string parsing System.getProperty("os.name");, which seems pretty horrible to me, but if jnaplatform does it, I guess that should be good enough for me.
Results (i.e. how I used the above idea to solve the issue in the question) at https://github.com/abligh/jnasockopt - specifically https://github.com/abligh/jnasockopt/blob/master/src/org/jnasockopt/JNASockOptionDetails.java

Jacob Fatal Error

I am trying to build a Java Drag and Drop that works with Outlook emails. I've been using Jacob because of an inability to transfer data from Outlook to Java using standard AWT Event stuff. That said, all of the solutions I've pulled from here or other sites have been causing a fatal crash in Java. Here's the code:
import java.awt.dnd.*;
import javax.swing.*;
import java.awt.event.*;
import java.awt.*;
import java.awt.datatransfer.*;
import java.io.*;
import java.util.List;
import sun.awt.datatransfer.*;
import com.jacob.com.*;
import com.jacob.activeX.*;
public class D2 extends JFrame
{
private static final String DIR = "FILES";
private static void saveSelectedOutlookMails(String directory) {
Dispatch xl = new Dispatch("Outlook.Application");
//Dispatch selection = Dispatch.get(xl, "Selection").toDispatch();
System.out.println(xl);
System.out.println(xl==null);
//PROGRAM CRASHES AFTER THIS LINE
Dispatch explorer = Dispatch.get(xl,"ActiveExplorer").toDispatch();
System.out.println("explorer");
Object selection = Dispatch.get(explorer, "Selection").toDispatch();
Variant count = Dispatch.get(selection, "Count");
for (int mailIndex = 1; mailIndex <= count.toInt(); mailIndex++ ) {
Object mailItem = Dispatch.call(selection, "Item", new Variant(mailIndex)).toDispatch();
Variant senderName = Dispatch.get(mailItem, "SenderName");
Variant subject = Dispatch.get(mailItem, "Subject");
Variant body = Dispatch.get(mailItem, "Body");
String emailFileName = subject.toString() +".txt";
String fullPath = directory + "/" + emailFileName;
try {
File email = new File(fullPath);
PrintWriter writer = new PrintWriter( new FileWriter(email) );
writer.println("From: "+ senderName );
writer.println("Subject: "+ subject);
writer.println("");
writer.print( body );
writer.close();
}
catch (IOException e) {
System.out.println(e.getMessage());
//logger.error("IOException writing e-mail with subject: '"+ subject +"'", e);
continue;
}
Object attachments = Dispatch.get(mailItem, "Attachments").toDispatch();
Variant attachmentCount = Dispatch.get(attachments, "Count");
if ( attachmentCount.toInt() > 0 ) {
for( int attachmentIndex = 1; attachmentIndex<=attachmentCount.toInt(); attachmentIndex++ ) {
Object attachment = Dispatch.call(attachments, "Item", new Variant(attachmentIndex)).toDispatch();
Variant fileNameVariant = Dispatch.get(attachment, "FileName");
String fileName = fileNameVariant.toString();
Variant saveResult = Dispatch.call(attachment, "SaveAsFile", directory, "/", fileName);
}
}
}
}
public D2() throws Exception
{
this.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
this.setBounds(0,0,300,300);
this.setVisible(true);
DropTarget dropTarget=new DropTarget();
dropTarget.setComponent(this);
dropTarget.addDropTargetListener(new DropTargetAdapter()
{
public void drop(DropTargetDropEvent dtde){
saveSelectedOutlookMails(DIR);
}
});
}
public static void main(String[] args)
{
try{
new D2();
}catch(Exception e){
e.printStackTrace();
}
}
}
Ok, firstly, you are creating Outlook.Application in a way I've never seen before - I've only ever seen the Active X Component way:
e.g.
ActiveXComponent xl = new ActiveXComponent("Outlook.Application");
Dispatch explorer = Dispatch.get(xl,"ActiveExplorer").toDispatch();
Dispatch selection = Dispatch.get(explorer, "Selection").toDispatch();
Variant count = Dispatch.get(selection, "Count");
// loop over selected mail items.
for (int mailIndex = 1; mailIndex <= count.getInt(); mailIndex++ ) {
Dispatch mailItem = Dispatch.call(selection, "Item", new Variant(mailIndex)).toDispatch();
Variant subject = Dispatch.get(mailItem, "Subject");
// .... and so on
}
Secondly, your code is not saving the mail, its pulling out all the fields and attachments and trying to recreate the mail, which seems inaccurate at best and will only be an approximation of what the message was.
Why don't you just use the COM objects to SaveAs the whole .msg file to disk? Then if you need to access it again you can use something like JDIC to launch Outlook and pop up the message in its original glory, including all attachments?
My guess is that you are trying to get PROPERTY with name "ActiveExplorer", but it is a method! Here is documentation of that particuliar method https://msdn.microsoft.com/en-us/library/office/ff870017.aspx . Try to use .call() Jacob method to invoke METHODS on MS objects.

Bug dereferencing after pig udf

The is actually related to the question How can I add row numbers for rows in PIG or HIVE?
The 3rd answer provided by srini works fine, but I have trouble to access the data after the udf.
The udf provided by srini is following
import java.io.IOException;
import java.util.Iterator;
import org.apache.pig.EvalFunc;
import org.apache.pig.backend.executionengine.ExecException;
import org.apache.pig.data.BagFactory;
import org.apache.pig.data.DataBag;
import org.apache.pig.data.Tuple;
import org.apache.pig.data.TupleFactory;
import org.apache.pig.impl.logicalLayer.schema.Schema;
import org.apache.pig.data.DataType;
public class RowCounter extends EvalFunc<DataBag> {
TupleFactory mTupleFactory = TupleFactory.getInstance();
BagFactory mBagFactory = BagFactory.getInstance();
public DataBag exec(Tuple input) throws IOException {
try {
DataBag output = mBagFactory.newDefaultBag();
DataBag bg = (DataBag)input.get(0);
Iterator it = bg.iterator();
Integer count = new Integer(1);
while(it.hasNext())
{ Tuple t = (Tuple)it.next();
t.append(count);
output.add(t);
count = count + 1;
}
return output;
} catch (ExecException ee) {
// error handling goes here
throw ee;
}
}
public Schema outputSchema(Schema input) {
try{
Schema bagSchema = new Schema();
bagSchema.add(new Schema.FieldSchema("RowCounter", DataType.BAG));
return new Schema(new Schema.FieldSchema(getSchemaName(this.getClass().getName().toLowerCase(), input),
bagSchema, DataType.BAG));
}catch (Exception e){
return null;
}
}
}
I wrote a simple test pig script as following
A = load 'input.txt' using PigStorage(' ') as (name:chararray, age:int);
/*
--A: {name: chararray,age: int}
(amy,56)
(bob,1)
(bob,9)
(amy,34)
(bob,20)
(amy,78)
*/
B = group A by name;
C = foreach B {
orderedGroup = order A by age;
generate myudfs.RowCounter(orderedGroup) as t;
}
/*
--C: {t: {(RowCounter: {})}}
({(amy,34,1),(amy,56,2),(amy,78,3)})
({(bob,1,1),(bob,9,2),(bob,20,3)})
*/
D = foreach C generate FLATTEN(t);
/*
D: {t::RowCounter: {}}
(amy,34,1)
(amy,56,2)
(amy,78,3)
(bob,1,1)
(bob,9,2)
(bob,20,3)
*/
The problem is how to use D in later operation. I tried multiple ways, but always got the following error
ava.lang.ClassCastException: java.lang.String cannot be cast to org.apache.pig.data.DataBag
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject.processInputBag(POProject.java:575)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject.getNext(POProject.java:248)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:316)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:332)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:284)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:459)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:427)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:407)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:261)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:572)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:414)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:256)
My guess is that because we don't have the schema for the tuple inside the bag. if this is the reason, how should I modify the udf?
ok, I found the solution by adding the outputSchema as following
public Schema outputSchema(Schema input) {
try{
Schema.FieldSchema counter = new Schema.FieldSchema("counter", DataType.INTEGER);
Schema tupleSchema = new Schema(input.getField(0).schema.getField(0).schema.getFields());
tupleSchema.add(counter);
Schema.FieldSchema tupleFs;
tupleFs = new Schema.FieldSchema("with_counter", tupleSchema, DataType.TUPLE);
Schema bagSchema = new Schema(tupleFs);
return new Schema(new Schema.FieldSchema("row_counter",
bagSchema, DataType.BAG));
}catch (Exception e){
return null;
}
}
}
Thanks.

Categories

Resources