I am exploring Impala for a POC, however I can't see any significant performance. I can't insert 5000 records/sec, at max I was able to insert mere 200/sec. This is really slow considering any database performance.
I tried two different methods but both are slow:
Using Cloudera
First, I installed Cloudera on my system and added latest CDH 6.2 cluster. I created a java client to insert data using ImpalaJDBC41 driver. I am able to insert record but speed is terrible. I tried tuning impala by increasing Impala Daemon Limit and my system RAM but it didn't help. Finally, I thought there is something wrong with my installation or something so I switched to another method.
Using Cloudera VM
Cloudera also ships there ready VM for test purpose. I tried my hands on to see if it gives better performance, but there is no big improvement. I still can't insert data 5k/sec speed.
I don't know where do I need to improvement. I have pasted my code below if any improvement can be done.
What is the ideal Impala configuration to achieve speed of (5k - 10k / sec)? This speed is still very less of which Impala is capable.
private static Connection connectViaDS() throws Exception {
Connection connection = null;
Class.forName("com.cloudera.impala.jdbc41.Driver");
connection = DriverManager.getConnection(CONNECTION_URL);
return connection;
}
private static void writeInABatchWithCompiledQuery(int records) {
int protocol_no = 233,s_port=20,d_port=34,packet=46,volume=58,duration=39,pps=76,
bps=65,bpp=89,i_vol=465,e_vol=345,i_pkt=5,e_pkt=54,s_i_ix=654,d_i_ix=444,_time=1000,flow=989;
String s_city = "Mumbai",s_country = "India", s_latt = "12.165.34c", s_long = "39.56.32d",
s_host="motadata",d_latt="29.25.43c",d_long="49.15.26c",d_city="Damouli",d_country="Nepal";
long e_date= 1275822966, e_time= 1370517366;
PreparedStatement preparedStatement;
int total = 1000*1000;
int counter =0;
Connection connection = null;
try {
connection = connectViaDS();
preparedStatement = connection.prepareStatement(sqlCompiledQuery);
Timestamp ed = new Timestamp(e_date);
Timestamp et = new Timestamp(e_time);
while(counter <total) {
for (int index = 1; index <= 5000; index++) {
counter++;
preparedStatement.setString(1, "s_ip" + String.valueOf(index));
preparedStatement.setString(2, "d_ip" + String.valueOf(index));
preparedStatement.setInt(3, protocol_no + index);
preparedStatement.setInt(4, s_port + index);
preparedStatement.setInt(5, d_port + index);
preparedStatement.setInt(6, packet + index);
preparedStatement.setInt(7, volume + index);
preparedStatement.setInt(8, duration + index);
preparedStatement.setInt(9, pps + index);
preparedStatement.setInt(10, bps + index);
preparedStatement.setInt(11, bpp + index);
preparedStatement.setString(12, s_latt + String.valueOf(index));
preparedStatement.setString(13, s_long + String.valueOf(index));
preparedStatement.setString(14, s_city + String.valueOf(index));
preparedStatement.setString(15, s_country + String.valueOf(index));
preparedStatement.setString(16, d_latt + String.valueOf(index));
preparedStatement.setString(17, d_long + String.valueOf(index));
preparedStatement.setString(18, d_city + String.valueOf(index));
preparedStatement.setString(19, d_country + String.valueOf(index));
preparedStatement.setInt(20, i_vol + index);
preparedStatement.setInt(21, e_vol + index);
preparedStatement.setInt(22, i_pkt + index);
preparedStatement.setInt(23, e_pkt + index);
preparedStatement.setInt(24, s_i_ix + index);
preparedStatement.setInt(25, d_i_ix + index);
preparedStatement.setString(26, s_host + String.valueOf(index));
preparedStatement.setTimestamp(27, ed);
preparedStatement.setTimestamp(28, et);
preparedStatement.setInt(29, _time);
preparedStatement.setInt(30, flow + index);
preparedStatement.addBatch();
}
preparedStatement.executeBatch();
preparedStatement.clearBatch();
}
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
connection.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
}
Data is updating at snails pace. I tried increasing the batch size but it's decreasing the speed. I don't know if my code is wrong or I need to tune Impala for better performance. Please guide.
I am using VM for testing, here is other details:
System.
Os - Ubuntu 16
RAM - 12 gb
Cloudera - CDH 6.2
Impala daemon limit - 2 gb
Java heap size impala daemon - 500mb
HDFS Java Heap Size of NameNode in Bytes - 500mb.
Please let me know if more details are required.
You can't benchmark on a VM with 12GB. Look at the Impala's hardware requirements and you'll see you need 128GB of memory minimum.
Memory
128 GB or more recommended, ideally 256 GB or more. If the intermediate results during query processing on a particular node exceed the amount of memory available to Impala on that node, the query writes temporary work data to disk, which can lead to long query times. Note that because the work is parallelized, and intermediate results for aggregate queries are typically smaller than the original data, Impala can query and join tables that are much larger than the memory available on an individual node.
Also, the VM is used to familiarize yourself with the toolset but it is not powerful enough to even be a development environment.
References
Impala Requirements:Hardware Requirements
Tuning Impala for Performance
If you had read my other question, you'll know I've spent this weekend putting together a 6502 CPU emulator as a programming exercise.
The CPU emulator is mostly complete, and seems to be fairly accurate from my limited testing, however it is running incredibly fast, and I want to throttle it down to the actual clock speed of the machine.
My current test loop is this:
// Just loop infinitely.
while (1 == 1)
{
CPU.ClockCyclesBeforeNext--;
if (CPU.ClockCyclesBeforeNext <= 0)
{
// Find out how many clock cycles this instruction will take
CPU.ClockCyclesBeforeNext = CPU.OpcodeMapper.Map[CPU.Memory[CPU.PC]].CpuCycles;
// Run the instruction
CPU.ExecuteInstruction(CPU.Memory[CPU.PC]);
// Debugging Info
CPU.DumpDebug();
Console.WriteLine(CPU.OpcodeMapper.Map[CPU.Memory[CPU.PC]].ArgumentLength);
// Move to next instruction
CPU.PC += 1 + CPU.OpcodeMapper.Map[CPU.Memory[CPU.PC]].ArgumentLength;
}
}
As you can tell, each opcode takes a specific amount of time to complete, so I do not run the next instruction until I count down the CPU Cycle clock. This provides proper timing between opcodes, its just that the entire thing runs way to fast.
The targeted CPU speed is 1.79mhz, however I'd like whatever solution to the clock issue to keep the speed at 1.79mhz even as I add complexity, so I don't have to adjust it up.
Any ideas?
I wrote a Z80 emulator many years ago, and to do cycle accurate execution, I divided the clock rate into a number of small blocks and had the core execute that many clock cycles. In my case, I tied it to the frame rate of the game system I was emulating. Each opcode knew how many cycles it took to execute and the core would keep running opcodes until the specified number of cycles had been executed. I had an outer run loop that would run the cpu core, and run other parts of the emulated system and then sleep until the start time of the next iteration.
EDIT: Adding example of run loop.
int execute_run_loop( int cycles )
{
int n = 0;
while( n < cycles )
{
/* Returns number of cycles executed */
n += execute_next_opcode();
}
return n;
}
Hope this helps.
Take a look at the original quicktime documentation for inspiration.
It was written a long time ago, when displaying video meant just swapping still frames at high enough speed, but the Apple guys decided they needed a full time-management framework. The design at first looks overengineered, but it let them deal with widely different speed requirements and keep them tightly synchronized.
you're fortunate that 6502 has deterministic time behaviour, the exact time each instruction takes is well documented; but it's not constant. some instructions take 2 cycles, other 3. Just like frames in QuickTime, a video doesn't have a 'frames per second' parameter, each frame tells how long it wants to be in screen.
Since modern CPU's are so non-deterministic, and multitasking OS's can even freeze for a few miliseconds (virtual memory!), you should keep a tab if you're behind schedule, or if you can take a few microseconds nap.
As jfk says, the most common way to do this is tie the cpu speed to the vertical refresh of the (emulated) video output.
Pick a number of cycles to run per video frame. This will often be machine-specific but you can calculate it by something like :
cycles = clock speed in Hz / required frames-per-second
Then you also get to do a sleep until the video update is hit, at which point you start the next n cycles of CPU emulation.
If you're emulating something in particular then you just need to look up the fps rate and processor speed to get this approximately right.
EDIT: If you don't have any external timing requirements then it is normal for an emulator to just run as fast as it possibly can. Sometimes this is a desired effect and sometimes not :)
I would use the clock cycles to calculate time and them sleep the difference in time. Of course, to do this, you need a high-resolution clock. They way you are doing it is going to spike the CPU in spinning loops.
Yes, as said before most of the time you don't need a CPU emulator to emulate instructions at the same speed of the real thing. What user perceive is the output of the computation (i.e. audio and video outputs) so you only need to be in sync with such outputs which doesn't mean you must have necessarily an exact CPU emulation speed.
In other words, if the frame rate of the video input is, let's say, 50Hz, then let the CPU emulator run as fast as it can to draw the screen but be sure to output the screen frames at the correct rate (50Hz). From an external point of view your emulator is emulating at the correct speed.
Trying to be cycle exact even in the execution time is a non-sense on a multi-tasking OS like Windows or Linux because the emulator instruction time (tipically 1uS for vintage 80s CPUs) and the scheduling time slot of the modern OS are comparable.
Trying to output something at a 50Hz rate is a much simpler task you can do very good on any modern machine
Another option is available if audio emulation is implemented, and if audio output is tied to the system/CPU clock. In particular I know that this is the case with the 8-bit Apple ][ computers.
Usually sound is generated in buffers of a fixed size (which is a fixed time), so operation (generation of data etc) of these buffers can be tied to CPU throughput via synchronization primitives.
I am in the process of making something a little more general use case based, such as the ability to convert time to an estimated amount of instructions and vice versa.
The project homepage is # http://net7mma.codeplex.com
The code starts like this: (I think)
#region Copyright
/*
This file came from Managed Media Aggregation, You can always find the latest version # https://net7mma.codeplex.com/
Julius.Friedman#gmail.com / (SR. Software Engineer ASTI Transportation Inc. http://www.asti-trans.com)
Permission is hereby granted, free of charge,
* to any person obtaining a copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction,
* including without limitation the rights to :
* use,
* copy,
* modify,
* merge,
* publish,
* distribute,
* sublicense,
* and/or sell copies of the Software,
* and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
*
*
* JuliusFriedman#gmail.com should be contacted for further details.
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
*
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
* DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
* TORT OR OTHERWISE,
* ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*
* v//
*/
#endregion
namespace Media.Concepts.Classes
{
//Windows.Media.Clock has a fairly complex but complete API
/// <summary>
/// Provides a clock with a given offset and calendar.
/// </summary>
public class Clock : Media.Common.BaseDisposable
{
static bool GC = false;
#region Fields
/// <summary>
/// Indicates when the clock was created
/// </summary>
public readonly System.DateTimeOffset Created;
/// <summary>
/// The calendar system of the clock
/// </summary>
public readonly System.Globalization.Calendar Calendar;
/// <summary>
/// The amount of ticks which occur per update of the <see cref="System.Environment.TickCount"/> member.
/// </summary>
public readonly long TicksPerUpdate;
/// <summary>
/// The amount of instructions which occured when synchronizing with the system clock.
/// </summary>
public readonly long InstructionsPerClockUpdate;
#endregion
#region Properties
/// <summary>
/// The TimeZone offset of the clock from UTC
/// </summary>
public System.TimeSpan Offset { get { return Created.Offset; } }
/// <summary>
/// The average amount of operations per tick.
/// </summary>
public long AverageOperationsPerTick { get { return InstructionsPerClockUpdate / TicksPerUpdate; } }
/// <summary>
/// The <see cref="System.TimeSpan"/> which represents <see cref="TicksPerUpdate"/> as an amount of time.
/// </summary>
public System.TimeSpan SystemClockResolution { get { return System.TimeSpan.FromTicks(TicksPerUpdate); } }
/// <summary>
/// Return the current system time in the TimeZone offset of this clock
/// </summary>
public System.DateTimeOffset Now { get { return System.DateTimeOffset.Now.ToOffset(Offset).Add(new System.TimeSpan((long)(AverageOperationsPerTick / System.TimeSpan.TicksPerMillisecond))); } }
/// <summary>
/// Return the current system time in the TimeZone offset of this clock converter to UniversalTime.
/// </summary>
public System.DateTimeOffset UtcNow { get { return Now.ToUniversalTime(); } }
//public bool IsUtc { get { return Offset == System.TimeSpan.Zero; } }
//public bool IsDaylightSavingTime { get { return Created.LocalDateTime.IsDaylightSavingTime(); } }
#endregion
#region Constructor
/// <summary>
/// Creates a clock using the system's current timezone and calendar.
/// The system clock is profiled to determine it's accuracy
/// <see cref="System.DateTimeOffset.Now.Offset"/>
/// <see cref="System.Globalization.CultureInfo.CurrentCulture.Calendar"/>
/// </summary>
public Clock(bool shouldDispose = true)
: this(System.DateTimeOffset.Now.Offset, System.Globalization.CultureInfo.CurrentCulture.Calendar, shouldDispose)
{
try { if (false == GC && System.Runtime.GCSettings.LatencyMode != System.Runtime.GCLatencyMode.NoGCRegion) GC = System.GC.TryStartNoGCRegion(0); }
catch { }
finally
{
System.Threading.Thread.BeginCriticalRegion();
//Sample the TickCount
long ticksStart = System.Environment.TickCount,
ticksEnd;
//Continually sample the TickCount. while the value has not changed increment InstructionsPerClockUpdate
while ((ticksEnd = System.Environment.TickCount) == ticksStart) ++InstructionsPerClockUpdate; //+= 4; Read,Assign,Compare,Increment
//How many ticks occur per update of TickCount
TicksPerUpdate = ticksEnd - ticksStart;
System.Threading.Thread.EndCriticalRegion();
}
}
/// <summary>
/// Constructs a new clock using the given TimeZone offset and Calendar system
/// </summary>
/// <param name="timeZoneOffset"></param>
/// <param name="calendar"></param>
/// <param name="shouldDispose">Indicates if the instace should be diposed when Dispose is called.</param>
public Clock(System.TimeSpan timeZoneOffset, System.Globalization.Calendar calendar, bool shouldDispose = true)
{
//Allow disposal
ShouldDispose = shouldDispose;
Calendar = System.Globalization.CultureInfo.CurrentCulture.Calendar;
Created = new System.DateTimeOffset(System.DateTime.Now, timeZoneOffset);
}
#endregion
#region Overrides
public override void Dispose()
{
if (false == ShouldDispose) return;
base.Dispose();
try
{
if (System.Runtime.GCSettings.LatencyMode == System.Runtime.GCLatencyMode.NoGCRegion)
{
System.GC.EndNoGCRegion();
GC = false;
}
}
catch { }
}
#endregion
//Methods or statics for OperationCountToTimeSpan? (Estimate)
public void NanoSleep(int nanos)
{
Clock.NanoSleep((long)nanos);
}
public static void NanoSleep(long nanos)
{
System.Threading.Thread.BeginCriticalRegion();
NanoSleep(ref nanos);
System.Threading.Thread.EndCriticalRegion();
}
static void NanoSleep(ref long nanos)
{
try
{
unchecked
{
while (Common.Binary.Clamp(--nanos, 0, 1) >= 2)
{
/* if(--nanos % 2 == 0) */
NanoSleep(long.MinValue); //nanos -= 1 + (ops / (ulong)AverageOperationsPerTick);// *10;
}
}
}
catch
{
return;
}
}
}
}
Once you have some type of layman clock implementation you advance to something like a Timer
/// <summary>
/// Provides a Timer implementation which can be used across all platforms and does not rely on the existing Timer implementation.
/// </summary>
public class Timer : Common.BaseDisposable
{
readonly System.Threading.Thread m_Counter; // m_Consumer, m_Producer
internal System.TimeSpan m_Frequency;
internal ulong m_Ops = 0, m_Ticks = 0;
bool m_Enabled;
internal System.DateTimeOffset m_Started;
public delegate void TickEvent(ref long ticks);
public event TickEvent Tick;
public bool Enabled { get { return m_Enabled; } set { m_Enabled = value; } }
public System.TimeSpan Frequency { get { return m_Frequency; } }
internal ulong m_Bias;
//
//Could just use a single int, 32 bits is more than enough.
//uint m_Flags;
//
readonly internal Clock m_Clock = new Clock();
readonly internal System.Collections.Generic.Queue<long> Producer;
void Count()
{
System.Threading.Thread Event = new System.Threading.Thread(new System.Threading.ThreadStart(() =>
{
System.Threading.Thread.BeginCriticalRegion();
long sample;
AfterSample:
try
{
Top:
System.Threading.Thread.CurrentThread.Priority = System.Threading.ThreadPriority.Highest;
while (m_Enabled && Producer.Count >= 1)
{
sample = Producer.Dequeue();
Tick(ref sample);
}
System.Threading.Thread.CurrentThread.Priority = System.Threading.ThreadPriority.Lowest;
if (false == m_Enabled) return;
while (m_Enabled && Producer.Count == 0) if(m_Counter.IsAlive) m_Counter.Join(0); //++m_Ops;
goto Top;
}
catch { if (false == m_Enabled) return; goto AfterSample; }
finally { System.Threading.Thread.EndCriticalRegion(); }
}))
{
IsBackground = false,
Priority = System.Threading.ThreadPriority.AboveNormal
};
Event.TrySetApartmentState(System.Threading.ApartmentState.MTA);
Event.Start();
Approximate:
ulong approximate = (ulong)Common.Binary.Clamp((m_Clock.AverageOperationsPerTick / (Frequency.Ticks + 1)), 1, ulong.MaxValue);
try
{
m_Started = m_Clock.Now;
System.Threading.Thread.BeginCriticalRegion();
unchecked
{
Start:
if (IsDisposed) return;
switch (++m_Ops)
{
default:
{
if (m_Bias + ++m_Ops >= approximate)
{
System.Threading.Thread.CurrentThread.Priority = System.Threading.ThreadPriority.Highest;
Producer.Enqueue((long)m_Ticks++);
ulong x = ++m_Ops / approximate;
while (1 > --x /*&& Producer.Count <= m_Frequency.Ticks*/) Producer.Enqueue((long)++m_Ticks);
m_Ops = (++m_Ops * m_Ticks) - (m_Bias = ++m_Ops / approximate);
System.Threading.Thread.CurrentThread.Priority = System.Threading.ThreadPriority.Lowest;
}
if(Event != null) Event.Join(m_Frequency);
goto Start;
}
}
}
}
catch (System.Threading.ThreadAbortException) { if (m_Enabled) goto Approximate; System.Threading.Thread.ResetAbort(); }
catch (System.OutOfMemoryException) { if ((ulong)Producer.Count > approximate) Producer.Clear(); if (m_Enabled) goto Approximate; }
catch { if (m_Enabled) goto Approximate; }
finally
{
Event = null;
System.Threading.Thread.EndCriticalRegion();
}
}
public Timer(System.TimeSpan frequency)
{
Producer = new System.Collections.Generic.Queue<long>((int)(m_Frequency = frequency).Ticks * 10);
m_Counter = new System.Threading.Thread(new System.Threading.ThreadStart(Count))
{
IsBackground = false,
Priority = System.Threading.ThreadPriority.AboveNormal
};
m_Counter.TrySetApartmentState(System.Threading.ApartmentState.MTA);
Tick = delegate { m_Ops += 1 + m_Bias; };
}
public void Start()
{
if (m_Enabled) return;
m_Enabled = true;
m_Counter.Start();
var p = System.Threading.Thread.CurrentThread.Priority;
System.Threading.Thread.CurrentThread.Priority = System.Threading.ThreadPriority.Lowest;
while (m_Ops == 0) m_Counter.Join(0); //m_Clock.NanoSleep(0);
System.Threading.Thread.CurrentThread.Priority = p;
}
public void Stop()
{
m_Enabled = false;
}
void Change(System.TimeSpan interval, System.TimeSpan dueTime)
{
m_Enabled = false;
m_Frequency = interval;
m_Enabled = true;
}
delegate void ElapsedEvent(object sender, object args);
public override void Dispose()
{
if (IsDisposed) return;
base.Dispose();
Stop();
try { m_Counter.Abort(m_Frequency); }
catch (System.Threading.ThreadAbortException) { System.Threading.Thread.ResetAbort(); }
catch { }
Tick = null;
//Producer.Clear();
}
}
Then you can really replicate some logic using something like
/// <summary>
/// Provides a completely managed implementation of <see cref="System.Diagnostics.Stopwatch"/> which expresses time in the same units as <see cref="System.TimeSpan"/>.
/// </summary>
public class Stopwatch : Common.BaseDisposable
{
internal Timer Timer;
long Units;
public bool Enabled { get { return Timer != null && Timer.Enabled; } }
public double ElapsedMicroseconds { get { return Units * Media.Common.Extensions.TimeSpan.TimeSpanExtensions.TotalMicroseconds(Timer.Frequency); } }
public double ElapsedMilliseconds { get { return Units * Timer.Frequency.TotalMilliseconds; } }
public double ElapsedSeconds { get { return Units * Timer.Frequency.TotalSeconds; } }
//public System.TimeSpan Elapsed { get { return System.TimeSpan.FromMilliseconds(ElapsedMilliseconds / System.TimeSpan.TicksPerMillisecond); } }
public System.TimeSpan Elapsed
{
get
{
switch (Units)
{
case 0: return System.TimeSpan.Zero;
default:
{
System.TimeSpan taken = System.DateTime.UtcNow - Timer.m_Started;
return taken.Add(new System.TimeSpan(Units * Timer.Frequency.Ticks));
//System.TimeSpan additional = new System.TimeSpan(Media.Common.Extensions.Math.MathExtensions.Clamp(Units, 0, Timer.Frequency.Ticks));
//return taken.Add(additional);
}
}
//////The maximum amount of times the timer can elapse in the given frequency
////double maxCount = (taken.TotalMilliseconds / Timer.Frequency.TotalMilliseconds) / ElapsedMilliseconds;
////if (Units > maxCount)
////{
//// //How many more times the event was fired than needed
//// double overage = (maxCount - Units);
//// additional = new System.TimeSpan(System.Convert.ToInt64(Media.Common.Extensions.Math.MathExtensions.Clamp(Units, overage, maxCount)));
//// //return taken.Add(new System.TimeSpan((long)Media.Common.Extensions.Math.MathExtensions.Clamp(Units, overage, maxCount)));
////}
//////return taken.Add(new System.TimeSpan(Units));
}
}
public void Start()
{
if (Enabled) return;
Units = 0;
//Create a Timer that will elapse every OneTick //`OneMicrosecond`
Timer = new Timer(Media.Common.Extensions.TimeSpan.TimeSpanExtensions.OneTick);
//Handle the event by incrementing count
Timer.Tick += Count;
Timer.Start();
}
public void Stop()
{
if (false == Enabled) return;
Timer.Stop();
Timer.Dispose();
}
void Count(ref long count) { ++Units; }
}
Finally, create something semi useful e.g. a Bus and then perhaps a virtual screen to emit data to the bus...
public abstract class Bus : Common.CommonDisposable
{
public readonly Timer Clock = new Timer(Common.Extensions.TimeSpan.TimeSpanExtensions.OneTick);
public Bus() : base(false) { Clock.Start(); }
}
public class ClockedBus : Bus
{
long FrequencyHz, Maximum, End;
readonly Queue<byte[]> Input = new Queue<byte[]>(), Output = new Queue<byte[]>();
readonly double m_Bias;
public ClockedBus(long frequencyHz, double bias = 1.5)
{
m_Bias = bias;
cache = Clock.m_Clock.InstructionsPerClockUpdate / 1000;
SetFrequency(frequencyHz);
Clock.Tick += Clock_Tick;
Clock.Start();
}
public void SetFrequency(long frequencyHz)
{
FrequencyHz = frequencyHz;
//Clock.m_Frequency = new TimeSpan(Clock.m_Clock.InstructionsPerClockUpdate / 1000);
//Maximum = System.TimeSpan.TicksPerSecond / Clock.m_Clock.InstructionsPerClockUpdate;
//Maximum = Clock.m_Clock.InstructionsPerClockUpdate / System.TimeSpan.TicksPerSecond;
Maximum = cache / (cache / FrequencyHz);
Maximum *= System.TimeSpan.TicksPerSecond;
Maximum = (cache / FrequencyHz);
End = Maximum * 2;
Clock.m_Frequency = new TimeSpan(Maximum);
if (cache < frequencyHz * m_Bias) throw new Exception("Cannot obtain stable clock");
Clock.Producer.Clear();
}
public override void Dispose()
{
ShouldDispose = true;
Clock.Tick -= Clock_Tick;
Clock.Stop();
Clock.Dispose();
base.Dispose();
}
~ClockedBus() { Dispose(); }
long sample = 0, steps = 0, count = 0, avg = 0, cache = 1;
void Clock_Tick(ref long ticks)
{
if (ShouldDispose == false && false == IsDisposed)
{
//Console.WriteLine("#ops=>" + Clock.m_Ops + " #ticks=>" + Clock.m_Ticks + " #Lticks=>" + ticks + "#=>" + Clock.m_Clock.Now.TimeOfDay + "#=>" + (Clock.m_Clock.Now - Clock.m_Clock.Created));
steps = sample;
sample = ticks;
++count;
System.ConsoleColor f = System.Console.ForegroundColor;
if (count <= Maximum)
{
System.Console.BackgroundColor = ConsoleColor.Yellow;
System.Console.ForegroundColor = ConsoleColor.Green;
Console.WriteLine("count=> " + count + "#=>" + Clock.m_Clock.Now.TimeOfDay + "#=>" + (Clock.m_Clock.Now - Clock.m_Clock.Created) + " - " + DateTime.UtcNow.ToString("MM/dd/yyyy hh:mm:ss.ffffff tt"));
avg = Maximum / count;
if (Clock.m_Clock.InstructionsPerClockUpdate / count > Maximum)
{
System.Console.ForegroundColor = ConsoleColor.Red;
Console.WriteLine("---- Over InstructionsPerClockUpdate ----" + FrequencyHz);
}
}
else if (count >= End)
{
System.Console.BackgroundColor = ConsoleColor.Black;
System.Console.ForegroundColor = ConsoleColor.Blue;
avg = Maximum / count;
Console.WriteLine("avg=> " + avg + "#=>" + FrequencyHz);
count = 0;
}
}
}
//Read, Write at Frequency
}
public class VirtualScreen
{
TimeSpan RefreshRate;
bool VerticalSync;
int Width, Height;
Common.MemorySegment DisplayMemory, BackBuffer, DisplayBuffer;
}
Here is how I tested the StopWatch
internal class StopWatchTests
{
public void TestForOneMicrosecond()
{
System.Collections.Generic.List<System.Tuple<bool, System.TimeSpan, System.TimeSpan>> l = new System.Collections.Generic.List<System.Tuple<bool, System.TimeSpan, System.TimeSpan>>();
//Create a Timer that will elapse every `OneMicrosecond`
for (int i = 0; i <= 250; ++i) using (Media.Concepts.Classes.Stopwatch sw = new Media.Concepts.Classes.Stopwatch())
{
var started = System.DateTime.UtcNow;
System.Console.WriteLine("Started: " + started.ToString("MM/dd/yyyy hh:mm:ss.ffffff tt"));
//Define some amount of time
System.TimeSpan sleepTime = Media.Common.Extensions.TimeSpan.TimeSpanExtensions.OneMicrosecond;
System.Diagnostics.Stopwatch testSw = new System.Diagnostics.Stopwatch();
//Start
testSw.Start();
//Start
sw.Start();
while (testSw.Elapsed.Ticks < sleepTime.Ticks - (Common.Extensions.TimeSpan.TimeSpanExtensions.OneTick + Common.Extensions.TimeSpan.TimeSpanExtensions.OneTick).Ticks)
sw.Timer.m_Clock.NanoSleep(0); //System.Threading.Thread.SpinWait(0);
//Sleep the desired amount
//System.Threading.Thread.Sleep(sleepTime);
//Stop
testSw.Stop();
//Stop
sw.Stop();
var finished = System.DateTime.UtcNow;
var taken = finished - started;
var cc = System.Console.ForegroundColor;
System.Console.WriteLine("Finished: " + finished.ToString("MM/dd/yyyy hh:mm:ss.ffffff tt"));
System.Console.WriteLine("Sleep Time: " + sleepTime.ToString());
System.Console.WriteLine("Real Taken Total: " + taken.ToString());
if (taken > sleepTime)
{
System.Console.ForegroundColor = System.ConsoleColor.Red;
System.Console.WriteLine("Missed by: " + (taken - sleepTime));
}
else
{
System.Console.ForegroundColor = System.ConsoleColor.Green;
System.Console.WriteLine("Still have: " + (sleepTime - taken));
}
System.Console.ForegroundColor = cc;
System.Console.WriteLine("Real Taken msec Total: " + taken.TotalMilliseconds.ToString());
System.Console.WriteLine("Real Taken sec Total: " + taken.TotalSeconds.ToString());
System.Console.WriteLine("Real Taken μs Total: " + Media.Common.Extensions.TimeSpan.TimeSpanExtensions.TotalMicroseconds(taken).ToString());
System.Console.WriteLine("Managed Taken Total: " + sw.Elapsed.ToString());
System.Console.WriteLine("Diagnostic Taken Total: " + testSw.Elapsed.ToString());
System.Console.WriteLine("Diagnostic Elapsed Seconds Total: " + ((testSw.ElapsedTicks / (double)System.Diagnostics.Stopwatch.Frequency)));
//Write the rough amount of time taken in micro seconds
System.Console.WriteLine("Managed Time Estimated Taken: " + sw.ElapsedMicroseconds + "μs");
//Write the rough amount of time taken in micro seconds
System.Console.WriteLine("Diagnostic Time Estimated Taken: " + Media.Common.Extensions.TimeSpan.TimeSpanExtensions.TotalMicroseconds(testSw.Elapsed) + "μs");
System.Console.WriteLine("Managed Time Estimated Taken: " + sw.ElapsedMilliseconds);
System.Console.WriteLine("Diagnostic Time Estimated Taken: " + testSw.ElapsedMilliseconds);
System.Console.WriteLine("Managed Time Estimated Taken: " + sw.ElapsedSeconds);
System.Console.WriteLine("Diagnostic Time Estimated Taken: " + testSw.Elapsed.TotalSeconds);
if (sw.Elapsed < testSw.Elapsed)
{
System.Console.WriteLine("Faster than Diagnostic StopWatch");
l.Add(new System.Tuple<bool, System.TimeSpan, System.TimeSpan>(true, sw.Elapsed, testSw.Elapsed));
}
else if (sw.Elapsed > testSw.Elapsed)
{
System.Console.WriteLine("Slower than Diagnostic StopWatch");
l.Add(new System.Tuple<bool, System.TimeSpan, System.TimeSpan>(false, sw.Elapsed, testSw.Elapsed));
}
else
{
System.Console.WriteLine("Equal to Diagnostic StopWatch");
l.Add(new System.Tuple<bool, System.TimeSpan, System.TimeSpan>(true, sw.Elapsed, testSw.Elapsed));
}
}
int w = 0, f = 0;
var cc2 = System.Console.ForegroundColor;
foreach (var t in l)
{
if (t.Item1)
{
System.Console.ForegroundColor = System.ConsoleColor.Green;
++w; System.Console.WriteLine("Faster than Diagnostic StopWatch by: " + (t.Item3 - t.Item2));
}
else
{
System.Console.ForegroundColor = System.ConsoleColor.Red;
++f; System.Console.WriteLine("Slower than Diagnostic StopWatch by: " + (t.Item2 - t.Item3));
}
}
System.Console.ForegroundColor = System.ConsoleColor.Green;
System.Console.WriteLine("Wins = " + w);
System.Console.ForegroundColor = System.ConsoleColor.Red;
System.Console.WriteLine("Loss = " + f);
System.Console.ForegroundColor = cc2;
}
}
I'm trying to get the Total Ram in Java using Sigar library, I do the following
return String.valueOf(sigar.getMem().getRam());
My total RAM is 4GB, so I was expecting 4 or 4.00 but the result is 4008.
I tried the following:
long x = sigar.getMem().getTotal();
final String[] units = new String[] { "B", "KB", "MB", "GB", "TB" };
int digitGroups = (int) (Math.log10(x) / Math.log10(1024));
return new DecimalFormat("#,##0.##")
.format(x / Math.pow(1024, digitGroups)) + " " + units[digitGroups];
but still I'm having the result of 3.91 GB.
Any Idea?
You probably have 3.91 physically available GigaBytes, and the math from Sigar is correct.
Check if your BIOS is reserving some RAM for the integrated graphics device or some other devilry.
Quite simply, how can I get the amount of memory (in MB) that my android app is currently using? This would need to be done in Java so I can display this information to the user.
I've looked at other stackoverflow posts but none give a simple or accurate answer to this problem.
adb shell dumpsys meminfo packagename
Try to execute this command with Java.
You can use ActivityManager for that purpose.
It's answered in this post.
Try to use this code if a performance is not critical:
Debug.MemoryInfo memInfo = new Debug.MemoryInfo();
Debug.getMemoryInfo(memInfo);
long res = memInfo.getTotalPrivateDirty();
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.KITKAT)
res += memInfo.getTotalPrivateClean();
return res * 1024L;
If performance is critical check this answer
1. dumpsys meminfo PKG_NAME
...
App Summary
Pss(KB)
------
Java Heap: 42868 <--
Native Heap: 52268 <--
Code: 23608
Stack: 96
Graphics: 5084
Private Other: 5620
System: 14900
TOTAL: 144444 <-- TOTAL SWAP PSS: 130
2. in java code
https://android.googlesource.com/platform/frameworks/base/+/master/core/java/android/os/Debug.java#640
// this is the value source of dumpsys meminfo.
Debug.MemoryInfo memInfo = new Debug.MemoryInfo();
Debug.getMemoryInfo(memInfo);
if (android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.M) {
String javaMem = memInfo.getMemoryStat("summary.java-heap");
String natiMem = memInfo.getMemoryStat("summary.native-heap");
String totalMem = memInfo.getMemoryStat("summary.total-pss");
//msg2 += "\n" + String.format("%s %s %s", javaMem, natiMem, totalMem);
msg2 += "\n\n"+String.format("java %8s\nnati %8s\ntotal %8s",
formatMB(Integer.parseInt(javaMem)), formatMB(Integer.parseInt(natiMem)), formatMB(Integer.parseInt(totalMem))
);
}
...
String formatMB(double KB){
return String.format("%.1f MB", KB/1024);
}
The other api need lots of calculate,
// you can read them in Debug.java src file.
This is the closest to dumpsys meminfo or android studio monitor.
How to get the Ram size and Hard disk size of the PC using Java? And Is it possible to get the currently logged user name on PC through java?
Disk size:
long diskSize = new File("/").getTotalSpace();
User name:
String userName = System.getProperty("user.name");
I'm not aware of a reliable way to determine total system memory in Java. On a Unix system you could parse /proc/meminfo. You can of course find the maximum memory available to the JVM:
long maxMemory = Runtime.getRuntime().maxMemory();
Edit: for completeness (thanks Suresh S), here's a way to get total memory with the Oracle JVM only:
long memorySize = ((com.sun.management.OperatingSystemMXBean) ManagementFactory
.getOperatingSystemMXBean()).getTotalPhysicalMemorySize();
For Ram Size , if you are using java 1.5
java.lang.management package
com.sun.management.OperatingSystemMXBean mxbean = (com.sun.management.OperatingSystemMXBean) ManagementFactory.getOperatingSystemMXBean();
System.out.println(mxbean.getTotalPhysicalMemorySize() + " Bytes ");
import java.lang.management.*;
import java.io.*;
class max
{
public static void main(String... a)
{
long diskSize = new File("/").getTotalSpace();
String userName = System.getProperty("user.name");
long maxMemory = Runtime.getRuntime().maxMemory();
long memorySize = ((com.sun.management.OperatingSystemMXBean) ManagementFactory.getOperatingSystemMXBean()).getTotalPhysicalMemorySize();
System.out.println("Size of C:="+diskSize+" Bytes");
System.out.println("User Name="+userName);
System.out.println("RAM Size="+memorySize+" Bytes");
}
}
Have a look at this topic, which goes into detail of how to get OS information such as this.
For Ram capacity:
//this step get ram capacity
long ram= ((com.sun.management.OperatingSystemMXBean) ManagementFactory.getOperatingSystemMXBean()).getTotalPhysicalMemorySize();
long sizekb = ram /1000;
long sizemb = sizekb / 1000;
long sizegb = sizemb / 1000 ;
System.out.println("System Ram ="+sizegb+"gb");