If you had read my other question, you'll know I've spent this weekend putting together a 6502 CPU emulator as a programming exercise.
The CPU emulator is mostly complete, and seems to be fairly accurate from my limited testing, however it is running incredibly fast, and I want to throttle it down to the actual clock speed of the machine.
My current test loop is this:
// Just loop infinitely.
while (1 == 1)
{
CPU.ClockCyclesBeforeNext--;
if (CPU.ClockCyclesBeforeNext <= 0)
{
// Find out how many clock cycles this instruction will take
CPU.ClockCyclesBeforeNext = CPU.OpcodeMapper.Map[CPU.Memory[CPU.PC]].CpuCycles;
// Run the instruction
CPU.ExecuteInstruction(CPU.Memory[CPU.PC]);
// Debugging Info
CPU.DumpDebug();
Console.WriteLine(CPU.OpcodeMapper.Map[CPU.Memory[CPU.PC]].ArgumentLength);
// Move to next instruction
CPU.PC += 1 + CPU.OpcodeMapper.Map[CPU.Memory[CPU.PC]].ArgumentLength;
}
}
As you can tell, each opcode takes a specific amount of time to complete, so I do not run the next instruction until I count down the CPU Cycle clock. This provides proper timing between opcodes, its just that the entire thing runs way to fast.
The targeted CPU speed is 1.79mhz, however I'd like whatever solution to the clock issue to keep the speed at 1.79mhz even as I add complexity, so I don't have to adjust it up.
Any ideas?
I wrote a Z80 emulator many years ago, and to do cycle accurate execution, I divided the clock rate into a number of small blocks and had the core execute that many clock cycles. In my case, I tied it to the frame rate of the game system I was emulating. Each opcode knew how many cycles it took to execute and the core would keep running opcodes until the specified number of cycles had been executed. I had an outer run loop that would run the cpu core, and run other parts of the emulated system and then sleep until the start time of the next iteration.
EDIT: Adding example of run loop.
int execute_run_loop( int cycles )
{
int n = 0;
while( n < cycles )
{
/* Returns number of cycles executed */
n += execute_next_opcode();
}
return n;
}
Hope this helps.
Take a look at the original quicktime documentation for inspiration.
It was written a long time ago, when displaying video meant just swapping still frames at high enough speed, but the Apple guys decided they needed a full time-management framework. The design at first looks overengineered, but it let them deal with widely different speed requirements and keep them tightly synchronized.
you're fortunate that 6502 has deterministic time behaviour, the exact time each instruction takes is well documented; but it's not constant. some instructions take 2 cycles, other 3. Just like frames in QuickTime, a video doesn't have a 'frames per second' parameter, each frame tells how long it wants to be in screen.
Since modern CPU's are so non-deterministic, and multitasking OS's can even freeze for a few miliseconds (virtual memory!), you should keep a tab if you're behind schedule, or if you can take a few microseconds nap.
As jfk says, the most common way to do this is tie the cpu speed to the vertical refresh of the (emulated) video output.
Pick a number of cycles to run per video frame. This will often be machine-specific but you can calculate it by something like :
cycles = clock speed in Hz / required frames-per-second
Then you also get to do a sleep until the video update is hit, at which point you start the next n cycles of CPU emulation.
If you're emulating something in particular then you just need to look up the fps rate and processor speed to get this approximately right.
EDIT: If you don't have any external timing requirements then it is normal for an emulator to just run as fast as it possibly can. Sometimes this is a desired effect and sometimes not :)
I would use the clock cycles to calculate time and them sleep the difference in time. Of course, to do this, you need a high-resolution clock. They way you are doing it is going to spike the CPU in spinning loops.
Yes, as said before most of the time you don't need a CPU emulator to emulate instructions at the same speed of the real thing. What user perceive is the output of the computation (i.e. audio and video outputs) so you only need to be in sync with such outputs which doesn't mean you must have necessarily an exact CPU emulation speed.
In other words, if the frame rate of the video input is, let's say, 50Hz, then let the CPU emulator run as fast as it can to draw the screen but be sure to output the screen frames at the correct rate (50Hz). From an external point of view your emulator is emulating at the correct speed.
Trying to be cycle exact even in the execution time is a non-sense on a multi-tasking OS like Windows or Linux because the emulator instruction time (tipically 1uS for vintage 80s CPUs) and the scheduling time slot of the modern OS are comparable.
Trying to output something at a 50Hz rate is a much simpler task you can do very good on any modern machine
Another option is available if audio emulation is implemented, and if audio output is tied to the system/CPU clock. In particular I know that this is the case with the 8-bit Apple ][ computers.
Usually sound is generated in buffers of a fixed size (which is a fixed time), so operation (generation of data etc) of these buffers can be tied to CPU throughput via synchronization primitives.
I am in the process of making something a little more general use case based, such as the ability to convert time to an estimated amount of instructions and vice versa.
The project homepage is # http://net7mma.codeplex.com
The code starts like this: (I think)
#region Copyright
/*
This file came from Managed Media Aggregation, You can always find the latest version # https://net7mma.codeplex.com/
Julius.Friedman#gmail.com / (SR. Software Engineer ASTI Transportation Inc. http://www.asti-trans.com)
Permission is hereby granted, free of charge,
* to any person obtaining a copy of this software and associated documentation files (the "Software"),
* to deal in the Software without restriction,
* including without limitation the rights to :
* use,
* copy,
* modify,
* merge,
* publish,
* distribute,
* sublicense,
* and/or sell copies of the Software,
* and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
*
*
* JuliusFriedman#gmail.com should be contacted for further details.
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
*
* IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
* DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
* TORT OR OTHERWISE,
* ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*
* v//
*/
#endregion
namespace Media.Concepts.Classes
{
//Windows.Media.Clock has a fairly complex but complete API
/// <summary>
/// Provides a clock with a given offset and calendar.
/// </summary>
public class Clock : Media.Common.BaseDisposable
{
static bool GC = false;
#region Fields
/// <summary>
/// Indicates when the clock was created
/// </summary>
public readonly System.DateTimeOffset Created;
/// <summary>
/// The calendar system of the clock
/// </summary>
public readonly System.Globalization.Calendar Calendar;
/// <summary>
/// The amount of ticks which occur per update of the <see cref="System.Environment.TickCount"/> member.
/// </summary>
public readonly long TicksPerUpdate;
/// <summary>
/// The amount of instructions which occured when synchronizing with the system clock.
/// </summary>
public readonly long InstructionsPerClockUpdate;
#endregion
#region Properties
/// <summary>
/// The TimeZone offset of the clock from UTC
/// </summary>
public System.TimeSpan Offset { get { return Created.Offset; } }
/// <summary>
/// The average amount of operations per tick.
/// </summary>
public long AverageOperationsPerTick { get { return InstructionsPerClockUpdate / TicksPerUpdate; } }
/// <summary>
/// The <see cref="System.TimeSpan"/> which represents <see cref="TicksPerUpdate"/> as an amount of time.
/// </summary>
public System.TimeSpan SystemClockResolution { get { return System.TimeSpan.FromTicks(TicksPerUpdate); } }
/// <summary>
/// Return the current system time in the TimeZone offset of this clock
/// </summary>
public System.DateTimeOffset Now { get { return System.DateTimeOffset.Now.ToOffset(Offset).Add(new System.TimeSpan((long)(AverageOperationsPerTick / System.TimeSpan.TicksPerMillisecond))); } }
/// <summary>
/// Return the current system time in the TimeZone offset of this clock converter to UniversalTime.
/// </summary>
public System.DateTimeOffset UtcNow { get { return Now.ToUniversalTime(); } }
//public bool IsUtc { get { return Offset == System.TimeSpan.Zero; } }
//public bool IsDaylightSavingTime { get { return Created.LocalDateTime.IsDaylightSavingTime(); } }
#endregion
#region Constructor
/// <summary>
/// Creates a clock using the system's current timezone and calendar.
/// The system clock is profiled to determine it's accuracy
/// <see cref="System.DateTimeOffset.Now.Offset"/>
/// <see cref="System.Globalization.CultureInfo.CurrentCulture.Calendar"/>
/// </summary>
public Clock(bool shouldDispose = true)
: this(System.DateTimeOffset.Now.Offset, System.Globalization.CultureInfo.CurrentCulture.Calendar, shouldDispose)
{
try { if (false == GC && System.Runtime.GCSettings.LatencyMode != System.Runtime.GCLatencyMode.NoGCRegion) GC = System.GC.TryStartNoGCRegion(0); }
catch { }
finally
{
System.Threading.Thread.BeginCriticalRegion();
//Sample the TickCount
long ticksStart = System.Environment.TickCount,
ticksEnd;
//Continually sample the TickCount. while the value has not changed increment InstructionsPerClockUpdate
while ((ticksEnd = System.Environment.TickCount) == ticksStart) ++InstructionsPerClockUpdate; //+= 4; Read,Assign,Compare,Increment
//How many ticks occur per update of TickCount
TicksPerUpdate = ticksEnd - ticksStart;
System.Threading.Thread.EndCriticalRegion();
}
}
/// <summary>
/// Constructs a new clock using the given TimeZone offset and Calendar system
/// </summary>
/// <param name="timeZoneOffset"></param>
/// <param name="calendar"></param>
/// <param name="shouldDispose">Indicates if the instace should be diposed when Dispose is called.</param>
public Clock(System.TimeSpan timeZoneOffset, System.Globalization.Calendar calendar, bool shouldDispose = true)
{
//Allow disposal
ShouldDispose = shouldDispose;
Calendar = System.Globalization.CultureInfo.CurrentCulture.Calendar;
Created = new System.DateTimeOffset(System.DateTime.Now, timeZoneOffset);
}
#endregion
#region Overrides
public override void Dispose()
{
if (false == ShouldDispose) return;
base.Dispose();
try
{
if (System.Runtime.GCSettings.LatencyMode == System.Runtime.GCLatencyMode.NoGCRegion)
{
System.GC.EndNoGCRegion();
GC = false;
}
}
catch { }
}
#endregion
//Methods or statics for OperationCountToTimeSpan? (Estimate)
public void NanoSleep(int nanos)
{
Clock.NanoSleep((long)nanos);
}
public static void NanoSleep(long nanos)
{
System.Threading.Thread.BeginCriticalRegion();
NanoSleep(ref nanos);
System.Threading.Thread.EndCriticalRegion();
}
static void NanoSleep(ref long nanos)
{
try
{
unchecked
{
while (Common.Binary.Clamp(--nanos, 0, 1) >= 2)
{
/* if(--nanos % 2 == 0) */
NanoSleep(long.MinValue); //nanos -= 1 + (ops / (ulong)AverageOperationsPerTick);// *10;
}
}
}
catch
{
return;
}
}
}
}
Once you have some type of layman clock implementation you advance to something like a Timer
/// <summary>
/// Provides a Timer implementation which can be used across all platforms and does not rely on the existing Timer implementation.
/// </summary>
public class Timer : Common.BaseDisposable
{
readonly System.Threading.Thread m_Counter; // m_Consumer, m_Producer
internal System.TimeSpan m_Frequency;
internal ulong m_Ops = 0, m_Ticks = 0;
bool m_Enabled;
internal System.DateTimeOffset m_Started;
public delegate void TickEvent(ref long ticks);
public event TickEvent Tick;
public bool Enabled { get { return m_Enabled; } set { m_Enabled = value; } }
public System.TimeSpan Frequency { get { return m_Frequency; } }
internal ulong m_Bias;
//
//Could just use a single int, 32 bits is more than enough.
//uint m_Flags;
//
readonly internal Clock m_Clock = new Clock();
readonly internal System.Collections.Generic.Queue<long> Producer;
void Count()
{
System.Threading.Thread Event = new System.Threading.Thread(new System.Threading.ThreadStart(() =>
{
System.Threading.Thread.BeginCriticalRegion();
long sample;
AfterSample:
try
{
Top:
System.Threading.Thread.CurrentThread.Priority = System.Threading.ThreadPriority.Highest;
while (m_Enabled && Producer.Count >= 1)
{
sample = Producer.Dequeue();
Tick(ref sample);
}
System.Threading.Thread.CurrentThread.Priority = System.Threading.ThreadPriority.Lowest;
if (false == m_Enabled) return;
while (m_Enabled && Producer.Count == 0) if(m_Counter.IsAlive) m_Counter.Join(0); //++m_Ops;
goto Top;
}
catch { if (false == m_Enabled) return; goto AfterSample; }
finally { System.Threading.Thread.EndCriticalRegion(); }
}))
{
IsBackground = false,
Priority = System.Threading.ThreadPriority.AboveNormal
};
Event.TrySetApartmentState(System.Threading.ApartmentState.MTA);
Event.Start();
Approximate:
ulong approximate = (ulong)Common.Binary.Clamp((m_Clock.AverageOperationsPerTick / (Frequency.Ticks + 1)), 1, ulong.MaxValue);
try
{
m_Started = m_Clock.Now;
System.Threading.Thread.BeginCriticalRegion();
unchecked
{
Start:
if (IsDisposed) return;
switch (++m_Ops)
{
default:
{
if (m_Bias + ++m_Ops >= approximate)
{
System.Threading.Thread.CurrentThread.Priority = System.Threading.ThreadPriority.Highest;
Producer.Enqueue((long)m_Ticks++);
ulong x = ++m_Ops / approximate;
while (1 > --x /*&& Producer.Count <= m_Frequency.Ticks*/) Producer.Enqueue((long)++m_Ticks);
m_Ops = (++m_Ops * m_Ticks) - (m_Bias = ++m_Ops / approximate);
System.Threading.Thread.CurrentThread.Priority = System.Threading.ThreadPriority.Lowest;
}
if(Event != null) Event.Join(m_Frequency);
goto Start;
}
}
}
}
catch (System.Threading.ThreadAbortException) { if (m_Enabled) goto Approximate; System.Threading.Thread.ResetAbort(); }
catch (System.OutOfMemoryException) { if ((ulong)Producer.Count > approximate) Producer.Clear(); if (m_Enabled) goto Approximate; }
catch { if (m_Enabled) goto Approximate; }
finally
{
Event = null;
System.Threading.Thread.EndCriticalRegion();
}
}
public Timer(System.TimeSpan frequency)
{
Producer = new System.Collections.Generic.Queue<long>((int)(m_Frequency = frequency).Ticks * 10);
m_Counter = new System.Threading.Thread(new System.Threading.ThreadStart(Count))
{
IsBackground = false,
Priority = System.Threading.ThreadPriority.AboveNormal
};
m_Counter.TrySetApartmentState(System.Threading.ApartmentState.MTA);
Tick = delegate { m_Ops += 1 + m_Bias; };
}
public void Start()
{
if (m_Enabled) return;
m_Enabled = true;
m_Counter.Start();
var p = System.Threading.Thread.CurrentThread.Priority;
System.Threading.Thread.CurrentThread.Priority = System.Threading.ThreadPriority.Lowest;
while (m_Ops == 0) m_Counter.Join(0); //m_Clock.NanoSleep(0);
System.Threading.Thread.CurrentThread.Priority = p;
}
public void Stop()
{
m_Enabled = false;
}
void Change(System.TimeSpan interval, System.TimeSpan dueTime)
{
m_Enabled = false;
m_Frequency = interval;
m_Enabled = true;
}
delegate void ElapsedEvent(object sender, object args);
public override void Dispose()
{
if (IsDisposed) return;
base.Dispose();
Stop();
try { m_Counter.Abort(m_Frequency); }
catch (System.Threading.ThreadAbortException) { System.Threading.Thread.ResetAbort(); }
catch { }
Tick = null;
//Producer.Clear();
}
}
Then you can really replicate some logic using something like
/// <summary>
/// Provides a completely managed implementation of <see cref="System.Diagnostics.Stopwatch"/> which expresses time in the same units as <see cref="System.TimeSpan"/>.
/// </summary>
public class Stopwatch : Common.BaseDisposable
{
internal Timer Timer;
long Units;
public bool Enabled { get { return Timer != null && Timer.Enabled; } }
public double ElapsedMicroseconds { get { return Units * Media.Common.Extensions.TimeSpan.TimeSpanExtensions.TotalMicroseconds(Timer.Frequency); } }
public double ElapsedMilliseconds { get { return Units * Timer.Frequency.TotalMilliseconds; } }
public double ElapsedSeconds { get { return Units * Timer.Frequency.TotalSeconds; } }
//public System.TimeSpan Elapsed { get { return System.TimeSpan.FromMilliseconds(ElapsedMilliseconds / System.TimeSpan.TicksPerMillisecond); } }
public System.TimeSpan Elapsed
{
get
{
switch (Units)
{
case 0: return System.TimeSpan.Zero;
default:
{
System.TimeSpan taken = System.DateTime.UtcNow - Timer.m_Started;
return taken.Add(new System.TimeSpan(Units * Timer.Frequency.Ticks));
//System.TimeSpan additional = new System.TimeSpan(Media.Common.Extensions.Math.MathExtensions.Clamp(Units, 0, Timer.Frequency.Ticks));
//return taken.Add(additional);
}
}
//////The maximum amount of times the timer can elapse in the given frequency
////double maxCount = (taken.TotalMilliseconds / Timer.Frequency.TotalMilliseconds) / ElapsedMilliseconds;
////if (Units > maxCount)
////{
//// //How many more times the event was fired than needed
//// double overage = (maxCount - Units);
//// additional = new System.TimeSpan(System.Convert.ToInt64(Media.Common.Extensions.Math.MathExtensions.Clamp(Units, overage, maxCount)));
//// //return taken.Add(new System.TimeSpan((long)Media.Common.Extensions.Math.MathExtensions.Clamp(Units, overage, maxCount)));
////}
//////return taken.Add(new System.TimeSpan(Units));
}
}
public void Start()
{
if (Enabled) return;
Units = 0;
//Create a Timer that will elapse every OneTick //`OneMicrosecond`
Timer = new Timer(Media.Common.Extensions.TimeSpan.TimeSpanExtensions.OneTick);
//Handle the event by incrementing count
Timer.Tick += Count;
Timer.Start();
}
public void Stop()
{
if (false == Enabled) return;
Timer.Stop();
Timer.Dispose();
}
void Count(ref long count) { ++Units; }
}
Finally, create something semi useful e.g. a Bus and then perhaps a virtual screen to emit data to the bus...
public abstract class Bus : Common.CommonDisposable
{
public readonly Timer Clock = new Timer(Common.Extensions.TimeSpan.TimeSpanExtensions.OneTick);
public Bus() : base(false) { Clock.Start(); }
}
public class ClockedBus : Bus
{
long FrequencyHz, Maximum, End;
readonly Queue<byte[]> Input = new Queue<byte[]>(), Output = new Queue<byte[]>();
readonly double m_Bias;
public ClockedBus(long frequencyHz, double bias = 1.5)
{
m_Bias = bias;
cache = Clock.m_Clock.InstructionsPerClockUpdate / 1000;
SetFrequency(frequencyHz);
Clock.Tick += Clock_Tick;
Clock.Start();
}
public void SetFrequency(long frequencyHz)
{
FrequencyHz = frequencyHz;
//Clock.m_Frequency = new TimeSpan(Clock.m_Clock.InstructionsPerClockUpdate / 1000);
//Maximum = System.TimeSpan.TicksPerSecond / Clock.m_Clock.InstructionsPerClockUpdate;
//Maximum = Clock.m_Clock.InstructionsPerClockUpdate / System.TimeSpan.TicksPerSecond;
Maximum = cache / (cache / FrequencyHz);
Maximum *= System.TimeSpan.TicksPerSecond;
Maximum = (cache / FrequencyHz);
End = Maximum * 2;
Clock.m_Frequency = new TimeSpan(Maximum);
if (cache < frequencyHz * m_Bias) throw new Exception("Cannot obtain stable clock");
Clock.Producer.Clear();
}
public override void Dispose()
{
ShouldDispose = true;
Clock.Tick -= Clock_Tick;
Clock.Stop();
Clock.Dispose();
base.Dispose();
}
~ClockedBus() { Dispose(); }
long sample = 0, steps = 0, count = 0, avg = 0, cache = 1;
void Clock_Tick(ref long ticks)
{
if (ShouldDispose == false && false == IsDisposed)
{
//Console.WriteLine("#ops=>" + Clock.m_Ops + " #ticks=>" + Clock.m_Ticks + " #Lticks=>" + ticks + "#=>" + Clock.m_Clock.Now.TimeOfDay + "#=>" + (Clock.m_Clock.Now - Clock.m_Clock.Created));
steps = sample;
sample = ticks;
++count;
System.ConsoleColor f = System.Console.ForegroundColor;
if (count <= Maximum)
{
System.Console.BackgroundColor = ConsoleColor.Yellow;
System.Console.ForegroundColor = ConsoleColor.Green;
Console.WriteLine("count=> " + count + "#=>" + Clock.m_Clock.Now.TimeOfDay + "#=>" + (Clock.m_Clock.Now - Clock.m_Clock.Created) + " - " + DateTime.UtcNow.ToString("MM/dd/yyyy hh:mm:ss.ffffff tt"));
avg = Maximum / count;
if (Clock.m_Clock.InstructionsPerClockUpdate / count > Maximum)
{
System.Console.ForegroundColor = ConsoleColor.Red;
Console.WriteLine("---- Over InstructionsPerClockUpdate ----" + FrequencyHz);
}
}
else if (count >= End)
{
System.Console.BackgroundColor = ConsoleColor.Black;
System.Console.ForegroundColor = ConsoleColor.Blue;
avg = Maximum / count;
Console.WriteLine("avg=> " + avg + "#=>" + FrequencyHz);
count = 0;
}
}
}
//Read, Write at Frequency
}
public class VirtualScreen
{
TimeSpan RefreshRate;
bool VerticalSync;
int Width, Height;
Common.MemorySegment DisplayMemory, BackBuffer, DisplayBuffer;
}
Here is how I tested the StopWatch
internal class StopWatchTests
{
public void TestForOneMicrosecond()
{
System.Collections.Generic.List<System.Tuple<bool, System.TimeSpan, System.TimeSpan>> l = new System.Collections.Generic.List<System.Tuple<bool, System.TimeSpan, System.TimeSpan>>();
//Create a Timer that will elapse every `OneMicrosecond`
for (int i = 0; i <= 250; ++i) using (Media.Concepts.Classes.Stopwatch sw = new Media.Concepts.Classes.Stopwatch())
{
var started = System.DateTime.UtcNow;
System.Console.WriteLine("Started: " + started.ToString("MM/dd/yyyy hh:mm:ss.ffffff tt"));
//Define some amount of time
System.TimeSpan sleepTime = Media.Common.Extensions.TimeSpan.TimeSpanExtensions.OneMicrosecond;
System.Diagnostics.Stopwatch testSw = new System.Diagnostics.Stopwatch();
//Start
testSw.Start();
//Start
sw.Start();
while (testSw.Elapsed.Ticks < sleepTime.Ticks - (Common.Extensions.TimeSpan.TimeSpanExtensions.OneTick + Common.Extensions.TimeSpan.TimeSpanExtensions.OneTick).Ticks)
sw.Timer.m_Clock.NanoSleep(0); //System.Threading.Thread.SpinWait(0);
//Sleep the desired amount
//System.Threading.Thread.Sleep(sleepTime);
//Stop
testSw.Stop();
//Stop
sw.Stop();
var finished = System.DateTime.UtcNow;
var taken = finished - started;
var cc = System.Console.ForegroundColor;
System.Console.WriteLine("Finished: " + finished.ToString("MM/dd/yyyy hh:mm:ss.ffffff tt"));
System.Console.WriteLine("Sleep Time: " + sleepTime.ToString());
System.Console.WriteLine("Real Taken Total: " + taken.ToString());
if (taken > sleepTime)
{
System.Console.ForegroundColor = System.ConsoleColor.Red;
System.Console.WriteLine("Missed by: " + (taken - sleepTime));
}
else
{
System.Console.ForegroundColor = System.ConsoleColor.Green;
System.Console.WriteLine("Still have: " + (sleepTime - taken));
}
System.Console.ForegroundColor = cc;
System.Console.WriteLine("Real Taken msec Total: " + taken.TotalMilliseconds.ToString());
System.Console.WriteLine("Real Taken sec Total: " + taken.TotalSeconds.ToString());
System.Console.WriteLine("Real Taken μs Total: " + Media.Common.Extensions.TimeSpan.TimeSpanExtensions.TotalMicroseconds(taken).ToString());
System.Console.WriteLine("Managed Taken Total: " + sw.Elapsed.ToString());
System.Console.WriteLine("Diagnostic Taken Total: " + testSw.Elapsed.ToString());
System.Console.WriteLine("Diagnostic Elapsed Seconds Total: " + ((testSw.ElapsedTicks / (double)System.Diagnostics.Stopwatch.Frequency)));
//Write the rough amount of time taken in micro seconds
System.Console.WriteLine("Managed Time Estimated Taken: " + sw.ElapsedMicroseconds + "μs");
//Write the rough amount of time taken in micro seconds
System.Console.WriteLine("Diagnostic Time Estimated Taken: " + Media.Common.Extensions.TimeSpan.TimeSpanExtensions.TotalMicroseconds(testSw.Elapsed) + "μs");
System.Console.WriteLine("Managed Time Estimated Taken: " + sw.ElapsedMilliseconds);
System.Console.WriteLine("Diagnostic Time Estimated Taken: " + testSw.ElapsedMilliseconds);
System.Console.WriteLine("Managed Time Estimated Taken: " + sw.ElapsedSeconds);
System.Console.WriteLine("Diagnostic Time Estimated Taken: " + testSw.Elapsed.TotalSeconds);
if (sw.Elapsed < testSw.Elapsed)
{
System.Console.WriteLine("Faster than Diagnostic StopWatch");
l.Add(new System.Tuple<bool, System.TimeSpan, System.TimeSpan>(true, sw.Elapsed, testSw.Elapsed));
}
else if (sw.Elapsed > testSw.Elapsed)
{
System.Console.WriteLine("Slower than Diagnostic StopWatch");
l.Add(new System.Tuple<bool, System.TimeSpan, System.TimeSpan>(false, sw.Elapsed, testSw.Elapsed));
}
else
{
System.Console.WriteLine("Equal to Diagnostic StopWatch");
l.Add(new System.Tuple<bool, System.TimeSpan, System.TimeSpan>(true, sw.Elapsed, testSw.Elapsed));
}
}
int w = 0, f = 0;
var cc2 = System.Console.ForegroundColor;
foreach (var t in l)
{
if (t.Item1)
{
System.Console.ForegroundColor = System.ConsoleColor.Green;
++w; System.Console.WriteLine("Faster than Diagnostic StopWatch by: " + (t.Item3 - t.Item2));
}
else
{
System.Console.ForegroundColor = System.ConsoleColor.Red;
++f; System.Console.WriteLine("Slower than Diagnostic StopWatch by: " + (t.Item2 - t.Item3));
}
}
System.Console.ForegroundColor = System.ConsoleColor.Green;
System.Console.WriteLine("Wins = " + w);
System.Console.ForegroundColor = System.ConsoleColor.Red;
System.Console.WriteLine("Loss = " + f);
System.Console.ForegroundColor = cc2;
}
}
Related
we are developing tool that's support For execute Jmx files in our Application
currently i have issue with get the Execution Duration Time Based On Steping thread Group Values.
I got the Duration time by using Java
public class SteppingThreadGroup {
public static void main(String[] args) {
int TotalThreads = 500,
firstWait = 25,
thenStart_threads = 5,
nextAdd_threads = 25,
threadsEverySeconds = 30,
usingRamupSeconds = 5,
holdForSeconds =600,
finallyStopThreads =25,
down_threadsEverySeconds =25,
//extra varaibles
RemaingThreads =0, Duration=0;
float RampDown =0,rampupTime =0;;
if(thenStart_threads>0){
rampupTime =(firstWait+usingRamupSeconds+threadsEverySeconds);
RemaingThreads =TotalThreads-thenStart_threads;
System.out.format("Inital_Remaing_threads=%S,Inital_rampupTime=%f \n\n",RemaingThreads,rampupTime);
// System.out.format(" ",rampupTime);
}else{
rampupTime = (firstWait);
RemaingThreads=TotalThreads;
}
while(RemaingThreads >0){
if(RemaingThreads>nextAdd_threads)
rampupTime+= (Duration+usingRamupSeconds+threadsEverySeconds);
else if(RemaingThreads==nextAdd_threads)
rampupTime +=(Duration+usingRamupSeconds);
else
rampupTime +=(Duration+(RemaingThreads*(usingRamupSeconds/nextAdd_threads)));
RemaingThreads=RemaingThreads-nextAdd_threads;
}
RampDown=((TotalThreads/finallyStopThreads-1)*down_threadsEverySeconds);
Duration=(int) (holdForSeconds+rampupTime+RampDown);
System.out.format("RamupTime =%S && HoldForSeconds =%S \n RamDown =%S && Total_Duration=%S",
rampupTime,holdForSeconds,RampDown,Duration);
}
}
but is it possible to find The Duration In formula Base ?
I implement one Formula but its not working Fine as i expected.
The sample Stepping ThreadGroup Value Images
Image1:
Image2:
I'm running a Spark application (Spark 1.6.3 cluster), which does some calculations on 2 small data sets, and writes the result into an S3 Parquet file.
Here is my code:
public void doWork(JavaSparkContext sc, Date writeStartDate, Date writeEndDate, String[] extraArgs) throws Exception {
SQLContext sqlContext = new org.apache.spark.sql.SQLContext(sc);
S3Client s3Client = new S3Client(ConfigTestingUtils.getBasicAWSCredentials());
boolean clearOutputBeforeSaving = false;
if (extraArgs != null && extraArgs.length > 0) {
if (extraArgs[0].equals("clearOutput")) {
clearOutputBeforeSaving = true;
} else {
logger.warn("Unknown param " + extraArgs[0]);
}
}
Date currRunDate = new Date(writeStartDate.getTime());
while (currRunDate.getTime() < writeEndDate.getTime()) {
try {
SparkReader<FirstData> sparkReader = new SparkReader<>(sc);
JavaRDD<FirstData> data1 = sparkReader.readDataPoints(
inputDir,
currRunDate,
getMinOfEndDateAndNextDay(currRunDate, writeEndDate));
// Normalize to 1 hours & 0.25 degrees
JavaRDD<FirstData> distinctData1 = data1.distinct();
// Floor all (distinct) values to 6 hour windows
JavaRDD<FirstData> basicData1BySixHours = distinctData1.map(d1 -> new FirstData(
d1.getId(),
TimeUtils.floorTimePerSixHourWindow(d1.getTimeStamp()),
d1.getLatitude(),
d1.getLongitude()));
// Convert Data1 to Dataframes
DataFrame data1DF = sqlContext.createDataFrame(basicData1BySixHours, FirstData.class);
data1DF.registerTempTable("data1");
// Read Data2 DataFrame
String currDateString = TimeUtils.getSimpleDailyStringFromDate(currRunDate);
String inputS3Path = basedirInput + "/dt=" + currDateString;
DataFrame data2DF = sqlContext.read().parquet(inputS3Path);
data2DF.registerTempTable("data2");
// Join data1 and data2
DataFrame mergedDataDF = sqlContext.sql("SELECT D1.Id,D2.beaufort,COUNT(1) AS hours " +
"FROM data1 as D1,data2 as D2 " +
"WHERE D1.latitude=D2.latitude AND D1.longitude=D2.longitude AND D1.timeStamp=D2.dataTimestamp " +
"GROUP BY D1.Id,D1.timeStamp,D1.longitude,D1.latitude,D2.beaufort");
// Create histogram per ID
JavaPairRDD<String, Iterable<Row>> mergedDataRows = mergedDataDF.toJavaRDD().groupBy(md -> md.getAs("Id"));
JavaRDD<MergedHistogram> mergedHistogram = mergedDataRows.map(new MergedHistogramCreator());
logger.info("Number of data1 results: " + data1DF.select("lId").distinct().count());
logger.info("Number of coordinates with data: " + data1DF.select("longitude","latitude").distinct().count());
logger.info("Number of results with beaufort histograms: " + mergedDataDF.select("Id").distinct().count());
// Save to parquet
String outputS3Path = basedirOutput + "/dt=" + TimeUtils.getSimpleDailyStringFromDate(currRunDate);
if (clearOutputBeforeSaving) {
writeWithCleanup(outputS3Path, mergedHistogram, MergedHistogram.class, sqlContext, s3Client);
} else {
write(outputS3Path, mergedHistogram, MergedHistogram.class, sqlContext);
}
} finally {
TimeUtils.progressToNextDay(currRunDate);
}
}
}
public void write(String outputS3Path, JavaRDD<MergedHistogram> outputRDD, Class outputClass, SQLContext sqlContext) {
// Apply a schema to an RDD of JavaBeans and save it as Parquet.
DataFrame fullDataDF = sqlContext.createDataFrame(outputRDD, outputClass);
fullDataDF.write().parquet(outputS3Path);
}
public void writeWithCleanup(String outputS3Path, JavaRDD<MergedHistogram> outputRDD, Class outputClass,
SQLContext sqlContext, S3Client s3Client) {
String fileKey = S3Utils.getS3Key(outputS3Path);
String bucket = S3Utils.getS3Bucket(outputS3Path);
logger.info("Deleting existing dir: " + outputS3Path);
s3Client.deleteAll(bucket, fileKey);
write(outputS3Path, outputRDD, outputClass, sqlContext);
}
public Date getMinOfEndDateAndNextDay(Date startTime, Date proposedEndTime) {
long endOfDay = startTime.getTime() - startTime.getTime() % MILLIS_PER_DAY + MILLIS_PER_DAY ;
if (endOfDay < proposedEndTime.getTime()) {
return new Date(endOfDay);
}
return proposedEndTime;
}
The size of data1 is around 150,000 and data2 is around 500,000.
What my code does is basically does some data manipulation, merges the 2 data objects, does a bit more manipulation, prints some statistics and saves to parquet.
The spark has 25GB of memory per server, and the code runs fine.
Each iteration takes about 2-3 minutes.
The problem starts when I run it on a large set of dates.
After a while, I get an OutOfMemory:
java.lang.OutOfMemoryError: GC overhead limit exceeded
at scala.collection.immutable.List.$colon$colon$colon(List.scala:127)
at org.json4s.JsonDSL$JsonListAssoc.$tilde(JsonDSL.scala:98)
at org.apache.spark.util.JsonProtocol$.taskEndToJson(JsonProtocol.scala:139)
at org.apache.spark.util.JsonProtocol$.sparkEventToJson(JsonProtocol.scala:72)
at org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:144)
at org.apache.spark.scheduler.EventLoggingListener.onTaskEnd(EventLoggingListener.scala:164)
at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:42)
at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:55)
at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:38)
at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(AsynchronousListenerBus.scala:87)
at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(AsynchronousListenerBus.scala:72)
at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(AsynchronousListenerBus.scala:72)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:71)
at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1181)
at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:70)
Last time it ran, it crashed after 233 iterations.
The line it crashed on was this:
logger.info("Number of coordinates with data: " + data1DF.select("longitude","latitude").distinct().count());
Can anyone please tell me what can be the reason for the eventual crashes?
I'm not sure that everyone will find this solution viable, but upgrading the Spark cluster to 2.2.0 seems to have resolved the issue.
I have ran my application for several days now, and had no crashes yet.
This error occurs when GC takes up over 98% of the total execution time of process. You can monitor the GC time in your Spark Web UI by going to stages tab in http://master:4040.
Try increasing the driver/executor(whichever is generating this error) memory using spark.{driver/executor}.memory by --conf while submitting the spark application.
Another thing to try is to change the garbage collector that the java is using. Read this article for that: https://databricks.com/blog/2015/05/28/tuning-java-garbage-collection-for-spark-applications.html. It very clearly explains why GC overhead error occurs and which garbage collector is best for your application.
I have three fields
hour -- minutes -- and total
If hour or minutes change i want to calculate a total.
If the total is changed i want to calculate the corresponding minutes and hours.
Example:
1h 30minutes = 1.5 Total
2.25 Total = 2h 15minutes
I am trying to achive this with watch
watch: {
hour: {
handler: (new_hour, old_hour) => {
if(isNaN(new_hour)){
manual.hour = old_hour
}else{
manual.setTotal();
}
}
},
minutes: {
handler: (new_minutes, old_minutes) => {
if(isNaN(new_minutes)){
manual.minutes = old_minutes
}else{
if(Number(new_minutes) > 60){
manual.minutes = old_minutes
}else{
manual.setTotal();
}
}
}
},
total:{
handler: (new_total, old_total) => {
if(isNaN(new_total)) {
manual.total = old_total;
}else{
const hour = new_total.split(",")[0];
const minutes = new_total.split(",")[1];
manual.hour = hour;
manual.minutes = (minutes * 60).toFixed(0);
}
}
}
}
However this is resulting in a loop because on handler allways calls the other handler. How can this be done in a smarter way?
you could use onkeyup listener (and/or change)
new Vue({
el: "#app",
data: {
input_h: 0,
input_m: 0,
input_t: 0
},
methods: {
update_h (e) {
this.input_h = Number(e.target.value)
this.update_t(null)
},
update_m (e) {
this.input_m = Number(e.target.value)
this.update_t(null)
},
update_t (e) {
if (e === null) {
this.input_t = Math.round((this.input_h + this.input_m / 60) * 100)/100
} else {
this.input_t = Number(e.target.value)
this.input_h = Math.floor(this.input_t)
this.input_m = Math.round(this.input_t%1 * 60)
}
},
}
})
<script src="https://cdnjs.cloudflare.com/ajax/libs/vue/2.4.4/vue.js"></script>
<div id="app">
<input :value="input_h" #keyUp="update_h"/>Hours<br/>
<input :value="input_m" #keyUp="update_m"/>Minutes<br/>
<input :value="input_t" #keyUp="update_t"/>Total<br/>
</div>
Otherwise, if you want to prevent circular dependency, you need to set up a single source of data, and use computed getters and setters to update the other fields. You don't even have to use a visible field for that.
https://jsbin.com/tagupab/edit has a working example
I haven't checked this out, but perhaps you could exchange watchers for computed properties with setter, see ref Computed setter
Note the comment here circular dependency on observed properties #622 (yyx990803 commented on Dec 11, 2014)
The thing is even when there is a circular dependency, the eventual
value should be able to stabilize after 1 additional iteration (which
Vue will then stop because the new evaluated value is the same)
OR
I guess you could put
if (newValue === oldValue) {
return
}
at the top of each watcher. This is sort of what computed does anyway.
My problem is that my JavaFx application has gone to be very slow. Both in the start of the application and in some of the triggered events. It is a calendar application that uses a GridPane, that I am modifying. I have this method:
t.setOnMouseClicked(event->{
long starttid = System.currentTimeMillis();
System.out.println("start");
if (markedTimeEnd != null && markedTimeStart != null) {
colorMinutes(markedTimeStart, markedTimeEnd, Color.BLACK, bakrundWhite);
} else if (markedTimeStart != null) {
colorMinutes(markedTimeStart, markedTimeStart, Color.BLACK, bakrundWhite);
}
long tidNu = System.currentTimeMillis();
long tid = tidNu-starttid;
System.out.println("Print first time:\n"+tid);
int minutTid = gridPane.getRowIndex(t);
int timmeTimme = minutTid / 60;
int minutMinut = minutTid - (60 * timmeTimme);
markedTimeStart = new TidPunkt(timmeTimme, minutMinut);
markedTimeEnd = null;
tid = System.currentTimeMillis() -tidNu;
tidNu = System.currentTimeMillis();
System.out.println("Time for the middel calculations:\n"+tid);
if (markedTimeEnd != null && markedTimeStart != null) {
colorMinutes(markedTimeStart, markedTimeEnd, Color.GREEN,bakrundGren);
} else if (markedTimeStart != null) {
colorMinutes(markedTimeStart, markedTimeStart, Color.GREEN,bakrundGren);
}
event.consume();
repaintAll();
System.out.println("Time to end:\n"+(System.currentTimeMillis()-tidNu));
});
And the code for the colorMinutes:
private void colorMinutes(TidPunkt markedTimeStart, TidPunkt markedTimeEnd, Color colorText, Background colorOther) {
System.out.println("The call is comming");
int startBothTogether = markedTimeStart.getTimme() * 100 + markedTimeStart.getMinut();
int endBothTogether = markedTimeEnd.getTimme() * 100 + markedTimeEnd.getMinut();
System.out.println("Befor filter");
gridPane.getChildren().stream()//parallelStream()
.filter(x-> x.getId()!=null)
.filter(y-> y.getId().matches("\\d\\d:\\d\\d"))
.filter(pp->{
int hoursForPart = Integer.parseInt(((Node) pp).getId().split(":")[0]);
int miutesForPart = Integer.parseInt(((Node) pp).getId().split(":")[1]);
int bothTogether = hoursForPart * 100 + miutesForPart;
if (bothTogether >= startBothTogether && bothTogether <= endBothTogether)
return true;
else
return false;
})
.forEach(pp->{
Platform.runLater(() -> {
System.out.println("Changing collor ----");
if(pp instanceof Pane){
((Pane) pp).setBackground(colorOther)
}else{
((Text) pp).setFill(colorText);
}
});
});
}
But it takes very long before it is change on the screen and it freezes sometime after what I can se is the finish of these methods by the System.out.println. I have been trying to profile but can't figure it out(best I have come up with it that is seems to be a big call tree, of javaFx "stuff" when exiting the method). The System.out.println prints is:
start
Print first time:
0
Time for the middel calculations:
0
The call is comming
Befor filter
Time to end:
373
Changing collor ----
Changing collor ----
Changing collor ----
Changing collor ----
But from button press to all is being color correct it takes many seconds.
The full code can be found here
You're posting to many Runnables using Platform.runLater. There is no reason to use Platform.runLater here at all, since the onMouseClicked event handler is executed on the javafx application thread anyways.
Using
.forEach(pp->{
if(pp instanceof Pane){
((Pane) pp).setBackground(colorOther)
}else{
((Text) pp).setFill(colorText);
}
});
Should improve the performance drastically.
Furthermore you seem to add a huge amout of Nodes in your repaintAll method (more specifically the ritaGrundKalender method) without removing Nodes which increases the number of Runnables for every click. I suggest you change this behavior.
I am trying to use hystrix to monitor a certain network call. But all the metrics I try to monitor are always empty. What am I doing wrong?
I simulate a network call by implementing a (somewhat) RESTful interface that returns a pow calculation:
GetPowerCommand gpc = new GetPowerCommand(5, 82);
powerMetrics = gpc.getMetrics();
This is how I call the hystrix command and expect to get some metrics (at least Requests: not 0)
boolean run = true;
while (run) {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
run = false;
}
System.out.println("GetPowerCommand.run(): " + gpc.run());
System.out.println("GetPowerCommand.run(): " + gpc.run());
System.out.println("getStatsStringFromMetrics(powerMetrics): " + getStatsStringFromMetrics(powerMetrics));
}
But all I get is:
GetPowerCommand.run(): <p>I guess .. </p><p>2^5 = 32</p>
GetPowerCommand.run(): <p>I guess .. </p><p>2^5 = 32</p>
getStatsStringFromMetrics(powerMetrics): Requests: 0 Errors: 0 (0%) Mean: 0 50th: 0 75th: 0 90th: 0 99th: 0
GetPowerCommand.run(): <p>I guess .. </p><p>2^5 = 32</p>
GetPowerCommand.run(): <p>I guess .. </p><p>2^5 = 32</p>
getStatsStringFromMetrics(powerMetrics): Requests: 0 Errors: 0 (0%) Mean: 0 50th: 0 75th: 0 90th: 0 99th: 0
edit: my metrics retrieval method:
private static String getStatsStringFromMetrics(HystrixCommandMetrics metrics) {
StringBuilder m = new StringBuilder();
if (metrics != null) {
HealthCounts health = metrics.getHealthCounts();
m.append("Requests: ").append(health.getTotalRequests()).append(" ");
m.append("Errors: ").append(health.getErrorCount()).append(" (").append(health.getErrorPercentage())
.append("%) ");
m.append("Mean: ").append(metrics.getTotalTimeMean()).append(" ");
m.append("50th: ").append(metrics.getExecutionTimePercentile(50)).append(" ");
m.append("75th: ").append(metrics.getExecutionTimePercentile(75)).append(" ");
m.append("90th: ").append(metrics.getExecutionTimePercentile(90)).append(" ");
m.append("99th: ").append(metrics.getExecutionTimePercentile(99)).append(" ");
}
return m.toString();
}
You have already answered your question: use execute() instead of run(). Have a look also here