wtangiit / qsim Goto Github PK

change bqsim and cqsim to be derived from qsim_base class for the convenience of further development.

att

in coscheduling simulation, two workload from different machine should be able to match in the same time span.

for further statistics.
one pair per line

using "hold" scheme on both machine may cause dead lock. two way to prevent:

Spatial way (sys util restrict)
Allow only x% of total nodes to be holden. job that may cause over holding will become yielding job.

Temporal way (temp yielding)
To prevent deadlock, add a hold threshold, if a job hold more than THRESH hours, give up the resources for a scheduling iteration to allow other job to use the resource. (if no other job take the resource, it will hold again). This way will also benefit for the scenario that holding a sub-partition blocks a large job for a long time even without any backfilling.

show on the screen general workload information (such total job numbers) and feature configuration (such as schemes of walltime prediction, walltime-aware job allocation, or coscheduling)

completely prevent deadlock and increase system utilization rate for "hold" scheme in coscheduling

adjust interval of job arrivals to make heavier or lighter workload

For example, while intrepid can use 'yield', eureka can use 'hold'.

job runtime should be dynamically adjusted according to I/O status

1. distinguish holden partitions via color
2. show number of holden bg midplanes and cluster nodes

1. add cluster simulator (cqsim)
2. enable event-manager drive scheduler iteration for both bqsim (blue gene qsim) and cqsim

display percentage of mate job pairs that start at same time

if yield time exceeds the threshold, the job can hold until it can run

input: utilization rate, time span
action: adjust job arrival intervals (Or job runtimes)
output: workload that has specified average utilization rate within the time span.

in batch execution mode, the print_screen will not be executed.

1. two queue-manager can exchange paired job information
2. implement co-scheduling algorithm 1, favoring system utilization
3. implement co-scheduling algorithm 2, favoring job response time

1. add command line arguments
2. add walltime-aware job allocation algorithm

currently two schemes: hold and yield. implemented as algorithm 2 and 1 in issue #2
a scheduled job holds or yields resource if mate cannot run.

merge two screens for bqsim and cqsim into a single one.

Currently making mate jobs is based on job trace. Iit is not accurate when workload is tuned.

given a proportion, make number of job pairs that conform to the proportion among the total jobs
(used for coscheduling simulation)

make some change in bgsched.py for coscheduling needs.

a line of the trace looks like:
jobid=xxx;qtime=xxx;nodes=xxx;walltime=xxx;runtime=xxx

wasted node hours for holding
wasted waiting time for yielding

jobs with qtime between the start and end point will be "submitted" in the simulation. other jobs in the job trace are truncated.

amount of I/O data (MB)
File type: N/1, N/N, N/M
historical I/O time (ratio of I/O time to total runtime)

att

code change in bqsim init_mate_qtime_pair()

check running jobs periodically and log I/O usage info based on running jobs I/O request

-P xyz (x,y,z=0|1), x: queuing job, y: backfilling job, z:running job

change components/evsim.py to components/qsim.py