GrADS ScaLAPACK Jan 2001 run results

This page contains results gleaned from the Contract Monitor output for the ScaLAPACK runs Antoine did in January 2001. Original log files are at http://www.cs.utk.edu/~petitet/grads/results/. All data was collected from sensors in mpi process 0 -- other processes produced sensor output but that was not collected by the monitor. Five experiments were conducted. In the first four, the predicted runtime was less than the actual; in the fifth, the actual was less than the predicted.

In addition, some notes about experiment parameters and measured durations are included.

The predicted durations were based on a formula supplied by the ScaLAPACK team that takes into account processor speed, latency, and bandwidth. Here is the routine that makes the prediction. I *believe* only the latency and bandwidth from/to the 'current process' are considered, not those between other processes working on other parts of the problem. <<After talking to Antoine I think I was wrong on the previous belief... It uses the latency and bandwidth "around the ring" from the current process. So, for node zero it first looks at 0->1 then 1->2, then 2->3 and so on. At some point it's going in the other direction since it's a split ring, and would be looking at 0->31, 31->30, etc. I'll ask Antoine to make sure I've written this down correctly. For now, take it with a grain of salt! Ruth >>

The fact (time to factor) and update (time to update columns) components of the prediction depend only on GAMMA, processor speed. The bcast (split-ring time for processor 0 to get the message) depends on latency and bandwidth. In a brief review of the results at ISI, it was clear that some important factors are not being considered in the initial model, as the actual results vary significantly (both in magnitude and shape) from what was predicted. It may be the bandwidth (I suspect this and not latency as the number of messages is consistent throughout the iterations for a given experiment, but the bytes transmitted decreases and the "time spikes" get shorter which wouldn't happen if were related to the latency instead of bandwidth) between the other processes are coming into play and slowing things down. << In talking to Antoine it could also be related to processor load.... stay tuned - Ruth>>

The "good news" is that the actual measurements point out areas where it may be possible to tweak the algorithm slightly to perform better in the Grid environment where the typical assumption that compute and communicate overlap fully (or almost fully) is likely not true. It seems from the output in experiments 3, 4, and 5, that the watched process is likely waiting for input from others to continue. This is pointed out when you notice the spikes every P iterations, where P is the number of processors. And, by observing that the peak comes one iteration *before* the predicted long duration by the model. The model expects the transmission iteration to take longer but in fact it's the compute one that does - perhaps because the wait time is included in the compute iteration. [Antoine updated the model slightly on 2/5 and I have not yet checked to see if this 'off by one iteration' still occurs in the updated model -will try to do that soon. Ruth]

John and Antoine can explain this better -- their insights into the algorithm and it's behavior are much better than mine and I'm trying to relay what they pointed out in the discussions at ISI. In some discussions since that time on the conf calls and mailing list, it is also possible that we do not want to 'tweak' the algorithm as the whole point is to use the basic one provided. Apparently other versions of the LAPACK code do things differently -that is, do not compute entire panels before transmission, and might offer better performance in low-bandwidth environments). We may also want to try this version with a different processor grid layout -- the runs here were all done with a 1xP layout, where P is the number of processes (= processors in this case). Perhaps a non-vector processor grid would hide some of the bandwidth bottlenecks better.

See the demo list mail archives for the mail about other options re: algorithms. Update: I've put some of the email to the demo list from Antoine that is related to this dicussion here.

Answers to questions on where speed and load came from (as reported below for the experiments) and how to interpret them:

Speed:
The load is the value returned by NWS, and the speed was handcoded in the filtering file. The value chosen there is the flops/percycle rate. Since we were using Intel Pentium boxes, this correspond to the Mhz rate. That number was scaled down in the performance modeler, contract monitor and everywhere it was
needed. - Antoine << This means speed as reported is alread adjusted for the current load and is not a 'raw' value - Ruth >>

Load:
I can answer the load question. The NWS reports the fraction of time-shared CPU cycles that are available. If the machine is completely unloaded, the fraction is 1.0. If there is an extra process, the fraction is 0.5. Multiprocessors present a special problem, for those of you who care, but the demo doesn't currently use them so it isn't currently an issue. also, the demo does not currently use the forecasting facilities so the readings are for the last measurement taken. The NWS takes CPU measurements every 10 seconds, by default, so those measurements are for, at most, the ten seconds before the program was launched (more or less). -- Cheers, Rich

Experiment 1:

Experiment 2:

Experiment 3:

UTK confirmed that some systems were involved in network benchmarks at this time so the variance, even on the same 'cluster', is not unexpected.

In the following zoom view, the peaks occur at iterations 5, 12, 19, etc. and the secondary "notches" at 7, 14, 21, etc.

Experiment 4:

In the following zoomed in view, the peak iterations occur at 7, 19, 31, etc with secondary 'peaklets' at 10, 22, 34, etc.

Experiment 5: Jump to plots for Experiment 5

-1.00	0.27	0.36	0.31	0.30	0.29	0.32	0.30	0.36	0.31	0.35	0.27	0.30	0.30	0.32	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	74.86	74.86	74.86	74.86	125.46	125.46	125.46	125.46	125.46
0.27	-1.00	0.28	0.26	0.32	0.37	0.26	0.31	0.28	0.30	0.26	0.26	0.31	0.29	0.26	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	74.86	74.86	74.86	74.86	125.46	125.46	125.46	125.46	125.46
0.36	0.28	-1.00	0.27	0.32	0.31	0.32	0.30	0.30	0.36	0.26	0.26	0.28	0.26	0.27	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	74.86	74.86	74.86	74.86	125.46	125.46	125.46	125.46	125.46
0.31	0.26	0.27	-1.00	0.34	0.33	0.32	0.28	0.31	0.35	0.26	0.32	0.26	0.26	0.26	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	74.86	74.86	74.86	74.86	125.46	125.46	125.46	125.46	125.46
0.30	0.32	0.32	0.34	-1.00	0.40	0.28	0.29	0.31	0.25	0.28	0.27	0.26	0.37	0.72	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	74.86	74.86	74.86	74.86	125.46	125.46	125.46	125.46	125.46
0.29	0.37	0.31	0.33	0.40	-1.00	0.30	0.31	0.30	0.31	0.29	0.30	0.28	0.32	0.25	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	74.86	74.86	74.86	74.86	125.46	125.46	125.46	125.46	125.46
0.32	0.26	0.32	0.32	0.28	0.30	-1.00	0.39	0.25	0.28	0.30	0.28	0.34	0.41	0.31	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	74.86	74.86	74.86	74.86	125.46	125.46	125.46	125.46	125.46
0.30	0.31	0.30	0.28	0.29	0.31	0.39	-1.00	0.32	7.55	0.28	65.45	0.27	0.29	0.28	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	74.86	74.86	74.86	74.86	125.46	125.46	125.46	125.46	125.46
0.36	0.28	0.30	0.31	0.31	0.30	0.25	0.32	-1.00	0.31	0.29	0.31	0.26	0.28	0.29	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	74.86	74.86	74.86	74.86	125.46	125.46	125.46	125.46	125.46
0.31	0.30	0.36	0.35	0.25	0.31	0.28	7.55	0.31	-1.00	0.32	88.04	0.30	0.37	0.30	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	74.86	74.86	74.86	74.86	125.46	125.46	125.46	125.46	125.46
0.35	0.26	0.26	0.26	0.28	0.29	0.30	0.28	0.29	0.32	-1.00	0.27	0.26	0.43	0.29	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	74.86	74.86	74.86	74.86	125.46	125.46	125.46	125.46	125.46
0.27	0.26	0.26	0.32	0.27	0.30	0.28	65.45	0.31	88.04	0.27	-1.00	0.28	0.32	0.38	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	74.86	74.86	74.86	74.86	125.46	125.46	125.46	125.46	125.46
0.30	0.31	0.28	0.26	0.26	0.28	0.34	0.27	0.26	0.30	0.26	0.28	-1.00	0.30	0.26	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	74.86	74.86	74.86	74.86	125.46	125.46	125.46	125.46	125.46
0.30	0.29	0.26	0.26	0.37	0.32	0.41	0.29	0.28	0.37	0.43	0.32	0.30	-1.00	0.38	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	74.86	74.86	74.86	74.86	125.46	125.46	125.46	125.46	125.46
0.32	0.26	0.27	0.26	0.72	0.25	0.31	0.28	0.29	0.30	0.29	0.38	0.26	0.38	-1.00	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	74.86	74.86	74.86	74.86	125.46	125.46	125.46	125.46	125.46
66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	-1.00	0.34	0.31	0.36	0.32	0.36	27.28	0.29	83.61	83.61	83.61	83.61	123.38	123.38	123.38	123.38	123.38
66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	0.34	-1.00	1.35	20.65	9.29	0.28	81.47	0.45	83.61	83.61	83.61	83.61	123.38	123.38	123.38	123.38	123.38
66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	0.31	1.35	-1.00	9.96	0.32	0.31	0.33	0.35	83.61	83.61	83.61	83.61	123.38	123.38	123.38	123.38	123.38
66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	0.36	20.65	9.96	-1.00	19.10	0.31	0.31	0.30	83.61	83.61	83.61	83.61	123.38	123.38	123.38	123.38	123.38
66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	0.32	9.29	0.32	19.10	-1.00	111.68	0.40	0.44	83.61	83.61	83.61	83.61	123.38	123.38	123.38	123.38	123.38
66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	0.36	0.28	0.31	0.31	111.68	-1.00	0.31	20.95	83.61	83.61	83.61	83.61	123.38	123.38	123.38	123.38	123.38
66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	27.28	81.47	0.33	0.31	0.40	0.31	-1.00	0.31	83.61	83.61	83.61	83.61	123.38	123.38	123.38	123.38	123.38
66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	66.22	0.29	0.45	0.35	0.30	0.44	20.95	0.31	-1.00	83.61	83.61	83.61	83.61	123.38	123.38	123.38	123.38	123.38
74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	83.61	83.61	83.61	83.61	83.61	83.61	83.61	83.61	-1.00	0.31	0.32	0.34	71.06	71.06	71.06	71.06	71.06
74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	83.61	83.61	83.61	83.61	83.61	83.61	83.61	83.61	0.31	-1.00	0.32	0.31	71.06	71.06	71.06	71.06	71.06
74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	83.61	83.61	83.61	83.61	83.61	83.61	83.61	83.61	0.32	0.32	-1.00	0.32	71.06	71.06	71.06	71.06	71.06
74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	74.86	83.61	83.61	83.61	83.61	83.61	83.61	83.61	83.61	0.34	0.31	0.32	-1.00	71.06	71.06	71.06	71.06	71.06
125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	123.38	123.38	123.38	123.38	123.38	123.38	123.38	123.38	71.06	71.06	71.06	71.06	-1.00	0.28	0.29	0.26	0.44
125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	123.38	123.38	123.38	123.38	123.38	123.38	123.38	123.38	71.06	71.06	71.06	71.06	0.28	-1.00	0.28	0.27	0.43
125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	123.38	123.38	123.38	123.38	123.38	123.38	123.38	123.38	71.06	71.06	71.06	71.06	0.29	0.28	-1.00	0.27	0.43
125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	123.38	123.38	123.38	123.38	123.38	123.38	123.38	123.38	71.06	71.06	71.06	71.06	0.26	0.27	0.27	-1.00	0.43
125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	125.46	123.38	123.38	123.38	123.38	123.38	123.38	123.38	123.38	71.06	71.06	71.06	71.06	0.44	0.43	0.43	0.43	-1.00

-1.00	214.78	204.96	208.38	187.51	197.03	181.29	83.00	153.62	211.66	189.41	66.10	103.76	207.97	79.91	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	1.06	1.06	1.06	1.06	0.64	0.64	0.64	0.64	0.64
214.78	-1.00	200.03	175.52	198.90	205.93	214.52	189.00	88.09	170.50	212.69	64.88	47.32	63.51	188.12	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	1.06	1.06	1.06	1.06	0.64	0.64	0.64	0.64	0.64
204.96	200.03	-1.00	189.07	119.02	101.12	173.09	208.46	197.03	86.25	60.71	75.46	63.32	212.09	87.09	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	1.06	1.06	1.06	1.06	0.64	0.64	0.64	0.64	0.64
208.38	175.52	189.07	-1.00	187.31	163.33	95.76	103.19	99.64	106.63	90.39	70.54	104.86	104.71	96.63	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	1.06	1.06	1.06	1.06	0.64	0.64	0.64	0.64	0.64
187.51	198.90	119.02	187.31	-1.00	176.53	100.81	53.73	175.23	72.41	85.67	218.09	209.13	157.82	96.34	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	1.06	1.06	1.06	1.06	0.64	0.64	0.64	0.64	0.64
197.03	205.93	101.12	163.33	176.53	-1.00	214.35	216.38	100.69	59.38	89.94	50.51	68.62	209.97	90.25	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	1.06	1.06	1.06	1.06	0.64	0.64	0.64	0.64	0.64
181.29	214.52	173.09	95.76	100.81	214.35	-1.00	203.69	205.44	215.58	71.40	186.25	209.63	64.81	44.48	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	1.06	1.06	1.06	1.06	0.64	0.64	0.64	0.64	0.64
83.00	189.00	208.46	103.19	53.73	216.38	203.69	-1.00	96.41	101.78	210.05	66.08	196.14	62.38	56.21	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	1.06	1.06	1.06	1.06	0.64	0.64	0.64	0.64	0.64
153.62	88.09	197.03	99.64	175.23	100.69	205.44	96.41	-1.00	198.97	72.77	204.64	65.68	92.91	96.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	1.06	1.06	1.06	1.06	0.64	0.64	0.64	0.64	0.64
211.66	170.50	86.25	106.63	72.41	59.38	215.58	101.78	198.97	-1.00	209.80	214.35	198.74	82.04	86.43	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	1.06	1.06	1.06	1.06	0.64	0.64	0.64	0.64	0.64
189.41	212.69	60.71	90.39	85.67	89.94	71.40	210.05	72.77	209.80	-1.00	130.39	44.71	86.98	88.44	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	1.06	1.06	1.06	1.06	0.64	0.64	0.64	0.64	0.64
66.10	64.88	75.46	70.54	218.09	50.51	186.25	66.08	204.64	214.35	130.39	-1.00	205.68	160.19	67.23	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	1.06	1.06	1.06	1.06	0.64	0.64	0.64	0.64	0.64
103.76	47.32	63.32	104.86	209.13	68.62	209.63	196.14	65.68	198.74	44.71	205.68	-1.00	81.47	67.28	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	1.06	1.06	1.06	1.06	0.64	0.64	0.64	0.64	0.64
207.97	63.51	212.09	104.71	157.82	209.97	64.81	62.38	92.91	82.04	86.98	160.19	81.47	-1.00	92.34	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	1.06	1.06	1.06	1.06	0.64	0.64	0.64	0.64	0.64
79.91	188.12	87.09	96.63	96.34	90.25	44.48	56.21	96.38	86.43	88.44	67.23	67.28	92.34	-1.00	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	1.06	1.06	1.06	1.06	0.64	0.64	0.64	0.64	0.64
18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	-1.00	81.83	80.14	68.75	43.98	31.13	39.91	12.82	0.94	0.94	0.94	0.94	0.64	0.64	0.64	0.64	0.64
18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	81.83	-1.00	81.83	58.38	55.45	81.41	35.61	56.38	0.94	0.94	0.94	0.94	0.64	0.64	0.64	0.64	0.64
18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	80.14	81.83	-1.00	81.37	56.30	81.84	81.92	65.90	0.94	0.94	0.94	0.94	0.64	0.64	0.64	0.64	0.64
18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	68.75	58.38	81.37	-1.00	81.03	41.78	44.41	59.71	0.94	0.94	0.94	0.94	0.64	0.64	0.64	0.64	0.64
18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	43.98	55.45	56.30	81.03	-1.00	80.46	69.03	13.93	0.94	0.94	0.94	0.94	0.64	0.64	0.64	0.64	0.64
18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	31.13	81.41	81.84	41.78	80.46	-1.00	59.32	45.10	0.94	0.94	0.94	0.94	0.64	0.64	0.64	0.64	0.64
18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	39.91	35.61	81.92	44.41	69.03	59.32	-1.00	61.76	0.94	0.94	0.94	0.94	0.64	0.64	0.64	0.64	0.64
18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	18.38	12.82	56.38	65.90	59.71	13.93	45.10	61.76	-1.00	0.94	0.94	0.94	0.94	0.64	0.64	0.64	0.64	0.64
1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	0.94	0.94	0.94	0.94	0.94	0.94	0.94	0.94	-1.00	190.72	185.79	189.82	1.13	1.13	1.13	1.13	1.13
1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	0.94	0.94	0.94	0.94	0.94	0.94	0.94	0.94	190.72	-1.00	194.11	188.66	1.13	1.13	1.13	1.13	1.13
1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	0.94	0.94	0.94	0.94	0.94	0.94	0.94	0.94	185.79	194.11	-1.00	191.77	1.13	1.13	1.13	1.13	1.13
1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	1.06	0.94	0.94	0.94	0.94	0.94	0.94	0.94	0.94	189.82	188.66	191.77	-1.00	1.13	1.13	1.13	1.13	1.13
0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	1.13	1.13	1.13	1.13	-1.00	76.99	60.27	39.96	8.06
0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	1.13	1.13	1.13	1.13	76.99	-1.00	62.01	82.20	7.78
0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	1.13	1.13	1.13	1.13	60.27	62.01	-1.00	82.10	7.56
0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	1.13	1.13	1.13	1.13	39.96	82.20	82.10	-1.00	8.20
0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	0.64	1.13	1.13	1.13	1.13	8.06	7.78	7.56	8.20	-1.00

The following plot shows the amount of time the various components contribute to the expected duration for each iteration.

In the following zoomed in view, the highest peaks occur at iterations 17, 49, etc. The secondary peaks before the highest in each processor cycle occur at 12, 44, etc.

The following zoomed in view shows the predicted duration each component of the solution would take for each iteration.

This material is based upon work supported by the National Science Foundation under Grant No. 9975020.

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

machine	cypher12	cypher15	cypher03	cypher09	cypher05	cypher04	cypher01
speeed	102.3	221.9	110.6	158.1	133.6	169.6	124.8
load	.58	1.25	.62	.89	.75	.96	.70

-1.00	0.27	0.27	0.29	0.31	0.28	7.67
0.27	-1.00	0.29	0.45	0.39	0.29	0.26
0.27	0.29	-1.00	0.33	0.29	0.31	0.29
0.29	0.45	0.33	-1.00	0.32	0.32	0.36
0.31	0.39	0.29	0.32	-1.00	0.27	0.35
0.28	0.29	0.31	0.32	0.27	-1.00	0.31
7.67	0.26	0.29	0.36	0.35	0.31	-1.00

-1.00	221.97	214.78	219.28	167.50	190.30	215.58
221.97	-1.00	201.26	192.40	197.77	216.20	57.79
214.78	201.26	-1.00	220.75	210.90	188.39	181.54
219.28	192.40	220.75	-1.00	214.43	118.72	195.27
167.50	197.77	210.90	214.43	-1.00	203.84	164.82
190.30	216.20	188.39	118.72	203.84	-1.00	185.98
215.58	57.79	181.54	195.27	164.82	185.98	-1.00

-1.00	0.29	0.31	0.35	0.33	0.30	0.28	0.32	0.25	0.32	0.36	0.28
0.29	-1.00	0.28	0.31	0.31	0.58	0.44	0.29	0.32	0.29	0.29	0.28
0.31	0.28	-1.00	0.31	0.31	0.39	0.27	0.29	0.28	0.30	0.26	0.33
0.35	0.31	0.31	-1.00	0.35	0.29	0.27	0.30	0.27	0.31	0.28	0.40
0.33	0.31	0.31	0.35	-1.00	0.33	0.29	0.40	0.31	0.31	0.32	0.30
0.30	0.58	0.39	0.29	0.33	-1.00	0.35	0.26	0.28	0.31	0.28	0.29
0.28	0.44	0.27	0.27	0.29	0.35	-1.00	0.31	0.28	0.26	0.30	0.27
0.32	0.29	0.29	0.30	0.40	0.26	0.31	-1.00	0.28	0.32	0.31	0.30
0.25	0.32	0.28	0.27	0.31	0.28	0.28	0.28	-1.00	0.34	0.29	0.26
0.32	0.29	0.30	0.31	0.31	0.31	0.26	0.32	0.34	-1.00	0.30	96.71
0.36	0.29	0.26	0.28	0.32	0.28	0.30	0.31	0.29	0.30	-1.00	0.38
0.28	0.28	0.33	0.40	0.30	0.29	0.27	0.30	0.26	96.71	0.38	-1.00

-1.00	221.69	219.46	165.86	206.33	95.34	184.61	219.28	77.44	59.83	118.03	205.60
221.69	-1.00	214.78	209.46	132.53	215.31	104.50	214.61	103.29	45.51	206.58	94.62
219.46	214.78	-1.00	177.42	173.72	211.15	88.06	108.30	218.18	211.24	207.80	45.93
165.86	209.46	177.42	-1.00	137.54	81.21	204.24	98.88	77.32	222.25	51.49	129.87
206.33	132.53	173.72	137.54	-1.00	192.89	184.28	136.25	179.98	163.28	80.04	206.17
95.34	215.31	211.15	81.21	192.89	-1.00	217.10	195.12	185.20	214.08	83.47	71.19
184.61	104.50	88.06	204.24	184.28	217.10	-1.00	136.78	186.91	76.17	201.18	97.71
219.28	214.61	108.30	98.88	136.25	195.12	136.78	-1.00	185.85	184.15	95.26	105.92
77.44	103.29	218.18	77.32	179.98	185.20	186.91	185.85	-1.00	186.05	188.93	175.70
59.83	45.51	211.24	222.25	163.28	214.08	76.17	184.15	186.05	-1.00	194.83	189.27
118.03	206.58	207.80	51.49	80.04	83.47	201.18	95.26	188.93	194.83	-1.00	106.09
205.60	94.62	45.93	129.87	206.17	71.19	97.71	105.92	175.70	189.27	106.09	-1.00

GrADS: Experimental results from Jan 2001 ScaLAPACK runs

Experiment 1:

Experiment 2:

Experiment 3:

Experiment 4:

Experiment 5: Jump to plots for Experiment 5