This application considers a 3-D numerical simulation of the Navier-Stokes equations in a geometry where the flow is periodic in at least one direction. Examples include flow past a cylinder, flow in a channel, and flow over a backward-facing step. A combination of spectral elements and Fourier modes are used to investigate the dynamics and transport properties of turbulent flow. Simply put, the goal is to integrate the Navier-Stokes equations from time t0 to tn to observe the changes in the state of a fluid. The input to the application defines the problem geometry and the initial conditions of the fluid. An initial velocity field is given by the input data, and the solution is integrated forward in time by numerically solving the equations that describe advection and diffusion of momentum in the fluid. A large number of modes are required to attain high resolution. The number of modes scales as the 9/4th power of the Reynolds number, which is a dimensionless measure of flow turbulence. Slices of the periodic domain are the units of work apportioned to the nodes. 

Input data is accessed by all processors. The mesh consists of quadrilateral elements with a uniform length in the periodic direction of the domain. Initially the number of elements and the x and y coordinates of their vertices are read in, followed by information regarding elements that have curved sides. The mesh is generated, and boundary information is accessed. Matrices for the discrete pressure and velocity equations are built using an additional input file. 

After the vectors are transformed to Fourier space, the Navier-Stokes calculations are applied to integrate the solution forward in time, and the final results are transformed back to physical space. The output file is written by all processors.

A set of images constructed from a simulation of the unsteady turbulent wake of a circular Cylinder are provided by Ron Henderson at Caltech.

Platforms

While PRISM has been used on a number of platforms, the version examined in this research has been specifically tuned for the Intel Paragon. Both the application source code and the required libraries are targeted to the specific platform to achieve the maximum possible performance.

The application consists of approximately 10000 lines of C code, specifically targeted to the Intel Paragon. Execution begins with a preprocessing segment. Three input files are used to initialize the system. The first file contains parameters such as the Reynolds number, the number of mesh elements, the coordinates for each element, the number of curved edges, and the boundary conditions for each edge. The initial conditions are then either read in from a separate "restart" file, or default initial conditions are engaged. The third data input file is the connectivity file. This information is used to setup the internal boundary system of the spectral element mesh. In the nonPFS version, all nodes open and read initialization files using serial Unix I/O calls. In the PFS version, the parameter and connectivity files are accessed through the M_GLOBAL file mode. The header of the restart file is accessed though the M_GLOBAL file mode and its body through the M_RECORD file mode. In the PFS2.0 version, the restart file is accessed using the M_ASYNC file mode. Additionally, the connectivity file is accessed using binary format. 

Once preprocessing has been completed, the integration function for the Navier-Stokes simulation integrates the status of the fluid from its current to its new state. History points are gathered to node 0 and written to disk during the integration. A measurement file is written by node 0 that includes lift and drag forces, viscous forces, and the kinetic energy of each mode. Flow statistics for velocity, vorticity, and turbulent stresses are gathered to node 0 at certain time steps and written to three separate files. Each of these files contains the mean, variance, skewness, and flatness associated with each field. In all three code versions, these files are written by node 0 using the default M_UNIX iomode.

During the final, or postprocessing, phase, the field file is written. In the nonPFS version, node 0 coordinates all writes using the default M_UNIX iomode. In the PFS and PFS2.0 versions, the field file is written using the M_ASYNC mode to allow unrestricted access to the file by all nodes.

PRISM uses libraries specifically designed for the application and tuned to the particular platform. Assembler code is used, where necessary, to achieve peak performance.

This application was provided by Ron Henderson at Caltech. The research is supported, in part, by the National Science Foundation under Grant No. CDA-9318145. Trace data was obtained using the Pablo trace capture library.

This research is supported in part by the  National Science Foundation.

back