| Main page | Site map
High Performance Computing
In Computational Fluid Dynamics and Aeroacoustics

Personal page of A. Gorobets


Parallel computing issues

How to prepare a DNS

Once an important CFD problem that will contribute to the whole humanity and save the civilization is chosen, its time to prepare a DNS. In fact to perform the primary DNS many preliminary auxiliary computations may be required. A typical procedure we follow is listed below.
  • Choice of the input conditions.
    Inflow must be close to reality and at the same time simple and easy to reproduce. Comparison with other authors and experimental data would be useful.
  • Geometry definition and verification.
    For example input boundaries must be far enough from obstacles, output as well. But at the same time the domain must be as small as possible time to save the mesh size.
  • Optimization of the mesh concentration factors.
    Preliminary DNS on smaller mesh is performed to obtain approximate flow characteristics which allow to optimize mesh concentrations.
  • Estimation of the time integration period.
    Preliminary DNS on smaller mesh also allows to evaluate time integration period needed to obtain converged average fields and statistics of the flow.
  • Estimation of the mesh size.
    Series of preliminary DNS on smaller mesh at lower Re / Ra numbers is performed for mesh refinement to evaluate required resolution. Once resolution at lower Re is known it is scaled by analytical formulas to estimate the mesh size required for the primary DNS. Together with the estimation of time integration period it gives an idea how much CPU time it would take.
After this stages the primary DNS is almost ready to start. The only thing missing is to get that frightening amount of the CPU hours required to perform it.

How to perform a DNS

Once required CPU hours and disk quota were granted we can start. But what to do with the immense amount of output data? The data during the simulation must be downloaded to a data storage that have enough terabytes free. Special attention must be paid on minimizing output size. The strategy of output we use is following.
The output data can be divided into categories:
  1. Restart points
  2. Averaged flow data
  3. Instantaneous flow fields - snapshots
  4. Time evolution (e.g. in monitoring points)

File output issues

Restart point
It is needed to restart the DNS after it was stopped when the fraction of the CPU time is over or due to a crash (which also may happen from time to time). Restart point contain arrays of all physical variables to reproduce exactly the instantaneous flow at that moment. Also it contains averaged flow data which is required to continue averaging. We store the 2 latest restart points (in case the task stops when writing data).

Averaging is fragmented into certain periods leading to a sequence of files with averaged data. It is needed to ensure first integration period starts after statistically stationary flow mode is reached and second convergence is achieved.

A set of reduced monitoring snapshots is being written after a certain interval. It allows to see the flow behavior. Sequence of snapshots can be saved for animation purposes. Snapshot can show all domain in reduced resolution, can show 2D slices in full resolution, specific fragments of the domain in full resolution.

Time evolution
History of values in the monitoring points and other quantities e.g. value of an integral of something over something are recorded for each time step and stored in a buffer in RAM memory. After a certain amount of time steps the values from buffer are being written to a file. This saves time on latency of the file system.

So the size of the disk quota must be large enough to store at least 2 restart points and one averaged flow record (the size of monitoring snapshots and history files can be assumed negligible). Everything else should be transferred to a data storage during the simulation.

What to do with the DNS results

Parallel processing tools required to deal with the immense amount of data DNS produces. We use local clusters connected to the data storage to perform postprocessing. Finally when this stage is done we obtain
  1. Averaged flow fields
  2. Turbulent statistics - turbulent kinetic energy, Reynolds stress tensor etc.
  3. Time evolution in monitoring points - spectral characteristics, etc.
  4. Visualization - instantaneous snapshots and animation of the flow

Main page  |   Contacts
(C) Web-design: Cherepock