Parallel computing issues
How to prepare a DNS
Once an important CFD problem that will contribute to the whole humanity and save the civilization is chosen, its time to prepare a DNS.
In fact to perform the primary DNS many preliminary auxiliary computations may be required. A typical procedure we follow is listed below.
After this stages the primary DNS is almost ready to start. The only thing missing is to get that frightening amount of the CPU hours required to perform it.
- Choice of the input conditions.
Inflow must be close to reality and at the same time simple and easy to reproduce.
Comparison with other authors and experimental data would be useful.
- Geometry definition and verification.
For example input boundaries must be far enough from obstacles, output as well. But at the same time the domain must be as small as possible time to save the mesh size.
- Optimization of the mesh concentration factors.
Preliminary DNS on smaller mesh is performed to obtain approximate flow characteristics which allow to optimize mesh concentrations.
- Estimation of the time integration period.
Preliminary DNS on smaller mesh also allows to evaluate time integration period needed to obtain converged average fields and statistics of the flow.
- Estimation of the mesh size.
Series of preliminary DNS on smaller mesh at lower Re / Ra numbers is performed for mesh refinement to evaluate required resolution.
Once resolution at lower Re is known it is scaled by analytical formulas to estimate the mesh size required for the primary DNS.
Together with the estimation of time integration period it gives an idea how much CPU time it would take.
How to perform a DNS
Once required CPU hours and disk quota were granted we can start. But what to do with the immense amount of output data?
The data during the simulation must be downloaded to a data storage that have enough terabytes free. Special attention must be paid on minimizing
output size. The strategy of output we use is following. |
The output data can be divided into categories:
- Restart points
- Averaged flow data
- Instantaneous flow fields - snapshots
- Time evolution (e.g. in monitoring points)
File output issues
It is needed to restart the DNS after it was stopped when the fraction of the CPU time is over or due to a crash (which also may happen from time to time). Restart point contain arrays of all physical variables to reproduce exactly the instantaneous flow at that moment. Also it contains
averaged flow data which is required to continue averaging. We store the 2 latest restart points (in case the task stops when writing data).
Averaging is fragmented into certain periods leading to a sequence of files with averaged data. It is needed to ensure first integration period starts after statistically stationary flow mode is reached and second convergence is achieved.
A set of reduced monitoring snapshots is being written after a certain interval. It allows to see the flow behavior. Sequence of snapshots can be saved for animation purposes. Snapshot can show all domain in reduced resolution, can show 2D slices in full resolution, specific fragments of the domain in full resolution.
History of values in the monitoring points and other quantities e.g. value of an integral of something over something are recorded for each time step and stored in a buffer in RAM memory. After a certain amount of time steps the values from buffer are being written to a file. This saves time on latency of the file system.
So the size of the disk quota must be large enough to store at least 2 restart points and one averaged flow record (the size of monitoring snapshots and history files can be assumed negligible). Everything else should be transferred to a data storage during the simulation.
What to do with the DNS results
Parallel processing tools required to deal with the immense amount of data DNS produces. We use local clusters connected to the data storage to perform postprocessing. Finally when this stage is done we obtain
- Averaged flow fields
- Turbulent statistics - turbulent kinetic energy, Reynolds stress tensor etc.
- Time evolution in monitoring points - spectral characteristics, etc.
- Visualization - instantaneous snapshots and animation of the flow