Paul Ricker Home Research Teaching Papers Interests Links
About Features Performance Test results Distribution Publications

Old research pages

I'm keeping these old research pages here for the benefit of anyone who might arrive here from an external page. They are no longer updated. Please visit my new web site at http://sipapu.astro.illinois.edu/~ricker/. Eventually some content from these pages will migrate to the new site.

COSMOS

Cluster collision image
Specific entropy and velocity in an off-center collision between two clusters of galaxies (mass ratio 1:2, gas fraction 15%), as computed with COSMOS 1.0.

COSMOS is a Fortran 90 code for parallel computers that solves the evolution equations for gravitationally coupled gas and collisionless matter in one, two, or three dimensions. It is oriented toward the solution of astrophysical and cosmological hydrodynamics problems. The hydrodynamical code uses the piecewise-parabolic method (PPM) to solve the Eulerian gas equations on a static grid. The collisionless (N-body) code integrates particle equations of motion using a variable-timestep leapfrog method. Both matter components evolve in their mutual gravitational field, which is determined at each timestep of a simulation by solving the Poisson equation using a multigrid or FFT method. These algorithms are discussed in further detail on the astrophysical hydrodynamics page. I wrote the hydrodynamics and Poisson codes as part of my Ph.D. thesis work with Don Lamb at the University of Chicago. Scott Dodelson of the Fermilab Theoretical Astrophysics Group wrote the N-body code.

Features

The modular design of COSMOS permits different sets of initial and boundary conditions to be compiled into the code in a straightforward manner with an easy-to-use setup script. Most parameter values are set through an ASCII parameter file, which can be edited and used without recompiling the COSMOS executable. In COSMOS 2.0, all of the variable buffers are allocated at runtime, enabling grid sizes also to be changed without recompilation. COSMOS currently handles the following physics:

* Nonrelativistic, compressible, inviscid Eulerian hydrodynamics in 1D spherical and 1/2/3D Cartesian coordinates, including flows with strong shocks
* Collisionless, self-gravitating particle dynamics in 3D Cartesian coordinates
* Coupled self-gravity of the collisional and collisionless fluids
* Static or comoving (cosmological) coordinate scaling
* Static, nonuniform meshes with: periodic, outflow, isolated, reflecting, or user-specified boundaries for hydrodynamics; periodic, Dirichlet, isolated, or given-value boundaries for self-gravity; and periodic or isolated boundaries for particle dynamics
* Point source functions for the gas, such as a temperature-dependent cooling function
* Additional advected quantities, such as fluid mixtures, tracer fields, and abundances of different elements; these can also have their own source and sink functions
* Different equations of state, including perfect gas and single-temperature gas plus radiation mixture

The code creates a log file for each job along with scalar (time-dependent global) data files for the gas and dark matter. Checkpointing is accomplished through alternating restart files which are written to disk at user-specified timestep intervals. In addition, visualization outputs (3D volumes and 2D slices) are written at user-specified simulation time intervals. Currently these use a special data format, which can be read using a collection of Fortran and IDL routines we have written, but they will soon be converted to use the Hierarchical Data Format to enable their use with a wider variety of visualization tools.

Parallel implementation

COSMOS uses message-passing with explicit domain decomposition to achieve scalability on parallel computers. COSMOS 2.0 uses MPI for parallel communications, while COSMOS 1.0 uses PVM. We have used COSMOS 1.0 for production runs on the Cray T3D and T3E at the Pittsburgh Supercomputing Center and the San Diego Supercomputer Center.

PE Grid Image
Each processor (PE) controls a contiguous subdomain of the complete grid. Communication of boundaries occurs between logically adjacent PEs.

Each processor (PE) handles a distinct subset of the full domain. This decomposition is the same for the hydrodynamics, gravity, and N-body components of the code. PEs are assigned coordinates in the PE-grid according to the scheme depicted here. For simplicity and efficiency the decomposition requires that the number of processors along each dimension be a factor of two. If self-gravity is used, the number of zones along each dimension must also be a factor of two.

For the hydrodynamics module, this decomposition produces high scalability and a balanced computational load. Prior to each timestep, PEs trade boundary information with their neighbors, then solve the hydrodynamic equations within their subdomains as though each were the only processor. Execution time scales as NPE-1, where NPE is the number of processors assigned, until the subdomain size becomes equal to the number of boundary zones required in each dimension (four for PPM), whereupon communication, a fixed cost, begins to dominate.

Progressive Redundancy Scheme
Progressive redundancy scheme used to parallelize the V-cycle multigrid algorithm for grids that are too coarse to simply decompose.

The multigrid code requires a more sophisticated approach. It requires that we subdivide not merely the grid with which the hydrodynamics is discretized, but also several levels of coarser grids, each of which has a factor of two fewer zones per dimension than the next finer one. As long as the number of PEs along a dimension is smaller than one-half the number of zones in a grid along that dimension, the grid can be divided evenly without redundancy. However, sufficiently coarse grid levels require that some of the PEs be redundant on such levels. We progressively increase the level of redundancy as grid coarseness increases so as to minimize the impact of this difficulty on the parallel efficiency of our code. Nevertheless, for a fixed problem size the execution time for this module scales approximately as aNPE-1 + blnNPE, where a and b are constants.

Information for each particle (position, velocity, mass, etc.) is stored by the PE which controls the subdomain in which it lies. When a particle leaves a subdomain, a message is sent to the processor which controls its destination. While this does not permit good load balancing under some circumstances, the overhead associated with computing particle densities and forces (via the particle-mesh procedure) is a small fraction of the total cost of a timestep; the force calculation easily dominates. We limit the cost of particle messages by double-buffering and by having each PE communicate only with its neighbors. This is possible because we limit the timestep in such a way that no particle can move more than one zone in a timestep.

Test results

As with any scientific code of this complexity it is important to apply COSMOS to a number of test problems with known solutions before using it on research problems. This is important not simply for the purpose of verifying that the code works as it should, but also because it gives the user an intuitive understanding of the code's strengths and weaknesses. Such an understanding is essential for making effective use of any hydrodynamical code.

We have performed a number of tests to exercise the various code modules in different combinations. The COSMOS test paper gives detailed descriptions and results of tests performed with COSMOS 1.0. We are in the process of documenting test results for COSMOS 2.0; these are described in pages linked from the following table.

Problem Hydro Gravity N-body Redshift Cooling
Sedov explosion X        
Sod shock tube X        
Square wave advection X        
Emery wind tunnel X        
Two blast waves X        
Cooling test         X
Radiative shock X       X
Redshift test       X  
Dust cloud potential   X      
Sine potential   X      
Dust collapse I X X      
Jeans instability X X      
Dust collapse II X X   X  
Zel'dovich pancake I X X   X  
Cooling flow X X     X
Two particle orbit   X X    
Two particle force   X X    
Dust collapse III   X X    
Dust collapse IV   X X X  
Zel'dovich pancake II   X X X  
SCDM cosmology I X X   X  
SCDM cosmology II   X X X  
SCDM cosmology III X X X X  
Santa Barbara cluster X X X X  

Distribution

At present COSMOS is not publicly distributed, though this may change in the future. We encourage scientists interested in using the code to contact us for up-to-date information.

Publications

COSMOS: A Hybrid N-Body/Hydrodynamics Code for Cosmological Problems
Ricker, P. M., Dodelson, S., and Lamb, D. Q. Ap. J. 536 122 (2000)

 
[Home] [Research] [Teaching] [Papers] [Interests] [Links] disclaimer 09/01/16