Mesh Splitting
MOOSE provides the ability to pre-split a mesh into several chunks for use with distributed mesh. This can be useful if the whole mesh is too big to fit into memory. A split-mesh workflow involves generating the mesh split configuration(s) and then telling moose to use them when you run simulations. This is described in the sections below.
Generating Split Configurations
You can run MOOSE with the --split-mesh <split1>[,<split2>]... flag which takes a comma-separated list of describing each split with an integer number of chunks to split the mesh into.
$ moose-app-opt -i your_input.i --split-mesh 13,42,1000
Splitting 13 ways...
- writing 13 files per process...
Splitting 42 ways...
- writing 42 files per process...
...
This will create three split configurations for your mesh that include 13, 42, and 1000 chunks each. If your input file uses the mesh file foo.e, this process will generate a directory named foo.cpr. In general, you should neither rename this directory nor modify its contents. All split configurations for the mesh will be stored in this directory (even across independent splitting operations).
For non-file-based mesh cases (e.g. GeneratedMesh), you will need to tell MOOSE what name to use for the split mesh files it will generate using the --split-file <file_name> flag; in general, you should use the .cpr file extension here. For example:
$ moose-app-opt -i your_input.i --split-mesh 42 --split-file foo.cpr
You can also run the splitting operation itself in parallel on several processors to speed up the operation:
$ mpiexec -n 9 -i moose-app-opt your_input.i --split-mesh 42
Splitting 42 ways...
- writing 5 files per process...
This will generate the same split configuration as the 42 chunk split generated by the first command but just generates it in parallel.
It is noted that if there are mesh meta data generated by mesh generators, these meta data will be written to a binary file under the generated directory that can be loaded when using split meshes.
The mesh splitter commands do not work with a distributed mesh. You can only split starting from a replicated mesh.
Using Split Meshes
There are two ways of referring to distributed meshes in MOOSE, as split or as distributed. A split mesh is a distributed mesh. A split mesh is read from a pre-split mesh file (from a Checkpoint or using the instructions above). A distributed mesh can be generated in parallel or loaded entirely replicated then distributed.
Loading a split mesh using FileMeshGenerator
A split mesh, i.e. a .cpr file, may be loaded using a FileMeshGenerator if access to the mesh is needed before the full simulation case is set-up, notably for additional mesh generation with mesh generators. In fact this approach is recommended; it leads to correct element ghosting whereas the approach we introduce in the following section may not.
Loading a split mesh using --use-split
Alternatively, a previously split mesh can be loaded using the --use-split flag (which takes no arguments):
$ mpiexec -n 42 moose-app-opt -i your_input.i --use-split
You may also use the Mesh/use_split input parameter.
This will cause MOOSE to look for a mesh split configuration with 42 chunks. If one exists, MOOSE will use it running in a distributed mesh mode, otherwise an error will occur. Note that you do not need to modify your input file - MOOSE automatically switches to using the pre-split distributed mesh regardless of the mesh type specified in the input file. If your input file does not use a file-based mesh, you will need to specify a split mesh file name using the --split-file <file_name> flag just as you did when splitting the mesh:
$ mpiexec -n 42 moose-app-opt -i your_input.i --use-split --split-file foo.cpr
The --use-split option skips entirely the [Mesh] block of the input file, and as such some data structure initializations required if further mesh operations are necessary at the simulation start-up stage.