Framework Software Test Plan

Test Scope

This plan details the testing implemented for the development of MOOSE or MOOSE-based application that leads to a stable revision, which upon a secondary review by the Project Lead can be released. The Nuclear Quality Assurance Level 1 (NQA-1) standard necessitates reviews and approvals for each release. This plan describes how automated testing fulfills obligatory testing of the software and how the Project Lead can leverage this information when performing a release.

MOOSE or MOOSE-based applications use an agile development method that tests all proposed changes prior to integration into the main repository. This allows developers to exercise proposed changes as they are written and ensure that existing code is not impacted in adverse ways. Testing is an integral part of the normal development process, as such the necessary testing and reviews to satisfy the NQA-1 standard are natural for contributors to the project.

All testing performed is "dynamic" that attempt to identify defects by executing the software. All testing for each revision is performed automatically using Continuous Integration for Verification, Enhancement, and Testing (CIVET). After automated testing has successfully completed and a technical review is performed, an automated merge is made into a "stable" revision. Each revision is eligible for release at the discretion of the Project Lead and subject to a complete release review.

Test Objectives

All test types, as detailed in Types of Tests to be Executed, for MOOSE or MOOSE-based application have a single objective: that when executed with the prescribed input the software shall produce the expected output. The type of test indicates the type of output that shall be considered, which can range from numerical verification to error testing. The overall object is to provide the necessary confidence that the software will perform as expected for each for the defined test cases to ensure that the software properly handles abnormal conditions and events as well as credible failures, does not perform adverse unintended functions, and does not degrade the system either by itself, or in combination with other functions.

By the nature of the software as a library it is not possible to guarantee the functionality of the software from an end-user perspective, since the input provided by the user cannot be controlled.

Assumptions

MOOSE and MOOSE-based applications are assumed to be dynamically-linked command-line UNIX (POSIX) compatible executable built on the target system. Being High Performance Computing (HPC) software, and the fact that our normal configuration relies on shared-libraries, it is generally not advisable to build MOOSE on one system and execute it on another system (with the exception of a homogeneous cluster environment).

MOOSE and MOOSE-based applications are assumed to be stateless, reading all inputs from local or network mounted file-systems. When deployed for parallel testing or use, standard Message Passing Interface (MPI) networking is expected to function among cluster compute nodes. MOOSE or MOOSE-based applications does not require any special file system (i.e., parallel file systems), however high performance file systems can improve performance of large simulations and also the speed at which the automated testing system can launch, run, and inspect test results.

Constraints

MOOSE and MOOSE-based applications are designed to be built and executed in-situ on the end-use machine. There is no requirement for separate testing or acceptance environments, as each independent invocation of a simulation maintains its own environment. Acceptance testing may be performed at full-scale provided resources are available. Therefore, there are no constraints on testing of MOOSE and MOOSE-based applications.

Types of Tests to be Executed

It is possible to categorize test cases in many ways such as "system", "integration", "performance", or "acceptance" testing. MOOSE and MOOSE-based applications do not categorize test cases; they are simply defined and executed as a complete set and automatically executed as defined in Test Automation (Scripting).

Identification of the stages at which testing is required

Testing for MOOSE and MOOSE-based applications shall include the levels of testing as defined in Figure 1. The testing is automated to the extent possible and the "Next" and "Devel" branch testing may be combined at the discretion of the application.

Figure 1: Required stages for testing of MOOSE, MOOSE-based applications, and . The "Next" and "Devel" branch testing may be combined at the discretion of the application.

Approval Requirements

All test cases are created by contributors during the development process and approved by independent reviewer. The creation of the test cases follows the change control process. These test cases can be in response to bug fixes or as a part of an enhancement.

Test Iteration

MOOSE and MOOSE-based applications are stateless, deterministic software for a given input. Therefore, a single testing iteration on each identified configuration is sufficient for completing the required tests necessary for deployment.

Test Automation (Scripting)

MOOSE and MOOSE-based applications rely heavily on full test automation. Since each application is stateless and command-line driven, developing a thorough test suite is generally more straightforward than it is for other business system type software. The MOOSE repository includes a general-purpose, extendable "Test Harness", which is heavily leveraged to run the test cases. The Test Harness is used throughout all phases of MOOSE development from initial development, the change request process, deployment testing, and finally end-use in-situ testing. The Test Harness is even suitable for testing on large deployment clusters and supports the "PBS" queuing system.

The Test Harness includes a suite of "Tester" types to enable complete testing of MOOSE and MOOSE-based applications. For each of the types, the Test Harness is able to execute the application with a developer designed input and verify the correct result automatically. A complete list of the built-in Testers is included here:

RunApp
A tester designed to assemble common command line arguments for executing MOOSE-based application including launching with MPI or threads.
Anticipated Output: A return code of 0 or 1.
Acceptance Criteria: A non-zero return code is acceptable (pass).
Reports and Records: The return code and the system standard output returned from the application.

RunCommand
A generic tester that can execute an arbitrary command.
Anticipated Output: A return code of 0 or 1.
Acceptance Criteria: A non-zero return code is acceptable (pass).
Reports and Records: The return code and the system standard output returned from the command.

FileTester
An intermediate base class that runs a MOOSE-based command and expects to process a file written out by the MOOSE-based simulation.
Anticipated Output: Files created as a result of the execution of an application.
Acceptance Criteria: If the files are created the result is acceptable (pass).
Reports and Records: This list of files missing upon failure

RunException
A tester that expects the application to produce a warning or error based on bad inputs, missing files, permissions, etc.
Anticipated Output: A non-zero exist status and an error message.
Acceptance Criteria: If the exit status is non-zero and the error message is present the result is acceptable (pass).
Reports and Records: The return code and the system standard output returned from the application.

CheckFiles
A tester that looks for the creation of specific files after a simulation runs without regard for the contents of those files.
Anticipated Output: Files created with the expected content.
Acceptance Criteria: If the files are created the desired content exists then result is acceptable (pass).
Reports and Records: This list of files created an the missing content.

Exodiff
Compares "ExodusII" format files output by the simulation to those checked into the repository as the "gold" standard for a given test within numeric tolerances.
Anticipated Output: ExodusII files created as a result of the execution of an application.
Acceptance Criteria: If the files created match a known "gold" standard the result is acceptable (pass).
Reports and Records: Upon failure, a report detailing the differences in the files.

CSVDiff:
Compares "CSV" format files output by the simulation to those checked into the repository as the "gold" standard for a given test within numeric tolerances.
Anticipated Output: CSV files created as a result of the execution of an application.
Acceptance Criteria: If the files created match a known "gold" standard the result is acceptable (pass).
Reports and Records: Upon failure, a report detailing the differences in the files.

ImageDiff
Compares various image format files output by the simulation to those checked into the repository as the "gold" standard for a given test within tolerance.
Anticipated Output: Image files created as a result of the execution of an application.
Acceptance Criteria: If the files created match a known "gold" standard the result is acceptable (pass).
Reports and Records: Upon failure, a report detailing the differences in the files.

VTKDiff
Compares "VTK" format files output by the simulation to those checked into the repository as the "gold" standard for a given test within numeric tolerances.
Anticipated Output: VTK files created as a result of the execution of an application.
Acceptance Criteria: If the files created match a known "gold" standard the result is acceptable (pass).
Reports and Records: Upon failure, a report detailing the differences in the files.

JacobianTester
Appends additional arguments to the command line to trigger special solver modes for the purpose of producing "finite-difference" Jacobians for which to compare the Jacobians produces by the simulation.
Anticipated Output: A return code of 0 or 1.
Acceptance Criteria: If the simulation values match the finite-difference the result is acceptable (pass).
Reports and Records: Upon failure, a report detailing the differences between the Jacobians.

PythonUnitTest
A tested designed to run Python scripts within the MOOSE test suite, generally containing unit tests.
Anticipated Output: A script return code of 0 or 1.
Acceptance Criteria: A 0 return code (meaning the script completed successfully) is acceptable (pass).
Reports and Records: Upon failure, a report detailing the failures experienced within the test script.

Resource Requirement

This section must clearly articulate what type of skill set is required for all the planned testing and when and for how long the resources are required.

In this section, the test planner must also identify the testing environment requirements, such as storage, servers, number of licenses for test automation tools...

Tasks and Responsibilities

The creation and execution of the test cases is part of the change control process, as such the associated roles and responsibilities are minimal.