Requirements, Trades, and Risks

V. Hunter Adams (vha3), MAE 4160/5160, Spring 2020

from IPython.display import Latex
from IPython.display import Image
from IPython.core.display import HTML

Other reading

1. SMAD 6, 24
2. JWST requirements document
3. ISS SRD document
4. Landsat document

In this lecture

1. Identifying stakeholders and stakeholder needs, goals, and objectives.
2. Technical requirements definition process.
3. Types of requirements.
4. How to write valid requirements.
5. Requirements verification.
6. Trade studies.
7. Evaluating risk.

Sources and further reading

1. https://www.nasa.gov/seh/4-design-process
2. https://www.nasa.gov/seh/appendix-c-how-to-write-a-good-requirement
3. https://www.nasa.gov/seh/appendix-e_creating-the-validation-plan

It all starts with stakeholders

A stakeholder is any group or individual that is affected by or has a stake in the product or project. The key players for a project are called the key stakeholders. One of the key stakeholders for your project is always the customer. The customer can be different depending which level in the systems heirarchy that one is working. For engineers working a few levels down the systems heirarchy, the customer may be the leader of a team which takes that engineer's product and integrates it into the larger system. At the highest level, the customer is the person or organization which is purchasing the product.

Other stakeholders may be more difficult to identify, and could include Congress, advisory planning teams, program managers, mission partners, the media, prime contractors, regulatory agencies, end users, etc. The below table shows some examples for stakeholders in a NASA science mission at various phases in its life cycle. For commercial missions, these stakeholders may be quite different.

Image("stake.png", width=800)

For your class projects, your TA's and I will play the role of the customer and/or principle investigator for each project. The success of a system depends entirely on satisfying stakeholders. It is not about maximizing performance or minimizing cost, it is about satisfying stakeholder needs.

Since the success of your mission depends on satisfying stakeholder needs, it is clearly very important to understand their expectations for the mission. This can be difficult, since the needs and expectations for many stakeholders may be qualitative and fuzzy. For the most part, these needs will be independent of the system itself, they will be some goal that the system that you design will accomplish. Stakeholder expectations are organized into needs, goals, and objectives, each of which is progressively more specific. Needs are defined in the answer to the question “What problem are we trying to solve?” Goals address what must be done to meet the needs; i.e., what the customer wants the system to do. Objectives expand on the goals and provide a means to document specific expectations. (Rationale should be provided where needed to explain why the need, goal, or objective exists, any assumptions made, and any other information useful in understanding or managing the NGO.)

Needs: A single statement that drives everything else. It should relate to the problem that the system is supposed to solve but not be the solution. The need statement is singular. Trying to satisfy more than one need requires a trade between the two, which could easily result in failing to meet at least one, and possibly several, stakeholder expectations.

Example from Landsat: Monitor changes in the Earth's surface.

Goals: An elaboration of the need, which constitutes a specific set of expectations for the system. Goals address the critical issues identified during the problem assessment. Goals need not be in a quantitative or measurable form, but they should allow us to assess whether the system has achieved them.

Example from Landsat Data Continuity Mission: The goal of the LDCM, consistent with U.S. law and government policy, is to continue the acquisition, archival, and distribution of multi-spectral imagery affording global, synoptic, and repetitive coverage of the Earth's land surfaces at a scale where natural and human- induced changes can be detected, differentiated, characterized, and monitored over time. - SMRD

Example from JWST: The primary goal of the JWST is to observe the early universe, at an age between 1 million and a few billion years. - JWST mission requirements doc

Objectives: Specific target levels of outputs the system must achieve. Each objective should relate to a particular goal. Generally, objectives should meet four criteria.

1. They should be specific enough to provide clear direction, so developers, customers, and testers will understand them. They should aim at results and reflect what the system needs to do but not outline how to implement the solution.
2. They should be measurable, quantifiable, and verifiable. The project needs to monitor the system’s success in achieving each objective.
3. They should be aggressive but attainable, challenging but reachable, and targets need to be realistic. Objectives “To Be Determined” (TBD) may be included until trade studies occur, operations concepts solidify, or technology matures. Objectives need to be feasible before requirements are written and systems designed.
4. They should be results-oriented focusing on desired outputs and outcomes, not on the methods used to achieve the target (what, not how). It is important to always remember that objectives are not requirements. Objectives are identified during pre-Phase A development and help with the eventual formulation of a requirements set, but it is the requirements themselves that are contractually binding and will be verified against the “as-built” system design.

Examples from Landsat Data Continuity Mission (SMRD):

1. Collect and archive moderate resolution (circa 30 m ground sample distance) multispectral image data affording seasonal coverage of the global landmass for a continuous period of not less than 5 years.
2. Ensure that LDCM data are sufficiently consistent with data from the earlier Landsat missions in terms of acquisition geometry, calibration, coverage characteristics, spectral characteristics, output product quality, and data availability to permit studies of land cover and land use change over multi-decadal periods.
3. Distribute LDCM data products to the general public on a nondiscriminatory basis and at a price no greater than the incremental cost of fulfilling a user request.

Objectives may be somewhat fuzzy/imprecise. They should specify what the system is supposed to do, without specifying how the system will do it. We derive requirements from these objectives. Requirements are not fuzzy at all. They are unambiguous, concise, measurable, unique, consistent, and isolated.

For your projects, you are given objectives. Your first task, for the SRR, is to derive requirements from these objectives.

Image("tasks.png", width=800)

Requirements Engineering

Requirements definition is an iterative process through which vague stakeholder needs are progressively refined into specific, unambiguous, quantitative requirements. This is often done in parallel with mission concept definition, since the concept and the requirements inform one another. The ambiguity about various concepts is reduced during this process.

After SRR, the requirements are placed into configuration management.

There are high-level (or system-level) requirements, and there are lower level requirements. We typically begin with the high level requirements and use those to inform and derive lower level requirements. There are different types of requirements, and a very specific set of guidelines for properly writing them.

What are requirements for?

Requirements are how we specify the system that is to be built. For spacecraft, the systems are simply too complex and the cost of design changes is too high to take the engineering approach that you might take for something like commercial product development. It would be too expensive to build, test, iterate, build, test, iterate, etc. for the entire system (though we may do that for components of the system). Instead, we must all agree (engineers and stakeholders) very precisely on the specifications to which the system should be built, and then we build to those specifications.

Requirements specify the system in terms of:

1. What the system must accomplish.
2. What constraints the system must satisfy.

Requirements specify the problem, not the solution, and they form the basis for the system's design, manufacture, verification, and operation.

The requirements generation process

The graphic below, from the NASA SEH, illustrates the requirements definition process and identifies typical inputs, outputs, and activities to consider in addressing technical requirements definition.

Image("requirements.png", width=800)

1. Define constraints, functional and behavioral expectations

The first step in the technical requirements definition process is to establish the top-level requirements. These exist in order to understand the technical problem to be solved, the scope of that problem, and the design boundary. This typically involves the following activities:

1. Defining constraints that the design needs to adhere to or that limit how the system will be used. The constraints typically cannot be changed based on trade-off analyses. The system shall cost less than 10M. The JWST wet mass shall not exceed 6,159 kg.
2. Identifying those elements that are already under design control and cannot be changed. This helps establish those areas where further trades will be made to narrow potential design solutions.
3. Identifying external and enabling systems with which the system should interact and establishing physical and functional interfaces (e.g., mechanical, electrical, thermal, human, etc.). The JWST Observatory shall meet the interface requirements to the Launch Segment defined in the Application to Use Ariane (DUA) IRD (JWST-IRD-003674). The operational JWST shall utilize the Deep Space Network.
4. Defining functional and behavioral expectations for the range of anticipated uses of the system as identified in the ConOps. The ConOps describes how the system will be operated and the possible use-case scenarios. The JWST shall orbit the second Lagrange point (L2) of the Sun-Earth system. The operational JWST System shall have at least one two-way communication contact between the Observatory and Ground Segment in a 24 hour period.

These top-level requirements come from stakeholder needs, the concept of operations, regulations, etc. With an overall understanding of the constraints, physical/functional interfaces, and functional/behavioral expectations, the requirements can be further defined by establishing performance and other technical criteria. The expected performance is expressed as a quantitative measure to indicate how well each product function needs to be accomplished.

Functional requirements: Functional requirements define what functions need to be performed to accomplish the objectives.

Performance requirements: Performance requirements define how well the system needs to perform the functions.

Technical requirements come from a number of sources, including functional, performance, interface, environmental, safety, human interfaces, standards, and in support of the "ilities" (reliability, sustainability, producibility, etc.). With the system-level requirements established, we then delegate and allocate requirements to successively lower subsystems. Each of these subsystems will also have functional and performance requirements, and a few other flavors of requirements.

2. Delegate/Allocate requirements to successively lower subsystems

We decompose/refine system requirements to successively lower level subsystems, to components, and to manufacturing processes, materials and tolerances, and integration back to the system. The figure below shows how this flowdown typically looks. This will generally involve the allocation of system budgets (mass, power, volume, $\Delta V$, data rate, reliability, etc.) to various subsystems. Deciding how much of each budget to allocate to each subsystem is an iterative process that can be informed by experience/rules of thumb, formal optimization methods, and guessing/iterating.

Image("flowdown.jpg", width=500)

These requirements come in a variety of flavors, each of which is explained below.

Functional requirements: Functional requirements define what functions need to be performed to accomplish the objectives. These are generally derived from system-level functional requirements.

1. The Thrust Vector Controller (TVC) shall provide vehicle control about the pitch and yaw axes.

Performance requirements: Performance requirements define how well the system needs to perform the functions. These are generally derived from system/subsystem level functional requirements.

1. The TVC shall gimbal the engine a maximum of 9 degrees, ± 0.1 degree.
2. The TVC shall gimbal the engine at a maximum rate of 5 degrees/second ± 0.3 degrees/second.
3. The TVC shall provide a force of 40,000 pounds, ± 500 pounds.
4. The TVC shall have a frequency response of 20 Hz, ± 0.1 Hz.

Interface requirements: Requirements that specify the functional or structural interfaces among subsystems.

1. The power subsystem shall provide 12V DC at up to 1.5 A to the payload.
2. The launch-vehicle upper stage shall provide the spacecraft with positive detection of separation via a +5 V signal on pin 8 of the umbilical connector.

Customer requirements: These will include product expectations, mission objectives, operational concerns, and/or measures of effectivity and suitability. It may require careful analysis to extract functions, and success criteria are generally provided.

1. The rock-abrasion tool shall be capable of grinding away some part of the surface of any rock encountered on the surface of Mars to permit scientists to analyze a portion that has not been exposed to weathering.

Design requirements: These are requirements derived from process specifications (e.g. MIL STDs), or internal best practices (tolerances, trade-secret guidelines, design for manufacturability, etc.). These are often associated with "design for X."

1. All control loops shall demonstrate at least 20 deg. phase margin by analysis.
2. Mechanisms shall be designed with torque margin in compliance with MIL-STD 1540-D.

Verification requirements: Requirements that specify the way in which verification must proceed—test requirements, analysis methodologies, etc. (We'll go over verification in some detail a bit later).

1. The rover wheel-bearing life tests shall include no accelerated component testing.
2. Corrosion testing shall be performed within the temperature range 10-120 deg. C.

Any of the above types of requirements could flowdown from a higher-level requirement. In that case, each will also fall into one or both of two categories for flowdown requirements: derived and allocated.

Derived requirements: Any requirements flowed down from a higher level.

1. SYS 1.4: The launch vehicle shall be capable of taking off from any NATO aircraft carrier.
2. STRUCT 1.0: The launch vehicle’s gross lift-off weight shall be less than 10,000 kg. [SYS 1.4]

Allocated requirements: Any requirement established by dividing or allocating a higher-level requirement into more than one requirement at a lower level.

1. SYS 1.4: The launch vehicle shall be capable of taking off vertically from any NATO aircraft carrier.
2. STRUCT 1.0: The launch vehicle gross lift-off weight shall be less than 10,000 kg. [SYS 1.4]
3. ENG 1.0: The sea-level standard-day thrust shall be no less than 209,000 N. [SYS 1.4]

Writing requirements

You'll have noticed that all of the example requirements that we've seen take a very particular form. There's a very specific way to write a valid requirement. A valid requirement is one which is unambiguous, isolated, concise, measurable, unique, and consistent. By following a set of rules, we can make certain that our requirements are valid ones.

1. Preferred verb is "shall."

Anything else ("should," "ought," etc.) implies a soft requirement, to which the system will not be held during verification.

In general, when you're putting together these requirements, you should imagine that you are sitting across from a lawyer. That lawyer is attempting to prove that your system does not meet requirements by exploiting any vague language, inconsistency, unmeasurable guarantee, etc. Write your requirements such that they stand up to this imaginary lawyer's scrutiny.

2. The grammar establishes the flow of the requirement.

A requirement should be a single sentence. The subject is a system, element, subsystem, component, etc., which establishes the functional level at which the requirement is relevant. The verb often implies the type of verification (test, inspect, analyze, etc.). The object of the verb is often a Technical Performance Measure (TPM).

Which of the below is a good example (bold), and which is a bad example?

1. The rover drive system shall weigh less than 5 kg.
2. The weight of the rover harness shall be less than 1 kg.

3. Requirements are unambiguous.

Unambiguous requirements are free of words and phrases such as "reasonable," "acceptable," "minimize," and "where applicable.“ Unambiguous requirements are not a matter of opinion, and cannot be misinterpreted. Quantitative requirements are often unambiguous, but qualitative ones can also be valid. Remember, don't give the imaginary lawyer any room for interpretation.

Which of the below is good (bold), and which is bad?

1. The rover shall be very fast.
2. The rover shall be capable of collecting three surface sample in less than 75 minutes.

4. Requirements are isolated.

Each "shall" statement belongs in a separate, unique requirement (i.e., no conjunctions). Constraining each paragraph to contain no more than one "shall" allows one to take full advantage of the viewing, reporting, and traceability functions of requirements-management tools. Isolation allows full traceability, discrete referencing, and one-to-one verification cross referencing.

1. The rover shall weigh less than 10 kg and shall operate when tethered to a 28 V nuclear power source
2. The rover shall weigh less than 10 kg.

5. Requirements are measureable.

Each requirement will be verified (by test, analysis, inspection, etc.). If the requirement cannot be verified, it cannot be tested. A measurable requirement is the only type that can be verified. (yes/no is a type of measurement)

1. The rover shall be robust to failures.
2. The rover shall be single-fault tolerant.

6. Requirements are concise.

Don’t include explanations, definitions, or other information unrelated to the specification; use a glossary, a list of acronyms, etc. in the documentation instead.

1. The rover shall be able to work in the dark, i.e. at 0.2 cd (where “cd” refers to a candela, one lumen per steradian)
2. The rover shall be capable of operating at light levels below 0.2 cd.

7. Requirements are unique.

It is easy in long documents created by teams of people to identify the same requirement multiple times in slightly different forms. The work to be done is deciding which version of the requirement to retain and which to delete.

1. The rover’s rubber seals shall function within the range 10-120 deg C” [source: Europa science spec]
2. The rover’s rubber seals shall function after having been exposed to temperatures in the range 15-50 deg C.” [source: Launch Vehicle ICD]

The benefits of well written requirements

Summarized below, from the NASA SEH.

Image("benefits.png", width=500)

Validating and managing requirements

Before any requirement is accepted, it must be validated. This is different from being verified, which will discuss momentarily. Requirements are validated against the stakeholder expectations, the mission objectives and constraints, the concept of operations, and the mission success criteria. Validating requirements can be broken into six steps:

Are the requirements written correctly?: Identify and correct requirements “shall” statement format errors and editorial errors. See above section.

Are the requirements technically correct?: A few trained reviewers from the technical team identify and remove as many technical errors as possible before having all the relevant stakeholders review the requirements. The reviewers should check that the requirement statements (a) have bidirectional traceability to the baselined stakeholder expectations; (b) were formed using valid assumptions; and (c) are essential to and consistent with designing and realizing the appropriate product solution form that will satisfy the applicable product life cycle phase success criteria.

Do the requirements satisfy stakeholders?: All relevant stakeholder groups identify and remove defects.

Are the requirementes feasible?: All requirements should make technical sense and be possible to achieve.

Are the requirements verifiable?: All requirements should be stated in a fashion and with enough information that it will be possible to verify the requirement after the end product is implemented.

Are the requirements redundant or over-specified?: All requirements should be unique (not redundant to other requirements) and necessary to meet the required functions, performance, or behaviors.

Verification, which we'll discus in a moment, answers the question Did we build the system right? Validation, by contrast, answers the question Did we build the right system?

For a complex system, maintaining traceability among all requirements is of critical importance. Typically, one uses a requirements management tool (e.g. DOORS or a SysML tool) to generate: Requirements statements, Requirements traceability (a matrix or tree), Verification cross-reference matrices, Lists of TBRs, TBDs, etc. For each requirement, the metadata shown in the table below is stored.

Image("metadata.png", width=800)

In your SRR, it should be clear which requirements are derived/allocated from other requirements.

Verifying requirements

A requirement is only as useful as it is verifiable. If we can't prove that our system satisfies a particular requirement, then that requirement is useless. Tests without requirements are wasted effort. Requirements without tests will prevent the development cycle from being complete. Every requirement must be verified. This verification may take the form of:

Test: uses special equipment to measure quantitative characteristics.

Demonstration: special kind of tests that qualitatively demonstrate correct operation of the system without physical measurements (e.g., reliability)

Inspection: e.g., visual examination

Analysis: e.g., theory, simulation

Analogy/similarity: e.g., if flight software is the same as previous mission and context has not changed, some tests can be waived.

As shown in the below diagram, system requirements are allocated to lower level subsystem requirements. Low-level requirements are verified, and then subsystems are verified, and finally the full system is verified to meet system requirements.

Image("v.png", width=800)

Trade Studies

With the functional and performance requirements for each subsystem established (the goal of your SRR assignment), we must then design an architecture that meets those requirements (the goal of your SDR assignment). For many of the subsytems, you may have multiple architecture options from which to choose. For attitude control, for example, perhaps you must decide whether you are going to use reaction wheels or CMG's. And perhaps you need to decide if you're going to dump accumulated momentum from those thrusters using torque coils or thrusters. For other subsystems too, you will have multiple options and the "correct" choice may not be clear.

There is a field of study devoted to making these sorts of decisions, and you have many options at your disposal. I am only going to present a single method.

Pugh Matrix

1. Define a set of alternative options.
2. Define a set of selection criteria (the attributes which drive the design, e.g. cost, risk, complexity, mass, etc.).
3. Choose a reference option.
4. Compare all of the options with the reference options across all criteria. Put a + of the option is better than the reference option, a 0 if it is the same, and a - if it is worse.
5. Compute the score for each alternative (score = # '+' # '-' # . . . ), adding weights as necessary/desired.
6. Choose the alternative with the highest score.
7. Eliminate poor alternatives, add new ones, eliminate non-distinguishing criteria, and iterate.

Image("pugh.png", width=800)

Image("pugh2.png", width=800)

Evaluating risk

You'll note that one of the criteria in the above Pugh matrix example is "risk." This has a formal definition.

Risk: A measure of the probability and severity of adverse effects.

Reliability: The ability of a system or component to perform its required functions under stated conditions for a specified period of time.

Opportunity: A measure of the probability and the benefit of beneficial effects.

When performing a risk analysis, you're asking yourself the following three questions about the alternative under consideration:

1. What can go wrong?
2. What is the likelihood that it would go wrong?
3. What are the consequences?

The answer to "what are the consequences" place a particular failure into one of two categories: a hard failure or a soft failure. A hard failure results in complete loss in functionality. These sorts of failures are easy to analyze, and rarely encountered in practice. A soft failure, by contrast, results in partial loss of funcitonality and is much more common and difficult to analyze.

Visualizing risk

"Risk" is some function of probability and consequence. Does it make sense that our contours for equal risk will look something like those shown below?

Image("risk.png", width=500)

Alternatives with low probability and low consequence are low risk. Alternatives with high probability and high consequence are high risk. Furthormore, low consequence alternatives are always low risk, regardless of their probability. High consequence events, by contrast, are low risk if they are low probability, and high risk if they are high probability. High probability events should have a risk directly proportional to their consequence.

This is sufficient information to draw the above, approximate curves. A common tool for evaluating risk is the Stoplight Chart, shown below, which approximates the above set of curves to a discrete number of blocks. Different spotlight charts may use different numbers of blocks, but they represent a quantization of the above continuous risk landscape.

Image("stoplight.png", width=500)

By quantifying the probability of a particular failure, and the consequence of that particular failure (which may not be a straightforward task), we can classify each alternative as high risk, medium risk, or low risk. This is a method for coming up with a rank metric for use in the trade studies described above.

An example story

Dr. Peck tells a story about a particularly interesting spacecraft failure, which was caused by the following chain of events:

1. Spacecraft attitude control failed.
2. Spacecraft attitude control failed because a thruster failed.
3. The thruster failed because the propellant flow was impeded.
4. The propellant flow was impeded because the manufacturing process failed.

The manufacturing process failed for the following reasons:

1. During the manufacturing process, technician A installed a foam plug meant to keep the thrust chamber clean, but did not document this fact in his log.
2. On the next shift, technician B installed another plug, thinking it had not yet been done.
3. Much later, during pre-launch activities, a technician removed technician B's plug, the single plug demanded by the operations process, unknowingly leaving technician A's inside.
4. Once in space, the foam plug prevented the thruster from firing correctly.

In this particular case, this was a soft failure which resulted in degraded (but not total loss of) functionality. The spacecraft was 8-for-7 thruster redundant, so there was an extra thruster which could pick up the slack for the failed one.

What is the risk of this particular failure? What is the consequence? Where would you place it in the stoplight chart?

Organizing faults

These potential faults are organized into fault trees, as shown below. Failures at the system, subsystem, and component levels are linked by logic gates in order to visualize which combinations of hard/soft failures result in subsystem or system failure. This is an SDR-level tool, since it requires an established spacecraft architecture to construct. Though, tools like this may also be used to inform architecture decisions.

Image("tree.png", width=500)