One of the features of the CERT Resiliency Engineering Framework is the REF-based capability appraisal for process improvement (REF Appraisal). The REF Appraisal is designed to objectively review an organization against the benchmark of the CERT Resiliency Engineering Framework processes and practices. It can be used internally by an organization to improve its processes for managing operational resiliency or applied externally to determine the capability of a third-party organization. Either way, the appraisal provides a foundation for long-term process improvement.

What distinguishes a REF Appraisal?

Unlike assessments, audits, or evaluations in the security, business continuity, or IT operations domain, the REF Appraisal is designed to help the organization understand its level of capability through an examination of process maturity. In other words, the REF Appraisal determines not only whether the organization is doing the right things right now, but whether it is capable of sustaining an acceptable level of performance during times of stress and over the long run as risk environments continue to evolve and change. In constrast, most practice-based assessments focus on how well the organization meets the prescribed practice at a point in time, which fails to tell the organization whether it can sustain an adequate level of performance after the assessment is over.

Why should an organization care about a REF Appraisal?

Managing operational resiliency is a challenge because it involves managing operational risk in complex environments. Because of technology and other factors, these environments (and corresponding threats and vulnerabilities) are continuously changing. An organization must be prepared to not only address the events it knows about, but also the events that will occur in the future. By considering the organization's process capability and maturity, the REF Appraisal tells the organization how well it is prepared to manage a changing risk environment.

Why is an expression of process capability and maturity important to an organization?

Organizations with lower levels of process capability and maturity tend to do things in an ad-hoc way, dependent on people and prone to heroics and fortunate circumstances. As process capability and maturity improves, the organization moves away from "getting lucky" to performing with an emphasis on predictable, repeatable, and consistent results. In other words, organizations with higher levels of process capability and maturity do things in a way that improves their potential for managing operational resiliency regardless of the risk environment. Knowing the organization's current level of process capability and maturity is a way to determine where on this scale the organization fits.

What will an organization learn in a REF Appraisal?

The REF Appraisal provides the organization insight into

• the current state of its processes for managing operational resiliency
• its process strengths and weaknesses
• opportunities for improvement relative to the REF model
• the potential value of improvements
• ways to prioritize improvement activities

What is the scope of a REF Appraisal?

Because the REF model allows for appraisals of individual capability areas, the scope of the REF Appraisal involves determining

• which REF capability areas will be included in the appraisal—the model scope
• which parts or levels of the organization will be appraised (the enterprise, a line of business, one or more operational units, a specific project, etc.)—the organizational scope

Both the model and the organizational scopes are determined during an appraisal workshop activity that considers criteria such as the organization's objectives for performing the appraisal, process improvement objectives, resiliency strategy, regulatory and compliance environment, and specific threats or risks that may be of concern.

What and who is involved in a REF Appraisal?

The appraisal is performed by appraisers who have been trained in the REF and the accompanying appraisal methodology and who are authorized by the Software Engineering Institute to perform the appraisal. The level and extent of involvement by the organization's personnel depend on the scope of the appraisal. The organization's personnel will assist in the appraisal by participating in interviews, supplying process artifacts (such as documents), facilitating process observation, and analyzing findings and drawing conclusions. Because the organization owns the appraisal results, the results can be a valuable learning tool for those involved in the appraisal and responsible for process improvement.

What can an organization do with the results of a REF Appraisal?

In addition to using the results to improve processes and set performance targets, the results of a REF Appraisal can be used to convey the organization's competency for managing operational resiliency. For the organization's customers, this may communicate confidence in creating a resilient partnership that can survive business and operational events. And, as appraisals are performed throughout the organization's core industry, the appraisal results can be used to benchmark the organization's performance against peers.

How can an organization make a business case for investing in a REF Appraisal?

A REF Appraisal is an investment in the organization's long-term ability to manage operational resiliency. It establishes the foundation for improving processes and helps the organization to efficiently focus on those areas that matter most, which, in turn, translates to less effort wasted on unnecessary improvements. In addition, improving processes can eliminate redundancies, streamline compliance activities, and increase efficiency in other ways. Some organizations may even be able to convince their insurers to reduce rates because of their demonstrated ability to manage risk. And, if the organization is a service provider to other organizations, the appraisal may help the organization increase its business and ability to secure contracts because it has an objective means to communicate its process capability and maturity with respect to resiliency.

How does an organization initiate a REF Appraisal?

CERT has approved REF appraisers who can work with you to establish an appraisal scope, perform the appraisal, and document and present appraisal results. We can even help you prioritize process improvement areas, develop action and implementation plans, and embark on an improvement process.

To learn more about REF Appraisals, become a licensed REF appraiser, or arrange for CERT to perform a REF Appraisal in your organization, contact Joe McLeod at jmcleod@sei.cmu.edu.