Air Traffic Control: Immature Software Acquisition Processes Increase FAA
System Acquisition Risks (Chapter Report, 03/21/97, GAO/AIMD-97-47).

Pursuant to a congressional request, GAO reviewed the Federal Aviation
Administration's (FAA) air traffic control (ATC) modernization software
acquisition efforts, focusing on the: (1) maturity of FAA's ATC
modernization software acquisition processes; and (2) steps/actions FAA
has underway or planned to improve these processes, including any
obstacles that may impede FAA's progress.

GAO noted that: (1) because of the number and severity of FAA ATC
modernization software acquisition process weaknesses, FAA did not fully
satisfy any of the seven key process areas (KPA) necessary to achieve
the "repeatable" level of process maturity; (2) as a result, its
processes for acquiring software, the most costly and complex component
of ATC systems, are ad hoc, sometime chaotic, and not repeatable across
projects; (3) in addition, serious process weaknesses prevented FAA from
satisfying the one KPA specified under the Software Engineering
Institute's "defined" maturity level; (4) while FAA showed process
strengths, primarily in the solicitation and evaluation KPAs, GAO found
extensive weaknesses in these and the remaining six KPAs; (5) some of
these weaknesses were systemic, recurring in each of the KPAs; (6) for
example, no software project teams measured or reported to management on
the status of activities performed, and management never verified that
critical activities were being done; (7) these types of problems are
some of the reasons for FAA's frequent failures to deliver promised ATC
system capabilities on time and within budget; (8) FAA has stated its
commitment to increasing ATC modernization process maturity; (9)
however, despite 4 years of activity in this area, FAA lacks an
effective management approach for improving software acquisition
processes; (10) currently, the Software Engineering Process Group (SEPG)
is responsible for process improvement, but the SEPG has neither
organizational nor budgetary authority over the product teams that
acquire software, and, therefore, cannot effectively implement or
enforce process change; (11) instead, it can only recommend and
encourage change; (12) additionally, FAA does not have an effective plan
to correctly target and prioritize improvements and measure improvement
progress; (13) in the absence of this plan, it has initiated a "hodge
podge" of software acquisition improvement efforts without any
analytical justification; and (14) as a result, FAA's process
improvement activities have yet to produce more repeatable, better
defined, more disciplined software acquisition processes.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  AIMD-97-47
     TITLE:  Air Traffic Control: Immature Software Acquisition 
             Processes Increase FAA System Acquisition Risks
      DATE:  03/21/97
   SUBJECT:  ADP procurement
             Computer software
             Air traffic control systems
             Strategic information systems planning
             Computer software verification and validation
             Requirements definition
IDENTIFIER:  Software Capability Maturity Model
             FAA Automated Radar Terminal System
             FAA Display System Replacement
             FAA Voice Switching and Control System
             FAA National Airspace System
             FAA Air Traffic Control Modernization Program
             FAA Advanced Automation System
             FAA Integrated Terminal Weather System
             FAA Air Route Surveillance Radar-4
             FAA Initial Sector Suite System
             FAA Aviation Weather Observing System
             NAVSTAR Global Positioning System
             FAA Weather and Radar Processor
             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                    <info@www.gao.gov>                        **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************


Cover
================================================================ COVER


Report to the Chairman, Subcommittee on Transportation, Committee on
Appropriations, House of Representatives

March 1997

AIR TRAFFIC CONTROL - IMMATURE
SOFTWARE ACQUISITION PROCESSES
INCREASE FAA SYSTEM ACQUISITION
RISKS

GAO/AIMD-97-47

Air Traffic Control

(511487)


Abbreviations
=============================================================== ABBREV

  AAR - Office of Aviation Research
  AAS - Advanced Automation System
  ACT - William J.  Hughes Technical Center
  AIMD - Accounting and Information Management Division
  AIT - Office of Information Technology
  AND - Office of Communication, Navigation, and Surveillance Systems
  ARA - Associate Administrator for Research and Acquisitions
  ARINC - Aeronautical Radio Incorporated
  ARTS - Automated Radar Terminal System
  ASD - Office of System Architecture and Investment Analysis
  ASU - Office of Acquisitions
  ATC - air traffic control
  ATCSCC - Air Traffic Control System Command Center
  ATS - Associate Administrator for Air Traffic Services
  AUA - Office of Air Traffic Systems Development
  CIO - Chief Information Officer
  CMM - Capability Maturity Model
  COTS - commercial, off-the-shelf
  DSR - Display System Replacement
  FAA - Federal Aviation Administration
  GAO - General Accounting Office
  IPT - Integrated Product Team
  ISSS - Initial Sector Suite System
  KPA - key process area
  NAS - National Airspace System
  NDI - non-development item
  NIMS - NAS Infrastructure Management System
  SA-CMM - Software Acquisition Capability Maturity Model
  SE-CMM - Systems Engineering Capability Maturity Model
  SCE - Software Capability Evaluation
  SEEC - Software Engineering Executive Committee
  SEI - Software Engineering Institute
  SEPG - Software Engineering Process Group
  SW-CMM - Capability Maturity Model for Software
  TRACON - Terminal Radar Approach Control
  VSCS - Voice Switching and Control System
  WARP - Weather and Radar Processor

Letter
=============================================================== LETTER


B-271531

March 21, 1997

The Honorable Frank R.  Wolf
Chairman, Subcommittee on
 Transportation
Committee on Appropriations
House of Representatives

Dear Mr.  Chairman: 

This report responds to your request that we review the Federal
Aviation Administration's (FAA) air traffic control modernization
software acquisition maturity and improvement activities.  FAA plans
to spend billions of dollars replacing its existing air traffic
control systems, but has a history of performing poorly when
acquiring these software-intensive systems.  We found that FAA's
software acquisition processes are immature, and are making
recommendations to the Secretary of Transportation for strengthening
them. 

We are sending copies of this report to the Secretary of
Transportation, the Director of the Office of Management and Budget,
the Administrator of the Federal Aviation Administration, and other
congressional committees.  We will also make copies available to
other interested parties upon request.  If you have questions or wish
to discuss the issues in this report, please contact me at (202)
512-6412.  Major contributors to this report are listed in appendix
II. 

Sincerely yours,

Dr.  Rona B.  Stillman
Chief Scientist for Computers
 and Telecommunications


EXECUTIVE SUMMARY
============================================================ Chapter 0


   PURPOSE
---------------------------------------------------------- Chapter 0:1

The Federal Aviation Administration (FAA) is modernizing the air
traffic control (ATC) systems upon which it will rely to ensure safe,
orderly, and efficient air travel well into the 21st century.  Since
software is the most expensive and complex component of these
systems, FAA must use defined and disciplined processes when it
acquires software. 

Recognizing software's growing importance and prevalence in ATC
systems, the Chairman, Subcommittee on Transportation and Related
Agencies, House Committee on Appropriations, asked GAO to determine
(1) the maturity of FAA's ATC modernization software acquisition
processes, and (2) the steps/actions FAA has underway or planned to
improve these processes, including any obstacles that may impede
FAA's progress. 


   BACKGROUND
---------------------------------------------------------- Chapter 0:2

To accommodate forecasted growth in air traffic and replace aging
equipment, FAA embarked on an ambitious ATC modernization program in
1981.  FAA estimates that it will spend about $20 billion to replace
and modernize software-intensive ATC systems between 1982 and 2003. 
Our work over the years has chronicled many FAA failures in meeting
ATC projects' cost, schedule, and performance goals, largely because
of software-related problems.  As a result of these failures as well
as the tremendous cost, complexity, and mission criticality of FAA's
ATC modernization program, we designated the program as a high-risk
information technology initiative in our 1995 and 1997 report series
on high-risk programs.\1

Software quality is governed largely by the quality of the processes
involved in developing or acquiring, and maintaining it.  Carnegie
Mellon University's Software Engineering Institute (SEI), recognized
for its expertise in software processes, has developed models and
methods that define and determine organizations' software process
maturity.  Together, they provide a logical framework for baselining
an organization's current process capabilities (i.e., strengths and
weaknesses) and providing a structured plan for incremental process
improvement. 

Using SEI's software acquisition capability maturity model
(SA-CMM),\2 SEI's software capability evaluation method, and SA-CMM
authors as consultants, GAO staff trained at SEI evaluated FAA's ATC
modernization software acquisition maturity in the seven key process
areas (KPA) necessary to attain a "repeatable" level of process
capability, and one KPA associated with the "defined" level of
process maturity.\3 Repeatability ensures that an organization has
the necessary process discipline in place to repeat earlier successes
on projects in similar domains.  Repeatable processes are at the
second level on SEI's five-level scale of process maturity. 
Organizations that do not satisfy the requirements for the
"repeatable" level are by default judged to be at the "initial" level
of maturity, meaning that their processes are ad hoc, sometimes even
chaotic, with few of the processes defined and success dependent
mainly on the heroic efforts of individuals.  The one KPA associated
with the third level of process maturity, which is called the
"defined" level, is acquisition risk management.  It was included
because many software experts consider it to be a very important
process area. 

As part of its evaluation, GAO examined five ongoing ATC
modernization projects selected by FAA.\4 These were the Automated
Radar Terminal System, Display System Replacement, National Airspace
System Infrastructure Management System, Voice Switching and Control
System, and the Weather and Radar Processor.  (See chapter 1 of this
report for more detailed information on GAO's evaluation scope and
methodology.)


--------------------
\1 High-Risk Series:  An Overview (GAO/HR-95-1, Feb.  1995);
High-Risk Series:  Information Management and Technology
(GAO/HR-97-9, Feb.  1997). 

\2 We used a draft version of the model for our evaluation (version
00.03, dated April 1996).  The first published version of the model
was released on October 1996, after we performed our evaluation. 
According to the model's authors, the published version differed only
editorially from the draft we used. 

\3 The seven KPAs relating to the repeatable level are software
acquisition planning, solicitation, requirements development and
management, project office management, contract tracking and
oversight, evaluation, and transition and support. 

\4 GAO asked FAA to choose five projects that are:  (1) major efforts
with large software acquisition components, (2) managed by different
FAA product teams, (3) at different life cycle stages, and (4) among
FAA's best managed. 


   RESULTS IN BRIEF
---------------------------------------------------------- Chapter 0:3

Because of the number and severity of FAA ATC modernization software
acquisition process weaknesses, FAA did not fully satisfy any of the
seven KPAs necessary to achieve the "repeatable" level of process
maturity.  As a result, its processes for acquiring software, the
most costly and complex component of ATC systems, are ad hoc,
sometimes chaotic, and not repeatable across projects.  In addition,
serious process weaknesses prevented FAA from satisfying the one KPA
specified under SEI's "defined" maturity level.  While FAA showed
process strengths, primarily in the solicitation and evaluation
(i.e., testing) KPAs,\5 GAO found extensive weaknesses in these and
the remaining six KPAs (i.e., software acquisition planning,
requirements development and management, project office management,
contract tracking and oversight, transition and support, and
acquisition risk management).\6 Some of these weaknesses were
systemic, recurring in each of the KPAs.  For example, no software
project teams measured or reported to management on the status of
activities performed, and management never verified that critical
activities were being done.  These types of problems are some of the
reasons for FAA's frequent failures to deliver promised ATC system
capabilities on time and within budget. 

FAA has stated its commitment to increasing ATC modernization process
maturity.  However, despite 4 years of activity in this area, FAA
lacks an effective management approach for improving software
acquisition processes.  Currently, the Software Engineering Process
Group (SEPG) is responsible for process improvement; but the SEPG has
neither organizational nor budgetary authority over the product teams
that acquire software, and, therefore, cannot effectively implement
or enforce process change.  Instead, it can only recommend and
encourage change.  Additionally, FAA does not have an effective plan
to correctly target and prioritize improvements and measure
improvement progress.  In the absence of this plan, it has initiated
a "hodge podge" of software acquisition improvement efforts without
any analytical justification.  As a result, FAA's process improvement
activities have yet to produce more repeatable, better defined, more
disciplined software acquisition processes. 


--------------------
\5 According to the SA-CMM, solicitation is the process of ensuring
that award is made to the contractor most capable of satisfying the
specified requirements, and evaluation is the process of determining
that acquired software products and services satisfy contract
requirements prior to acceptance. 

\6 According to the SA-CMM, software acquisition planning is the
process for ensuring that reasonable planning for all elements of the
software acquisition occur; requirements development and management
is the process for establishing an unambiguous and agreed upon set of
software requirements; project office management is the process for
effective and efficient management of project office activities;
contract tracking and oversight is the process of ensuring that
contractor activities, products, and services satisfy contract
requirements; transition and support is the process of transferring
acquired software products to the eventual support organization; and
acquisition risk management is the process of identifying software
risks early and adjusting the acquisition strategy to mitigate those
risks. 


   PRINCIPAL FINDINGS
---------------------------------------------------------- Chapter 0:4


      ATC MODERNIZATION SOFTWARE
      ACQUISITION PROCESSES ARE
      IMMATURE
-------------------------------------------------------- Chapter 0:4.1

To attain a given SEI-defined maturity level, an organization must
satisfy the key practices for the KPAs associated with that level. 
FAA's ATC modernization organization had too many weaknesses to
satisfy any of the "repeatable" KPAs (i.e., software acquisition
planning, solicitation, requirements development and management,
project office management, contract tracking and oversight,
evaluation, and transition and support), nor does it satisfy the
acquisition risk management KPA from the "defined" or third maturity
level. 

For FAA to satisfy any of these eight KPAs, it must eliminate the key
practice weaknesses identified in this report.\7 Each practice that
is performed effectively constitutes a strength, and each practice
not performed or performed poorly constitutes a weakness.  While
FAA's ATC modernization has some strengths, it has more weaknesses. 
Table 1 tallies these strengths on the five projects that GAO
evaluated.  In summary, of the total number of KPA practices rated,
38 percent constituted strengths, 50 percent were weaknesses, and 12
percent were observations.  An observation indicates that the
evidence was inconclusive and did not clearly support a determination
of either strength or weakness. 



                                Table 1
                
                  Collective Number of KPA Strengths,
                Weaknesses, and Observations on the Five
                                Projects

                                                             Number of
                                     Number of   Number of  observatio
Key Process Area                     strengths  weaknesses          ns
----------------------------------  ----------  ----------  ----------
Software acquisition planning               16          37           7
Solicitation                                36          28          14
Requirements development and                17          35           6
 management
Project office management                   26          35           6
Contract tracking and oversight             26          32           6
Evaluation                                  43          21           8
Transition and support                      27          32           8
Acquisition risk management                 16          46           7
======================================================================
Totals                                     207         266          62
----------------------------------------------------------------------
Additionally, GAO found that while the five projects varied as to
practice strengths, weaknesses, and observations under three of the
five "common features" or practice groupings (commitment to perform,
ability to perform, and activities performed), the projects were
consistently weak in all practices under the remaining two groupings
(measurement and analysis and verifying implementation).  As a
result, software project teams and FAA management consistently lack
reliable information on project team performance. 


--------------------
\7 SEI groups each of its KPA practices into one of five "common
features" or practice categories.  These are "commitment to perform,
ability to perform, activities performed, measurement and analysis,
and verifying implementation."


      FAA'S APPROACH FOR IMPROVING
      ATC MODERNIZATION SOFTWARE
      ACQUISITION PROCESSES IS NOT
      EFFECTIVE
-------------------------------------------------------- Chapter 0:4.2

To be effective, the FAA organization responsible for software
acquisition process improvement must have (1) organizational and/or
budgetary authority over the ATC modernization units acquiring the
software; and (2) an effective plan to guide improvement efforts and
measure progress on each.  The FAA organizational entity currently
responsible for ATC modernization software acquisition process
improvement, the SEPG, has neither.  As a result, little progress has
been made over the last 4 years in instituting definition and
discipline into ATC modernization software acquisition processes. 

The SEPG is a multilevel committee structure chaired by a member of
FAA's Chief Information Officer's (CIO) staff.  The SEPG is directed
by the Software Engineering Executive Committee, which is chaired by
the head of the ATC modernization program.  The SEPG has no authority
to implement and enforce process change.  Consequently, it can only
attempt to encourage and persuade software acquirers to establish and
follow defined and disciplined software acquisition processes. 

The SEPG and its predecessors have advocated and initiated a
collection of efforts intended to strengthen ATC modernization
software-related processes, including software acquisition processes. 
However, there is no analytical basis for the focus, content, timing,
and interrelationships of these efforts.  Specifically, the efforts
(1) are not based on any assessment of current software acquisition
process strengths and weaknesses; and (2) are not detailed in a
formal plan that specifies measurable goals, objectives, milestones,
and needed resources, prioritizes efforts, fixes responsibility and
accountability, and defines metrics for measuring progress.  Instead,
these efforts were undertaken with no sound analytical basis and,
rather than being part of a comprehensive plan, are discussed in
general terms without detail and specificity in briefing documents,
minutes of meetings, and working group recommendations.  While the
SEPG is now taking steps to establish the analytical basis needed to
formulate a comprehensive software process improvement plan, that
plan does not yet exist, and no schedule has been established for
completing it. 


   RECOMMENDATIONS
---------------------------------------------------------- Chapter 0:5

Given the importance and the magnitude of information technology at
FAA, GAO reiterates its earlier recommendation that a CIO management
structure similar to the department-level CIOs as prescribed in the
Clinger-Cohen Act of 1996 be established for FAA.\8

To improve FAA's software acquisition capability for its ATC
modernization, GAO recommends that the Secretary of Transportation
direct the FAA Administrator to: 

  -- assign responsibility for software acquisition process
     improvement to FAA's CIO;

  -- provide FAA's CIO with the authority needed to implement and
     enforce ATC modernization software acquisition process
     improvement;

  -- require the CIO to develop and implement a formal plan for ATC
     modernization software acquisition process improvement that is
     based on the software capability evaluation results contained in
     this report and specifies measurable goals and time frames,
     prioritizes initiatives, estimates resource requirements, and
     assigns roles and responsibilities;

  -- allocate adequate resources to ensure that planned initiatives
     are implemented and enforced; and

  -- require that, before being approved, every ATC modernization
     acquisition project have software acquisition processes that
     satisfy at least SA-CMM level 2 requirements. 


--------------------
\8 Air Traffic Control:  Complete and Enforced Architecture Needed
for FAA Systems Modernization (GAO/AIMD-97-30, February 3, 1997). 


   AGENCY COMMENTS AND GAO'S
   EVALUATION
---------------------------------------------------------- Chapter 0:6

In its written comments on a draft of this report, the Department of
Transportation recognized the importance of mature software
acquisition processes, agreed that FAA's processes are insufficiently
mature, acknowledged that FAA process improvement activities have yet
to produce greater software acquisition process discipline, and
reaffirmed FAA's commitment to improving its software acquisition
capabilities using the SA-CMM.  However, the Department did not state
what, if any, specific action it would take on GAO's recommendations. 
Additionally, it took the positions that (1) the SA-CMM by itself is
inadequate to evaluate ATC system acquisition capabilities, is too
new to use as an authoritative source of guidance, and "may" have
been misapplied by GAO, (2) the report does not sufficiently
recognize FAA's process improvement organization and progress nor the
difficulties and time required to affect process improvement change,
(3) the SEPG, which is FAA's designated agent for software
acquisition process change, is organized as the Department
"understands" other SEPGs to be organized, and (4) the report "leads
the reader to erroneously conclude that the five programs reviewed
are in trouble" relative to attainment of cost and schedule goals. 

None of these positions are valid.  First, the SA-CMM, like the
SW-CMM (another SEI software-specific capability maturity model),
focuses on software because it is widely recognized as the most
difficult and costly component of modern computer systems; the one
most frequently associated with late deliveries, cost overruns, and
performance shortfalls; and the one in greatest need of special
management attention.  Further, while the SA-CMM is relatively new,
the processes it requires are well established, experience-based
tenets of effective software acquisition that are widely supported
throughout industry and government.  Moreover, GAO applied the SA-CMM
at FAA properly and with extraordinary diligence:  Every member of
the evaluation team was trained at SEI; the team leader was certified
by SEI as a lead evaluator; and three SEI professionals, including
two authors of the SA-CMM, participated in the evaluation and
concurred with every practice determination (e.g., strength,
weakness). 

Second, FAA's many software acquisition process improvement
activities were undertaken without assessing current software
acquisition process strengths and weaknesses, and were not part of a
comprehensive plan for process improvement.  Therefore, FAA had no
analytical basis for deciding what improvement activities to
initiate, or what priorities to assign them.  Further, although FAA
began drafting a plan during the course of GAO's evaluation, it has
no schedule for completing it.  In describing FAA's progress to date
in improving its processes, the report delineates a wide array of FAA
process improvement activities, but distinguishes these activities
from actual progress.  In fact, after 4 years of activity, FAA could
not point to a single case in which it had instituted a more
disciplined software acquisition process.  Since SEI published
statistics show that the median time to improve from software
development CMM level 1 to level 2 is 26 months, and from level 2 to
level 3 is 17 months, it is entirely reasonable to expect FAA to be
able to demonstrate some improvement in its processes after 4 years. 

Third, the issue is not whether FAA's SEPG is organized as the
Department "understands" other SEPGs to be organized, but whether the
SEPG, or any FAA organizational entity responsible for implementing
and enforcing software acquisition process change, has the authority
needed to accomplish the task.  Currently, no organizational entity
in FAA has the requisite authority. 

Last, the report addresses the maturity of FAA's software acquisition
processes and concludes that these processes are ad hoc and
undisciplined, reducing the probability that software-intense ATC
modernization projects will consistently perform as intended and be
delivered on schedule and within budget.  The report does not address
the overall status of the projects covered by GAO's review, and,
therefore, provides no basis for drawing conclusions about the
projects' overall cost or schedule performance. 

The Department's comments and GAO's evaluation of them are presented
in greater detail in chapter 11 of this report. 


INTRODUCTION
============================================================ Chapter 1

The Federal Aviation Administration's (FAA) primary mission is to
ensure safe, orderly, and efficient air travel in the national
airspace.  FAA's ability to fulfill this mission depends on the
adequacy and reliability of the nation's air traffic control (ATC)
system, a vast network of computer hardware, software, and
communications equipment.\1 The ATC system, however, is being
strained by aging equipment, much of which is 1960's technology, and
growing air traffic.  This growth should continue as the number of
passengers traveling on U.S.  airlines is expected to increase by 38
percent between 1995 and 2003, from about 580 million to nearly 800
million. 

To accommodate the forecasted growth in air traffic and to relieve
the problems of the aging ATC system, FAA embarked on an ambitious
ATC modernization program in 1981.  FAA estimates that it will spend
about $20 billion to replace and modernize software-intensive ATC
systems between 1982 and 2003.  Our work over the years has
chronicled many FAA failures in meeting ATC projects' cost, schedule,
and performance goals.\2 As a result of these failures as well as the
tremendous cost, complexity, and mission criticality of FAA's ATC
modernization program, we designated the program as a high-risk
information technology initiative in our 1995 and 1997 report series
on high-risk programs.\3


--------------------
\1 The ATC system is a major component of the National Airspace
System (NAS). 

\2 Air Traffic Control:  Status of FAA's Modernization Program
(GAO/RCED-95-175FS, May 26, 1995); Air Traffic Control:  Status of
FAA's Modernization Program (GAO/RCED-94-167FS, Apr.  15, 1994); Air
Traffic Control:  Status of FAA's Modernization Program
(GAO/RCED-93-121FS, Apr.  16, 1993). 

\3 High-Risk Series:  An Overview (GAO/HR-95-1, Feb.  1995);
High-Risk Series:  Information Management and Technology
(GAO/HR-97-9, Feb.  1997). 


   OVERVIEW OF ATC
---------------------------------------------------------- Chapter 1:1

Automated information processing and display, communication,
navigation, surveillance, and weather resources permit air traffic
controllers to view key information, such as aircraft location,
aircraft flight plans, and prevailing weather conditions, and to
communicate with pilots.  These resources reside at, or are
associated with, several ATC facilities--air traffic control towers,
terminal radar approach control (TRACON) facilities, air route
traffic control centers (en route centers), flight service stations,
and the Air Traffic Control System Command Center (ATCSCC).  These
facilities' ATC functions are described below. 

  -- Airport towers control aircraft on the ground and before landing
     and after take-off when they are within about 5 nautical miles
     of the airport, and up to 3,000 feet above the airport.  Air
     traffic controllers rely on a combination of technology and
     visual surveillance to direct aircraft departures and
     approaches, maintain safe distances between aircraft, and
     communicate weather-related information, clearances, and other
     instructions to pilots and other personnel. 

  -- Approximately 180 TRACONs sequence and separate aircraft as they
     approach and leave busy airports, beginning about 5 nautical
     miles and ending about 50 nautical miles from the airport, and
     generally up to 10,000 feet above the airport, where en route
     centers' control begins. 

  -- Twenty en route centers control planes over the continental
     United States in transit and during approaches to some airports. 
     Each en route center handles a different region of airspace,
     passing control from one to another as respective borders are
     reached until the aircraft reaches TRACON airspace.  En route
     center controlled airspace usually extends above 18,000 feet for
     commercial aircraft.  En route centers also handle lower
     altitudes when dealing directly with a tower, or when agreed
     upon with a TRACON. 

  -- Two en route centers--Oakland and New York--also control
     aircraft over the ocean.  Controlling aircraft over oceans is
     radically different from controlling aircraft over land because
     radar surveillance only extends 175 to 225 miles offshore. 
     Beyond the radars' sight, controllers must rely on periodic
     radio communications through a third party--Aeronautical Radio
     Incorporated (ARINC), a private organization funded by the
     airlines and FAA to operate radio stations--to determine
     aircraft locations. 

  -- About 90 flight service stations provide pre-flight and
     in-flight services, such as flight plan filing and weather
     report updates, primarily for general aviation aircraft. 

See figure 1.1 for a visual summary of air traffic control over the
continental United States and oceans. 

   Figure 1.1:  Summary of ATC
   Over the Continental United
   States and Oceans

   (See figure in printed
   edition.)


   THE ATC MODERNIZATION PROGRAM
   IS COMPLEX, COSTLY, AND
   HISTORICALLY PROBLEMATIC
---------------------------------------------------------- Chapter 1:2

FAA faced two problems in continuing to fulfill its mission to ensure
safe, orderly, and efficient air travel in the national airspace. 
First, the ATC system of the late 1970s was a blend of several
generations of automated and manual equipment, much of it
labor-intensive and obsolete.  Second, air traffic was projected to
increase dramatically as a result of airline deregulations of the
late 1970s.  FAA recognized that it could increase ATC operating
efficiency by increasing automation.  It also anticipated that
meeting the demand safely and efficiently would require improved and
expanded services, additional facilities and equipment, improved work
force productivity, and the orderly replacement of aging equipment. 
Accordingly, in December 1981, FAA initiated its plan to modernize,
automate, and consolidate its enormous ATC system infrastructure by
the year 2000.  In doing so, it chose to acquire new ATC systems by
contracting for systems development services from vendors rather than
building new ATC systems in-house. 

This ambitious modernization program includes the acquisition of new
surveillance, data processing, navigation, and communication
equipment in addition to new facilities and support equipment. 
Totaling over 200 separate projects, the modernization is estimated
to cost over $34 billion through the year 2003.  Software-intensive
ATC systems make up a large portion of this total, accounting for 169
projects costing $20.7 billion.  The Congress will have provided FAA
with approximately $14.7 billion of the $20.7 billion through fiscal
year 1997.  Many of these projects, for example the Display System
Replacement and the Voice Switching and Control System, each involve
the acquisition of over a million lines of code.  Moreover, because
the software must operate in a real-time environment in which human
life is at stake, it must be fault tolerant, meaning that it must be
able to monitor its own execution and recover from failures without
losing any data. 

Over the past 15 years, FAA's modernization projects have experienced
substantial cost overruns, lengthy schedule delays, and significant
performance shortfalls.  To illustrate, the long-time centerpiece of
this modernization program--the Advanced Automation System (AAS)--was
restructured in 1994 after estimated costs tripled from $2.5 billion
to $7.6 billion and delays in putting significantly
less-than-promised system capabilities into operation were expected
to run 8 years or more over original estimates.  Similarly, increases
in costs for three other ATC projects\4 have ranged from 51 to 511
percent, and schedule delays have averaged almost 4 years.  For
example, the per-unit cost estimate for the Voice Switching and
Control System increased 511 percent, and the first site
implementation was delayed 6 years from the original estimate. 

AAS and other ATC projects have also experienced shortfalls in
performance.  For example, the critical Initial Sector Suite System
component of AAS, which was intended to replace controllers'
workstations at en route centers, faced so many technical problems
that its functionality was severely scaled back.  In addition,
difficulties in developing the Air Route Surveillance Radar-4
software and integrating it with other ATC systems delayed its
implementation for years. 


--------------------
\4 The three projects and their respective percentage increase in
unit costs are the Voice Switching and Control System (511 percent),
the Integrated Terminal Weather System (129 percent), and the
Aviation Weather Observing System (51 percent). 


   ATC MODERNIZATION NOW
   PROCEEDING UNDER A NEW
   ACQUISITION MANAGEMENT SYSTEM
---------------------------------------------------------- Chapter 1:3

Because of FAA's contention that many of its modernization problems
were rooted in the Federal Acquisition System, the Congress enacted
legislation in October 1995 that exempted FAA from most federal
procurement and personnel laws and regulations and directed FAA to
develop and implement a new acquisition system that would address the
unique needs of the agency.\5 At a minimum, the system was to provide
for more timely and cost-effective acquisitions.  On April 1, 1996,
in response to the Act, the FAA Administrator began implementation of
a new acquisition management system. 

The new system is intended to reduce the time and cost to field new
products and services by introducing a new investment management
system that spans the investments' entire life cycles, a new
procurement system that provides flexibility in selecting and
managing contractors, and organizational learning and cultural reform
that supports the new investment management and procurement systems. 

This high-level policy promulgated by the new acquisition management
system is intended to be supplemented by guidelines in three areas: 
software/systems acquisition, facilities acquisition, and services
acquisition.  These guidelines will be available to FAA staff via the
Internet and were scheduled to be online by October 1, 1996.  As of
February 1, 1997, these guidelines were still in draft form and not
available to FAA staff. 


--------------------
\5 Department of Transportation and Related Agencies Appropriations
Act 1996, P.L.  No.  104-50, sec.  348, 109 Stat.  436, 460 (1995). 


   FAA ORGANIZATIONS RESPONSIBLE
   FOR ATC SYSTEMS ACQUISITION AND
   MAINTENANCE
---------------------------------------------------------- Chapter 1:4

Two major FAA organizations play key roles in the modernization of
ATC systems--the Office of the Associate Administrator for Research
and Acquisitions (ARA) and the Office of the Associate Administrator
for Air Traffic Services (ATS).  Briefly, ARA is responsible for
acquiring ATC systems, while ATS is responsible for operating and
maintaining ATC systems.  Cross-functional integrated product teams
(IPT) residing in ARA are responsible for specific ATC system
acquisition projects. 

ARA manages ATC modernization research and development and
acquisition activities.  According to the Associate Administrator for
ARA, only about one-half of the total ATC systems development budget
is spent by ARA, while the other one-half is spent by ATS
implementing system enhancements.  Within ARA, two groups are
responsible for acquiring systems, while other groups handle
cross-cutting management functions, such as budget formulation and
program evaluation.  These two groups are the Office of Air Traffic
Systems Development (AUA) and the Office of Communication,
Navigation, and Surveillance Systems (AND). 

Five IPTs reside in AUA and are organized by ATC business areas
(i.e., en route, terminal, weather and flight service, air traffic
management, oceanic), and five IPTs reside in AND and are organized
by ATC functional areas (i.e., infrastructure, communications,
surveillance, Global Positioning System/navigation,
aircraft/avionics).  IPTs are responsible for research, development,
and acquisition as well as for ensuring that new equipment is
delivered, installed, and working properly.  For example, the en
route IPT comprises product teams for the Display Channel Complex
Rehost, the Display System Replacement, the Voice Switching and
Control System, and several other en route systems.  Each IPT
includes systems and specialty engineers, logistics personnel,
testing personnel, contract personnel, and lawyers as well as
representatives from the organizations responsible for operating and
maintaining the ATC equipment.  The organization chart below shows
the structure of the ARA organization. 

   Figure 1.2:  ARA Organization
   Chart

   (See figure in printed
   edition.)


   OBJECTIVES, SCOPE, AND
   METHODOLOGY
---------------------------------------------------------- Chapter 1:5

The Chairman, Subcommittee on Transportation and Related Agencies,
House Committee on Appropriations, requested that we review FAA's
ability to acquire software for ATC systems.  Our objectives were to
determine (1) the maturity of FAA's ATC modernization software
acquisition processes; and (2) the steps/actions FAA has underway or
planned to improve these processes, and any obstacles that may impede
FAA's improvement actions. 

To determine FAA's software acquisition process maturity, we applied
the Software Engineering Institute's Software Acquisition Capability
Maturity Model (SA-CMM)\6 and its Software Capability Evaluation
(SCE) method.  SEI's expertise in software process assessment is
accepted throughout the industry.  Our evaluators were all
SEI-trained software specialists.  In addition, we employed SEI
consultants, two of whom are authors of the model, as advisors to
ensure proper application of the model. 

The SA-CMM ranks organizational maturity according to five levels
(see figure 1.3).  Maturity levels 2 through 5 require the verifiable
existence and use of certain software acquisition processes, known as
key process areas (KPA).  According to the SEI, an agency that has
these acquisition processes in place is in a much better position to
successfully acquire software than an organization that does not have
these processes in place.  We evaluated FAA's software acquisition
processes against all level 2 KPAs and one level 3 KPA (see table
1.1).  We included one level 3 KPA--acquisition risk
management--because it is considered by software experts to be a very
important process area. 

   Figure 1.3:  SA-CMM Levels and
   Descriptions

   (See figure in printed
   edition.)



                               Table 1.1
                
                SA-CMM KPAs Used to Assess FAA Software
                          Acquisition Maturity

SA-CMM Level 2 Key
process areas       Description
------------------  --------------------------------------------------
Software            Ensuring that reasonable planning for the software
acquisition         acquisition is conducted and that all elements of
planning            the project are included.

Solicitation        Ensuring that award is made to the contractor most
                    capable of satisfying the specified requirements.

Requirements        Establishing a common and unambiguous definition
development and     of software acquisition requirements understood by
management          the acquisition team, system user, and the
                    contractor.

Project office      Managing the activities of the project office and
management          supporting contractor(s) to ensure a timely,
                    efficient, and effective software acquisition.

Contract tracking   Ensuring that the software activities under
and oversight       contract are being performed in accordance with
                    contract requirements, and that products and
                    services will satisfy contract requirements.

Evaluation          Determining that the acquired software products
                    and services satisfy contract requirements prior
                    to acceptance and transition to support.

Transition and      Providing for the transition of the software
support             products being acquired to their eventual support
                    organization.

CMM Level 3         Description
Key process area

Acquisition risk    Identifying risks as early as possible, adjusting
management          acquisition strategy to mitigate those risks, and
                    developing and implementing a risk management
                    process as an integral part of the acquisition
                    process.
----------------------------------------------------------------------
As established by the model, each KPA contains five common attributes
that indicate whether the implementation and institutionalization of
a KPA can be effective, repeatable, and lasting.  The five common
features are: 

  -- Commitment to perform.  Commitment to perform describes the
     actions that the organization must take to establish the process
     and ensure that it can endure.  Commitment to perform typically
     involves establishing organizational policies and sponsorship. 

  -- Ability to perform.  Ability to perform describes the
     preconditions that must exist in the project or organization to
     implement the software acquisition process competently.  Ability
     to perform typically involves resources, organizational
     structures, and training. 

  -- Activities performed.  Activities performed describes the roles
     and procedures necessary to implement a KPA.  Activities
     performed typically involve establishing plans and procedures,
     performing the work, tracking it, and taking appropriate
     management actions. 

  -- Measurement and analysis.  Measurement and analysis describes
     activities performed to measure the process and analyze the
     measurements.  Measurement and analysis typically includes
     defining the measurements to be taken and the analyses to be
     conducted to determine the status and effectiveness of the
     activities performed. 

  -- Verifying implementation.  Verifying implementation describes
     the steps to ensure that the activities are performed in
     compliance with the process that has been established. 
     Verification typically encompasses reviews by management. 

In accordance with SEI's SCE method, for each KPA in level 2 and the
one KPA in level 3 (risk management), we evaluated institutional FAA
policies and practices and compared project-specific guidance and
practices against the five common attributes.  This project-specific
comparison can result in one of four possible outcomes:  (1) project
strength--an effective implementation of the key practice; (2)
project weakness--ineffective implementation of a key practice or
failure to implement a key practice; (3) project observation--key
practice evaluated but evidence inconclusive and cannot be
characterized as either strength or weakness; and (4) not rated--key
practice not currently relevant to project, therefore not evaluated. 

We performed the project-specific evaluations on five ongoing ATC
modernization projects, each of which is described below.  We asked
FAA to choose these projects using the following criteria:  (1) the
projects are major efforts with large software acquisition
components; (2) the projects are managed by different integrated
product teams, (3) the projects are in different stages of their life
cycles, and (4) the projects are among FAA's best-managed
acquisitions.  The projects that FAA selected for our evaluation are: 

  -- Automated Radar Terminal System (ARTS) IIIE:  ARTS gathers
     information from surveillance sensors, processes it, and sends
     it to air traffic controllers in terminal radar approach control
     facilities and control towers at airports.  A series of
     improvements to ARTS have provided increased processor capacity
     and the ability to support a greater number of controller
     displays.  The ARTS IIIE improvements provide for more
     controller positions and surveillance sensor inputs at selected
     large facilities.  ARTS IIIE is operational at New York,
     Chicago, and Dallas/Fort Worth with additional systems planned
     for Southern California and Denver.  FAA estimates that the
     enhancement will cost $383.8 million to develop and deploy. 

  -- Display System Replacement (DSR):  DSR is intended to replace
     air traffic controllers' display-related systems in each of the
     en route centers.  DSR consists of controller work stations
     connected via a local area network to three interfacing systems
     (Host Computer System, Enhanced Direct Access Radar Channel, and
     Weather and Radar Processor).  FAA plans to deploy DSR to all 20
     en route centers in the continental United States, as well as
     ATC facilities in Anchorage and potentially in Honolulu.  FAA is
     now conducting system acceptance testing.  FAA estimates that
     DSR will cost $1,055.3 million to develop and deploy. 

  -- National Airspace System (NAS) Infrastructure Management System
     (NIMS):  NIMS is intended to provide the system infrastructure,
     including data architecture and network communications, to
     permit remote ATC system operational monitoring and maintenance. 
     This program will provide a three-tiered architecture consisting
     of a national control center, 4 to 10 operational control
     centers, and over 300 local work centers.  NIMS is in the
     pre-solicitation phase, and FAA estimates that it will cost
     about $500 million to develop and deploy. 

  -- Voice Switching and Control System (VSCS):  VSCS is intended to
     provide air-to-ground and ground-to-ground communications
     between en route centers and aircraft.  The VSCS is to replace
     the aging ground-to-ground switching equipment and the
     air-to-ground circuits with a single integrated,
     computer-controlled, digital voice switching system.  The
     development of VSCS is completed and all systems are
     operational.  FAA estimates that VSCS will cost about $1.5
     billion to develop and deploy. 

  -- Weather and Radar Processor (WARP):  WARP is a next generation
     weather and radar processing and display system that is intended
     to permit consolidation of weather data from several sources
     into a single, integrated display for controllers.  Currently,
     the weather information provided to controllers in the en route
     centers comes from long-range aircraft surveillance radars,
     which are not well-suited for this purpose.  Next generation
     weather radars are to replace the surveillance radars as the
     source of weather information.  WARP is to collect, process, and
     disseminate this and other weather data to controllers, traffic
     management specialists, pilots, and meteorologists.  WARP is
     currently under development, and FAA estimates that it will cost
     $124.6 million to develop and deploy. 

To address our second objective (what steps/actions FAA has underway
or planned to improve its software acquisition processes and what
obstacles, if any, may impede FAA's progress), we interviewed FAA's
Chief Scientist for Software Engineering and his staff to determine: 
(1) process improvements that are planned and underway; (2) the
rationale for each initiative; (3) the relative priority of each; (4)
progress made on each initiative; and (5) obstacles, if any, impeding
progress.  We also analyzed process improvement plans, meeting
minutes, and related documentation to further address these areas. 
Finally, we interviewed representatives from the ATC modernization
product teams and the SEPG to obtain their perspectives in assessing
process improvement support, activities, progress, and obstacles. 

The Department of Transportation provided written comments on a draft
of this report.  These comments are presented and evaluated in
chapter 11, and are reprinted in appendix I.  We performed our work
at FAA headquarters offices in Washington, D.C.  between March 1996
and February 1997, in accordance with generally accepted government
auditing standards. 


--------------------
\6 We used a draft version of the model for our evaluation (version
00.03, dated April 1996).  The first published version of the model
was released in October 1996, after we performed our evaluation. 
According to the model's authors, the published version differed only
editorially from the draft we used. 


SOFTWARE ACQUISITION PLANNING
============================================================ Chapter 2

The purpose of software acquisition planning is to ensure that
reasonable planning for the software acquisition is conducted and
that all aspects of the total software acquisition effort are
included in these plans.  According to the SA-CMM, a repeatable
software acquisition planning process, among other things, includes
(1) addressing software life cycle support in acquisition plans, (2)
preparing life cycle software cost estimates, (3) having a written
software acquisition policy, (4) measuring and reporting on the
status of software acquisition planning activities, and (5) having
guidance on software training and experience requirements for project
personnel. 


   FAA'S SOFTWARE ACQUISITION
   PLANNING PROCESS IS NOT
   EFFECTIVE
---------------------------------------------------------- Chapter 2:1

All five projects have some ability and/or activity strengths in this
KPA.  For example, every project addresses software life cycle
support in planning documents and software life cycle cost estimates
were prepared for four of the projects.  However, we found many more
process weaknesses than strengths.  For example, FAA has a systems
acquisition policy, but the policy does not specifically address or
provide guidance on software acquisition.  Therefore, FAA management
has not formally recognized the importance and uniqueness of software
acquisition issues in the system acquisition process, and has not
formally committed to managing software acquisition in a disciplined
manner.  Also, the product teams do not measure or report on the
status of software acquisition planning activities.  As a result,
management is not always aware of problems in project performance,
and cannot always take corrective action expeditiously. 
Additionally, none of the five projects has specific guidance on
software training or experience requirements for project
participation.  As a result, software training is ad hoc, and
decisions about project personnel assignments are subjective. 

Figure 2.1 provides a comprehensive listing of the five projects'
strengths, weaknesses, and observations for the software acquisition
planning KPA.  The specific findings supporting the practice ratings
cited in figure 2.1 are in tables 2.1 through 2.5. 

   Figure 2.1:  Software
   Acquisition Planning

   (See figure in printed
   edition.)



                                    Table 2.1
                     
                      Software Acquisition Planning Findings
                                   for ARTSIIIE

                      Automated Radar Terminal System IIIE
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           The system acquisition    Weakness
               organization has a        policy does not
               written policy for        adequately address
               planning the software     software, e.g., it does
               acquisition.              not address items that
                                         should be included in
                                         software planning such
                                         as contract tracking and
                                         oversight, requirements
                                         development, evaluation,
                                         and risk management.

Ability 1      Personnel are assigned    No personnel are          Weakness
               the responsibility for    assigned the
               performing software       responsibility for
               acquisition planning.     software acquisition
                                         planning.

Ability 2      The acquisition           The acquisition           Observation
               organization has          organization has no
               experienced software      guidance regarding
               acquisition management    training or experience
               personnel.                requirements for project
                                         participation.

Ability 3      Software acquisition      There are no guidelines   Weakness
               management personnel are  that define domain
               experienced in the        knowledge or experience.
               domain of the project.

Activity 1     Software acquisition      No one on the product     Weakness
               planning personnel are    team is specifically
               involved in system        assigned responsibility
               acquisition planning.     for software
                                         acquisition.

Activity 2     The project's software    There is no documented    Weakness
               acquisition planning is   software acquisition
               documented and the        plan.
               planning documentation
               is maintained over the
               life of the project.

Activity 3     The software acquisition  There is no software      Weakness
               strategy for the project  acquisition strategy.
               is developed.

Activity 4     Software acquisition      The product team ensures  Strength
               planning includes         that life cycle support
               provisions for ensuring   is included in planning
               that the life cycle       documentation.
               support of the system is
               included in planning
               documentation.

Activity 5     A life cycle cost         The life cycle cost       Weakness
               estimate for the          estimate was prepared
               software activity is      but not independently
               prepared and              verified.
               independently verified.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         and used to determine
               software acquisition      the status of activities
               planning activities.      for any of the key
                                         process areas.

Verification   The activities for        While the Integrated      Weakness
1              software acquisition      Product Team leader
               planning are reviewed by  reviews the status of
               acquisition organization  the contract and the
               management on a periodic  contractor's cost and
               basis.                    schedule, he does not
                                         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              software acquisition      leader reviews the
               planning are reviewed by  status of the contract
               the project manager on    and the contractor's
               both a periodic and       cost and schedule, he
               event-driven basis.       does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------


                                    Table 2.2
                     
                      Software Acquisition Planning Findings
                                     for DSR

                           Display System Replacement
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           The system acquisition    Weakness
               organization has a        policy does not
               written policy for        adequately address
               planning the software     software, e.g., it does
               acquisition.              not address items that
                                         should be included in
                                         software planning such
                                         as contract tracking and
                                         oversight, requirements
                                         development, evaluation,
                                         and risk management.

Ability 1      Personnel are assigned    Personnel are assigned    Strength
               the responsibility for    the responsibility for
               performing software       performing software
               acquisition planning.     acquisition planning.

Ability 2      The acquisition           The acquisition           Observation
               organization has          organization has no
               experienced software      guidance regarding
               acquisition management    training or experience
               personnel.                requirements for project
                                         participation.

Ability 3      Software acquisition      There are no guidelines   Weakness
               management personnel are  that define domain
               experienced in the        knowledge or experience.
               domain of the project.

Activity 1     Software acquisition      Software acquisition      Strength
               planning personnel are    personnel are involved
               involved in system        in system acquisition
               acquisition planning.     planning.

Activity 2     The project's software    There is no software      Weakness
               acquisition planning is   acquisition planning
               documented and the        documentation.
               planning documentation
               is maintained over the
               life of the project.

Activity 3     The software acquisition  Officials stated that     Weakness
               strategy for the project  the software acquisition
               is developed.             strategy is developed,
                                         however, the documents
                                         provided did not address
                                         such things as
                                         objectives,
                                         technologies, and
                                         schedule.

Activity 4     Software acquisition      Software acquisition      Strength
               planning includes         planning includes life
               provisions for ensuring   cycle support planning.
               that the life cycle
               support of the system is
               included in planning
               documentation.

Activity 5     A life cycle cost         A life cycle cost         Strength
               estimate for the          estimate is prepared and
               software activity is      independently verified.
               prepared and
               independently verified.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         and used to determine
               software acquisition      the status of activities
               planning activities.      for any of the key
                                         process areas.

Verification   The activities for        While the Integrated      Weakness
1              software acquisition      Product Team leader
               planning are reviewed by  reviews the status of
               acquisition organization  the contract and the
               management on a periodic  contractor's cost and
               basis.                    schedule, he does not
                                         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              software acquisition      leader reviews the
               planning are reviewed by  status of the contract
               the project manager on    and the contractor's
               both a periodic and       cost and schedule, he
               event-driven basis.       does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------


                                    Table 2.3
                     
                      Software Acquisition Planning Findings
                                     for NIMS

                      NAS Infrastructure Management System
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           The system acquisition    Weakness
               organization has a        policy does not
               written policy for        adequately address
               planning the software     software, e.g., it does
               acquisition.              not address items that
                                         should be included in
                                         software planning such
                                         as contract tracking and
                                         oversight, requirements
                                         development, evaluation,
                                         and risk management.

Ability 1      Personnel are assigned    The team members are      Strength
               the responsibility for    assigned the
               performing software       responsibility for
               acquisition planning.     software acquisition
                                         planning.

Ability 2      The acquisition           The acquisition           Observation
               organization has          organization has no
               experienced software      guidance regarding
               acquisition management    training or experience
               personnel.                requirements for project
                                         participation.

Ability 3      Software acquisition      There are no guidelines   Weakness
               management personnel are  that define domain
               experienced in the        knowledge or experience.
               domain of the project.

Activity 1     Software acquisition      The team members for      Strength
               planning personnel are    software acquisition are
               involved in system        assigned collective
               acquisition planning.     responsibility and are
                                         actively involved in
                                         system acquisition
                                         planning.

Activity 2     The project's software    At this early stage in    Observation
               acquisition planning is   the program, the
               documented and the        software acquisition
               planning documentation    planning documentation
               is maintained over the    is being written but is
               life of the project.      not complete.

Activity 3     The software acquisition  Officials stated that     Strength
               strategy for the project  the software acquisition
               is developed.             strategy will be
                                         contained within the
                                         acquisition plan.

Activity 4     Software acquisition      Software acquisition      Strength
               planning includes         planning includes
               provisions for ensuring   provisions for ensuring
               that the life cycle       that life cycle support
               support of the system is  is included in planning
               included in planning      documentation.
               documentation.

Activity 5     A life cycle cost         A life cycle cost         Strength
               estimate for the          estimate for software
               software activity is      has been prepared and
               prepared and              independently verified.
               independently verified.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         to determine the status
               software acquisition      of activities for any of
               planning activities.      the key process areas.

Verification   The activities for        While the Integrated      Weakness
1              software acquisition      Product Team leader
               planning are reviewed by  reviews the status of
               acquisition organization  the contract and the
               management on a periodic  contractor's cost and
               basis.                    schedule, he does not
                                         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              software acquisition      leader reviews the
               planning are reviewed by  status of the contract
               the project manager on    and the contractor's
               both a periodic and       cost and schedule, he
               event-driven basis.       does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------


                                    Table 2.4
                     
                      Software Acquisition Planning Findings
                                     for VSCS

                       Voice Switching and Control System
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           The system acquisition    Weakness
               organization has a        policy does not
               written policy for        adequately address
               planning the software     software, e.g., it does
               acquisition.              not address items that
                                         should be included in
                                         software planning such
                                         as contract tracking and
                                         oversight, requirements
                                         development, evaluation,
                                         and risk management.

Ability 1      Personnel are assigned    Personnel are assigned    Strength
               the responsibility for    the responsibility for
               performing software       performing software
               acquisition planning.     acquisition planning.

Ability 2      The acquisition           The acquisition           Observation
               organization has          organization has no
               experienced software      guidance regarding
               acquisition management    training or experience
               personnel.                requirements for project
                                         participation.

Ability 3      Software acquisition      There are no guidelines   Weakness
               management personnel are  that define domain
               experienced in the        knowledge or experience.
               domain of the project.

Activity 1     Software acquisition      Software acquisition      Strength
               planning personnel are    personnel are involved
               involved in system        in system acquisition
               acquisition planning.     planning.

Activity 2     The project's software    The project's software    Weakness
               acquisition planning is   acquisition planning is
               documented and the        not documented.
               planning documentation
               is maintained over the
               life of the project.

Activity 3     The software acquisition  No software acquisition   Weakness
               strategy for the project  strategy exists. The
               is developed.             system acquisition
                                         strategy does not
                                         address software.

Activity 4     Software acquisition      The life cycle support    Strength
               planning includes         of the system is
               provisions for ensuring   included in the
               that the life cycle       acquisition planning
               support of the system is  documentation.
               included in planning
               documentation.

Activity 5     A life cycle cost         The life cycle cost       Strength
               estimate for the          estimate has been
               software activity is      prepared and
               prepared and              independently verified.
               independently verified.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         or used to determine the
               software acquisition      status of activities for
               planning activities.      any of the key process
                                         areas.

Verification   The activities for        While the Integrated      Weakness
1              software acquisition      Product Team leader
               planning are reviewed by  reviews the status of
               acquisition organization  the contract and the
               management on a periodic  contractor's cost and
               basis.                    schedule, he does not
                                         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              software acquisition      leader reviews the
               planning are reviewed by  status of the contract
               the project manager on    and the contractor's
               both a periodic and       cost and schedule, he
               event-driven basis.       does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------


                                    Table 2.5
                     
                      Software Acquisition Planning Findings
                                     for WARP

                          Weather and Radar Processor
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           The system acquisition    Weakness
               organization has a        policy does not
               written policy for        adequately address
               planning the software     software, e.g., it does
               acquisition.              not address items that
                                         should be included in
                                         software planning such
                                         as contract tracking and
                                         oversight, requirements
                                         development, evaluation,
                                         and risk management.

Ability 1      Personnel are assigned    Although the product      Observation
               the responsibility for    team stated that they
               performing software       are assigned collective
               acquisition planning.     responsibility for
                                         systems acquisition,
                                         they could not provide
                                         documentation to show a
                                         specific assignment for
                                         software acquisition.

Ability 2      The acquisition           The acquisition           Observation
               organization has          organization has no
               experienced software      guidance regarding
               acquisition management    training or experience
               personnel.                requirements for project
                                         participation.

Ability 3      Software acquisition      There are no guidelines   Weakness
               management personnel are  that define domain
               experienced in the        knowledge or experience.
               domain of the project.

Activity 1     Software acquisition      Although the product      Weakness
               planning personnel are    team is responsible for
               involved in system        systems acquisition,
               acquisition planning.     there is no one
                                         specifically assigned
                                         for software nor does
                                         any document expressly
                                         state that software is
                                         part of systems
                                         acquisition.

Activity 2     The project's software    There is no software      Weakness
               acquisition planning is   acquisition plan.
               documented and the
               planning documentation
               is maintained over the
               life of the project.

Activity 3     The software acquisition  There is no software      Weakness
               strategy for the project  acquisition strategy for
               is developed.             the project. The system
                                         acquisition strategy
                                         covers only software
                                         enhancements.

Activity 4     Software acquisition      The acquisition plan      Strength
               planning includes         includes provisions for
               provisions for ensuring   ensuring that life cycle
               that the life cycle       support is included.
               support of the system is
               included in planning
               documentation.

Activity 5     A life cycle cost         The life cycle cost       Strength
               estimate for the          estimate is prepared and
               software activity is      independently verified.
               prepared and
               independently verified.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         and used to determine
               software acquisition      the status of activities
               planning activities.      for any of the key
                                         process areas.

Verification   The activities for        While the Integrated      Weakness
1              software acquisition      Product Team leader
               planning are reviewed by  reviews the status of
               acquisition organization  the contract and the
               management on a periodic  contractor's cost and
               basis.                    schedule, he does not
                                         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              software acquisition      leader reviews the
               planning are reviewed by  status of the contract
               the project manager on    and the contractor's
               both a periodic and       cost and schedule, he
               event-driven basis.       does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------

   CONCLUSIONS
---------------------------------------------------------- Chapter 2:2

Effective planning is the cornerstone of successful software
acquisition.  While FAA showed some strengths in this KPA, its many
weaknesses render the software acquisition planning capability ad hoc
and chaotic.  Therefore, it is unlikely that projects are effectively
measuring and monitoring software acquisition progress and taking
corrective actions expeditiously. 



SOLICITATION
============================================================ Chapter 3

The purpose of solicitation is to prepare a request for proposal that
delineates a project's software-related requirements, and select a
contractor that can most cost-effectively satisfy these requirements,
while complying with relevant solicitation laws and regulations. 
According to the SA-CMM, specific requirements for a repeatable
solicitation process include, among other things, (1) having and
following a solicitation plan, (2) assigning responsibility and
ensuring sufficient resources for coordinating and conducting
solicitation activities, (3) preparing and reviewing cost and
schedule estimates for the software products and services being
acquired, and (4) periodically measuring solicitation work completed
and effort and funds expended, comparing these measures to plans, and
reporting the results to management. 


   PRODUCT TEAMS PERFORMING MANY
   SOLICITATION PRACTICES
---------------------------------------------------------- Chapter 3:1

All five projects have strengths in many of the practices required by
this KPA.  For example, most projects have written solicitation
plans, assign responsibility for coordinating and conducting the
solicitation activities, and prepare and review contract-related
software cost and schedule estimates. 

However, the projects are weak in several areas.  For example, even
though most projects had a written solicitation plan, not all
projects followed their plans.  Also, none of the projects adequately
identified the resources needed to conduct solicitation activities. 
While FAA personnel stated that they had adequate solicitation
resources, they provided no evidence of either a mechanism for
identifying required resources or for ensuring that required
resources are provided.  These weaknesses increase the risk of FAA
not adequately evaluating the offerors' proposals, and making a
suboptimal selection.  Additionally, none of the five measured or
reported on the status of product team solicitation activities.  As a
result, management cannot identify solicitation problems early and
resolve them expeditiously. 

Figure 3.1 provides a comprehensive listing of the five projects'
strengths, weaknesses, and observations for the solicitation KPA. 
The specific findings supporting the practice ratings cited in figure
3.1 are in tables 3.1 through 3.5. 

   Figure 3.1:  Solicitation
   Summary

   (See figure in printed
   edition.)



   (See figure in printed
   edition.)




                                    Table 3.1
                     
                        Solicitation Findings for ARTSIIIE

                      Automated Radar Terminal System IIIE
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           FAA Order 1810.1F is the  Strength
               organization has a        written policy.
               written policy for the
               conduct of the
               solicitation.

Commitment 2   Responsibility for the    Officials gave            Weakness
               software portion of the   conflicting answers as
               solicitation is           to who is responsible
               designated.               for the software portion
                                         of the solicitation, and
                                         could not provide a
                                         document that formally
                                         designates
                                         responsibility.

Commitment 3   A selection official has  The Administrator was     Strength
               been designated to be     the selection official
               responsible for the       for the sole-source
               selection process and     contract.
               the decision.

Ability 1      A group that is           A group (matrix team) is  Strength
               responsible for           responsible for
               coordinating and          coordinating and
               conducting the            conducting the
               solicitation activities   solicitation activities.
               exists.

Ability 2      Adequate resources are    No mechanism exists for   Weakness
               provided for the          identifying resources
               solicitation activities.  required and for
                                         ensuring that the needed
                                         resources are provided
                                         to the project.

Ability 3      Individuals performing    The acquisition           Observation
               the solicitation          organization has no
               activities have           guidance regarding
               experience or receive     training or experience
               training.                 requirements for project
                                         participation.

Activity 1     The project team          The team documents its    Strength
               documents its plans for   plans for solicitation
               solicitation activities.  activities.

Activity 2     The project's             While officials stated    Observation
               solicitation activities   that solicitation
               are performed in          activities are performed
               accordance with its       in accordance with its
               plans.                    plans, they could not
                                         provide documentation to
                                         support this.

Activity 3     The project team          The team does not         Weakness
               documents its plans for   document its plans for
               proposal evaluation       proposal evaluation
               activities.               activities.

Activity 4     The project team's        No evaluation plan        Weakness
               proposal evaluation       exists, therefore, the
               activities are performed  team could not perform
               in accordance with its    in accordance with its
               plans.                    plan.

Activity 5     A cost estimate and       A cost estimate and       Strength
               schedule for the          schedule were prepared.
               software activity being
               sought are prepared.

Activity 6     The software cost         The cost estimate and     Strength
               estimate and schedule     schedule were
               are independently         independently reviewed.
               reviewed for
               comprehensiveness and
               realism.

Activity 7     The groups supporting     No orientation briefing   Weakness
               the solicitation (e.g.,   occurred.
               end user, systems
               engineering, support
               organization, and
               application domain
               experts) receive
               orientation on the
               solicitation's
               objectives and
               procedures.

Activity 8     The project team and the  Officials stated that     Observation
               offeror review the        meetings were held with
               project's software        the contractor to ensure
               requirements during       mutual understanding,
               negotiations to ensure    however, they could not
               mutual understanding.     provide documents to
                                         support this.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         and used to determine
               solicitation activities.  the status of activities
                                         for any of the key
                                         process areas.

Verification   The activities for        While the Integrated      Weakness
1              solicitation are          Product Team leader
               reviewed by the           reviews the status of
               designated selection      the contract and the
               official or acquisition   contractor's cost and
               organization management   schedule, he does not
               on a periodic basis.      review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the various
                                         key process areas.

Verification   The activities for        While the product team    Weakness
2              solicitation are          leader reviews the
               reviewed by the project   status of the contract
               manager on both a         and the contractor's
               periodic and event-       cost and schedule, he
               driven basis.             does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------



                                    Table 3.2
                     
                          Solicitation Findings for DSR

                           Display System Replacement
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           There is an FAA policy    Strength
               organization has a        that addresses
               written policy for the    solicitation conduct.
               conduct of the
               solicitation.

Commitment 2   Responsibility for the    Officials stated that     Observation
               software portion of the   responsibility for the
               solicitation is           software portion of the
               designated.               solicitation has been
                                         assigned, however, they
                                         could not provide
                                         documents to support
                                         this.

Commitment 3   A selection official has  Not applicable. DSR was   Not rated
               been designated to be     a change order from an
               responsible for the       existing larger contract
               selection process and     that went through the
               the decision.             acquisition phase in
                                         1984. Current team
                                         members joined the team
                                         after the change order
                                         was negotiated and,
                                         therefore, could not
                                         address this key
                                         practice.

Ability 1      A group that is           A group responsible for   Strength
               responsible for           the solicitation exists.
               coordinating and
               conducting the
               solicitation activities
               exists.

Ability 2      Adequate resources are    No mechanism exists for   Weakness
               provided for the          identifying resources
               solicitation activities.  required and for
                                         ensuring that the needed
                                         resources are provided
                                         to the project.

Ability 3      Individuals performing    The acquisition           Observation
               the solicitation          organization has no
               activities have           guidance regarding
               experience or receive     training or experience
               training.                 requirements for project
                                         participation.

Activity 1     The project team          Not applicable. DSR was   Not rated
               documents its plans for   a change order from an
               solicitation activities.  existing larger contract
                                         that went through the
                                         acquisition phase in
                                         1984. Current team
                                         members joined the team
                                         after the change order
                                         was negotiated and,
                                         therefore, could not
                                         address this key
                                         practice.

Activity 2     The project's             Not applicable. DSR was   Not rated
               solicitation activities   a change order from an
               are performed in          existing larger contract
               accordance with its       that went through the
               plans.                    acquisition phase in
                                         1984. Current team
                                         members joined the team
                                         after the change order
                                         was negotiated and,
                                         therefore, could not
                                         address this key
                                         practice.

Activity 3     The project team          The product team uses a   Strength
               documents its plans for   change order evaluation
               proposal evaluation       plan to document plans
               activities.               for proposal evaluation
                                         activities.

Activity 4     The project team's        Not applicable. DSR was   Not rated
               proposal evaluation       a change order from an
               activities are performed  existing larger contract
               in accordance with its    that went through the
               plans.                    acquisition phase in
                                         1984. Current team
                                         members joined the team
                                         after the change order
                                         was negotiated and,
                                         therefore, could not
                                         address this key
                                         practice.

Activity 5     A cost estimate and       A cost estimate and       Strength
               schedule for the          schedule were generated.
               software activity being
               sought are prepared.

Activity 6     The software cost         Officials could not       Observation
               estimate and schedule     produce documentation
               are independently         that supported their
               reviewed for              statement that the
               comprehensiveness and     software cost estimate
               realism.                  and schedule were
                                         independently reviewed.

Activity 7     The groups supporting     Not applicable. DSR was   Not rated
               the solicitation (e.g.,   a change order from an
               end user, systems         existing larger contract
               engineering, support      that went through the
               organization, and         acquisition phase in
               application domain        1984. Current team
               experts) receive          members joined the team
               orientation on the        after the change order
               solicitation's            was negotiated and,
               objectives and            therefore, could not
               procedures.               address this key
                                         practice.

Activity 8     The project team and the  A series of scheduled     Strength
               offeror review the        meetings were held to
               project's software        ensure mutual
               requirements during       understanding of
               negotiations to ensure    requirements.
               mutual understanding.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         and used to determine
               solicitation activities.  the status of activities
                                         for any of the key
                                         process areas.

Verification   The activities for        While the Integrated      Weakness
1              solicitation are          Product Team leader
               reviewed by the           reviews the status of
               designated selection      the contract and the
               official or acquisition   contractor's cost and
               organization management   schedule, he does not
               on a periodic basis.      review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              solicitation are          leader reviews the
               reviewed by the project   status of the contract
               manager on both a         and the contractor's
               periodic and event-       cost and schedule, he
               driven basis.             does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------



                                    Table 3.3
                     
                          Solicitation Findings for NIMS

                      NAS Infrastructure Management System
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           The Acquisition           Strength
               organization has a        Management System is the
               written policy for the    written policy.
               conduct of the
               solicitation.

Commitment 2   Responsibility for the    Responsibility for the    Strength
               software portion of the   software portion of the
               solicitation is           solicitation has been
               designated.               designated to software
                                         experts on the team.

Commitment 3   A selection official has  A selection official has  Strength
               been designated to be     been designated to be
               responsible for the       responsible for the
               selection process and     selection process and
               the decision.             decision.

Ability 1      A group that is           The contracting officer,  Strength
               responsible for           support staff, and the
               coordinating and          parent ASU organization
               conducting the            are responsible for
               solicitation activities   coordinating and
               exists.                   conducting the
                                         solicitation activities.

Ability 2      Adequate resources are    No mechanism exists for   Weakness
               provided for the          identifying resources
               solicitation activities.  required and for
                                         ensuring that the needed
                                         resources are provided
                                         to the project.

Ability 3      Individuals performing    The acquisition           Observation
               the solicitation          organization has no
               activities have           guidance regarding
               experience or receive     training or experience
               training.                 requirements for project
                                         participation.

Activity 1     The project team          The product team          Strength
               documents its plans for   documents its plans for
               solicitation activities.  solicitation activities.

Activity 2     The project's             Officials stated that     Strength
               solicitation activities   solicitation activities
               are performed in          will be performed in
               accordance with its       accordance with its
               plans.                    plans. NIMS is in the
                                         presolicitation phase.

Activity 3     The project team          The project team has      Strength
               documents its plans for   documented its plans for
               proposal evaluation       proposal evaluation
               activities.               activities.

Activity 4     The project team's        In accordance with the    Strength
               proposal evaluation       plan, prequalification
               activities are performed  was completed and
               in accordance with its    vendors were down-
               plans.                    selected from it.

Activity 5     A cost estimate and       The Acquisition Program   Strength
               schedule for the          Baseline includes a cost
               software activity being   estimate and schedule
               sought are prepared.      for the software
                                         acquisition.

Activity 6     The software cost         An independent            Strength
               estimate and schedule     assessment was done.
               are independently
               reviewed for
               comprehensiveness and
               realism.

Activity 7     The groups supporting     Solicitation activities   Strength
               the solicitation (e.g.,   orientation was
               end user, systems         conducted for NIMS
               engineering, support      personnel.
               organization, and
               application domain
               experts) receive
               orientation on the
               solicitation's
               objectives and
               procedures.

Activity 8     The project team and the  The NIMS project has not  Not rated
               offeror review the        yet reached this stage,
               project's software        therefore, this activity
               requirements during       was not rated.
               negotiations to ensure
               mutual understanding.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         to determine the status
               solicitation activities.  of activities for any of
                                         the key process areas.

Verification   The activities for        While the Integrated      Weakness
1              solicitation are          Product Team leader
               reviewed by the           reviews the status of
               designated selection      the contract and the
               official or acquisition   contractor's cost and
               organization management   schedule, he does not
               on a periodic basis.      review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              solicitation are          leader reviews the
               reviewed by the project   status of the contract
               manager on both a         and the contractor's
               periodic and event-       cost and schedule, he
               driven basis.             does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------



                                    Table 3.4
                     
                          Solicitation Findings for VSCS

                       Voice Switching and Control System
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           FAA Order 1810.1F is the  Strength
               organization has a        written policy.
               written policy for the
               conduct of the
               solicitation.

Commitment 2   Responsibility for the    Officials gave            Weakness
               software portion of the   conflicting answers as
               solicitation is           to who is responsible
               designated.               for software
                                         acquisition, and could
                                         not provide
                                         documentation that
                                         formally designates
                                         responsibility.

Commitment 3   A selection official has  The Administrator was     Strength
               been designated to be     the selection official.
               responsible for the
               selection process and
               the decision.

Ability 1      A group that is           Officials stated that a   Observation
               responsible for           group was responsible
               coordinating and          for coordinating and
               conducting the            conducting solicitation
               solicitation activities   activities; however,
               exists.                   they could not provide
                                         documentation to support
                                         this claim.

Ability 2      Adequate resources are    No mechanism exists for   Weakness
               provided for the          identifying the
               solicitation activities.  resources required and
                                         for ensuring that the
                                         needed resources are
                                         provided to the project.

Ability 3      Individuals performing    The acquisition           Observation
               the solicitation          organization has no
               activities have           guidance regarding
               experience or receive     training or experience
               training.                 requirements for project
                                         participation.

Activity 1     The project team          Officials said that       Observation
               documents its plans for   solicitation activities
               solicitation activities.  were documented in the
                                         Acquisition Plan and
                                         Source Evaluation Plan;
                                         however, they could not
                                         provide these documents.

Activity 2     The project's             Solicitation activities   Weakness
               solicitation activities   were not performed in
               are performed in          accordance with plans.
               accordance with its
               plans.

Activity 3     The project team          Officials stated that     Observation
               documents its plans for   proposal evaluation
               proposal evaluation       activities were
               activities.               documented; however,
                                         they could not provide
                                         the documentation.

Activity 4     The project team's        Officials could not       Weakness
               proposal evaluation       describe how or if
               activities are performed  proposal evaluation
               in accordance with its    activities were
               plans.                    performed in accordance
                                         with plans;
                                         additionally, they could
                                         not provide documents to
                                         support this activity.

Activity 5     A cost estimate and       A cost estimate and       Strength
               schedule for the          schedule for the
               software activity being   software activity were
               sought are prepared.      developed.

Activity 6     The software cost         The software cost         Strength
               estimate and schedule     estimate and schedule
               are independently         were independently
               reviewed for              reviewed for
               comprehensiveness and     comprehensiveness and
               realism.                  realism.

Activity 7     The groups supporting     Officials stated that     Weakness
               the solicitation (e.g.,   orientation was held,
               end user, systems         however, the
               engineering, support      documentation provided
               organization, and         did not indicate that
               application domain        the product team
               experts) receive          received orientation on
               orientation on the        the solicitation
               solicitation's            objectives and
               objectives and            procedures.
               procedures.

Activity 8     The project team and the  Officials stated that     Observation
               offeror review the        the product team and the
               project's software        offeror reviewed project
               requirements during       software requirements
               negotiations to ensure    during pre-award
               mutual understanding.     negotiations, however,
                                         they could not provide
                                         documentation to support
                                         this.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         or used to determine the
               solicitation activities.  status of activities for
                                         any of the key process
                                         areas.

Verification   The activities for        While the Integrated      Weakness
1              solicitation are          Product Team leader
               reviewed by the           reviews the status of
               designated selection      the contract and the
               official or acquisition   contractor's cost and
               organization management   schedule, he does not
               on a periodic basis.      review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              solicitation are          leader reviews the
               reviewed by the project   status of the contract
               manager on both a         and the contractor's
               periodic and event-       cost and schedule, he
               driven basis.             does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------



                                    Table 3.5
                     
                          Solicitation Findings for WARP

                          Weather and Radar Processor
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           There is a written        Strength
               organization has a        policy for solicitation.
               written policy for the
               conduct of the
               solicitation.

Commitment 2   Responsibility for the    Responsibility for the    Strength
               software portion of the   solicitation has been
               solicitation is           designated.
               designated.

Commitment 3   A selection official has  Because there was only    Not rated
               been designated to be     one offeror, this
               responsible for the       commitment was not
               selection process and     rated.
               the decision.

Ability 1      A group that is           A group is responsible    Strength
               responsible for           for coordinating and
               coordinating and          conducting the
               conducting the            solicitation activities.
               solicitation activities
               exists.

Ability 2      Adequate resources are    No mechanism exists for   Weakness
               provided for the          identifying and for
               solicitation activities.  ensuring that the needed
                                         resources are provided
                                         to the project.

Ability 3      Individuals performing    The acquisition           Observation
               the solicitation          organization has no
               activities have           guidance regarding
               experience or receive     training or experience
               training.                 requirements for project
                                         participation.

Activity 1     The project team          The project team          Strength
               documents its plans for   documents its plans for
               solicitation activities.  solicitation activities.

Activity 2     The project's             The project's             Strength
               solicitation activities   solicitation activities
               are performed in          were performed in
               accordance with its       accordance with its
               plans.                    plans.

Activity 3     The project team          The team documented its   Strength
               documents its plans for   plans for proposal
               proposal evaluation       evaluation activities.
               activities.

Activity 4     The project team's        Proposal evaluation       Strength
               proposal evaluation       activities were
               activities are performed  performed in accordance
               in accordance with its    with the plan.
               plans.

Activity 5     A cost estimate and       An independent            Strength
               schedule for the          government cost estimate
               software activity being   was prepared which
               sought are prepared.      included major milestone
                                         data.

Activity 6     The software cost         The software cost         Strength
               estimate and schedule     estimate and schedule
               are independently         were independently
               reviewed for              reviewed.
               comprehensiveness and
               realism.

Activity 7     The groups supporting     Officials stated that     Observation
               the solicitation (e.g.,   team members received
               end user, systems         orientation at the
               engineering, support      beginning of the
               organization, and         solicitation, however,
               application domain        they could not provide
               experts) receive          documentation to support
               orientation on the        this.
               solicitation's
               objectives and
               procedures.

Activity 8     The project team and the  Numerous negotiation      Strength
               offeror review the        sessions were held.
               project's software
               requirements during
               negotiations to ensure
               mutual understanding.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         and used to determine
               solicitation activities.  the status of activities
                                         for any of the key
                                         process areas.

Verification   The activities for        While the Integrated      Weakness
1              solicitation are          Product Team leader
               reviewed by the           reviews the status of
               designated selection      the contract and the
               official or acquisition   contractor's cost and
               organization management   schedule, he does not
               on a periodic basis.      review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              solicitation are          leader reviews the
               reviewed by the project   status of the contract
               manager on both a         and the contractor's
               periodic and event-       cost and schedule, he
               driven basis.             does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------

   CONCLUSIONS
---------------------------------------------------------- Chapter 3:2

While FAA has many strengths in this KPA, systemic weaknesses in
areas including measurement and analysis and management verification
of practices, along with other project-specific weaknesses, render
this KPA non-repeatable and dependent upon the capabilities and
commitment of individual employees. 


REQUIREMENTS DEVELOPMENT AND
MANAGEMENT
============================================================ Chapter 4

The purpose of requirements development and management is to
establish and maintain a common and unambiguous definition of
software requirements among the acquisition team, the system users,
and the software development contractor.  This KPA involves two
subprocesses:  (1) developing a baseline set of software-related
contractual requirements, and (2) managing these requirements and
changes to these requirements for the duration of the acquisition. 

The SA-CMM specifies a number of requirements development and
management practices necessary to achieve a repeatable maturity
level.  These include (1) having a written organizational policy for
establishing and managing requirements allocated to software; (2)
documenting plans for the development and management of requirements;
(3) having documented processes for requirements development,
including elicitation, analysis, and verification; (4) measuring and
reporting on the status of requirements development and management
activities to management; (5) appraising the impact on software of
system-level requirements changes; and (6) having a mechanism to
ensure that contractor-delivered work products meet specified
requirements. 


   REQUIREMENTS DEVELOPMENT AND
   MANAGEMENT PROCESS IS NOT
   EFFECTIVE
---------------------------------------------------------- Chapter 4:1

In the past, we have attributed ATC modernization problems, in part,
to FAA's failure to effectively manage requirements.  For example, we
reported in 1994 that FAA did not adequately specify or effectively
control changes to the requirements of its Initial Sector Suite
System (ISSS) component of the Advanced Automation System.\1

Our evaluation of FAA's capability relative to this KPA's
requirements reiterates our earlier reported concerns in this area
and pinpoints specific weaknesses.  For example, while FAA has a
policy on requirements development and management, this policy does
not address establishing and managing software requirements. 
Further, product teams do not always document their requirements
development and management plans, and while two had a defined process
for controlling changes to existing, baselined requirements, they did
not have a documented process for developing new software
requirements, including requirements planning, elicitation, analysis,
or verification.  Additionally, management does not oversee or verify
requirements development and management activities, which means that
management has no assurance that specified requirements are correct
and complete, and does not know when management action is warranted. 

We also found some practice strengths.  For example, most projects
(1) are assessing the impact on software requirements of system-level
requirements changes and (2) have a mechanism to ensure that
contractor-delivered work products and services satisfied specified
software requirements. 

Figure 4.1 provides a comprehensive listing of the five projects'
strengths, weaknesses, and observations for the requirements
development and management KPA.  The specific findings supporting the
practice ratings in figure 4.1 are in tables 4.1 through 4.5. 

   Figure 4.1:  Requirements
   Development and Management
   Summary

   (See figure in printed
   edition.)



                                    Table 4.1
                     
                     Requirements Development and Management
                              Findings for ARTSIIIE

                      Automated Radar Terminal System IIIE
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           Officials stated that     Weakness
               organization has a        FAA Order 1810.1F is the
               written policy for        written policy for
               establishing and          requirements development
               managing the system       and management, however,
               requirements allocated    it does not address
               to software.              software requirements.

Ability 1      A group that is           A group responsible for   Strength
               responsible for           requirements development
               performing requirements   and management exists.
               development and
               management activities
               exists.

Ability 2      Adequate resources are    No mechanism exists for   Weakness
               provided for              identifying and for
               requirements development  ensuring that the needed
               and management            resources are provided
               activities.               to the project.

Ability 3      Individuals performing    The acquisition           Observation
               requirements development  organization has no
               and management            guidance regarding
               activities have           training or experience
               experience or receive     requirements for project
               training.                 participation.

Activity 1     The project team          The project team does     Weakness
               documents its plans for   not document its plans
               requirements development  for requirements
               and management            development, planning,
               activities.               elicitation, analysis,
                                         and verification.

Activity 2     The project's             There is no requirements  Weakness
               requirements development  development and
               and management            management plan,
               activities are performed  therefore, the
               in accordance with its    activities are not
               plans.                    performed in accordance
                                         with it.

Activity 3     The project team          A requirements baseline,  Strength
               baselines the software    which is under change
               requirements and places   control, was established
               them under change         prior to the release of
               control early in the      the solicitation.
               project, but not later
               than release of the
               solicitation.

Activity 4     The project team          The project team's        Strength
               appraises system          appraisals of the impact
               requirements change       of system requirements
               requests for their        changes on software are
               impact on software.       documented.

Activity 5     The project team          The project team's        Strength
               appraises software        appraisal of software
               requirements changes for  requirements changes'
               their impact on           impact on performance,
               performance, schedule,    schedule, cost, and
               cost, system capacities,  system capacities is
               supportability, and       documented, but not the
               architecture.             impact on system
                                         supportability or
                                         architecture.

Activity 6     The project team          There is a mechanism for  Strength
               maintains a requirements  traceability of software
               mechanism for             requirements
               traceability during the   implementation.
               software effort to
               ensure requirements have
               been included in the
               implemented work
               products and services.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         and used to determine
               requirements development  the status of activities
               and management            for any of the key
               activities.               process areas.

Verification   The activities for        While the Integrated      Weakness
1              requirements development  Product Team leader
               and management are        reviews the status of
               reviewed by acquisition   the contract and the
               organization management   contractor's cost and
               (and the contractor) on   schedule, he does not
               a periodic basis.         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              requirements development  leader reviews the
               and management are        status of the contract
               reviewed by the project   and the contractor's
               manager on both a         cost and schedule, he
               periodic and event-       does not review the
               driven basis.             status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------



                                    Table 4.2
                     
                     Requirements Development and Management
                                 Findings for DSR

                           Display System Replacement
--------------------------------------------------------------------------------
           Key Practice                  Finding                        Rating
---------  ----------------------------  -----------------------------  --------
Commitmen  The acquisition organization  Officials stated that FAA      Weakness
t 1        has a written policy for      Order 1810.1F is the written
           establishing and managing     policy for requirements
           the system requirements       development and management,
           allocated to software.        however, it does not address
                                         software requirements.

Ability 1  A group that is responsible   The group responsible for      Strength
           for performing requirements   requirements development and
           development and management    management is the product
           activities exists.            team.

Ability 2  Adequate resources are        Adequate resources are         Strength
           provided for requirements     provided for requirements
           development and management    development and management
           activities.                   activities.

Ability 3  Individuals performing        The acquisition organization   Observat
           requirements development and  has no guidance regarding      ion
           management activities have    training or experience
           experience or receive         requirements for project
           training.                     participation.

Activity   The project team documents    While processes exist for      Weakness
1          its plans for requirements    milestone review, these
           development and management    documents do not cover the
           activities.                   activities to be performed
                                         such as user involvement,
                                         elicitation, and requirements
                                         development.

Activity   The project's requirements    There is no requirements       Weakness
2          development and management    development and management
           activities are performed in   plan, therefore, the
           accordance with its plans.    activities are not performed
                                         in accordance with it.

Activity   The project team baselines    Software requirements are      Weakness
3          the software requirements     baselined as part of the
           and places them under change  contract process, but not
           control early in the          explicitly prior to
           project, but not later than   solicitation.
           release of the solicitation.

Activity   The project team appraises    The product team appraises     Strength
4          system requirements change    system requirements changes
           requests for their impact on  for their impact on software.
           software.

Activity   The project team appraises    Software requirements were     Weakness
5          software requirements         not appraised for their
           changes for their impact on   impact on performance,
           performance, schedule, cost,  schedule, cost, system
           system capacities,            capacities, supportability,
           supportability, and           and architecture.
           architecture.

Activity   The project team maintains a  The product team maintains a   Strength
6          requirements mechanism for    mechanism for requirements
           traceability during the       traceability.
           software effort to ensure
           requirements have been
           included in the implemented
           work products and services.

Measureme  Measurements are made and     No internal measurements are   Weakness
nt 1       used to determine the status  taken and used to determine
           of the requirements           the status of activities for
           development and management    any of the key process areas.
           activities.

Verificat  The activities for            While the Integrated Product   Weakness
ion 1      requirements development and  Team leader reviews the
           management are reviewed by    status of the contract and
           acquisition organization      the contractor's cost and
           management (and the           schedule, he does not review
           contractor) on a periodic     the status of the activities
           basis.                        that are required to be
                                         performed for any of the key
                                         process areas.

Verificat  The activities for            While the product team leader  Weakness
ion 2      requirements development and  reviews the status of the
           management are reviewed by    contract and the contractor's
           the project manager on both   cost and schedule, he does
           a periodic and event-driven   not review the status of the
           basis.                        activities that are required
                                         to be performed for any of
                                         the key process areas.
--------------------------------------------------------------------------------



                                    Table 4.3
                     
                     Requirements Development and Management
                                Findings for NIMS

                      NAS Infrastructure Management System
--------------------------------------------------------------------------------
              Key Practice                 Finding                      Rating
------------  ---------------------------  ---------------------------  --------
Commitment 1  The acquisition              Officials cited the          Weakness
              organization has a written   Acquisition Management
              policy for establishing and  System and FAA Order
              managing the system          1810.1F, but these do not
              requirements allocated to    address software
              software.                    requirements.

Ability 1     A group that is responsible  The team members are         Strength
              for performing requirements  assigned collective
              development and management   responsibility for the
              activities exists.           requirements process.

Ability 2     Adequate resources are       No mechanism exists for      Weakness
              provided for requirements    identifying resources
              development and management   required and for ensuring
              activities.                  that the needed resources
                                           are provided to the
                                           project.

Ability 3     Individuals performing       The acquisition              Observat
              requirements development     organization has no          ion
              and management activities    guidance regarding training
              have experience or receive   or experience requirements
              training.                    for project participation.

Activity 1    The project team documents   The project has not reached  Not
              its plans for requirements   the point where these        rated
              development and management   activities are performed.
              activities.

Activity 2    The project's requirements   The project has not reached  Not
              development and management   the point where these        rated
              activities are performed in  activities are performed.
              accordance with its plans.

Activity 3    The project team baselines   It is too early in the       Not
              the software requirements    project life to assess: the  rated
              and places them under        software requirements have
              change control early in the  not been developed.
              project, but not later than
              release of the
              solicitation.

Activity 4    The project team appraises   It is too early in the       Not
              system requirements change   project life to assess: no   rated
              requests for their impact    software has been developed
              on software.                 or specified.

Activity 5    The project team appraises   It is too early in the       Not
              software requirements        project life to assess: no   rated
              changes for their impact on  software requirements have
              performance, schedule,       been developed or
              cost, system capacities,     specified.
              supportability, and
              architecture.

Activity 6    The project team maintains   No traceability matrix of    Not
              a requirements mechanism     software requirements has    rated
              for traceability during the  been developed at this
              software effort to ensure    point in the project.
              requirements have been
              included in the implemented
              work products and services.

Measurement   Measurements are made and    No internal process          Weakness
1             used to determine the        measurements are taken to
              status of the requirements   determine the status of
              development and management   activities for any of the
              activities.                  key process areas.

Verification  The activities for           While the Integrated         Weakness
1             requirements development     Product Team leader reviews
              and management are reviewed  the status of the contract
              by acquisition organization  and the contractor's cost
              management (and the          and schedule, he does not
              contractor) on a periodic    review the status of the
              basis.                       activities that are
                                           required to be performed
                                           for any of the key process
                                           areas.

Verification  The activities for           While the product team       Weakness
2             requirements development     leader reviews the status
              and management are reviewed  of the contract and the
              by the project manager on    contractor's cost and
              both a periodic and event-   schedule, he does not
              driven basis.                review the status of the
                                           activities that are
                                           required to be performed
                                           for any of the key process
                                           areas.
--------------------------------------------------------------------------------



                                    Table 4.4
                     
                     Requirements Development and Management
                                Findings for VSCS

                       Voice Switching and Control System
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           Officials stated that     Weakness
               organization has a        FAA Order 1810.1F is the
               written policy for        written policy for
               establishing and          requirements development
               managing the system       and management, however,
               requirements allocated    it does not address
               to software.              software requirements.

Ability 1      A group that is           The product team is       Strength
               responsible for           responsible for
               performing requirements   requirements development
               development and           and management planning.
               management activities
               exists.

Ability 2      Adequate resources are    No mechanism exists for   Weakness
               provided for              identifying the
               requirements development  resources required and
               and management            for ensuring that the
               activities.               needed resources are
                                         provided to the project.

Ability 3      Individuals performing    The acquisition           Observation
               requirements development  organization has no
               and management            guidance regarding
               activities have           training or experience
               experience or receive     requirements for project
               training.                 participation.

Activity 1     The project team          There are no documented   Weakness
               documents its plans for   plans for requirements
               requirements development  development and
               and management            management activities.
               activities.

Activity 2     The project's             There is no requirements  Weakness
               requirements development  development and
               and management            management plan,
               activities are performed  therefore, the
               in accordance with its    activities cannot be
               plans.                    performed in accordance
                                         with it.

Activity 3     The project team          Officials stated that     Observation
               baselines the software    requirements are
               requirements and places   baselined at contract
               them under change         award, but no
               control early in the      documentation was
               project, but not later    provided to support this
               than release of the       statement.
               solicitation.

Activity 4     The project team          The project team          Strength
               appraises system          appraises system
               requirements change       requirements change
               requests for their        requests for their
               impact on software.       impact on software.

Activity 5     The project team          The project team          Weakness
               appraises software        appraises software
               requirements changes for  requirements changes for
               their impact on           their impact on
               performance, schedule,    performance, schedule,
               cost, system capacities,  cost, and system
               supportability, and       capacities, but not on
               architecture.             system supportability
                                         and architecture.

Activity 6     The project team          A mechanism for           Strength
               maintains a requirements  traceability during the
               mechanism for             software effort is
               traceability during the   maintained.
               software effort to
               ensure requirements have
               been included in the
               implemented work
               products and services.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         or used to determine the
               requirements development  status of activities for
               and management            any of the key process
               activities.               areas.

Verification   The activities for        While the Integrated      Weakness
1              requirements development  Product Team leader
               and management are        reviews the status of
               reviewed by acquisition   the contract and the
               organization management   contractor's cost and
               (and the contractor) on   schedule, he does not
               a periodic basis.         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              requirements development  leader reviews the
               and management are        status of the contract
               reviewed by the project   and the contractor's
               manager on both a         cost and schedule, he
               periodic and event-       does not review the
               driven basis.             status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------



                                    Table 4.5
                     
                     Requirements Development and Management
                                Findings for WARP

                          Weather and Radar Processor
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           Officials stated that     Weakness
               organization has a        FAA Order 1810.1F is the
               written policy for        written policy for
               establishing and          requirements development
               managing the system       and management, however,
               requirements allocated    it does not address
               to software.              software requirements.

Ability 1      A group that is           The team is responsible   Strength
               responsible for           for requirements
               performing requirements   development and
               development and           measurement activities.
               management activities
               exists.

Ability 2      Adequate resources are    No mechanism exists for   Weakness
               provided for              identifying and for
               requirements development  ensuring that the needed
               and management            resources are provided
               activities.               to the project.

Ability 3      Individuals performing    The acquisition           Observation
               requirements development  organization has no
               and management            guidance regarding
               activities have           training or experience
               experience or receive     requirements for project
               training.                 participation.

Activity 1     The project team          While a process for       Weakness
               documents its plans for   managing requirements
               requirements development  changes exists, there is
               and management            no documented process
               activities.               for requirements
                                         development and
                                         management.

Activity 2     The project's             There is no plan, thus,   Weakness
               requirements development  it cannot be followed.
               and management
               activities are performed
               in accordance with its
               plans.

Activity 3     The project team          The product team          Strength
               baselines the software    baselined the software
               requirements and places   requirements and placed
               them under change         them under change
               control early in the      control before the
               project, but not later    release of the
               than release of the       solicitation.
               solicitation.

Activity 4     The project team          The product team          Strength
               appraises system          appraises system
               requirements change       requirements change
               requests for their        requests for their
               impact on software.       impact on software.

Activity 5     The project team          WARP has not had a        Not rated
               appraises software        software requirement
               requirements changes for  change yet; therefore,
               their impact on           this activity was not
               performance, schedule,    rated.
               cost, system capacities,
               supportability, and
               architecture.

Activity 6     The project team          There is a traceability   Strength
               maintains a requirements  matrix for tracking
               mechanism for             software requirements
               traceability during the   implementation in the
               software effort to        system specification.
               ensure requirements have
               been included in the
               implemented work
               products and services.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         and used to determine
               requirements development  the status of activities
               and management            for any of the key
               activities.               process areas.

Verification   The activities for        While the Integrated      Weakness
1              requirements development  Product Team leader
               and management are        reviews the status of
               reviewed by acquisition   the contract and the
               organization management   contractor's cost and
               (and the contractor) on   schedule, he does not
               a periodic basis.         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              requirements development  leader reviews the
               and management are        status of the contract
               reviewed by the project   and the contractor's
               manager on both a         cost and schedule, he
               periodic and event-       does not review the
               driven basis.             status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------

--------------------
\1 Advanced Automation System:  Implications of Problems and Recent
Changes (GAO/T-RCED-94-188, Apr.  13, 1994). 


   CONCLUSIONS
---------------------------------------------------------- Chapter 4:2

Requirements management has been a pervasive and longstanding problem
with FAA's ATC modernization, and the results of our evaluation point
to many software-specific weaknesses in this area.  Because of these
weaknesses, it is likely that requirements management problems will
continue to jeopardize projects' cost, schedule, and performance
goals. 


PROJECT OFFICE MANAGEMENT
============================================================ Chapter 5

The purpose of project office management is to manage the activities
of the project office and supporting contractors to ensure a timely,
efficient, and effective software acquisition.  According to the
SA-CMM, effective project office management requires, among other
things, that project teams (1) be organized to accomplish the
project's objective, (2) have a written policy for the management of
the software project, (3) document its plans for the activities of
the project team, (4) have the authority to alter either the
project's performance, cost, or schedule baseline while maintaining
the other two, and (5) periodically brief management on the status of
project office management activities. 


   FAA'S PROJECT OFFICE MANAGEMENT
   PROCESS AREA IS NOT EFFECTIVE
---------------------------------------------------------- Chapter 5:1

ATC modernization teams are organized to accomplish project
objectives, with each team including representatives from key
functional areas (e.g., software engineering, contracting, test and
evaluation, operations and maintenance).  However, serious weaknesses
in other KPA requirements undermine FAA's project office management
capability.  For example, most teams lack a written policy for
software project management, do not document its plans for software
acquisition management activities, and could not identify which team
member(s) is responsible for different team activities (e.g.,
software, support, requirements, testing, and/or reviews).  As a
result, lines of accountability and decision-making are blurred,
increasing the chances of delays and mistakes.  Additionally, the
product lead cannot adjust either software performance, cost, or
schedule baseline while holding the other two constant.  This
inflexibility limits the teams' ability to effectively and
efficiently respond to such events as valid requirements changes and
funding changes.  Also, project teams do not periodically brief
management on the status of project office activities, which means
that management may not be able take corrective action when
warranted. 

Figure 5.1 provides a comprehensive listing of the five projects'
strengths, weaknesses, and observations for the project office
management KPA.  The specific findings supporting the practice
ratings cited in figure 5.1 are in tables 5.1 through 5.5. 

   Figure 5.1:  Project Office
   Management Summary

   (See figure in printed
   edition.)



   (See figure in printed
   edition.)




                                    Table 5.1
                     
                      Project Office Management Findings for
                                     ARTSIIIE

                      Automated Radar Terminal System IIIE
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           FAA Order 1810.1F was     Weakness
               organization has a        cited as the written
               written policy for        policy, but it does not
               execution of the          contain policy for
               software project.         software project
                                         execution.

Commitment 2   Performance, cost, and    Performance, cost, and    Strength
               schedule baselines are    schedule baselines are
               supported.                supported.

Ability 1      A project team that is    The product team is       Strength
               responsible for           responsible for
               performing the project's  performing software
               software acquisition      acquisition management
               management activities     activities.
               exists.

Ability 2      Adequate resources for    No mechanism exists for   Weakness
               the project team and      identifying resources
               matrix support            required and for
               organization(s) are       ensuring that the needed
               provided for the          resources are provided
               duration of the software  to the project.
               acquisition project.

Ability 3      The project manager is    The acquisition baseline  Weakness
               permitted to alter        process does not allow
               either the performance,   the project manager to
               cost, or schedule         alter the baseline.
               software acquisition
               baseline while
               maintaining the other
               two constant.

Ability 4      The project team and      The acquisition           Observation
               matrix support            organization has no
               individual(s) have        guidance regarding
               experience or receive     training or experience
               training in project       requirements for project
               office software           participation.
               acquisition management
               activities.

Activity 1     The project team          The project plans do not  Weakness
               documents its plans for   address software
               software acquisition      acquisition project
               management activities.    management.

Activity 2     The project team is       The product team is       Strength
               organized to accomplish   organized to accomplish
               the project's             the project's
               objectives.               objectives.

Activity 3     The software acquisition  The activities of the     Strength
               activities of the         product team are
               project team are          directed and controlled
               directed to accomplish    to accomplish the
               the project's             project's objectives.
               objectives.

Activity 4     The software acquisition  The software activities   Strength
               activities of the         of the product team are
               project team are          controlled.
               controlled.

Activity 5     Measurements are used to  Measurements are used to  Strength
               track project status,     track project status,
               execution, and funding    execution, and funding
               expenditures.             expenditures.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         and used to determine
               project office            the status of activities
               management activities.    for any of the key
                                         process areas.

Verification   The activities for        While the Integrated      Weakness
1              project office            Product Team leader
               management are reviewed   reviews the status of
               by acquisition            the contract and the
               organization management   contractor's cost and
               on a periodic basis.      schedule, he does not
                                         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              project office            leader reviews the
               management are reviewed   status of the contract
               by the project manager    and the contractor's
               on both a periodic and    cost and schedule, he
               event-driven basis.       does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------



                                    Table 5.2
                     
                      Project Office Management Findings for
                                       DSR

                           Display System Replacement
--------------------------------------------------------------------------------
              Key Practice                 Finding                      Rating
------------  ---------------------------  ---------------------------  --------
Commitment 1  The acquisition              FAA Order 1810.1F was cited  Weakness
              organization has a written   as the written policy, but
              policy for execution of the  it does not contain policy
              software project.            for software project
                                           execution.

Commitment 2  Performance, cost, and       Performance, cost, and       Strength
              schedule baselines are       schedule baselines are
              supported.                   supported.

Ability 1     A project team that is       The product team is          Strength
              responsible for performing   responsible for managing
              the project's software       the software acquisition
              acquisition management       management activities.
              activities exists.

Ability 2     Adequate resources for the   No mechanism exists for      Weakness
              project team and matrix      identifying resources
              support organization(s) are  required and for ensuring
              provided for the duration    that the needed resources
              of the software acquisition  are provided to the
              project.                     project.

Ability 3     The project manager is       The product team leader      Weakness
              permitted to alter either    cannot alter the
              the performance, cost, or    performance, cost, or
              schedule software            schedule software
              acquisition baseline while   acquisition baseline.
              maintaining the other two
              constant.

Ability 4     The project team and matrix  The acquisition              Observat
              support individual(s) have   organization has no          ion
              experience or receive        guidance regarding training
              training in project office   or experience requirements
              software acquisition         for project participation.
              management activities.

Activity 1    The project team documents   Plans for software           Weakness
              its plans for software       acquisition management
              acquisition management       activities are not
              activities.                  documented.

Activity 2    The project team is          The product team is          Strength
              organized to accomplish the  organized to achieve the
              project's objectives.        project's objectives.

Activity 3    The software acquisition     The software acquisition     Strength
              activities of the project    activities of the product
              team are directed to         team are directed to
              accomplish the project's     accomplish the project's
              objectives.                  objectives.

Activity 4    The software acquisition     The software acquisition     Strength
              activities of the project    activities of the product
              team are controlled.         team are controlled by the
                                           Integrated Product Team
                                           leader.

Activity 5    Measurements are used to     The product team is using    Strength
              track project status,        measurements to track
              execution, and funding       product status, execution,
              expenditures.                and funding expenditures.

Measurement   Measurements are made and    No internal process          Weakness
1             used to determine the        measurements are taken and
              status of the project        used to determine the
              office management            status of activities for
              activities.                  any of the key process
                                           areas.

Verification  The activities for project   While the Integrated         Weakness
1             office management are        Product Team leader reviews
              reviewed by acquisition      the status of the contract
              organization management on   and the contractor's cost
              a periodic basis.            and schedule, he does not
                                           review the status of the
                                           activities that are
                                           required to be performed
                                           for any of the key process
                                           areas.

Verification  The activities for project   While the product team       Weakness
2             office management are        leader reviews the status
              reviewed by the project      of the contract and the
              manager on both a periodic   contractor's cost and
              and event-driven basis.      schedule, he does not
                                           review the status of the
                                           activities that are
                                           required to be performed
                                           for any of the key process
                                           areas.
--------------------------------------------------------------------------------



                                    Table 5.3
                     
                      Project Office Management Findings for
                                       NIMS

                      NAS Infrastructure Management System
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           The policy for project    Strength
               organization has a        office management is
               written policy for        contained in the
               execution of the          Acquisition Management
               software project.         System.

Commitment 2   Performance, cost, and    Performance, cost, and    Strength
               schedule baselines are    schedule baselines were
               supported.                developed and are being
                                         reviewed through the
                                         Acquisition Program
                                         Baseline process.

Ability 1      A project team that is    The product team is       Weakness
               responsible for           assigned the
               performing the project's  responsibility for
               software acquisition      acquisition management
               management activities     activities. However, no
               exists.                   one is assigned
                                         responsibility for
                                         software acquisition
                                         management activities.

Ability 2      Adequate resources for    No mechanism exists for   Weakness
               the project team and      identifying resources
               matrix support            required and for
               organization(s) are       ensuring that the needed
               provided for the          resources are provided
               duration of the software  to the project.
               acquisition project.

Ability 3      The project manager is    Officials said they       Observation
               permitted to alter        could change the cost,
               either the performance,   performance, or schedule
               cost, or schedule         baseline, but could not
               software acquisition      provide documentation to
               baseline while            support this.
               maintaining the other
               two constant.

Ability 4      The project team and      The acquisition           Observation
               matrix support            organization has no
               individual(s) have        guidance regarding
               experience or receive     training or experience
               training in project       requirements for project
               office software           participation.
               acquisition management
               activities.

Activity 1     The project team          The Integrated Program    Weakness
               documents its plans for   Plan and the Product
               software acquisition      Team Plan provide plans
               management activities.    for acquisition
                                         management, but these do
                                         not address software
                                         acquisition management
                                         activities.

Activity 2     The project team is       The product team is       Strength
               organized to accomplish   being organized to
               the project's             accomplish the project's
               objectives.               objectives.

Activity 3     The software acquisition  Too early in project      Not rated
               activities of the         life to assess: no
               project team are          software acquisition
               directed to accomplish    activities performed.
               the project's
               objectives.

Activity 4     The software acquisition  Too early in project      Not rated
               activities of the         life to assess: no
               project team are          software acquisition
               controlled.               activities performed.

Activity 5     Measurements are used to  The project has not       Not rated
               track project status,     reached a stage where
               execution, and funding    this activity applies.
               expenditures.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         to determine the status
               project office            of activities for any of
               management activities.    the key process areas.

Verification   The activities for        While the Integrated      Weakness
1              project office            Product Team leader
               management are reviewed   reviews the status of
               by acquisition            the contract and the
               organization management   contractor's cost and
               on a periodic basis.      schedule, he does not
                                         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              project office            leader reviews the
               management are reviewed   status of the contract
               by the project manager    and the contractor's
               on both a periodic and    cost and schedule, he
               event-driven basis.       does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------



                                    Table 5.4
                     
                      Project Office Management Findings for
                                       VSCS

                       Voice Switching and Control System
--------------------------------------------------------------------------------
              Key Practice                 Finding                      Rating
------------  ---------------------------  ---------------------------  --------
Commitment 1  The acquisition              FAA Order 1810.1F was cited  Weakness
              organization has a written   as the written policy, but
              policy for execution of the  it does not contain policy
              software project.            for software project
                                           execution.

Commitment 2  Performance, cost, and       Performance, cost, and       Weakness
              schedule baselines are       schedule baselines are not
              supported.                   supported.

Ability 1     A project team that is       The product team is          Strength
              responsible for performing   responsible for performing
              the project's software       software acquisition
              acquisition management       management activities.
              activities exists.

Ability 2     Adequate resources for the   No mechanism exists for      Weakness
              project team and matrix      identifying the resources
              support organization(s) are  required and for ensuring
              provided for the duration    that the needed resources
              of the software acquisition  are provided to the
              project.                     project.

Ability 3     The project manager is       The product team leader      Weakness
              permitted to alter either    does not have the
              the performance, cost, or    flexibility to alter cost,
              schedule software            performance, or schedule
              acquisition baseline while   while maintaining the other
              maintaining the other two    two.
              constant.

Ability 4     The project team and matrix  The acquisition              Observat
              support individual(s) have   organization has no          ion
              experience or receive        guidance regarding training
              training in project office   or experience requirements
              software acquisition         for project participation.
              management activities.

Activity 1    The project team documents   The Product Team Plan and    Strength
              its plans for software       the Contract Management
              acquisition management       Plan document acquisition
              activities.                  activities.

Activity 2    The project team is          The product team is          Strength
              organized to accomplish the  organized to accomplish the
              project's objectives.        project's objectives.

Activity 3    The software acquisition     While officials stated that  Weakness
              activities of the project    software activities of the
              team are directed to         team are directed to
              accomplish the project's     accomplish the project's
              objectives.                  objectives, documents
                                           provided do not specify the
                                           activities that the team
                                           members must accomplish.

Activity 4    The software acquisition     The software acquisition     Strength
              activities of the project    activities are controlled.
              team are controlled.

Activity 5    Measurements are used to     Measurements are used to     Strength
              track project status,        track project status,
              execution, and funding       execution, and funding
              expenditures.                expenditures.

Measurement   Measurements are made and    No internal process          Weakness
1             used to determine the        measurements are taken or
              status of the project        used to determine the
              office management            status of activities for
              activities.                  any of the key process
                                           areas.

Verification  The activities for project   While the Integrated         Weakness
1             office management are        Product Team leader reviews
              reviewed by acquisition      the status of the contract
              organization management on   and the contractor's cost
              a periodic basis.            and schedule, he does not
                                           review the status of the
                                           activities that are
                                           required to be performed
                                           for any of the key process
                                           areas.

Verification  The activities for project   While the product team       Weakness
2             office management are        leader reviews the status
              reviewed by the project      of the contract and the
              manager on both a periodic   contractor's cost and
              and event-driven basis.      schedule, he does not
                                           review the status of the
                                           activities that are
                                           required to be performed
                                           for any of the key process
                                           areas.
--------------------------------------------------------------------------------



                                    Table 5.5
                     
                      Project Office Management Findings for
                                       WARP

                          Weather and Radar Processor
--------------------------------------------------------------------------------
              Key Practice                 Finding                      Rating
------------  ---------------------------  ---------------------------  --------
Commitment 1  The acquisition              FAA Order 1810.1F was cited  Weakness
              organization has a written   as the written policy, but
              policy for execution of the  it does not contain policy
              software project.            for software project
                                           execution.

Commitment 2  Performance, cost, and       Performance, cost, and       Strength
              schedule baselines are       schedule baselines are
              supported.                   generated and supported.

Ability 1     A project team that is       The product team is          Strength
              responsible for performing   responsible for performing
              the project's software       software acquisition
              acquisition management       management activities.
              activities exists.

Ability 2     Adequate resources for the   No mechanism exists for      Weakness
              project team and matrix      identifying and for
              support organization(s) are  ensuring that the needed
              provided for the duration    resources are provided to
              of the software acquisition  the project.
              project.

Ability 3     The project manager is       The acquisition baseline     Weakness
              permitted to alter either    process does not allow the
              the performance, cost, or    project manager to alter
              schedule software            the baseline.
              acquisition baseline while
              maintaining the other two
              constant.

Ability 4     The project team and matrix  The acquisition              Observat
              support individual(s) have   organization has no          ion
              experience or receive        guidance regarding training
              training in project office   or experience requirements
              software acquisition         for project participation.
              management activities.

Activity 1    The project team documents   The product team documents   Strength
              its plans for software       its plans for software
              acquisition management       acquisition management
              activities.                  activities.

Activity 2    The project team is          The product team is          Strength
              organized to accomplish the  organized to accomplish the
              project's objectives.        project's objectives.

Activity 3    The software acquisition     The software acquisition     Strength
              activities of the project    activities of the project
              team are directed to         team are directed to
              accomplish the project's     accomplish the project's
              objectives.                  objectives.

Activity 4    The software acquisition     No individual is             Weakness
              activities of the project    controlling the software
              team are controlled.         acquisition activities of
                                           the product team.

Activity 5    Measurements are used to     Measurements are used to     Strength
              track project status,        track project status,
              execution, and funding       execution, and funding
              expenditures.                status.

Measurement   Measurements are made and    No internal process          Weakness
1             used to determine the        measurements are taken and
              status of the project        used to determine the
              office management            status of activities for
              activities.                  any of the key process
                                           areas.

Verification  The activities for project   While the Integrated         Weakness
1             office management are        Product Team leader reviews
              reviewed by acquisition      the status of the contract
              organization management on   and the contractor's cost
              a periodic basis.            and schedule, he does not
                                           review the status of the
                                           activities that are
                                           required to be performed
                                           for any of the key process
                                           areas.

Verification  The activities for project   While the product team       Weakness
2             office management are        leader reviews the status
              reviewed by the project      of the contract and the
              manager on both a periodic   contractor's cost and
              and event-driven basis.      schedule, he does not
                                           review the status of the
                                           activities that are
                                           required to be performed
                                           for any of the key process
                                           areas.
--------------------------------------------------------------------------------

   CONCLUSIONS
---------------------------------------------------------- Chapter 5:2

Numerous ad hoc project office management practices, including a
pervasive lack of measurement, analysis, and verification of project
status and progress, are limiting FAA's ability to meet ATC
modernization project commitments.  More discipline and definition in
this KPA is needed before ATC modernization teams can consistently
repeat project successes. 


CONTRACT TRACKING AND OVERSIGHT
============================================================ Chapter 6

The purpose of contract tracking and oversight is to ensure that (1)
the software development contractor performs according to the terms
of the contract; (2) needed contract changes are identified,
negotiated, and incorporated into the contract; and (3) contractor
performance issues are identified early, when they are easier to
address.  According to the SA-CMM, a repeatable contract tracking and
oversight process, among other things, includes (1) having a written
organizational policy for contract tracking and oversight, (2) having
a documented plan for contract tracking and oversight, (3) conducting
tracking and oversight activities in accordance with the plan, and
(4) ensuring that individuals performing contract tracking and
oversight are suitably experienced or trained. 


   FAA LACKS AN EFFECTIVE CONTRACT
   TRACKING AND OVERSIGHT PROCESS
---------------------------------------------------------- Chapter 6:1

Our past work on ATC modernization projects has raised concerns about
contract tracking and oversight.  For example, in 1994 we reported
that FAA did not provide adequate oversight of its contractor during
the initial development of the ISSS.\1 As a result, development
problems and lack of progress were not always recognized in a timely
manner.  The results of this software capability evaluation indicate
that these problems persist and pinpoint the underlying contract
tracking and oversight weaknesses.  For example, FAA does not have a
written organizational policy for contract tracking and oversight,
and most teams have no documented plan for contract tracking and
oversight activities.  Furthermore, the team that has a plan does not
always follow the plan, and none of the teams ensure that persons
responsible for managing software contracts have suitable experience
or training.  As a result, the product teams cannot formulate an
independent assessment of contract progress and are forced to rely on
data provided by the contractor.  Since contractor reports do not
always identify problems expeditiously, FAA is not always positioned
to correct them promptly. 

Figure 6.1 provides a comprehensive listing of the five projects'
strengths, weaknesses, and observations for the contractor tracking
and oversight KPA.  The specific findings supporting the practice
ratings cited in figure 6.1 are in tables 6.1 through 6.5. 

   Figure 6.1:  Contract Tracking
   and Oversight Summary

   (See figure in printed
   edition.)



   (See figure in printed
   edition.)




                                    Table 6.1
                     
                     Contract Tracking and Oversight Findings
                                   for ARTSIIIE

                      Automated Radar Terminal System IIIE
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           FAA Order 1810.1F was     Weakness
               organization has a        cited as the written
               written policy for the    policy, but it does not
               contract tracking and     provide the policy for
               oversight of the          contract tracking and
               contracted software       oversight.
               effort.

Commitment 2   Responsibility for the    Responsibility for        Strength
               contract tracking and     contract tracking and
               oversight activities is   oversight is designated.
               designated.

Commitment 3   The project team is       Contract specialists are  Strength
               supported by contracting  assigned to the product
               specialists in the        team.
               execution of the
               contract.

Ability 1      A group that is           Officials gave            Weakness
               responsible for managing  conflicting statements
               contract tracking and     as to who has the
               oversight activities      responsibility for
               exists.                   contract tracking and
                                         oversight activities,
                                         and could not provide
                                         documentation that
                                         formally delegates
                                         responsibility.

Ability 2      Adequate resources are    No mechanism exists for   Weakness
               provided for contract     identifying resources
               tracking and oversight    required and for
               activities.               ensuring that the needed
                                         resources are provided
                                         to the project.

Ability 3      Individuals performing    The acquisition           Observation
               contract tracking and     organization has no
               oversight activities      guidance regarding
               have experience or        training or experience
               receive training.         requirements for project
                                         participation.

Activity 1     The project team          Contract tracking and     Weakness
               documents its plans for   oversight plans do not
               contract tracking and     exist.
               oversight activities.

Activity 2     The project's contract    No plan exists,           Weakness
               tracking and oversight    therefore, activities
               activities are performed  cannot be performed in
               in accordance with its    accordance with it.
               plans.

Activity 3     The project team reviews  While officials stated    Observation
               required contractor       that the product team
               software planning         reviews contractor
               documents which, when     software planning
               satisfactory, are made    documents, they could
               part of the contractor's  not provide
               baseline.                 documentation to support
                                         this.

Activity 4     The project team, with    Periodic reviews are      Strength
               end user input, conducts  held.
               periodic reviews and
               interchanges with the
               contractor.

Activity 5     The project team tracks   The product team tracks   Strength
               the contractor's          the development of the
               development of the        software engineering
               software engineering      environment.
               environment required to
               support the software.

Activity 6     Any problems or issues    Issues found during       Strength
               found by the project      meetings and reviews are
               team during contract      documented in minutes
               tracking and oversight    and tracked.
               are recorded in the
               appropriate corrective
               action system and
               tracked to closure.

Activity 7     The project team          While product team        Weakness
               maintains the integrity   members (including the
               of the contract           contracting officer)
               throughout the contract   stated that the
               performance period.       contracting officer is
                                         responsible for
                                         maintaining the
                                         integrity of the
                                         contract, they could not
                                         provide any documents
                                         that support this.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         and used to determine
               contract tracking and     the status of activities
               oversight activities.     for any of the key
                                         process areas.

Verification   The activities for        While the Integrated      Weakness
1              contract tracking and     Product Team leader
               oversight are reviewed    reviews the status of
               by acquisition            the contract and the
               organization management   contractor's cost and
               on a periodic basis.      schedule, he does not
                                         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              contract tracking and     leader reviews the
               oversight are reviewed    status of the contract
               by the project manager    and the contractor's
               on both a periodic and    cost and schedule, he
               event-driven basis.       does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------



                                    Table 6.2
                     
                     Contract Tracking and Oversight Findings
                                     for DSR

                           Display System Replacement
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           FAA Order 1810.1F was     Weakness
               organization has a        cited as the written
               written policy for the    policy, but it does not
               contract tracking and     provide the policy for
               oversight of the          contract tracking and
               contracted software       oversight.
               effort.

Commitment 2   Responsibility for the    Officials gave            Weakness
               contract tracking and     conflicting answers on
               oversight activities is   who is responsible for
               designated.               contract tracking and
                                         oversight, and they
                                         could not provide
                                         documentation that
                                         formally delegates
                                         responsibility.

Commitment 3   The project team is       The product team is       Strength
               supported by contracting  supported by a
               specialists in the        contracting specialist
               execution of the          in execution of the
               contract.                 contract.

Ability 1      A group that is           The product team is       Strength
               responsible for managing  responsible for managing
               contract tracking and     contract tracking and
               oversight activities      oversight activities.
               exists.

Ability 2      Adequate resources are    No mechanism exists for   Weakness
               provided for contract     identifying resources
               tracking and oversight    required and for
               activities.               ensuring that the needed
                                         resources are provided
                                         to the project.

Ability 3      Individuals performing    The acquisition           Observation
               contract tracking and     organization has no
               oversight activities      guidance regarding
               have experience or        training or experience
               receive training.         requirements for project
                                         participation.

Activity 1     The project team          There are no written      Weakness
               documents its plans for   plans for contract
               contract tracking and     tracking and oversight
               oversight activities.     activities.

Activity 2     The project's contract    No plan exists,           Weakness
               tracking and oversight    therefore, the product
               activities are performed  team cannot perform in
               in accordance with its    accordance with it.
               plans.

Activity 3     The project team reviews  Reviews are used to       Strength
               required contractor       approve contract
               software planning         planning documents,
               documents which, when     which, when
               satisfactory, are made    satisfactory, are made
               part of the contractor's  part of the contractor's
               baseline.                 baseline.

Activity 4     The project team, with    While it was stated that  Observation
               end user input, conducts  continuous interactions
               periodic reviews and      with the contractor are
               interchanges with the     held, officials could
               contractor.               provide no documents to
                                         support this.

Activity 5     The project team tracks   The product team tracks   Strength
               the contractor's          the contractor's
               development of the        development of the
               software engineering      software engineering
               environment required to   environment required to
               support the software.     support the software.

Activity 6     Any problems or issues    The product team records  Strength
               found by the project      problems and issues
               team during contract      found during contract
               tracking and oversight    tracking and oversight
               are recorded in the       and tracks them to
               appropriate corrective    closure.
               action system and
               tracked to closure.

Activity 7     The project team          The product team          Strength
               maintains the integrity   maintains the integrity
               of the contract           of the contract
               throughout the contract   throughout the contract
               performance period.       performance period.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         and used to determine
               contract tracking and     the status of activities
               oversight activities.     for any of the key
                                         process areas.

Verification   The activities for        While the Integrated      Weakness
1              contract tracking and     Product Team leader
               oversight are reviewed    reviews the status of
               by acquisition            the contract and the
               organization management   contractor's cost and
               on a periodic basis.      schedule, he does not
                                         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              contract tracking and     leader reviews the
               oversight are reviewed    status of the contract
               by the project manager    and the contractor's
               on both a periodic and    cost and schedule, he
               event-driven basis.       does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------



                                    Table 6.3
                     
                     Contract Tracking and Oversight Findings
                                     for NIMS

                      NAS Infrastructure Management System
--------------------------------------------------------------------------------
               Key Practice              Findings                  Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           Since NIMS is not under   Not rated
               organization has a        contract yet, it was not
               written policy for the    evaluated against this
               contract tracking and     KPA.
               oversight of the
               contracted software
               effort.

Commitment 2   Responsibility for the    Too early to assess: the  Not rated
               contract tracking and     project has not reached
               oversight activities is   a stage where this
               designated.               applies.

Commitment 3   The project team is       Too early to assess: the  Not rated
               supported by contracting  project has not reached
               specialists in the        a stage where this
               execution of the          applies.
               contract.

Ability 1      A group that is           Too early to assess: the  Not rated
               responsible for managing  project has not reached
               contract tracking and     a stage where this
               oversight activities      applies.
               exists.

Ability 2      Adequate resources are    Too early to assess: the  Not rated
               provided for contract     project has not reached
               tracking and oversight    a stage where this
               activities.               applies.

Ability 3      Individuals performing    Too early to assess: the  Not rated
               contract tracking and     project has not reached
               oversight activities      a stage where this
               have experience or        applies.
               receive training.

Activity 1     The project team          Too early to assess: the  Not rated
               documents its plans for   project has not reached
               contract tracking and     a stage where this
               oversight activities.     applies.

Activity 2     The project's contract    Too early to assess: the  Not rated
               tracking and oversight    project has not reached
               activities are performed  a stage where this
               in accordance with its    applies.
               plans.

Activity 3     The project team reviews  Too early to assess: the  Not rated
               required contractor       project has not reached
               software planning         a stage where this
               documents which, when     applies.
               satisfactory, are made
               part of the contractor's
               baseline.

Activity 4     The project team, with    Too early to assess: the  Not rated
               end user input, conducts  project has not reached
               periodic reviews and      a stage where this
               interchanges with the     applies.
               contractor.

Activity 5     The project team tracks   Too early to assess: the  Not rated
               the contractor's          project has not reached
               development of the        a stage where this
               software engineering      applies.
               environment required to
               support the software.

Activity 6     Any problems or issues    Too early to assess: the  Not rated
               found by the project      project has not reached
               team during contract      a stage where this
               tracking and oversight    applies.
               are recorded in the
               appropriate corrective
               action system and
               tracked to closure.

Activity 7     The project team          Too early to assess: the  Not rated
               maintains the integrity   project has not reached
               of the contract           a stage where this
               throughout the contract   applies.
               performance period.

Measurement 1  Measurements are made     Too early to assess: the  Not rated
               and used to determine     project has not reached
               the status of the         a stage where this
               contract tracking and     applies.
               oversight activities.

Verification   The activities for        Too early to assess: the  Not rated
1              contract tracking and     project has not reached
               oversight are reviewed    a stage where this
               by acquisition            applies.
               organization management
               on a periodic basis.

Verification   The activities for        Too early to assess: the  Not rated
2              contract tracking and     project has not reached
               oversight are reviewed    a stage where this
               by the project manager    applies.
               on both a periodic and
               event-driven basis.
--------------------------------------------------------------------------------



                                    Table 6.4
                     
                     Contract Tracking and Oversight Findings
                                     for VSCS

                       Voice Switching and Control System
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           There is no policy on     Weakness
               organization has a        contract tracking and
               written policy for the    oversight.
               contract tracking and
               oversight of the
               contracted software
               effort.

Commitment 2   Responsibility for the    Responsibility for        Strength
               contract tracking and     contract tracking and
               oversight activities is   oversight is designated.
               designated.

Commitment 3   The project team is       The product team is       Strength
               supported by contracting  supported by a
               specialists in the        contracting specialist.
               execution of the
               contract.

Ability 1      A group that is           A group exists that is    Strength
               responsible for managing  responsible for managing
               contract tracking and     contract tracking and
               oversight activities      oversight.
               exists.

Ability 2      Adequate resources are    No mechanism exists for   Weakness
               provided for contract     identifying the
               tracking and oversight    resources required and
               activities.               for ensuring that the
                                         needed resources are
                                         provided to the project.

Ability 3      Individuals performing    The acquisition           Observation
               contract tracking and     organization has no
               oversight activities      guidance regarding
               have experience or        training or experience
               receive training.         requirements for project
                                         participation.

Activity 1     The project team          The product team has      Strength
               documents its plans for   documented its plans.
               contract tracking and
               oversight activities.

Activity 2     The project's contract    The product team could    Weakness
               tracking and oversight    not provide evidence
               activities are performed  that shows its
               in accordance with its    contracting tracking and
               plans.                    oversight activities are
                                         performed in accordance
                                         with its plans.

Activity 3     The project team reviews  While reviews are         Weakness
               required contractor       conducted, officials
               software planning         could not provide
               documents which, when     documentation that shows
               satisfactory, are made    the results are made
               part of the contractor's  part of the contractor's
               baseline.                 baseline.

Activity 4     The project team, with    The product team          Strength
               end user input, conducts  conducts periodic
               periodic reviews and      reviews and interchanges
               interchanges with the     with the contractor.
               contractor.

Activity 5     The project team tracks   The contractor is         Strength
               the contractor's          required to provide a
               development of the        list of common tools and
               software engineering      support equipment, which
               environment required to   the product team uses to
               support the software.     track the software
                                         engineering environment
                                         development.

Activity 6     Any problems or issues    The contractor has an     Strength
               found by the project      extensive software
               team during contract      development environment
               tracking and oversight    and problems or issues
               are recorded in the       are tracked to closure.
               appropriate corrective
               action system and
               tracked to closure.

Activity 7     The project team          There is no evidence      Weakness
               maintains the integrity   that either the
               of the contract           contracting officer or
               throughout the contract   the product team are
               performance period.       following the process
                                         and maintaining the
                                         integrity of the
                                         contract.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         or used to determine the
               contract tracking and     status of activities for
               oversight activities.     any of the key process
                                         areas.

Verification   The activities for        While the Integrated      Weakness
1              contract tracking and     Product Team leader
               oversight are reviewed    reviews the status of
               by acquisition            the contract and the
               organization management   contractor's cost and
               on a periodic basis.      schedule, he does not
                                         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              contract tracking and     leader reviews the
               oversight are reviewed    status of the contract
               by the project manager    and the contractor's
               on both a periodic and    cost and schedule, he
               event-driven basis.       does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------



                                    Table 6.5
                     
                     Contract Tracking and Oversight Findings
                                     for WARP

                          Weather and Radar Processor
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           There is no written       Weakness
               organization has a        policy for contract
               written policy for the    tracking and oversight
               contract tracking and     activities.
               oversight of the
               contracted software
               effort.

Commitment 2   Responsibility for the    The product team is       Strength
               contract tracking and     responsible for contract
               oversight activities is   tracking and oversight
               designated.               activities.

Commitment 3   The project team is       The product team is       Strength
               supported by contracting  supported by contracting
               specialists in the        specialists.
               execution of the
               contract.

Ability 1      A group that is           The product team is       Strength
               responsible for managing  collectively responsible
               contract tracking and     for contract tracking
               oversight activities      and oversight
               exists.                   activities.

Ability 2      Adequate resources are    No mechanism exists for   Weakness
               provided for contract     identifying and for
               tracking and oversight    ensuring that the needed
               activities.               resources are provided
                                         to the project.

Ability 3      Individuals performing    The acquisition           Observation
               contract tracking and     organization has no
               oversight activities      guidance regarding
               have experience or        training or experience
               receive training.         requirements for project
                                         participation.

Activity 1     The project team          Plans for contract        Weakness
               documents its plans for   tracking and oversight
               contract tracking and     activities are not
               oversight activities.     documented.

Activity 2     The project's contract    Since there is no         Weakness
               tracking and oversight    contract tracking and
               activities are performed  oversight plan, there is
               in accordance with its    no way to assess whether
               plans.                    activities are being
                                         performed in accordance
                                         with the plan.

Activity 3     The project team reviews  The product team reviews  Strength
               required contractor       required contractor
               software planning         software planning
               documents which, when     documents which, when
               satisfactory, are made    satisfactory, are made
               part of the contractor's  part of the contractor's
               baseline.                 baseline.

Activity 4     The project team, with    The product team          Strength
               end user input, conducts  conducts periodic
               periodic reviews and      reviews and interchanges
               interchanges with the     with the contractor.
               contractor.

Activity 5     The project team tracks   As a deliverable, the     Strength
               the contractor's          status of the software
               development of the        support environment
               software engineering      development is reviewed
               environment required to   and tracked.
               support the software.

Activity 6     Any problems or issues    The contracting           Strength
               found by the project      officer's technical
               team during contract      representative is
               tracking and oversight    responsible for managing
               are recorded in the       and tracking action
               appropriate corrective    items, and these are
               action system and         recorded in an
               tracked to closure.       appropriate correction
                                         system.

Activity 7     The project team          The contracting officer   Strength
               maintains the integrity   maintains the integrity
               of the contract           of the contract and is
               throughout the contract   responsible for doing so
               performance period.       throughout the contract
                                         performance period.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         and used to determine
               contract tracking and     the status of activities
               oversight activities.     for any of the key
                                         process areas.

Verification   The activities for        While the Integrated      Weakness
1              contract tracking and     Product Team leader
               oversight are reviewed    reviews the status of
               by acquisition            the contract and the
               organization management   contractor's cost and
               on a periodic basis.      schedule, he does not
                                         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              contract tracking and     leader reviews the
               oversight are reviewed    status of the contract
               by the project manager    and the contractor's
               on both a periodic and    cost and schedule, he
               event-driven basis.       does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------

--------------------
\1 Advanced Automation System:  Implications of Problems and Recent
Changes (GAO/T-RCED-94-188, Apr.  13, 1994). 


   CONCLUSIONS
---------------------------------------------------------- Chapter 6:2

To effectively and efficiently acquire software, FAA must have a
well-defined and enforced process that provides for proactive
tracking and oversight of its software development contractors. 
FAA's current process for ATC modernization contractor tracking and
oversight is ad hoc and reactive, thereby increasing the chances of
its ATC software acquisitions being late, costing more than expected,
and not performing as intended. 



EVALUATION
============================================================ Chapter 7

The purpose of evaluation, or testing, is to determine that the
acquired software products and services satisfy contract requirements
prior to acceptance.  According to the SA-CMM, a repeatable
evaluation process includes (1) documenting evaluation plans and
conducting evaluation activities in accordance with the plan, (2)
developing and managing evaluation requirements in conjunction with
developing software technical requirements, (3) incorporating
evaluation requirements into the solicitation and the resulting
contract, (4) tracking contractor performance of evaluation
activities for compliance with the contract, (5) ensuring that
adequate resources are provided for evaluation activities, and (6)
measuring and reporting on the status of evaluation activities to
management. 


   FAA IS STRONG IN MOST BUT NOT
   ALL EVALUATION KPA PRACTICES
---------------------------------------------------------- Chapter 7:1

All of the projects were strong in many evaluation practice areas. 
For example, all rated projects have documented test and evaluation
plans and conduct test and evaluation activities in accordance with
the plans.  In addition, most teams develop evaluation requirements
for contractor-conducted software tests concurrent with developing
software technical requirements, and all teams incorporate evaluation
requirements into the solicitation and resulting contract.  Also,
most teams track contractor performance of test activities for
compliance with the contract. 

Despite these many strengths, several weaknesses prevented FAA from
meeting this KPA.  For example, only one of the teams ensures that
adequate resources are provided for evaluation activities. 
Additionally, none of the teams measure and report on the status of
all evaluation activities to management.  As a result, management
does not have a complete and accurate picture of evaluation status
and progress, which could impair decision-making. 

Figure 7.1 provides a comprehensive listing of the five projects'
strengths, weaknesses, and observations for the evaluation KPA.  The
specific findings supporting the practice ratings cited in figure 7.1
are in tables 7.1 through 7.5. 

   Figure 7.1:  Evaluation Summary

   (See figure in printed
   edition.)



   (See figure in printed
   edition.)




                                    Table 7.1
                     
                         Evaluation Findings for ARTSIIIE

                      Automated Radar Terminal System IIIE
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           FAA Order 1810.4B is the  Strength
               organization has a        written policy.
               written policy for
               managing the evaluation
               of the acquired software
               products and services.

Commitment 2   Responsibility for        Evaluation                Strength
               evaluation activities is  responsibility is
               clearly defined.          clearly defined.

Ability 1      A group that is           The product team is       Strength
               responsible for           responsible for
               planning, managing, and   evaluation activities.
               performing evaluation
               activities for the
               project exists.

Ability 2      Adequate resources are    No mechanism exists for   Weakness
               provided for evaluation   identifying resources
               activities.               required and for
                                         ensuring that the needed
                                         resources are provided
                                         to the project.

Ability 3      Individuals performing    The acquisition           Observation
               evaluation activities     organization has no
               have experience or        guidance regarding
               receive training.         training or experience
                                         requirements for project
                                         participation.

Ability 4      Members of the project    The product team did not  Weakness
               team and groups           receive orientation on
               supporting the software   the evaluation approach.
               acquisition receive
               orientation on the
               objectives of the
               evaluation approach.

Activity 1     The project team          The Test and Evaluation   Strength
               documents its plans for   Master Plan delineates
               evaluation activities.    all evaluation
                                         activities.

Activity 2     The project's evaluation  Evaluation activities     Strength
               activities are performed  are performed in
               in accordance with its    accordance with plans.
               plans.

Activity 3     Evaluation requirements   Evaluation requirements   Strength
               are developed and         are developed and
               managed in conjunction    managed in conjunction
               with development of the   with development of the
               system or software        system or software
               technical requirements.   technical requirements.

Activity 4     The evaluation            Evaluation requirements   Strength
               requirements are          are part of the
               incorporated into the     contract.
               solicitation and
               resulting contract.

Activity 5     The project team tracks   The evaluation            Strength
               contractor's performance  activities performed by
               in terms of evaluation    the contractor are
               requirements for          tracked.
               compliance with the
               contract.

Activity 6     Planned evaluations are   Planned evaluations are   Strength
               performed on the          performed to ensure that
               acquired software         technical and contract
               products and services     requirements are met
               prior to acceptance for   prior to acceptance.
               operational use.

Activity 7     Results of the            Technical and contract    Strength
               evaluations are analyzed  requirements are met
               and compared to the       prior to acceptance.
               contract's requirements
               to establish a basis for
               acceptance.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         and used to determine
               evaluation activities.    the status of activities
                                         for any of the key
                                         process areas.

Verification   The activities for        While the Integrated      Weakness
1              evaluation are reviewed   Product Team leader
               with acquisition          reviews the status of
               organization management   the contract and the
               on a periodic basis.      contractor's cost and
                                         schedule, he does not
                                         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              evaluation are reviewed   leader reviews the
               with the project manager  status of the contract
               on both a periodic and    and the contractor's
               event-driven basis.       cost and schedule, he
                                         does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------



                                    Table 7.2
                     
                           Evaluation Findings for DSR

                           Display System Replacement
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           FAA Order 1810.4B is the  Strength
               organization has a        written policy.
               written policy for
               managing the evaluation
               of the acquired software
               products and services.

Commitment 2   Responsibility for        The product team leader   Strength
               evaluation activities is  is responsible for
               clearly defined.          evaluation activities.

Ability 1      A group that is           The test and maintenance  Strength
               responsible for           leader, along with the
               planning, managing, and   product team, are
               performing evaluation     responsible for
               activities for the        planning, managing, and
               project exists.           performing evaluation
                                         activities.

Ability 2      Adequate resources are    Adequate resources are    Strength
               provided for evaluation   provided for evaluation
               activities.               activities.

Ability 3      Individuals performing    The acquisition           Observation
               evaluation activities     organization has no
               have experience or        guidance regarding
               receive training.         training or experience
                                         requirements for project
                                         participation.

Ability 4      Members of the project    Members of the project    Strength
               team and groups           team received
               supporting the software   orientation on the
               acquisition receive       evaluation approach.
               orientation on the
               objectives of the
               evaluation approach.

Activity 1     The project team          The product team          Strength
               documents its plans for   documents its plans for
               evaluation activities.    evaluation activities.

Activity 2     The project's evaluation  The project's evaluation  Strength
               activities are performed  activities are performed
               in accordance with its    in accordance with its
               plans.                    plans.

Activity 3     Evaluation requirements   Evaluation requirements   Strength
               are developed and         are developed in
               managed in conjunction    conjunction with the
               with development of the   system requirements.
               system or software
               technical requirements.

Activity 4     The evaluation            The evaluation            Strength
               requirements are          requirements are
               incorporated into the     incorporated into the
               solicitation and          contract.
               resulting contract.

Activity 5     The project team tracks   The product team tracks   Strength
               contractor's performance  the contractor's
               in terms of evaluation    performance, in terms of
               requirements for          evaluation requirements,
               compliance with the       using the A-level and B-
               contract.                 level specifications in
                                         the contract as
                                         criteria.

Activity 6     Planned evaluations are   Planned evaluations are   Strength
               performed on the          performed on the
               acquired software         acquired software
               products and services     products and services
               prior to acceptance for   prior to acceptance for
               operational use.          operational use.

Activity 7     Results of the            Results of evaluations    Strength
               evaluations are analyzed  are analyzed and
               and compared to the       compared to the
               contract's requirements   contract's requirements
               to establish a basis for  to establish a basis for
               acceptance.               acceptance.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         and used to determine
               evaluation activities.    the status of activities
                                         for any of the key
                                         process areas.

Verification   The activities for        While the Integrated      Weakness
1              evaluation are reviewed   Product Team leader
               with acquisition          reviews the status of
               organization management   the contract and the
               on a periodic basis.      contractor's cost and
                                         schedule, he does not
                                         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              evaluation are reviewed   leader reviews the
               with the project manager  status of the contract
               on both a periodic and    and the contractor's
               event-driven basis.       cost and schedule, he
                                         does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------



                                    Table 7.3
                     
                           Evaluation Findings for NIMS

                      NAS Infrastructure Management System
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           The Acquisition           Strength
               organization has a        Management System is the
               written policy for        written policy.
               managing the evaluation
               of the acquired software
               products and services.

Commitment 2   Responsibility for        Officials stated that     Observation
               evaluation activities is  the test leader is
               clearly defined.          responsible for
                                         evaluation activities;
                                         however, they could not
                                         provide documentation to
                                         support this.

Ability 1      A group that is           The product team has      Strength
               responsible for           responsibility for
               planning, managing, and   planning, managing, and
               performing evaluation     performing evaluation
               activities for the        activities.
               project exists.

Ability 2      Adequate resources are    No mechanism exists for   Weakness
               provided for evaluation   identifying resources
               activities.               required and for
                                         ensuring that the needed
                                         resources are provided
                                         to the project.

Ability 3      Individuals performing    The acquisition           Observation
               evaluation activities     organization has no
               have experience or        guidance regarding
               receive training.         training or experience
                                         requirements for project
                                         participation.

Ability 4      Members of the project    Orientation on the        Weakness
               team and groups           objectives of the
               supporting the software   evaluation approach was
               acquisition receive       not received.
               orientation on the
               objectives of the
               evaluation approach.

Activity 1     The project team          The project team          Strength
               documents its plans for   documents its plans for
               evaluation activities.    evaluation activities.

Activity 2     The project's evaluation  NIMS has not yet reached  Not rated
               activities are performed  the stage where
               in accordance with its    evaluation activities
               plans.                    are being performed.

Activity 3     Evaluation requirements   Too early to assess:      Observation
               are developed and         system software
               managed in conjunction    requirements are not
               with development of the   developed sufficiently
               system or software        to develop evaluation
               technical requirements.   requirements.

Activity 4     The evaluation            The evaluation            Strength
               requirements are          requirements are
               incorporated into the     incorporated into the
               solicitation and          solicitation through the
               resulting contract.       statement of work, which
                                         will be part of the
                                         contract.

Activity 5     The project team tracks   NIMS has not yet reached  Not rated
               contractor's performance  the stage where
               in terms of evaluation    evaluation activities
               requirements for          are being performed.
               compliance with the
               contract.

Activity 6     Planned evaluations are   NIMS has not yet reached  Not rated
               performed on the          the stage where
               acquired software         operational evaluation
               products and services     activities are being
               prior to acceptance for   performed.
               operational use.

Activity 7     Results of the            NIMS has not yet reached  Not rated
               evaluations are analyzed  the stage where
               and compared to the       evaluation activities
               contract's requirements   are analyzed and
               to establish a basis for  compared to the
               acceptance.               contract.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         to determine the status
               evaluation activities.    of activities for any of
                                         the key process areas.

Verification   The activities for        While the Integrated      Weakness
1              evaluation are reviewed   Product Team leader
               with acquisition          reviews the status of
               organization management   the contract and the
               on a periodic basis.      contractor's cost and
                                         schedule, he does not
                                         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              evaluation are reviewed   leader reviews the
               with the project manager  status of the contract
               on both a periodic and    and the contractor's
               event-driven basis.       cost and schedule, he
                                         does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------



                                    Table 7.4
                     
                           Evaluation Findings for VSCS

                       Voice Switching and Control System
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           FAA Order 1810.4B         Strength
               organization has a        provides policy
               written policy for        guidance.
               managing the evaluation
               of the acquired software
               products and services.

Commitment 2   Responsibility for        The Test and Evaluation   Strength
               evaluation activities is  Master Plan defines the
               clearly defined.          responsibility for
                                         evaluation.

Ability 1      A group that is           A group is assigned       Strength
               responsible for           responsibility for
               planning, managing, and   planning, managing, and
               performing evaluation     performing evaluation
               activities for the        activities.
               project exists.

Ability 2      Adequate resources are    No mechanism exists for   Weakness
               provided for evaluation   identifying the
               activities.               resources required and
                                         for ensuring that the
                                         needed resources are
                                         provided to the project.

Ability 3      Individuals performing    The acquisition           Observation
               evaluation activities     organization has no
               have experience or        guidance regarding
               receive training.         training or experience
                                         requirements for project
                                         participation.

Ability 4      Members of the project    An orientation was        Strength
               team and groups           conducted.
               supporting the software
               acquisition receive
               orientation on the
               objectives of the
               evaluation approach.

Activity 1     The project team          Test and evaluation       Strength
               documents its plans for   activities are
               evaluation activities.    documented.

Activity 2     The project's evaluation  Evaluation activities     Strength
               activities are performed  are performed in
               in accordance with its    accordance with plans.
               plans.

Activity 3     Evaluation requirements   Test and evaluation       Strength
               are developed and         requirements are
               managed in conjunction    developed and managed in
               with development of the   conjunction with system
               system or software        requirements.
               technical requirements.

Activity 4     The evaluation            Evaluation requirements   Strength
               requirements are          are part of the
               incorporated into the     contract.
               solicitation and
               resulting contract.

Activity 5     The project team tracks   The contractor's          Strength
               contractor's performance  performance in terms of
               in terms of evaluation    evaluation requirements
               requirements for          is tracked through
               compliance with the       weekly meetings.
               contract.

Activity 6     Planned evaluations are   Evaluation procedures     Strength
               performed on the          are documented in the
               acquired software         Test and Evaluation
               products and services     Master Plan and are
               prior to acceptance for   performed prior to
               operational use.          acceptance.

Activity 7     Results of the            FAA conducts factory      Strength
               evaluations are analyzed  acceptance tests and
               and compared to the       compares results to the
               contract's requirements   contract's requirements.
               to establish a basis for
               acceptance.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         or used to determine the
               evaluation activities.    status of activities for
                                         any of the key process
                                         areas.

Verification   The activities for        While the Integrated      Weakness
1              evaluation are reviewed   Product Team leader
               with acquisition          reviews the status of
               organization management   the contract and the
               on a periodic basis.      contractor's cost and
                                         schedule, he does not
                                         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              evaluation are reviewed   leader reviews the
               with the project manager  status of the contract
               on both a periodic and    and the contractor's
               event-driven basis.       cost and schedule, he
                                         does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------



                                    Table 7.5
                     
                           Evaluation Findings for WARP

                          Weather and Radar Processor
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           FAA Order 1810.4B is the  Strength
               organization has a        written policy.
               written policy for
               managing the evaluation
               of the acquired software
               products and services.

Commitment 2   Responsibility for        Responsibility for        Strength
               evaluation activities is  evaluation activities
               clearly defined.          has been clearly
                                         defined.

Ability 1      A group that is           A group is responsible    Strength
               responsible for           for evaluation
               planning, managing, and   activities for the
               performing evaluation     project.
               activities for the
               project exists.

Ability 2      Adequate resources are    No mechanism exists for   Weakness
               provided for evaluation   identifying and for
               activities.               ensuring that the needed
                                         resources are provided
                                         to the project.

Ability 3      Individuals performing    The acquisition           Observation
               evaluation activities     organization has no
               have experience or        guidance regarding
               receive training.         training or experience
                                         requirements for project
                                         participation.

Ability 4      Members of the project    Orientation was not       Observation
               team and groups           needed because the test
               supporting the software   leader and his team were
               acquisition receive       familiar with the
               orientation on the        evaluation approach.
               objectives of the
               evaluation approach.

Activity 1     The project team          The Test and Evaluation   Strength
               documents its plans for   Master Plan documents
               evaluation activities.    the team's plan for
                                         evaluation activities.

Activity 2     The project's evaluation  WARP has not reached the  Not rated
               activities are performed  stage where evaluation
               in accordance with its    activities are being
               plans.                    performed and can be
                                         compared to a plan.

Activity 3     Evaluation requirements   Evaluation requirements   Strength
               are developed and         are developed and
               managed in conjunction    managed in conjunction
               with development of the   with development of
               system or software        system or software
               technical requirements.   technical requirements.

Activity 4     The evaluation            The evaluation            Strength
               requirements are          requirements are
               incorporated into the     incorporated into the
               solicitation and          solicitation and
               resulting contract.       resulting contract.

Activity 5     The project team tracks   WARP has not reached the  Not rated
               contractor's performance  stage where the
               in terms of evaluation    contractor is performing
               requirements for          evaluation activities
               compliance with the       that can be tracked for
               contract.                 compliance with
                                         contract.

Activity 6     Planned evaluations are   WARP has not yet reached  Not rated
               performed on the          the stage where
               acquired software         evaluations are being
               products and services     performed.
               prior to acceptance for
               operational use.

Activity 7     Results of the            WARP has not yet reached  Not rated
               evaluations are analyzed  the stage where
               and compared to the       evaluations are being
               contract's requirements   performed.
               to establish a basis for
               acceptance.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         and used to determine
               evaluation activities.    the status of activities
                                         for any of the key
                                         process areas.

Verification   The activities for        While the Integrated      Weakness
1              evaluation are reviewed   Product Team leader
               with acquisition          reviews the status of
               organization management   the contract and the
               on a periodic basis.      contractor's cost and
                                         schedule, he does not
                                         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              evaluation are reviewed   leader reviews the
               with the project manager  status of the contract
               on both a periodic and    and the contractor's
               event-driven basis.       cost and schedule, he
                                         does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------

   CONCLUSIONS
---------------------------------------------------------- Chapter 7:2

FAA performs most but not all evaluation KPA practices.  To satisfy
all evaluation practices and thereby have reasonable assurance that
its software acquisition projects will be effectively evaluated and
tested on a repeatable basis, FAA must ensure that its product teams
identify evaluation resource, training, and experience needs, and
that they measure and brief management on the status of all
evaluation activities. 



TRANSITION AND SUPPORT
============================================================ Chapter 8

The purpose of transition and support is to provide for the effective
and efficient "hand-off" of the acquired software products to the
support organization responsible for software maintenance.  According
to the SA-CMM, repeatable transition and support processes, among
other things, include (1) having a written policy for transitioning
software products to the support organization, (2) designating a
group that is responsible for coordinating transition and support
activities, (3) having a complete inventory of all software and
related items that are to be transitioned, (4) including members of
the support organization in transition and support planning, (5)
requiring the support organization to demonstrate its capability to
modify and support the software, and (6) measuring and reporting to
management on the status of transition and support activities. 


   TRANSITION AND SUPPORT NOT
   BEING PERFORMED EFFECTIVELY
---------------------------------------------------------- Chapter 8:1

All of the projects were strong in several transition and support
practice areas.  For example, FAA has a written policy for
transitioning software products to the support organization. 
Additionally, all five projects have designated a group responsible
for transition and support.  However, various weaknesses in other
practices prevented FAA from satisfying this KPA.  In particular,
some of the teams lack a complete inventory of all the software and
related products to be transitioned, thus jeopardizing the efforts of
the support organization to effectively maintain the full software
configuration.  Additionally, one team did not include the software
support organization in planning for transition and support, and some
teams do not have plans to require the support organization to
demonstrate its capability to maintain the software after transition. 
As a result, support problems, such as the inability to perform
required maintenance, may result.  Further, none of the projects are
measuring and reporting to management on the status of transition and
support activities, precluding management from addressing transition
and support problems expeditiously. 

Figure 8.1 provides a comprehensive listing of the five projects'
strengths, weaknesses, and observations for the transition and
support KPA.  The specific findings supporting the practice ratings
cited in figure 8.1 are in tables 8.1 through 8.5. 

   Figure 8.1:  Transition and
   Support Summary

   (See figure in printed
   edition.)



   (See figure in printed
   edition.)




                                    Table 8.1
                     
                       Transition and Support Findings for
                                     ARTSIIIE

                      Automated Radar Terminal System IIIE
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           FAA Order 1800.58 is the  Strength
               organization has a        written policy.
               written policy for the
               transitioning of
               software products to the
               support organization.

Commitment 2   The acquisition           The Product Team Plan     Weakness
               organization ensures      identifies the
               that the software         organization responsible
               support organization is   for software support,
               involved in planning for  but transition is not
               transition and support.   mentioned.

Ability 1      A group that is           A group responsible for   Strength
               responsible for           coordinating the
               coordinating the          transition to support
               transition and support    activities exists.
               activities exists.

Ability 2      Responsibility for        Responsibility for        Strength
               transition and support    transition and support
               activities is             activities is
               designated.               designated.

Ability 3      Adequate resources are    No mechanism exists for   Weakness
               provided for transition   identifying resources
               and support activities.   required and for
                                         ensuring that the needed
                                         resources are provided
                                         to the project.

Ability 4      The organization          The Product Team Plan     Strength
               responsible for           designates
               providing support of the  responsibility for
               software products is      providing support of the
               identified no later than  software products before
               release of the            release of the
               solicitation.             solicitation.

Ability 5      The software support      The software support      Strength
               organization has a        organization has a
               complete inventory of     complete inventory of
               all software and related  all software and related
               items that are to be      items that are to be
               transitioned.             transitioned.

Ability 6      Individuals performing    The acquisition           Observation
               transition and support    organization has no
               activities have           guidance regarding
               experience or receive     training or experience
               training.                 requirements for project
                                         participation.

Ability 7      The members of            Orientation was not       Weakness
               organizations             given.
               interfacing with the
               transition and support
               activities receive
               orientation on the
               salient aspects of the
               transition and support
               activities.

Activity 1     The project team          The product team          Strength
               documents its plans for   documents its plans for
               transition and support    transition and support
               activities.               activities.

Activity 2     The project team's        The product team's        Strength
               transition and support    activities are being
               activities are performed  conducted in accordance
               in accordance with its    with its plans.
               plans.

Activity 3     The project team          No certification          Weakness
               transfers responsibility  procedures to test the
               for the software          capability of the
               products only after the   support organization was
               support organization      provided.
               demonstrates its
               capability to modify and
               support the software
               products.

Activity 4     The project team          The product team          Strength
               oversees the              oversees the
               configuration control of  configuration management
               the software products     of the software
               throughout the            throughout transition.
               transition.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         and used to determine
               transition and support    the status of activities
               activities.               for any of the key
                                         process areas.

Verification   The activities for        While the Integrated      Weakness
1              transition and support    Product Team leader
               are reviewed by           reviews the status of
               acquisition and software  the contract and the
               support organizations'    contractor's cost and
               management on a periodic  schedule, he does not
               basis.                    review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              transition and support    leader reviews the
               are reviewed by the       status of the contract
               project manager on both   and the contractor's
               a periodic and event-     cost and schedule, he
               driven basis.             does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------



                                    Table 8.2
                     
                     Transition and Support Findings for DSR

                           Display System Replacement
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           FAA has a written policy  Strength
               organization has a        for transition and
               written policy for the    support.
               transitioning of
               software products to the
               support organization.

Commitment 2   The acquisition           The Product Team Plan     Strength
               organization ensures      identifies the
               that the software         individual responsible
               support organization is   for support planning.
               involved in planning for
               transition and support.

Ability 1      A group that is           The product team is       Strength
               responsible for           responsible for
               coordinating the          transition and support
               transition and support    activities.
               activities exists.

Ability 2      Responsibility for        Responsibility is         Strength
               transition and support    assigned to the product
               activities is             team for transition and
               designated.               support activities.

Ability 3      Adequate resources are    No mechanism exists for   Weakness
               provided for transition   identifying resources
               and support activities.   required and for
                                         ensuring that the needed
                                         resources are provided
                                         to the project.

Ability 4      The organization          Although DSR is in the    Weakness
               responsible for           development stage, the
               providing support of the  support organization has
               software products is      not been identified.
               identified no later than
               release of the
               solicitation.

Ability 5      The software support      While documents are       Weakness
               organization has a        planned for
               complete inventory of     configuration audits as
               all software and related  part of the transition
               items that are to be      process, a complete
               transitioned.             inventory does not
                                         exist.

Ability 6      Individuals performing    The acquisition           Observation
               transition and support    organization has no
               activities have           guidance regarding
               experience or receive     training or experience
               training.                 requirements for project
                                         participation.

Ability 7      The members of            While officials stated    Observation
               organizations             that orientation had
               interfacing with the      occurred, they could not
               transition and support    provide documents to
               activities receive        support this.
               orientation on the
               salient aspects of the
               transition and support
               activities.

Activity 1     The project team          The Program               Strength
               documents its plans for   Implementation Plan
               transition and support    contains the plans for
               activities.               transition and support.

Activity 2     The project team's        Transition planning is    Weakness
               transition and support    being delayed pending a
               activities are performed  decision on who will
               in accordance with its    provide maintenance.
               plans.

Activity 3     The project team          There is a plan for the   Strength
               transfers responsibility  support organization to
               for the software          demonstrate its
               products only after the   capabilities prior to
               support organization      transition of
               demonstrates its          responsibilities.
               capability to modify and
               support the software
               products.

Activity 4     The project team          The product team          Strength
               oversees the              oversees the contractor,
               configuration control of  who will maintain
               the software products     configuration management
               throughout the            throughout the
               transition.               transition.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         and used to determine
               transition and support    the status of activities
               activities.               for any of the key
                                         process areas.

Verification   The activities for        While the Integrated      Weakness
1              transition and support    Product Team leader
               are reviewed by           reviews the status of
               acquisition and software  the contract and the
               support organizations'    contractor's cost and
               management on a periodic  schedule, he does not
               basis.                    review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              transition and support    leader reviews the
               are reviewed by the       status of the contract
               project manager on both   and the contractor's
               a periodic and event-     cost and schedule, he
               driven basis.             does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------



                                    Table 8.3
                     
                     Transition and Support Findings for NIMS

                      NAS Infrastructure Management System
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           The Acquisition           Strength
               organization has a        Management System is the
               written policy for the    written policy.
               transitioning of
               software products to the
               support organization.

Commitment 2   The acquisition           The software support      Strength
               organization ensures      organization is
               that the software         represented in the
               support organization is   product team.
               involved in planning for
               transition and support.

Ability 1      A group that is           Officials gave            Weakness
               responsible for           conflicting answers as
               coordinating the          to who is responsible
               transition and support    for coordinating the
               activities exists.        transition and support
                                         activities, and they
                                         could not provide
                                         documentation that
                                         formally designates
                                         responsibility.

Ability 2      Responsibility for        Responsibility for        Strength
               transition and support    transition and support
               activities is             activities is
               designated.               designated.

Ability 3      Adequate resources are    It is too early in the    Not rated
               provided for transition   project's cycle to
               and support activities.   evaluate this ability.

Ability 4      The organization          It is too early in the    Not rated
               responsible for           project's cycle to
               providing support of the  evaluate this ability.
               software products is
               identified no later than
               release of the
               solicitation.

Ability 5      The software support      It is too early in the    Not rated
               organization has a        project's cycle to
               complete inventory of     evaluate this ability.
               all software and related
               items that are to be
               transitioned.

Ability 6      Individuals performing    It is too early in the    Not rated
               transition and support    project's cycle to
               activities have           evaluate this ability.
               experience or receive
               training.

Ability 7      The members of            It is too early in the    Not rated
               organizations             project's cycle to
               interfacing with the      evaluate this ability.
               transition and support
               activities receive
               orientation on the
               salient aspects of the
               transition and support
               activities.

Activity 1     The project team          It is too early in the    Not rated
               documents its plans for   project's cycle to
               transition and support    evaluate this activity.
               activities.

Activity 2     The project team's        It is too early in the    Not rated
               transition and support    project's cycle to
               activities are performed  evaluate this activity.
               in accordance with its
               plans.

Activity 3     The project team          It is too early in the    Not rated
               transfers responsibility  project's cycle to
               for the software          evaluate this activity.
               products only after the
               support organization
               demonstrates its
               capability to modify and
               support the software
               products.

Activity 4     The project team          It is too early in the    Not rated
               oversees the              project's cycle to
               configuration control of  evaluate this activity.
               the software products
               throughout the
               transition.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         to determine the status
               transition and support    of activities for any of
               activities.               the key process areas.

Verification   The activities for        While the Integrated      Weakness
1              transition and support    Product Team leader
               are reviewed by           reviews the status of
               acquisition and software  the contract and the
               support organizations'    contractor's cost and
               management on a periodic  schedule, he does not
               basis.                    review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              transition and support    leader reviews the
               are reviewed by the       status of the contract
               project manager on both   and the contractor's
               a periodic and event-     cost and schedule, he
               driven basis.             does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------



                                    Table 8.4
                     
                     Transition and Support Findings for VSCS

                       Voice Switching and Control System
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           A written policy for      Observation
               organization has a        transition exists;
               written policy for the    however, the officials
               transitioning of          interviewed did not
               software products to the  identify it.
               support organization.

Commitment 2   The acquisition           The acquisition           Strength
               organization ensures      organization ensured
               that the software         that the software
               support organization is   support organization was
               involved in planning for  involved in planning for
               transition and support.   transition and support.

Ability 1      A group that is           The product team is       Strength
               responsible for           responsible for
               coordinating the          coordinating transition
               transition and support    and support activities.
               activities exists.

Ability 2      Responsibility for        Responsibility for        Strength
               transition and support    transition and support
               activities is             activities is
               designated.               designated.

Ability 3      Adequate resources are    No mechanism exists for   Weakness
               provided for transition   identifying the
               and support activities.   resources required and
                                         for ensuring that the
                                         needed resources are
                                         provided to the project.

Ability 4      The organization          The organization          Strength
               responsible for           responsible was
               providing support of the  identified before the
               software products is      release of the
               identified no later than  solicitation.
               release of the
               solicitation.

Ability 5      The software support      The software support      Weakness
               organization has a        organization does not
               complete inventory of     have a complete
               all software and related  inventory of all
               items that are to be      software and related
               transitioned.             items that are to be
                                         transitioned.

Ability 6      Individuals performing    The acquisition           Observation
               transition and support    organization has no
               activities have           guidance regarding
               experience or receive     training or experience
               training.                 requirements for project
                                         participation.

Ability 7      The members of            While officials said      Observation
               organizations             that members of
               interfacing with the      organizations
               transition and support    interfacing with the
               activities receive        transition and support
               orientation on the        activities received
               salient aspects of the    orientation, they could
               transition and support    not provide
               activities.               documentation to support
                                         this.

Activity 1     The project team          Only a draft plan         Weakness
               documents its plans for   exists.
               transition and support
               activities.

Activity 2     The project team's        There is no plan for      Weakness
               transition and support    transition and support,
               activities are performed  therefore, the activity
               in accordance with its    cannot be performed in
               plans.                    accordance with it.

Activity 3     The project team          No demonstrations were    Weakness
               transfers responsibility  held for the support
               for the software          organization to
               products only after the   demonstrate its
               support organization      capability to support
               demonstrates its          the software products.
               capability to modify and
               support the software
               products.

Activity 4     The project team          Since there is no         Weakness
               oversees the              transition and support
               configuration control of  plan, there is no
               the software products     evidence that this
               throughout the            activity is being
               transition.               performed.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         or used to determine the
               transition and support    status of activities for
               activities.               any of the key process
                                         areas.

Verification   The activities for        While the Integrated      Weakness
1              transition and support    Product Team leader
               are reviewed by           reviews the status of
               acquisition and software  the contract and the
               support organizations'    contractor's cost and
               management on a periodic  schedule, he does not
               basis.                    review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         processes.

Verification   The activities for        While the product team    Weakness
2              transition and support    leader reviews the
               are reviewed by the       status of the contract
               project manager on both   and the contractor's
               a periodic and event-     cost and schedule, he
               driven basis.             does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------



                                    Table 8.5
                     
                     Transition and Support Findings for WARP

                          Weather and Radar Processor
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           There is a written        Strength
               organization has a        policy for transition
               written policy for the    and support.
               transitioning of
               software products to the
               support organization.

Commitment 2   The acquisition           The support organization  Strength
               organization ensures      is part of the product
               that the software         team and is involved in
               support organization is   planning for transition.
               involved in planning for
               transition and support.

Ability 1      A group that is           The product team has the  Strength
               responsible for           responsibility for
               coordinating the          coordinating the
               transition and support    transition and support
               activities exists.        activities.

Ability 2      Responsibility for        Responsibility for        Strength
               transition and support    transition and support
               activities is             activities is designated
               designated.               to the product team.

Ability 3      Adequate resources are    No mechanism exists for   Weakness
               provided for transition   identifying and for
               and support activities.   ensuring that the needed
                                         resources are provided
                                         to the project.

Ability 4      The organization          The organization          Weakness
               responsible for           responsible for
               providing support of the  providing support of the
               software products is      software products was
               identified no later than  not chosen before the
               release of the            release of the
               solicitation.             solicitation.

Ability 5      The software support      It is too early in the    Not rated
               organization has a        project's development
               complete inventory of     for it to have developed
               all software and related  an inventory.
               items that are to be
               transitioned.

Ability 6      Individuals performing    The acquisition           Observation
               transition and support    organization has no
               activities have           guidance regarding
               experience or receive     training or experience
               training.                 requirements for project
                                         participation.

Ability 7      The members of            It is too early in the    Not rated
               organizations             project's development
               interfacing with the      for it to define
               transition and support    support.
               activities receive
               orientation on the
               salient aspects of the
               transition and support
               activities.

Activity 1     The project team          The Product Team Plan     Strength
               documents its plans for   documents the initial
               transition and support    plans for transition and
               activities.               support activities.

Activity 2     The project team's        It is too early in the    Not rated
               transition and support    project's development
               activities are performed  for it to be evaluated
               in accordance with its    against this key
               plans.                    practice.

Activity 3     The project team          Officials gave            Observation
               transfers responsibility  conflicting answers as
               for the software          to whether or not the
               products only after the   demonstration has been
               support organization      planned.
               demonstrates its
               capability to modify and
               support the software
               products.

Activity 4     The project team          It is too early in the    Not rated
               oversees the              project's development
               configuration control of  for it to be evaluated
               the software products     against this key
               throughout the            practice.
               transition.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         and used to determine
               transition and support    the status of activities
               activities.               for any of the key
                                         process areas.

Verification   The activities for        While the Integrated      Weakness
1              transition and support    Product Team leader
               are reviewed by           reviews the status of
               acquisition and software  the contract and the
               support organizations'    contractor's cost and
               management on a periodic  schedule, he does not
               basis.                    review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              transition and support    leader reviews the
               are reviewed by the       status of the contract
               project manager on both   and the contractor's
               a periodic and event-     cost and schedule, he
               driven basis.             does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------

   CONCLUSIONS
---------------------------------------------------------- Chapter 8:2

FAA's transition and support process for its ATC modernization
suffers from weaknesses which render it undefined and undisciplined. 
In light of FAA's enormous investment in ATC-related software and the
fact that over 65 percent of the life cycle cost of software is
incurred during maintenance, it is essential that these weaknesses be
corrected. 



ACQUISITION RISK MANAGEMENT
============================================================ Chapter 9

SEI defines risk as the possibility of suffering a loss.  The purpose
of acquisition risk management is to formally identify risks as early
as possible and adjust the acquisition to mitigate those risks. 
According to the SA-CMM, an effective risk management process, among
other things, includes (1) having a written policy on acquisition
risk management, (2) developing a software acquisition risk
management plan, (3) conducting software risk management activities
in accordance with the plan (e.g., identifying risks, taking
mitigation actions, and tracking risk mitigation actions to
completion), and (4) measuring and reporting on the status of
acquisition risk management activities to management. 


   ATC MODERNIZATION SOFTWARE RISK
   MANAGEMENT IS INEFFECTIVE
---------------------------------------------------------- Chapter 9:1

FAA is not effectively performing ATC modernization software
acquisition risk management.  Although FAA has a written policy on
risk management, it has many weaknesses in this KPA.  For example,
most teams have no risk management plan, and the one team that has a
plan failed to follow it.  As a result, the teams are not identifying
risks, taking risk mitigation actions, and tracking risk mitigation
actions to completion.  Moreover, none of the teams measure and
report to management on the status of acquisition risk management
activities.  Consequently, management is not in a position to correct
problems promptly. 

Figure 9.1 provides a comprehensive listing of the five projects'
strengths, weaknesses, and observations for the acquisition risk
management KPA.  The specific findings supporting the practice
ratings in figure 9.1 are in tables 9.1 through 9.5. 

   Figure 9.1:  Acquisition Risk
   Management Summary

   (See figure in printed
   edition.)



   (See figure in printed
   edition.)



                                    Table 9.1
                     
                     Acquisition Risk Management Findings for
                                     ARTSIIIE

                      Automated Radar Terminal System IIIE
--------------------------------------------------------------------------------
              Key Practice                 Finding                      Rating
------------  ---------------------------  ---------------------------  --------
Commitment 1  The acquisition              There is a policy for        Strength
              organization has a written   acquisition risk
              policy for the management    management.
              of acquisition risk.

Ability 1     A group that is responsible  No group is assigned         Weakness
              for coordinating             acquisition risk management
              acquisition risk management  tasks.
              activities exists.

Ability 2     Adequate resources are       No mechanism exists for      Weakness
              provided for acquisition     identifying resources
              risk management activities.  required and for ensuring
                                           that the needed resources
                                           are provided to the
                                           project.

Ability 3     Individuals performing       The acquisition              Observat
              acquisition risk management  organization has no          ion
              activities have experience   guidance regarding training
              or receive required          or experience requirements
              training.                    for project participation.

Activity 1    Risk identification,         No risk identification,      Weakness
              analysis, and mitigation     analysis, or mitigation
              activities are integrated    activities are integrated
              into software acquisition    into software acquisition
              planning.                    planning.

Activity 2    The Software Risk            There is no risk management  Weakness
              Management Plan is           plan.
              developed according to the
              project's defined software
              acquisition process.

Activity 3    The project's acquisition    No software risk management  Weakness
              risk management activities   plan exists, therefore, the
              are performed in accordance  activities are not
              with its Software Risk       performed in accordance
              Management Plan.             with it.

Activity 4    The project team             Officials stated that the    Observat
              identifies, analyzes, and    product team identifies,     ion
              takes appropriate software   analyzes, and mitigates
              risk mitigation actions      risks, but could not
              during acquisition           provide documents to
              planning.                    support this.

Activity 5    Risk identification,         Officials could provide no   Weakness
              analysis, and mitigation     evidence of risk
              are conducted as an          identification, analysis,
              integral part of the         and mitigation being
              project performance          conducted as part of
              management and contract      project and contract
              performance management       management.
              processes.

Activity 6    Software risk mitigation     Risks are not tracked to     Weakness
              actions are tracked to       completion.
              completion.

Measurement   Measurements are made and    No internal process          Weakness
1             used to determine the        measurements are taken and
              status of the acquisition    used to determine the
              risk management activities.  status of activities for
                                           any of the key process
                                           areas.

Measurement   Resources expended for       FAA does not track           Weakness
2             acquisition risk management  resources.
              activities are recorded and
              tracked.

Verification  The activities for           While the Integrated         Weakness
1             acquisition risk management  Product Team leader reviews
              are reviewed by acquisition  the status of the contract
              organization management on   and the contractor's cost
              a periodic basis.            and schedule, he does not
                                           review the status of the
                                           activities that are
                                           required to be performed
                                           for any of the key process
                                           areas.

Verification  The activities for           While the product team       Weakness
2             acquisition risk management  leader reviews the status
              are reviewed by the project  of the contract and the
              manager on both a periodic   contractor's cost and
              and event-driven basis.      schedule, he does not
                                           review the status of the
                                           activities that are
                                           required to be performed
                                           for any of the key process
                                           areas.
--------------------------------------------------------------------------------



                                    Table 9.2
                     
                     Acquisition Risk Management Findings for
                                       DSR

                           Display System Replacement
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           There is a written        Strength
               organization has a        policy for acquisition
               written policy for the    risk management.
               management of
               acquisition risk.

Ability 1      A group that is           The product team is       Strength
               responsible for           responsible for
               coordinating acquisition  acquisition risk
               risk management           management.
               activities exists.

Ability 2      Adequate resources are    No mechanism exists for   Weakness
               provided for acquisition  identifying resources
               risk management           required and for
               activities.               ensuring that the needed
                                         resources are provided
                                         to the project.

Ability 3      Individuals performing    The acquisition           Observation
               acquisition risk          organization has no
               management activities     guidance regarding
               have experience or        training or experience
               receive required          requirements for project
               training.                 participation.

Activity 1     Risk identification,      Risk identification,      Strength
               analysis, and mitigation  analysis, and mitigation
               activities are            are integrated into the
               integrated into software  software acquisition
               acquisition planning.     planning.

Activity 2     The Software Risk         While there is a risk     Weakness
               Management Plan is        management plan that
               developed according to    addresses software at a
               the project's defined     high level, there is no
               software acquisition      software risk management
               process.                  plan.

Activity 3     The project's             Because there is no       Weakness
               acquisition risk          software risk management
               management activities     plan, activities are not
               are performed in          performed in accordance
               accordance with its       with it.
               Software Risk Management
               Plan.

Activity 4     The project team          The risk identification   Weakness
               identifies, analyzes,     activities are not well
               and takes appropriate     defined. The product
               software risk mitigation  team used risk
               actions during            interchangeably with
               acquisition planning.     currently identified
                                         problems, action items,
                                         issues, and schedule
                                         tracking.

Activity 5     Risk identification,      What the product team     Strength
               analysis, and mitigation  considers risks
               are conducted as an       (problems, action items,
               integral part of the      issues, etc.) are
               project performance       identified, analyzed,
               management and contract   and mitigated.
               performance management
               processes.

Activity 6     Software risk mitigation  Software risk mitigation  Weakness
               actions are tracked to    is not tracked to
               completion.               completion.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         and used to determine
               acquisition risk          the status of activities
               management activities.    for any of the key
                                         process areas.

Measurement 2  Resources expended for    FAA does not track        Weakness
               acquisition risk          resources.
               management activities
               are recorded and
               tracked.

Verification   The activities for        While the Integrated      Weakness
1              acquisition risk          Product Team leader
               management are reviewed   reviews the status of
               by acquisition            the contract and the
               organization management   contractor's cost and
               on a periodic basis.      schedule, he does not
                                         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              acquisition risk          leader reviews the
               management are reviewed   status of the contract
               by the project manager    and the contractor's
               on both a periodic and    cost and schedule, he
               event-driven basis.       does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------



                                    Table 9.3
                     
                     Acquisition Risk Management Findings for
                                       NIMS

                      NAS Infrastructure Management System
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           The Acquisition           Weakness
               organization has a        Management System is the
               written policy for the    written policy for
               management of             acquisition risk
               acquisition risk.         management, but it does
                                         not address software.

Ability 1      A group that is           Both the Integrated       Strength
               responsible for           Program Plan and the
               coordinating acquisition  Product Team Plan assign
               risk management           the risk management plan
               activities exists.        activities to the team.

Ability 2      Adequate resources are    No mechanism exists for   Weakness
               provided for acquisition  identifying resources
               risk management           required and for
               activities.               ensuring that the needed
                                         resources are provided
                                         to the project.

Ability 3      Individuals performing    The acquisition           Observation
               acquisition risk          organization has no
               management activities     guidance regarding
               have experience or        training or experience
               receive required          requirements for project
               training.                 participation.

Activity 1     Risk identification,      Acquisition risk          Strength
               analysis, and mitigation  management is part of
               activities are            acquisition planning as
               integrated into software  called for in both the
               acquisition planning.     Integrated Program Plan
                                         and the Product Team
                                         Plan.

Activity 2     The Software Risk         No software risk          Weakness
               Management Plan is        management plan exists.
               developed according to    The risk management plan
               the project's defined     does not address
               software acquisition      software.
               process.

Activity 3     The project's             There is no software      Weakness
               acquisition risk          risk management plan,
               management activities     therefore, activities
               are performed in          cannot be performed in
               accordance with its       accordance with it.
               Software Risk Management
               Plan.

Activity 4     The project team          The project team does     Weakness
               identifies, analyzes,     not identify, analyze,
               and takes appropriate     or mitigate software
               software risk mitigation  risks.
               actions during
               acquisition planning.

Activity 5     Risk identification,      While plans call for      Not rated
               analysis, and mitigation  risk identification,
               are conducted as an       analysis, and
               integral part of the      mitigation, the project
               project performance       has not reached a stage
               management and contract   where this activity
               performance management    applies.
               processes.

Activity 6     Software risk mitigation  A risk tracking system    Observation
               actions are tracked to    is being developed as
               completion.               called for in the
                                         Integrated Program Plan.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         to determine the status
               acquisition risk          of activities for any of
               management activities.    the key process areas.

Measurement 2  Resources expended for    FAA does not track        Weakness
               acquisition risk          resources.
               management activities
               are recorded and
               tracked.

Verification   The activities for        While the Integrated      Weakness
1              acquisition risk          Product Team leader
               management are reviewed   reviews the status of
               by acquisition            the contract and the
               organization management   contractor's cost and
               on a periodic basis.      schedule, he does not
                                         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              acquisition risk          leader reviews the
               management are reviewed   status of the contract
               by the project manager    and the contractor's
               on both a periodic and    cost and schedule, he
               event-driven basis.       does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------



                                    Table 9.4
                     
                     Acquisition Risk Management Findings for
                                       VSCS

                       Voice Switching and Control System
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           FAA Order 1810.1F is the  Strength
               organization has a        policy for acquisition
               written policy for the    risk management.
               management of
               acquisition risk.

Ability 1      A group that is           The product team leader   Strength
               responsible for           is the acquisition risk
               coordinating acquisition  manager.
               risk management
               activities exists.

Ability 2      Adequate resources are    No mechanism exists for   Weakness
               provided for acquisition  identifying the
               risk management           resources required and
               activities.               for ensuring that the
                                         needed resources are
                                         provided to the project.

Ability 3      Individuals performing    The acquisition           Observation
               acquisition risk          organization has no
               management activities     guidance regarding
               have experience or        training or experience
               receive required          requirements for project
               training.                 participation.

Activity 1     Risk identification,      The risk management plan  Weakness
               analysis, and mitigation  does not call for risk
               activities are            activities to be
               integrated into software  integrated into software
               acquisition planning.     acquisition planning.

Activity 2     The Software Risk         There is a software risk  Strength
               Management Plan is        management plan.
               developed according to
               the project's defined
               software acquisition
               process.

Activity 3     The project's             Although there is a risk  Weakness
               acquisition risk          management plan, there
               management activities     is no evidence that
               are performed in          acquisition risk
               accordance with its       management is performed
               Software Risk Management  in accordance with the
               Plan.                     plan.

Activity 4     The project team          The project team does     Weakness
               identifies, analyzes,     not identify, analyze,
               and takes appropriate     and take appropriate
               software risk mitigation  actions during
               actions during            acquisition planning.
               acquisition planning.

Activity 5     Risk identification,      The product team does     Weakness
               analysis, and mitigation  not ensure that risks
               are conducted as an       are identified, or that
               integral part of the      analyses and mitigations
               project performance       are conducted as an
               management and contract   integral part of project
               performance management    performance management
               processes.                and contract performance
                                         management.

Activity 6     Software risk mitigation  There is no evidence      Weakness
               actions are tracked to    that risk mitigation is
               completion.               tracked to completion.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         or used to determine the
               acquisition risk          status of activities for
               management activities.    any of the key process
                                         areas.

Measurement 2  Resources expended for    FAA does not track        Weakness
               acquisition risk          resources.
               management activities
               are recorded and
               tracked.

Verification   The activities for        While the Integrated      Weakness
1              acquisition risk          Product Team leader
               management are reviewed   reviews the status of
               by acquisition            the contract and the
               organization management   contractor's cost and
               on a periodic basis.      schedule, he does not
                                         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              acquisition risk          leader reviews the
               management are reviewed   status of the contract
               by the project manager    and the contractor's
               on both a periodic and    cost and schedule, he
               event-driven basis.       does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------



                                    Table 9.5
                     
                     Acquisition Risk Management Findings for
                                       WARP

                          Weather and Radar Processor
--------------------------------------------------------------------------------
               Key Practice              Finding                   Rating
-------------  ------------------------  ------------------------  -------------
Commitment 1   The acquisition           FAA Order 1810.1F is the  Strength
               organization has a        written policy.
               written policy for the
               management of
               acquisition risk.

Ability 1      A group that is           The product team is       Strength
               responsible for           responsible for
               coordinating acquisition  acquisition risk
               risk management           management activities.
               activities exists.

Ability 2      Adequate resources are    No mechanism exists for   Weakness
               provided for acquisition  identifying and for
               risk management           ensuring that the needed
               activities.               resources are provided
                                         to the project.

Ability 3      Individuals performing    The acquisition           Observation
               acquisition risk          organization has no
               management activities     guidance regarding
               have experience or        training or experience
               receive required          requirements for project
               training.                 participation.

Activity 1     Risk identification,      Risk identification,      Strength
               analysis, and mitigation  analysis, and mitigation
               activities are            activities are
               integrated into software  integrated into software
               acquisition planning.     acquisition planning.

Activity 2     The Software Risk         Although there is a risk  Weakness
               Management Plan is        management plan, it
               developed according to    incorrectly identifies
               the project's defined     risks as currently
               software acquisition      identified problems,
               process.                  action items, issues,
                                         etc.

Activity 3     The project's             While issues are          Weakness
               acquisition risk          reviewed at team
               management activities     meetings, they are not
               are performed in          specifically identified
               accordance with its       and managed as risks.
               Software Risk Management
               Plan.

Activity 4     The project team          The product team          Strength
               identifies, analyzes,     identifies and analyzes
               and takes appropriate     risks as defined in the
               software risk mitigation  risk management plan and
               actions during            takes appropriate
               acquisition planning.     actions during
                                         acquisition planning.

Activity 5     Risk identification,      The product team          Strength
               analysis, and mitigation  identifies, analyzes,
               are conducted as an       and mitigates risks (as
               integral part of the      defined in the risk
               project performance       management plan) as part
               management and contract   of project and contract
               performance management    performance management.
               processes.

Activity 6     Software risk mitigation  Software risks as         Strength
               actions are tracked to    defined in the risk
               completion.               management plan are
                                         tracked to completion.

Measurement 1  Measurements are made     No internal process       Weakness
               and used to determine     measurements are taken
               the status of the         and used to determine
               acquisition risk          the status of activities
               management activities.    for any of the key
                                         process areas.

Measurement 2  Resources expended for    FAA does not track        Weakness
               acquisition risk          resources.
               management activities
               are recorded and
               tracked.

Verification   The activities for        While the Integrated      Weakness
1              acquisition risk          Product Team leader
               management are reviewed   reviews the status of
               by acquisition            the contract and the
               organization management   contractor's cost and
               on a periodic basis.      schedule, he does not
                                         review the status of the
                                         activities that are
                                         required to be performed
                                         for any of the key
                                         process areas.

Verification   The activities for        While the product team    Weakness
2              acquisition risk          leader reviews the
               management are reviewed   status of the contract
               by the project manager    and the contractor's
               on both a periodic and    cost and schedule, he
               event-driven basis.       does not review the
                                         status of the activities
                                         that are required to be
                                         performed for any of the
                                         key process areas.
--------------------------------------------------------------------------------

   CONCLUSIONS
---------------------------------------------------------- Chapter 9:2

FAA's software acquisition risk management process has many
weaknesses and is, therefore, undefined and undisciplined.  To become
an effective acquirer of software, FAA must adopt a more structured,
rigorous, and disciplined approach to ATC modernization software
acquisition risk management. 


FAA LACKS AN EFFECTIVE APPROACH
FOR IMPROVING ATC MODERNIZATION
SOFTWARE ACQUISITION PROCESSES
=========================================================== Chapter 10

In order to be effective, the organization responsible for improving
software acquisition processes must have (1) organizational and/or
budgetary authority over units acquiring software; and (2) a
comprehensive plan to guide software acquisition process improvement
efforts and measure progress on each.  At FAA, responsibility for ATC
software acquisition process maturity and improvement has been
assigned to three organizational entities over the last 4 years, and
currently is assigned to FAA's Software Engineering Process Group
(SEPG), a multilevel committee structure chaired by a member of FAA's
Chief Information Officer's (CIO) staff.  However, the SEPG, like the
entities previously responsible for software acquisition process
improvement, has neither organizational nor budgetary authority over
the ATC modernization product teams that acquire software.  Further,
the SEPG does not have a software acquisition process improvement
plan to guide its maturation efforts. 

As a result, management of FAA's software acquisition process
improvement effort is ad hoc and has not instilled software
acquisition process discipline into the ATC modernization program. 
While the SEPG is now taking steps to establish the analytical basis
for a defined and comprehensive software process improvement plan,
that plan does not yet exist, no schedule has been established for
completing it, and the SEPG, like the organizational entities before
it that have failed to institute process improvements, has no
authority to implement or to enforce process change. 


   FAA ORGANIZATION RESPONSIBLE
   FOR ATC SOFTWARE ACQUISITION
   PROCESS IMPROVEMENT LACKS
   AUTHORITY TO AFFECT CHANGE
--------------------------------------------------------- Chapter 10:1

FAA has been attempting to strengthen its software acquisition
processes since 1993.  At that time it established the Software
Engineering Specialty Group to, among other things, incrementally
improve FAA's software acquisition processes, first establishing
repeatable processes [SEI maturity level 2], then defined processes
[SEI maturity level 3], and eventually managed processes [SEI
maturity level 4].  This group was to be FAA's focal point for
software process assessment and improvement initiatives through 2002,
and it developed a 10-year strategy and implementation plan for doing
so.  It also produced guidance addressing software acquisition
management, software management indicators, and other
software-related processes and practices. 

In 1995, FAA established the Office of the Chief Scientist for
Software Engineering under FAA's CIO to lead FAA's software-related
process improvement efforts, including software acquisition. 
According to FAA officials, this change was made to strengthen
software process improvement through increased interaction with the
ATC modernization project offices.  In May 1995, the Chief Scientist
reaffirmed SEI's CMM as the basis for all FAA process improvement and
set forth three broad "strategies for improving the quality of FAA
software."\1 The three strategies were (1) "improve the software
acquisition, development, certification, and maintenance processes of
the FAA and its suppliers,"\2 (2) "accelerate the adoption of open
systems, COTS, NDI,\3 re-engineering, and reuse by FAA programs
(without jeopardizing system safety or integrity),"\4 and (3)
"promote the use of best software engineering practices throughout
the FAA and its supplier community."\5 The Chief Scientist also began
about a dozen loosely defined process improvement activities. 
Examples of these activities include participating in national and
international software engineering activities, interacting with
governmental and professional software engineering organizations,
meeting with FAA suppliers and aviation groups, and assessing
software engineering methods, tools, and best practices. 

Another of the Chief Scientist's dozen activities was to form the
SEPG as FAA's "focal point for initiating, planning, motivating, and
facilitating the improvement of 'acquisition life cycle processes'
for software intensive systems."\6 Formed in October 1995, the SEPG
includes representatives from ATC modernization product teams and
their parent organizations as well as other FAA organizations, and it
is chaired by the Chief Scientist for Software Engineering.  The SEPG
is directed by the Software Engineering Executive Committee (SEEC),
which is chaired by the Associate Administrator for Research and
Acquisition (i.e., the head of the ATC modernization), and is
composed of senior FAA executives.  The SEEC is responsible for
recommending and providing guidance on software engineering issues. 
Additionally, some of the ATC modernization product teams' parent
organizations have SEPGs. 

None of the organizations responsible for ATC modernization software
acquisition process improvement over the last 4 years, including the
SEPG, have had organizational and/or budgetary authority over the ATC
modernization product teams that acquire ATC software.  As a result,
neither the SEPG nor its predecessor organizations have been
positioned to effectively and efficiently implement and enforce
process changes.  Instead, they can only attempt to encourage and
persuade product teams to undertake process improvement activities. 

To illustrate the ineffectiveness of this management structure, the
Chief Scientist proposed that each product team earmark 5 percent of
its software-related budgets to implement approved process
improvement initiatives.  According to the Chief Scientist, however,
the teams refused to spend product team money on the FAA-wide
software improvement activities being proposed.  One product team
leader stated that the teams are understaffed and there is not enough
time and resources to support these activities. 


--------------------
\1 "Strategies and Tactics for FAA to Improve Software, CIP Steering
Committee Meeting," May 16, 1995, page 8. 

\2 See footnote 1. 

\3 COTS is commercial, off-the-shelf; NDI is non-development item. 

\4 See footnote 1. 

\5 See footnote 1. 

\6 In a document entitled "SEPG Purpose" Nov.  18, 1996, FAA defines
a software intensive system as "any system that is entirely software
or whose principle (sic) functionality depends on the correct
functioning of software."





   FAA PLANNING FOR SOFTWARE
   PROCESS IMPROVEMENT HAS NOT
   BEEN EFFECTIVE
--------------------------------------------------------- Chapter 10:2

To properly focus and target software acquisition process
improvements, current process strengths and weaknesses (the
capability baseline) must be fully identified and assessed, and an
effective plan for systematically correcting weaknesses must be
developed.  At a minimum, this plan should specify measurable goals,
objectives, milestones, and needed resources, should clearly assign
responsibility and accountability for accomplishing well- defined
tasks, and should be documented and approved by FAA leadership. 

Despite 4 years of software acquisition process improvement effort,
FAA has not effectively baselined FAA's ATC modernization software
acquisition processes, and has not developed a comprehensive plan for
software acquisition process improvement on the basis of this
baseline.  In 1995, the Chief Scientist began an effort to assess
software acquisition processes, but completion of the effort was
delayed 8 months and, because it lacked the requisite depth and
scope, it could not be used to produce an effective baseline to guide
improvement activities.  Subsequent plans to perform more detailed
and comprehensive assessments were dropped. 

FAA also has no comprehensive plan for software acquisition process
improvement.  As a proxy, the Chief Scientist claims that a variety
of documents produced and activities conducted over the last 2 years
collectively form a complete and comprehensive plan.  These include
(1) a document containing the "preliminary process improvement
recommendations" of a process improvement working group dated
September 4, 1996, (2) minutes of SEPG and SEEC meetings, (3)
briefings on software process improvement activities, and (4) the
business plan for the ATC modernization organization.  However, these
documents and activities, which only briefly address process
improvement initiatives, cite broad strategies, and sometimes mention
general schedules and resource needs, do not constitute an effective
plan.  They do not specify well-defined and measurable goals and
milestones, assign responsibility, identify resource needs for each
initiative, or define how and when process improvements will actually
be implemented and enforced.  Further, without a capability maturity
baseline, there is no analytical basis for selecting these
initiatives.  According to product team officials, the modernization
effort has no software acquisition process improvement plan. 


   IMPROVEMENT INITIATIVES HAVE
   THUS FAR NOT INSTILLED SOFTWARE
   PROCESS DISCIPLINE
--------------------------------------------------------- Chapter 10:3

Since 1995, the Chief Scientist has planned or initiated a dozen
activities.  Some have never been started, some are behind schedule,
and several have proceeded according to plan.  For example, while the
Chief Scientist planned to complete an assessment of ATC
modernization software acquisition capabilities using SEI level 2 and
level 3 requirements by December 1996 and June 1997, respectively,
these efforts were never performed.  Other efforts are behind
schedule.  For example, software engineering policies, guidance, and
standards that were to be issued by September 1996 are now scheduled
for issuance the third quarter of calendar year 1997; and a software
life cycle tool that was to be developed by October 1996 has been
postponed indefinitely. 

Several efforts have met their milestones and begun to build a
foundation for undertaking process improvements.  For example, a
software engineering training plan was developed in May 1996, 1 month
ahead of schedule; product teams have been trained to use the SEI
software development CMM and the associated capability evaluation
methodology to evaluate contractors' capabilities to develop
software; one product team used the results of a CMM evaluation as
part of its source selection criteria; and, according to the Chief
Scientist, hundreds of members of various ATC modernization product
teams have been trained in software management techniques, such as
defining software processes, using software management indicators,
estimating software costs, and using standards such as open systems
standards. 

Other FAA activities relating to software process improvement include
establishing an SEPG infrastructure, pilot testing selected software
product and process metrics, creating a software quality assurance
capability, reengineering configuration management processes, and
studying software cost estimation tools.  In addition, the SEPG and
the Chief Scientist are now taking steps to establish an analytical
basis for software acquisition process improvement.  For example, the
SEPG and the Chief Scientist intend to use the results of this GAO
evaluation as the basis for planning future software acquisition
process improvement activities.  Also, the SEPG has begun to analyze
the interrelationships among FAA's various process improvement
activities, link the activities to strategic goals and measurable
outcomes, and explore ways to manage these activities in a
coordinated fashion.  Further, the Chief Scientist intends to
formalize the results of these steps in a comprehensive plan of
action, although no schedule has been set for doing so. 

However, none of these activities or steps, either individually or in
the aggregate, have yet resulted in more repeatable, better defined,
more disciplined ATC modernization software acquisition processes. 
In interviews, product team and SEPG officials confirmed that the
software acquisition processes had not yet been improved.  Instead,
the activities have begun to lay a foundation for potential process
improvements in the future. 


   CONCLUSIONS
--------------------------------------------------------- Chapter 10:4

FAA has neither an effective management structure nor plan of action
for improving its software acquisition processes.  As a result,
software acquisition processes will remain immature and will not
support effective, efficient, and economical acquisition of
mission-critical software costing billions of dollars.  Until
responsibility and accountability for software acquisition process
improvement is assigned to an FAA organizational entity with the
requisite authority to affect process change, and until this
organizational entity pursues a plan for change based on a complete
and objective assessment of current process strengths and weaknesses,
it is unlikely that significant ATC modernization software
acquisition process improvements will be made, and ATC software
acquisition processes will remain immature. 


OVERALL CONCLUSIONS AND
RECOMMENDATIONS
=========================================================== Chapter 11

Leading software acquisition organizations rely on defined and
disciplined software acquisition processes to deliver promised
software capabilities on time and within budget, first on a
project-by-project basis, and later, as the organization's processes
become more mature, consistently across the institution.  FAA's ATC
modernization software acquisition processes are ad hoc and sometimes
chaotic, and are not repeatable even on a project-by-project basis. 
As a result, FAA's success or failure in acquiring ATC software
depends largely on specific individuals, rather than on well-defined
and disciplined software acquisition management practices.  This
greatly reduces the probability that software-intense ATC
modernization projects will consistently perform as intended and be
delivered on schedule and within budget.  For FAA to mature beyond
this initial level, it must implement basic management controls and
instill self-discipline in its software projects. 

FAA recognizes the importance of software process maturity and the
need to improve its software acquisition processes.  However, it
lacks an effective management structure for accomplishing this
because the FAA organization responsible for process improvement, the
SEPG, lacks the authority to implement management controls and
instill process discipline within the ATC modernization software
acquiring organizations.  Additionally, despite 4 years of FAA
process improvement activity, no well-targeted, comprehensive, and
coordinated plan of action for software acquisition process
improvement exists. 


   RECOMMENDATIONS
--------------------------------------------------------- Chapter 11:1

Given the importance and the magnitude of information technology at
FAA, we reiterate our recent recommendation that a CIO organizational
structure similar to the department-level CIOs as prescribed in the
Clinger-Cohen Act of 1996 be established for FAA.\1

To improve FAA's software acquisition capability for its ATC
modernization and thereby take the first step in institutionalizing
mature acquisition processes, we recommend that the Secretary of
Transportation direct the FAA Administrator to: 

  -- assign responsibility for software acquisition process
     improvement to FAA's CIO;

  -- provide FAA's CIO the authority needed to implement and enforce
     ATC modernization software acquisition process improvement;

  -- require the CIO to develop and implement a comprehensive plan
     for ATC modernization software acquisition process improvement
     that is based on the software capability evaluation results
     contained in this report and specifies measurable goals and time
     frames, prioritizes initiatives, estimates resource
     requirements, and assigns roles and responsibilities;

  -- allocate adequate resources to ensure that these improvement
     efforts are implemented and enforced; and

  -- require that, before being approved, every ATC modernization
     acquisition project have software acquisition processes that
     satisfy at least SA-CMM level 2 requirements. 


--------------------
\1 Air Traffic Control:  Complete and Enforced Architecture Needed
for FAA Systems Modernization (GAO/AIMD-97-30, Feb.  3, 1997). 


   AGENCY COMMENTS AND OUR
   EVALUATION
--------------------------------------------------------- Chapter 11:2

In its written comments on a draft of this report, the Department of
Transportation recognized the importance of mature software
acquisition processes, agreed that FAA's processes are insufficiently
mature, acknowledged that FAA process improvement activities have yet
to produce greater process discipline, and reaffirmed FAA's
commitment to improving its software acquisition capabilities using
the SA-CMM.  However, the Department did not state what, if any,
specific action it would take on our recommendations.  Additionally,
the Department expressed several concerns, each of which is presented
below, along with our rebuttal. 

First, the Department stated that FAA does not separately procure
software for its ATC systems.  Rather, it procures systems that use
software as a major component.  Therefore, its policies and
procedures (i.e., processes) are "geared" to system acquisitions, and
evaluating only the software-related aspects of its acquisition
processes "is not an adequate approach."

GAO Rebuttal:  All major system modernizations, like the ATC
modernization, involve the acquisition of hardware, software, and
firmware operating interdependently.  However, as FAA's own
experience with the Advanced Automation System clearly proves, the
software component is the source of most system risk, and the
component most frequently associated with late deliveries, cost
increases, and performance shortfalls.  Moreover, there is widespread
recognition throughout the computer industry that the billions of
dollars being invested in complex, real-time, fault tolerant systems,
like FAA ATC systems, are jeopardized by inadequate management
attention to software in general, and undisciplined, ill-defined
software acquisition and development processes in particular.  This
is precisely why SEI developed its software-related CMMs, why the
CMMs have been endorsed and accepted throughout industry and
government, and why the scope of this evaluation focused on software
acquisition processes rather than on broader systems issues. 
Further, the FAA system acquisition policies and procedures that the
Department references do not explicitly or adequately address
software issues.  For example, they do not address software-specific
acquisition planning for such KPAs as contract tracking and
oversight, requirements development and management, and risk
management.  Additionally, they do not provide for measuring and
verifying the performance of software-specific acquisition
activities. 

Second, the Department commented that the SA-CMM "is not widely used,
adopted, or validated" and, while it has "significant merit, it is
certainly not to be taken as the same authoritative source for
process improvement guidance as the SW-CMM,\2 which has been in use
worldwide by thousands of organizations for several years."

GAO Rebuttal:  This position is clearly inconsistent with the
Department's and FAA's stated commitment to using the SA-CMM as the
basis for efforts to improve FAA's software acquisition capabilities. 
More important however is that this position is without substance. 
The SA-CMM does not promulgate original or novel concepts of
debatable value.  Instead, it presents as requisite processes and
practices those activities that common sense have validated as
essential to effective software acquisition.  For example, it
requires disciplined requirements development and management,
solicitation, contract tracking and oversight, and evaluation.  Our
evaluations have for years made the same points without rational
dispute.  The SA-CMM simply provides a coherent framework and
standard terminology for these concepts.  The findings in this
report, which have been corroborated by SEI, are compelling not
because of the age of the model used, but because the criticality of
the processes and practices examined is undeniable. 

Third, the Department claimed that "GAO may have misapplied the
model" by (1) giving inadequate consideration to equivalent
alternative practices when determining whether SA-CMM specified
practices were performed (e.g., DSR system acquisition planning being
judged as an insufficient proxy for software acquisition planning
specified in the SA-CMM), (2) not effectively tailoring the SA-CMM to
focus only on project activities that occurred after the cancellation
of the Advanced Automation System, and (3) reporting evaluation
results in a way that "does not create an environment conducive to
process improvement" (i.e., reporting the results for each project,
rather than either aggregating the results or disguising the identity
of the projects). 

GAO Rebuttal:  We applied the model properly and correctly, and SEI
has attested to this.  Every member of our evaluation team was
trained by SEI; the team leader was an SEI designated "lead
evaluator" and has been authorized by SEI to submit results for
inclusion in SEI studies; three senior SEI professionals, two of whom
are authors of the SA-CMM, participated in the evaluation to ensure
that the model was properly used; and SEI concurred with each
practice determination in the report (e.g., strength, weakness). 
With respect to each of the Department's subsidiary points regarding
our application of the model: 

(1)During the course of extensive interviews with FAA designated
officials, no evidence of reasonable alternative practices was
provided.  If such evidence had been provided, we would have
considered it.  For example, when FAA provided a system acquisition
plan for DSR as evidence of software acquisition planning, we
reviewed the document and found that it was not a reasonable
alternative practice because it did not adequately address the
software component of the acquisition. 

(2)As agreed with SEI, those practices that were deemed inapplicable
were not rated, and those performed years ago were so designated. 
Moreover, even if all practices predating the Advance Automation
System's cancellation were ignored, none of our conclusions and
recommendations would change. 

(3)The model and evaluation method do not specify any reporting
format.  In particular, they do not address whether results should be
reported for each project, or whether the identity of the projects
should be disguised or results reported only in the aggregate.  Given
the mission-critical importance and billion dollar cost of these
projects, full disclosure of all relevant facts to the Congress and
the public is both warranted and appropriate. 

Fourth, concerning FAA's software acquisition process improvement
efforts, the Department stated that the report does not sufficiently
appreciate the "progress made to date, the difficulties involved in
achieving that progress, and the time that it takes for .  .  . 
changes of
this .  .  .  magnitude." Specifically, the many efforts underway are
not a "hodge podge" of activities, but are "a very healthy sign of
the seriousness and enthusiasm" that FAA assigns to process
improvement and are "organized with respect to specific directorate
objectives." Also, since FAA's process improvement activities have
been underway for less than 2 years, it is too early to expect
results. 

GAO Rebuttal:  The Department's position that FAA's many process
improvement activities are not a "hodge podge," but rather are part
of an organized and coordinated comprehensive plan of action, is not
supported by the facts.  While FAA began drafting a plan during the
course of our evaluation, it had no schedule for finalizing this
plan, and no analytical basis for the software acquisition process
improvement activities underway.  Just as its software acquisition
processes lack maturity and discipline, so do its efforts to improve
these processes.  Claims that FAA has been engaged in software
improvement efforts for less than 2 years, and thus it is too early
to evaluate results, are also unsupported.  In fact, software
acquisition process maturity and improvement efforts began in 1993. 
Since SEI published statistics show that the median time to improve
from SW-CMM level 1 to level 2 is 26 months, and from SW-CMM level 2
to level 3 is 17 months, it is entirely reasonable to expect FAA to
be able to demonstrate some improvement in its processes after 4
years. 

Fifth, while the report states that the SEPG has neither the
organizational nor the budgetary authority to effectively implement
process change, the Department stated that its "understanding .  .  . 
is that organizations do not normally give their SEPG authority over
product teams." In FAA's case, the SEPG provides advice and counsel
to the Software Engineering Executive Committee, which consists of
senior managers who have authority and responsibility to direct
process improvement actions.  The SEPG is the committee's agent for
implementing these improvements. 

GAO Rebuttal:  The issue is not whether FAA's SEPG is organized as
the Department "understands" other SEPGs to be organized, but whether
the SEPG, or any FAA organizational entity responsible for
implementing and enforcing software process change, has the authority
needed to accomplish this task.  Currently, no organizational entity
in FAA has the requisite authority.  Accordingly, we have recommended
that a CIO organizational structure similar to the department-level
CIOs prescribed in the Clinger-Cohen Act of 1996 be established for
FAA, and that it be assigned the responsibility and resources needed
to affect and enforce software acquisition process improvement. 

Sixth, the Department contends that the report "leads the reader to
erroneously conclude that the five programs reviewed are in trouble"
relative to their cost and schedule goals. 

GAO Rebuttal:  The report addresses the maturity of FAA's software
acquisition processes and concludes that these processes are ad hoc
and undisciplined, reducing the probability that software-intense ATC
modernization projects will consistently perform as intended and be
delivered on schedule and within budget.  The report does not address
the overall status of the projects covered by GAO's review, and,
therefore, provides no basis for drawing conclusions about the
projects' overall cost and schedule performance. 



(See figure in printed edition.)Appendix I

--------------------
\2 SW-CMM is SEI's software development capability maturity model. 


COMMENTS FROM THE DEPARTMENT OF
TRANSPORTATION
=========================================================== Chapter 11



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)


MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix II


   ACCOUNTING AND INFORMATION
   MANAGEMENT DIVISION,
   WASHINGTON, D.C. 
-------------------------------------------------------- Appendix II:1

Randolph C.  Hite, Senior Assistant Director
Keith A.  Rhodes, Technical Director
Leonard J.  Latham, Technical Assistant Director
Suzanne M.  Burns, Senior Information Systems Analyst
Madhav S.  Panwar, Senior ADP/Telecommunications Analyst
Ira S.  Sachs, Senior Information Systems Analyst

*** End of document. ***