1 Introduction

1.1 Purpose

The System Integration Test Plan describes the methodology and process used for executing the System Test of the application software. The purpose of System Testing is to verify that the application works in an integrated manner and to validate that the requirements have been implemented into the software according to the topic paper designs and business scenario system flows. The System Test phase within the development of any large-scale system is an important milestone. It serves as the bridge between Development and User Acceptance Test, and it confirms that the system performs as designed and fulfills functional requirements. The remainder of this introduction presents the scope and objectives of this plan along with a description of how System Test fits in with the overall test process.

This document, System Test Plan, will be utilized as the process for System Testing throughout the life-cycle of the Florida Safe Families Network (FSFN) Project. This deliverable document represents the System Test Plan for Release 1 of the FSFN application. Lessons learned and subsequent changes due to the lessons learned will be updated for redelivery of the Plan for future releases.

1.2 Objective and Scope of the System Test Plan

The objective of the System Integration Test Plan is to define the methodology to be used throughout System Testing activities associated with the FSFN project. The plan will identify the goals and objectives of System Test, the activities necessary to support System Testing, the execution of System Test, the documentation of the results of System Test, and the identification of the roles and responsibilities of the project team members.

System Test is the scripted test of business scenarios within the FSFN application. During System Test, members of the CGI Functional Team will execute test scripts to ensure that the FSFN application functions as described in the topic paper designs and business scenario system flows.

The scope of System Test includes:

· Testing the functionality of the FSFN application as it pertains to the features within the design. This includes web-based functions, batch processes, reports, and interfaces (when required). For Release 1, report and interface system testing will continue through user acceptance and pilot test timeframes;

· Ensuring all acceptance criteria associated with each test script have been tested and function as designed;

· Validation of the conversion process by ensuring that the converted data integrates correctly with the application (when applicable);

· Stress and volume testing tasks (excludes backups, restore procedures, and integrations). This is described in detail in Section 5 of this document; and

· Validation of interface processes by ensuring that the transferred data integrates correctly with the application.

System Test activities, Interface Test activities and Conversion Test activities are separate processes, and the methodology for testing the conversion and interface software will be addressed in the Conversion and Interface plans. However, converted and interfaced data will be incorporated, in part, during System Test and User Acceptance Test to make sure that the converted and interfaced data is successfully integrated with the FSFN application and adheres to DCF business rules. The overall purpose of System Testing is to verify that the application works in an integrated manner. System Test scripts will capture and test all aspects of the application.

1.3 Phases of Testing

In the overall lifecycle of an application project, at a very high level, testing activities begin after design and at the end of development. There are three phases of testing planned for each Release of the FSFN application prior to Pilot Test: Unit Test, System Test, and User Acceptance Test. These three phases each reflect a different approach in testing the application. System Test is the second phase of testing of the FSFN application. System Test Activities begin after the design of functionality comprising a Release. This is when test scripts are developed. The execution of the System Test scripts occurs following the successful Unit Test of a page or module within the application. Exhibit #1 depicts the different phases.

Text Box: Unit Test
CGI developers and designers test inputs and outputs of individual components.

Exhibit #1

Phases of Testing

Text Box: System Integration Test
CGI testers test interaction between components. Testers execute System Test scripts, report defects.

Text Box: User Acceptance Test
DCF tests system to validate that business requirements have been met. Functionality tested by simulating business scenarios.

1.3.1 Code and Unit Test

Using the specifications from the approved topic paper designs, the CGI development team will code and subsequently Unit Test the application and its’ interfaces. Unit Testing will occur both in a discrete Unit Test of the applicable layer and then in an integrated Test, to validate that specific requirements pertaining to each layer are met, and ultimately to verify that the unit correctly integrates the presentation layer, the business object layer, and the data access layer processes. Unit testing will also occur for each discrete interface program unit. The integrated Test will be supported by both the CGI developer and designer.

1.3.2 System Integration Test

In a Release, the final testing phase prior to User Acceptance Test is System Integration Test. During this phase CGI staff using System Test scripts will test the application. Topic paper designs and business scenario system flows created during the design phase will be used in the creation of the System Test scripts. The purpose of System Testing is to verify that the application works in an integrated manner and to validate that the Requirements have been implemented into the software as designed. When incidents are found outside of the normal System Test cycles, incidents will be prioritized by the level and time to fix, test and deploy to the next environment. CGI will work with DCF to determine the best process for maintenance fixes. All incidents that are fixed must go through the testing cycles of Unit, System and User Acceptance testing.

The execution of the System Test scripts occurs following the successful Unit Test of a page or module within the application. Release 1 System Test incorporates an iterative approach to testing the topic areas. As pages and modules are developed, system test scripts will be executed for those areas of the application. End-to-end system testing will occur once all the pages and modules are developed and moved to the system test environment.

1.3.3 User Acceptance Test

User Acceptance Test closely resembles the System Test process, with two important differences. First, the target environment for the User Acceptance Test includes activities to validate the size of database and transaction volumes associated with that environment. Secondly, DCF staff will lead the test process, with CGI staff providing support. User Acceptance Testing provides DCF with the opportunity to validate that the business requirements have been met in the implementation of the system, and to verify that the software works according to design. User Acceptance Testing procedures and methodology will be discussed in detail in the User Acceptance Test Plan.

2 Assumptions

There are certain assumptions upon which the System Test Plan is based. The following is a list of processes, procedures, and equipment that must be in place in order for the CGI Test Team to effectively accomplish the System Test in the required time frame.

· System Testing will occur at the FSFN Project Site.

· The CGI testing environment (e.g. computers, network, and software) will be sufficient to accomplish the required testing assignments. The environment consists of the test server with a test version of the database and access to any external system environments required to test interfaces for systems maintained by DCF. Per the Interfaces Plan, Interface Partnering Agreements will be established with the partner agencies for each interface. These agreements will dictate the System Integration testing approach for each interface.

· Success of the system test will be based upon the fulfillment of the approved detailed design. Any design changes that may occur during the testing processes will be processed through the project change control procedures.

· Interface System Test of systems not maintained by DCF will verify data output formats from FSFN and proper processing of data received by FSFN in specified formats. Per the Interfaces Plan, Interface Partnering Agreements will be established with the partner agencies for each interface. These agreements will dictate the System Integration testing support for each interface.

· As it relates to Volumetric Testing, CGI will work with DCF so the testing schedule is not impacted by access to the database and servers.

· The Incident Tracking Database will be available to all testers and developers, both DCF and CGI. This tool is described in detail in section 8 of this document.

· CGI designated software-testing support tools will be available to all testers requiring them (e.g Incident Tracking Database).

· Although some dependency data may be required for testing, CGI will create and maintain the data, as needed, for System Test scripts.

· The CGI testing team will be trained in the functional specifications of the system. The team will also be trained in the use of the Incident Tracking Database, as well as any designated tools to assist in the testing process.

· The CGI Test Team will have security access for the FSFN system, the Incident Tracking Database, and other testing tools. Access to the DCF network, legacy data (HSn), and mainframe products will follow the established DCF security policies and practices.

· Development team resources will be available for support of System Test and for prompt resolution of critical defects that would prohibit further testing.

· At the start of System Test, all test scripts and the script tracking tool will be stored in the Subversion (SVN) Repository. Execution, comments, and changes will all be tracked. At the conclusion of System Test, final incident reports and final test results will also be stored within SVN Repository.

· CGI will provide DCF with a copy of the System Test Scripts per the agreed to project schedule.

· Automated Regression Testing will not be utilized for Release 1 System Testing. Many functions developed in Release 1 will be updated with Release 2 and therefore comprehensive regression scripts will not be available for automated testing. An automated regression tool and a process for regression testing will be researched for Release 2. Manual regression testing is described in sections 4.3.1 and 6 of this document.

· CGI will work with DCF to determine the appropriate environment to conduct a valid performance test of the application. Volumetric Testing is described in detail in section 5 of this document. The project schedule identifies the timeframes for Volumetric Testing.

· Volumetric testing will be run during off-peak times to minimize the impact on the current production environments.

· Volumetric performance testing and system testing will occur in two different environments. System test may use VPN access but the performance test environment will not use VPN.

· CGI will confirm that the system testing environment is properly configured and fully functional prior to the start of system test.

3 Overview of System Test

3.1 Goals and Objectives

The System Test phase of the FSFN application development is the scripted test of designed functionality and business scenarios within the application. During System Test, members of the integrated team will execute test scripts to ensure that the FSFN application functions as described in the topic paper designs and business scenario system work flows. This includes testing of the online FSFN components as well as the interfaces and reports. Only those reports and interfaces included in the detail design will be included in the timeframes of System Test. The other reports, interfaces, and data warehouse processes will be system tested as they are ready and included in the FSFN application.

3.1.1 Goals

The goals of System Test are defined as follows.

· System Test serves as a bridge between development and User Acceptance Test of a Release. It is the initial point where the system can be viewed and tested as an integrated application.

· System Test confirms that the system performs properly, both from a functional perspective as the business requirements are tested, and from a technical perspective, since the System Test environment will use a similar configuration as the production environment.

· System Test addresses topics from a business functionality perspective, replicating the use of the system in daily work processes.

· System Test confirms that FSFN business requirements are met and FSFN solution works in total.

· System Test validates previous R1 functionality through manual end-to-end regression testing. Release 1 of the FSFN application centers on the following functional processes and system elements:

o Intake – the process of documenting abuse and neglect calls and creating abuse and neglect reports, as well as information and referral type inquiries that do not become investigations of abuse or neglect.

§ SM02 – Intake

o Investigation – the process of investigating and determining and documenting the safety situation of the child.

§ SM06a – Child Investigation

§ SM06d – Adult Investigation

o Case Management – the process of creating and maintaining cases for organizing the ongoing provision of services and assessments for children under the care of DCF.

§ SM04a – Maintain Case

§ SM05 – Case Closure and SM13a – Medical/Mental Health

§ SM10a – Out of Home Placement

§ SM20 – Interim Child Information

o Navigation – the method used to access and move between screens and functions within the SACWIS application.

§ SM04b – Visual Metaphor and Desktop

o Common System Functions – basic system elements, including Security Administration, Help Function, Person Maintenance, Workers and Organization Maintenance, Assignments, Approvals, Alerts and Checklists, and Notes.

§ CM09 – Security

§ CM12 – System Help

§ CM01 – Person Management

§ CM25 – Organization

· CM02 – Worker Assignment

· CM18 – Manage Worker

· CM19 – Approvals

§ CM08 – Alerts

§ CM06 – Notes

§ CM10 – Search

o Provider – the process of collecting and documenting basic information about the persons who provide services for the child or family.

§ PM02a – Home Provider

§ PM02b – Organization Provider

§ PM07 – Home Inquiry

§ PM08 – Provider Address Maintenance

o Reports – scheduled and ad hoc reporting normally required to perform the operation and management of ongoing business processes within DCF and its related agencies.

o Interfaces – Interfaces that integrate other systems with the online application and the data warehouse.

3.1.2 Objectives

In order to fulfill the goals of System Testing, certain objectives must be met. The objectives of System Test are outlined below and apply to the functions and interfaces that are in-scope for Release 1.

· To ensure that modules function as an integrated unit as defined in the approved detailed design;

· To verify data accuracy of reference data, converted data, and input/output for interfaces, as well as user-entered data and related processes;

· To verify step-by-step functionality and that deviation from a workflow does not produce unexpected results (supported by the execution of System Test scripts and off-script testing methodology);

· To provide requirements traceability through the tracing of functions from the approved design to the test scripts;

· To conduct preliminary tests on reliability and maintainability; and

· To identify potential technical issues.

3.2 Management of System Test Activities

The CGI Testing Manager will lead the System Testing effort. As the coordinator of this testing phase, the CGI Testing Manager will:

· Direct the execution of the System Test in accordance with the provisions of the test plan;

· Establish the System Test schedule with the coordination of the Technical Manager;

· Monitor the test process to identify incidents and mitigate against schedule slippage;

· Manage the team’s assignments, with assistance from the Team Leads, and coordinate assignments;

· Monitor test results to identify problems and any potential rework;

· Provide scheduled reports on test status and progress;

· Work directly with each Functional Team Lead to review and map System Test Scripts to the functions of the application using the approved Topic Paper designs and the business scenario system workflows (Release 1 high-level use cases). This will ensure that System Test uses a comprehensive set of test scripts to test all aspects of the system.

4 System Test Process

System Test incorporates three major processes:

1. Preparation, including staff, environmental and procedural readiness;

2. Execution of the System Test according to structured methodology; and

3. Documentation, quality assurance, and risk management of System Test activities.

The successful integration and completion of each of these three processes is necessary for a thorough, ordered System Test.

4.1 Preparation for System Test

4.1.1 Introduction

System Test preparation incorporates staff, environmental, and procedural readiness activities. Planning and providing adequate preparation time is the starting point for ensuring the success of the each Release’s System Test.

4.1.2 Staff Readiness

The business analysts that are involved in the design activities of the application will become the System Testers of the application. These CGI staff will be trained to fully understand the goals and objectives of System Test, to correctly implement the methodologies in creating and executing System Test scripts, and to complete the required documentation associated with System Test.

4.1.3 Schedule

The schedule for System Test will follow the approved Release 1 FSFN Work plan.

4.1.4 Training

The CGI Functional Team will be trained in the functional specifications of the system and the technical environment under which the system will operate once in production. The training of the CGI Staff will be accomplished using in-house experienced staff, including the CGI Testing Manager, CGI Technical Manager, CGI Training Manager and CGI Team Leads. Training will also be available from less formal sources such as documentation, code samples, web sites, Computer-Based Training (CBT), and third party publishers.

The CGI Functional Team have been trained in the skills required to write test cases from the topic paper designs, requirements, and business scenario system workflows (Release 1 high-level use cases) created during design activities. Additionally, developers and testers are being trained in the use of the Incident Tracking Database.

4.1.5 Environmental Readiness

During the Release 1 preparation phase, the CGI Technical Manager, the DBA, and the CGI Testing Manager will validate the following:

· The test environment is established, and all hardware to be used in testing is in place and operational;

· The System Test environment is established and will be structurally similar to the production environment.

· The Legacy database (HSn DB2) and other conversion data sources will be production size for purpose of Conversion System Test.

System Test and Volumetric Test will occur in 2 separate environments. The scripted system test activities will be performed in the System Test environment and the volumetric and converted data testing will occur in a second test environment on the mainframe. These 2 environments will allow for volumetric and conversion testing to occur without impacting the performance of the scripted test.

4.1.6 Procedural Readiness

To assure that the goals and objectives of System Test are implemented throughout the process, criteria are established to identify the expected inputs and outcomes of System Test. These include the establishment of exit and success criteria, and the creation of System Test Scripts.

4.1.6.1 Exit Criteria

The criteria used to define the successful completion of System Test and approval for code to be migrated to the User Acceptance Test environment is identified as the following:

· Each online System Test Script has been executed successfully;

· Conversion, reports, batch, and interface testing scheduled for Release 1 System Test is executed successfully. Report and Interface system testing will continue through user acceptance test and pilot and therefore only those reports and interfaces included in the detail design will be included in the timeframes of System Test. It is estimated that 3 interfaces (phoeniX, Client Photo, and FDLE/MCTS) and 17 reports will be included in the detail design document and therefore will be part of the formal system test period. The other reports, interfaces, and data warehouse processes will be system tested as they are developed and included in the FSFN application;

· Off-script testing (Ad Hoc testing of alternative/diverse paths) is complete;

· All incidents identified as “1-Critical” and “2-Serious” have been corrected or resolved and retested;

· Testing documentation is complete.

Five primary methodologies for test cases will be used in execution of test scripts. (1) Test scripts will use test cases input by the tester. (2) Test scripts will test the creation of new data, as appropriate, for each module. (3) Ad hoc test scripts will use converted data to validate integration with the application (when available/applicable). (4) Test scripts will use data transferred to FSFN via interfaces (when available/applicable). (5) Test scripts will verify data sent to other systems via interfaces provided from FSFN according to interface specifications.

4.2 System Test Scripts

4.2.1 Design of System Test Scripts

System Test scripts are system-specific descriptions of the Florida child welfare business workflow (scenarios). Test Scripts are written from the perspective of the worker and describe the business workflow in terms that reflect the application design. System Test scripts are developed by the design analysts who have an in-depth understanding of the system functionality. System Test scripts will be designed so that trained testers may execute each script. The basis of the test scripts will be the design documentation created during the design phase. The test scripts will be designed to facilitate tracking the scenario tested, results, and requirements associated with a particular Release. These System Test scripts will be provided to DCF to assist DCF in creating the User Acceptance Test scripts. The dates for System Test script delivery are identified in the project schedule.

4.2.2 Development of System Test Scripts

System Test scripts will be developed, as described above, using the topic paper designs, business scenario system workflows and requirements as the basis. This applies to the Release 1 online functionality, reports, and interfaces. Test scripts will be written with a level of detail necessary to assure the testing of each discrete function. Performance of the scripts will require the tester to have a basic knowledge of the application functionality. Test scripts will be checked into the SVN Repository to ensure version control and to document changes made to test scripts that correspond to modifications to design documentation.

4.2.3 Format of System Test Scripts

To ensure that each test script contains the necessary elements so that thorough testing methodology is incorporated into each test, a standard test script format will be used.

Each test script will consist of the following components:

Script Name - The script name will be consistent with the topic area which it is based.

Script Number - The script number will be consistent with the topic area that it is based on. The number will consist of the topic area, then a dash, then the number of the script. (i.e. 2^nd test script for CM03 would be CM03-2).

Case/Provider Name – The name of the case or Provider used for testing.

Dependencies – Data that must exist or be created in the test environment, prior to the execution of the test script. This section may also include preconditions that may be required for a script to be executed (i.e. a converted case and/or person).

# of Scenarios – The number of scenarios associated with this test script.

Scenarios Tested – The scenario(s) being tested.

Script Overview - The description of the business process to be tested.

Requirements Tested by this Script - Requirements associated with the scenario will be documented.

Tester Name(s) - Identifies the tester(s) who execute(s) the script.

Execution # Start Date/End Date - Identifies the start and end date of a test execution.

For each Scenario:

Step Number - Each step will be numbered for reference.

Test Instructions - Each action required by the user will be noted.

Expected Results - Each step will identify the anticipated system response.

Actual Results – For each execution, the tester will document the actual result for each step.

Pass/Fail – For each execution, the tester will document whether the step passed or failed.

Comments – The tester may document any comments, such as training issues or script modifications.

Screen Print - The tester will attach screen shots upon completion of the final test execution.

Section 4.2.3.1 displays a sample System Test script.

4.2.3.1 Sample Test Script

SYSTEM TEST SCRIPT DEFINITION
Script Name	Existing Checklists
Script Number	CM03-1
Case/Provider Name	N/A
Dependencies	Cortny Chi (OHC Supervisor)
# of Scenarios	1
Scenarios Tested	Update an existing Checklist
Script Overview	The Supervisor would like to add an additional checklist item to an existing checklist to ensure certain steps will be taken by their workers.
Requirements Tested by this Script	123, 124, 125 ,126, 175
Testers Name(s)	1. Tim Tester 2. Tina Tester
Execution # Start Date/End Date	# 1 - 01/10/2007 - 01/12/2007 Failed
Execution # Start Date/End Date

Step #	Test Instruction	Expected Result	Actual Result	Pass Fail
Login to the system as Cortny Chi.
	On the Main Menu bar select Maintain> Checklist Template...	This will open the Select Checklist Template window. The Continue button is disabled and no template is selected. The Close button is enabled.	1. Opened select checklist template, the continue button was disabled, no template was selected, close button was enabled.	1. Pass
	Select ‘Case Adoption Closing Checklist’ from the Select Template drop-down and click the Continue button.	Selecting a template enables the Continue button. The Checklist Template page is launched with the checklist data.	1. Selected template, enabled continue button. Template was launched with checklist.	1. Pass
	Check the Due Date Allowed check box.	This enables the Due Date field.	1. Due date enabled	1. Pass
4.	Click the Insert button.	A new blank row is inserted below the last item on the list in increments of 10.	1. Blank row inserted as expected.	1. Pass
	Enter 65 in the Number field .	N/A	1. Entered 65 in number field.	1. Pass
	Enter text in the Item field.	N/A	1. Entered text into the item field	1. Pass
	Check the Due Date box beside the new item.	This enables the Days field.	1. Checked box and enabled the days field.	1. Pass
9.	Enter 10 in the Days field.	The number will appear in the window.	The field is disabled and will not allow the user to enter anything in the field. Reviewed the DSD Page XX and found that the field should be enabled. Incident 1234.	1. Fail
10.	Click the Save button.	The changes will be saved and the Save button is disabled.
11.	Click the Close button.	The Checklist Template page is close and the Supervisor is returned to the desktop.
12.	Re-enter the same Checklist.	The change will appear in the appropriate place on the Checklist.
Tester Comments
Step # 9 failed, even checked the DSD to ensure that the design was correct. Created Incident # 1234, for build # 123456.

Attached Screen Shots: (For this sample, no screen shots will be included)

4.3 Execution of System Test

4.3.1 Introduction

Release 1 System Test incorporates an iterative approach to testing the topic areas. As pages and modules are developed, system test scripts will be executed for those areas of the application. End-to-end system testing will occur once all the pages and modules are developed and moved to the system test environment. If testing anomalies (incidents) are identified during System Test then they will be tracked as incidents, researched by functional/technical staff, and follow the incident correction process identified in section 6 of this plan.

In order to conduct a thorough and well-documented System Test, testers will complete a series of activities throughout the System Test process.

Execution of Test Scripts - Test scripts will be executed as written, with documentation of each step completed. Changes to scripts that are due to incorrect or incomplete steps will be documented through the comments and the step will be changed to reflect the correction. Each test script will be executed twice, successfully, before being considered to have met exit criteria. Upon the completion of the first successful execution, the script will be handed-off to a different tester to perform the second execution. The second execution, executed by a different tester, is essential for the quality of System Testing.
Validation of Converted Data - In addition to testing activities associated specifically with the conversion process, System Test will provide an additional measure to validate converted data, when available, and to ensure that converted data integrates correctly with application software. The CGI Testing Manager will work with the Technical and Data Conversion Managers to successfully coordinate the two activities. Ad hoc test scripts that follow business processes will use converted data to validate integration with the online application. The plan for System Testing the actual conversion programs and processes is not included in this plan but is documented in the Data Conversion Plan.
Validation of Operational Data - Data created or modified through the course of testing will be validated to ensure that the window is functioning correctly and that data is processed as specified in the design documentation.
Off-Script Testing - After the script has been thoroughly tested, the testers will conduct tests of alternative/diverse paths to ensure that the application supports legitimate alternative paths or activates the required roadblocks (error messages, security constraints and incidents). Off-script testing results will be documented following the methodology established for reporting test activities and defects. Off-script testing will be completed and documented prior to the conclusion of testing activities for each module, in accordance with the exit criteria guidelines. Off-Script Testing will be documented by the use of a blank testing script, named and stored within SVN Repository. This enables the reuse of off-script testing as documented scripts.
Report Testing - All applicable reports will be tested during Reports System Test. Steps for testing these reports will be described within the steps of the test scripts. The test scripts for reports will include the verification of the field elements and data display. For the most part, report testing will occur after the online functionality has been tested because reports require entry of the online data before it can be displayed appropriately on the report. For Release 1, only those reports that are included in the detail design document will be tested as part of the formal system test timeframes. The other reports will be tested as they are developed and included in the FSFN application.
Interface Testing - Batches and interfaces, as documented in the functional and technical designs, will be tested to support the requirements associated with each of these activities. Steps for testing these processes will be described within the steps of the test scripts. System Test will provide an additional measure to validate interfaced data, when available, and to see that interfaced data integrates correctly with application software. Testing of the Interface Process will follow multiple paths:

Real Time - For those interfaces that are integrated with the online FSFN application and provide real time interfacing of data (phoeniX, Client Photo, and FDLE/MCTS), testing of these interfaces will take place as part of the testing of the online application functionality. Designs for these interfaces will be included in the detail design document and the interface activities will be included in the test scripts for those appropriate topic areas. This ensures that the interface processing logic is integrated with the online application code and supporting the online business processes. The other interfaces that do not integrate with the online FSFN application will be system tested as they are developed.
Interfaces that dump into the Data Warehouse – testing of these will require validation of record counts and visually inspect the loaded tables to validate that the data loaded correctly.
Data Warehouse, Staging tables, and Data Marts – Testing of these will require execution of the ETL processing, and the execution of reports and ad hoc queries to validate processing results.
Data Extracts – Testing of the data extract processes will require execution of the ETL processes to build the extracts, and manual examination of the content of the extracts.
Batch Processing to the Data Marts - Testing of processing loading data to the Data Marts will require execution of the ETL processing (BO, COBOL, JCL) and examination of the load processing and data mart tables to validate the results.
As part of the design and development activities for each interface processing model, test scripts will be constructed and updated to support testing.

Regression Testing – Upon the completion of Development, Unit Test and System Test, with the exception of incident fixes, manual end-to-end testing will occur. This may include the rerun of previous test scripts to make sure the overall functionality of the release works as designed. End-to-end and off-script testing will be documented using a test script template and logging the test on the script tracking spreadsheet. Future releases will also utilize the previous release scripts during Regression Testing, when applicable, to test and verify functionality from these previous releases.
Software Installs and Upgrades – All new and upgrades of releases and software associated with FSFN will go through the System Test process. For additional information on Software Install and Upgrades, please reference the Technical Standards and Procedures document for the FSFN project.

4.3.2 Performance of Tests

System Test is structured by the use of standard test instruments and an organized process to ensure that each tester follows uniform and thorough procedures. The development of scripts that follow standard business processes allows a full test of the functionality of the application. The creation of test databases ensures that consistent and appropriate data is available to support actions by testers throughout the test process.

Any incidents or testing anomalies identified by the testers will be documented according to a structured process. An electronic incident form will be completed for each defect found. After an incident has been resolved and unit tested by the developer, the incident will be reassigned to the tester. The tester will test the specific repair to verify resolution and then will execute the related test script steps, when applicable, to verify successful integration of the repair.

4.3.3 System Integration Test Schedule

System Integration Testing activity dates are identified in the project schedule and will be tracked in the project schedule.

4.4 Documentation, Quality Assurance, and Risk Management of the System Test Process

4.4.1 Introduction

Documentation, quality assurance, and risk management activities will be incorporated throughout the System Test process. Implementation of these activities will be monitored during the course of System Test, and the results of the implementation will be analyzed during reviews of ongoing documentation and practice.

4.4.2 Documentation of System Test Activities and Results

Documentation of the System Test includes both system test activities and system test results, as described below.

4.4.2.1 Review of Incidents, Issues and Test Results

The CGI Testing Manager and Leads will meet weekly to review incidents, issues and testing results. This meeting will also be a venue to discuss updated and estimated changes that may impact testing.

Regularly scheduled briefings with the DCF Full Time SMEs will be held to review testing issues and progress. Incidents will be reviewed during this briefing. Any adjustments to the severity of the incidents will be discussed during these meetings, and concurrence reached by the team during the course of the meeting. Any issues identified will be documented in the Issues Management Database to ensure prompt resolution. Progress will be reviewed regularly to identify any potential areas where additional resources may be needed.

4.4.2.2 Methodology for Documenting System Test Results

The script tracking spreadsheet will be developed to track daily testing activity. Each day testers will document their progress and activities while executing the test scripts. As each piece of a script is tested or retested, the assigned tester will document the successful execution of that script component in the script tracking spreadsheet. The tester will indicate if an incident is encountered in the execution of a script. Incidents will also be documented in detail through the formal Incident Tracking Database, which provides information for diagnosis and prioritization of the incident for resolution. All activity tracking reports will be reviewed daily and maintained throughout the test period as a means of monitoring test results.

The successful completion of all script components (including initial tests, required retests, and off-script testing) as documented by individual testers and verified by a Team Lead provide the basis for a script’s release from System Test. When all the System Test scripts have been executed successfully, System Test will be considered complete, as specified in the exit criteria.

The CGI Testing Manager will review the script tracking spreadsheet daily to determine if; (1) each component of the release has been tested, and (2) if Regression Testing has been completed.

Throughout the execution process, screen prints will be used to document incidents, as needed for clarification purposes. Upon the completion of System Test activities, final screen shots will be created for documentation purposes.

4.4.3 Quality Assurance of the System Testing Process

Quality assurance of the System Test process includes both quality assurance and validation and verification activities, as described below.

4.4.3.1 Quality Assurance

Throughout the testing process, quality assurance reviews will be conducted to make sure that testers adhere to standards and to identify any potential issues associated with the quality of the System Test process. The CGI Testing Manager and designated DCF Management/Staff will review documentation that will include:

Review of Incident Tracking Database to identify communication issues between the testing and development teams;
Review of Incident Tracking Database to identify types and severity of incidents identified;
Review of test script tracking report to identify any modules with significant numbers of defects or issues;
Review of scheduled tasks completed to ensure timely completion; and
Review of incident resolution to ensure that repairs and retests are completed within the approved Project Schedule. Incidents that are categorized as Critical are expected to be corrected immediately. Developers will notify the tester on the estimated amount of time to fix the incident. Other categories will be prioritized as to when the fix is completed. CGI and DCF will work together on the prioritization of non-Critical and non-Serious incidents.

4.4.3.2 Validation and Verification of the System Testing Process

Validation and verification activities will utilize the requirements tracking tool and design documentation to support validation and verification activities. The validation and verification process makes use of FSFN design products at several levels to assure the software works as designed. Additionally, the software testing process provides iterative levels of testing, with System Test providing the final verification of design and web development activities prior to the release of the software for DCF User Acceptance Test.

4.4.4 Risk Management of the System Testing Process

Software testing mitigates against the risk of implementing software that fails to meet the business requirements and system design specifications. The test process, therefore, is a critical phase in the software life cycle, and management of the test process is crucial to mitigate against the risks of improperly conducted testing.

Risk management measures will include the following:

Implementing the test methodologies outlined in this plan;
Identifying staff who will be part of the testing team and ensuring their availability for participation;
Providing adequate training for each testing resource;
Completing verification checks on the test environment in ample time prior to the commencement of testing to correct any issues to ensure that testing begins as scheduled;
Completing and verifying test scripts prior to testing to correct any errors in the flows and to ensure that testers have accurate test scripts, which will reduce delays during the script executions (Functional Team members creating System Test scripts verify their work utilizing the approved topic paper designs. Additionally, during testing, the tester running the first execution completes the final verification against the application);
Reviewing incident documentation to identify any critical issues at the earliest possible time;
Conversion and reference data are ready;
External systems are available and ready for interface testing; and
User ID, roles and access are identified and ready.

5 Volumetric Testing

5.1 Purpose

Performance testing, as with other forms of testing that occur throughout the application project lifecycle, is a risk mitigation tool. More specifically, performance testing is used to verify that the system can sustain a given service level under load through the use of volumetric, load, and stress testing.

5.2 Objective and Scope of the Volumetric Test

Florida FSFN has three planned releases comprising of Release 1 – Intake, Release 2 – Full SACWIS functionality and Release 3 – Additional Interfaces. The performance test will be executed prior to each release and the test and tuning activities will focus on the release in hand. An initial Release 1 Volumetric Test was performed by CGI at the beginning of the FSFN project utilizing the CGI baseline code base. The subsequent Volumetric Testing will be similar to the initial test but will be executed after the Florida modifications have been made to the code base and the code has been system tested. CGI will work with DCF to determine the appropriate environment to conduct a valid performance test of the application.

5.2.1 Volumetric Test Objectives

The following are the objectives for the Release 1 Volumetric Test:

Determine if the application will perform acceptably in accordance with the requirements of FSFN users based on predicted load forecasts.
Determine FSFN end user response time under normal usage conditions.
Determine FSFN application and database server response time and server behavior under full load conditions.
Resolve any code related performance issues.
Resolve any configuration (Linux, Application Server, ZOS, DB2 8.1) issues that arise.
Resolve any database (indexes, SQL) issues.
Document the performance results, configuration and tuning information

5.2.2 Volumetric Test Assumptions

There are certain assumptions upon which the Release 1 Volumetric Test is based. The following is a list of targeted transaction rates and response times that will be used for the test.

5.2.2.1 Target Transaction Arrival Rate

The basis for the determination for the anticipated FSFN transaction counts will be derived from past CGI SACWIS projects and analysis of key business activities in HSn data. Our production experience has continually demonstrated that our SACWIS systems have a concurrency rate of 33%. This concurrency rate has consistently generated an arrival rate of 4.53 transactions/second for 1,320 concurrent users in Wisconsin: (4000 * .33).

Given Florida’s stated concurrency rate of 66%, this would translate to an arrival rate of 16.20 transactions per second for 4,620 concurrent users (7,000 * .66). For database I/O, each CGI SACWIS web transaction generates on average 7 – 15 database calls. This means that for Florida, the system would generate between 113 and 243 database calls per second. Therefore, the Florida FSFN database must be able to handle up to 243 database calls per second.

5.2.2.2 Target Response Times

The targeted on-line response time targets as specified in the ITN are listed below.

CGI will use these targets to determine the success of the volumetric tests.

	Update			Query
	Simple	Typical	Complex	Simple	Typical	Complex
Total Transaction Time (seconds)	1.00	1.72	5.90	1.00	2.35	5.90

5.2.3 Volumetric Test Environments

CGI will work with DCF to determine the appropriate environment to conduct a valid performance test of the application. Minimum requirements of this environment include the following:

MIPS: 300
Memory: 11.5 GB
Disk Capacity: 125 GB

The following is CGI’s environment recommendations:

The volumetric test will be executed for the on-line FSFN application. The application server and database server are the components that will be performance tested. The CGI Team will perform the Release 1 Volumetric Test using a test environment that is similar to the production environment. Volumetric Testing will begin with full production volume testing on a CGI hosted application servers and databases running in the Linux environments. Once the performance of the application has been verified on this environment, a second series of full production volume test runs will be executed on BEA Web Logic application servers running on Linux with the DB2 Version hosted on z/OS for the database. These tests will be run during off-peak load hours with work-load manager CPU limit rules in place to protect the HSn production environment from being impacted by the workload. These final volume test runs on the planned deployment environment will verify that all SQLs perform as expected on the mainframe DB2 system and that the environments are properly deployed in a way that supports the planned workload.

In order to monitor all web and database calls that will be occurring, the CGI Team will use a number of performance monitoring tools. These tools included:

Silk Performer 7.

The enterprise load generation tool used to mimic system load

SAS/MXG

Produces batch performance reports

TMON for DB2

Monitors CPU times for active DB2 SQL transactions

DB2 RMF Reports

Gathers point in time statistics for the DB2 database and zOS environment

SACWIS Transaction Statistics

An internal SACWIS statistics gathering tool. The SACWIS Application has its own internal monitoring capability that tracks the amount of time each transaction spends on the application and database server. This combined time is recorded in the TRAN_STATS table.

netstat, vmstat, top

Linux OS Commands to monitor CPU usage, virtual memory, and network traffic

Web Logic Internal Monitoring

Allows tracking of connection pool usage, web container threads, current servlet threads and provides advanced logging

5.2.4 Volumetric Test Scenarios

Stress Test:

During the Stress Test portion of the performance testing, the CGI Team will run the different performance test scripts running 100 concurrent transactions for a total of 60 minutes. The CGI Team will utilize this approach for stress testing the application to determine if the production code/hardware/configuration fulfills the requirements.

Extended Load Test:

During the Extended Load test, the CGI Team will perform tests that will closely match the maximum usage of the system during a normal working day. In order to achieve this, the CGI Team will leverage the transfer SACWIS production numbers to determine usage patterns during an 8 hour period. Using these patterns and what is predicted to be Florida’s worker count and case count, the CGI Team will simulate the hourly usage of the FSFN system. The CGI Team will then configure the performance test tool to match the calculated usage as well as to run our various scripts utilizing user lag, or thinking, times between transactions. This will provide the CGI Team an accurate view of what a typical usage day will look like in Florida, and result in an accurate Extended Load test. This test will verify the stability of the system under load for extended periods of time.

5.2.5 Volumetric Test Scripts

The CGI SACWIS system is an extensive application with over one thousand distinct functional transactions. It is critical that performance testing reflects the real world usage of the system, and that the test strategy include those transactions that are key to the system and those transactions that are of high volume.

Before writing the performance test scripts, the actual system usage of the transferred SACWIS system will be analyzed. Key and high volume transactions will be identified, and they will be included in the test scripts. For the Florida performance tests, the CGI Team will keep the intent and the structure of the original tests. The scripts will perform logins by random workers who are assigned to random cases with searches of random case and providers. This allows for a more real-life test.

5.2.6 Volumetric Test Data

The anticipated data source for volumetric testing is HSn production. The CGI Data Conversion team will convert the HSn data to FSFN in preparation for UAT. The volumetric testing will then be performed using the converted data. The initial volumetric test that was performed at the beginning of the project did not use production data. However, it is important that this volumetric test utilizes production data to provide a close representation of the expected database size of FSFN in Release 1. It will also provide greater accuracy in determining the performance of FSFN under a load.

5.2.7 Monitoring and Tuning

Monitoring system health is a key factor while executing a performance test as it forms the basis for tuning. There are various elements that should be monitored on each layer and selecting the right granularity is always the challenge. For purposes of this performance test, the elements that need to be monitored are listed below. These are the same elements used to tune the system for performance.

· HTTP Server

· Bea Web Logic Application Server

· Database Server

· Network / IO

5.2.8 Volumetric Test Results

At the completion of the volumetric test, results will be compiled and summarized.

These results will include the following:

· A description of the test scenarios and test data to provide a better understanding of the business cases that were used to validate the application performance;

· Throughput;

· http hits;

· end user response times;

· application transaction statistics;

· problems encountered;

· pre and post tuning activities; and

· recommendations for moving forward

These results will be focused on validating the FSFN application performance and satisfying the volumetric test objectives listed above. The performance test outcomes will be summarized to identify any key resource restrictions and potential changes to what is currently in place to accommodate what is needed to support the FSFN application.

5.3 Management of Volumetric Test Activities

The CGI Technical Manager will lead the Volumetric Testing effort. As the coordinator of this testing phase, the CGI Technical Manager will:

· Direct the execution of the Volumetric Test in accordance with the provisions of the test plan;

· Establish the Volumetric Test schedule;

· Monitor the test process to identify issues and ensure against schedule slippage;

· Manage the team’s assignments, with assistance from the Team Leads, and coordinate assignments;

· Monitor test results to identify problems, any potential rework, and tuning activities;

· Provide reports on test status and progress.

6 Incident Tracking Process

The incident tracking process includes the definitions of software deficiencies, the process for the analysis of software deficiencies, a description of the prioritization of software deficiencies, and the methodologies for tracking, resolving, communicating, and retesting repaired defects, as defined in the sections below.

6.1 Overview

Incident and defect management facilitates the process of actively tracking incidents from early identification and confirmation that they are defects to final resolution. Incident reports are produced to provide management with insight into problem areas and highlight areas for potential improvement.

All problems identified during testing will be tracked in the Incident Tracking Database. Incidents will be tracked from the time they are found and logged through the fix and retesting. Items are not considered resolved until they have been retested and the testers confirm that the fix addressed and resolved the identified problem. As problems are corrected, unit and system tests will be repeated.

6.2 Definition of Software Deficiencies

An incident is an unexpected or unexplained result. With other similar SACWIS projects, an incident is usually identified when actual results do not match expected results during testing. A defect is an incident that identifies a true problem in a system component that must be corrected. While defects are most often associated with software components, defects can be identified in any part of the solution, such as designs, documentation, test scripts, environment configuration, data set-up, or business processes. During System Test, testers will categorize the incident per the definition. Disagreement in categorization will be resolved by the CGI Testing and Technical Managers. Defects will be classified by severity:

· 1-Critical – A system error which prevents usage of the application and causes significant disruption to agency operations. Critical errors should be remedied as soon as possible and migrated to production as soon a fix is ready.

· 2-Serious – Serious system errors prevent use of the system without significant work arounds. Serious system errors should be remedied as soon as possible and migrated to production on the next scheduled build.

· 3-Moderate – Moderate system errors cause a low degree of disruption and should be scheduled for remedy as time permits.

· 4-Minor – Minor system errors are system deficiencies which do no inhibit use of the system and cause little to no disruption in operations. They should be fixed only if time permits.

· 5-Enhancements – Enhancements are requests for new functionality outside the scope of the original design. They should only be addressed through the change control process.

· 6-Usability – Usability issues with the application are not incidents and should be brought to the attention of the CGI Testing Manager.

· 7-Training – Training issues associated with training documentation are not incidents and should be brought to the attention of the CGI Testing Manager.

In addition, each Incident is further categorized with an Incident Type that helps determine who or what team holds responsibility for further assessing the Incident, determining if a fix is needed, and completing the necessary fix. Exhibit 6.2-1 presents a list of proposed Incident Types:

Exhibit 6.2-1

Incident Type Definition

Incident Type	Description
Batch	The batch code is not performing as expected
Conversion	The conversion code is not performing as expected
Database	The database is not performing as expected
Interfaces	The interface code is not performing as expected
Online	The online application code is not performing as expected
Performance	The application transactions are not providing the expected response time or are using too many system resources
Reference Data	The reference data was not set up correctly
Report	A report is not performing as expected
Template	A template is not performing as expected

6.3 Usability and Training Issues

Testing outcomes pertaining to the usability of a feature are not incidents. These issues should be brought to the attention of the CGI Testing Manager. Usability issues will then be directed to the Project Management Team for further disposition. (e.g. Tabbing sequence on a window)

Testing outcomes pertaining to the specific training needs of the user are not incidents. These issues should be brought to the attention of the CGI Testing Manager. Training issues will then be directed to the Training Manager(s) for further disposition. (e.g. Assignment of an Intake and the roles associated with each.)

6.4 Analysis of Software Deficiencies

Throughout System Testing, testers will document the successful completion of test activities and their detection of incidents according to a structured process. At a scheduled time, the CGI Technical Manager, the CGI Testing Manager, and members of the project team (including both DCF and QA Vendor representatives) will meet to examine the documented incidents. During this analysis, this group will:

Review completion of test scripts;

Determine the number and types of incidents identified;

Rank incidents according to their priority; and

Assign incidents for correction.

This review will allow validation of incident reports entered by testers. Entry of incidents into a tracking system allows the Team Leads to document, categorize, prioritize, and assign incidents for correction. The use of the script-tracking tool also allows for the development of daily and summary incident reports.

6.5 Prioritization of Software Deficiencies

Defect severity levels are described in detail in the ‘Definition of Software Deficiencies’ section of this plan. “Critical” will receive the highest priority rating, followed by “Serious”. As “Critical” and “Serious” incidents are resolved, “Moderate” and “Minor” defects will be repaired. Any adjustments to the assigned severity of a defect will be discussed and agreed upon by the project team during the regularly scheduled meetings.

6.6 Tracking of Software Deficiencies

Software deficiencies will be tracked using the Incident Tracking Database. The Incident Tracking Database is discussed in detail in the Incident Tracking Database section (Section 8) of this plan, while the methodology of tracking is discussed in this section.

Each incident logged will incorporate a clear, concise description of the incident, identifying test case information and a detailed description of the steps to recreate the defect. Additionally, the documentation will include the names of each person involved with this incident, such as the tester, programmer, or approver, as well as the dates associated with each step. Defect type (e.g., code, reference data), severity of defect, and testing phase (e.g., System Test, User Acceptance Test) will also be identified for each incident. Reports and/or Queries can be generated with specified parameters to provide detailed information for review purposes. The Incident Tracking Database allows for the individual assignment for resolution and retesting of software deficiencies.

6.7 Communication

Clear communication between the testing team and the development team is essential in increasing the efficiency of the test process. During the testing phases of an increment, development and design staff will have regularly scheduled meetings to discuss the status of incidents and communicate relevant issues. In addition, the incident tracking database serves as a key communication tool on incident and testing status and will be monitored frequently by development and design staff throughout System Test. The process and format for tracking and communicating the status of test scripts is discussed in Section 9.1 of this document.

6.8 Retest Process for Corrected Deficiencies

Upon repair of a defect during System Test, the programmer will conduct an initial unit test of the resolution. Subsequently, a member of the test team will be assigned to retest the repair in the code/unit test environment. If the tester approves the repair, the incident will be closed, the corrected code will be migrated to the appropriate environment, and the related System Test script, when applicable, will be executed again in its’ entirety (e.g., end to end) to verify successful integration of the repair. The following flow depicts the retest process.

7 Roles and Responsibilities

The CGI Team leads System Test with coordinated assistance from DCF and QA Vendor. “DCF staff” as used here is intended to include staff from one or more of the following: DCF FSFN project staff; IS staff; and non-project DCF staff enlisted to assist with conducting System Test. The System Test will reflect the integrated team approach used throughout the FSFN project. The project work plan will include tasks and timeframes for DCF staff assignments in support of System Test.

7.1 CGI Staff

The CGI team will consist of the Functional Team members, Functional/Testing Manager, the Technical Team members, Technical Manager, and the Database Manager. Each team member will have specific roles and responsibilities.

7.1.1 Functional Team Members

The members of the Functional team will carry out test activities for the software releases. The team members will construct and execute structured test scripts that describe typical business processes that will be performed using FSFN. They will follow the defined processes for documenting the completion of test scripts and for recording and reporting any incidents encountered during testing.

The Functional Team will be responsible for:

Creating all components of the System Test scripts;
Ensuring configuration management of the test scripts through the use of SVN Repository;
Ensuring that all components of the assigned test scripts are tested and documented;
Validating the outcome of batch processes, reports, interfaces, and integration of conversion data associated with scripts (as necessary). The technical team will test the interface and conversion programs;
Documenting and reporting incidents, and retesting incident repairs;
Identifying issues for elevation to the Team Leads or Testing Manager, as applicable;
Retesting applicable script steps when incidents have been resolved; and
Participating as required in the reporting of test results.

7.1.2 Functional/Testing Manager

The Testing Manager’s testing role and responsibilities are outlined in detail in Section 3.2 - Management of System Test Activities. The Testing Manager will be responsible for the coordination and management of all System Test activities, and will also be responsible for communicating any updates to the joint test team. Updates may include information regarding modifications to process, changes to test scripts, or other vital details regarding System Test activities.

7.1.3 Technical Team

Members of the Technical Team will construct the software that is tested during each Release. Prior to its release to the System Test team, the features will have been tested in the Unit Test phase and will be certified ready for System Test by its developer and tester. The Technical Team’s major responsibilities during System Test include the following:

Assisting the System Test team in setting up the System Test environment and workstations;
Resolving and conducting initial retest of any incidents identified during System Test;
Maintaining version control on the code through the use of SVN Repository;

· Migrating fixes to the System Test environment; and

· Assist in setup and system testing of batch processes, reports, interfaces, and conversion programs.

7.1.4 Technical Manager

The Technical Manager provides senior oversight to the development side of the System Test process. This manager will approve the System Test schedule from the standpoint of coordination with the development schedule and will approve build releases for testing. Additionally, the Technical Manager will oversee the work of the Developers assigned to resolve incidents and will provide approval of software for retest by the Functional team. The Technical Manager will work with the Functional Manager to prioritize the incidents identified during System Testing.

The Technical Manager is also responsible for the coordination and management of all Volumetric Test activities. The Technical Manager’s volumetric test role and responsibilities are outlined in detail in section 5.3 – Management of Volumetric Test Activities.

7.2 DCF and IS Staff

DCF members will work with the CGI Functional Team to prepare for the System Test. DCF may also choose to enlist members from IS, selected non-project DCF staff, or select other state organizations staff. These DCF-provided staff will accomplish the following tasks:

Provide subject-matter expertise during the development of System Test scripts;
Assist CGI staff with identification of the data necessary to properly exercise the steps contained within the test scripts;
Provisioning and configuring resources for System Test and Volumetric Performance Test
Creation of or access to applications maintained by DCF that have interfaces to or from FSFN; for Release 1 this includes:

Creation of HSn application test region on the Mainframe;
Provision of data download capability from the HSn test region to a CGI DB2 FSFN database server;
Please refer to the individual interface partnering agreements for a detailed list of activities required to support testing of the Release 1 interfaces and reports (PhoeniX interface, MCTS/FDLE interface, Client Photo interface, and data downloads to the data warehouse to support reporting and CBC extracts).

Assist the CGI conversion team with the identification and validation of any conversion data to be used in System Test, prior to the actual start of System Test;
Data cleanup for converted data and data verification for interface programs. This includes pre conversion data cleanup and cleaning up the data issues in HSn that are uncovered when validating convert data in the FSFN application;
Assist CGI staff with the identification and setup of security profiles to be tested in conjunction with the running of the test scripts;
Review the System Test Plan;
Participate in the pre- and post-test briefings (i.e. System Test Kick-off Meeting and Test Results Review Meeting);
Participate in the observation during execution of System Test scripts;
Monitor the progress of System Test and results with CGI management;

· Review and disposition of incidents identified during System Test in accordance with the incident tracking process; and

· Assist the CGI team working on interfaces with the identification and validation of any interfaced data to be used in System Test, prior to the actual start of System Test.

The following represents the anticipated resource level and involvement of DCF and partner staff to perform the tasks listed above:

· DCF IS Network team: 20 hours

· DCF IS DBA: 20 hours

· DCF IS Security team: 40 hours

· DCF IS Server support: 20 hours

· DCF Release 1 Interface Testing: 410 hours (breakdown by system and primary contact is included in the FSFN Interfaces Plan)

· Mainframe system programmer for performance test: 40 hours

· DCF Conversion support and data clean-up: This is included in the Data Conversion Plan

· DCF Subject Matter Experts: 80 hours

· DCF Management: 30 hours

7.3 QA and IV&V Vendor

The DCF QA and IV&V Vendors are responsible for reviewing and approving contractual deliverables in order to determine whether standards are being met, plans are conducted as agreed, and documentation is being completed.

For System Test, the QA and IV&V vendors will be responsible for the following tasks:

Reviewing the System Test Plan;
Monitoring the System Test for compliance with System Test Plan; and

· Participating in other tasks as requested by DCF.

8 Incident Tracking Database

8.1 Purpose

The Incident Tracking Database is the tool used to track relevant issues during both the pre and post design phases of FSFN—from initial system testing throughout the production phase. The Incident Tracking Database provides an organized and efficient method to record any issues, “bugs,” requested enhancements, and fixes to FSFN. Reports can be run using various sets of parameters and values that are chosen by the user, depending on the type of information sought. Users of the Incident Tracking Database include all management, functional and technical staff on the project, as well as QA vendors and client staff. The Incident Tracking Database facilitates coordination between roles on the FSFN project, and as a result saves valuable time and energy. It is important to note that the Database maintains information related to all types of changes to the FSFN application; even though the database is commonly used to record problems or issues within the application, it is also used to record and track events such as enhancement requests, change order requests, template updates, and revised reference data values. For that reason, the Database acts as an inherent history record of events related to the FSFN application.

The CGI Incident Management tool is a mechanism for capturing detailed information regarding the progress of incidents found during the testing process. This tracking tool provides utilities by which:

· Testers document system Incidents;

· Development, Test, and other Team Leads determine the severity and priority of the defect, then assign it to a developer for analysis;

· Developers create and document the necessary fixes;

· The Testing Lead reviews the Incident resolution and assigns to a Tester; and

· A Tester verifies the resolution by re-testing the module and associated Test Case(s).

The Incident Tracking tool is used to document and track Incidents detected during the subsequent phases of testing and implementation. The review and assignment process are maintained within the test team. After re-test by developers and testers, the corrected software or other artifact is released. Exhibit 8.1-1 illustrates the flow of the Incident Management Process.

Exhibit 8.1-1

Incident Management Process Flow

The following is a typical scenario involving the Incident Tracking Database:

During the design/pre-production phase, Designer Dave is system testing a placement screen in FSFN. He clicks on the Save button and receives an error message that should not have occurred. Designer Dave opens the Incident Tracking Database and searches to see if, perhaps, this issue has already been documented. He searches using a value of “SM10a Placement” in the Topic Number field. This query will retrieve any incident related to the Placements topic that has been recorded in the Database. Designer Dave’s search retrieves no incidents relating to his issue, so he creates a new incident in the database. After filling out all of the fields in the Incident Detail screen of the Incident Tracking Database and choosing a status of “Open,” he clicks on the Save button. Designer Dave notes that he has successfully created incident #13, and then continues his system testing of FSFN.

Technical Lead Libby reviews the Master Defect List (MDL), assigns Incident #13 to herself, and changes the status to “In Progress”. After reviewing the information that Designer Dave recorded, Technical Lead Libby has an understanding of what happened, and uses the information entered by Designer Dave in the Steps to Recreate field to recreate the incident. Technical Lead Libby now works on the incident and fixes the problem. Once it is fixed, she records data in the Comments and Resolution fields and changes the status to “Fixed/Awaiting Build,” then to “In System Test” after the next build is run by the technical team that night.

The next morning Designer Dave checks the status of incidents that he has recorded in the Incident Tracking Database and sees that Incident #13 has a status of “In System Test” and is ready to be retested. Designer Dave retests, sees that the issue is no longer occurring, and changes the status to “Resolved”.

Three months later during another round of system testing Designer Sandy comes across an error message that she does not believe should be occurring in FSFN. Designer Sandy asks Designer Dave about this message, because she vaguely remembers something similar occurring in Placement several months ago. Designer Dave cannot remember anything about an error message, so Designer Sandy searches the Incident Tracking Database by entering “SM10a Placement” in the Topic Number field, and “Designer Dave” in the Reported By field. Incident #13 is returned from this query and Designer Sandy quickly reviews the history of the incident, realizing that her issue is indeed different. Designer Sandy records this new incident in the Database so that it will be reviewed by the technical staff.

In the following sections, you will see the screens and fields used in the above scenario. The purpose of this is to gain a better understanding of the Incident Tracking Database.

8.2 Access

The Incident Tracking Database is web-based and will be stored at a URL accessible to all project staff. The web-based nature of the Database allows for the storage and retrieval of large amounts of information, without affecting system performance. Also, the web-based Incident Tracking Database allows for multiple users to be simultaneously working from their own PCs without system lock-ups.

8.3 Navigation

Upon accessing the Incident Tracking Database, the following screen will display:

The user is able to:

Search out an existing incident using criteria such as Incident ID, Topic Number, Reported By, etc. The user can search using one or many of these criteria—the greater number of criteria used, the more narrow the search becomes. Searching the system without data entered into the criteria fields is not recommended, as it will retrieve all incident records in the Database.

Create a new incident by clicking on the Create button. (Note: It is preferable to search out an incident prior to creating a new one in an effort to avoid duplicates.)

8.4 Searching an Existing Record

If the user decides to search in the Database, they will enter appropriate search criteria on the Incident Search page. After the Search button is selected, the results will appear in the box labeled Search Results at the bottom of the Incident Search page. If there are no records that match the search criteria entered, the user will see the following screen with a message in the bottom left corner, “0 Incident(s) Returned”.

If, instead, the search criteria yield results, the user will see the following screen:

To select a specific record on the Search Results screen, the user will click on the underlined (hyperlinked) number under the Incident ID column. This will bring the user to the Incident Detail page (please note scroll):

The Incident Detail Page displays all of the information that was entered when the incident was created, and all fields are editable. Once the user has made changes, they will need to select the Save button in order to save those changes. When the Save button becomes disabled, the user knows that the changes have been saved. To navigate back to the main Incident Search page the user will click on the Close button. The user will once again have the option to either search an existing incident, or create a new incident record. The fields found on the Incident Search and Incident Detail page are described in the Field Descriptions (tables) section of this manual.

8.5 Creating a New Incident Record

If the incident identified by the user is not returned in a search, then a new incident record needs to be created. This is accomplished by selecting the Create Incident button in the bottom right corner of the Incident Search page. This will launch a blank Incident Detail page. If an Incident Detail page has already been opened by the user, but not yet saved, the user will get an error message when they click on the Create Incident button, “Incident Detail is already open. Please complete the current work.” This is to prevent the creation of duplicate “new” incidents. Once the user clicks on the Create Incident button the new Incident Detail page will display:

Once the user has filled out the appropriate fields on the new Incident Detail page, they will select the Save button to save their work. The user may also click on the Save button at any time during the creation of the new record to save their work up until that point. At that time the Incident ID automatically assigned to the record will appear in the upper left hand corner of the record. Users recording and/or maintaining incidents, in the Database should regularly review the status of their incidents in order to be responsive to any status updates in the Database.

8.6 Field Descriptions (tables)

The tables and screen shots below describe the fields displayed in the Incident Tracking Database pages. The tables for each type of page in the Incident Tracking Database are preceded by a screen shot of the page. The blue shaded fields indicate required fields, and the user must complete these fields prior to saving the record in order for a successful save to occur. If the user attempts to save the page without completing these required fields, an error message will display, “You must correct the following errors before proceeding:,” and lists the fields that have not been filled out by the user. Once the user corrects these fields, they must click on the Save button again in order for the record to save.

8.6.1 Incident Search Page

The following is an example of an Incident Search page.
Field level detail descriptions are listed in the table below the screen shots:

8.6.1.1 Incident Search Page - Search Criteria group box

The majority of the fields in the Search Criteria group box will be discussed in the Incident Detail Page - Incident Summary section of this document. The exceptions are discussed below:

Field Name	Description	Comments
Text Search	Allows user to search for incidents that contain certain text in a specific (or all) field(s)	User entered, not required.
Search Button	Searches based on the criteria entered in the Search Criteria group box.

8.6.1.2 Incident Search Page - Search Results group box

Field Name	Description	Comments
Incident ID	Unique identifier for each record in the Incident Tracking Database.	Hyperlinked number that will take the user to the specific Incident Detail page.
Topic	Displays the value that was selected in the Topic Number field for the Incident.	Display Only
Title	Displays the Title that was entered in the Problem Area field for the Incident.	Display Only
Severity	Displays the value that was selected in the Severity field for the Incident.	Display Only
Priority	Displays the value that was selected in the Priority field for the Incident.	Display Only
Status	Displays the value that was selected in the Status field for the Incident.	Display Only
Reported By	Displays the value that was selected in the Reported By field for the Incident.	Display Only
Assigned To	Displays the value that was selected in the Assigned To field for the Incident.	Display Only
Create Incident Button	Launches a new Incident Detail window.
Close Button	Closes the Incident Tracker window.

8.6.2 Incident Detail Page

The following is an example of a completed and saved Incident Detail page.
Field level detail descriptions are listed in the table below the screen shots.