2.0 INTRODUCTION

The purpose of this User Acceptance Test (UAT) Plan for Post Statewide Implementation is to formally define the revised testing processes used by the ACESS User Acceptance Test (UAT) Team when testing defect corrections and approved/funded change requests. This plan is based on testing procedures used for Increment One but has been modified to consider considerations necessary to support using the new build/release process, on-going introduction of change requests to the system (as opposed to a one-time deployment) and post-implementation activity. This plan will be in use until UAT for the Cúram Upgrade begins and then re-implemented following the deployment of the upgrade to production.

MAXIMUS has worked with the State and IBM teams to refine the build process and to establish proposed timelines for the testing process and potential production deployment dates. The process described in this document reflects the known testing procedures as developed by the State, IBM, and MAXIMUS since the ACESS Statewide Implementation.

2.1 Testing Scope and Objectives of UAT

The defined scope of the post implementation user testing focuses on specific functionality impacted by changes to the application because of defect correction or change requests. Testers also complete automated and/or manual regression testing to test the ongoing overall functionality of the system. The UAT objective is to focus on the application’s appearance, workflow, and requirements verification. .

2.2 Key Assumptions

There are number of assumptions that will guide the ongoing UAT process. These assumptions include:

q State business staff will prioritize all defects and change requests on an ongoing basis. IBM and/or the State developers will focus work efforts on the highest priorities using the prioritized list as a guide.

q Potential builds for user acceptance testing are created as coding changes are completed and checked into the development (DEV) environment. This process can occur daily.

q All builds undergo system testing. Only builds selected as candidate builds for production will undergo UAT.

q The UAT environment is refreshed only on an exception basis. This will ensure defects and change requests are tested under as similar conditions to production as possible including any data conversion requirements that may be appropriate.

q Staff will continue to use ClearQuest to document the status of specific defects and change requests. For tracking purposes, each change request is assigned a defect number when ready for promotion to a build.

q Minimum development time for a fix (including coding and system testing) is two (2) weeks.

q Once a build is created and selected for UAT and possible deployment to production, no new defects will be added to that build. Only corrections for existing defects included in that specific build will be considered using the exception process. The exception process is invoked only if agreed to by the State and IBM and the changes can be coded and tested within the timeframes that allow deployment for the scheduled release date. For the exception process developers make fixes within the build and re-promote that build to UAT for testing. The fix must also be included in a subsequent build.

q If for some reason the State wants to add defects or change requests for deployment, a new build that includes those defects must be selected and fully tested. A new production deploy/release date would be established considering minimum development and testing timeframes.

q All defect corrections and changes must have adequate development time and adequate testing prior to deployment to production. Only “emergency” or “hot fixes” for critical issues will be deployed using an accelerated development, testing, and deployment process.

2.2.1 Test Documents

As a part of the UAT process, the project will use several documents and/or files to guide the testing efforts. These documents and/or files include:

q System Test Build Status Report (Excel spreadsheet) – documents the contents of each build and the system test status. Used by UAT to identify candidate builds for testing. Each build is a compilation of all software modules that have been checked into the DEV environment through the point in time when the build is created.

q Defects and/or Change Requests as entered into ClearQuest. Entry provides tracking of the defect and/or change request through each stage of development. Requirements and design specifications are attached to the ClearQuest records to further define the changes and to obtain concurrence from the State regarding the proposed changes.

q Test scripts to document the specific steps used to test a defect correction or change and the expected outcomes. The test scripts are in paper format as well as automated. If not already identified in the user requirements or technical design documents, MAXIMUS would recommend including references to the specific script names, numbers, and/or location of the scripts used within the ClearQuest record.

q Test results prepared by each tester. This includes copies of screen prints and/or written description of the test results. These are maintained by the UAT Lead.

q Recommendation to deploy a build to production. The UAT Team Lead (with concurrence from the business leads) will generate the recommendation to the State Project Director.

q Authorization from the State Project Director to IBM and/or State technical team to deploy a specific build to production and the requested deployment date. This will most often be in the form of an email.

q Release Notes documenting the changes to be deployed to production and any impact to the users. Change readiness staff has responsibility for development of this material.

q Notification to training of changes to be deployed as a part of a specific production release. Change readiness leads have responsibility for sending this notification. At the present time, training has asked that this information be shared at the time the actual contents of the build to be deployed have been identified.

3.0 Build/Release Process

A build/release process implemented post-ACESS statewide implementation guides the on-going testing of the system within the system and user acceptance environments. This process allows greater independence among the major lifecycle components of the project including business analysis and issue prioritization, development, user testing, and production releases. The process also strategically places authority for promotion of code at various points within the appropriate teams.

Developers will complete work on defect and change requests based upon the prioritization provided by the business analysts. As work is completed, developers check the completed code into the development (DEV) environment. When the next build is created (usually on a daily basis) that new code is included in the build and the associated defect number is associated with that build.

As builds are created, the System Test Lead documents the contents of each new build and initiates system testing. The results of the testing are documented in the System Test Build Status Report (excel spreadsheet). Primarily the status indicates if the defect correction or change was validated or returned to the developer for some reason. ClearQuest is also updated to reflect the validation of the correction/change or return to the developers for more work. If returned to the developers, any corrections will appear in subsequent builds when the code is corrected and then checked in again to the DEV environment.

The excel spreadsheet maintained by the system testers includes the following data elements:

q Defect Number

q Build Number

q Description

q System Test Status

q UAT Status

Each tab within the build only identifies those defects that were added as a part of the build process. However, each build is a compilation of all promoted defects included in prior builds and any new defects. For tracking purposes, change requests are assigned a defect number when promoted to a build and is tracked as a defect from that point forward.

3.1 Selection of Candidate Build for UAT and production

The System Test lead shares the excel spreadsheet of available builds periodically with the UAT lead. Using the information provided in the spreadsheet and any additional information provided by the system testers and developers, the UAT lead identifies candidate builds for promotion to the UAT environment and ultimately the production environments. The UAT Test Lead also notifies the State Technical Lead or IBM regarding any additional requirements to be considered (for example, refresh of data, data conversion) for testing of the selected build.

The following criterion provides a guide for determining the best build for deployment to the UAT (and production) environments:

q build has undergone testing at the next lowest environment

q number of defects included within a build and availability of testing resources to meet timelines

q number of defects within a build validated by testers versus those rejected and returned to developers for correction

q nature of problem(s) identified during system test would not hinder UAT or degrade production environment should a build be deployed

q critical need of specific defect(s) and/or timing and coordination restrictions (e.g., associated training required; implementation has to be a specific date; and communication to field staff) that would hinder deployment within the defined timeframe for the testing cycle. The number of workdays could vary depending upon the number of defects and/or holidays. In most circumstances, the proposed timeline is approximately five (5) weeks.

q existence of isolated functionality that allows a defect to be promoted without impacting existing working functionality within the system (e.g., certain report functionality would never occur if batch jobs are not run)

Under normal circumstances, the build selection occurs two (2) times during each testing cycle. The first identifies build candidate A for testing during the first two weeks of a UAT cycle. If candidate A is acceptable, candidate B is then selected and deployed to UAT for testing.

Once UAT testing of both candidate builds is completed, the UAT lead (in concert with the project’s business and technical leads) notifies the State Project Director via email of the test results and recommended build for deployment (candidate A or candidate B). The State Project Director, if in agreement with the recommendation, then sends a formal request to IBM and/or the State Technical Lead identifying the recommended build and the proposed deployment date. The Project Management Office (PMO), business analysts, and technical leads then work together to determine if the proposed deployment date is viable and/or select an alternate date that allows the deployment with minimal downtime for the system.

Once a build is deployed to production, the UAT testing cycle then repeats itself. There will be times when two candidate builds are not available for user testing. As such, the cycle is modified to include testing of only one build. If that build is acceptable, the build is promoted to production. If there are problems with the build that prevent deployment to production, the UAT testing cycle would restart with adjustments made to the proposed release deploy dates.

4.0 Testing Cycle Timeline and Steps

All testing cycles start with the selection and promotion of a candidate Build A to the UAT environment based upon the criteria defined above. Selection of the specific build for promotion to UAT is based on input from various project resources including, but not limited to, the Development Team Lead, ACESS Business Lead and the UAT Lead. Once testing of candidate A is completed, the development and/or testing path can vary depending upon the decision regarding viability of candidate A for deployment to production. The following table identifies key milestones or decision points within the testing phase and the associated paths. The timeline shown assumes no holidays and reflects workdays.

Table X: User Acceptance Testing Timeline (Workdays)

Step		Production Release 1
		Normal Build Process		Exception Process
1. Promote Candidate A for Testing	Day 1
2. Complete Testing of Candidate A	Day 10
3. Determine If Build A is Candidate “as is” or Invoke Exception Process - If candidate “as is” times in Normal column are followed (go to Step 4) - If exception process invoked – timeline in Exception column is followed (go to Step 7)	Day 10
	NORMAL		EXCEPTION
4. Identify Candidate B for Testing	Day 10
5. Promote Candidate B to UAT for Testing	Day 11
6. Complete Testing of Candidate Build B (go to Step 11)	Day 20
7. Return Candidate Build A to Developers			Day 10
8. Complete Correction and System Testing of Candidate Build A			Day 20
9. Promote CORRECTED Build A to UAT			Day 21
10. Complete testing of Corrected Candidate Build A (go to Step 12)			Day 25
11. Select Candidate Build A or Build B for Promotion to UAT (go to Step 13)	Day 20
12. Accept Corrected Candidate A for Promotion (go to Step 13)			Day 25
13. Promote Selected Build to Production		Day 31

Once the UAT cycle begins, there is key decision points within the timelines described above. These include:

q Day 10 – UAT must decide if candidate Build A is acceptable for production. If the build is not acceptable, the business analysts and developers (IBM and State) must determine if the exception process should be invoked. The primary factor considered is whether the fix can be completed (development and system test) within the proposed timelines.

NORMAL TEST CYCLE:

q Day 10 – UAT must select a candidate Build B if the normal timeline is used.

q Day 20 – UAT must recommend candidate Build A or Build B for deployment to production if the normal timeline is used. Recommendation is forwarded to the State Project Director.

EXCEPTION TEST CYCLE:

q Day 25 – UAT determines if revised candidate Build A is acceptable for deployment to production and make recommendations to the State Project Director. Testing of the revised build includes testing the specific correction and regression testing of all functionality.

5.0 Testing Approach

IBM and/or State developers will complete integration testing in the development environment. When code is promoted to a specific build (usually on a daily basis), State testers will complete system test on each build and update an internal excel spreadsheet with the system test status of each defect. System testing occurs on all builds, UAT only occurs on builds selected as candidate for production.

For each candidate build, UAT staff must determine the testing strategy used for each defect correction; required baseline data, data entry needs, integration impacts and testing requirements. As needed, existing scripts are refined to ensure the scripts include case scenarios that simulate the successful entry and usage of data into the ACESS application by staff on a day-to-day basis. The scripts include generation of tasks/alerts, reassignments to other workers, generation of communications, generation of reports, integration, and other automated required by ACESS to satisfy the approved requirements and detailed design. Testing includes both positive and negative testing approaches. Negative testing addresses the entry of invalid data such as character data in numeric fields, invalid dates, data strings too large for the field size, and unpopulated mandatory fields. The scripts also address circumstances when the entered data is valid, but fail the business rules, such as an application for assistance where all criteria are met except income is too much to qualify

UAT also maintains and uses a series of standard scripts for automated and/or manual regression testing of the overall functionality of the system. Regression testing is necessary to ensure that changes to the system do not introduce defects to other parts of the system. As time allows, staff also perform ad hoc testing as business analysts or other stakeholders identify specific issues/concerns.

The UAT testing effort focuses on the use of the ACESS application based on the documented and approved system requirements and detailed design as documented in Rational RequisitePro or modified via the approved User Requirements Documents (URDs) or Technical Design Documents (TDD) attached to the specific defect within ClearQuest. In the process of testing these requirements, users will test and verify online functionality and usability of the system. While UAT does not specifically focus on the technical functionality of ACESS, UAT does validate specific technical requirements for the system such as security features, accessibility, performance, data conversion, and integration with legacy systems.

5.1 Defect Levels and Definitions

Defect definitions are established using two levels. First, there is the severity of the defect, which indicates how damaging the defect is to the testing process. Next, there is a priority assigned, which in the past indicated to development what the prioritization of the defect repair was and in what order it should be scheduled for resolution. Priority levels designated within the ClearQuest are not as important now as the ongoing prioritization completed by project business analysts prioritizing all outstanding defects and approved/funded change requests. This external prioritization completed by the business analysts should drive the development work efforts.

Please note the defect level definitions are different from those used for System Testing. UAT must also consider impact to the pilot and statewide environment when assigning a priority level.

5.1.1 Severity Level

Prior to November 20, 2006, the following severity level definitions were assigned to any defects identified in the ACESS environment including:

q Severity 1 (SV1) - Critical defects are errors that completely halt testing. These critical defects are critical failures of the application such as application crashes, data loss, etc.

q Severity 2 (SV2) - High defects are errors that halt testing in a particular function. They are a defect encountered while using the application that there is no workaround for and cannot be bypassed.

q Severity 3 (SV3) - Medium defects are errors that can be avoided or where an acceptable work-around exists, but impair the function of the application. The State Team Leads will make the decision regarding whether a work-around is acceptable

q Severity 4 (SV4) - Low defects are minor errors that do not require a workaround, such as spelling errors or incorrect images. The application functions without any additional problems.

On November 20, 2006, new severity definitions were implemented to mitigate risks associated with the Cúram Version 4.5 upgrade. In essence, the new severity levels limit the coding efforts that will be required in both the existing Cúram Version 3.0 and Version 4.5. The new severity level definitions are:

q Severity 1:

o ACESS or major functional areas are not accessible or working (e.g., ability to perform searches or establish household, client, or program cases, integration, RFS, communications)

o ACESS has encountered data loss that requires restores from backups or manipulation of the data in the system.

o No Workarounds Exist.

q Severity 2:

o Defect affects 25% or more CPI reports, intakes, and/or investigations.

o Workaround requires entry of false or manipulated data into the system

o Defect requires work activities on cases within the system to stop until a fix is promoted.

o Defect prevents the correct exchange of data with TIPS preventing payment processes from occurring.

o Defect causes communications to the courts to include incorrect case data (e.g., pulling data from the wrong field) or correct insertion into envelopes if centrally mailed.

q Severity 3:

o Acceptable workaround exists for the defect.

o Work around does not require entry of false or manipulated information.

o ACESS functionality is impaired.

o Defect affects very small number of cases or unique circumstances.

q Severity 4:

o Defect exists but does not require a workaround.

o Defect is cosmetic (spelling errors, incorrect image on page)

o System functions without any additional problems.

On a weekly basis, designated staff will review the severity level assignment for all new defects to ensure they are properly assigned.

5.1.2 Priority Level

The following are the priority levels and their definitions:

q Priority Level 1 (P1) - Critical priority is for defects that should be placed at the top of the list for immediate repair. Critical priority is assigned to those requests that must be fixed to allow all testing or testing of specific scenarios to proceed. No defects that are lower than Critical priority should be worked on until all Critical priority items are complete.

q Priority Level 2 (P2) - High priority is for defects that are important to get repaired as soon as possible, but an acceptable work around that can be used in the testing and production environments exists.

q Priority Level 3 (P3) - Medium priority is for defects that need repair, but there is no immediate need. They should be repaired as soon as possible, but not at the cost of higher priority defects.

q Priority Level 4 (P4) - Low priority is for defects that need to be repaired for release, but that there is no pressing need for immediate repair. Examples of these changes include corrections of spelling on screens or other cosmetic changes that if not fixed, do not impair the operation of the system or accurate collection of data.

5.2 Defect Status in ClearQuest

All defects identified in any of the ACESS environments are tracked in ClearQuest. The following states are used to track the status of the defect:

q Submitted – defect has been submitted but is waiting initial review.

q Analyzed – defect has been analyzed by IBM

q Change Control Board – defect, as reported, has been determined to be a change request and referred to Change Control Board

q Assigned – defect has been assigned to a business analyst, developer, or management to address.

q Open – defect is being actively addressed by a developer.

q Resolved – developer has resolved the defect. This could mean the defect has been fixed OR the defect could not be duplicated.

q Rejected – defect has been rejected as the system works as designed OR a cost change request will be developed to request changes to the system.

q Postponed – work on defect has been postponed.

q Ready for System Test - Defect has been corrected and tested by the developer and is now ready for System Test

q Validated by System Test (Validation 1) – defect has been tested by system test staff and is now ready for UAT testing

q Validated by UAT (Validation 2) – defect has been tested by UAT test staff and has been approved

q Closed – defect has been addressed in some manner and is now closed.

5.3 Testing Metrics

On a weekly basis, designated project staff prepare and communicate metrics regarding the number of defects and their associate states to appropriate stakeholders. The metrics include:

q Number of Remedy Help Desk Tickets that are open or resolved

q Number and status of ACESS Help Desk Tickets entered in ClearQuest

q Number and status of defects remaining from Increment One UAT after July 1, 2006 by functional area within the system (Administration, CPI, RFS, Communications, Reports, and Interfaces)

q Number and status of Pilot/Production defects identified since the start of Pilot Implementation (Administration, CPI, RFS, Communications, Reports, and Interfaces)

Tracking the number of defects by function allows testing to decide where to most concentrate their efforts in the next iteration of regression testing and for new features in the same function or similar functions.

5.4 Configuration Management

A separate UAT testing environment exists to support the ongoing user testing of defect correction and changes resulting from change requests.

Version	Version Date	Nature of Change	Submitted
1.0	11/29/2006	Original
1.1	1/22/2007	Revised to Include Informal Comments Provided by State after Approval	1/22/2007