eWiSACWIS
User Acceptance
Test Plan
May 9, 2003
Table of Contents
1. Purpose
of Acceptance Test...................................................................................... 1
2. Scope of
Acceptance Test.......................................................................................... 1
3. Roles and
Responsibilities for Acceptance Test......................................................... 3
3.1 Quality Assurance Team........................................................................................ 3
3.2 eWiSACWIS State Technical Team...................................................................... 4
3.3 Technical
Services................................................................................................. 4
3.4 Operations............................................................................................................ 4
3.5 Program Management Team Members.................................................................. 5
3.6 AMS Staff............................................................................................................ 5
3.7 End-User Testers.................................................................................................. 6
4. The Acceptance Test Environment............................................................................... 7
4.1 Test Databases...................................................................................................... 8
4.2 Testing Aids.......................................................................................................... 8
4.3 Application Software............................................................................................. 9
5.0 The Testing Process................................................................................................. 10
5.1 Test Planning & Preparation................................................................................ 10
5.1.1 Scenario
Development / Utilization.................................................................. 10
5.1.2 Test
Scheduling................................................................................................. 11
5.1.3 Test Preparation............................................................................................ 12
5.1.4 eWiSACWIS UAT Help Line......................................................................... 12
5.2 Test Execution....................................................................................................... 12
5.2.1
Test Support Staff............................................................................................ 12
5.2.2 Test
Verification, Documentation and Resolution............................................ 14
5.2.3 Tester Activity
Tracking................................................................................... 14
5.3 Test Result
Reporting.......................................................................................... 19
5.3.1 Completion Criteria........................................................................................ 19
5.3.2 Acceptance Test Report.................................................................................... 19
6.0 Incident Management.............................................................................................. 21
6.1 Incident Tracking Database................................................................................... 22
7.0 Risks and Assumptions............................................................................................ 243
8.0 Acceptance Testing
Timeline................................................................................... 24
The successful
execution of acceptance test activities involves numerous members of the
eWiSACWIS project team and end-users outside of the project team. These activities range from direct and
primary involvement in task performance to the provision of support or advisory
assistance to test team members.
Test participants will come from two primary populations: the eWiSACWIS
project team and the end-user community.
Project team participants will be provided primarily from the Quality
Assurance Team and from the Program Management Team.
End-users will be provided from areas that will be directly interfacing
with and using the system once it is installed. eWiSACWIS UAT participants will
be drawn from the following user organizations.
Organization Test Location
·
Statewide Adoptions Wisconsin Rapids, Madison
·
Clark County Clark County
·
Lafayette County Lafayette
County | eWiSACWIS Test Lab
·
Milwaukee
County Milwaukee County
·
Ozaukee
County Ozaukee County
·
St. Croix County St.
Croix County
·
Winnebago
County Winnebago County
·
BFS Madison
·
BPP Madison
Users from these locations will undergo training and orientation prior to
performing test activities. The training
portion will consist of completing eWiSACWIS Web Based Training (WBTs) to
familiarize testers with the look and feel of the application. The goal of orientation is to set
expectations regarding the UAT process, its goals and outcomes.
UAT participants will be guided through two approaches. 1) Following UAT
test scenarios developed by the QA team to become familiar with the system; and
2) their own work reflecting regular as well as extraordinary scenarios. They
will be supported by project team members in the execution and validation of
tests.
The state’s Quality
Assurance Team will lead the acceptance testing effort and will perform the
following activities:
·
Acceptance
test planning tasks, including establishing the acceptance test schedule
·
Directing
and coordinating the acceptance test in accordance with the provisions of the test plan
·
Providing
support and assistance in the execution of acceptance test scenarios
·
Executing
acceptance test scenarios as necessary
·
Documenting
and reporting test results
·
Documenting
and reporting unexpected results as incidents
·
Working with
other project team members to analyze and resolve incidents
·
Monitoring
test progress to ensure against schedule slippage
·
Providing
routine and periodic reports on test status and progress
·
Providing a
final acceptance test report
The State Technical Team consists of eWiSACWIS project staff
and BIS staff who manage database, Websphere, iChain, and network
resources. Staff from these
organizations will be responsible for the following.
·
Establishing
the acceptance test environment
·
Installing
application software in the acceptance test environment
·
Performing
all DBA functions in support of testing
·
Managing and
troubleshooting the Websphere server
·
Managing and
troubleshooting iChain
·
Managing and
troubleshooting technical and environmental issues
·
Publishing
eWiSACWIS builds to the acceptance test environment
·
Supporting
County technical staff for remote test locations
·
Ad hoc
report testing
·
ePASS
testing
·
Replication
testing
Technical Services staff will be responsible for setting up testing
workstations, servers and network connectivity. The eWiSACWIS database administrators (DBAs)
will participate in acceptance test preparation, and will work with the test
team members to create and populate the eWiSACWIS test databases as necessary
to support the test. The DBAs will also provide database backup, recovery and
data refresh activities during the testing effort.
Operations staff will be responsible for coordinating the execution of
batch test jobs.
Program management staff
represents the end-user perspective to the project team. They are familiar with
the functional tasks to be performed and have detailed knowledge of the
application being tested. They will perform
the following activities:
·
Assist with
development of acceptance test plan, testing schedule, and test scenarios
·
Executing
acceptance test scenarios
·
Documenting
and reporting test results
·
Documenting
and reporting unexpected results as incidents
·
Working with
other project team members to analyze and resolve incidents
·
Providing
support and coaching to other testers
·
Off-scenario
testing for flow through the application
AMS has responsibility for the following Acceptance testing activities:
·
Training for
acceptance testers
·
Providing
support during acceptance testing — both in the eWiSACWIS test lab and at the
following County locations: Milwaukee County, Ozaukee
County, Winnebago County
·
Documenting
and reporting unexpected results as incidents
·
Resolution
of incidents detected during acceptance testing
·
Providing
updated builds of the application as needed
The following activities are outside the scope of responsibility for AMS.
·
Ad hoc
reporting
·
ePASS
Replication
Those organizations identified
earlier in this document have expressed a desire to be involved in the
acceptance test. These resources are
available on a limited basis to test the portions of the system that impact
their area(s) of expertise. These
testers will come from the following populations:
·
ePilot site: Employees
from Lafayette County
·
County Users: Clark,
Lafayette, Milwaukee,
Ozaukee, St. Croix, Winnebago
Table 2 lists personnel outside
of the project team who have been identified for performance of acceptance test
activities, listed here.
·
Complete eWiSACWIS WBTs
·
Execute assigned acceptance test scenarios
·
Execute ad hoc testing specific to their own
work conditions
·
Document test results
·
Work with the QA team to evaluate and/or report
unexpected test results
·
Provide feedback regarding the application
within their area(s) of expertise
Table 2: User Acceptance Test Participants
|
Source
|
Name
|
Area(s) of Expertise
|
|
Statewide Adoptions
|
|
|
|
Lafayette
County
|
|
|
|
Milwaukee
County
|
|
|
|
Ozaukee
County
|
|
|
|
Winnebago
County
|
|
|
|
Bureau of Financial Services
|
|
|
|
Bureau of Program and Policy
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Acceptance testing
will be supported by two dedicated test databases. The existence of two test database maximizes
the flexibility of the testing effort. The creation of the test databases
assures that testers will be able to access data representing clients, staff,
providers, events and service types, and other information that reflects
standard business information within eWiSACWIS.
Data that testers create during test operations will also be stored in
this database. The existence of a test
database allows testers to fully test eWiSACWIS without being affected by
database changes that occur in the development environment.
The scenario development process takes into account the testers’ need to
access pre-existing data. Each scenario
includes a specification of the data that must be available for successful test
execution. During scenario development,
the baseline data is identified for the tester.
It is expected that baseline data will be part of the initial load of
test data.
Scenarios are designed to be independent of each other. Execution of one scenario will not impact the
data used by another. This ensures that
multiple scenarios can be executed at the same time without data contention
issues.
A backup copy of the database will be created after the database is
initially populated with reference and test data. Additional backups may be
taken at various points throughout testing if it is determined that a need may
exist to to restore the database to a specific point in time.
It is anticipated that there will be multiple points during acceptance
testing when the database will need to be restored to the original
baseline. The primary reason for
restoring the database will be to facilitate re-execution of test
scenarios. Due to the nature of
acceptance testing, multiple testers will be executing the same scenario. This will require that the database be reset
prior to re-executing a scenario.
The testing process incorporates a person-based approach to testing;
automated testing tools will not be utilized.
The acceptance test
team will employ an incident reporting database to report and track incidents
found during testing. An incident is
defined as an unexpected result, or a system feature that does not appear to be
operating as designed. The eWiSACWIS
Incident Tracking database will be used for incident reporting. Please refer to the incident tracking section
of this document for more information regarding this database.
AMS is responsible for installing the application
software to be tested into the acceptance-testing environment. The application will be installed on the
network. If updates are necessary to the
application software, due to incident resolution, AMS will install new build(s)
into the acceptance test environment. Test builds will be kept under version control,
using PVCS, to assure adequate control over the software being tested.
The acceptance testing process has three major stages:
·
Test Planning
& Preparation
·
Test Execution
·
Reporting of Test Results
In previous WiSACWIS acceptance test efforts, the QA team
has observed that meticulously detailed test scripts create an environment
where test participants become focused on the process of completing the script
rather than concentrating on the application and workflow being tested. In developing the test approach for eWiSACWIS
UAT it has been decided that high-level test scenarios will be used rather than
detailed step-by-step test scripts. With
this approach it is important that testers complete the eWiSACWIS Web Based
Training in order to familiarize themselves with the application screens and
workflow. Additionally, it will be
important to have project staff available to guide users through unfamiliar
areas of the application.
A test scenario
provides a framework for standard testing and familiarization of eWiSACWIS
components. Each scenario includes a
collection of eWiSACWIS functions which approximates the use of the system in a
typical business situation. Table 3
lists the scenarios that have been identified.
Table 3: UAT Scenarios
|
|
To
be completed by QA staff
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
·
Scenarios
are developed to enable users to navigate through the application to test the
normal and abnormal usage across the various business process flows.
Test scenarios guide the entry of data into the test database, just as a
user would enter data into eWiSACWIS in day-to-day use. The scenario defines
the case data to be entered by the tester.
At the same time, the successful execution of part of a scenario may
depend on the existence of test data already resident in the test
database. For example, in order to test
the search functionality in a scenario that matches an intake to an existing
case, certain historical data must be resident in the test database to provide
the tester with a matching case. The
scenario will guide the user in searching for that data through system
processes. The existing data referenced
by the scenario will reside in the test database.
Many resources will be executing test scenarios as part of the acceptance
test. Resources from BPP, BFS, several
counties and vendor representatives will be available on a limited, as-needed,
basis for acceptance testing. Many
testers are coordinating their acceptance-testing role with their ‘regular’
jobs and will need advance notice of when they will be scheduled for testing
activities.
The Quality Assurance Team will create an initial acceptance-testing
schedule, which will reflect when resources will be needed for testing. The Program team will be responsible for
communicating the testing schedule to testers outside of the project team. The schedule may need to be adjusted to
accommodate individual availability.
Preparation for
Acceptance test will include the following activities:
·
Preparation of the test environment as defined
in section 4 including hardware, systems software, database(s) and the creation
of test data
·
Training of acceptance testers
·
Installation of necessary tools and software in
test workstations and on the network
·
Any necessary clarifications or modifications to
the acceptance test schedule or testing procedure
In order to provide additional support to those participants
testing in county facilities, the test lab includes a telephone and voice mail
system that will be monitored by QA and Program staff. These staff will check voice mail throughout
the day and answer incoming calls.
Anyone in need of assistance during UAT may contact the Help Line at
608.267.3292.
The acceptance testing process is
designed to assure that the eWiSACWIS application complies with requirements
and to discover and document any incidents. Following the test schedule,
testers will navigate the application windows and request batch processes to ensure that they function
according to eWiSACWIS requirements.
Using business scenarios, they will enter data, select options and open
and close windows following the instructions on the eWiSACWIS System Test
scenarios. Testers will test a single
path through selected functionality, following a scenario to ensure that the
most common sequence of user actions is supported.
After the scenario has been
thoroughly tested, as time permits, project team members may conduct additional
tests of diverse paths to ensure that the application supports legitimate
alternative paths or activates the required roadblocks (error messages,
security constraints). Also, the end-users are encouraged to bring actual work to
further test the application.
Testers will follow the
established schedule for each scenario’s execution. State project team members will be available
to provide assistance to other testers.
There will be at least one eWiSACWIS
representative drawn from the QA and Program staff on-site at the test lab
during acceptance test execution. For
those counties testing in their own facilities, AMS will be available on-site
Tuesdays and Thursdays from 10:00 to 3:30.
In addition, test support staff will be responsible for monitoring the
eWiSACWIS UAT Help Line described in section 5.1.4.
These support staff will provide
assistance with test execution, help the tester with any scenario-related
questions and provide a level of assurance that testers are executing scenarios
and generating verification documentation as expected.
As part of the acceptance testing
process, verification points will be inserted into the test scenarios. These verification activities will consist of
screen prints, reports and on-line data verification. Testers will be asked to check off that they
have completed each scenario and will also be asked to generate documents at
specific points.
Testers will be asked to complete the eWiSACWIS test
activity tracking form to verify that they have completed the scenarios. Testers will also be given an opportunity to
incorporate any comments onto the tracking form. End-user testers will also be asked for their
feedback and comments on a short survey to be completed at the end of their
testing session. The QA Team will be responsible for maintaining the library of
test activity tracking forms and for monitoring the overall progress of the
test effort.
QA Team members will work with
testers to assess any unexpected results and to determine whether the incident
should be reported in the incident tracking database. Due to the number and
varied background of the acceptance testers, it is not recommended that each
tester record his/her own incidents. QA
will be the arbiter of whether or not something should be entered into the
incident tracking database, and will enter incidents in a consistent fashion.
The documentation maintained during testing provides a basis for consistent and
continuous monitoring of the test process and status, and for the development
of test reports at the conclusion of test activities.
When reported defects have been
fixed, the test scenario that originally surfaced the defect will be re-tested
to verify that the fix has corrected the original defect. Additional tests will also be performed to
assure that the fix has not introduced additional defects. Retests will be performed by Program and QA
staff.
A test activity tracking form will be developed to track
daily tester activity. Each day, testers
will complete the form, reporting their progress and activity for the day. PPS staff will collate and communicate UAT
user comments within the team and to end users.
As each piece of a scenario is
tested or retested, the assigned tester will document the execution of that
scenario component on the activity tracking report. The tester will indicate if an incident is
encountered in the execution of a scenario.
Incidents will also be documented in detail through the formal
incident-tracking database, which provides information for diagnosis and
prioritization of the incident for resolution.
All activity tracking reports will be collected daily and maintained
throughout the test period as a means of monitoring test results. A sample
activity tracking form is included as Exhibit 1.
The successful completion of all scenario components
(including initial test and required retests) as documented by individual
testers and verified by the QA Team provide the basis for a scenario’s release
from acceptance test. All scenarios must
be successfully executed with the final version of the application prior to
release from acceptance test.
Please refer to Exhibit 2 for an overview of the test
execution process.
Figure 2 Test Execution Process
Test status and incident summary reports will be
developed for daily and weekly status reporting. Status reports will be provided to both State
and AMS project leaders. Initially,
daily incident review meetings will be held with State and AMS staff. The frequency of these meetings may decrease
depending on the volume of incidents.
For purposes of the acceptance
test, a defect is defined as a reproducible failure of an application software
deliverable to perform materially in accordance with its design documentation.
The acceptance test will be
considered complete when the following criteria have been met:
·
Successful completion of all acceptance test
scenarios.
·
Successful resolution of all material defects
that have been identified during acceptance testing.
·
Sign-off of Acceptance tests results by the
State Technical Project Manager. The QA
and Program management teams will provide recommendations related to the
approval of the acceptance test.
The QA team will develop a
preliminary acceptance test report when user testing is complete summarizing
test activities and any outstanding defects or issues. It is anticipated that there will be
outstanding defects, which will be resolved and retested after user involvement
in UAT but prior to implementation.
A final acceptance report will be
developed prior to implementation which formally summarizes test activities and
presents the overall results of the testing effort. The final report of acceptance test will
include:
·
An overview of test processes and activities
·
Documentation of the incidents encountered and
resolved
·
A summary of final test results
·
Recommendation(s) for approval of the acceptance
test
Both the preliminary and final
acceptance test reports will be delivered to the State Technical Project
Manager for review and consideration.
6.0 Test
Schedule
May 19-23 User preparation
May 27-30 Remote testers in test environment
No
onsite support
eUAT
Help Line staffed by QA
June 2-20 AMS Supports Ozaukee, Winnebago
·
June 3, 5, 10, 12
·
June 17, 19 tentative
Remote testing
eUAT
Help Line staffed by QA
Testers in Test
Lab at Fairchild
An Access database will be used to enter, track and
report on incidents. This tool provides
the ability to monitor the status of a single incident, or to prepare summary
reports regarding incidents, such as the number of resolved incidents awaiting
retest. Incident tracking reports will
be developed for periodic review by interested parties, and will be used during
incident review sessions.
The incident database will provide a defect repository. For purposes of acceptance testing, the
incident database will represent the master defect list. However, not all incidents will be classified
as application defects. Incidents will
be logged as they are encountered during acceptance testing. The incident database will be available to
AMS and AMS will be included in regular incident review sessions.
Recorded incidents will be
reviewed on a regular basis. The
frequency of these meetings will be dictated by the frequency with which
incidents are detected. It is
anticipated that daily meetings will be held at the onset of testing.
Participants in the incident
review meeting will include:
·
The Technical Manager
·
The Program Manager
·
A QA Representative
·
The AMS Development Manager
·
The AMS Test Manager
·
Other project team members as needed
During the incident review
meeting, the group will:
·
Determine appropriate disposition of the
incident
·
Determine whether an incident represents a
defect which must be corrected prior to completion of acceptance test
·
Rank incidents according to their severity and
priority
·
Assign incidents for correction or response
The use of an online tool also
allows for the development of daily and summary incident reports. The incident-tracking tool proposed for use
is a Microsoft Access database. A sample
window appears in Figure 3.
Figure 3
Please refer to figure 4 for an
overview of the incident management process.
Figure 4: Incident Management
Process
The acceptance test schedule is contingent upon the
successful completion of a variety of preceding project tasks. This plan has been developed under the
assumption that the project will continue to hit key milestones. Slippage in prerequisite tasks will have an
adverse affect on the ability to complete acceptance testing according to the
project schedule. Assumptions for successful
acceptance testing include:
·
The AMS team will complete prerequisite tasks as
scheduled on the current project plan:
Ø
System Testing will be completed no later than
May 23, as indicated on the current project plan.
Ø
A regression test, to validate the acceptance
test environment, will be conducted no later than May 9
Ø
Training will be provided to acceptance testers
prior to testing
Ø
Topic Papers complete by May 9
Ø
Web Based Training complete by June 6.
·
The application software that is delivered into
Acceptance testing will contain a low-incidence of defects. Acceptance testing will take longer and be
more problematic if there is a high incidence of problems within the
application software.
·
The User Acceptance Test environment will be
available no later than May 9.
Workstations and servers will be dedicated to acceptance testing for the
duration of the test.
·
AMS will be responsible for analyzing and
resolving applications problems found during User Acceptance testing.
·
AMS will provide functional and operational
support during Acceptance Testing.
Please refer to the attached Microsoft Project Timeline for
the proposed acceptance test schedule.
