Chapter 1: Introduction

From March 3^rd to March 11^th the eWiSACWIS team took the application to the AMS Center for Advanced Technology lab to execute performance testing. 36 Separate performance tests were run, with user loads from 50-270 concurrent users, and durations from 10 minutes to 15 hours. This document contains the performance testing results.

This document is organized as follows:

· Chapter 1 contains an introduction, stating the objectives of the performance test, summarizing the results, and listing the performance improvements made to the eWiSACWIS application.

· Chapter 2 contains a description of the test script.

· Chapter 3 contains performance test results of selected test runs.

1.1 Objectives

The objectives of the performance test activities were as follows:

Determine if the eWiSACWIS application will meet the requirements specified in the eWiSACWIS Production Environment Requirements document;
Determine eWiSACWIS response time and server behavior under normal usage conditions;
Determine eWiSACWIS response time and server behavior under extreme load conditions;
Determine eWiSACWIS behavior under continuous load over a period of 12 hours;
Resolve any Java performance related issues;
Resolve any WebSphere configuration issues that arise; and
Resolve any database issues.

1.2 Configuration Information

It should be noted that the test lab did not have an exact configuration match of the eWiSACWIS production environment. In addition to hardware differences, the test lab also did not have the eWiSACWIS production authentication component, Novell’s iChain. Below is a list of the hardware and software components that were used.

Hardware

Ø Database Server - Dell 1550(Windows 2000 Server, 2 1.2 GHz CPU, 2GB RAM, 3x18GB drives)

Ø Web/Application Server – Sun Sunfire 4800(Solaris, 4 CPU, 4GB RAM, 1 JVM)

Software

Ø Oracle 8.1.7

Ø WebSphere Advanced Application Server 4.04

Tools Used

Ø Segue Silk Performer

Ø IBM HTTP Server 1.3.19

Ø Quest Software Toad

Database

Ø A production volume data set was used

1.3 Test Results

The primary objective of the performance test was to determine if the application could support targeted load based on WiSACWIS system utilization. The projections from the eWiSACWIS Production Environment Requirements document are as follows:

The current number of total users is 2200. The projected future total is 4000.
The estimated maximum number of future concurrent users is 96.
The current database transaction load is 84,500 per hour, or 24 per second.
The projected database transaction load with 4000 users is 169,000 per hour, or 48 per second.

Therefore, eWiSACWIS must be able to process 169,000 database transactions per hour, with a concurrent user load of 96 users. Note that the determination of the number of database transactions, as stated in the eWiSACWIS Production Environment Requirements document, is to take the number of Oracle “User Calls” and divide them by 3.

On all 36 runs that were executed, the eWiSACWIS system exceeded the stated performance objectives. The following table contains the results of the final “normal usage” performance test that was done on March 8^th.

	PerformanceTarget	Performance Achieved
Concurrent Users	96	100
Database Transactions per Second	48	433
Database Transactions per Hour	169,000	1,558,800

1.3.1 Types of Tests

As stated above, 36 separate test runs were performed. There were 3 types of performance tests done; normal usage tests, extreme load usage tests, and duration tests.

Normal usage tests were short in duration, lasting from 10 minutes to one hour. The objective of these tests was to determine how the system performs under normal conditions. The first test that was executed was a normal usage test. This first run established baseline performance metrics. After this and subsequent runs, changes were applied to the eWiSACWIS application, the eWiSACWIS database, and to the WebSphere application server. Tests were then re-run to determine the impact of the modifications made.

It is also important to test the server under extreme load conditions to make sure no one component of the architecture failed. This was achieved by increasing the load to 150 and then to 270 concurrent users. It was determined that increasing the load past 100 concurrent users actually starts to reduce the number of transactions processed per second as the server starts spending more time on managing connections versus processing requests. The solution to this situation would be to load balance by adding another JVM. We were limited to the use of 1 JVM, so we were not able to apply this, but adding another JVM should nearly double the number of concurrent users that the application server can effectively handle.

The system was also tested for stability by running a 12-hour load simulation with 100 concurrent users. The results showed that the system suffered no ill effects and completed the test without incident. The system also released its resources correctly and had no impact on performance afterwards.

1.4 Application Performance Tuning

By changing JVM settings, database settings, and tweaking application components, we were able to effectively double the number of transactions processed. To summarize:

· With tuning, database transactions processed per hour improved from 601,200 to 1,558,800.

· With tuning, average eWiSACWIS transaction response time (such as Update Case, Search Person) was trimmed from 3.93 seconds to 1.94 seconds, an improvement of over 100%.

· The optimal number of concurrent users for 1 JVM is 100.

Below is a list of the changes that were made that provided the performance improvements. Please note that the changes that provided the most significant benefit are the Database Changes.

1.4.1 WebSphere Configuration Changes

Most of the WebSphere changes made were not necessarily for performance reasons but for load reasons.

Ø Increased JVM Heap Memory from 512MB to 1.2GB. At 512MB the JVM did not have the proper space to execute a high number of concurrent transactions. Increasing the JVM past 1.2GB does not increase transaction throughput.

Ø Set the minimum number of database connection pool to 20.

Ø Set the maximum number of connections to 200. The pool should never get close to this number but it allows some cushion if the system experiences heavy load. Connections not in use will always revert back to Oracle.

Ø Increased web container maximum thread size from 50 to 500. The minimum thread size is still 25 and is set to grow automatically. Under extreme the web server turned down requests.

Ø Increased maximum keep alive from 25 to 450. This number should be 90% of the maximum web containers thread number.

1.4.2 eWiSACWIS Application Changes

Most application changes dealt with optimizing SQL and using proper constructs within Java code.

Ø Added SQL hint to PopulateUserProfile function.

Ø Modified Assessment to use used SQL from the static initializer. Since the SQL is static for selects, it now loads once at runtime and is re-used by all other objects. Previously it was being loaded every time the class was invoked.

Ø Modified the search programs not to throw exceptions when no search results are found. Using try/catch is very expensive in Java and should only be used for actual exceptions, not business logic.

Ø Tweaked assessment to use ArrayLists instead of Vectors.

Ø Streamlined the system connection manager further.

Ø Modified SQL for expand provider on the outliner.

Ø Modified Assessment not to use Enumerations when populating data.

1.4.3 Database Changes

The most significant changes that affected database performance involved tuning application SQL statements ( i.e. hints, joins, and SQL structures ), creating indexes where needed, and optimally tuning oracle initialization parameters to support the hardware configuration and make the most efficient use of physical RAM available. The indexes that were created will also benefit performance for the current production C/S WiSACWIS system.

Ø CREATE INDEX INVESTIGATION_1 ON INVESTIGATION (ID_INVS);

Ø CREATE INDEX ADDRESS_3 ON ADDRESS (CD_GRP, ID_ADRSS_GRP, CD_GRP_TYPE

Ø CREATE INDEX CPS_REPORT_4 ON CPS_REPORT (ID_INVS);

Ø CREATE INDEX INVESTIGATION_CNTC_2 ON INVESTIGATION_CNTC (ID_INVS, ID_CAN_EVNT);

Ø CREATE INDEX ASSIGN_CATEGORY_1 ON ASSIGN_CATEGORY (CD_ASGN_CTGRY, CD_ASGN_TYPE, CD_ASGN_ROLE, CD_ASGN_RSPNS);

Ø CREATE INDEX JOB_CLASS_1 ON JOB_CLASS (CD_JOB_CLS);

Ø CREATE INDEX LOCATION_1 ON LOCATION (CD_LCTN, CD_OFC_DIV, CD_DEPT_UNIT);

Ø CREATE INDEX PROVIDER_SERVICE_1 ON PROVIDER_SERVICE (ID_PRVD_ORG, CD_STAT);

Chapter 2: Test Description

eWiSACWIS is a very large system with over one thousand (1,000) distinct functional transactions. It is critical that performance testing reflect the real world usage of the system, and that the test strategy includes those transactions that are key to the system and those transactions that are of high volume.

Before writing the performance test scripts, the actual usage of the WiSACWIS system was analyzed. Key and high volume transactions were identified, and they are included in the test script.

The following is the list of scripts used for testing and the user actions in each script. During performance testing, all scripts were run in stress mode. This mode does not account for user lag time between submits.

2.1 Test Script

Script 1 – Expand Outliner

Logon
Expand Case
Open Case Maintenance
Expand Case Details
Expand Case Assignment
Expand Case Assessment
Expand Provider
Open Provider Maintenance
Expand Provider Details
Expand Provider Basics
Expand Worker
Expand Worker Details
Expand Approval
Expand Approval Details
Expand Intake
Logout

Script 2 – Search

Logon
Case Sounds Like Search using First and Last Name
Person Sounds Like Search using First and Last Name
Provider Sounds Like Search using Last Name
Worker Sounds Like Search using First and Last Name
Case Search using First and Last Name
Person Search using First and Last Name
Provider Search using Last Name
Worker Search using First and Last Name
Case Search using Case Id
Person Search using Person Id
Provider Search using Provider Id
Logout

Script 3 – Create Case

Logon
Create PS Report
Screen In and convert PS Report to a Case
Create Assessment
Open Mandated Reporter Document
Create IA Primary Caregiver
Open IA Primary Caregiver document
Create Legal Action
Create and Approve Court Report
Create and Approve Legal Document (Referral TPR)
Create Safety Assessment
Create Permanency Plan
Create Secondary Assignment
Logout

Individual Scripts (4-10)

Create Provider

Create Case Notes
Create Court Report
Create Assignment
Create Legal Document
Create Permanency Plan
Create Safety Assessment

On a percentage basis 50% of the users were allocated to Expand Outliner and Search scripts, remaining users were divided up among the rest of the scripts.

2.2 Test Results Description

The following charts and graphs are used to document the performance test results.

2.2.1 User Load

This chart shows the overall number of active virtual users. A virtual user is considered as active if the user has started and is currently in one of the following states: executing, wait DB, document downloading, and think time.

2.2.2 Throughput

The amount of data sent to and received from the server; this includes header and body content information, all TCP/IP-related traffic (HTTP, native TCP/IP, IIOP, POP3, SMTP, FTP, LDAP and WAP), and secures traffic over SSL/TLS. This measurement does not include data overhead caused by SSL/TLS encryption and WTLS encryption in case of WAP.

The Blue line running at the bottom of the chart represents the request data sent.

The Aqua line running at the top of the chart represents the Response data received.

2.2.3 HTTP Hits

The number of HTTP requests that arrive at the Web server per second.

2.2.4 End User Average Response Time

The time it takes a virtual user to download a Web page from the server, in seconds.

It should be noted that the average response time that users of eWiSACWIS experience will vary based on infrastructure and network performance. The times stated in this document are “best case”, as they do not include propagation delays that will occur when data is transferred to the many outlying areas in the State of Wisconsin. The propagation delay will especially affect those areas serviced by low-speed data communication lines.

Another factor that will affect end user response time is the services that will be performed by the authentication/proxy service, Novell’s iChain. As stated previously, AMSCAT did not have an iChain authentication server available.

2.2.5 Application Transaction Statistics

eWiSACWIS has its own internal monitoring capability that tracks the amount of time each transaction spends on the application and database server. This combined time is recorded in the TRAN_STATS table.

The TRAN_STATS data will capture the time each application transaction spends in the application and database server. Note, an application transaction does NOT equate to the transaction mentioned in the eWiSACWIS Production Environment Requirements document. The Production Environment Requirements transaction consists of a single SQL statement. An application transaction measured in TRAN_STATS is a user interaction, and can consist of multiple SQL statements in order to complete a logical unit of work.

Below is a sample report that will be generated.

eWiSACWIS Transaction Performance - Slowest 30 - 03/04/2003

Rank	AVG	Count	Class	Path	Action
1	36.600	10	com.ams.sacwis.sm06a.SM06a_AssessmentAction	/SM06A_Assessment	EDIT
2	29.984	10	com.ams.sacwis.sm06a.SM06a_AssessmentAction	/SM06A_Assessment	CREATE_ASSESSMENT
3	18.985	627	com.ams.sacwis.sm04b.SM04B_OutlinerAction	/SM04B_Outliner	EXPANDPROVIDER
4	14.614	398	com.ams.sacwis.cm10.CM10_UtilitySearchAction	/CM10_UtilitySearch	SEARCHPRVD
5	14.524	104	com.ams.sacwis.sm06b.SM06B_SafetyAssessmentAction	/SM06B_SafetyAssessment	CREATE
6	11.607	48	com.ams.sacwis.pm07.PM07_HomeInquiryAction	/PM07_HomeInquiry	VIEW
7	9.973	1154	com.ams.sacwis.security.SecurityAction	/LOGON	LOGON
8	9.831	1	com.ams.sacwis.cm09.CM09_SecurityProfileAction	/CM09_SecurityProfile	UPDATE
9	9.562	715	com.ams.sacwis.pm02a.PM02A_HomeProviderAction	/PM02A_HomeProvider	UPDATE
10	9.304	20	com.ams.sacwis.sm06a.SM06a_AssessmentAction	/SM06A_Assessment	INSERT
11	8.488	593	com.ams.sacwis.sm04b.SM04B_OutlinerAction	/SM04B_Outliner	EXPANDCASE
12	8.261	11	com.ams.sacwis.sm01a.SM01A_ProtectiveServicesReportAction	/SM01A_ProtectiveServicesReport	VIEW
13	7.586	85	com.ams.sacwis.sm08.SM08_PermanencyPlanAction	/SM08_PermanencyPlan	INSERT
14	7.187	83	com.ams.sacwis.sm08.SM08_PermanencyPlanAction	/SM08_PermanencyPlan	OpenDocument100800
15	7.128	102	com.ams.sacwis.sm06b.SM06B_SafetyAssessmentAction	/SM06B_SafetyAssessment	VIEW
16	6.889	87	com.ams.sacwis.sm09.SM09_LegalDocumentationAction	/SM09_LegalDocumentation	VIEW
17	6.461	10	com.ams.sacwis.sm06a.SM06A_InitialAssessmentPrimaryAction	/SM06A_InitialAssessmentPrimary	EDIT
18	6.444	102	com.ams.sacwis.sm09.SM09_LegalDocumentationAction	/SM09_LegalDocumentation	UPDATE
19	6.106	1	com.ams.sacwis.sm04b.SM04B_OutlinerAction	/SM04B_Outliner	EXPANDTICKLER
20	4.733	381	com.ams.sacwis.sm09.SM09_LegalDocumentationAction	/SM09_LegalDocumentation	INSERT
21	4.470	1	com.ams.sacwis.cm18.CM18_MaintainWorkerInformationAction	/CM18_MaintainWorkerInformation	OPEN
22	4.265	271	com.ams.sacwis.cm19.CM19_ApprovalHistoryAction	/CM19_ApprovalHistory	EDIT
23	4.261	180	com.ams.sacwis.sm09.SM09_LegalDocumentationAction	/SM09_LegalDocumentation	OpenDocument
24	4.166	11	com.ams.sacwis.sm09.SM09_LegalActionAction	/SM09_LegalAction	UPDATE

Chapter 3: Test Results

3.1 First Test: 90 Concurrent Users, 30 Minutes – March 4, 2003

The purpose of Run 1 was to execute the scripts with 90 concurrent users without making any modifications to the system. This was done to establish a baseline for future performance tests. Note: an application transaction is a round trip through the application and can comprise numerous Database transactions.

Ø 90 Concurrent Users

Ø 167 Oracle Database Transactions per second.

Ø 601,200 Oracle Database Transactions per hour.

Ø 15,248 eWiSACWIS Application Transactions (Update Case, Search Person)

Ø 4.5 seconds - Average Time per Application Transaction

3.1.1 User Load

(90 concurrent users)

3.1.2 Throughput

Throughput / Concurrency

Throughput [kB]: 1,360,949

Average throughput [kB]/sec: 751.91

3.1.3 HTTP Hits

The number of HTTP requests that arrive at the Web server.

Number of hits: 91,203

Average number of hits/sec: 50.39

3.1.4 End User Average Response Time

Average page time[s]: 3.93

3.1.5 Application Transaction Statistics

eWiSACWIS Transaction Performance - Slowest 30 - 03/04/2003

Rank	AVG	Count	Class	Path	Action
1	36.600	10	com.ams.sacwis.sm06a.SM06a_AssessmentAction	/SM06A_Assessment	EDIT
2	29.984	10	com.ams.sacwis.sm06a.SM06a_AssessmentAction	/SM06A_Assessment	CREATE_ASSESSMENT
3	18.985	627	com.ams.sacwis.sm04b.SM04B_OutlinerAction	/SM04B_Outliner	EXPANDPROVIDER
4	14.614	398	com.ams.sacwis.cm10.CM10_UtilitySearchAction	/CM10_UtilitySearch	SEARCHPRVD
5	14.524	104	com.ams.sacwis.sm06b.SM06B_SafetyAssessmentAction	/SM06B_SafetyAssessment	CREATE
6	11.607	48	com.ams.sacwis.pm07.PM07_HomeInquiryAction	/PM07_HomeInquiry	VIEW
7	9.973	1154	com.ams.sacwis.security.SecurityAction	/LOGON	LOGON
8	9.831	1	com.ams.sacwis.cm09.CM09_SecurityProfileAction	/CM09_SecurityProfile	UPDATE
9	9.562	715	com.ams.sacwis.pm02a.PM02A_HomeProviderAction	/PM02A_HomeProvider	UPDATE
10	9.304	20	com.ams.sacwis.sm06a.SM06a_AssessmentAction	/SM06A_Assessment	INSERT
11	8.488	593	com.ams.sacwis.sm04b.SM04B_OutlinerAction	/SM04B_Outliner	EXPANDCASE
12	8.261	11	com.ams.sacwis.sm01a.SM01A_ProtectiveServicesReportAction	/SM01A_ProtectiveServicesReport	VIEW
13	7.586	85	com.ams.sacwis.sm08.SM08_PermanencyPlanAction	/SM08_PermanencyPlan	INSERT
14	7.187	83	com.ams.sacwis.sm08.SM08_PermanencyPlanAction	/SM08_PermanencyPlan	OpenDocument100800
15	7.128	102	com.ams.sacwis.sm06b.SM06B_SafetyAssessmentAction	/SM06B_SafetyAssessment	VIEW
16	6.889	87	com.ams.sacwis.sm09.SM09_LegalDocumentationAction	/SM09_LegalDocumentation	VIEW
17	6.461	10	com.ams.sacwis.sm06a.SM06A_InitialAssessmentPrimaryAction	/SM06A_InitialAssessmentPrimary	EDIT
18	6.444	102	com.ams.sacwis.sm09.SM09_LegalDocumentationAction	/SM09_LegalDocumentation	UPDATE
19	6.106	1	com.ams.sacwis.sm04b.SM04B_OutlinerAction	/SM04B_Outliner	EXPANDTICKLER
20	4.733	381	com.ams.sacwis.sm09.SM09_LegalDocumentationAction	/SM09_LegalDocumentation	INSERT
21	4.470	1	com.ams.sacwis.cm18.CM18_MaintainWorkerInformationAction	/CM18_MaintainWorkerInformation	OPEN
22	4.265	271	com.ams.sacwis.cm19.CM19_ApprovalHistoryAction	/CM19_ApprovalHistory	EDIT
23	4.261	180	com.ams.sacwis.sm09.SM09_LegalDocumentationAction	/SM09_LegalDocumentation	OpenDocument
24	4.166	11	com.ams.sacwis.sm09.SM09_LegalActionAction	/SM09_LegalAction	UPDATE
25	3.996	90	com.ams.sacwis.sm09.SM09_CourtReportSelectionAction	/SM09_CourtReportSelection	CREATE
26	3.674	49	com.ams.sacwis.pm07.PM07_HomeInquiryAction	/PM07_IntakeMembersData	UPDATE
27	3.231	30	com.ams.sacwis.sm06a.SM06A_AssessmentNoticeAction	/SM06A_AssessmentNotice	EDIT
28	3.163	11	com.ams.sacwis.sm04a.SM04A_SearchCaseAction	/SM04A_SearchCase	EDIT
29	2.841	397	com.ams.sacwis.cm10.CM10_UtilitySearchAction	/CM10_UtilitySearch	SEARCHCASE
30	2.745	47	com.ams.sacwis.sm01a.SM01A_ProtectiveServicesReportAction	/SM01A_ProtectiveServicesReport	INSERT

3.2 Post Tuning Test: 100 Concurrent Users, 30 Minutes – March 8, 2003

The last “normal usage” run was conducted after all performance improvements had been applied. Significant improvement was made in both transaction volume and time per transaction.

Ø 100 Concurrent Users

Ø 433 Oracle Database Transactions per second.

Ø 1,558,800 Oracle Database Transactions per hour.

Ø 38,180 eWiSACWIS Application Transactions (Update Case, Search Person)

Ø 1.8 seconds - Average Time per Application Transaction

Some selected topic improvements:

Ø Edit Assessment was reduced from 36.6 seconds to 14.8 seconds, a 59.5% decrease

Ø The Logon process was reduced from 10 seconds to 4.6 seconds, a 54% decrease

Ø Expand provider was reduced from 19 seconds to 4.4 seconds, a 77% decrease

Ø Search case was reduced from 2.8 seconds to 2.3 seconds, a 17.9% decrease

3.2.1 User Load

(100 concurrent users)

3.2.2 Throughput

Throughput / Concurrency

Throughput[kB]: 2,849,106

Average throughput[kB]/sec: 1,574.09

3.2.3 HTTP Hits

The number of HTTP requests that arrive at the Web server.

Number of hits: 213,095

Average number of hits/sec: 117.73

3.2.4 End User Response Times

Average page time[s]: 1.94

3.2.5 Application Transaction Statistics

eWiSACWIS Transaction Performance - Slowest 30 - 03/07/2003

Rank	AVG	Count	Class	Path	Action
1	14.831	20	com.ams.sacwis.sm06a.SM06a_AssessmentAction	/SM06A_Assessment	EDIT
2	13.228	22	com.ams.sacwis.sm06a.SM06a_AssessmentAction	/SM06A_Assessment	CREATE_ASSESSMENT
3	7.113	758	com.ams.sacwis.cm10.CM10_UtilitySearchAction	/CM10_UtilitySearch	SEARCHPRVD
4	6.931	1	com.ams.sacwis.sm06a.SM06A_InitialAssessmentPrimaryAction	/SM06A_InitialAssessmentPrimary	OpenDocument100029
5	6.662	212	com.ams.sacwis.sm06b.SM06B_SafetyAssessmentAction	/SM06B_SafetyAssessment	CREATE
6	6.434	102	com.ams.sacwis.pm07.PM07_HomeInquiryAction	/PM07_HomeInquiry	VIEW
7	4.652	2768	com.ams.sacwis.security.SecurityAction	/LOGON	LOGON
8	4.599	175	com.ams.sacwis.sm09.SM09_LegalDocumentationAction	/SM09_LegalDocumentation	VIEW
9	4.439	1320	com.ams.sacwis.sm04b.SM04B_OutlinerAction	/SM04B_Outliner	EXPANDPROVIDER
10	4.258	207	com.ams.sacwis.sm06b.SM06B_SafetyAssessmentAction	/SM06B_SafetyAssessment	VIEW
11	4.245	1494	com.ams.sacwis.pm02a.PM02A_HomeProviderAction	/PM02A_HomeProvider	UPDATE
12	4.215	44	com.ams.sacwis.sm06a.SM06a_AssessmentAction	/SM06A_Assessment	INSERT
13	4.053	22	com.ams.sacwis.sm01a.SM01A_ProtectiveServicesReportAction	/SM01A_ProtectiveServicesReport	VIEW
14	3.931	201	com.ams.sacwis.sm09.SM09_LegalDocumentationAction	/SM09_LegalDocumentation	UPDATE
15	3.722	20	com.ams.sacwis.sm06a.SM06A_InitialAssessmentPrimaryAction	/SM06A_InitialAssessmentPrimary	EDIT
16	3.598	166	com.ams.sacwis.sm08.SM08_PermanencyPlanAction	/SM08_PermanencyPlan	INSERT
17	3.567	1356	com.ams.sacwis.sm04b.SM04B_OutlinerAction	/SM04B_Outliner	EXPANDCASE
18	3.411	166	com.ams.sacwis.sm08.SM08_PermanencyPlanAction	/SM08_PermanencyPlan	OpenDocument100800
19	2.983	322	com.ams.sacwis.cm06.CM06_CaseNoteAction	/CM06_CaseNote	UPDATE
20	2.281	754	com.ams.sacwis.cm10.CM10_UtilitySearchAction	/CM10_UtilitySearch	SEARCHCASE
21	2.151	770	com.ams.sacwis.sm09.SM09_LegalDocumentationAction	/SM09_LegalDocumentation	INSERT
22	2.011	371	com.ams.sacwis.cm02.CM02_CreateWorkerAssignmentAction	/CM02_CreateWorkerAssignment	UPDATE
23	1.916	364	com.ams.sacwis.sm09.SM09_LegalDocumentationAction	/SM09_LegalDocumentation	OpenDocument
24	1.846	323	com.ams.sacwis.cm06.CM06_CaseNoteAction	/CM06_CaseNote	INSERT
25	1.772	20	com.ams.sacwis.sm09.SM09_LegalActionAction	/SM09_LegalAction	UPDATE
26	1.734	102	com.ams.sacwis.pm07.PM07_HomeInquiryAction	/PM07_IntakeMembersData	UPDATE
27	1.664	66	com.ams.sacwis.sm06a.SM06A_AssessmentNoticeAction	/SM06A_AssessmentNotice	EDIT
28	1.657	550	com.ams.sacwis.cm19.CM19_ApprovalHistoryAction	/CM19_ApprovalHistory	EDIT
29	1.609	89	com.ams.sacwis.sm01a.SM01A_ProtectiveServicesReportAction	/SM01A_ProtectiveServicesReport	INSERT
30	1.323	22	com.ams.sacwis.sm04a.SM04A_CaseAction	/SM04A_Case	INSERT

3.3 Final Endurance Test: 12 Hour, 100 Concurrent Users – March 11, 2003

The system needs to perform under load for long periods of time and suffer no down time. Also, it was important to monitor the system for memory leaks and performance degradation over time. You will notice that the average time per transaction slipped a little. This is due to the increasing data for each of the 100 users over a 12-hour period. In the real application, each individual user would never have this much data assigned to them.

The system was able to handle this load without any impact to the database or the application server. After the test was completed, the system was able to process requests in a normal fashion. No memory leaks or locked connections were ever observed.