TSA 3rd Party Testing Program

Background TSA's Test & Evaluation (T&E) processes evaluate the effectiveness and suitability of technology solutions prior to their acquisition and deployment to the field. Currently, this phase of the capability lifecycle is lengthy and iterative, driving high costs to TSA and increasing time to market for new capability solutions. In FY '13, TSA spent just under 20% of its test execution budget on re-testing systems that were deemed unsuitable to meet operational demands because they experienced failures and required additional development. Discussion TSA will reduce the time of the formal T&E process through multiple initiatives and will employ innovative approaches to ensure new capabilities will be sufficiently tested early in the process and matured to a level of readiness for formal testing, thus reducing the amount of formal testing required and reducing the need for re-testing. TSA will: � Increase transparency into its testing process and plans; � Update and streamline requirements for vendor Qualification Data Packages; and � Adopt a third-party testing requirement. TSA believes these initiatives will discover issues earlier in the process rather than during formal testing, allowing for an expedited and successful transition into and through the formal T&E process, and potentially reducing T&E timelines and time to market. It is TSA's ultimate goal to reduce the time in formal testing for high priority technologies and systems by 50%. Policy TSA intends to significantly reduce the iterative and costly nature of its current test-fix-test T&E rubric. Effective July 1, 2014, systems encountering a significant failure during formal TSA Qualification Testing, with the exception of detection testing, will be temporarily suspended from further TSA testing until sufficient data is provided from a third-party test entity attesting the failed requirement(s) have been met. Significant failure TSA uses a widely accepted T&E best practice scoring system to assess all requirement failures in formal testing. Each failure is scored by a board of experts according to a system with seven failure levels, as defined in the attachment. Within this scoring system, any failure that is scored as a "1", "2", or in cases of numerous and clustered "3s", is determined to be a significant failure and continued testing is not possible until additional development to resolve the failure is accomplished. This methodology is currently used to determine readiness of a system to move from QT into OT, as well as in the scoring of results from OT. Therefore, this same methodology will be used to determine when this policy will take effect. Third-party testing Upon completion of development activities to correct significant failures, the OEM must provide for the release of the documented test results from a third-party clearly proving any failed requirement(s) have been resolved. This required testing can be performed by a third-party testing entity or performed by the OEM with third-party oversight witnessing the testing and documenting the results. TSA may also witness the testing, but not in lieu of the third-party requirement. Acceptance The decision to allow a system back into the formal T&E process (QT), and the degree of regression testing required to be done in QT, will be made by the OSC Program Manager, with concurrence by the TSA Operational Test Agent. This decision will be made on an individual basis, based on the following criteria: - Adequacy of the OEM/third-party test plan - Results of the testing - Statistical significance of the data - Validation by observation of testing by a TSA representative, if this occurred Based on the assessment of the testing effort and the results, the TSA may perform the QT in its entirety, conduct regression tests on areas that previously failed, or conduct a focused test to validate the results of the third-party tester.