Figure 1: Workflow of an ADT

The ADT solution uses an AI/ML model to auto-categorize failed test scripts into predefined categories based on test execution and failure patterns. It helps identify failed execution root causes and suggests recommended actions based on knowledge from previous executions.

It consists of the following steps:

Step 1: Test script execution result collation

Irrespective of the test automation framework used — whether based on open-source tools like selenium/extent reporter or on commercial off-the-shelf (COTS) tools like UFT — the test results need to be gathered in a commonly accessible location for the ADT agent.

Step 2: Monitoring through ADT agent

The agent will continuously poll the reports and test execution result stack trace to gather all the results and move them to a database.

Step 3: Automated Defect Triaging (ADT)

The ADT solution picks up the data that the ADT agent sends to a database. After initial data preparation, the data is fed into a classifier model, which helps determine the reason for failure. The test failure is classified based on the result description and by pattern-matching with the historical data. This classification determines if the failure was due to a configuration failure, test infrastructure failure, or a product failure. If at any moment the classifier model cannot auto-classify the defect due to low confidence, the human in the loop (HITL) comes into the picture to assist with the workflow.

Step 4: Defect de-duplication

Once the cause of failure is determined, it passes on to the defect de-duplication, and the model tries to identify similar predicate defects that already exist within the defect management tool. If the classifier model cannot succeed because of low confidence, the human in the loop comes into the picture to assist with the workflow.

Step 5: Defect logging

Defect logging is the final step in the process, closing the loop. The solution should update the existing defect if a duplicate is found, or it should log a new defect within the defect management tool.

Key benefits:

The output of the ADT solution is extensive, as it improves the following processes:

Classification of test failures
Root cause analysis of the defects
Recommendation of resolution steps
Identification of duplicate defects
Creation of a summary of test defects, including drill-down details
The provision of alerts/notifications to subject matter experts (SMEs) regarding critical, high-severity defects; this enables quick, decisive action

The right-fit, popular-classification algorithms can be used for various classifier models. Moreover, a proof of concept (PoC) should be performed on existing data in order to determine the right algorithm. The ADT team should work with the quality assurance (QA) team to retrain the model regularly and triage the defects. The ADT solution helps reduce the test script failure triage efforts by a considerable margin.

In a sprint model with a high level of DevOps automation, there is constant pressure on the test engineers to perform continuous testing within the sprint, and to do so effectively. Bottlenecks within the manual defect triaging process consume SMEs’ bandwidth — often when they should be focused on testing. This effect is magnified when there are thousands of automated scripts. It would be wise for companies to invest in a solution that addresses these challenges.

Services

Industries

Insights

Careers

About