Viewing Continuous Testing Results

Overview

The Continuous Testing Results page shows the results and statistics for all existing Continuous Tests. For long-running Tests, you can see the entire history of the Tests. To see the results, in the Login Enterprise sidebar menu > Results, navigate to Continuous testing.

Сharts

In the Сharts, you can access additional historical login information. This feature allows you to track data over extended periods of time and utilize a prediction model to forecast future performance accurately.

To choose the specific information you wish to view, click the corresponding radio button from the Measurements menu on the top left.

Exporting Data and Creating SLA Reports

You can export data as needed and create SLA (Service Level Agreement) reports to notify individuals or groups when thresholds have been exceeded.

For more information on creating and configuring SLA reports, see SLA Reports.

Examining Application Performance

To analyze application performance metrics such as application start times and virtual user workload functions, in the Continuous testing charts, select Applications from the Measurements menu on the top left. In Applications, select the specific application data you wish to examine.

Monitoring Application Performance

To view high-level data on applications that exceed defined performance thresholds within the environment, in the Login Enterprise sidebar menu, navigate to Results > Dashboard > Test dashboard, and review the Applications section.

Application Performance Metrics

When you click into an application with available data, you can quickly view detailed performance metrics and observe the duration of various functions performed within the workload.

EUX Data

You can calculate an End User Experience Score (EUX) used to determine the responsiveness of the digital workspace during a Continuous Test. Alongside Latency, Login Performance, and Application Performance, you can visualize the overall end-user experience over a day, week, or month.

To view the EUX Score, click the corresponding radio button.

You can surface the EUX timers that are aggregated to calculate the EUX Score. To view the EUX Score and related timers, click the "EUX METRICS" radio button. You can use the checkboxes to select or deselect the EUX timers, which will add or remove them from the chart.

For more information on the EUX score, see Calculating the EUX score and VSImax.

Performance Data

Clicking the Applications radio button gives you access to visualize the performance of your applications over time.

If you look at the image above, the left-hand menu lets you configure the chart for your specific requirements per continuous test and performance metric. Below you can see the headers that, when clicked, will quickly turn the line on or off so you can see the specific performance metrics you want to investigate. Hovering over each data point on the chart will reveal the exact value for that point in time.

To define a specific date, simply click the Calendar icon located above the chart. On the top right-hand side, you can select a resolution for your data, such as Hour, Day, Week, and Monthly views. You can then use the time-slider underneath the chart itself to choose the time of the day (per hour) you want to investigate further. After browsing different times, you can always get back the the current hour/day/week/month by selecting the Now button.

Downloading Charts as an Image

You can download charts as images for sharing or use in custom reports and presentations. Downloading a chart as an image applies to all charts: Load Test, Continuous Test, and Appliance Health.

To download a chart, follow these steps:

Locate the Download icon: The download icon is located at the top right corner of the chart.
Select image format: When you hover over the icon, three download options will appear: PNG, SVG, and JPG.
Download the image: The selected image will be saved to your Downloads folder, or the directory you've configured as your default Downloads location.

Prediction view

There is also an option to predict application performance. This will take a look at past performance measurements and calculate what your application might perform like next. Please note that this is only possible when there is at least 30 hours of application data available.

Events

The Events section includes events related to Continuous Test and system events.

To access Events:

In the Login Enterprise sidebar menu, navigate to Results > Continuous testing.
In Continuous testing results, select Events from the top menu.

Additionally, you can open each logged event to get more detailed information. For this, simply click on the event you're interested in:

Possible Events

accountCapacityExceeded
accountDisabled
appExecutionAbandoned
applicationFailure
applicationThresholdExceeded
connectionInitializationTimeout
customUserSessionEvent
dataRetentionFailed
dataRetentionFinished
dataRetentionStarted
databaseFailure
emailFailure
emailRequest
emailSent
engineLogs
enginePaused
engineResumed
euxExecutionFailure
euxInitializationFailure
latencyMeasurementFailed
latencyThresholdExceeded
launcherCapacityExceeded
launcherLogs
launcherOffline
licenseSessionLimit
loginFailure
loginTimeMeasurementFailed
loginTimeThresholdExceeded
remoteSessionDisconnected
screenshotFailure
scriptEvent
scriptScreenshot
sessionDiscoveryError
sessionFailure
sessionRequestEndedBeforeEngineBecameOnline
testRunCancelled
testRunFailed
testRunFinished
testRunStarted

SLA Reports

SLA reporting provides a concise and straightforward document that delivers key performance information. These reports are generated from historical data gathered by Login Enterprise during its operational processes, making them ideal for monitoring service-level agreements, understanding infrastructure performance over time, and facilitating operations reporting to management and service customers.

Benefits of SLA reporting

Using SLA reporting offers several advantages:

Automated history of Test Results: SLA reports provide a clear and automatically generated history of test scenario results, making it easy to review and analyze past performance.
Multiple email recipients: This feature allows the distribution of results to multiple stakeholders. For example, an IT manager can quickly review the overall health and ROI indicators through a PDF attachment, while Login Enterprise administrators can focus on specific details relevant to their responsibilities within the same email.
Simplified review process: SLA reports are designed to be easy to read, streamlining the process of reviewing results within a defined time frame. This simplifies the task compared to examining Continuous test results and charts directly in the UI.

For reference, an example of an SLA report is attached to this article.

Configuring SLA Reports

To create a new report, in the Login Enterprise sidebar menu, navigate to Results > Continuous Testing.
In the Continuous Testing results, select SLA Reports from the top menu.

In the SLA reports, click the green "+" in the upper-right corner to add a new SLA report (The Report info window will open).

In the Report info, provide the following information:

Name and description: Specify a name and description for your reporting configuration.
Test and time frame: Select the test and the time frame for which the data should be retrieved.
Email recipients: Optionally, specify the email address(es) to receive the reports when automatically generated. To enter multiple email addresses, separate them with commas.
Time zone: Specify a time zone (UTC).
Report logo: Customize the report with your company logo. Supported image formats are JPEG, JPG, PNG, and SVG, with a maximum file size of 1 MB.

Once you're done, click Next to add measurements and thresholds.

Email Notification Requirements

The Email Notification function requires that the email settings be configured. For more information, see Managing Notifications. If Email Notification is enabled but no email addresses are entered here, the global email settings will be inherited.

Adding Measurements and Thresholds

Adding a new threshold lets you specify the following for your SLA report:

Application: Select the application to be monitored.
Measurement: Choose the specific measurement of the application.
Time limit: Set the time limit for the measurement.

Any measurements that exceed this limit will be included in the SLA report.

Click Save to apply your SLA report configurations.

Generating SLA Reports

Reports will not be generated immediately unless you click Manually generate report now. If you have set the schedule to daily, you will receive your first report after a day.

To generate a manual report, in Available reports:

Click Manually generate report.

Select the desired date.
Click Generate.

To view the report, in Available reports, click on the report that you've just generated manually (The PDF report will start downloading immediately).

You can download a sample SLA report from here.

Scheduled Reports

Daily reports: Generated daily.
Weekly reports: Generated weekly.
Monthly reports: Generated monthly.

Daily, weekly, and monthly reports can be deleted as necessary.

The SLA report displays the report date, indicating the timeframe the report covers.

The Application Success value in the SLA Report represents the status of the entire workload script to complete successfully.

Exporting Data

You can export data from Continuous Tests in CSV format. This feature is particularly useful for integrating data from different sources.

Benefits of Using Raw Data Export

Comprehensive data analysis: Exporting raw data allows you to perform detailed analysis and compare metrics from different tests or sources.
Data integration: Easily combine and correlate data from various tests or external systems for more robust reporting.
Error tracking: Access detailed information about application execution and performance, which can help in troubleshooting issues.

Contents of the RawData.zip File

The RawData.zip file is named according to the following schema:

RawData <TEST_NAME> <DATE> <TIME>.zip

AppExecutions
- This file contains data on application script executions. Each entry represents a Virtual User starting an app and includes details on whether any errors occurred during the execution.
Measurements
- This file includes all measurements captured during the Test run. It covers application response times and other timers configured in the test settings.
UserSessionMetrics
- This file provides CPU and Memory utilization metrics for each user session in the Test run, expressed as percentages.
UserSessions
- This file contains information about the Virtual User accounts used during the Test run. It details the Launcher of origin, the target host, and the logon results.

Exporting Raw Data from Continuous Tests

Navigate to the Continuous Testing Results page.
Click the Export Data tab.

Click the green + icon to create a new export.

Specify the desired timeframe using the From date and To date fields.

Select the configuration for which you want to export data and click Export.

Click the folder icon to download the .zip file containing your data.

Exporting Data from Charts

Alternatively, you can export data from the Charts view:

From the Continuous Test Results page, click Charts.

Use the drop-down menu to select the configuration.

Click Export selected data to CSV under the chart window.

The RawData.zip file will be prepared and downloaded.

The data you downloaded will also be displayed in the Export Data tab for your reference.

Example Output

Below is an example of the RawData.zip file generated by the Raw data export:

EUX Raw Data

To enhance debuggability and facilitate root cause analysis, the EUX raw timer data can be essential. Login Enterprise stores the raw data by default, and the data retention can be adjusted based on your specific needs. This article provides instructions on how to retrieve the EUX raw data and how to change the data retention period.

Retrieving the EUX Raw Data

You can retrieve the EUX raw data via the Public API with the following request (the testRunId is a Continuous Test Id):

CODE

GET https://appliance.loginvsi.com/publicApi/v7-preview/test-runs/{testRunId}/eux-script-executions

The API will return all the EUX measurements for each user session. Below is an example of how the response is structured:

CODE

{
"items": [
{
"userSessionId": "6f838f90-b9ed-4e63-b04e-2291a50e34cb",
"startTime": "2025-01-06T15:20:55.288615Z",
"endTime": "2025-01-06T15:22:08.481351Z",
"successful": true,
"euxMeasurements": [
{
"timestamp": "2025-01-06T15:21:55.998094Z",
"timer": "appSpeedUserInput",
"duration": 635
},
{
"timestamp": "2025-01-06T15:21:32.025Z",
"timer": "appSpeed",
"duration": 5656
},
{
"timestamp": "2025-01-06T15:22:00.373194Z",
"timer": "cpuSpeed",
"duration": 210623
},
{
"timestamp": "2025-01-06T15:21:37.61784Z",
"timer": "diskAppDataLatency",
"duration": 4329
},
{
"timestamp": "2025-01-06T15:22:02.388249Z",
"timer": "diskAppData",
"duration": 16952
},
{
"timestamp": "2025-01-06T15:22:03.918992Z",
"timer": "diskMyDocsLatency",
"duration": 4424
},
{
"timestamp": "2025-01-06T15:22:04.419015Z",
"timer": "diskMyDocs",
"duration": 14851
},
{
"timestamp": "2025-01-06T15:21:42.172781Z",
"timer": "fastCompression",
"duration": 920
},
{
"timestamp": "2025-01-06T15:21:19.825964Z",
"timer": "highCompression",
"duration": 925
}
],

Cleaning the EUX Data (Data Retention)

You can configure the data retention period for EUX raw data, ranging from 1 to 9999 days. Customizing this setting is crucial to managing storage resources effectively. New installations and upgrades default to a 30-day retention period, but any customizations persist through subsequent upgrades.

To adjust the EUX raw data:

In the Login Enterprise sidebar menu, navigate to Other > System.
In the System, click General.
In Data retention, configure EUX raw data.

Additional Resources

For general guidelines on how to analyze Continuous Testing Session Metrics, refer to Viewing Session Metrics.