Regression Intelligence API

Table of Contents

What is a Regression?

In the most general sense, a regression is any specific thing that makes something worse. It has opposite word-roots to the term "progress". A software regression is usually the reappearance of a bug that had previously been solved, and preventing regressions is a major motivation for writing test suites. HeadSpin's Regression Intelligence feature focuses on both hard-to-detect integration issues and on performance regressions, which is when the quantitative measurements of an app get worse. Performance regressions might include slow-downs in loading times or any other metric associated with how "snappy" an app feels when a user engages with an app.

Unlike software features, that tend to get more complete as the software development lifecycle continues, performance tends to degrade as more and more features are added because more and more computing resources are being used. If an app's performance gets bad enough, your application can stop working entirely, but more often it just gets unacceptably slow.

Discovering and taking action on regressions is a vital part of the software development lifecycle. Because regressions can come from a wide variety of sources, from network issues to code changes, pinpointing root causes can be frustrating and error-prone. HeadSpin offers detailed metrics to help you track and debug regressions as soon as they occur. The API for accessing those metrics and alerting based on dramatic changes in performance is documented below.

Overview: Available API Routes

Common Parameters

Selecting a User Flow

• <code class="dcode">"user_flow_name"</code>: The name of the User Flow to select sessions from.

Regression analytics are only supported for sessions associated with the same User Flow. See the Performance Monitoring UI and Performance Monitoring API documentation for details. For sessions to be considered for regression analysis, the sessions must be marked as "passed" in the User Flow. This indicates that the individual session completed without issues and validates that the data from that session is useful to include in the regression analysis and other aggregate data views.

Session selection parameters

The HeadSpin Regression API supports two methods for selecting sets of sessions in set A (the reference or control) and set B (the target or treatment) for comparison: session tags and HeadSpin Selector strings.

● <code class="dcode">"a"</a>: Session tags or HeadSpin Selector string used to select the reference or control set of sessions.

• Example tags: <code class="dcode">[{"build": "0.1.0"}, {"day_of_week": "Mon"}]</code>
• Example selector: <code class="dcode">"build: prev_build(1) day_of_week: Mon"</code>

● <code class="dcode">"b"</code>: Session tags or HeadSpin Selector string used to select the target or treatment set of sessions for comparison.

• Example tags: <code class="dcode">[{"build": "0.2.0"}, {"day_of_week": "Mon"}]</code>
• Example selector: <code class="dcode">"build: current_build() day_of_week: Mon"</code>

Session Tags

A JSON array of key, value session tag objects may be used to select sets of sessions for analysis. Session tags are logically ANDed together, meaning that sessions selected must contain all of the tags provided.

Standard HeadSpin Session Tags

Session tags are the foundation for regression queries. For a list of standard tags applied by HeadSpin when a session is added to a User Flow, see the Annotating Your Session documentation.

Setting a threshold for pass/fail

• <code class="dcode">"threshold"</code>: The tolerable percent increase in network latency before failing a build.

The threshold expects a float value for the percent increase in total network latency that is tolerable before a build should be failed. If the network latency exceeds the specified threshold, a <code class="dcode">"failed"</code> status is returned. Otherwise a <code class="dcode">"passed"</code> status is returned.

Configuring the number of sessions considered

• <code class="dcode">"limit"</code> (optional): The maximum number of sessions to consider for each set of sessions.

The optional <code class="dcode">"limit"</code> argument must be a positive integer value and defaults to 15 sessions per set (30 total) when requested via <code class="dcode">/v0/regression</code> or 50 sessions per set (100 total) when requested via <code class="dcode">/v0/regression/worker</code>. Setting a different value in the request body will increase or decrease the number of sessions considered in the population of sessions used to select a representative session for <code class="dcode">"a"</code> and <code class="dcode">"b"</code>.

• <code class="dcode">"min_limit"</code> (optional): The minimum number of sessions required to make a useful comparison.

The optional <code class="dcode">"min_limit"</code> argument must be a positive integer value. If either set of sessions selected by the tags in <code class="dcode">"a"</a> or <code class="dcode">"b"</code> has fewer than <code class="dcode">"min_limit"</code> sessions (and more than 0), the API will skip the regression analysis because no meaningful comparison can be made.

Make a Regression Comparison

This API analyzes two sets of sessions for regressions and returns a pass/fail status and performance report based on parameters selected by the caller.

Parameters

See the section on Common Parameters for more details on the standard regression comparison parameters.

Required

• <code class="dcode">"user_flow_name":</code> The name of the User Flow to select sessions from.
• <code class="dcode">"a"</code>: Session tags or HeadSpin Selector string used to select the reference or control set of sessions.
• <code class="dcode">"b"</code>: Session tags or HeadSpin Selector string used to select the target or treatment set of sessions for comparison.

Optional

• <code class="dcode">"kpi"</code> (optional): A string representing the metric for setting the passed/failed status of the report. We use JMESpath expressions to select or construct the metric from the JSON provided in the regression report. If omitted, the report status will be marked as <code class="dcode">"complete"</code> on completion instead of passed/failed. See the guide for selecting a custom regression KPI for more details.
• <code class="dcode">"threshold"</code> (optional): The tolerable percent increase in the selected KPI before failing a build. The <code class="dcode">"threshold"</code> argument is required when a KPI is specified.
• <code class="dcode">"limit"</code> (optional): The maximum number of sessions to consider for each set of sessions. If omitted, the regression comparison will consider 15 sessions per set (30 sessions total) by default.
• <code class="dcode">"min_limit"</code> (optional): The minimum number of sessions required to make a useful comparison. If omitted, a regression comparison will be made if there is at least one session matching each set of selection criteria.
• <code class="dcode">"percentile_rank"</code> (optional): If provided, the corresponding percentile will be estimated for each metric independently, before performing the final comparison that determines regression status. (e.g., <code class="dcode">percentile_rank = 50</code> will estimate the median for each metric).
• <code class="dcode">"notifications"</code>: Configuration for how performance notifications (passed/failed status) should be pushed.
• <code class ="dcode">"email"</code>: An email address in your domain.
• Example: <code class="dcode">{"email": "test@company.io"}</code>

Route-Specific

• <code class="dcode">"percentile"</code> (optional): The two sessions most closely matching the specified percentile in total network latency will be selected for comparison. Note that <code class="dcode">"percentile"</code> is not currently compatible with selecting a custom <code class="dcode">"kpi"</code>.
• Example: <code class="dcode">50</code> (the median).

(Optional) Selecting representative sessions to compare

The <code class="dcode">"percentile"</code> argument uses the provided percentile to select a single representative session for comparison from the sessions selected by <code class="dcode">"a"</code> and <code class="dcode">"b"</code>. The API pseudo-randomly selects up to 15 sessions per set (30 total) by default or up to <code class="dcode">"limit"</code> sessions per set to use in comparison. The total network latency is analyzed for all pseudo-randomly sampled sessions and the sessions most closely matching the selected <code class="dcode">"percentile"</code> are selected for comparison.

Omitting this optional argument will execute regression comparisons for the mean of aggregated metrics across the sets of sessions defined in <code class="dcode">"a"</code> and <code class="dcode">"b"</code>.

Example Request

curl -X POST -H "Authorization: Bearer <your_api_token>" https://api-dev.headspin.io/v0/regression -d '{"a": "build: \"0.1.0\" day_of_week: Mon", "b": [{"build": "0.1.0"}, {"day_of_week": "Tue"}], "threshold": 10, "user_flow_name": "Example Regression User Flow", "kpi": "summary.issues.regression.\"Network Latency\"", "notifications": {"email": "test@headspin.io"}}' 

Response Details

A successful response returns the full regression report. See the Regression Report Schema.

Example Error Status Responses

Not Enough Sessions to Compare

If there are not enough sessions to satisfy the <code class="dcode">min_limit</code> constraint for either <code class="dcode">"a"</code> or <code class="dcode">"b"</code>, the API will return a 200 status code with the following response.

{
    "status": "skipped",
    "message": "Insufficient sessions for analysis."
}

Bad Input

Invalid arguments will recieve a 400 status code response with a description of what arguments were invalid or missing.

{
    "status": "<status message>",
    "status_code": 400
}

Internal Errors

Internal errors may return a 200 status code with the following response.

{
    "_headspin_progress": "....",
    "status": "error",
    "message": "Internal service error."
}

Create a Regression Worker

This API starts an asynchronous worker for a regression comparison. This route is an asynchronous version of the POST <code class="dcode">/v0/regression</code> route described in Make a Regression Comparison.

Parameters

See the section on Common Parameters for more details on the standard regression comparison parameters.

Required

• <code class="dcode">"user_flow_name"</code>: The name of the User Flow to select sessions from.
• <code class="dcode">"a"</code>: Session tags or HeadSpin Selector string used to select the reference or control set of sessions.
• <code c;ass="dcode">"b"</code>: Session tags or HeadSpin Selector string used to select the target or treatment set of sessions for comparison.

Optional

• <code class="dcode">"kpi"</code> (optional): A string representing the metric for setting the passed/failed status of the report. We use JMESpath expressions to select or construct the metric from the JSON provided in the regression report. If omitted, the report status will be marked as <code class="dcode">"complete"</code> on completion instead of passed/failed. See the guide for selecting a custom regression KPI for more details.
• <code class="dcode">"threshold"</code> (optional): The tolerable percent increase in the selected KPI before failing a build. The "threshold" argument is required when a KPI is specified.
• <code class="dcode">"limit"</code> (optional): The maximum number of sessions to consider for each set of sessions. If omitted, the regression comparison will consider 50 sessions per set (100 sessions total) by default.
• <code class="dcode">"min_limit"</code> (optional): The minimum number of sessions required to make a useful comparison. If omitted, a regression comparison will be made if there is at least one session matching each set of selection criteria.
• <code class="dcode">"percentile_rank"</code> (optional): If provided, the corresponding percentile will be estimated for each metric independently, before performing the final comparison that determines regression status. (e.g., percentile_rank = 50 will estimate the median for each metric).
• <code class="dcode">"notifications"</code>: Configuration for how performance notifications (passed/failed status) should be pushed.
• <code class="dcode">"email"</code>: An email address in your domain.
• Example: <code class="dcode">{"email": "test@headspin.io"}</code>

Route-Specific

• <code class="dcode">"percentile"</code> (optional): The two sessions most closely matching the specified percentile in total network latency will be selected for comparison. Note that <code class="dcode">"percentile"</code> is not currently compatible with selecting a custom <code class="dcode">"kpi"</code>.
• Example: <code class="dcode">50</code> (the median)

Example Request

curl -X POST -H "Authorization: Bearer <your_api_token>" https://api-dev.headspin.io/v0/regression/worker -d '{"a": "build: prev_build(1) geo: \"Mountain View, US\"", "b": "build: current_build() geo: \"Mountain View, US\"", "threshold": 10, "user_flow_name": "Example Regression User Flow", "kpi": "summary.issues.regression.\"Network Latency\"", "notifications": {"email": "test@headspin.io"}}' 

Response Details

Returns the unique ID for the worker-driven regression report. The report status may be polled using the regression worker <code class="dcode">GET</code> route as shown in the section below.

{
    "regression_id": "<UUID>"
}

Poll for Regression Worker Status

This route queries the regression report status and downloads the report JSON described in the section on the Regression Report Schema.

This route is an alias of Download a Regression Report.

Download a Regression Report

This route queries the regression report status and downloads the report JSON described in the section on the Regression Report Schema.

This route is an alias of Poll for Regression Worker Status.

Parameters

Required:

• <code class="dcode">regression_id</code>: The unique ID associated with your regression report.

Optional:

• <code class="dcode">download</code>: Optional query parameter. If <code class="dcode">download=true</code>, downloads the regression report when the analysis completes. If <code class="dcode">download=false</code>, the API only sends progress updates.
• <code class="dcode">jmespath_expression</code>: Optional query parameter. If specified, the expression will be used to select a subset of the regression report to return. JMESpath expressions support complex queries of JSON objects. Simpler versions of the syntax include each step in a JSON object path seperated by a period. For example: "labels.user.categories.\"Label Category\"" will return the subset of the regression report describing the labels in "Label Category".

Example Request

curl -H "Authorization: Bearer <your_api_token>" https://api-dev.headspin.io/v0/regression/worker/{regression_id}?download={bool} 

Response Details

While the worker is in progress:

{
    "regression_id": "<UUID>",
    "status": "<status>",
    "message": "<message>",
    "progress": <int>,
    "last_updated": <float>,
}

The <code class="dcode">"status"</code> field indicates whether the build exceeded the reference network latency by threshold amount.

Possible values:

• <code class="dcode">"pending"</code>: The worker is scheduled, but the regression comparison job has not yet started.
• <code class="dcode">"in progress"</code>: The worker has started processing your request.
• <code class="dcode">"skipped"</code>: A passed/failed status could not be set because the selected KPI was not available or there were not enough sessions for the provided selection criteria to satisfy the provided <code class="dcode">min_limit</code> constraint.
• <code class="dcode">"error"</code>: The API encountered an unexpected error.
• <code class="dcode">"passed"</code>: The target did not exceed the reference network latency by at least <code class="dcode">threshold</code> amount.
• <code class="dcode">"failed"</code>: The target exceeded the reference by at least <code class="dcode">threshold</code> amount.
• <code class="dcode">"complete"</code>: A full report was successfully generated. However, no KPI was selected therefore a passed/failed status cannot be set.

The <code class="dcode">"status"</code> indicator is determined by comparing the difference in the total network latency for the sessions selected in <code class="dcode">"b"</code> to those in <code class="dcode">"a"</code> using the <code class="dcode">"threshold"</code>. If the total network latency of sessions in <code class="dcode">"b"</code> is greater than or equal to the threshold percent above the total network latency of sessions in <code class="dcode">"a"</code>, the regression status is set to <code class="dcode">"failed"</code>.

For example, if the average total network latency for sessions in <code class="dcode">"a"</code> is 100 ms and the <code class="dcode">"threshold"</code> was specified as 10 percent, then the status will be set to <code class="dcode">"failed"</code> if the average total network latency of sessions in <code class="dcode">"b"</code> is greater than or equal to 110 ms.

The <code class="dcode">"complete"</code> status only occurs when the kpi argument is omitted in session comparison requested through <code class="dcode">/v0/regression</code>and <code class="dcode">/v0/regression/worker</code>.

The <code class="dcode">"message"</code> field is an optional description string explaining the <code class="dcode">"status"</code>.

The <code class="dcode">"progress"</code> field is the percent progress for analysis completion in [0, 100].

The <code class="dcode">"last_updated"</code> field is the UNIX timestamp of the last worker update recorded.

By default, returns the full regression report after the worker completes (<code class="dcode">"progress"</code> = 100). See the Regression Report Schema. If the query parameter download=false is provided, only the progress updates are reported.

List Regression Reports

Optional Parameters

• <code class="dcode">"user_flow_id"</code>: The ID of the User Flow to select sessions from.

Example Request

curl -X GET -H "Authorization: Bearer <your_api_token>" https://api-dev.headspin.io/v0/regression/list 

Response Details

The response is a list of regression reports. See the Regression Report Schema for details.

Create a Pro-Active Regression Alert Monitoring Watcher

This route configures a pro-active monitoring agent to run one or more regression comparisons at set intervals.

Parameters

See the section on Common Parameters for more details on the standard regression comparison parameters.

Required

• <code class="dcode">"user_flow_name"</code> or <code class="dcode">"user_flow_id"</code>: The name or ID of the User Flow to select sessions from.
• <code class="dcode">"a"</code>: Session tags or HeadSpin Selector string used to select the reference or control set of sessions.
• <code class="dcode">"b"</code>: Session tags or HeadSpin Selector string used to select the target or treatment set of sessions for comparison.
• <code class="dcode">"threshold"</code>: The tolerable percent increase in network latency before failing a build.
• <code class="dcode">"frequency"</code>: A valid <code class="dcode">cron</code> expression specifying the frequency at which the agent should execute regression comparisons. See our guide to cron expressions.

Optional

• <code class="dcode">"kpi"</code> (optional): A string representing the metric for setting the passed/failed status of the report. We use JMESpath expressions to select or construct the metric from the JSON provided in the regression report. If omitted, the aggregated total network latency will be used by default. See the guide for selecting a custom regression KPI for more details.
• <code class="dcode">"partition"</code> (optional): The desired Session Tag key that you wish to control for in your regression comparisons. See our section on how to partition your regression alerts.
• <code class="dcode">"ts_start"</code> (optional): The desired UNIX timestamp when the agent should start monitoring for performance regressions. If <code classs="dcode">ts_start</code> is before the current timestamp, <code class="dcode">ts_start</code> will be ignored.
• <code class="dcode">"ts_end"</code> (optional): The desired UNIX timestamp when the agent should stop monitoring for performance regressions.
• <code class="dcode">"limit"</code> (optional): The maximum number of sessions to consider for each set of sessions. If omitted, each regression comparison will consider 50 sessions per set (100 sessions total) by default.
• <code class="dcode">"min_limit"</code> (optional): The minimum number of sessions required to make a useful comparison. If omitted, a regression comparison will be made if there is at least one session matching each set of selection criteria.
• <code class="dcode">"percentile_rank"</code> (optional): If provided, the corresponding percentile will be estimated for each metric independently, before performing the final comparison that determines regression status. (e.g., <code class="dcode">percentile_rank</code> = 50 will estimate the median for each metric).

Route-Specific

Example Request

The following request creates a regression alert watcher called "My Performance Alert" to monitor for performance regressions of 10% or more across all HeadSpin Performance Issues between the previous build and the current build in the Mountain View geo-location running every 20 minutes until Tuesday February 9, 2021 at 01:09:48 UTC.

curl -X POST -H "Authorization: Bearer <your_api_token>" https://api-dev.headspin.io/v0/regression/alert/watcher -d '{"user_flow_name": "Example Regression User Flow", "a": "build: current_build() geo: \"Mountain View, US\"", "b": "build: prev_build(1) geo: \"Mountain View, US\"", "kpi": {"path": ["summary", "issues"], "metric": "Impact Time"}, "threshold": 10, "frequency": "*/20 * * * *", "name": "My Performance Alert", "ts_end": 1612832988, "notifications": {"email": "test@HeadSpin.io"}}' 

Response Details

{
    "name": "<str>",
    "watcher_id": "<UUID>",
    "alert_id": "<UUID>"
}

Create Multiple Regression Alert Watchers

This route configures multiple monitoring agents to run one or more regression comparisons at set intervals.

Parameters

This route supports creating multiple regression alert watchers from multiple configurations. See the section on how to create a Regression Alert Watcher for more details on the parameters per watcher configuration.

Required

• <code class="dcode">"watchers"</code>: An array of regression alert watcher configurations.

Optional

• <code class="dcode">"overrides"</code>: Any regression alert watcher configuration parameters that should apply to all watchers created.

Example Request

The following request creates two regression alert watchers called "My Build Performance Alert" and "My Time Performance Alert" to monitor for performance regressions of 10% or more across all HeadSpin Performance Issues between the selected sessions in the Mountain View geo-location running every 20 minutes until Tuesday February 9, 2021 at 01:09:48 UTC.

curl -X POST -H "Authorization: Bearer <your_api_token>" https://api-dev.headspin.io/v0/regression/alert/watcher -d '{"watchers": [{"a": "build: current_build() geo: \"Mountain View, US\"", "b": "build: prev_build(1) geo: \"Mountain View, US\"", "name": "My Build Performance Alert", }, {"a": "end_timestamp: minutes_ago(20) geo: \"Mountain View, US\"", "b": "end_timestamp: minutes_ago(80) geo: \"Mountain View, US\"", "name": "My Time Performance Alert"}], "overrides": {"user_flow_name": "Example Regression User Flow", "kpi": "summary.issues.regression.\"Impact Time\""}, "threshold": 10, "frequency": "*/20 * * * *", "ts_end": 1612832988, "notifications": {"email": "test@HeadSpin.io"}}}' 

Response Details

Responds with a <code class="dcode">watcher_id</code> and the first pending <code class="dcode">alert_id</code> per watcher created.

{
    "watchers": [
        {
            "name": "<str>",
            "watcher_id": "<UUID>",
            "alert_id": "<UUID>"
        },
        {
            "name": "<str>",
            "watcher_id": "<UUID>",
            "alert_id": "<UUID>"
        }
}

Delete a Pro-Active Regression Alert Watcher and All Associated Alerts and Reports

This route deletes a pro-active regression alert watcher and all associated regression alerts and reports.

Example Request

A request body containing the ID of the watcher to be deleted is expected to confirm the deletion request. Note that the watcher ID is expected in both the path and the request body.

curl -X DELETE -H "Authorization: Bearer <your_api_token>" https://api-dev.headspin.io/v0/regression/alert/watcher/{watcher_id} -d '{"watcher_id": ""}' 

Response Details

{
    "status_code": 200,
    "status": "OK"
}

Get Information About a Pro-Active Regression Alert Watcher

Example Request

curl -H "Authorization: Bearer <your_api_token>" https://api-dev.headspin.io/v0/regression/alert/watcher/{watcher_id} 

Response Details

{
    "watcher_id": "<UUID>",
    "name": <string>,
    "user_flow_id": "<UUID>",
    "a": <string>,
    "b": <string>,
    "kpi": <string>, // JMESpath Expression
    "threshold": <float>,
    "notifications": {
        "email": <string>
    },
    "frequency": <string>,
    "ts_created": <float>,
    "ts_start": <float>,
    "ts_end": <float>,
    "min_limit": <int>,
    "limit": <int>,
    "partition": <string>,
}

List Regression Alert Watchers

Optional Parameters

• <code class="dcode">"user_flow_id"</code>: The ID of the User Flow to select sessions from.

Example Request

curl -H "Authorization: Bearer <your_api_token>" https://api-dev.headspin.io/v0/regression/alert/watcher/list 

Response Details

{"watchers": [
        {
            "watcher_id": "<UUID>",
            "name": <string>,
            "user_flow_id": "<UUID>",
            "a": <string>,
            "b": <string>,
            "kpi": <string>, // JMESpath expression
            "threshold": <float>,
            "notifications": {
                "email": <string>
            },
            "frequency": <string>,
            "ts_created": <float>,
            "ts_start": <float>,
            "ts_end": <float>,
            "min_limit": <int>,
            "limit": <int>,
            "partition": <string>
        },
    ]
}

List Alerts

Optional Parameters

• <code class="dcode">"user_flow_id"</code>: The ID of the User Flow to select sessions from.

Example Request

curl -H "Authorization: Bearer <your_api_token>" https://api-dev.headspin.io/v0/regression/alert/list 

Response Details

{"alerts": [
        {
            "alert_id": "<UUID>",
            "watcher_id": "<UUID>",
            "ts_start": <float>,
            "ts_created": <float>,
            "last_updated": <float>,
            "status": <string>,
            "partition": <string>,
            "progress": <int>
        },
    ]
}

Get Information About an Alert

Example Request

curl -H "Authorization: Bearer <your_api_token>" https://api-dev.headspin.io/v0/regression/alert/{alert_id} 

Response Details

{
    "alert_id": "<UUID>",
    "watcher_id": "<UUID>",
    "ts_start": <float>,
    "ts_created": <float>,
    "last_updated": <float>,
    "status": <string>,
    "partition": <string>,
    "progress": <int>
}

Regression Report Schema

{
    "_headspin_progress": ".....",  // a progress indicator used to keep the connection alive
    "version": <int>,  // schema version
    "progress": <int>, // analysis percent completion in [0, 100]
    "status": "<complete|passed|failed|skipped|error>",  // the pass or fail status of the performance regression analysis
    "message": "<string>",  // additional information regarding the "status" field
    "url": "https://ui-dev.headspin.io/...",  // the deep link into the respective regression UI
    "sessions": {  // information about the sessions selected using the provided session tags
        "a": {
            "total": <int>,  // the total number of sessions matching the selected tags
            "considered": <int>,  // the total number of sessions to be analyzed
            "included": <int>  // the number of sessions with data included in this payload
        },
        "b": {...}
    },
    "summary": {  // issue and network summaries across all sessions
        "issues": {  // across all issues
            "a": {
                "Impact Count": <float>,  // mean number of issue impacts
                "Impact Time": <float>,  // mean impact time in milliseconds
            },
            "b": {...},
            "regression": {...},
        },
        "network": {  // across all hosts and network traffic
            "a": {
                "HTTP Latency": <float>,  // mean total HTTP latency in milliseconds
                "Network Latency": <float>,  // mean total latency in milliseconds
                "HTTP Receive": <float>,  // mean HTTP download time in milliseconds
                "TCP Connect": <float>,  // mean TCP connect time in milliseconds
                "TLS Handshake": <float>,  // mean TLS handshake time in milliseconds
                "Transport Latency": <float>, // mean transport latency in milliseconds
                "HTTP Wait": <float>,  // mean HTTP wait time in milliseconds
            },
            "b": {...},
            "regression": {...},
        },
    },
    "issues": {  // issue data by category
        "<issue_category>": {  // the respective issue category
            "a": {
                "Impact Time": <float>,  // mean impact time in milliseconds for affected sessions
                "Impact Count": <float>,  // mean number issue impacts of this type per session affected
                "sessions": <int>, // sessions affected
            },
            "b": {...},
            "regression": {...},
            "descriptions": {
                "short": "<text>",  // brief Issue card description
                "explanation": "<text>",  // longer explanation of the Issue card
                "suggestions": "<text>",  // suggestions for how to resolve this Issue
                "group": "<text>",  // meta-grouping for Issue card (e.g. "User Experience" or "Root Cause")
            },
        },  // ...
    },
    "network": {  // network data by host
        "<subdomain>:<port>": {  // host
            "a": {
                "Connection Count": <float>,  // mean number of connections
                "HTTP Receive": <float>,  // mean HTTP download time in milliseconds
                "HTTP Latency": <float>,  // mean HTTP latency in milliseconds
                "HTTP Request Count": <float>,  // mean HTTP request count
                "HTTP Wait": <float>,  // mean HTTP wait time in milliseconds
                "TCP Connect":  <float>,  // mean TCP connect time in milliseconds
                "TLS Handshake": <float>,  // mean TLS handshake time in milliseconds
                "Network Latency": <float>,  // mean total network latency in milliseconds
                "sessions": <int>,  // total number of sessions containing traffic to this host
            },
            "b": {...},
            "regression": {...},
            "urls": [
                    {
                        "method": "<text>",  // the HTTP request method
                        "url": "<text>"  // the full request URL
                        "a": {
                            "HTTP Receive": <float>,  // mean HTTP download time in milliseconds for this request
                            "HTTP Latency": <float>,  // mean HTTP latency in milliseconds
                            "HTTP Wait": <float>,  // mean HTTP wait time in milliseconds
                            "sessions": <int>,  // total number of sessions containing this request
                        },
                        "b": {...},
                        "regression": {...}
                    },  // ...
                ],
                "base urls": [
                    {
                        "method": "<text>",  // the HTTP request method
                        "url": "<text>"  // the request URL stripped of any query parameters
                        "a": {
                            "HTTP Receive": <float>,  // mean HTTP download time in milliseconds
                            "HTTP Latency": <float>,  // mean HTTP latency in milliseconds
                            "HTTP Wait": <float>,  // mean HTTP wait time in milliseconds
                            "sessions": <int>,  // total number of sessions containing requests matching this request method and base URL
                        },
                        "b": {...},
                        "regression": {...}
                    },  // ...
                ],
        },
    },
    "labels": {
        "<label_type>": {  // metrics for labels of type <label_type>, typically "user" for labels applied via UI or the Session Annotation API
            "a": {
                "Duration": <float>,  // mean label duration per session in seconds
                "Count": <int>,  // mean number of labels per session
                "sessions": <int>,  // sessions affected
            },
            "b": {...},
            "regression": {...},
            "categories": {
                "<category_name>": {  // metrics for labels with type <label_type> and category <category_name>
                    "a": {
                        "Duration": <float>,
                        "Count": <int>,
                        "sessions": <int>,
                    },
                    "b": {...},
                    "regression": {...},
                    "names": {
                        "<label_name>": {  // metrics for labels with type <label_type>, category <category_name>, and name <label_name>
                            "a": {
                                "Duration": <float>,
                                "Count": <int>,
                                "sessions": <int>,
                            },
                            "b": {...},
                            "regression": {...},
                        }
                    },
                }
            }
        },  // ...
    },
}

HeadSpin progress indicator

This is a progress indicator used to keep the connection alive. This indicator can be ignored.

• <code class="dcode">"progress"</code>: An indicator of the analysis progress in percent completion. This will currently always be <code class="dcode">100</code>.

Pass/fail status indicator

The <code class="dcode">"status"</code> field indicates whether the build exceeded the reference KPI by <code class="dcode">threshold</code> percent.

Possible values:

• <code class="dcode">"skipped"</code>: A passed/failed status could not be set because the selected KPI was not available or there were not enough sessions for the provided selection criteria to satisfy the provided <code class="dcode">min_limit</code> constraint.
• <code class="dcode">"error"</code>: The API encountered an unexpected error.
• <code class="dcode">"passed"</code>: The target did not exceed the reference network latency by at least <code class="dcode">threshold</code> amount.
• <code class="dcode">"failed"</code>: The target exceeded the reference by at least <code class="dcode">threshold</code> amount.
• <code classs="dcpde">"complete"</code>: A full report was successfully generated. However, no KPI was selected therefore a passed/failed status cannot be set.

This <code class="dcode">"status"</code> indicator is determined by comparing the difference in the selected <code class="dcode">"kpi"</code> for the sessions selected in <code class="dcode">"b"</code> to those in <code class="dcode">"a"</code> using the <code class="dcode">"threshold"</code>. If the KPI of sessions in <code class="dcode">"b"</code> is greater than or equal to the threshold percent above the KPI of sessions in <code class="dcode">"a"</code>, the regression status is set to <code class="dcode">"failed"</code>.

For example, if the average total network latency for sessions in <code class="dcode">"a"</code> is 100 ms and the <code class="dcode">"threshold"</code> was specified as <code class="dcode">10</code> percent, then the status will be set to <code class="dcode">failed</code> if the average total network latency of sessions in <code class="dcode">"b"</code> is greater than or equal to 110 ms.

The <code class="dcode">"complete"</code> status only occurs when the kpi argument is omitted in session comparison requested through <code class="dcode">/v0/regression</code> and <code class="dcode">/v0/regression/worker</code>. Since alert watchers fall back to using <code class="dcode">Network Latency</code> as the default KPI, a passed/failed status will always be set at the end of the regression analysis.

Additional information regarding the status indicator

The <code class="dcode">"message"</code> field contains a human-readable message providing more context regarding the status of the regression comparison.

URL for the Regression Intelligence dashboard

• <code class="dcode">"url"</code>: The URL to the regression intelligence dashboard for this report.

This URL links to the regression dashboard where details regarding the regressions for the sessions selected in <code class="dcode">"a"</code> and <code class="dcode">"b"</code> can be inspected.

Context for any observed regressions

• <code class="dcode">"a"</code>: Data from the reference set of sessions.
• <code class="dcode">"network"</code>: Any network data observed.
• <code class="dcode">"issues"</code>: Any HeadSpin Performance Issues observed.
• <code class="dcode">"b"</code>: Data from the target or comparison set of sessions.
• <code class="dcode">"regressions"</code>: The observed regressions between the target and the reference. Note that regressions may have negative values if there was an improvement between the target and the reference.

Any hosts and API requests observed in the network traffic and any HeadSpin AI-based Issues identified are included in the response for context. Both regressions and improvements are included and all data values are in units of counts or milliseconds.

HeadSpin Selector Syntax

A HeadSpin Selector string may be used for more complex selection expressions. Selectors support logical AND, OR, NOT, value comparisons, glob expressions, and more.

Syntax

Selector expressions may be used to make assertions about the sessions tags applied to the sessions selected for comparison.

Logical AND

A logical AND may be expressed implicitly using a single space or explicitly using a + between session tag expressions.

Selector strings that are logically equivalent to the session tag argument <code class="dcode">[{"build": "0.1.0"}, {"day_of_week": "Mon"}]</code> are:

• <code class="dcode">build: 0.1.0 day_of_week: Mon</code>: The single space between <code class="dcode">build: 0.1.0</code> and <code class="dcode">day_of_week: Mon</code> is an implicit AND.
• <code class="dcode">build: 0.1.0 + day_of_week: Mon</code>: The <code class="dcode">+</code> operator is an explicit AND.

Logical OR

A logical OR may be expressed using a comma between session tag expressions. For example, <code class="dcode">build: 0.1.0, build: 0.2.0</code> would select sessions tagged with exactly <code class="dcode">build: 0.1.0 or build: 0.2.0</code>.

Logical NOT

An expression may be negated using the ! symbol.

For example, <code class="dcode">build: 0.1.0 !geo: "Mountain View, US"</code> would match all sessions tagged with <code class="dcode">build: 0.1.0</code> that were not captured in the Mountain View geolocation.

Literals

Session tags with keys or values consisting of multiple words or containing symbols may be expressed as string literals using double quotation marks.

Examples:

• <code class="dcode">geo: "Palo Alto, US"</code> matches the tag <code class="dcode">{"geo": "Palo Alto, US"}</code>
• <code class="dcode">"some complex tag key with symbols !@,": 1</code> matches the tag <code class="dcode">{"some complex tag key with symbols !@,": "1"}</code>

Note that when passing a HeadSpin Selector string literal as an argument to the Regression API you will need to escape the quotation marks in the string literal. For example, to select sessions recorded in Palo Alto with the selector <code class="dcode">geo: "Palo Alto, US"</code> as the reference set for a regression comparison, you would pass the following argument to the Regression API: <code class="dcode">"a": "geo: \"Palo Alto, US\""</code> where <code class="dcode">"a"</code> indicates the selector or tag expression for selecting the reference sessions and <code class="dcode">"geo: \"Palo Alto, US\""</code> is the HeadSpin Selector string argument.

Complex Value Expressions

The HeadSpin Selector syntax supports value assertions using the following unary operators:

• <code class="dcode">=</code>: exactly equal to
• <code class="dcode">!=>/code>: not equal to
• <code class="dcode"><</code>: less than
• <code class="dcode"><=</code>: less than or equal to
• <code class="dcode">></code>: greater than
• <code class="dcode">>=</code>: greater than or equal to

Simple selector expressions implicitly express exact matches equivalent to using the = symbol. For example, the following expressions are identical:

• <code class="dcode">build: 0.1.0</code>
• <code class="dcode">build: =0.1.0</code>
• <code class="dcode">=build: =0.1.0</code>

Complex expressions require explicit unary operators. For example, you might use this expression <code class="dcode">day_of_week: !=Sat !=Sun</code> to select sessions captured during the week and compare those sessions to sessions captured over the weekend using this expression <code class="dcode">day_of_week: =Sat, =Sun</code>.

Other examples:

Select sessions greater than some build version.

build: >1.7.0

Select sessions with durations greater than a minute but less than two minutes.

duration_seconds: >60 <120

Select sessions captured within a specific time window using UNIX timestamps

start_timestamp: >1605323758.323 <1605327358.323 end_timestamp: >1605323758.323 <1605327358.323

Precedence

Parentheses may be used to indicate precedence in selector expressions.

Select sessions associated with builds greater than 100 and less than 200 or otherwise equal to build 300.

build: >100 (<200, =300)

Glob Expressions

Selectors work with glob-like syntax.

Select all sessions captured in US geos.

geo: *US

Select all builds of a similar version.

build: 0.4.*

Select all iPhone devices.

device_sku: iPhone*

Selector Functions

In some cases, it may be desirable to continuously monitor performance regressions using a relative selector. The HeadSpin Selector syntax supports the following functions to simplify these kinds of queries:

Relative Time-Based Selectors

The <code class="dcode">hours_ago(N)</code> and <code class="dcode">minutes_ago(N)</code> selector functions support relative time-based queries from the UNIX timestamp when the regression request was received. These functions are implemented only for the standard <code class="dcode">start_timestamp</code> and <code class="dcode">end_timestamp</code> session tags that correspond to the UNIX timestamp for the session capture start and end times, respectively. You can think of <code class="dcode">hours_ago(N)</code> as a function that subtracts <code class="dcode">N</code> hours from the current timestamp and returns the resulting timestamp.

Examples

Compare sessions ending in the last hour to those ending between an hour and two hours ago.

{
    "b": "end_timestamp: >hours_ago(1)",
    "a": "end_timestamp: <hours_ago(1) >hours_ago(2)",
    // ...
}

Compare sessions ending in the last 15 minutes to those ending between 15 minutes and 30 minutes ago.

{
    "b": "end_timestamp: >minutes_ago(15)",
    "a": "end_timestamp: <minutes_ago(15) >minutes_ago(30)",
    // ...
}

Relative Build Selectors

The <code class="dcode">current_build()</code> and <code class="dcode">prev_build(x)</code> selector functions support relative queries for build regression selectors. The <code class="dcode">current_build()</code> function automatically matches the largest observed build version (numeric and SemVer versions are supported). The <code class="dcode">prev_build(N)</code> function matches an integer number (<code class="dcode">N</code>) of builds prior to the current build.

Compare the previous build to the current build.

{
    "a": "build: current_build()",
    "b": "build: prev_build(1)",
    // ...
}

Note that the selector <code class="dcode">build: current_build()</code> is equivalent to <code class="dcode">build: prev_build(0)</code>.

Guide: Setting Pro-Active Alert Frequency Using Cron Expressions

Creating a pro-active regression alert watcher requires selecting an execution schedule using a valid <code class="dcode">cron</code> schedule expression. A <code class="dcode">cron</code> expression has five fields passed in order for specifying the minutes, hours, days of month, month, and days of week that the agent should execute a process.

The values column can have a <code class="dcode">*</code> or a list of elements separated by commas. An element is either a value or a value range between two values separated by a hyphen. For example a value range for Monday through Friday would be <code class="dcode">1-5</code> or <code class="dcode">Mon-Fri</code>. All value ranges are inclusive.

Repeat patterns may be specified using a <code class="dcode">/</code>. For example, an expression to repeat every 30 minutes is <code class="dcode">*/30 * * * *</code>.

Guide: How to Partition Pro-Active Regression Alerts

A "partition" is a convenient way to configure multiple regression comparisons while controlling for variation that you may not be interested in monitoring using a pro-active regression alert watcher. A partition is a single Session Tag key that splits each alert into segments grouping sessions matching each value associated with the given tag key.

For example, you may wish to compare performance of the current build of your application over the previous build. The selectors <code class="dcode">build: current_build()</code> and <code class="dcode">build: prev_build(1)</code> may be used to match all sessions tagged with your latest and previous build numbers, respectively. If you wished to control for variation across geolocation, you may partition your alerts by passing <code class="dcode">"geo"</code> as the argument for the <code class="dcode">"partition"</code> parameter. This will group sessions into matching geolocations such as <code class="dcode">"geo": "Mountain View, US" and "geo": "Palo Alto, US"</code>. Regression comparisons and notifications are handled independently for each partition, so you would recieve notifications of performance regressions for each geolocation as well as notifications when each performance incident has been resolved.

Guide: Selecting a Custom Regression Alert KPI

A regression comparison executed via the regression comparison, regression worker, or pro-active regression alert watcher APIs may be configured with a custom Key Performance Indicator (KPI) metric and performance threshold to determine when a regression should be indicated and, in the case of the regression alert watcher, when alert notifications should be pushed. If a KPI is omitted, the default KPI is the aggregated total Network Latency across all hosts across all sessions.

A KPI selector consists of a JMESpath expression slicing into the Regression Report Schema. Most JMESpath selectors will be a string of keys specifying a path to a leaf of the regression report schema's JSON object. Each step in the path should be separated by periods. For example, the default KPI selector is <code class="dcode">summary.network.regression.\"Network Latency\</code>". See the JMESpath tutorial for a detailed guide to the syntax.

Tip

Since some custom KPI metrics may not be known to the system until after a regression comparison is made, we strongly recommend testing your selection criteria and custom KPI criteria using the regression comparison or regression worker APIs prior to configuring a pro-active regression alert watcher. In most cases, running a quick comparison using the regression comparison API with a <code class="dcode">low</code> limit argument to reduce the number of sessions analyzed will be sufficient to ensure that your selectors are configured appropriately.

Example: Reproduce the Default KPI

The default KPI is the total Network Latency for all hosts aggregated accross sessions. To use the default KPI implicitly the <code class="dcode">"kpi"</code> arg may be omitted. To use the default KPI explicitly, pass the following args when using the regression comparison, regression worker, or pro-active regression alert watcher APIs:

{
    ...,  // other args
    "kpi": "summary.network.regression.\"Network Latency\""
}

Example: Alert Based on the Duration of Custom Session Labels with Category "Label of Interest" and Name "Login"

To trigger an alert if the duration of session labels with category "Label of Interest" and name "Login" increases by 10% in the target sessions (represented by the selection criteria for <code class="dcode">"b"</code>) relative to the reference sessions (represented by the selection criteria for <code class="dcode">"a"</code>), pass the following args when creating a pro-active regression alert watcher:

{
    ...,  // other args
    "threshold": 10,  // a notification will be sent if the KPI in b increases by more than "threshold" percent
    "kpi": "labels.user.categories.\"Label of Interest\".names.Login.regression.Duration"
    }
}

Example: Alert Based on Poor Loading Experiences using the HeadSpin Loading Animation Issue Card Impact Time

To trigger an alert for performance regressions measured using time that preloader or buffering animations are displayed on screen, pass the following args when creating a pro-active regression alert watcher:

{
    ...,  // other args
    "kpi": "issues.\"Loading Animation\".regression.\"Impact Time\""
}