Measuring Visual Load Time in Sessions

The HeadSpin platform offers several loading metrics that can estimate how quickly content becomes available to the end user. Our loading metrics are developed from computer vision and machine learning algorithms that analyze the video screen recording captured in HeadSpin sessions for indicators of loading content.

This visual approach is advantageous for two main reasons:

1. Rather than measuring proxies for loading events that may not align with the end-user experience, our visual loading metrics directly measure what gets rendered on the screen, i.e., what the end user actually sees.
2. Since instrumenting application logs for event listening is not required, the visual loading analyses can run on any HeadSpin session with a video recording. This means that our visual loading metrics have the flexibility to work across the different environments in the HeadSpin device cloud, whether that be browsers or apps on mobile phones, desktops, or media streaming devices.

The visual loading metrics currently available on the HeadSpin platform are Low Page Content, Loading Animation, and Page Load, which can be used synergistically to get a more comprehensive picture of user-experienced loading time.

Summary

Page Load

Page Load highlights the loading region in its entirety for any arbitrary page transitions. This is achieved by detecting where the first and last meaningful page changes occur and then highlighting the time interval between them. It is designed to be used in sessions or annotated regions that have known start and end states. For example, a cold start test session that starts in the home screen and ends with a fully loaded app page is a good candidate for this analysis. In this case, the analysis detects the first frame that deviates from the home screen (start of the page load) and then the first frame at which the fully loaded app page occurs (end of the page load).

To reiterate, given an annotated region that starts at Screen 1 and ends at Screen 2 (shown by the gray bar in the illustration below), this analysis is useful if the goal is to measure the time it takes to offload from Screen 1 to reach Screen 2 (shown by the purple bar).

Here we measured "Total Load Time" or "Time to Fully Load" by specifying Screen 1 as the page before clicking on the "Contact Us" button and Screen 2 as the fully loaded "Contact Us" page. Depending on what we use as Screen 1 and Screen 2, we can measure a variety of "Time to X" metrics (e.g., "Time to Interact"), making this analysis flexible and suitable for many use cases. You can also tune how sensitive the analysis should be in detecting the page changes (see the section below on Tunable Sensitivities for details).

There are several ways to run the analysis and retrieve the analysis results:

Create a Session Label in the Waterfall UI

On the timeline, click and drag your mouse across the region on which you would like to run the analysis. This opens the <code class="dcode">Create Label</code> popup with the pre-filled <code class="dcode">Start Time</code> and <code class="dcode">End Time</code> that correspond to the region that you highlighted. From the <code class="dcode">Type</code> drop-down menu, select <code class="dcode">Analysis: Page Load Request</code>. Fill out the required <code class="dcode">Name</code> field and then click the <code class="dcode">Submit</code> button to create the annotation. Optionally, you can fill out the Data to specify custom sensitivities, e.g., <code class="dcode">{"end_sensitivity": 0.8}</code>. This triggers the Page Load analysis to run on the region specified by the label, which creates a label on the resulting page load region from the analysis.

Shown below is an example of this process:

The light gray bar is the user-annotated region (<code class="dcode">page-load-request</code> label), whereas the purple bar overlaid on top is the resulting page load region from the analysis (<code class="dcode">page-load-result</code> label).

Use the Session Annotation API for Page Load Analysis

• Add <code class="dcode">"page-load-request"</code> labels to provide the time intervals in a session (and optionally spatial regions on the screen) to be analyzed for page load time.
• Retrieve <code class="dcode">"page-load-result"</code> labels by looking up the label group ids of the created <code class="dcode">"page-load-request"</code> labels.

Run page load analysis by adding labels

To run the page load analysis, simply add labels on the time intervals of interest with the <code class="dcode">"label_type"</code> specified to <code class="dcode">"page-load-request"</code>. For each <code classs="dcode">"page-load-request"</code> label that you add, you can provide the following optional customizations:

• Specify custom sensitivies in the <code class="dcode">"data"</code> field. See the section below Tunable Sensitivities.
• Specify the coordinates of a bounding box in the <code class="dcode">"video_box"</code> field. See our documentation on applying spatial filtering on video analyses.

Example: Add <code class="dcode">"page-load-request"</code> labels

curl -X POST -H "Authorization: Bearer <your_api_token>" https://api-dev.headspin.io/v0/sessions/{session_id}/label/add -d '{"labels": [{"name": "no customizations", "label_type": "page-load-request", "start_time": "24.37", "end_time": "29.39"}, {"name": "customized", "label_type": "page-load-request", "start_time": "24.37", "end_time": "29.39", "video_box": [[0, 600, 450, 700]], "data": {"start_sensitivity": 0.98, "end_sensitivity": 0.99}}]}' 

Response

{"label_ids": ["85e731d2-65cd-11eb-b3c0-e7adb5682894", "85e731f0-65cd-11eb-b3c0-e7adb5682894"]}

These are the label IDs of the two <code class="dcode">"page-load-request"</code> labels that you added. If the page load regions are found, then <code class="dcode">"page-load-result"</code> labels are created on them during the analysis.

Retrieve page load analysis results by fetching labels

You can check the session Waterfall UI to see if any <code class="dcode">"page-load-result"</code> labels have been created.

You can also look up the labels with a label group ID since a <code class="dcode">"page-load-request"</code> label and its resulting <code class="dcode">"page-load-result"</code> label are <code class="dcode">"linked"</code> by the same label group ID.

Example: Look up a <code class="dcode">"page-load-request"</code> label

First, look up the "page-load-request" label with its label ID to obtain its label group ID:

curl -H "Authorization: Bearer <your_api_token>" https://api-dev.headspin.io/v0/sessions/label/85e731f0-65cd-11eb-b3c0-e7adb5682894 

Response

{
    "label_id": "85e731f0-65cd-11eb-b3c0-e7adb5682894",
    "label_group_id": "85e731f1-65cd-11eb-b3c0-e7adb5682894",
    "label_type": "page-load-request",
    "session_id": "2b1312f9-55ec-11eb-9382-067dc7f9db8c",
    "name": "customized",
    "category": "page load",
    "start_time": 24370,
    "end_time": 29390,
    "ts_start": 1610575880.728,
    "ts_end": 1610575885.748,
    "data": {"start_sensitivity": 0.98, "end_sensitivity": 0.99},
    "pinned": false,
    "video_box": [[0, 600, 450, 700]]
}

Example: Get <code class="dcode">"page-load-result"</code> labels by looking up all labels associated with a label group

Then look up the labels that share the same label group ID to find the <code class="dcode">"page-load-result"</code> label:

curl -H "Authorization: Bearer <your_api_token>" https://api-dev.headspin.io/v0/sessions/label/group/6cd99b5b-6280-11eb-aa94-e5af0b819d09 

Response

{
    "labels": [
        {
            "label_id": "85e731f0-65cd-11eb-b3c0-e7adb5682894",
            "label_group_id": "85e731f1-65cd-11eb-b3c0-e7adb5682894",
            "label_type": "page-load-request",
            "session_id": "2b1312f9-55ec-11eb-9382-067dc7f9db8c",
            "name": "customized",
            "category": "page load",
            "start_time": 24370,
            "end_time": 29390,
            "ts_start": 1610575880.728,
            "ts_end": 1610575885.748,
            "data": {"start_sensitivity": 0.98, "end_sensitivity": 0.99},
            "pinned": false,
            "video_box": [[0, 600, 450, 700]]
        },
        {
            "label_id": "87cfc70c-65cd-11eb-b3c0-e7adb5682894",
            "label_group_id": "85e731f1-65cd-11eb-b3c0-e7adb5682894",
            "label_type": "page-load-result",
            "session_id": "2b1312f9-55ec-11eb-9382-067dc7f9db8c",
            "name": "customized",
            "category": "page load",
            "start_time": 26160,
            "end_time": 28640,
            "ts_start": 1610575882.518,
            "ts_end": 1610575884.998,
            "data": {"start_sensitivity": 0.98, "end_sensitivity": 0.99},
            "pinned": false,
            "video_box": [[0, 600, 450, 700]]
        },
    ]
}

Tuneable sensitivities

The following sensitivities can be tuned (default values shown) in the page load analysis:

{
    "start_sensitivity": 0.835,
    "end_sensitivity": 0.975
}

You can provide values beween 0 (not at all sensitive) to 1 (extremely sensitive) for both <code class="dcode">"start_sensitivity"</code> and <code class="dcode">"end_sensitivity"</code>. The higher the sensitivity value, the more sensitive the analysis is to detecting minute screen changes between frames.

If you wish to use custom sensitivities on a time interval of interest, specify the sensitivities in the <code class="dcode">"data"</code> field when adding a <code class="dcode">"page-load-request"</code> label.

Page Load Time API

Run page load analysis

Unless you need the page load analysis results directly returned, we recommend that you run the page load analysis via adding "page-load-request" labels over calling this API.

This API does exactly the same thing as adding <code class="dcode">"page-load-request"</code> labels but runs a blocking analysis that returns the analysis results directly. When this API is called, <code class="dcode">"page-load-request"</code> labels are first created on the specified time intervals, on which the analysis is run. Then for each of the <code class="dcode">"page-load-request"</code> label, a <code class="dcode">"page-load-result"</code> label is created if a page load region is found.

Request Body

The request body is optional. If omitted, the API will run the analysis over the full session with the default values. If provided, the request body must be a JSON object with the following optional keys:

An example body:

{
    "regions":
    [
        {
            "name": "Page A to B",
            "start_time": "20.0",
            "end_time": "30.0",
            "video_box": [[100, 125, 300, 550]]
        },
        {
            "name": "Page B to C",
            "start_time": "40.0",
            "end_time": "50.0",
            "start_sensitivity": 0.95
        }
    ],
    "end_sensitivity": 0.99,
    "wait_timeout_sec": 300
}

Example

curl -X POST -H "Authorization: Bearer <your_api_token>" https://api-dev.headspin.io/v0/sessions/analysis/pageloadtime/{session_id} -H "Authorization: Bearer {api_token}" -d '{"regions": [{"name": "Page A to B", "start_time": "20.0", "end_time": "30.0", "video_box": [[100, 125, 300, 550]]}, {"name": "Page B to C", "start_time": "40.0", "end_time": "50.0", "start_sensitivity": 0.95}], "end_sensitivity": 0.99", "wait_timeout_sec": 300}'

Recommended: Unless you need the page load analysis result directly returned, run the analysis via adding "page-load-request" labels instead. Instead of the request above, you can use the example request below which does exactly the same thing:

curl -X POST -H "Authorization: Bearer <your_api_token>" https://api-dev.headspin.io/v0/sessions/{session_id}/label/add  -d '{"labels": [{"name": "Page A to B", "label_type": "page-load-request", "start_time": "20.0", "end_time": "30.0", "data": {"end_sensitivity": 0.99}, "video_box": [[100, 125, 300, 550]]}, {"name": "Page B to C", "label_type": "page-load-request", "start_time": "40.0", "end_time": "50.0", "data": {"start_sensitivity": 0.95, "end_sensitivity": 0.99}}]}' 

Response: Result successfully returned

An example response is shown here where a visual load event was identified for the first request region but not for the second one:

{
    "units": "milliseconds since session start for start_time and end_time, seconds since epoch for ts_start and ts_end",
    "page_load_regions":
    [
        {
            "request_label_id": "c9a58208-8ce1-11eb-b51d-eb23d338cc0b",
            "request_label_name": "Page A to B",
            "request_start_time": 20000,
            "request_end_time": 30000,
            "request_ts_start": 1585227174.565,
            "request_ts_end": 1585227184.565,
            "start_sensitivity": 0.835,
            "end_sensitivity": 0.99,
            "video_box": [[100, 125, 300, 550]],
            "result_label_id": "632e8f8e-72e7-11ea-aa54-9cb6d08b256d",
            "start_time": 20280,
            "end_time": 24000,
            "ts_start": 1585227174.845,
            "ts_end": 1585227178.565
        },
        {
            "request_label_id": "c9a5820a-8ce1-11eb-b51d-eb23d338cc0b",
            "request_label_name": "Page B to C",
            "request_start_time": 40000,
            "request_end_time": 50000,
            "request_ts_start": 1585227194.565,
            "request_ts_end": 1585227204.565,
            "start_sensitivity": 0.95,
            "end_sensitivity": 0.99,
            "video_box": null,
            "error_msg": "Unable to identify a page change."
        }
    ],
    "session_id": "6d05aaa1-8377-11eb-8fb7-067dc7f9db8c",
    "_headspin_progress": "."
}

Response: Timeout

If the request takes longer than <code class="dcode">"wait_timeout_sec"</code> to complete and return the result, then the following response is returned:

{
    "_headspin_progress": ".",
    "status": "timeout",
    "request_label_ids": ["c9a58208-8ce1-11eb-b51d-eb23d338cc0b", "c9a5820a-8ce1-11eb-b51d-eb23d338cc0b"],
    "message": "Page load request labels were created on the specified regions, but the analysis did not complete within 300 seconds and timed out.",
    "session_id": "6d05aaa1-8377-11eb-8fb7-067dc7f9db8c"
}

This does not mean that the analysis failed to run. Please check the Waterfall UI or retrieve the page load result labels later. If you do not need to wait for the analysis result, consider running the analysis via adding "page-load-request" labels instead.

Retrieve the existing page load results

Retrieve any previously generated page load results for a session.

Example

curl -H "Authorization: Bearer <your_api_token>" https://api-dev.headspin.io/v0/sessions/analysis/pageloadtime/b0d36a38-a296-11e9-8169-f21898a483e52 

Response

Example response for a successful request with a valid session:

{
    "session_id": "b0d36a38-a296-11e9-8169-f21898a483e52",
    "page_load_regions":
    [
        {
            "request_label_id": "83b0ff91-7480-11ea-aa54-9cb6d08b256d",
            "request_label_name": "Page A to B",
            "request_start_time": 20000,
            "request_end_time": 30000,
            "request_ts_start": 1585227174.565,
            "request_ts_end": 1585227184.565,
            "start_sensitivity": 0.835,
            "end_sensitivity": 0.99,
            "result_label_id": "632e8f8e-72e7-11ea-aa54-9cb6d08b256d",
            "start_time": 20280,
            "end_time": 24000,
            "ts_start": 1585227174.845,
            "ts_end": 1585227178.565
        }
    ],
    "units": "milliseconds since session start for start_time and end_time, seconds since epoch for ts_start and ts_end"
}

Add a Session Tag

Add the tag <code class="dcode">session type: first load</code> to first load test sessions to measure the app launch time. Although designed specifically for first load tests, this tag is simply a convenient way of calling the analysis to run on the entire session.

This tag can be added via the Session Tag API or in the Waterfall UI.

Example: Session Tag API

curl -X POST -H "Authorization: Bearer <your_api_token>" https://api-dev.headspin.io/v0/sessions/tags/{session_id} -d '[{"session type": "first load"}]' 

Example: Waterfall UI

Click on the <code class="dcode">Add New Tag</code> button on the right panel in the Waterfall UI and enter in "session type" as <code class="dcode">Key</code> and "first load" as <code class="dcode">Value</code>.

Low Page Content

Low Page Content highlights time intervals in a session where little to no visible content is displayed on the screen. Such regions are often associated with loading or transition pages before the page fully loads. Blank screens, simple splash screens, and partially loaded screens would be highlighted as Low Page Content.

We use edge detection to measure the amount of visible content available on the screen. Once text and images load on the screen, more edges are detected. This maps to more visible content on the screen, which then ends the Low Page Content region.

This analysis is automatically run in every HeadSpin session.

Loading Animation

Loading Animation highlights time intervals in a session (as HeadSpin issues) where preloading or buffering animations are displayed on the screen. These animations are unambiguous signals that content is either fully or partially unavailable to the end user. Shown below are a few examples of the wildly different loading animations detected by our analysis:

In order to detect a wide variety of loading animations, we've trained a model on a myriad of sessions to recognize traits that a diverse set of loading animations may share. The model has learned to assign high probabilities (of a loading animation existing) to regions where repeated localized movements occur.

This analysis is automatically run in every HeadSpin session, but if you're not getting the desired results, we suggest that you try running the analysis by creating a loading animation request label, either in the Waterfall UI or via the Session Annotation API. If you're creating loading animation request labels, please note that there must not be any overlapping time intervals between the labels. If you delete a loading animation request label, any Loading Animation Issues associated with it will also be deleted.

Create a Loading Animation Request Label in the Waterfall UI

On the timeline, click and drag your mouse across the region on which you would like to run the analysis. This opens the <code class="dcode">Create Label</code> popup with the pre-filled <code class="dcode">Start Time</code> and <code class="dcode">End Time</code> that correspond to the region that you highlighted. From the <code class="dcode">Type</code> drop-down menu, select <code class="dcode">Analysis: Loading Animation Request</code>. Fill out the required <code class="dcode">Name</code> field and then click the <code class="dcode">Submit</code> button to create the annotation. Optionally, you can fill out the <code class="dcode">Data</code> to configure analysis parameters, e.g., <code class="dcode">{"min_radius": 10, "max_radius": 20}</code>. Creating a loading animation request label triggers the analysis to run on the time interval specified by the label, which creates a Loading Animation Issue for every loading animation region found.

Create a Loading Animation Request Label via the Session Annotation API

Add labels on the time intervals of interest with the <code class="dcode">"label_type"</code> specified to <code class="dcode">"loading-animation-request"</code>. For each <code class="dcode">"loading-animation-request"</code> label that you add, you can provide the following optional customizations:

• Specify custom analysis parameters in the <code class="dcode">"data"</code> field. See the section below Configurable Parameters.
• Specify the coordinates of bounding boxes in the <code class="dcode">"video_box"</code> field. See our documentation on applying spatial filtering on video analyses.

Example: Add <code class="dcode">"loading-animation-request"</code> labels

curl -X POST https://api-dev.headspin.io/v0/sessions/{session_id}/label/add -d '{"labels": [{"name": "no customizations", "label_type": "loading-animation-request", "start_time": "10.5", "end_time": "30.0"}, {"name": "customized", "label_type": "loading-animation-request", "start_time": "40.0", "end_time": "60.5", "video_box": [[0, 0, 300, 400], [50, 450, 400, 680]], "data": {"min_radius": 10, "max_radius": 20}}]}' 

Response

{"label_ids": ["85e731d2-65cd-11eb-b3c0-e7adb5682894", "85e731f0-65cd-11eb-b3c0-e7adb5682894"]}

These are the label IDs of the two <code class="dcode">"loading-animation-request"</code> labels that you added. The Loading Animation analysis runs on the time intervals specified by the labels (and for each lable, if <code class="dcode">video_box</code> is specified, the analysis runs only on the bounding boxes). For each loading animation region detected from the analysis, a Loading Animation Issue gets created.

Configurable Parameters

The default Loading Animation analysis is shape- and color-agnostic and relies on small localized movements to detect loading animations. This generalized approach, while great for detecting a wide variety of loading animations, can lead to false positives, especially when video content is playing. To combat this, we provide the option to perform circle detection as another layer of analysis on top of the default Loading Animation analysis in order to filter out the false positives.

Developed primarily for reducing false positives that occur during video play, this circle detection is useful only if all of the following criteria are met:

1. You're seeing false positives from the default Loading Animation analysis.
2. The loading animation you're trying to detect has a circular shape.
3. The loading animation you're trying to detect is light in color (e.g., white) against a dark background.

In order to enable circle detection to run on top of the default Loading Animation analysis, you must provide both of the following parameters:

{
    "min_radius": <integer>,
    "max_radius": <integer>
}

• <code class="dcode">"min_radius"</code>: The minimum radius of the spinner to be detected (in pixels). Must be an integer and smaller than <code class="dcode">max_radius</code>.
• <code class="dcode">"max_radius"</code>: The maximum radius of the spinner to be detected (in pixels). Must be an integer and larger than <code class="dcode">min_radius</code>.

We recommend keeping the difference between <code class="dcode">max_radius</code> and <code class="dcode">min_radius</code> within ~20 pixels. You can get the dimensions of the session video to help with selecting the values for the radius.

Misidentification

Visual changes on the screen that mimic loading animations (e.g., animated logos and flashing icons) can be incorrectly identified by the model. If loading animations appear on dynamic backgrounds, the model may have a hard time delineating exactly where the loading animations occur. When you encounter examples that do not meet your expectations, we recommend that you try running the analysis with custom configurations, especially if the loading animation you're trying to detect meets the criteria laid out in Configurable Parameters. You can also provide feedback by creating feedback labels on such regions so we can continue to improve the model. Instructions and guidelines for providing useful feedback are detailed in Providing Feedback on HeadSpin Performance Sessions.