Etsy has filed a patent for a method to detect and correct imbalances in online experiments. The method involves obtaining user information, generating alternative samplings of devices, and generating a threshold to detect imbalances. Once an imbalance is detected, a corrective action is implemented to mitigate it in the multivariate testing framework. GlobalData’s report on Etsy gives a 360-degree view of the company including its patenting strategy. Buy the report here.
According to GlobalData’s company profile on Etsy, dynamic premium pricing was a key innovation area identified from patents. Etsy's grant share as of September 2023 was 52%. Grant share is based on the ratio of number of grants to total number of patents.
Imbalance detection in online experiments using multivariate testing framework
A recently filed patent (Publication Number: US20230273874A1) describes a method for detecting and mitigating imbalances in a multivariate testing framework. The method involves obtaining user information that indicates which devices are assigned to different versions of a service. Alternative samplings of devices are generated, and an expected set of devices for each version of the service is also generated. A threshold is then generated to detect imbalances using the alternative samplings and the expected sets of devices. The method detects imbalances by comparing the obtained user information with the generated threshold and implements corrective actions to mitigate the detected imbalances in the multivariate testing framework.
The user information used in the method can be obtained from a computer system that provides the different versions of the service to the assigned devices. The alternative samplings of devices assigned to each version of the service can be generated by performing permutations on values representing segments or variants of the user information. Additionally, the alternative samplings can be generated by sampling from a distribution that indicates the numerical range of device types assigned to each version of the service, where the assignment of device types is independent of the types of devices.
The expected sets of devices assigned to each version of the service are generated by determining the number of devices of a specific type that receive each version. These specific types can include devices operating in a specific region, with a specific language preference, on specific hardware, or with specific software.
To detect imbalances, the method generates one or more difference values between the obtained user information and the expected sets of devices. If the cumulative difference value exceeds a threshold difference value generated using the alternative samplings and the expected sets of devices, an imbalance is detected.
The method also includes implementing corrective actions to mitigate the detected imbalances. These actions can include generating notifications for testing devices and transmitting them to halt data processing processes on the testing device. Alternatively, a signal can be generated to assign additional devices to a specific version of the service to reduce the detected imbalance. The additional devices can be of a particular type represented by device or user features associated with the detected imbalance.
The patent also describes a non-transitory computer-readable medium that stores instructions for performing the method and a system comprising processors and machine-readable media that execute the instructions to perform the method.