Better Together Connectivity Quality (BeToCQ) is a new test tool built by Android to test the cross-device connectivity performance that isn't covered by the existing Android tests.
This tool is built on the top of the Nearby Connections API. Under Nearby Connections, it has Android connectivity stack including Bluetooth, Wi-Fi, NFC, and UWB technologies.
BeToCQ is designed to catch connectivity software and hardware performance issues by measuring detailed quality signals including the discovery, connection latency, transfer speed, and overall success rate.
Depending on the device capabilities, the test takes two to six hours to complete.
BeToCQ consists of three parts:
Function test
The function test ensures hardware and software readiness for each radio technology.
Directed test
The directed test measures performance of each wireless medium against expectations. To discover the radio concurrency issue, sometimes multiple mediums are enabled at the same time during the test.
The function and direct tests are the foundational tests running with fixed wireless mediums. This helps isolate the issue to an individual medium and makes the debugging process more straightforward.
Critical user journey (CUJ) test
The CUJ test tests the real use case. Different from function and directed test, the CUJ test can use multiple radios in a more dynamic way. So the debugging is typically more difficult in CUJ tests. That's why CUJ tests run as the last step when the other tests have already passed.
CUJ test are implemented as the test cases defined in the compound_test
directory, and are dynamically configured based on the CUJ requirements. The term compound_test
refers to the fact that it uses multiple radios in a dynamic way.
The test suite currently supports three CUJs: Quick Start, Quick Share, and eSIM transfer. We plan to add more CUJs in later releases.
The exact connectivity performance depends on the device capability. For example, the low-cost 2 GHz-only Wi-Fi device achieves a lower speed than the dual-band Wi-Fi device. On the other hand, the dual-band-simultaneous (DBS) capable device can support a 2G infrastructure-STA connection and 5G device-to-device connection in parallel and thus can support higher device-to-device transfer speed.
As a result, this test suite uses the wireless capabilities of test devices as inputs to customize the test case and set the right performance expectations.
In the directed and CUJ tests, depending on the device capabilities, test cases are defined to cover:
The test cases uses the following naming convention:
ConcurrencyMode_MediumBand_MediumName_StaBand_sta_test
For example, scc_indoor_5g_wfd_sta_test
means:
Similarly, mcc_5g_all_wifi_non_dbs_2g_sta_test
means: outmod betocq_test_suite
Note that some test cases are skipped if they aren't supported by the device capabilities. For example:
scc_indoor_5g_wfd_sta_test
is skipped if the device doesn't support WFD group owner (GO) at the 5G indoor channel.
mcc_5g_all_wifi_non_dbs_2g_sta_test
is skipped for DBS capable devices.
Each test case runs multiple iterations to collect the following stats:
MCC test cases run more iterations than SCC test cases. 5G test cases transfer larger files than 2G test cases.
The test cases execution depends on the device capability, so it's important to fill in the device capabilities section correctly in the test configuration file. We'll discuss this in more detail in the following sections.
Running the test is straightforward, but it can be difficult to get insights out of the test results and determine further action to take.
BetoCQ takes three steps to address this issue:
Simplifies the test report review with the visualized test summary.
Sets the proper performance expectations based on devices capabilities. The test results are compared against the expectations so that there are clear pass/failure signals.
Makes debugging job more straightforward, with the test isolating each failure to a single component. The tool also provides the most likely failure reasons and suggest next steps for debugging and appropriate component owner.
Environment.
We recommend an RF shielding box or room to run the test. At minimum, avoid running tests with many other devices nearby.
Wi-Fi AP and network.
The test AP must be a dual-band capable Wi-Fi AP with two SSIDs (one at 2 GHz and one at 5 GHz) with support for DFS channels. Example of routers that meet the testing requirements include NETGEAR RAX50 AX5400, NETGEAR RAX120 AX6000, and NETGEAR R8000b AC3200. Ideally two APs are required to support all test cases.
Target device.
The target device must run a userdebug image of the latest Android version, for example, Android 14. This is the device that is being validated.
Source device.
Run the suite and pass the quality bar with one source device running a userdebug image of the latest Android version. We recommend a model with known good connectivity performance. Some options are:
Follow these steps to prepare and execute tests and review test results.
Prepare the following materials to be used for the tests.
betocq_test_suite
(Linux and macOS)betocq_test_suite_windows.zip
(Windows only)local_mobly_runner.py
cuj_and_test_config.yml
chmod +x betocq_test_suite chmod +x local_mobly_runner.py
Check and install Python version 3.11 or later:
python3 --version
sudo apt install python3
Or install the latest version from python.org for Windows.
Windows only: Download adb and add its path to the Path
environment variable.
There are three Wi-Fi channels to be tested: 2437, 5180, and 5260. 5260 is a [DFS channel] (https://en.wikipedia.org/wiki/List_of_WLAN_channels).
If there are two dual-band APs, all three Wi-Fi channels can be supported.
Modify the test config file cuj_and_test_config.yml
as follows:
Find device serial numbers:
adb devices -l List of devices attached 17011FDEE0002N device usb:1-1 product:raven model:Pixel_6_Pro R3CN90YNAR device usb:1-2 product:p3sksx model:SM_G998N
In this example, the source device is 17011FDEE0002N and the target device is R3CN90YNAR.
Specify the target and source device serial numbers:
- serial: "17011FDEE0002N" role: "source_device"
- serial: "R3CN90YNAR" role: "target_device"
Specify wifi_ssid
and wifi_password
for each Wi-Fi channel:
wifi_2g_ssid: "NETGEAR62-2G" wifi_2g_password: "yourpassword" wifi_5g_ssid: "NETGEAR62-5G-1" wifi_5g_password: "yourpassword" wifi_dfs_5g_ssid: "ASUS_5G" wifi_dfs_5g_password: "yourpassword"
Where wifi_2g_ssid
is for the channel of 2437, wifi_2g_ssid
is for the channel of 5180 and wifi_dfs_5g_ssid
is for the channel of 5260.
Leave wifi_password
as an empty string ""
if it's an open network.
Split the test into two runs if the required channels can't be supported at the same time:
""
so that the 5G DFS test cases are skipped.Configure device capabilities for both source and target devices.
For example, the following configuration means the device uses Wi-Fi chipset WCN6710, and supports two spatial streams with the maximum PHY rate of 2402 Mbps (2x2, 11AX, 160 MHz) at 5G and 287 Mbps (2x2, 11AX, 20 MHz) at 2G. This device doesn't support STA + WFD concurrency in DBS mode. It doesn't support starting WFD group owner mode at an STA-associated DFS or indoor channel.
wifi_chipset: "wcn6710" # The max number of spatial streams max_num_streams: 2 # The max PHY rate at 5G, in Mbps max_phy_rate_5g_mbps: 2402 # The max PHY rate at 2G, in Mbps max_phy_rate_2g_mbps: 287 # if the device supports 5G Wi-Fi supports_5g: True # if the device supports DBS in STA and Wi-Fi Direct concurrency mode supports_dbs_sta_wfd: False # The max number of spatial streams in DBS mode. max_num_streams_dbs: 1 # if the device supports to start WFD group owner at a STA-associated DFS channel enable_sta_dfs_channel_for_peer_network: False # if the device supports to start WFD group owner at a STA-associated indoor channel enable_sta_indoor_channel_for_peer_network: False
For the last two parameters, review config_wifiEnableStaDfsChannelForPeerNetwork
and config_wifiEnableStaIndoorChannelForPeerNetwork
in the Wi-Fi device overlay file [config.xml
] (https://cs.android.com/android/platform/superproject/main/+/main:packages/modules/Wifi/service/ServiceWifiResources/res/values/config.xml).
Check with the Wi-Fi engineering team about device capabilities details.
To run the test on Linux and macOS, run the following commands from the local directory:
python3 local_mobly_runner.py -p ./betocq_test_suite -tb Quickstart -i --novenv -c cuj_and_test_config.yml
Note that Quickstart
is the CUJ test name and there are a few other supported CUJ tests listed in cuj_and_test_config.yml
.
If there are more than two devices connected to USB ports, specify the device serial number explicitly:
python3 local_mobly_runner.py -p ./betocq_test_suite -tb Quickstart -i --novenv -s <serial1>,<serial2> -c cuj_and_test_config.yml
Note that no space is allowed between two device serial numbers in the above command.
To run the test on Windows:
python3 local_mobly_runner.py -p ./betocq_test_suite_windows.zip -tb Quickstart -i -c cuj_and_test_config.yml
Artifacts are saved in <TestResultDirectory> Test summary saved in <TestResultDirectory>/test_summary.yaml
Where `<TestResultDirectory>` is something like `/tmp/logs/mobly/<CujTestName>/<TestDateTime>`.
Use Result Uploader to upload the artifact folder to Google's result storage service. The latest version of the tool and instructions are provided in the results_uploader.
Click MoblyTest to display the overall test results.
If the test passes, no further action is required.
Click each test case (for example, test_scc_5g_wfd_sta
) to display the status of each iteration under Repeats.
If the test fails, follow the following steps to triage the results:
For each failed test case:
Review the test case details including the transfer medium, concurrency mode, the channel bands of STA, and the transfer medium.
Check if the device capabilities are configured correctly.
Review the failed iterations and reasons. Follow the debugging tips to triage and work with the internal engineering team.
Click the failed (red) iterations to see the timestamp and detailed failure signatures. Here is the list of failure signatures:
Wi-Fi STA connection failure signature:
Failed to connect to SSID
or
Failed to remove networks
Discovery failure signature:
Timed out after waiting 30.0s for event "onEndpointFound" triggered by startDiscovery
BT connection failure signature:
com.google.android.gms.common.api.ApiException: 8007: STATUS_RADIO_ERROR at com.google.android.nearby.mobly.snippet.connection.ConnectionsClientSnippet.requestConnection(
Wi-Fi medium upgrade failure signature:
Timed out after waiting 25.0s for event "onBandwidthChanged" triggered by requestConnection
Transfer failure signature:
Timed out after 110.0s waiting for an "onPayloadTransferUpdate" event
Failure signature due to the GMS updates, which repeats a few times before the test exits:
test_log.INFO: In send_rpc_request No response from server. Check the device logcat for crashes.
logcat and bug report: stop com.google.android.gms due to installPackage
Review the logcat and bug report of each failing iteration on both source and target sides. You can find them as boxed links under the test name.
Search the following keywords for the related logs in the bug report: WifiP2pService
, wpa_supplicant
, NearbyConnections
, and NearbyMediums
.
Review the Wi-Fi Direct logs in the bug report if the WIFI_DIRECT
medium is used. Check if it's a group owner or client side error when bandwidth upgrade fails.
DUMP OF SERVICE wifip2p: WifiP2pMetrics: mConnectionEvents: connectionType=FAST, groupRole=CLIENT, freq=5745, sta freq=2437, connectivityLevelFailureCode=NONE
Check the above STA frequency and P2P frequency values. If both have valid values but the values are different, the device likely operates in multiple channel concurrency (MCC) mode unless it supports 2G + 5G concurrency. In MCC mode, firmware could have the bugs resulting in bandwidth upgrade failure or transfer issues. Check with the Wi-Fi chip vendor if there are any known bug fixes for MCC mode.
To rule out the test setup issue or device issue, repeat the test with a pair of known good devices (or a pair of new devices).
If the issue can't be resolved by the internal engineering team and there is strong evidence that there might be an issue on the Google side, create an issue for Google. Be sure to include all test artifacts.
Skip this unless you want to release the test binary from AOSP.
Get AOSP codes from AOSP.
Build the test binary for Linux and macOS:
source build/envsetup.sh lunch aosp_arm-trunk_staging-eng make betocq_test_suite outmod betocq_test_suite
Upload these files to a shared drive:
tools/test/mobly_extensions/scripts/local_mobly_runner.py out/host/linux-x86/nativetest64/betocq_test_suite/betocq_test_suite out/host/linux-x86/nativetest64/betocq_test_suite/cuj_and_test_config.yml
Generate the zip file for Windows execution:
mkdir ~/betocq_windows cp platform_testing/tests/bettertogether/betocq/betocq_test_suite.py ~/betocq_windows/__main__.py echo mobly > ~/betocq_windows/requirements.txt cp -r platform_testing/tests/bettertogether/betocq ~/betocq_windows cp out/host/linux-x86/nativetest64/betocq_test_suite/*.apk ~/betocq_windows cd ~/betocq_windows zip -r ~/betocq_test_suite_windows.zip ./
Upload these files to a shared drive:
tools/test/mobly_extensions/scripts/local_mobly_runner.py out/host/linux-x86/nativetest64/betocq_test_suite/cuj_and_test_config.yml ~/betocq_test_suite_windows.zip