Introduction
The report details the background and resolution to an issue affecting a subset of Android users that started 2 March 2024.
Issue Summary
On 2nd March 2024, the wildcard TLS security certificate for the quoox.com domain was automatically regenerated. This is a routine process that occurs roughly every 60 days, and has done so since Quoox launched. This process ensures that all data transmitted by the Quoox system remains encrypted and secure.
This particular certificate issue used a new “root certificate”. Whilst this is nothing out of the ordinary, in this instance, it appears to have caused an issue with a subset of Android users. It appears that some Android devices have not had the root certificate installed. This normally occurs as part of a regular Android system update and is a general security process unrelated to Quoox.
Because the affected devices could not validate the certificate chain, they deemed the Quoox servers “untrusted”. Consequently, they blocked the quoox.com domain, meaning the Quoox mobile app could not pull the needed data. This manifested in i) members being logged out of the app, then ii) the login dropdown box being empty as the app couldn’t fetch a customer list.
We are still trying to determine what the affected devices had in common, as it was only a subset of Android devices.
Timeline and Resolution
The Quoox support team operates Monday-Friday. The Quoox systems are monitored minute-by-minute by multiple systems around the globe, flagging to the out-of-hours technical team if there is any outage. The Quoox systems were fully functional throughout and thus, the monitors did not flag any issues.
Once we became aware of the issue, we immediately started exploring. First reports were that “all members were affected”, but this rapidly became clear as not being the case. We were able to ascertain it was a subset of Android users. However, our efforts to track down the issue were hampered as all of the devices we could immediately access were working correctly.
We knew that our systems were functioning, and there hadn’t been an app update, so it wasn’t immediately obvious what the cause might be. We explored several theories, which we crossed off one by one.
When we got access to a device exhibiting the problems, we could use it to debug the app and see the “trust failure” issue raised by the underlying Android system. This led to us identifying the issue of the new certificate, which proved valid and correct. Further analysis then led us to identify the device missing the root certificate that should have been installed as part of a regular Android update.
With a limited number of options, the decision was made to make an emergency update to the Android app to “force install” the missing certificate. This was swiftly written, tested on multiple test devices, and then deployed to Google for review, approval and deployment. This happened overnight 3/3/24, and was on full rollout by 6am on 4/3/24.
Summary
Feedback is that the updated Android app (version 1.8.6) has resolved the issue by providing a workaround. Affected members should download and install this version from Google Play.
The FitnessHub site remained fully functional throughout, and we encourage all clients to make sure members are aware of this alternative.
Whilst not directly a Quoox issue, we continue to explore which devices appear to have the certificate missing so that we can better understand and provide feedback through the appropriate channels.
We have also deployed an alternate TLS certificate to our network using a different root certificate.
At this point, we believe the steps we have taken have resolved the issue. We apologise for the inconvenience which, as described, appears to have been a certificate missing from Android system updates on some devices.
