Since the beginning of the ftc-ml project, our goal has been to give teams the best overall experience possible - and that included training TensorFlow models using Tensor Processing Units (TPUs) for optimal training performance. Training a TensorFlow model with TPU resources (versus GPU or CPU resources) significantly reduces the time necessary to train models within the Google Cloud Infrastructure. Google, a long-time FIRST Strategic Partner, has been a strong supporter of this project along the way and has provided many of the resources necessary for this project to run within the Google Cloud.
With the increasing use of Machine Learning and Artificial Intelligence, it has come to our attention very recently that the global demand for TPU resources has significantly outstripped global capacity. In every one of Google’s worldwide datacenters TPU resources are at or near capacity around the clock, making it impossible for us to provide a consistent experience for teams using TPU resources.
Therefore, we have temporarily disabled TensorFlow model training while we are developing solutions to use Google Cloud GPU resources to train TensorFlow models. Teams will still be able to read documentation, create/upload videos, label videos, prepare training datasets, and explore the existing TensorFlow models using their new-found knowledge. Please understand that documentation will be updated once GPU training has been fully implemented and tested, and another announcement will be posted once this happens.
At this time there is no guaranteed target date for reenabling model training, but we appreciate your patience and your continued support as we work as hard as possible to implement and test this major change to our core product.
-Danny
__________________
Danny Diaz (he/him)
FTC Senior Engineering Manager ,FIRST ®