Machine Learning this season? In the future? Hunting for details

[Danneallstar]

Hi, I’m the programmer for the 13590 team in the U.S. I was wondering what machine learning technologies/software is available to use in this Into The Deep Season. I know that this question is coming fairly late into the season; however, I’d like to context my situation by stating that I was operating on my knowledge about the resources we were provided last season to carry over and be available this season.

Recently, I’ve gathered a decent amount of information about the machine learning softwares, that I know of, which were available in previous seasons. I’ve been informed through the unofficial FTC discord that Vuforia simply was no longer available to use in the current SDKs by some removal of the support infrastructure. The following forum post talks about how Tensorflow has also suffered this support infrastructure removal as of the previous season’s official SDK.

Is the Machine Learning Toolchain is closed for the 2024 offseason

Hey Kyle. We had a number of setbacks force our hand to drop support of TensorFlow within the core FTC SDK. The straw that broke the camel’s back, however, was a breaking change in TensorFlow itself that was not compatible with our SDK support infrastructure. Our current (9.x) infrastructure is not compatible with newer TensorFlow toolchains, and so in order to support teams making models with current tools we would have to make significant changes that we’re not able to make at this time. Tens…

This forum post talks about how HuskyLens is a viable alternative in the future for object detection, but it does not talk about its legality in the current season. I’ve also heard of Limelight, but I have yet to educate myself properly on its usage, current legality, or its potential in the future.

A certain part I found important in the forum posted by ddiaz reads - “As we move forward we’ll be constantly evaluating the device landscape looking for better ways to incorporate machine learning, AI, and other similar technologies. But for right now it doesn’t make sense for us to pour resources into technologies that are not directly helping teams (sure, they’re learning a lot about ML and AI, but we need them to learn more than “it’s really difficult to use effectively”).”
This paragraph quite clearly, with a certain perspective, gives off the idea that “it takes a good amount of effort to maintain ML within FTC SDKs and since no one’s using our ML models, we might as well remove them for *** a while*** ( forever).” I know I’m probably being pessimistic but with how these types of things turn out I’m seeking out a clear answer from the source (especially as the more I search the more I see the scarcity of FTC documented/reliable information on this topic.)

So, I was just wondering if machine learning, object detection, and anything related to the sort is available in the current (10.x) SDK version/season? And regardless of the answer, will it be available in the future? If not, why so? If you know about anything remotely of the sort please spill your thoughts; I’m pretty desperate at this point.

Thank you,
Dante

[ddiaz]

Hey Dante,

There are two primary ways to do machine learning in FTC today, and they both require external hardware - a DFRobot HuskyLens (or similar device compliant with rule R703) or a Limelight 3A. If you were to search the Competition Manual for either device we made sure to leave breadcrumbs to indicate that the use of the devices is legal. If you search the documentation of either device for “Object Detection” you will undoubtedly find their instructions on how to use that functionality within their devices.

However, there are also non-Machine Learning alternatives out there if you’re looking for vision processing. If you’re simply looking for vision processing algorithms to look for patterns of colors (such as looking for game pieces this season) I think you would do well looking at the Camera Color Processing APIs that were added to SDK 10.1.

Programming Resources — FIRST Tech Challenge Docs 0.2 documentation

-Danny

[Danneallstar]

Thank you @ddiaz ,

Yes, I am now aware of both of the external hardware systems that you mentioned being legal and the Color Processor API that had been added.

Do you have any knowledge about the future of machine learning and perhaps object detection within FTC?

[ddiaz]

It is unlikely that we will be able to provide a native machine learning solution in the current Control Hub architecture. However, in the latest blog about the new mobile robot controller being introduced in the 2027-2028 FTC season, it was announced that it is a goal to have first-class AI and machine learning capabilities in the new system. Limelight will be bringing their expertise in vision and AI, including machine learning, to the platform.