Our FTC team has issues with ML model, after using the latest model it crashes more often. May be the model is too big for the memory. Before everygame we restart the controlhub and driver hub to work. Is it possible to add more training time to create a new model?
Which model in Team 15089’s FTC-ML account crashes when it’s loaded?
Just FYI, a few things to note:
-
You have a massive dataset for an incredibly simple object. There’s virtually no reason to have more than 1,000 (maximum, ever) images for an object, and that should really be done in less than 300 images. With an object like yours that is virtually the same from every angle, it should be even less. There’s no benefit having hundreds of images of the same object with virtually no difference in background, pose (orientation) or lighting. In your case, with an object that is symmetrical, the vast majority of your images are duplicated. That provides the model with unwanted confirmation bias.
-
Do not train on objects that are illegal. If the model trains on an object, it expects the object within reason how you trained it. The green in the cup is not legal, but the model absolutely can “trigger” off the pattern that includes an illegal section.
-
Having combinations of clear and blurry images rarely train well. I have to remove overly blurry images from my datasets, otherwise the training metrics show incredibly poor training. I also have to be EXTREMELY conscious about having good data to provide to the model, blurry images cause big problems. Some people have told me, “Oh, but if I give it blurry images it’ll be better because it’s seeing the worse-case, right?” and the answer is “absolutely not.” This is an extreme example, but an example nonetheless.
-
3,000 steps is the gold standard. Don’t exceed 3,000 steps - if your math tells you that you can’t do 100 epochs in less than 3,000 steps (and you have 1-2 labels), you have too many images (too many duplicates). I also recommend 3,000 steps regardless - only in cases where you have less than 100 images per label (and the label is of a single “object” that is very symmetric or the same) have I seen it over-train a model.
-Danny