We have at least two out of our four new CHUBS exhibiting this problem. Anytime they hit the rigging, they reset.
We switched to the IMU in the expansion hub and that works fine, but it doesn’t update data fast enough for something like RoadRunner to work well.
We have older CHUBS that didn’t seem to have this issue. Or at least we didn’t run into it last year at tournaments, state, and worlds…
Danny, is there a particular set of serial numbers or batch number we can look at? We have a tournament this Saturday and could put the effort in to swap CHUBS, but we need to know if this is a manufacturing issue or software issue with the new IMUs.
I apologize profusely for not seeing these posts here. We are aware of the issue, and we’re investigating.
This issue seems to be mostly limited to the new Control Hubs with the BHI260AP IMUs (though REV says it could still happen with the old IMUs though we’ve never seen that in “real life”), if you go to the Management Interface on the Hub and look at the IMUs you can see which one it’s using.
REV is currently investigating the issue, per the Blog posted here:
Seeing the same from our Control Hub purchased a few months ago.
Pretty sure it’s ESD, seems to act up on the cold-est/dry-est day we had so far. IMU resets during auto within the first 2 seconds of motion. Reproduced reliably for an hour+ then intermittently after that. Moved to different section of the field with more human contact, no wall contact and IMU started to behave.
We have grounding resistor, chokes on the USB, no other cabled sensors and the hubs are mounted on wood panels.
This is pretty catastrophic for teams with IMU-based autos. We will reach out to our local competition for Staticide requests but can’t rely on this for Semis/Regionals/States etc.
Please keep us updated on any fix. We’re happy to try somethign experimental.
Anyone know if this is an issue with overly susceptible IMUs or low ESD suppression on I2C bus? There are some I2C compatible, 3-pin ESD suppressors that cold be soldered into a cable and connected to I2C bus0. But without FIRST permission (and REVs review), I think these may not be legal.
Danny, is that something FIRST would entertain allowing if it helped?
Alternatives to an Expansion Hub with the old IMU could be an off-the-shel IMU board. I haven’t tested this but it’s possible that the Adafruit BNO055 board could be connected to I2C port 1 on the Control Hub, added to the robot configuration as a BNO055 IMU and used in Blocks/Java using the built-in driver.
Yep, we have one on the way as well. I don’t see an ESD suppressor on the AdaFruit schematic though. If the ESD issue is with the Control Hub I2C bus itself (is it the new Control Hub design that is resetting the bus?) then adding an external cable without ESD suppression seems like it might not make the issue better.
If it’s really the new IMU version that has ESD issues, then substituting the external BNO055 device would make sense. I’m not sure from the REV blog if the issue is the IMU itself, or the Control Hub I2C bus/interface.
We’ll give it a try and also report back. ESD is so hard to debug… I need a static gun, and a pile of sacrificial Control Hubs.
From what we know at this point, this would not likely help the IMU issue; it appears to reproduce when EMI from an ESD event interferes with an internal circuit. Nothing plugged into the I2C port would have any effect.
We’ll keep the community informed as we get more information from REV.
REV has not yet provided any solutions. I understand that they are testing multiple possible solutions, but they do not have anything that has yet shown enough merit for external testing by users.
An external IMU will avoid using the internal Control Hub IMU, but the biggest question is whether or not external IMUs will be affected by ESD similar to the internal IMU. None of the external IMUs that I am aware of have specific ESD protection circuitry - it’s important to understand that there’s two levels of “protection”, the first level is “ESD will not physically harm the device” which the IMU on the Control Hub has, and the second level is “ESD will not cause the device to behave incorrectly” which very few devices actually have. Most external IMUs that I am aware of have neither specifically in their circuitry, though in their design they have features that help them be more resistant.
With that said, I have never heard reports of the Adafruit IMU featuring the BNO055 (like this one) having ESD problems. It also has software support already built in to the FTC SDK.
You can only purchase one if someone is willing to sell it to you. I wouldn’t hold my breath.
We mounted an external one and it has the same problem. We monitored all three IMUs and the only one that didn’t fail was the one in the old expansion hub. My test consisted of just pushing the bot around by hand on the field and then after a bit, pushing it into the structure. All the angles came back as 0.
Is there a way to poll the IMU directly to find out if it is in a bad state? If it is then we can switch to the expansion hub one or something like that.
In the mean time, what is being done to resolve this issue? Should we reach out to REV directly?
938
01-04 15:50:24.958
Warning
BNO055IMU
(new): getRobotOrientationAsQuaternion(): Failed to retrieve valid quaternion from IMU. Returning the identity quaternion.
939
01-04 15:50:24.972
Warning
BHI260IMU
getRobotOrientationAsQuaternion(): Failed to retrieve valid quaternion from IMU. Returning the identity quaternion.
We’ve had our IMU fail only once (that I’m aware of or observed), but when it failed I seem to recall that orientation.getAcquisitionTime() was returning zero values instead of valid timestamps. That might be a useful way to determine that an IMU has entered a bad state.
Fortunately our IMUs don’t seem to fail often, so I haven’t had adequate time to repeat this test (and tbh, I don’t want to try to force our IMUs to fail). But if others could confirm that acquisition time is a useful surrogate for this, that would be really helpful.
Code snippet:
YawPitchRollAngles orientation = imu.getRobotYawPitchRollAngles();
long imuTime = orientation.getAcquisitionTime();
if (imuTime == 0) {
// imu has perhaps failed...?
}
Quick update from our experimentation. We ran the evening with the garage door open. 55F @ 60% humidity.
Grounding strap on.
Ferrite chokes on the USB cables.
Electronics are isolated on a wooden board.
External IMU is added to Control Hub I2C port 3.
We experienced an ESD event on the Control Hub 10/10 runs (then stopped counting).
We sample the IMUs on the Control Hub, Expansion Hub (old), and external IMU through our entire auto.
The Control Hub IMU returned garbage about 500ms into the auto. Both the Expansion Hub and External IMU reported valid data (even when the CH IMU failed.) This was repeatable 5/5 attempts (until we stopped trying.)
We can confirm that the Orientation class Acquisition Time does start to read ‘0’ with or very near the ESD event.
This is incredibly useful information, our thanks to your team for conducting the experiments and reporting on them.
We’ve ordered an Adafruit external IMU. Our internal CH IMU has seemed to work so far, but we at least want to have something ready as a backup, if not switch to the external sensor outright. (Competition day is not when we want to discover it starts failing.)
We implemented a class that has two IMUs in it: a primary that could be the one in the control hub and a secondary from the expansion hub. It essentially uses orientation.getAcquistionTime() returning 0 (thanks pmichaud) to determine if the primary one has failed. If it determines that such a thing happened, it remembers it and then on returns the orientation etc from the secondary imu.
Don’t know how useful this might be or if it does solve the problem or not but haven’t had any issues after this implementation. There have been cases where the primary IMU does seem to have failed but the robot run was not affected, at least not noticeably.
/**
* Create the primary IMU, initialize it and set the Expansion Hub IMU as its secondary IMU
*/
IMU controlHubIMU = hardwareMap.get(IMU.class, "imu");
IMU.Parameters controlHubParameters = new IMU.Parameters(new RevHubOrientationOnRobot(
RevHubOrientationOnRobot.LogoFacingDirection.LEFT, RevHubOrientationOnRobot.UsbFacingDirection.UP
));
controlHubIMU.initialize(controlHubParameters);
/**
* Create a secondary IMU, the one from the expansion hub. Initialize it.
*/
IMU expansionHubIMU = hardwareMap.get(IMU.class, "imu2");
IMU.Parameters expansionHubParameters = new IMU.Parameters(new RevHubOrientationOnRobot(
RevHubOrientationOnRobot.LogoFacingDirection.RIGHT, RevHubOrientationOnRobot.UsbFacingDirection.UP
));
expansionHubIMU.initialize(expansionHubParameters);
silverTitansIMU = new SilverTitansIMU(controlHubIMU, expansionHubIMU);
/**
* Reset the yaw on the IMU
*/
resetYaw();
We have done a similar thing where we use our externalIMU until it goes bad and then switch to the expansion hub. Both the external and expansion are the old BNO IMUs.
There’s definitely an environmental factor to all of this. Our Control Hub IMU fails rarely when in our normal practice space (a room in my house), but when we go to another team’s practice facility the Control Hub IMU fails 7 of 10 times or more. When we’re back to our normal space it works more reliably again.
To me, a Control Hub IMU that works for months and then starts failing may simply be reacting to the environment, such as changes in the weather and HVAC operation. Since September I’ve watched the humidity in my house go from averaging 50%-60% to being regularly 35% or lower (we keep a temp/humidity sensor in our practice space for precisely this reason).
Similarly, some of the events in our area are noticing increased ESD issues as we get into colder weather – venues that were “just fine” back in October and November are now seeing increased disconnects as things have gotten colder and heating systems are running more constantly.
We’ve switched to an external IMU for the time being, and so far it seems to run well in both environments that we regularly practice, as well as at our League Meet last Saturday. I think that’s really the best bet for now, as well as getting event hosts to treat fields for static.
could well be. Our practice field is in a school that was closed from mid December until mid January when temps were below zero for a week or so. so we had very low humidity in that room
all the ESD best practices are already implemented so I am going to use the IMU in our expansion as an alternate and check for a zero time from the primary control hub imu.
