My team has been experiencing similar issues. Our control HUB stops responding, the robot continues the last command for a second or two then stops. We maintain connection to the SSID and can connect to the SSID with the REV hardware client but we cannot see the HUB from the player controller or the REV client.
Sometimes this crash and disconnect happens in thirty seconds and other times it takes 5 or 6 minutes before this happens. We have replaced our control HUB, expansion HUB, a couple cables, tested others, and run the robot with only the four motors for the wheels. The problem persists.
Any help figuring out how to resolve with would be great. We have a state competition this Friday and Saturday.
I’ve been talking with Rowan with REV Tech Support, and he provided me with your crash logs. Unfortunately there’s a lot here, but we’re not able to find any specific smoking guns in your logs (16MB of logs will do that to you). What I would REALLY appreciate is this:
First, make sure the time/date on your Driver Hub (or Driver Station phone) is correct and up to date.
Pick a program that causes the problem. Rename it to “CrashTester” so that it’s easy to find in the logs. When you run that program, we can easily track its progress.
Cause the program to reproduce the issue. Please reproduce the issue up to 3 times, and make a note of the time at which the robot demonstrates the issue. As long as your Driver Hub / Phone has the correct time, we will be able to see the time in the logs. The time doesn’t have to be exact, but close enough so that we can narrow our focus.
Collect your Robot Controller Log and your Wi-Fi Log. You can use pastebin.com to copy-paste the log file contents into a pastebin, click on the “Create New Paste”, then give us the URL to the paste. Do it separately for the two logs.
I can trim the logs down to our runs last night. Basically each time there is a time gap of more than a few minutes, we had a crash. It does appear to stop logging when it crashes as best I could tell from when the crashes happened after I pulled the logs. The time on the Driver Hub and the stamps in the logs are correct. I will trim the logs and upload them to pastebin.com. I will post the links for the uploads shortly.
Here is the pastbin 12-06 19:43:46.110 974 1169 I JniTime : [time.cpp:83] settimeofday() succeede - Pastebin.com with the logs from at least one crash, but I think actually two crashes at about 19:46 and 19:51. We will have a practice tonight and we can create new logs as you requested in the steps above. This is our last chance to fix issues before competition Friday and Saturday.
The Wi-Fi logs were not complete, it seems to only have logged a short period. When I have looked at those before they were the same as this really:
--------- beginning of main
--------- beginning of system
12-06 19:26:09.303 1075 1075 I hostapd : wlan0: STA 18:47:3d:a6:33:87 IEEE 802.11: associated
12-06 19:26:09.303 1075 1075 I hostapd : wlan0: STA 18:47:3d:a6:33:87 IEEE 802.11: associated
12-06 19:26:09.358 1075 1075 I hostapd : wlan0: AP-STA-CONNECTED 18:47:3d:a6:33:87
12-06 19:26:09.359 1075 1075 I hostapd : wlan0: STA 18:47:3d:a6:33:87 WPA: pairwise key handshake completed (RSN)
12-06 19:26:09.359 1075 1075 I hostapd : wlan0: STA 18:47:3d:a6:33:87 WPA: pairwise key handshake completed (RSN)
12-06 19:26:10.340 948 976 I FtcAccessPointService: HostapdMonitor: Received hostapd message: <3>AP-STA-CONNECTED 18:47:3d:a6:33:87
12-06 19:26:12.146 1133 1133 I dnsmasq : DHCPDISCOVER(wlan0) 18:47:3d:a6:33:87
12-06 19:26:12.146 1133 1133 I dnsmasq : DHCPOFFER(wlan0) 192.168.43.28 18:47:3d:a6:33:87
12-06 19:26:12.162 1133 1133 I dnsmasq : DHCPREQUEST(wlan0) 192.168.43.28 18:47:3d:a6:33:87
12-06 19:26:12.162 1133 1133 I dnsmasq : DHCPACK(wlan0) 192.168.43.28 18:47:3d:a6:33:87 Tempest-Laptop
So there are just a couple things I want to point out in this log:
Is your Text-to-Speech necessary? Can you disable those in the program for the purposes of debugging? They’re putting a fair amount of “I wonder…” into the logs, and I’d love to just remove that. The exceptions that the Text-to-Speech is providing are likely nothing, but I’d love to just make sure.
Are you using the TEST_TENSORFLOW.blk file? It appears to be corrupt, and is throwing an exception - again, I think that’s just on program registration with Blocks, but if you’re not using it I’d love for you to delete that file if you can.
Also, is there anything you’re doing when you experience this crash? Like is your robot touching something that might have electronic discharge, if you wiggle the power wires (I assume you’re using a battery extension cable maybe?) does the robot reboot, or is there anything else you can think of that is helping to reproduce this faster?
I am not sure where that text to speech is coming from, we don’t have it in the program. Could that be a default featured enabled on the player control HUB?
I think we are or were using the test_tensorflow.blk, but we will definitely remove it and if needed replace with a valid file.
We haven’t really found a pattern for what we are doing with the robot when this happens. While we can’t rule out discharge, it doesn’t seem likely based on our observations. One of our last tests the disconnect happened while we were driving forward not touching anything and only have the 4 wheel motors connected.
We have rechecked our connections a bunch of times. We even bypassed the power switch and extension cables and plugged the battery directly into the control HUB.
It has been very confusing. No consistent cause that we can point to.
One last question - what SDK version are you using? Can you upgrade to 8.1.1? Since you’re a Blocks user, it shouldn’t have any ill effect on your performance. You also have Firmware 1.1.2 on your Control Hub?
We are up to date on the OS and firmware. We removed the corrupt test_tensorflow blocks. Last night we ran the robot for a bit. We had two crashes. After the second crash we ran for a bit with no crashes.
Are you (or someone else) holding the Driver Station device while you’re running your program? I’m just trying to make sense of what I’m seeing.
In your first “crash”, I see normal running and normal termination of the program after 6 seconds of operation (unfortunately your Wi-Fi log looks incomplete, perhaps you only pasted what you felt was necessary?). So I’m trying to figure out why all of your logs look like everything is happening “normally”, and then it hit me. The DS App will force termination of a program under these circumstances:
You press the Stop button while the program is running.
You tap the OpMode selection menu while the program is running.
You leave the main DS screen while the program is running.
We will even disconnect from the robot under the following scenarios:
You swap to another app, or swap the view to the main OS window (pressing the “home” button, et. al.)
You close the app at any time.
Most of your logs tend to show me that something like this is happening in all of these cases. I remember when my team had phones and students held the phones in their hand, they would often unknowingly press on the screen in the wrong places, or even in our phone holders the holder would be too tight and would cause intermittent presses to occur. Is this something that could be happening here?
In your second log, I can see your Webcam perform a surprise removal and insertion during the match, that’s either a loose USB connection or perhaps the system got hit by static electricity (the robot touched something that was charged, like a field wall, game piece, pole, etc…).
The kids are not holding the driver station while running the robot. The driver station is in a mount on a board and no part of it is touching the screen or any buttons.
This is more than just a disconnect. The robot is locking up. After the second crash I tried to connect to it by USB and there was not response. I think the crash is preventing logs from being saved. For the first crash we were running it for more than 6 seconds. The match log is missing the time for of the crash completely.
For Wi-Fi logs, I did delete some old data from a couple weeks ago that was in the log to reduce the size some. I had cleared the logs on the robot but some how old data was added to logs. Some of the parts of the logs were also out of order which is really odd. One of the logs after clearing the old ones just didn’t capture very much at all.
The USB for the camera is pretty secure. However, the robot does bump into things which might cause static discharge.
Oh, another note, we are not using any speech to text anywhere. We asked another team if they knew anything about that process. They also have that all over the logs but are not using anything related.
Sure, I can gather that. I may take a couple of days as I do not have access to the robot right now. We were at the SE MI State Championship this weekend so I was not able to do this or respond sooner.
The good news is that your code doesn’t cause any problems on my setup. The bad news is that your code doesn’t cause any problems on my setup. On code inspection everything appears fine, and when running it on my testbed everything runs fine as well.
The odd thing from your logs is that when your OpModes are running, they’re being exited “normally”, meaning it appears that you’re doing something normal (mentioned above) to disconnect the robot. If you say the robot is crashing, I just don’t see that in the logs - there’s no “sudden shutdown” that I can see. I really wish I could determine what’s going on, but from the looks of things I cannot find anything in the logs that indicates problems.
Can you possibly give me some idea about how the hardware is connected? Do you have just a Control Hub, how is it mounted to the robot, do you have a resistive grounding strap, where is it connected, etc…?
From the timing of the OpModes logs and the crashes, I don’t think the logs were written at all for the time of the crashes. For the first set of logs, it didn’t look like the robot didn’t create and record the logs at all until we powered it off and back on.
The control HUB and one expansion HUB are mounted in the middle of the robot. The control hub is mounted to long barrel nuts which are mounted to the expansion HUB directly under it. The expansion HUB is mount to an acrylic plate in the cent of the robot.
We do have a grounding wire on the robot. I don’t know how to best explain the connection and positioning.
This issue is perplexing. We were fine for a couple weeks, then the problem came back for about a week. The day we were trying to pull logs, we got it to crash/lock up twice pretty quickly. Then it ran with out issue for about 45 minutes that night. It ran fine for next two days at the state champion ship for 11 or 12 games plus about as many practice runs.
We hadn’t changed anything. I don’t know where to go from here. We will be starting some off season activities in a couple weeks and we hope to figure this out so we can have a less frustrating season next year. I can get some pictures of how the robot is assembled and the grounding wire connection.