Occasional crash with Invalid address passed to free: value not allocated

Occassionaly Robot Controller crashes with Invalid address 0x74542d00c0 passed to free: value not allocated. This is on a Rev Control Hub and an Expansion Hub. Seems to happen at different point in match play. Sometimes in init_loop, sometimes in teleop, etc. We have the log but it’s too big to post it here.
Any help would be appreciated.

It would be immensely helpful to post your robot logs and robot code to a Google Drive and link to it here.

-Danny

Thanks Danny.

The log is at https://drive.google.com/file/d/1QUMi81XPTvwk2YdDKaDtU0vu7w_Msmmc/view?usp=drive_link. The crash starts at line 5320.

The code is on github at GitHub - Alokxmathur/CenterStageV9.

Let us know if we need to clarify anything about the code. It is rather complex unfortunately.

That log is… interesting.

It has

Fatal signal 6 (SIGABRT), code -6 in tid 17508 (RenderThread)

But then no backtrace showing the native call stack. I’m not sure if that is because debug symbols are being stripped when the release version of the native executable is generated or ?

Are you able to extract the contents of the /data/tombstones/ directory on the Control Hub using ADB?

Actually, before we go down that rabbit hole…

The “RenderThread” is a bit of a clue. The invalid free is happening on the Android thread responsible for rendering the application UI. I notice you are using a custom VisionProcessor (ObjectDetectionVisionProcessor.java). I believe that the onDrawFrame() function is invoked by the Android RenderThread.

Try this: in SilverTitansVisionPortal.java, where you build the VisionPortal, try NOT adding your objectDetectionVisionProcessor to the portal on line 40.

Can you share the stack trace of the thread causing the invalid free? Is there some tool we can use to analyze the system dump ourselves? You seem to have gotten to the crux of it rather rapidly.

If we don’t add ObjectDetectionVisionProcessor and the crash doesn’t happen, we won’t know where the problem lies. Also, of course, we do want to use our vision processor.

Firstly it’s not a Java crash so there isn’t going to be a stacktrace pointing to your code, only a native backtrace as Android calls it. But we don’t have that either, that’s what I was asking about when I mentioned the tombstone earlier.

I just made an educated guess at what’s probably going on. I am the author of both VisionPortal and EasyOpenCV :slight_smile:

If the crash doesn’t happen without your processor, we at least have an extremely narrowed scope of investigation for where the issue may be.

Can you help us with the VisionProcessor? I think we know where the problem lies. We implement the processFrame(Mat frame, long captureTimeNanos) and return an object which is really a Map of objects found by the processor indexed by the object type. When onDrawFrame is called, we expect the Map to passed in. We work based on this map to paint on the canvas the objects seen and returned in processFrame. I don’t know if the two calls happen on the same thread or not. There is no synchronization between them and thus the issue with free. We’ve change the onDrawFrame to do nothing to see if it removes the problem.

Also, didn’t know we were being helped by essentially Royalty! The author of VisionPortal and EasyOpenCV. Wow!

You have the right idea for how to implement the coordination between processFrame() and onDrawFrame().

Correct, there is no synchronization between processFrame() and onDrawFrame(), and they will be called on different threads. They may also be called concurrently, and moreover there is no guarantee that you will get an onDrawFrame() call for every processFrame() call. If the viewport renderer is lagging behind, it will drop frames to prevent backlogging the actual vision processing. Also, onDrawFrame() itself may be called from multiple threads, one for the local view and one for the remote view on the DS.

Synchronization is not forced between the two functions because it is not needed in all cases. You can always add a mutex if you need it. That being said, the easiest way to deal with it is to just return new object instances from processFrame() each time. Then no explicit synchronization is necessary. This is how the implementation of the AprilTagProcessor is done. The onDrawFrame() function has a mutex just for itself to prevent being called from two threads simultaneously for reasons that I don’t completely remember but I think it was because I was using the same canvasAnnotator instance and I didn’t want to let the scale change mid-draw - though looking at it now I see it probably could have been avoided by passing the draw params directly to the drawOutlineMarker() etc. calls instead of having a noteDrawParams() function.

If you’re thinking to yourself “gee this is a lot more complicated than drawing directly on the image buffer in a standard EOCV pipeline,” you’re not wrong… the complexity stems from the fact that we needed to support multiple processors annotating the same frame while leaving ourselves the future option to run processors in parallel on different threads (although currently they are run in series).

At the time, I was of the opinion that people advanced enough to write their own vision code would just want to keep working with the EasyOpenCV APIs directly, although evidently that doesn’t seem to be the case as you are one of many to be interested in using custom VisionProcessors.

All that being said, even if you have concurrency issues in your processor, I’d usually expect that it would manifest as a Java crash. The free() error is not even something that’s possible to do in Java, it’s only possible in C/C++. The fact that it’s a native crash is very odd, it makes me think perhaps that the crash is happening inside the native code of the Canvas graphics API.

Thanks for the detailed explanation. We do have synchronization in code on the returned object so maybe that is not the problem. As you mentioned, even if we did not have proper synchronization, it should only result in a java issue, not a “free” issue.

Good to know that the processors are not called in parallel, we thought they were. I think it might be best for use to move to a pure EasyOpenCV implementation that finds our objects and the april tags in a pipeline we write.

That’s what I originally anticipated more advanced teams would gravitate towards. I’m attaching a sample from our internal repository that shows using the Tfod VisionProcessor from EOCV, you could use it as a reference for using the AprilTag one.

That being said, we still have not confirmed whether it is in fact an issue with your custom processor or not.

/*
 * Copyright (c) 2023 FIRST
 *
 * All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without modification,
 * are permitted (subject to the limitations in the disclaimer below) provided that
 * the following conditions are met:
 *
 * Redistributions of source code must retain the above copyright notice, this list
 * of conditions and the following disclaimer.
 *
 * Redistributions in binary form must reproduce the above copyright notice, this
 * list of conditions and the following disclaimer in the documentation and/or
 * other materials provided with the distribution.
 *
 * Neither the name of FIRST nor the names of its contributors may be used to
 * endorse or promote products derived from this software without specific prior
 * written permission.
 *
 * NO EXPRESS OR IMPLIED LICENSES TO ANY PARTY'S PATENT RIGHTS ARE GRANTED BY THIS
 * LICENSE. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
 * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
 * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
 * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
 * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
 * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
 * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
 * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
 * TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
 * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */

package org.firstinspires.ftc.robotcontroller.tests;

import android.graphics.Canvas;

import com.qualcomm.robotcore.eventloop.opmode.Disabled;
import com.qualcomm.robotcore.eventloop.opmode.LinearOpMode;
import com.qualcomm.robotcore.eventloop.opmode.TeleOp;

import org.firstinspires.ftc.robotcore.external.hardware.camera.WebcamName;
import org.firstinspires.ftc.robotcore.external.tfod.Recognition;
import org.firstinspires.ftc.robotcore.internal.camera.calibration.CameraCalibration;
import org.firstinspires.ftc.robotcore.internal.camera.calibration.CameraCalibrationHelper;
import org.firstinspires.ftc.robotcore.internal.camera.calibration.CameraCalibrationIdentity;
import org.firstinspires.ftc.vision.tfod.TfodProcessor;
import org.firstinspires.ftc.vision.VisionProcessor;
import org.opencv.core.Mat;
import org.openftc.easyopencv.OpenCvCamera;
import org.openftc.easyopencv.OpenCvCameraFactory;
import org.openftc.easyopencv.OpenCvCameraRotation;
import org.openftc.easyopencv.OpenCvInternalCamera2;
import org.openftc.easyopencv.OpenCvWebcam;
import org.openftc.easyopencv.TimestampedOpenCvPipeline;

@TeleOp
@Disabled
public class TestTfodProcessorWithEocv extends LinearOpMode
{
    boolean USE_WEBCAM = false;
    OpenCvCamera camera;
    TfodProcessor tfProcessor;

    private static final String TFOD_MODEL_ASSET = "CenterStage.tflite";
    private static final String[] TFOD_LABELS ={
            "Pixel"
    };

    @Override
    public void runOpMode()
    {
        tfProcessor = new TfodProcessor.Builder()
            .setModelAssetName(TFOD_MODEL_ASSET)
            .setModelLabels(TFOD_LABELS)
            .setIsModelTensorFlow2(true)
            .setIsModelQuantized(true)
            .setModelInputSize(300)
            .setModelAspectRatio(16.0 / 9.0)
            .build();

        int cameraMonitorViewId = hardwareMap.appContext.getResources().getIdentifier("cameraMonitorViewId", "id", hardwareMap.appContext.getPackageName());

        if (USE_WEBCAM)
        {
            camera = OpenCvCameraFactory.getInstance().createWebcam(hardwareMap.get(WebcamName.class, "Webcam 1"), cameraMonitorViewId);
        }
        else
        {
            camera = OpenCvCameraFactory.getInstance().createInternalCamera2(OpenCvInternalCamera2.CameraDirection.BACK, cameraMonitorViewId);
        }

        camera.setViewportRenderer(OpenCvCamera.ViewportRenderer.NATIVE_VIEW);
        camera.setViewportRenderingPolicy(OpenCvCamera.ViewportRenderingPolicy.OPTIMIZE_VIEW);
        camera.openCameraDeviceAsync(new OpenCvCamera.AsyncCameraOpenListener()
        {
            @Override
            public void onOpened()
            {
                MyPipeline myPipeline = new MyPipeline(tfProcessor);

                if (camera instanceof OpenCvWebcam)
                {
                    myPipeline.noteCalibrationIdentity(((OpenCvWebcam) camera).getCalibrationIdentity());
                }

                camera.startStreaming(640, 480, OpenCvCameraRotation.SENSOR_NATIVE);
                camera.setPipeline(myPipeline);
            }

            @Override
            public void onError(int errorCode)
            {

            }
        });

        waitForStart();

        while (opModeIsActive())
        {
            for (Recognition recognition : tfProcessor.getRecognitions())
            {
                telemetry.addData("Image", "%s (%.0f %% Conf.)", recognition.getLabel(), recognition.getConfidence() * 100 );
            }
        }

        telemetry.update();
    }

    static class MyPipeline extends TimestampedOpenCvPipeline
    {
        private VisionProcessor processor;
        private CameraCalibrationIdentity ident;

        public MyPipeline(VisionProcessor processor)
        {
            this.processor = processor;
        }

        public void noteCalibrationIdentity(CameraCalibrationIdentity ident)
        {
            this.ident = ident;
        }

        @Override
        public void init(Mat firstFrame)
        {
            CameraCalibration calibration = CameraCalibrationHelper.getInstance().getCalibration(ident, firstFrame.width(), firstFrame.height());
            processor.init(firstFrame.width(), firstFrame.height(), calibration);
        }

        @Override
        public Mat processFrame(Mat input, long captureTimeNanos)
        {
            Object drawCtx = processor.processFrame(input, captureTimeNanos);
            requestViewportDrawHook(drawCtx);
            return input;
        }

        @Override
        public void onDrawFrame(Canvas canvas, int onscreenWidth, int onscreenHeight, float scaleBmpPxToCanvasPx, float scaleCanvasDensity, Object userContext)
        {
            processor.onDrawFrame(canvas, onscreenWidth, onscreenHeight, scaleBmpPxToCanvasPx, scaleCanvasDensity, userContext);
        }
    }
}

@Windwoes Returning to this, our team has working pipelines under the old EOCV. We already have it for our team element. They also have an AprilTagJNI detector pipeline, but it appears to find a different pose (I assume 2.0.0 is built into the SDK as well? We haven’t compiled yet, but Android Studio finds all the import files).

So, my question is: Is there a quick way to get the quality FTC pose from the current AprilTagJNI? I know there’s some relatively simple transformation happening, but it’s not clear whether it’s complete or what it is.
Thanks!

I’m not completely clear on what you are asking, but, yes: AprilTag plugin 2.0.0 is included in the SDK.

You can see the transformation that happens here: Extracted-RC/Vision/src/main/java/org/firstinspires/ftc/vision/apriltag/AprilTagProcessorImpl.java at d6468783e0248b92369171d988ce0581e47bae09 · OpenFTC/Extracted-RC · GitHub

Thanks - this is very helpful. I was asking how to implement the new VisionProcessor in a pipeline without having to redo all the VisionProcessor setup w/respect to our Team Element since that pipeline works (using the OpenCVCameraFactory, StartStreaming and Listeners, etc. I thought I saw that posted somewhere, but I can’t find it again. But since our current apriltag pipeline uses AprilTagJNI directly, I suppose we can just do our own transformation.

In a previous post on this thread I posted a sample OpMode showing how to run the TFOD processor from a pipeline. It should be straightforward to adapt for the AprilTag processor.

Thanks - I read that, which is what prompted my question. Our current pipelines are separate class files, and they don’t take a processor as a parameter (though I think we added a different class as a parameter because we directly change the other class inside the pipeline). I guess we can just adjust the parameters as we see fit? Or create the processor inside the pipeline class?

Either of those would work.

Thanks, I think we’ve got it now.