Robotics & Vision
A pan-tilt camera mount that finds a boat in frame and keeps it centred.
BoatTrack is a pan-tilt camera mount that finds a boat in frame and keeps it centred. A Mac handles the vision work, an ESP32-S3 relays commands, and two Feetech STS3215 smart servos move the camera.
Since I didn't have a real boat nearby, I tested with a photo of a sailing boat on an iPad screen, mounted on a stand.
The system has three layers, and each one runs its own loop:
Camera → detect / track → P controller → pan & tilt velocity in [−1, 1]
Integrates velocity into position targets, applies soft limits, runs the watchdog
Internal position PID holds each target
Everything on the Mac side is Python: Ultralytics, OpenCV and NumPy. The firmware is C++ built with PlatformIO.
Phase 1
I began without any hardware. For video I used my iPhone as a webcam through Camo Studio, which provides the RGB feed, and the servo output went to a MockServo that just printed commands. The actuator sits behind a small ServoInterface class, so later swapping in the real serial driver didn't touch anything upstream.
The default detector is YOLO11n filtered to the COCO boat class, running on the Metal GPU through MPS. I also added a YOLOWorld mode. It's an open-vocabulary model, so the target can be described in text, with prompts like “small toy boat”.
Running YOLO on every frame was too slow, so the tracker runs detection every 15 frames and uses OpenCV's CSRT tracker in between. It also re-detects right away when CSRT loses lock.
New detections are matched to existing tracks by centroid distance, which keeps the primary target from jumping to a different boat. You can also click the preview to force a re-detect near a specific point.
One gotcha: CSRT is only included in opencv-contrib-python, not the plain wheel.
The error is the centroid's offset from frame centre, normalised by frame size, so the gains work at any resolution. Frames get downscaled to 960px wide before processing, which was the biggest single improvement to loop rate.
I started with a full PID and a P → D → I tuning pass, but ended up with plain P control plus:
The ESP32 already integrates velocity into position, so the loop has an integrator built in, and the extra terms weren't helping much.
Each run writes a per-frame CSV trace with error, commands, FPS, latency, servo steps and kinematics, and I plotted those while tuning. On my longest logged session (about 23 minutes), the loop's median frame rate was around 30 FPS. Median per-loop processing time was about 12 ms.
Beyond Centring
I also wanted distance and speed estimates for the target, not just a centred image.
This uses the pinhole model:
distance = real_width × focal_px / bbox_widthIt depends on knowing the target's real width and the camera's horizontal FOV, and the estimate drifts when the target rotates or is partially hidden.
A constant-velocity Kalman filter on the centroid gives image-plane velocity plus a short look-ahead position. Tangential speed is distance × angular velocity. Radial speed comes from how fast the distance changes.
I added a FrameSource abstraction so the pipeline could also take LiDAR depth, which the iPhone sends through the Record3D app. With depth available, distance is the median depth inside the bounding box instead of an estimate from box size.
It works over USB and over WiFi via WebRTC. The WiFi mode means one phone in the mount handles everything with no cable. The trade-off is that depth arrives HSV-encoded and compressed, with a range of only 0–3 m.
Phase 2
The firmware runs on a Seeed XIAO ESP32-S3. It reads newline-terminated ASCII commands:
| Command | Purpose |
|---|---|
| V | Velocity |
| C | Centre |
| H | Halt |
| K | Write the servo's PID registers |
| I | Set a servo's ID |
| P | Ping a servo |
| F | Free mode — torque off to position the axes by hand |
The firmware also streams position telemetry back to the Mac. At 100 Hz it integrates the commanded velocity into position targets and clamps them to soft limits. I found those limits by moving each axis to its hard stops in free mode and backing off about 25 steps.
A 500 ms watchdog sets velocity to zero if the Mac stops sending, so a dropped USB link leaves the gimbal holding its position.
I command so I could renumber them one at a time from the serial monitor instead of installing Feetech's tool.--swap-axes, --invert-pan and --invert-tilt are CLI flags.
The full loop works end to end on the bench: camera → YOLO + CSRT → P controller → ESP32 → servos, with a live dashboard showing the feed, a LiDAR view, and distance, speed and servo data. When the target is lost, the gimbal currently just holds still.
The next things I want to add are search/recentre behaviour after a timeout, and feeding the Kalman look-ahead position into the controller to reduce tracking lag. Beyond that is testing on actual boats at real distances, where LiDAR won't reach and monocular estimation has to do the work.