PhotogrammetricWAAM

Open-source Photogrammetry-driven WAAM (Wire-Arc Additive Manufacturing) framework.

This mdBook is generated from the same Markdown source as the Astro / Starlight edition.

telemetry_bridge__py

ROS 2 node that receives KUKA robot telemetry over TCP and publishes to ROS topics.

Overview

Listens for incoming TCP connections on port 60002 (configurable), receives 128-byte telemetry packets from KUKA robot, and publishes decoded data to standard ROS message types.

Published Topics

• /kuka/pose (geometry_msgs/PoseStamped) - Cartesian position and orientation (quaternion)
• /kuka/velocity (geometry_msgs/TwistStamped) - Calculated Cartesian velocity
• /kuka/velocity_cartesian (std_msgs/Float32) - Velocity magnitude in mm/s
• /kuka/joint_states (sensor_msgs/JointState) - Joint angles for axes A1-A6
• /kuka/sequence_number (std_msgs/Int32) - Telemetry packet sequence number
• /kuka/queue_size (std_msgs/Int32) - Command queue size on robot

Usage

ros2 run telemetry_bridge__py ROS_telemetry_server

Parameters

• port (default: 60002) - TCP port to listen on
• frame_id (default: "kuka_base") - Frame ID for published messages

Example with custom parameters

ros2 run telemetry_bridge__py ROS_telemetry_server --ros-args -p port:=60003 -p frame_id:=robot_base

Telemetry Packet Format (output from KUKA - consumed here)

128-byte binary packet:

• Bytes 0-23: Cartesian position (X, Y, Z in mm) and orientation (A, B, C in degrees)
• Bytes 24-47: Joint angles A1-A6 (degrees)
• Bytes 48-63: Sequence number, in-flight count, queue size, flags
• Bytes 64-75: Actual velocity, active tool/base indices

Requirements

• ROS 2 Jazzy
• Python 3
• Standard ROS message packages: geometry_msgs, sensor_msgs, std_msgs

License

Apache-2.0

Parametric Pose Array Publisher

USAGE:

the publisher ..

ros2 run ScanPlanPkg ScanPlanNode

a node to change params:

ros2 run ScanPlanPkg fastparam

what is it?

the parametric pose array publisher receives the following parameters and runs it's algorithm to generate a PoseArray message responsively (event driven) to parameter changes.

Params:

• grid_size (i.e. 1 = a grid of 1x1, 2 = a grid of 2x2, etc.) .. default: 3
• grid_spacing (i.e. 0.5 = 0.5m between each grid point) .. default: 0.5

Output:

• PoseArray message

Topic:

• /parametric_pose_array

WAAM Path Weave Modifier

Given a G-code toolpath in the XY plane, superimposes a continuous sine-wave weave along the path, parameterized by amplitude and wavelength.

The input path "threads" the centre of the produced output sine-wave path; the weave is always perpendicular to the instantaneous heading.

Usage

python weave_modifier.py \
    --input_gcode <input_gcode_file> \
    [--output_gcode <output_gcode_file>] \
    [--output_plot <output_plot_file>] \
    [--amplitude <mm>] \
    [--wavelength <mm>] \
    [--first_layer_amplitude <mm>] \
    [--first_layer_wavelength <mm>] \
    [--resolution <mm>]

Arguments

• --input_gcode: Path to input G-code file (required)
• --output_gcode: Path to output G-code file (auto-inferred beside input if omitted)
• --output_plot: Path to output plot image file (auto-inferred beside input if omitted)
• --amplitude: Weave amplitude in mm (default: 2.0)
• --wavelength: Weave wavelength in mm (default: 3.0)
• --first_layer_amplitude: Override amplitude for the first layer in mm (defaults to --amplitude)
• --first_layer_wavelength: Override wavelength for the first layer in mm (defaults to --wavelength)
• --resolution: Resampling resolution in mm (default: 0.1)

When --output_gcode / --output_plot are omitted they are placed beside the input file as:

{stem}_amp{A}mm_wave{W}mm.gcode
{stem}_comparison_amp{A}mm_wave{W}mm.png

Example

python weave_modifier.py \
    --input_gcode input_data/test_path.gcode \
    --amplitude 3.0 \
    --wavelength 1.5

Expected output

On a successful run the tool prints a per-path summary, the output G-code path, and the comparison plot path. Example for a multi-layer slicer input invoked with --amplitude 3 --wavelength 1.5:

Parsing  SANDBOX/v2_whispers_walnut.gcode ...
  Found 28 extrusion path(s)

  Path 0:  1842 pts,  length =   3421.7 mm  (amp=3.0, wlen=1.5)
           resampled -> 34218 pts,  weaved length = 7214.9 mm
  Path 1:  1517 pts,  length =   2874.2 mm  (amp=3.0, wlen=1.5)
           resampled -> 28743 pts,  weaved length = 6056.3 mm
  ...
  Path 27:  1433 pts, length =   2711.0 mm  (amp=3.0, wlen=1.5)
           resampled -> 27110 pts,  weaved length = 5713.8 mm

Writing G-code ...
  Wrote SANDBOX/v2_whispers_walnut_amp3mm_wave1.5mm.gcode
Generating plot ...
  Saved SANDBOX/v2_whispers_walnut_comparison_amp3mm_wave1.5mm.png

Done.

Things worth noting in the output:

• Found N extrusion path(s) — every contiguous run of G1 moves with E != 0 is counted as one path. Non-extrusion lines (travels, comments, M-codes) are preserved verbatim between paths.
• Path i: <pts> pts, length = <mm> — the input polyline for that path.
• resampled -> <pts> — after arc-length resampling at --resolution (default 0.1 mm).
• weaved length = <mm> — the arc length of the weaved output for that path. It will always be longer than the input (by roughly a factor set by amplitude/wavelength), which is why proportional E-distribution is used when emitting G-code.
• Final two lines name the two artefacts written: the .gcode (weaved toolpath) and the .png (2-panel comparison plot described below).

Plot explainer

The comparison plot is a single PNG with two side-by-side panels:

• Left — Full Path Overview. Every extrusion path in the file, drawn at true scale and equal aspect. Input paths are steelblue, weaved paths are crimson. Layer alpha increases slightly with layer index so stacked geometry is visually separable. At typical WAAM scales (wavelengths of 1–3 mm) the weave appears as a dense red "band" enveloping the blue centreline — use this panel to confirm extent, coverage, and that no layer was dropped.
• Right — Detail (Layer 1). A zoomed window (±18 mm) centred ~25 % along the first extrusion path. This is where you visually verify the weave itself: continuous sinusoid, perpendicular-to-heading, threading the input path through its centre, with the expected amplitude (peak transverse offset, half of peak-to-peak) and wavelength (one full period along arc length). The blue centreline should always pass through every zero crossing of the red curve; lobes should be symmetric about it.

If the detail panel shows discontinuities, asymmetric lobes, or amplitude/wavelength that don't match the CLI arguments, the usual culprit is a --resolution value too coarse relative to --wavelength (rule of thumb: resolution ≤ wavelength / 20).

spec

Given a path in the XY plane (Z constant) continuously assess the instantaneous vector heading of that path.

Create a new path parametrically as a function of:

• the input path
• an amplitude value
• a wavelength value

given the amplitude and wavelength modulate a sine wave which is continuous (never any instantaneous discontinuity) and follows the input path (the input path should "thread" the center of the produced output sine wave path)

the function returns:

• output path

validate

input_path: input_data/test_path.gcode amplitude: 2mm wavelength: 3mm

;;; save result path to output_data also save a render of a 2D plot comparing the input path and the output path.

aggregate_error_stackup

-- Multi-layer error aggregation and trend visualization.
aggregate_error_stackup :: StateDir -> (AggregateHTML, SummaryCSV)
-- Reads all L{N}.csv files from the print loop state directory,
-- stacks them into a unified 3D Plotly scene at their physical Z heights,
-- and charts error statistics over time.

Post-print analysis tool that aggregates per-layer surface-height error data from the WAAM print loop into a single interactive visualization. The 3D "pancake stackup" shows every layer's robot path alongside its error surface at the correct Z height, while a 2D trend chart below confirms that the proportional controller is reducing error over time.

Usage

# from ros2_ws/lib/control/aggregate_error_stackup/
uv run python aggregate_error_stackup.py /path/to/STATE/ERR/job_dir/

This reads all L*.csv files in the directory and produces:

• stackup.html — interactive 3D + 2D Plotly visualization
• stackup_summary.csv — per-layer error statistics table

Serve for remote viewing

The output HTML is self-contained. To view from another machine on the same Tailscale / LAN:

cd /path/to/STATE/ERR/job_dir/
python3 -m http.server 8080 --bind 0.0.0.0
# browse to http://<robot-host>:8080/stackup.html

Options

positional arguments:
  state_dir             Directory containing L*.csv error maps

options:
  -o, --output          Output HTML path (default: <state_dir>/stackup.html)
  --summary             Output summary CSV (default: <state_dir>/stackup_summary.csv)
  --title               Custom plot title
  --layers              Comma-separated layer subset (e.g. 19,20,21,22)
  --no-trend            Omit the 2D error-trend subplot

Examples

# Full stackup of all layers
uv run python aggregate_error_stackup.py \
    /home/r76/PhotogrammetryWAAM/STATE/ERR/mayorvase_LH2.1_40x_-20y/

# Only layers 40-60 with custom title
uv run python aggregate_error_stackup.py \
    /home/r76/PhotogrammetryWAAM/STATE/ERR/mayorvase_LH2.1_40x_-20y/ \
    --layers 40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60 \
    --title "Mayor Vase — Layers 40–60"

# Stackup without trend chart
uv run python aggregate_error_stackup.py \
    ~/PhotogrammetryWAAM/STATE/ERR/mayorvase_LH2.1_40x_-20y/ \
    --no-trend

What it shows

3D Stackup (top)

Every layer is plotted at its physical Z height. Two traces per layer:

Layers are toggle-able via the legend. "Show All" / "Hide All" buttons allow quick comparison.

Error Trend (bottom)

Per-layer statistics plotted against layer number:

A downward slope confirms the proportional controller is converging.

Integration with the print loop

This tool is designed as Step 4 — a post-hoc analysis that runs after layers have been printed. It reads the same state_dir that the print_loop orchestrator writes to:

print_loop (Steps 1-3)          aggregate_error_stackup (Step 4)
        │                                     │
        ▼                                     ▼
  state_dir/                            state_dir/
  ├── L19.csv                           ├── stackup.html        ← NEW
  ├── L19.html                          ├── stackup_summary.csv ← NEW
  ├── L20.csv                           ├── L19.csv
  ├── L20.html                          ├── L19.html
  ├── ...                               ├── ...
  └── mayor_1_state.json                └── mayor_1_state.json

Run it at any point during or after a print job to see accumulated progress.

Files

Full specification (../spec/)

aggregate_error_stackup — SPEC

Purpose

Aggregate per-layer surface-height error CSVs from the print loop's state_dir into a unified 3D "pancake stackup" visualization. The plot reveals how the proportional controller's corrections reduce error over successive layers.

Domain context

Each layer of a WAAM print produces a CSV (L{N}.csv) with columns X, Y, Z, err_mm mapping every robot pose sample to its measured surface-height error. Individually these are viewed as single-layer 3D plots (robot path vs error surface). This module stacks all available layers into one interactive Plotly scene so the operator can visually confirm that error magnitudes diminish as the build progresses — i.e., the proportional controller is converging.

Type signatures

type LayerCSV       = FilePath   -- state_dir/L{N}.csv with columns X, Y, Z, err_mm
type AggregateHTML  = FilePath   -- interactive Plotly 3D stackup
type SummaryCSV     = FilePath   -- per-layer error statistics

-- Core pipeline
discover_layers   :: StateDir -> [(LayerNumber, LayerCSV)]
load_layer        :: LayerCSV -> DataFrame  -- X, Y, Z, err_mm
compute_stats     :: DataFrame -> LayerStats  -- mean_abs, rms, max_abs, n_samples
build_stackup     :: [(LayerNumber, DataFrame)] -> PlotlyFigure
build_trend       :: [LayerStats] -> PlotlyFigure

aggregate_error_stackup :: StateDir -> (AggregateHTML, SummaryCSV)

Inputs

Per-layer error CSVs

Files L{N}.csv in the state directory produced by map_surface_err_to_xyz_pos.py (Step 2 of the print loop).

X,Y,Z,err_mm
12.2021,47.3637,43.2479,4.0644
10.7972,47.6810,43.2649,4.0644
...

Discovery: glob state_dir/L*.csv, extract layer number N from filename.

Optional: job state JSON

{state_dir}/{job_name}_state.json — if present, used to extract nominal Z heights per layer. Otherwise, Z is inferred from the CSV data (median Z per layer).

Outputs

1. Aggregate 3D HTML (stackup.html)

A Plotly Scatter3d figure with two traces per layer:

All layers rendered simultaneously in a single 3D scene. The Z axis naturally separates layers by their physical height. A dropdown/slider allows toggling individual layers on/off.

Layout:

• Title: "WAAM Aggregate Error Stackup — {state_dir.name}"
• Scene: aspect mode "data", axis labels X/Y/Z in mm
• Legend groups by layer number

2. Error trend subplot (embedded in same HTML)

A secondary 2D subplot (below the 3D scene) charting per-layer error statistics vs layer number:

X-axis: layer number. Y-axis: error magnitude (mm). If the controller is working, these curves should trend downward.

3. Summary CSV (stackup_summary.csv)

layer,nominal_z,n_samples,mean_abs_err,rms_err,max_abs_err,std_err
19,63.8,1842,0.312,0.418,1.204,0.278
20,65.9,1910,0.287,0.391,1.103,0.266
...

Algorithm

1. Discover: glob state_dir/L*.csv, sort by layer number
2. For each CSV:
   a. Load into numpy arrays (X, Y, Z, err_mm)
   b. Compute statistics: n, mean_abs, rms, max_abs, std
   c. Build error surface: Z_err = Z + err_mm
3. Build 3D figure:
   a. For each layer, add robot-path trace (X, Y, Z) with layer-indexed color
   b. For each layer, add error-surface trace (X, Y, Z_err) colored by err_mm
   c. Add updatemenus for layer toggle
4. Build 2D trend subplot with error statistics over layer progression
5. Combine into single HTML with subplots
6. Write summary CSV

CLI interface

uv run python aggregate_error_stackup.py <state_dir> [options]

Dependencies

• numpy — array operations and statistics
• plotly — interactive 3D + 2D visualization
• Standard library: csv, pathlib, argparse, re

Constraints

• CSV files must follow L{N}.csv naming convention.
• At least two layers must be present for the trend plot to be meaningful.
• Large layer counts (>50) may require trace simplification (downsampling) for browser performance.

print_loop

-- The closed-loop WAAM print cycle, orchestrated.
print_loop :: PrintJobConfig -> IO ()
-- For each layer N in [start..end]:
--   1. record_bag + execute_gcode   :: GCode -> IO BagDirectory
--   2. map_surface_error             :: BagDirectory -> (HTML, CSV)
--   3. proportional_correction       :: (CSV, GCode_next) -> GCode_corrected
--   [operator inspects HTML, confirms]
--   loop.

Interactive CLI orchestrator that drives the three-step WAAM print loop from a single YAML config file. Eliminates manual path copy-pasting between steps; the only human input required is a visual sanity-check of the error-map plot before each subsequent layer prints.

Usage

# from ros2_ws/lib/control/print_loop/
uv run python print_loop.py my_job.yaml

Verify path resolution without touching the robot:

uv run python print_loop.py my_job.yaml --dry-run

What it automates

Config

Copy example_config.yaml and fill in your job parameters:

job_name: mayor_1
slices_dir: /path/to/layer_slices/
bag_output_dir: /path/to/BAGS/
state_dir: ~/PhotogrammetryWAAM/STATE/ERR/
start_layer: 21
end_layer: 43
z_base: 5.0
z_per_layer: 2.0

error_mapping:
  smooth: 12
  trim_first: 32
  trim_last: 32

control:
  k_p: 0.25
  lower_bounds_corrected_feed: 4.2

Operator interaction

Per layer, the orchestrator runs Steps 1-2 automatically, then:

  Error map: ~/STATE/ERR/L21.html
  Opening in browser...

  Review the error map. Continue? [Y/n/q]

On confirmation it runs Step 3 and prompts once more before printing the next layer. Answering q at any prompt saves state and exits; rerunning the same command resumes from where you left off.

Resumable state

Progress is persisted to {state_dir}/{job_name}_state.json after every step. If the process is interrupted (Ctrl-C, SSH drop, power loss), relaunch the same command and it picks up at the last incomplete step.

Architecture

The orchestrator calls the existing scripts as subprocesses (uv run / ros2 run) rather than importing them, keeping the uv environments independent:

• Step 1: ros2 bag record (Popen + SIGINT) and ros2 run gcode_file_parser_client
• Step 2: uv run python map_surface_err_to_xyz_pos.py (in shared/rosbag_parser/)
• Step 3: uv run python simple_proportional.py (in ros2_ws/lib/control/simple_proportional/)

Files

Full specification (../spec/)

print_loop — SPEC

Purpose

Orchestrate the WAAM closed-loop print cycle so the operator only provides a YAML config and per-layer visual confirmation. All path derivation, subprocess management, and state tracking are automated.

Domain context

Wire-Arc Additive Manufacturing (WAAM) deposits metal layer-by-layer. A camera array measures surface-height error during deposition. That error feeds back into the next layer's toolpath via proportional feedrate correction. The loop runs once per layer.

Type signatures

type GCode           = FilePath
type BagDirectory    = FilePath
type SurfaceErrorMap = CSV  -- columns: X, Y, Z, err_mm
type HTMLPlot        = FilePath

-- Existing modules (called as subprocesses)
execute_gcode            :: GCode -> IO ()
record_bag               :: [Topic] -> IO BagDirectory
map_surface_err_to_xyz   :: (BagDirectory, GCode, Params) -> (HTMLPlot, SurfaceErrorMap)
proportional_correction  :: (GCode_next, SurfaceErrorMap, K_p) -> GCode_corrected

-- This module
print_loop :: PrintJobConfig -> IO ()

Config schema

A YAML file with these fields (validated by Pydantic):

Path derivation rules

Given config and layer number N:

gcode_path  = slices_dir / *_layers_{N}-{N}_1layers[_corrected].gcode
bag_dir     = bag_output_dir / L{N}_Z{z_base + N * z_per_layer}
error_html  = state_dir / L{N}.html
error_csv   = state_dir / L{N}.csv
corrected   = slices_dir / *_layers_{N+1}-{N+1}_1layers_corrected.gcode

Corrected gcode is preferred over nominal when available for a given layer.

Loop algorithm

for layer N in [start_layer .. end_layer]:
    1. EXECUTE + RECORD
       - Popen("ros2 bag record -o {bag_dir} --topics ...")
       - sleep(2)  // let recorder initialise
       - run("ros2 run gcode_file_parser_client ... {gcode_path}")
       - SIGINT bag recorder; wait for shutdown

    2. MAP SURFACE ERROR
       - run("uv run python map_surface_err_to_xyz_pos.py {bag_dir} {gcode_path}
              --nominal_z {z} -o {html} --smooth ... --trim_first ... --trim_last ...")
       - outputs: {html}, {csv}

    GATE: open {html} in browser; prompt operator [Y/n/q]
       - Y: continue to step 3
       - n: skip correction, advance to next layer
       - q: save state, exit

    3. PROPORTIONAL CORRECTION (skip if last layer)
       - run("uv run python simple_proportional.py {next_gcode} {csv}
              -k {k_p} --lower_bounds_corrected_feed {f_min}")
       - output: {next_gcode}_corrected.gcode

    GATE: prompt operator before printing next layer [Enter/q]

    save state after every step

State persistence

JSON file at {state_dir}/{job_name}_state.json. Tracks:

• current_layer: int
• last_completed_step: 0 | 1 | 2 | 3
• layers: dict mapping layer number to per-layer state (gcode_path, bag_dir, step statuses, output paths)

On restart, the loop resumes from the first incomplete step of current_layer.

Subprocess strategy

Each external tool is invoked via subprocess.run (or Popen for bag recording). The orchestrator does not import any of the tool modules directly, keeping uv environments isolated:

• ros2 bag record / ros2 run gcode_file_parser_client -- ROS2 CLI
• uv run python map_surface_err_to_xyz_pos.py -- run in shared/rosbag_parser/
• uv run python simple_proportional.py -- run in ros2_ws/lib/control/simple_proportional/

Dry-run mode

--dry-run prints derived paths for every layer without executing anything. Useful for verifying config correctness.

Constraints

• Gcode files must follow *_layers_{N}-{N}_1layers.gcode naming convention (produced by divide_gcode_into_n_layers.py).
• ROS2 environment must be sourced for Step 1 subprocesses.
• uv must be available on PATH.
• Operator must be able to view HTML files (local browser or file transfer).

simple_proportional

-- Measured surface contour: maps each sampled (x,y,z) point to its
-- height deviation from the target surface (mm).
-- Concretely a CSV with columns X, Y, Z, err_mm.
type SurfaceErrorMap = [((X, Y, Z), ZErr_mm)]

-- Primary signature of this module
simple_proportional_compensation :: (SurfaceErrorMap, GcodeProg) -> GcodeProg

Adjusts G-code feedrates for the next WAAM layer using surface-height errors measured on the previous layer, applying the proportional correction F_corrected = F_nominal * (1 + K_p * z_err_mm). Where the prior layer is too high the robot speeds up (depositing less material), and where it is too low the robot slows down.

Usage

uv run python simple_proportional.py <next_layer_to_be_corrected.gcode> <prev_layer_map__xyz_to_err_z.csv> [-k K_P] [-o output.gcode]

cypher grid generator thingy

IN: (x, y) bounds
OUT: cypher mutation to generate graph nodes

usage

uv run generate_grid_points.py \
        --x_min -0.5 --x_max 0.5 \
        --y_min -0.5 --y_max 0.5 \
        --grid_spacing 0.005 \
        --z_height 0.5

introspect

points within bounds...

MATCH (p:Point) 
WHERE p.x >= -0.1 AND p.x <= 0.1 
AND p.y >= -0.1 AND p.y <= 0.1
RETURN p

other bells other wistles

--write_gcode=./path/to/file.gcode

if --write_gcode=... is passed then points are additionally written to file as

G1 X0 Y0 Z0
G1 X1 Y0 Z0
... etc

kalibr_based_eye_in_hand

through this procedure we will validate a bespoke eye-in-hand calibration procedure using Kalibr.

the steps are:

1. a kalibr target grid is generated

our fork of kalibr is here: https://github.com/nargetdev/kalibr_w_static_focal

generate a target grid

the pattern (soon to be dockerized) to generate a PDF grid looks like

cd kalibr_w_static_focal/aslam_cv/aslam_cameras_april/src/gen_pdf
# sanity check the uv version .. should be >=0.9.x 
uv --version
uv python pin 3.13 # or whatever .. just be explicit .. i guess this could be in pyproject.toml
uv sync --locked
uv run generate_aprilgrid_from_yaml.py april_24x24_size20mm_space0.3.yaml --border-bits 1 --corner-fillet-micrometers 500

MQTT gphoto2 Delegate

A Python-based MQTT service that provides remote camera control capabilities using the gphoto2 library. This delegate follows the same recipient-based topic format as other camera services in the photogrammetry system for consistent device targeting and communication.

Part of the DSLR pipeline (role-1 only) in the Camera Pipelines overview.

Features

• Remote Camera Control: Control compatible cameras via MQTT commands
• Flexible Parameter Support: Supports any gphoto2 configuration parameter
• Recipient-Based Routing: Target specific devices or broadcast to all
• Automatic File Management: Organized capture directories with session management
• Status Reporting: Real-time status updates during capture operations
• Sync Integration: Automatic file sync notifications for downstream processing
• Error Handling: Comprehensive error reporting and logging

Hardware Requirements

• Computer with USB port for camera connection
• Compatible camera (DSLR/mirrorless with gphoto2 support)
• Network connection for MQTT communication

Installation

System Dependencies

# Ubuntu/Debian
sudo apt update
sudo apt install gphoto2 libgphoto2-dev

# macOS with Homebrew
brew install gphoto2

# Verify installation
gphoto2 --version
gphoto2 --auto-detect

Python Dependencies

pip install paho-mqtt

Usage

Starting the Delegate

python mqtt__gphoto2_delegate.py --mqtt-broker <broker_ip> --base-data-dir ./captures

Command Line Options

• --mqtt-broker: MQTT broker address (default: localhost)
• --mqtt-port: MQTT broker port (default: 1883)
• --base-data-dir: Base directory for captures (default: ./captures)

MQTT Topics

• Subscribe: {hostname}/gphoto2 or ALL/gphoto2 (broadcast)
• Publish: {hostname}/gphoto2/response
• Publish: photogrammetry/sync/available

Example MQTT Request

{
  "action": "capture",
  "session_id": "photo_session_001",
  "filename": "portrait.jpg",
  "capture_params": {
    "aperture": "8",
    "shutterspeed": "1/60",
    "iso": "200",
    "imageformat": "RAW+JPEG",
    "whitebalance": "Auto"
  }
}

Testing

Use the included test script to verify functionality:

# Basic targeted test
python test_gphoto2_delegate.py --mqtt-broker <broker_ip> --target-device <hostname>

# Broadcast test
python test_gphoto2_delegate.py --mqtt-broker <broker_ip> --test-type broadcast

# Advanced parameter test
python test_gphoto2_delegate.py --mqtt-broker <broker_ip> --capture-type advanced

Supported Capture Parameters

The delegate supports any gphoto2 configuration parameter. Common parameters include:

File Structure

Each capture session creates organized directories:

captures/
└── session_id/
    └── captured_image.jpg

MQTT Response Messages

The delegate publishes status updates during capture operations:

Starting

{
  "timestamp": "2024-12-01T14:30:22.123456",
  "hostname": "camera-station-1",
  "status": "starting",
  "session_id": "photo_session_001",
  "action": "capture"
}

Capturing

{
  "timestamp": "2024-12-01T14:30:25.123456",
  "hostname": "camera-station-1",
  "status": "capturing",
  "session_id": "photo_session_001",
  "command": ["gphoto2", "--set-config", "aperture=8", "--capture-image-and-download"]
}

Completed

{
  "timestamp": "2024-12-01T14:30:28.123456",
  "hostname": "camera-station-1",
  "status": "completed",
  "session_id": "photo_session_001",
  "capture": {
    "filename": "portrait.jpg",
    "file_size": 2048576,
    "absolute_path": "/home/user/captures/photo_session_001/portrait.jpg"
  }
}

Camera Compatibility

The delegate works with any camera supported by gphoto2. Check compatibility:

# List supported cameras
gphoto2 --list-cameras

# Test camera detection
gphoto2 --auto-detect

# Test basic capture
gphoto2 --capture-image-and-download

Popular compatible cameras include:

• Canon EOS series (DSLR and mirrorless)
• Nikon D series and Z series
• Sony Alpha series
• Fujifilm X series
• Many point-and-shoot cameras

Integration

With File Sync Systems

The delegate automatically publishes sync availability notifications:

{
  "source_role": "gphoto2_camera",
  "source_host": "192.168.1.100",
  "source_hostname": "camera-station-1",
  "source_directory": "/home/user/captures/session_001",
  "session": "session_001"
}

With Other Camera Systems

The delegate can work alongside other camera systems (like the focus stack server) using the same MQTT broker and topic structure.

Troubleshooting

Camera Not Detected

# Check USB connection
lsusb

# Check gphoto2 detection
gphoto2 --auto-detect

# Check for conflicting processes
sudo killall gvfs-gphoto2-volume-monitor
sudo killall gvfsd-gphoto2

Permission Issues

# Add user to camera group (if exists)
sudo usermod -a -G camera $USER

# Or set up udev rules for camera access
sudo gphoto2 --udev-rules > /etc/udev/rules.d/90-libgphoto2.rules
sudo udevadm control --reload-rules

MQTT Connection Issues

• Verify MQTT broker address and port
• Check network connectivity
• Ensure MQTT broker is running and accessible
• Check firewall settings

Development

Running Tests

# Install development dependencies
pip install pytest

# Run basic functionality test
python test_gphoto2_delegate.py --mqtt-broker localhost

# Test different capture types
python test_gphoto2_delegate.py --capture-type advanced

Adding New Features

The delegate is designed to be extensible. To add new capture actions:

1. Update the process_gphoto2_request method
2. Add new action handlers
3. Update the specification document
4. Add corresponding tests

License

This project follows the same license as the parent photogrammetry system.

Contributing

Please follow the established patterns and conventions when contributing:

• Use the same MQTT topic structure
• Follow the existing error handling patterns
• Add appropriate logging and status reporting
• Include tests for new features

CameraWebServer_for_esp-arduino_3.0.x — Project-specific notes

This is the firmware half of the ESP32-S3 + OV2640/OV5640 image pipeline, one of three hardware pipelines in the PhotogrammetricWAAM stack. See the parent overview for the bigger picture and how this pipeline relates to the IMX708 (RPi) and DSLR (gphoto2) pipelines.

Upstream README.md is preserved as-is — it is just an Espressif/Seeed compatibility note. This doc captures the project-specific divergence: our DEVICE_ID convention, MQTT telemetry, port layout, and the open role-switching work.

Hardware

• MCU board: Seeed Studio XIAO ESP32-S3 Sense (PSRAM-equipped — required).
• Sensor: OV2640 (default board) or OV5640 (5 MP daughter sensor). Both work with the same sketch; only frame_size upper-bound differs.
• Network: WiFi (most boards) or wired ETH via an external PHY (the "TSO" boards — see esp32s3_eth.tmuxp.yml).
• PSRAM is required for any frame_size above SVGA — the firmware refuses high-res mode if psramFound() is false.

Per-board identity — the DEVICE_ID convention

Every flashed board is uniquely identified by one preprocessor define near the top of the sketch:

#define DEVICE_ID 143

That single number drives everything else on the network:

To deploy a new board: change just that one number, reflash, done.

A small preprocessor trick concatenates DEVICE_ID into MQTT topic strings at compile time:

#define _STRINGIFY(x) #x
#define _TOSTRING(x) _STRINGIFY(x)
#define DEVICE_ID_STR _TOSTRING(DEVICE_ID)

const char *mqtt_topic = "esp32s3/" DEVICE_ID_STR "/log";

Network endpoints

A flashed board exposes two HTTP servers and one MQTT client:

:81/stream is the canonical Espressif port — different from the IMX708 streamer's :8000/stream. Wire that into all image_publisher_node filenames accordingly.

MQTT telemetry topics

The firmware publishes (one-way, no callback) at fixed intervals:

Reconnect uses non-blocking 2-second backoff so a missing broker never stalls the camera/HTTP loop.

Current camera operating point (as flashed)

The firmware is presently hard-pinned to a role-(1)-leaning still-quality configuration. This is not yet runtime-switchable — see Open work below.

// From CameraWebServer_for_esp-arduino_3.0.x.ino, setup()
config.frame_size      = FRAMESIZE_5MP;          // 2592 × 1944 (OV5640 limit)
config.jpeg_quality    = 27;                     // (lower number = higher quality after the override below)
config.fb_count        = 1;                      // explicit override of the more usual fb_count=2
config.grab_mode       = CAMERA_GRAB_LATEST;
config.fb_location     = CAMERA_FB_IN_PSRAM;
config.xclk_freq_hz    = 20_000_000;
config.pixel_format    = PIXFORMAT_JPEG;

// Then via sensor_t setters:
s->set_framesize(s, FRAMESIZE_5MP);
s->set_quality(s, 6);                            // 6 = high quality (range 0..63, lower = better)
s->set_wb_mode(s, 3);                            // fixed white balance mode
s->set_exposure_ctrl(s, 0);                      // AE off
s->set_aec_value(s, 800);                        // manual exposure value
s->set_gain_ctrl(s, 0);                          // AGC off
s->set_whitebal(s, 0);                           // AWB off
s->set_awb_gain(s, 0);                           // AWB gain off
s->set_raw_gma(s, 1);                            // gamma correction on

Why this combination is role-(1)-leaning today:

• FRAMESIZE_5MP + jpeg_quality=6 produces ~150–400 KB JPGs at the largest size the sensor can natively output.
• fb_count=1 means only one frame's worth of PSRAM is reserved — necessary to fit a 5 MP JPG in PSRAM at all on the XIAO.
• All the …_ctrl(s, 0) / set_aec_value / set_whitebal(0) calls lock exposure / gain / white balance, which is what you want for photogrammetry (frame-to-frame consistency) but not what you want for a viewfinder in changing lighting.

Despite this still-leaning configuration the boards are also currently acting as role (2) MJPG streamers — the ROS 2 host pulls :81/stream and republishes at 16 Hz (WiFi) / 24 Hz (ETH). They get away with it because the JPGs are small enough to push, but the encode latency is higher than it needs to be for real-time monitoring.

Open work — role switching (todo-esp32s3-fb-roles)

The firmware needs runtime support for switching between role (1) and role (2) without reflashing. Target settings, copied from ros2_ws/edge/README.md:

Why fb_count matters specifically on this MCU:

• Role (1) — fb_count=1 is essentially mandatory. With the OV5640 at full 5 MP and JPG quality 6, a single framebuffer can already approach the PSRAM ceiling; a second buffer either won't allocate or will force a smaller resolution.
• Role (2) — fb_count=2 lets the JPG encoder run on buffer N while the sensor DMA is filling buffer N+1, hiding encode latency end-to-end. This is the dominant lever for "fastest stream".

Switch trigger candidates (decision pending — see also todo-mqtt-bridge):

1. New MQTT topic esp32s3/<DEVICE_ID>/role/{request,response} that takes {"role": "stills" | "stream"} and reconfigures the sensor in-place. This is the cleanest fit with the existing mqtt__gphoto2_delegate contract and would make the ESP32-S3 schedulable by the batch_request_delegate.
2. New HTTP endpoint :81/role?… parallel to Espressif's existing /control. Faster to wire up but doesn't unify the control plane.

Implementation hazard: esp_camera_init cannot be re-run without esp_camera_deinit() first, and switching frame_size at runtime is safer through the sensor_t setters than through a full reinit. Test on a single board before fleet rollout.

OTA flashing

ElegantOTA is mounted on the secondary WebServer at port 8080:

http://172.31.1.<DEVICE_ID>:8080/update

Drag-and-drop the new .bin from PlatformIO/Arduino IDE's build artifacts. The board reboots into the new firmware on completion.

Quick checks at boot

The serial console prints (at 115200 baud):

BEGIN SETUP
===========
...
v0.0
FIRMWARE COMPILED: Apr 8th, 2025
JPEG quality 6
awb: OFF
framebuffer - 2          # (note: subsequently overridden to 1 — see informConnectionURL())
CAMERA_GRAB_LATEST
VFLIP!!!
HMIRROR!!!

Camera Stream: http://172.31.1.143:81/stream
OTA Update:    http://172.31.1.143:8080/update

DEVICE_ID    : 143
MQTT broker  : 172.31.1.252:1883
MQTT client  : esp32s3-143
MQTT topics  : esp32s3/143/log , esp32s3/143/temp , esp32s3/143/rssi
OVERRIDING FB COUNT - 1

Use this as a checklist when bringing up a new board.

Related

• Architectural overview: ros2_ws/edge/README.md
• ROS 2 client side (which consumes :81/stream): ros2_ws/launch/image_publisher_client/README.md
• WiFi XIAO launcher: ros2_ws/launch/xiao_sense_esp32s3_eyes.py
• Wired ETH XIAO + Lepton launcher: ros2_ws/launch/image_publisher_client/esp32s3_eth.tmuxp.yml
• Operator browser-pane view: INBOX/TMUXP_VIEWS/xiao_sense_eyes.tmuxp.yml
• IMX708 sibling pipeline: ros2_ws/edge/simple_picamera2_streamer/README.md
• DSLR sibling pipeline: mqtt__gphoto2_delegate

EKI Pose Action Server

This ROS 2 action server receives requests for KUKA Cartesian poses and sends them to the robot.

Action Specification

The action is defined in action/CartesianPose.action.

Action Goal

The goal is a CartesianPose.action message containing the desired pose.

Action Result

The result is an actionlib_msgs/GoalStatus message indicating the success or failure of the goal.

Robot Interface

The action server uses the rclpy library to interact with the robot.

Publisher

The action server publishes /kuka/pose messages to command the robot to a new pose.

Subscriber

The action server subscribes to /kuka/pose messages to receive updates on the robot's current pose.

Main Loop

The main loop of the action server listens for incoming goals. When a goal is received, the action server publishes a /kuka/pose message with the desired pose and waits for a feedback message from the robot indicating the success of the command.

When a feedback message is received, the action server sends a result message indicating the success or failure of the goal.

Note: The implementation of the action server is not provided here, as it is a complex task that requires knowledge of the robot's specific interface and the ROS 2 rclpy library.

copy python files into Blender

./__DEV_TOOLING/TOOLS/USING_MCP__copy_python_file_into_Blender.py \
    --server tcp://localhost:9876 \
    BLENDER_PLUGIN_COMMON/is_CLI_or_Blender_context.py \
    --open-in-editor

{
  "status": "ok",
  "mode": "tcp",
  "response": {
    "status": "success",
    "result": {
      "executed": true,
      "result": "Created/updated: is_CLI_or_Blender_context.py\n"
    }
  }
}

Example Guide

Guides lead a user through a specific task they want to accomplish, often with a sequence of steps. Writing a good guide requires thinking about what your users are trying to do.

Further reading

• Read about how-to guides in the Diátaxis framework

using GCODE with KUKA

The gcode_sender can be commanded via MQTT.

Que up commands...

mosquitto_pub -h 172.31.1.252 -t kuka/gcode -m 'G1 X0.0 Y0 Z400 A0 B0 C0 F80'
mosquitto_pub -h 172.31.1.252 -t kuka/gcode -m 'G1 X0.0 Y0 Z500 A0 B0 C0 F80'

then click the "RUN TRIGGER: ON" button.

This step physically deposits one layer of metal and simultaneously captures all sensor telemetry into a ROS 2 bag.

What Happens

1. The operator (or automation script) launches ros2 bag record to capture the topics listed below.
2. gcode_file_parser_client sends the G-code toolpath to the KUKA robot via the EKI interface. The robot moves along the commanded path while the wire-arc welding system deposits material.
3. During the motion the wire_stickout node continuously publishes /wire_stickout/err_surface_mm — the deviation of the observed surface from the expected height, measured by an array of welding-glass-shielded cameras mounted on the tool tip that track the arc's brightest point via blob detection.

gcode_file_parser_client :: GCode -> IO PhysicalDeposition
-- side-effects: robot motion, arc welding, sensor streaming

Commands

# Terminal 1 — start bag recording before the print
ros2 bag record -o L<N>_Z<ZZ> \
  --topics \
    /TSO_1/image_raw/compressed \
    /TSO_0/image_raw/compressed \
    /TSO_2/image_raw/compressed \
    /cams/table_1/image_raw/compressed \
    /cams/table_0/image_raw/compressed \
    /kuka/pose \
    /wire_stickout/err_surface_mm \
    /kuka/velocity_cartesian \
    /therm/image_raw/compressed \
    /kuka/sequence_number

# Terminal 2 — execute the layer gcode
ros2 run gcode_file_parser_client gcode_file_parser_client <layer.gcode>

Naming Convention

Bag directories follow the pattern L<layer>_Z<nominal_z>, e.g. L21_Z48 means layer 21 at nominal Z = 48 mm.

Key Sensor: wire_stickout

The wire_stickout node uses an array of cameras mounted on the tool tip, each shielded by a small glass welding-shade to reduce arc brightness. These cameras detect the y-position of the arc's brightest point (blob detection) and translate that into the real-time weld surface position. During a calibration pass on the flat substrate (layer 0), a baseline blob (x, y) position is established at a known 10 mm stickout. Subsequent layers compare the observed blob position against this baseline to derive err_surface_mm — the signed deviation of the real surface from the expected height.

Outputs

Step 2 — Map Surface-Z Error

After a layer is printed and its rosbag is captured, the rosbag_parser associates the err_surface_mm signal with the robot's (X, Y, Z) position at each moment in time, producing both a human-readable 3D visualization and a machine-readable error map.

What Happens

1. Read /kuka/pose messages → extract (timestamp, X, Y, Z).
2. Read /wire_stickout/err_surface_mm messages → extract (timestamp, value).
3. Interpolate error values onto pose timestamps (nearest-neighbor by time).
4. Optionally parse the G-code for the commanded toolpath overlay.
5. Apply trimming/smoothing to suppress transient sensor noise at layer start/end.
6. Write error_map.csv — each row maps a robot position to its error.
7. Render an interactive 3D Plotly scatter plot and save to plot.html.

rosbag_parser :: BagDirectory -> GCode -> (HTML, CSV)

Command

uv run python map_surface_err_to_xyz_pos.py \
  /path/to/L<N>_Z<ZZ>/ \
  /path/to/<layer>.gcode \
  --nominal_z <ZZ> \
  --no-gcode \
  -o ~/PhotogrammetryWAAM/STATE/ERR/L<N>.html \
  --smooth 12 \
  --trim_first_n_err_vals 32 \
  --trim_last_n_err_vals 32

Working directory: shared/rosbag_parser/

Key Parameters

Processing Order

1. Detect (or accept) nominal Z.
2. Trim leading samples where Z deviates > 1 mm from nominal.
3. Exclude samples outside ±exclude_outside_z_bounds of nominal Z.
4. Interpolate error to remaining pose timestamps.
5. Zero first/last N error values.
6. Apply smoothing window.

Outputs

error_map.csv Schema

X,Y,Z,err_mm
12.2021,47.3637,43.2479,4.0644
10.7972,47.6810,43.2649,4.0644
...

All positions are in the robot frame (mm). Error is signed: positive means the surface is above nominal, negative means below.

3D Plot Traces

Step 3 — Proportional Control Correction

The error map from Step 2 is fed into a proportional controller that adjusts the G-code feedrates for the next layer to compensate for measured surface-height deviations. This closes the control loop.

Control Law

F_corrected = F_nominal * (1 + K_p * z_err_mm)

Where:

• F_nominal — the feedrate in the original (uncorrected) G-code for the next layer.
• K_p — proportional gain (default: 1.0). Tuned experimentally.
• z_err_mm — the surface height error at the nearest measured point from the previous layer's error map.

Intuition

• Positive error (surface too high) → increase feedrate → robot moves faster → deposits less material → surface drops.
• Negative error (surface too low) → decrease feedrate → robot moves slower → deposits more material → surface rises.

Spatial Matching

For each extruding G-code point (E > 0) in the next layer's toolpath, the controller finds the nearest point in the error map CSV (by 3D Euclidean distance) and applies that point's err_mm value to the feedrate formula. Only extruding moves are modified; travel moves are passed through unchanged.

Command

uv run python simple_proportional.py \
  <next_layer.gcode> \
  <L<N>_error_map.csv> \
  -k <K_p> \
  -o <next_layer_corrected.gcode>

Working directory: ros2_ws/lib/control/simple_proportional/

Arguments

Inputs

1. Next Layer G-Code (nominal)

Standard slicer output. Only G1 moves with non-zero E values have their F parameter adjusted.

G1 X-7.119 Y-17.261 E24.0 F3.4 ;N4
G1 X-5.676 Y-16.964 E24.0 F3.4 ;N5

2. Previous Layer Error Map (CSV)

X,Y,Z,err_mm
12.2021,47.3637,43.2479,4.0644
10.7972,47.6810,43.2649,4.0644

Output

A corrected G-code file identical to the input except that feedrate (F) values on extruding moves have been scaled according to the control law. This file becomes the input to Step 1 for the next iteration.

Future Work

The current controller is purely proportional (P-only). Planned extensions include:

• Integral (I) term — accumulate historical error across multiple layers to eliminate steady-state offset.
• Derivative (D) term — respond to the rate of error change between consecutive layers.
• Full PID tuning with per-region gain scheduling.

The per-layer error CSVs produced by Step 2 accumulate in the state directory over the course of a print job. Step 4 aggregates them into a single interactive visualization that confirms the proportional controller's performance: error should diminish as layers progress.

What it produces

3D Pancake Stackup

An interactive Plotly scene showing every layer's robot path and error surface at their physical Z heights — a literal "pancake stack" of the build. Each layer has two traces:

Layers are toggle-able via the legend. Hover reveals coordinates and error values.

Error Trend Chart

A 2D subplot below the 3D scene charting per-layer error statistics against layer number:

A downward slope confirms the controller is converging.

Summary CSV

stackup_summary.csv — one row per layer with columns: layer, nominal_z, n_samples, mean_abs_err, rms_err, max_abs_err, std_err

Command

# from ros2_ws/lib/control/aggregate_error_stackup/
uv run python aggregate_error_stackup.py <state_dir> [options]

Working directory: ros2_ws/lib/control/aggregate_error_stackup/

Arguments

Inputs

All L{N}.csv files in the state directory, each with schema:

X,Y,Z,err_mm
12.2021,47.3637,43.2479,4.0644
10.7972,47.6810,43.2649,4.0644

These are produced by Step 2 (map_surface_err_to_xyz_pos.py) and accumulate naturally as the print loop runs.

When to run

• During a print job — run between layer batches to check that error is trending downward. If not, consider adjusting k_p.
• After a completed job — produce a final report of controller performance across all layers.
• Comparing configurations — run on state directories from different print jobs to compare k_p values or other parameter changes.

Remote viewing

The output HTML is self-contained (Plotly JS is inlined). Serve it over the network from the robot host:

cd /path/to/STATE/ERR/job_dir/
python3 -m http.server 8080 --bind 0.0.0.0
# browse to http://<robot-host>:8080/stackup.html

Relationship to Steps 1–3

Steps 1–3 form the real-time closed loop. Step 4 is a post-hoc analysis tool that reads the same artifacts the loop produces. It does not modify any state and can be run repeatedly as new layers become available.

Steps 1–3 (per layer, real-time)     Step 4 (aggregate, any time)
       │                                       │
       ▼                                       ▼
 state_dir/L{N}.csv  ──────────────→  stackup.html
                                      stackup_summary.csv

The WAAM print loop is a closed-loop cycle that prints a layer of metal, measures the resulting surface error, and feeds that measurement back into the toolpath for the next layer. Each iteration drives the surface height closer to its nominal value.

┌──────────────────────────────────────────────────────┐
│                                                      │
│   ┌─────────────────────────────────────────────┐    │
│   │  1. EXECUTE G-CODE  +  RECORD ROSBAG        │    │
│   │     gcode_file_parser_client                 │    │
│   │     ros2 bag record …                        │    │
│   └──────────────────┬──────────────────────────┘    │
│                      │                               │
│                      ▼                               │
│   ┌─────────────────────────────────────────────┐    │
│   │  2. MAP SURFACE-Z ERROR                      │    │
│   │     rosbag_parser                            │    │
│   │     → plot.html  +  error_map.csv            │    │
│   └──────────────────┬──────────────────────────┘    │
│                      │                               │
│                      ▼                               │
│   ┌─────────────────────────────────────────────┐    │
│   │  3. PROPORTIONAL CONTROL CORRECTION          │    │
│   │     simple_proportional_control              │    │
│   │     → corrected_layer.gcode                  │    │
│   └──────────────────┬──────────────────────────┘    │
│                      │                               │
│                      └───────────────────────────┘   │
│                        loop back to step 1           │
└──────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────┐
│  4. AGGREGATE ERROR STACKUP  (post-hoc, any time)    │
│     aggregate_error_stackup                          │
│     → stackup.html  +  stackup_summary.csv           │
│     Reads all L*.csv from state_dir                  │
│     Confirms controller convergence over layers      │
└──────────────────────────────────────────────────────┘

Data Flow (type signatures)

Using Haskell-style notation to express what each stage consumes and produces:

-- Step 1: execute toolpath, observe physical result
gcode_file_parser_client :: GCode -> IO PhysicalDeposition

-- (concurrently)
ros2_bag_record :: [Topic] -> IO BagDirectory

-- Step 2: associate error with robot XYZ
rosbag_parser :: BagDirectory -> GCode -> (HTML, CSV)
--   CSV schema: X, Y, Z, err_mm

-- Step 3: correct next layer feedrates
simple_proportional :: GCode_next -> CSV_err -> K_p -> GCode_corrected
--   F_corrected = F_nominal * (1 + K_p * z_err_mm)

-- Step 4: aggregate multi-layer error for convergence analysis
aggregate_error_stackup :: StateDir -> (AggregateHTML, SummaryCSV)
--   Reads all L{N}.csv, produces 3D stackup + error trend chart

Recorded ROS Topics

Each layer print records the following topics into an MCAP bag:

Quick-Reference Commands

Step 1 — Execute & Record

# start recording (in one terminal)
ros2 bag record -o L<N>_Z<ZZ> \
  --topics \
    /TSO_1/image_raw/compressed \
    /TSO_0/image_raw/compressed \
    /TSO_2/image_raw/compressed \
    /cams/table_1/image_raw/compressed \
    /cams/table_0/image_raw/compressed \
    /kuka/pose \
    /wire_stickout/err_surface_mm \
    /kuka/velocity_cartesian \
    /therm/image_raw/compressed \
    /kuka/sequence_number

# execute gcode (in another terminal)
ros2 run gcode_file_parser_client gcode_file_parser_client <layer.gcode>

Step 2 — Map Surface Error

uv run python map_surface_err_to_xyz_pos.py \
  <bag_directory>/ \
  <layer.gcode> \
  --nominal_z <ZZ> \
  --no-gcode \
  -o <STATE/ERR/L<N>.html> \
  --smooth 12 \
  --trim_first_n_err_vals 32 \
  --trim_last_n_err_vals 32

Step 3 — Correct Next Layer

uv run python simple_proportional.py \
  <next_layer.gcode> \
  <L<N>_error_map.csv> \
  -k <K_p> \
  -o <next_layer_corrected.gcode>

Then go back to Step 1 with the corrected G-code.

Step 4 — Aggregate Error Stackup (post-hoc)

# from ros2_ws/lib/control/aggregate_error_stackup/
uv run python aggregate_error_stackup.py <STATE/ERR/job_dir/>

Run at any time during or after a print job. Reads all L*.csv in the state directory and produces stackup.html (3D stackup + error trend) and stackup_summary.csv (per-layer error statistics). Serve for remote viewing with python3 -m http.server 8080.

Automated Orchestrator (print_loop)

Instead of running each step manually, the print-loop orchestrator drives the entire cycle from a single YAML config file:

# from ros2_ws/lib/control/print_loop/
uv run python print_loop.py my_job.yaml

The orchestrator:

1. Resolves all gcode file paths from the slices_dir automatically.
2. Starts/stops bag recording and gcode execution as subprocesses.
3. Runs map_surface_err_to_xyz_pos.py and simple_proportional.py.
4. Opens the error-map HTML for visual inspection between layers.
5. Waits for operator confirmation before printing the next layer.
6. Saves resumable state to {state_dir}/{job_name}_state.json.

Use --dry-run to verify path resolution without executing anything:

uv run python print_loop.py my_job.yaml --dry-run

See example_config.yaml in ros2_ws/lib/control/print_loop/ for a template configuration.

See the sub-pages for detailed descriptions of each step.

another

Reference pages are ideal for outlining how things work in terse and clear terms. Less concerned with telling a story or addressing a specific use case, they should give a comprehensive outline of what you're documenting.

Further reading

• Read about reference in the Diátaxis framework

Example Reference

Reference pages are ideal for outlining how things work in terse and clear terms. Less concerned with telling a story or addressing a specific use case, they should give a comprehensive outline of what you're documenting.

Further reading

• Read about reference in the Diátaxis framework

Camera Image Ingestion — Overview

This document is the single entry point for how images get from a physical sensor into the central ROS 2 host in the PhotogrammetricWAAM system.

There are exactly three hardware pipelines and two roles each camera can serve. Everything else in this folder (and in PhotogrammetricWAAM-Edge/ and ros2_ws/launch/) is one concrete instance of that 3 × 2 matrix.

TL;DR — The 3 × 2 matrix

ROS 2 client side (the kernel host) is uniform across all three: it runs image_publisher_node against either an MJPG stream URL (roles 2, IMX708 + ESP32S3) or consumes the on-disk JPG/RAW that the gphoto2 delegate dumps (role 1, DSLR and on-demand IMX708/ESP32S3). See ros2_ws/launch/image_publisher_client/README.md.

The two roles, in detail

Every photogrammetry-grade camera in this system serves one of two roles at any given moment. The IMX708 and the ESP32-S3-attached OV2640/OV5640 are capable of either role; the DSLR is permanently locked to role (1).

Role (1) — Highest-fidelity still producer

Goal: Best possible JPG (or RAW) of one moment in time, on demand or at a slow cadence. Latency does not matter.

• Maximum sensor resolution (e.g. IMX708 4608×2592, OV5640 2592×1944, DSLR ≥24 MP).
• Highest JPG quality (low quantisation) — or RAW where available.
• Capture is triggered, not free-running. One trigger → one (or one stack of) frames written to durable storage with a session ID.
• Consumed by the photogrammetry / SfM pipeline downstream — not by RViz/Foxglove.
• Control plane: MQTT (recipient-based topics — see mqtt__gphoto2_delegate.spec.md for the canonical request/response shape).

Role (2) — Lowest-latency MJPG streamer

Goal: Real-time monitoring on the ROS 2 graph (visible in rqt_image_view, Foxglove, RViz). Per-frame fidelity is sacrificed for timeliness.

• Down-rezzed and/or higher JPG compression (e.g. IMX708 at 2304×1296 @ 8 Hz, ESP32-S3 OV5640 typically HD/SVGA).
• Continuous MJPG over plain HTTP (multipart/x-mixed-replace).
• Encoded once at the edge, decoded once on the ROS 2 client by image_publisher_node, republished as sensor_msgs/Image on a per-camera namespace (/cam0/image_raw, /xiao_143/image_raw, …).
• Control plane: HTTP (/set on RPi today; open work for ESP32-S3 — see TODOs below).

                 role-(1)                          role-(2)
              "snapshot mode"                  "viewfinder mode"
        ┌────────────────────────┐        ┌─────────────────────────┐
   any  │  highest quality JPG    │        │  smallest-possible JPG  │
   cam  │  on demand, slow rate   │        │  fast as possible,      │
        │  → durable storage      │        │  free-running           │
        │  → SfM / photogrammetry │        │  → ROS 2 image topic    │
        │  → RViz/Foxglove via    │        │  → live monitoring      │
        │     image_publisher of  │        │     (rqt_image_view)    │
        │     a *file* path       │        │                         │
        └────────────────────────┘        └─────────────────────────┘
                  ▲                                     ▲
                  │ MQTT request/response               │ HTTP GET /stream
                  │ (gphoto2-style topics)              │ (multipart MJPG)

The three hardware pipelines, in detail

1. IMX708 on Raspberry Pi (CSI)

[ IMX708 sensor ]──CSI──▶[ Raspberry Pi ]──HTTP MJPG──▶[ ROS 2 host ]
                          libcamera/picamera2           image_publisher_node
                          + cv2.imencode JPG            → /camN/image_raw
                          app.py @ :8000

Edge: ros2_ws/edge/simple_picamera2_streamer/app.py. A single Python process that owns the camera, runs a capture thread at the configured FrameDurationLimits, and serves three endpoints:

Role today: running role (2) only — see TODO todo-imx708-fb-roles for the mode-switch work.

Client: see Both tmuxp variants below.

2. OV2640 / OV5640 on XIAO ESP32-S3 Sense (DVP)

[ OV2640 / OV5640 ]──DVP──▶[ XIAO ESP32-S3 ]──HTTP MJPG──▶[ ROS 2 host ]
                            esp_camera + httpd            image_publisher_node
                            CameraWebServer_for_*.ino     → /xiao_NNN/image_raw
                            stream @ :81/stream
                            OTA    @ :8080/update
                            telemetry → MQTT broker

Edge: PhotogrammetricWAAM-Edge/photogrammetricWAAM_xiao_eyes_ov2640_ov5640/CameraWebServer_for_esp-arduino_3.0.x/.

Custom Arduino-ESP32 (3.0.x) firmware derived from Espressif's CameraWebServer. Each board is statically configured by a single #define DEVICE_ID 1xx which also drives:

• Static IP 172.31.1.<DEVICE_ID>
• MQTT topics esp32s3/<DEVICE_ID>/{log,temp,rssi}
• MQTT client id esp32s3-<DEVICE_ID>

The MJPG stream is served on port 81 (the canonical Espressif port — not the same as the IMX708 streamer's :8000). OTA flashing lives on :8080/update.

Currently the firmware is hard-configured for role (1)-leaning settings (FRAMESIZE_5MP, set_quality(s, 6), set_aec_value(s, 800), awb=OFF, fb_count=1, CAMERA_GRAB_LATEST) — see TODO todo-esp32s3-fb-roles for the dynamic role-switching work.

Client: see Both tmuxp variants below.

3. DSLR on Raspberry Pi (USB)

[ Canon/Nikon/Sony ]──USB──▶[ Raspberry Pi ]──gphoto2 capture──▶[ shared FS ]
                              mqtt__gphoto2_delegate.py        ─sync─▶ ROS 2 host
                              MQTT request/response                    image_publisher_node
                                                                      → /dslr_NN/image_raw

Edge: PhotogrammetricWAAM-Blender-UI/02__STILLS/_EDGE_CAMERA_DAEMON/.../mqtt__gphoto2_delegate.py.

A Python service that subscribes to {hostname}/gphoto2 (or ALL/gphoto2), shells out to gphoto2 --set-config … --capture-image-and-download …, writes the result into a session-ID'd directory, and publishes a structured {hostname}/gphoto2/response along with a photogrammetry/sync/available notification for the file-sync layer.

Role today: role (1) only. DSLRs do not stream MJPG in this stack. (gphoto2 --capture-movie exists but is intentionally out of scope — the DSLR is the fidelity reference.)

SSH operator view: INBOX/TMUXP_VIEWS/DSLR.tmuxp.yml opens parallel SSH sessions to the DSLR-hosting Pis (id2-rpi4.local, pi3m50.local).

Batch coordination across many DSLRs + Pi cams is the job of batch_request_delegate.py — one MQTT batch request fans out to N services and aggregates N responses into a single batch response.

Edge ↔ ROS 2 contract — the two halves

Edge half (server)

ROS 2 client half

The ROS 2 host runs image_publisher_node (one per camera URL or per file path), which:

1. Decodes the MJPG / JPG into an OpenCV Mat.
2. Publishes sensor_msgs/Image on <__ns>/image_raw (and camera_info if provided).
3. Republishes at the rate set by publish_rate.

Critical empirical finding (see simple_picamera2_streamer/README.md): publish_rate MUST match the edge capture rate exactly, otherwise OpenCV internally buffers MJPG frames and rqt_image_view shows stale frames from seconds in the past. With the IMX708 streamer at 8 Hz, the client must be launched with publish_rate:=8. — not 7.9, not 10.

Two tmuxp launch styles exist for this client side, depending on where you're running from:

• Mac/laptop with pixi (no system ROS install): pixi_image_publishers.yml — prefixes every command with pixi run -e kilted ros2 …
• ROS host / docker container with ros2 on PATH: image_publishers.tmuxp.yml — same commands without the pixi run prefix.

Plus a parameterised Python launch file at xiao_sense_esp32s3_eyes.py for the ESP32-S3 fleet, and esp32s3_eth.tmuxp.yml for the wired ESP32-S3 + Lepton thermal mix.

See ros2_ws/launch/image_publisher_client/README.md for the full namespace map and per-host IP allocation.

Open implementation work (tracked, not yet implemented)

These are intentional gaps — documented here so the architecture page is the source of truth, then mirrored in the project todo list.

todo-esp32s3-fb-roles — Runtime role switching on the ESP32-S3 XIAO

The OV2640/OV5640 firmware is currently hard-pinned to one operating point. Add a runtime mode-switch (over MQTT or HTTP) that reconfigures the camera without a reflash:

Rationale: with PSRAM at a premium, fb_count=1 lets a 5MP JPG actually fit; fb_count=2 hides JPG-encode latency for streaming.

todo-imx708-fb-roles — Mode switching in simple_picamera2_streamer/app.py

Today app.py is built around picam2.create_video_configuration(main={"size": (2304,1296)}, buffer_count=4) — fixed at role (2). Add an endpoint (e.g. GET /mode?role=stills / …?role=stream) that:

• For role (1): picam2.switch_mode_and_capture_file(...) against a still configuration at full sensor resolution, optionally RAW+JPEG, then revert.
• For role (2): keep the current free-running 8 Hz video path.

This makes one IMX708 host serve both the SfM batch capture and the live viewfinder without contention.

todo-mqtt-bridge — Unify the control plane

Right now control is heterogeneous:

• IMX708 is controlled by HTTP GET /set?ExposureTime=….
• ESP32-S3 has only MQTT telemetry (log/temp/rssi) — control is via the Espressif web UI on :81/.
• DSLR is fully MQTT (request/response, recipient-based).

Decide whether the RPi streamer and the ESP32-S3 firmware should adopt the same recipient-based MQTT contract as the gphoto2 delegate. If yes, the batch_request_delegate already aggregates across services and would Just Work.

See also

• simple_picamera2_streamer/README.md — IMX708 pipeline detail
• ros2_ws/launch/image_publisher_client/README.md — ROS 2 client side, namespace map, tmuxp variants
• PhotogrammetricWAAM-Edge/.../CameraWebServer_for_esp-arduino_3.0.x/PROJECT_README.md — ESP32-S3 firmware specifics
• mqtt__gphoto2_delegate.spec.md — DSLR MQTT contract
• batch_request_delegate.spec.md — multi-camera batch coordinator

simple_picamera2_streamer — IMX708 edge stack

A tiny, dependency-light Python service that owns one IMX708 (Raspberry Pi Camera Module 3) on a Raspberry Pi over CSI and exposes it as:

• a continuous MJPG stream (multipart/x-mixed-replace) for the live ROS 2 image graph (role 2 — low-latency viewfinder), and
• a single-shot JPG endpoint (/jpg) plus a control endpoint (/set).

It is one of three hardware ingestion pipelines in this system — see ros2_ws/edge/README.md for the bigger picture.

Status: runs role (2) only today. Role (1) high-fidelity still capture is a known gap — see TODO todo-imx708-fb-roles in the parent README.

Edge — the server side (app.py)

[ IMX708 ]──CSI──▶[ picamera2 capture_array ]──cv2.imencode JPG──▶ shared frame_jpg
                                                                          │
                                                                  ┌───────┴────────┐
                                                                  ▼                ▼
                                                          GET /stream       GET /jpg
                                                          (multipart MJPG)  (one frame)

Capture loop

config = picam2.create_video_configuration(main={"size": (2304, 1296)}, buffer_count=4)
picam2.set_controls({"AeEnable": 1, "AwbEnable": 1, "AfMode": 2})  # AfMode=2 = continuous AF
picam2.set_controls({"FrameDurationLimits": (125000, 125000)})     # 125 ms ≈ 8 Hz, hard-capped

A single background thread does picam2.capture_array() → cv2.imencode(".jpg", arr, [cv2.IMWRITE_JPEG_QUALITY, 100]) and atomically swaps the result into a shared frame_jpg under a threading.Lock(). Every HTTP handler reads from that single shared buffer — there is no per-client encoder.

HTTP endpoints

The server is a ThreadingHTTPServer so /stream, /jpg, and /set can be served concurrently to multiple clients.

Running it on the Pi

# on the Raspberry Pi (e.g. as a systemd service, or under tmux)
python3 ros2_ws/edge/simple_picamera2_streamer/app.py
# listens on 0.0.0.0:8000

Quick sanity checks from anywhere on the network (substitute the Pi's IP):

# stream — open in browser or VLC
open http://172.31.1.97:8000/stream

# one-shot JPG into a file
curl -o frame.jpg http://172.31.1.97:8000/jpg

# bias the exposure / gain / focus
curl 'http://172.31.1.97:8000/set?ExposureTime=10000&AnalogueGain=2.0'
curl 'http://172.31.1.97:8000/set?LensPosition=2.5'   # diopters → ~40 cm

ROS 2 client side — image_publisher_node

The matching ROS 2 client is the stock image_publisher package. One node per camera URL, wrapped in a per-camera namespace.

ros2 run image_publisher image_publisher_node \
  --ros-args \
  -p filename:=http://172.31.1.97:8000/stream \
  -p publish_rate:=8. \
  -r __ns:=/cam2

This produces:

• /cam2/image_raw — sensor_msgs/Image
• /cam2/image_raw/compressed — sensor_msgs/CompressedImage
• /cam2/camera_info — empty unless a calibration is provided

⚠️ The single most important parameter — publish_rate

publish_rate MUST equal the edge capture rate, exactly.

image_publisher_node opens the MJPG URL via OpenCV (cv::VideoCapture), which buffers decoded frames internally. If publish_rate is slower than the edge produces frames, OpenCV's queue fills up and the node ends up republishing frames from seconds — sometimes tens of seconds — in the past. rqt_image_view and Foxglove will show stale, lagging video that looks "smooth" but is actually time-shifted.

Concrete contract for this streamer:

If you change app.py's frame rate, change every tmuxp / launch file's publish_rate in lockstep.

Where the client is launched

Two equivalent tmuxp variants live in ros2_ws/launch/image_publisher_client/ — one for Mac/laptop without a system ROS install (uses pixi run -e kilted ros2), one for hosts with ros2 already on PATH. Both fan out one image_publisher_node per camera URL into separate tmux panes:

• image_publishers.tmuxp.yml
• pixi_image_publishers.yml

See ros2_ws/launch/image_publisher_client/README.md for the full IP-and-namespace map.

Operational notes

• Bandwidth. At 2304×1296 JPG-quality 100 @ 8 Hz, expect roughly 500 KB per frame ≈ 32 Mbit/s per camera. Plan WiFi accordingly — on a 2.4 GHz AP with two active streams you will saturate.
• CPU on the Pi. cv2.imencode at quality 100 on a Pi 5 is the dominant cost. Drop quality to ~85 if you need headroom; visually indistinguishable for monitoring purposes.
• Latency budget. Roughly: 125 ms (sensor) + ~20 ms (encode) + ~30 ms (network) + 50–100 ms (OpenCV decode + republish) ≈ 200–300 ms glass-to-RViz.
• AF behaviour. AfMode=2 is continuous AF; setting LensPosition via /set flips to manual (AfMode=0) and stays there until restart. To return to continuous AF, restart the process — there is intentionally no "go back to auto" verb yet.

Related

• Parent overview: ros2_ws/edge/README.md
• ROS 2 client launch directory: ros2_ws/launch/image_publisher_client/README.md
• ESP32-S3 sibling pipeline: PhotogrammetricWAAM-Edge/.../CameraWebServer_for_esp-arduino_3.0.x/PROJECT_README.md
• DSLR sibling pipeline: mqtt__gphoto2_delegate

image_publisher_client — the ROS 2 host side

This directory holds the client half of the camera ingestion stack. The server halves (one per hardware pipeline) live elsewhere — see the parent overview.

The client side is uniform: for every MJPG URL coming off an edge node, run exactly one image_publisher_node in its own tmux pane (or its own Node in a launch file). Each pane:

• decodes the MJPG / JPG stream,
• republishes it as sensor_msgs/Image on <__ns>/image_raw,
• runs at the rate set by publish_rate.

The single most important rule is that publish_rate must equal the edge capture rate exactly. See Why publish_rate matters below.

Files in this directory

A second, ESP32-S3-only Python launch file lives one level up: ../xiao_sense_esp32s3_eyes.py — for the WiFi XIAO eyes (172.31.1.139/142/143/144 @ :81/stream, 16 Hz).

Choosing between image_publishers.tmuxp.yml and pixi_image_publishers.yml

Functionally identical. The only difference is how ros2 is invoked:

# image_publishers.tmuxp.yml — assumes ROS is installed system-wide
- shell_command:
    - ros2 run image_publisher image_publisher_node ...

# pixi_image_publishers.yml — assumes pixi env "kilted" is set up locally
- shell_command:
    - pixi run -e kilted ros2 run image_publisher image_publisher_node ...

Use cases:

• image_publishers.tmuxp.yml — running on the ROS host (the ros2 Docker container, a real Ubuntu/ROS box, or any environment where which ros2 works at the prompt). This is the production launcher when the kernel host comes up.
• pixi_image_publishers.yml — running on macOS / a laptop during development. There is no native ROS 2 install; pixi brings in a per-project ROS distribution (here, the kilted env defined in the repo's pixi config). Use this when you want to view the streams from your laptop using rqt_image_view or Foxglove without spinning up a Docker container.

The pixi variant is intentionally a subset of the full client — it only launches the two cameras you typically need to monitor from a laptop (172.31.1.97:8000 and :8001).

Camera namespace map

The default tmuxp session (image_publishers.tmuxp.yml) publishes the following ROS 2 topics. 172.31.1.x here are the static IPs of the Raspberry Pi hosts running simple_picamera2_streamer/app.py (IMX708 cams over CSI). Each Pi may host more than one streamer on different ports.

All published at publish_rate:=8. — matching the IMX708 streamer's hard 8 Hz cap.

The ESP32-S3 XIAO fleet (xiao_sense_esp32s3_eyes.py):

All published at publish_rate:=16..

The wired-ETH ESP32-S3 + Lepton mix (esp32s3_eth.tmuxp.yml):

IP convention reminder. ESP32-S3 boards on this network are bound to 172.31.1.<DEVICE_ID> by their firmware (#define DEVICE_ID …). Pi cam hosts use whatever DHCP/static you've assigned them; the "port +1 per cam" trick (:8000, :8001) is what lets one Pi host two IMX708 streamers.

Running them

# default — ROS host / docker container
tmuxp load ros2_ws/launch/image_publisher_client/image_publishers.tmuxp.yml

# from a Mac/laptop (no system ROS)
tmuxp load ros2_ws/launch/image_publisher_client/pixi_image_publishers.yml

# wired ESP32-S3 fleet + Lepton thermal
tmuxp load ros2_ws/launch/image_publisher_client/esp32s3_eth.tmuxp.yml

# WiFi XIAO ESP32-S3 fleet
ros2 launch ros2_ws/launch/xiao_sense_esp32s3_eyes.py

Verify in another shell:

ros2 topic list | grep image_raw
ros2 topic hz /cam0/image_raw          # should report ~8 Hz
ros2 run rqt_image_view rqt_image_view /cam0/image_raw

Why publish_rate matters

image_publisher_node opens MJPG URLs through OpenCV (cv::VideoCapture), which buffers decoded frames internally. If publish_rate is slower than the rate the edge produces frames, that buffer fills up and the node republishes frames from seconds — sometimes tens of seconds — in the past.

The downstream symptom is "smooth but wrong" video: rqt_image_view / Foxglove / RViz show motion that looks fluid but lags the real world by a slowly growing offset.

The fix is to match the rate exactly. Per pipeline:

If you bump the edge rate in the firmware / streamer, bump it here in lockstep — every tmuxp file and every launch file in this directory.

Related

• Architectural overview: ros2_ws/edge/README.md
• IMX708 edge server: ros2_ws/edge/simple_picamera2_streamer/README.md
• ESP32-S3 firmware: PhotogrammetricWAAM-Edge/.../CameraWebServer_for_esp-arduino_3.0.x/PROJECT_README.md
• DSLR (role-1 only) pipeline: mqtt__gphoto2_delegate
• xiao_sense_eyes.tmuxp.yml (browser/operator view) at INBOX/TMUXP_VIEWS/

grid_generator.py generates very simple 2D grid of RigidTransform[]

parameters

• Optional: scan_id (autogenerated if not provided)
• Required: center_x, center_y, grid_size_x, grid_size_y, grid_spacing, z_height

relevant schema

• schema/Scan/Scan.py
• schema/RigidTransform/RigidTransform.py

Rotations

if optional params are provided, the rotations should be applied to each pose in the grid.

For our rotations about Z axis, we will use the following convention: --bracket_range_A_yaw_deg (float) --step_size_A_deg (float)

For our rotations about Y axis, we will use the following convention: --bracket_range_B_pitch_deg (float) --step_size_B_deg (float)

For our rotations about X axis, we will use the following convention: --bracket_range_C_roll_deg (float) --step_size_C_deg (float)

Always apply the A rotation first. For each A position apply the full bracket of B positions. For each AB position apply the full bracket of C positions.

Name each position accordingly to its integer index within the location bracket.

i.e. "pose_x0-y0_A0-B2-C1"

Generating from Kalibr AprilGrid metadata

The AprilGrid metadata file is generated by the Kalibr tool.

{
  "description": "AprilGrid placement metadata for Blender. Kalibr origin is bottom-left corner of tag ID 0.",
  "units": "millimeters",
  "no_margin": true,
  "tag_size_mm": 20.0,
  "spacer_size_mm": 6.0,
  "tag_pitch_mm": 26.0,
  "grid_dimensions_mm": {
    "width": 624.0,
    "height": 624.0
  },
  "image_dimensions_mm": {
    "width": 630.0,
    "height": 630.0
  },
  "kalibr_origin_offset_from_image_bottom_left_mm": {
    "x": 6.0,
    "y": 6.0
  },
  "grid_top_right_from_kalibr_origin_mm": {
    "x": 618.0,
    "y": 618.0
  },
  "tags": [
    {
      "id": 0,
      "bottom_left_mm": {
        "x": 0.0,
        "y": 0.0
      },
      "top_right_mm": {
        "x": 20.0,
        "y": 20.0
      }
    },
    {
      "id": 1,
      "bottom_left_mm": {
        "x": 26.0,
        "y": 0.0
      },
      "top_right_mm": {
        "x": 46.0,
        "y": 20.0
      }
    },
    ...

generate_grid_scan_plan_from_aprilgrid_metadata(metadata_file: Path, z_height: float, ratio_of_target_coverage: float, density_of_poses_relative_fortynine: float, bracket_range_A_yaw_deg: float, step_size_A_deg: float, bracket_range_B_pitch_deg: float, step_size_B_deg: float, bracket_range_C_roll_deg: float, step_size_C_deg: float) -> ScanPlan

this does something sensible.

for example if our target is 1000mm and our ratio_of_target_coverage is 0.5, then we should generate a grid that covers 500mm of the target.

the default grid size is 7x7 poses (49 total). If density_of_poses_relative_fortynine is provided, it should be used to scale the number of poses generated. i.e. if density is 2.0, then we should generate a 14x14 grid (196 total poses).

all of bracket_range_A_yaw_deg: float, step_size_A_deg: float, bracket_range_B_pitch_deg: float, step_size_B_deg: float should default to 10 degrees with 5 degree step size

bracket_range_C_roll_deg: float, step_size_C_deg: float should default to 0 degrees with 5 degree step size

pinhole-radtan is the most commonly used camera model.

in this case pinhole refers to intrinsics (focal length and principal point). This alone could express the lambda which maps x,y,z points in 3D space to u,v 2D points on the image plane - i.e. a simple pinhole camera with no lens, just an infinitesimally small point aperture.

# a pure function:
u, v <= x, y, z

radtan refers to distortion coefficients (k1, k2, p1, p2) which model the effects of a lens mapping points in 2D space incident on the image plane after the influence of the lens, to where those points would have landed with no lens (i.e. a simple pinhole camera)..

# a pure function:
u', v' <= u, v

consuming Kalibr calibration output

we take the raw data from Kalibr and transform it into a pydantic model in our schema.

cam0:
  cam_overlaps: [1, 2, 3, 4, 5]
  camera_model: pinhole
  distortion_coeffs: [0.047671970231275874, -0.052126945675713646, 0.00035674261728729675, -0.00024633230889745236]
  distortion_model: radtan
  intrinsics: [982.4521342422584, 981.4319920949041, 1161.8729899564785, 649.9965088472593]
  resolution: [2304, 1296]
  rostopic: /cam0/image_raw

distortion_coeffs

radtan distortion_coeffs is (as per https://github.com/ethz-asl/kalibr/wiki/supported-models):

• k1 (second order radial distortion)
• k2 (fourth order radial distortion)
• p1 (tangential distortion)
• p2 (tangential distortion)

i.e.

distortion_coeffs: [0.047671970231275874, -0.052126945675713646, 0.00035674261728729675, -0.00024633230889745236]
// k1 = 0.047671970231275874
// k2 = -0.052126945675713646
// p1 = 0.00035674261728729675
// p2 = -0.00024633230889745236

as related by equation:

x_distorted = x(1 + k1*r² + k2*r⁴) + 2p1*xy + p2(r² + 2x²)
y_distorted = y(1 + k1*r² + k2*r⁴) + p1(r² + 2y²) + 2p2*xy

intrinsics

pinhole intrinsics is (as per https://github.com/ethz-asl/kalibr/wiki/supported-models):

• fx (focal length x)
• fy (focal length y)
• cx (principal point x)
• cy (principal point y)

i.e.

intrinsics: [982.4521342422584, 981.4319920949041, 1161.8729899564785, 649.9965088472593]
// fx = 982.4521342422584
// fy = 981.4319920949041
// cx = 1161.8729899564785
// cy = 649.9965088472593

The pinhole intrinsics equation projects 3D camera coordinates to 2D pixel coordinates:

u = fx * (X/Z) + cx
v = fy * (Y/Z) + cy

run a test which imports kalibr data into our schema

From the schema/ uv project root:

uv run -p 3.12 --package camera-schema pytest -q Camera/tests/test_CameraPinholeRadtan.py -s

CameraRig

class CameraRig(BaseModel):
            """
            CameraRig is a set of cameras mounted together.
            Each camera is positioned at a RigidTransform relative to an origin point
            which is where the physical ToolPose is controlled via the robot.
            """
            
            cameras: List[CameraPinholeRadtan]
            camera_poses: List[RigidTransform]
            rig_target_point: Optional[RigidTransform] = None

description

CameraRig is a set of Camera[] mounted together. Each Camera is positioned at RigidTransform relative to an origin point which is where the physical ToolPose is controlled via the robot.

derived from the Kalibr calibration output

eki_interface_srv

ROS2 service for sending one off gcode commands to the robot.

example:

ros2 service call /send_gcode_command eki_interface_srv/srv/SendGcodeCommand "{gcode_line: 'G1 X-400 Y-350 Z400 F100'}"

wire_stickout

Real-time surface height error measurement for WAAM, derived from wire stickout (torch-tip to arc distance) observed through welding-glass cameras.

How it works

Each TSO camera looks through a welding glass at the arc. The brightest saturated blob in the image is the arc core. Its y-position in the image is proportional to the wire stickout distance (torch tip to substrate). When the surface is higher than expected the arc moves up in the image (shorter stickout); when lower, it moves down (longer stickout).

Pipeline:

/TSO_n/image_raw/compressed
    -> detect brightest blob centroid (x, y) normalized to [0,1]
    -> subtract baseline_y (captured at known 10mm stickout on layer 0)
    -> per-channel delta_y
    -> average across all channels -> consensus_delta_y
    -> scale by delta_y_to_delta_z_scale_factor -> err_surface_mm

Quick start

# build
colcon build --packages-select wire_stickout

# run (source first)
source install/setup.bash
ros2 run wire_stickout wire_stickout_blob_measure \
  --ros-args -p tso_namespaces:="['/TSO_0']"

Baseline calibration

During layer 0 on a flat substrate with 10mm wire stickout:

# arm on launch
ros2 run wire_stickout wire_stickout_blob_measure \
  --ros-args -p tso_namespaces:="['/TSO_0']" -p capture_baseline:=true

# or arm at runtime
ros2 param set /wire_stickout_blob_measure capture_baseline true

Collects 128 samples per stream, averages, writes the baseline parameters, and saves to ~/PhotogrammetryWAAM/STATE/TSO/baseline.json. On subsequent launches the baseline is loaded automatically.

Topics

Parameters

Example: Modifying delta_y_to_delta_z_scale_factor

You can override the scale factor parameter (default: 1.0) to change how delta_y is converted to physical units in mm. For example, to set it to 20:

ros2 run wire_stickout wire_stickout_blob_measure \
  --ros-args -p delta_y_to_delta_z_scale_factor:=20

Or, at runtime:

ros2 param set /wire_stickout_blob_measure delta_y_to_delta_z_scale_factor 20.0

This will multiply the consensus delta_y value by 20 to yield /wire_stickout/err_surface_mm.

How to Print

Welcome to the printing tutorial! This guide will walk you through the basic steps to print a model using photogrammetry and Wire Arc Additive Manufacturing (WAAM).

Steps Overview

1. Prepare your 3D model: Ensure your model is clean and ready for printing.
2. Set up the printer: Calibrate and check your WAAM printer.
3. Load the model: Import your 3D model into the printer software.
4. Start printing: Begin the printing process and monitor progress.

For more detailed instructions, refer to the specific sections or consult the reference documentation.