FoundationPose perception pipeline setup

Purpose

Prepare the FoundationPose perception pipeline for depth, segmentation, and pose inference. Environment installation and engine construction belong here; dataset adaptation, inference, and pose evaluation belong to foundationpose-pipeline when that skill is installed.

Requirements

Work from the product checkout, not the installed skill directory. Find the user's checkout by checking for pyproject.toml (project foundationpose-perception-pipeline), tools/build_tao_engine.py, and config/defaults.yaml. If absent and setup was requested, clone the product URL above into the user's workspace and enter it. For advice-only requests, use the supplied diagnostics without cloning or installing anything. Commands below use paths relative to the product root; references/ links are relative to this skill.

Read the checkout's README.md Requirements and Install sections for the matching revision. The supported stack requires Linux x86_64, glibc >= 2.38, GLIBCXX_3.4.31, NVIDIA driver >= 580, a CUDA toolkit >= 12.8 with nvcc, Python 3.12, uv, Git, Docker with GPU access, wget, and unzip. Start GPU sizing at 24 GB and measure the densest scene; 32 GB was tested. Budget depth-cache disk as roughly width * height * 4 * 3 bytes per scene, plus predictions and models.

Keep sibling directories for sam3/, foundation-pose-inference-library/, and models/ beside the product checkout. models/ contains the deployable ONNX and engine, not FoundationStereo source.

Instructions

1. Preflight before installing. Check glibc, GLIBCXX, driver, nvcc, tools, and Docker GPU access. An Ubuntu 22.04 host with glibc 2.35 cannot load the shipped FoundationPose library; report the unsupported runtime and stop setup there. Do not replace system libc or try to solve this with LD_LIBRARY_PATH. See installation.

2. Install into the product's Python 3.12 venv. Follow installation for uv, SAM3, the FoundationPose build, and TAO Deploy. On a fresh venv use uv sync --extra foundationpose; on an existing venv use uv sync --inexact --extra foundationpose to preserve out-of-band packages.

3. Verify checkpoint access. SAM3 is gated at Hugging Face; an existing authorized token or usable cached checkpoint is sufficient. Request user action only if access is missing. FoundationPose and the documented FoundationStereo export are public Hugging Face downloads; credential hunting is not the first response to a network failure.

4. Prepare the depth engine. Read engine construction. Use the user's ONNX location or the sibling models/ directory. Adapt the dataset before measuring the engine shape; use tools/bop_adapt/adapt.py --config <profile> --src <source> as described in the checkout's README Dataset adaptation section. Only registered adapters are supported. Build with --shape-from-scene on an adapted scene and FP32 unless the user requests a precision experiment. Set overrides.depth.engine in config/<profile>.yaml.

5. Set runtime paths and verify. From the product root:

   source .venv/bin/activate
   export FOUNDATIONPOSE_ROOT="$(realpath ../foundation-pose-inference-library)"
   PIPELINE_SITE="$(realpath .venv/lib/python3.12/site-packages)"
   export LD_LIBRARY_PATH="${PIPELINE_SITE}/tensorrt_libs:${PIPELINE_SITE}/nvidia/cu13/lib:${LD_LIBRARY_PATH:-}"
   python tools/verify_sam3.py
   python tools/verify_foundationpose.py
   python tools/verify_foundationstereo.py --config <profile> --engine <engine-path>
   python test/check_engine_depth_smoke.py --config <profile> --engine <engine-path>
   

   The first three verify components; the last also needs an adapted dataset. Expect backend=tao, normalization=imagenet, the intended fixed shape, and no cropping N rows warning. An unloaded or unavailable engine is an incomplete verification, not a pass.

Troubleshooting

Symptom	Action
GLIBC_2.38 not found	Use a supported OS/runtime; a venv or library search path cannot upgrade host libc.
libcudart.so.13 missing	Check the product venv runtime wheels and absolute library paths before retrying pose.
SAM3 breaks after sync	Use --inexact; confirm numpy 1.26.x and reinstall the sibling SAM3 package if pruned.
pycuda build cannot find cuda.h	Check the CUDA toolkit, nvcc on PATH, or CUDA_ROOT.
TAO import or dependency conflict	Use TAO Deploy 7.1.0 with --no-deps; sync declared dependencies with --inexact.
Engine sidecar mismatch or cropping	Rebuild for this GPU, TensorRT version, precision, and adapted scene shape.

Examples

• "Install the FoundationPose perception pipeline on this Ubuntu 24.04 GPU machine."
• "The pipeline cannot load libcudart.so.13 after I moved the checkout."
• "Build the TAO depth engine for my adapted T-LESS scenes."

Limitations and completion

Engines are machine-specific and must not be committed. With no dataset, download the ONNX and report shape-dependent engine construction and scene validation as pending; do not invent a rig shape. The pipeline's Apache license does not cover separately downloaded model weights; retain their upstream terms and SAM3's access requirements.

Report which preflight, install, checkpoint, engine, and verification steps actually passed, the checkout and engine paths, versions used, and remaining blockers. Do not equate installation or a smoke check with measured pose accuracy.