Posted inAI

Auto-labeling with Grounding DINO and SAM: Shrink Dataset Prep from Weeks to Hours

AI tutorial - IT technology blog
AI tutorial - IT technology blog

The 2 AM Data Bottleneck

It was 2 AM, and my team was staring at a hard deadline for a custom object detection project. We needed to identify oxidized rivets on turbine blades—components that don’t exist in any public dataset. We had 15,000 raw images. The quote for manual labeling via CVAT was three weeks. We had exactly three days.

This is the hidden friction point in Computer Vision. You have the latest transformer architecture and plenty of GPU compute. Yet, you’re stuck waiting for humans to click corners of boxes for $15 an hour. If you have ever spent a weekend outlining microscopic cells or shipping containers, you know the frustration. The bottleneck isn’t the training cycle; it’s the sheer volume of manual labor required before the first epoch even starts.

Why Manual Labeling Hits a Wall

Our delay wasn’t caused by a lack of manpower. It was the inherent inefficiency of the human-in-the-loop workflow. When we audited our process, three major friction points emerged:

  • The Exhaustion Gap: After image 400, human precision nose-dives. Bounding boxes shift by 5-10 pixels, and similar-looking classes get swapped.
  • The Expert Requirement: You can’t outsource the labeling of 22nm semiconductor defects to a generic labeling farm. You need engineers who know what a defect looks like. Their time is too valuable for drawing boxes.
  • The Cold Start Paradox: You need a model to help label data, but you can’t train that model because you don’t have labeled data yet. It’s a frustrating loop.

Evaluating the Alternatives

Before building our current pipeline, we explored several ways to break the deadlock:

1. Pre-trained YOLO Models

Using a YOLOv8 model pre-trained on COCO is the standard first step. It’s incredibly fast, but it is limited to 80 generic classes. If your target isn’t a “person” or a “car,” the model is essentially blind. It fails to generalize to niche industrial parts without existing labels.

2. Model-in-the-Loop

This approach involves a human correcting a model’s rough guesses. While it’s faster than starting from scratch, the human remains the primary speed limiter. You are still paying for every second of human attention and mouse movement.

3. Zero-Shot Foundation Models

This was our breakthrough. By pairing Grounding DINO with the Segment Anything Model (SAM), we built a pipeline that understands context without prior training. DINO finds the object using text prompts; SAM handles the pixel-perfect masking. No manual boxes required.

The Logic: Grounding DINO + SAM

Think of Grounding DINO as the “eyes” that understand natural language. You provide a prompt like “rusted bolt on steel beam,” and it returns a bounding box. However, these boxes are often loose or slightly off-center. That is where SAM comes in. SAM takes that box as a spatial prompt and shrink-wraps it into a precise, high-fidelity mask.

I have deployed this stack in production environments ranging from ag-tech to heavy manufacturing. It consistently slashes the time to iterate on new datasets from weeks to a few hours of script execution.

Building the Auto-Labeling Pipeline

You will need a GPU with at least 12GB of VRAM to run this effectively. Attempting to run these foundation models on a CPU will result in inference times of 30+ seconds per image—far too slow for large batches.

Step 1: Environment Setup

pip install torch torchvision
pip install git+https://github.com/IDEA-Research/GroundingDINO.git
pip install git+https://github.com/facebookresearch/segment-anything.git
pip install opencv-python supervision pycocotools

Step 2: Object Detection with Grounding DINO

We initialize Grounding DINO to detect objects based on text strings. Unlike traditional detectors, we don’t use class IDs. We use English descriptions.

from groundingdino.util.inference import load_model, load_image, predict
import cv2

CONFIG_PATH = "GroundingDINO_SwinT_OGC.py"
WEIGHTS_PATH = "groundingdino_swint_ogc.pth"
model = load_model(CONFIG_PATH, WEIGHTS_PATH)

IMAGE_PATH = "turbine_blade_01.jpg"
TEXT_PROMPT = "oxidized rivet"
BOX_THRESHOLD = 0.35
TEXT_THRESHOLD = 0.25

image_source, image = load_image(IMAGE_PATH)

boxes, logits, phrases = predict(
    model=model, 
    image=image, 
    caption=TEXT_PROMPT, 
    box_threshold=BOX_THRESHOLD, 
    text_threshold=TEXT_THRESHOLD
)

Step 3: Generating Masks with SAM

Now we feed those DINO boxes into SAM. This step is vital for instance segmentation tasks where box-level accuracy isn’t enough.

from segment_anything import SamPredictor, sam_model_registry

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth").to("cuda")
predictor = SamPredictor(sam)
predictor.set_image(image_source)

# DINO boxes are normalized [cx, cy, w, h]. 
# We convert them to pixel-based [x1, y1, x2, y2] for SAM.
for box in boxes:
    masks, scores, _ = predictor.predict(
        box=box.numpy(),
        multimask_output=False
    )
    # The result is a high-quality boolean mask for training

Moving to Production: The Distillation Strategy

Grounding DINO and SAM are heavy. Running a ViT-H SAM model for real-time inference on an edge device is impossible. Instead, use this pipeline to generate 10,000 “silver-standard” labels overnight. Then, use those labels to train a lightweight model like YOLOv8 or RT-DETR.

This strategy gives you the best of both worlds. You get the deep reasoning of foundation models during the labeling phase and millisecond-level inference speeds on your production hardware.

Filtering for Quality

Automated systems occasionally hallucinate. In our tests, Grounding DINO sometimes flags shadows as objects. I recommend a confidence filter: if the logit score is below 0.45, flag that image for a 5-second human review. This reduces the manual workload by roughly 92% while maintaining high dataset integrity.

Final Thoughts

The era of clicking boxes for weeks is ending. If your team is still manually labeling every frame, you are losing time that should be spent on model optimization or edge-case engineering. By chaining Grounding DINO and SAM, you transform labeling from a manual grind into a prompt engineering task.

Run a small pilot batch to tune your thresholds. Once the masks look sharp, let the script process your entire backlog. You will wake up to a fully labeled dataset ready for the training pipeline.

Share:
Tags: