Integrate with Segment Anything Model (SAM)

[wkentaro]

• Close segment anything support #1260
• Close Implement SAM #1257

TODO

• Positive/negative click to run AI-assisted polygon annotation
• Fix weird behavior when annotating many images Integrate with Segment Anything Model (SAM) #1262 (comment)
• Support portrait images Integrate with Segment Anything Model (SAM) #1262 (comment)
• Support 1-click annotation
• Support double_click="close"
• Real-time feedback without click Sam realtime render #1265
• Dropdown menu to change AI models

Installation

1. Install conda

# Linux/Mac: command to install anaconda3 and labelme
curl -L https://github.com/wkentaro/dotfiles/raw/main/local/bin/install_anaconda3.sh | bash -s .
source .anaconda3/bin/activate

# Windows: check https://www.continuum.io/downloads

2. Install Labelme with sam branch.

git clone https://github.com/wkentaro/labelme.git
cd labelme
git checkout sam
pip install -e .

Usage

labelme examples/instance_segmentation/data_annotated

1. Select "Create AI-Polygon"
2. Do click for the region you want to include
3. Do shift+click for the region you want to exclude
4. Do ctrl/cmd+click or ctrl/cmd+shift+click or double-click to finish
5. Press esc to go back and continue clicking

Untitled.mp4

---

NOTE: The response for the first annotation is SLOW (~20s) because the AI model needs to extract image embedding from the entire image, but it is cached in the second you annotate.

[wkentaro]

@Johnny-liqiang Yes, I'm using ONNX exported model of Segment-Anything: https://github.com/facebookresearch/segment-anything#onnx-export

This is the forked repo for download and export scripts: https://github.com/wkentaro/segment-anything/tree/labelme

[not-a-swiss-conductor]

I use this formal command for test, labelme examples/instance_segmentation/data_annotated, but it can't work!!!
I create a new environment to develop it, python = 3.10, numpy = 1.24, onnxruntime = 1.14.
NOTICE: python need >=3.9, for onnxruntime and numpy compatibility.

Exception in thread Thread-1 (get_image_embedding):
Traceback (most recent call last):
  File "C:\Users\ysccc\anaconda3\envs\labelme\lib\threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "C:\Users\ysccc\anaconda3\envs\labelme\lib\threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "d:\code\labelme-sam\labelme\ai\models\segment_anything.py", line 55, in get_image_embedding
    self._image_embedding = compute_image_embedding(
  File "d:\code\labelme-sam\labelme\ai\models\segment_anything.py", line 107, in compute_image_embedding
    output = encoder_session.run(output_names=None, input_feed={"x": x})
  File "C:\Users\ysccc\anaconda3\envs\labelme\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 196, in run
    raise ValueError("Model requires {} inputs. Input Feed contains {}".format(num_required_inputs, num_inputs))
ValueError: Model requires 6 inputs. Input Feed contains 1

[healthonrails]

You need to switch to the "sam" branch and then proceed to install it.

@not-a-swiss-conductor Are you running on Mac or Windows?, for me the Window, this Labelme with SAM doesn't work as well, it doesn't show creat AI Polygon this function when I right-clicked!

I believe the following install command only works for Mac or Linux Systems:

git clone https://github.com/wkentaro/labelme.git
cd labelme
curl -L https://github.com/wkentaro/dotfiles/raw/main/local/bin/install_anaconda3.sh | bash -s .
source .anaconda3/bin/activate
pip install -e .

If you want to test SAM with Labelme on Windows: SAM model 91M ViT-B SAM 508M ViT-L or 636M ViT-H have been integrated into Anylabeling tool:

I have installed it on Windows as well! very easy to use:

Install Anaconda
then conda activate any label environment
then just run anylabeling

The best model I would say from SAM is this ViT-H with 636M numbers of parameters. It basically segments most the objects well! the other two models from SAM are not performing as well as this!

[wkentaro]

@not-a-swiss-conductor @healthonrails
If you install Anaconda from https://www.continuum.io/downloads, it should also work on Windows. (I updated the instruction in my first post.)

[wkentaro]

@f-macfarlane95 The SAM model can definitely handle splitted objects, but we need to support it with group_id correctly in Labelme. Let's add the feature of supporting splitted object annotation later.

[d710055071]

hello，
Do you have any plans to add rectangular boxes?

[FreeSoaring]

The first time I clicked 'Create AI-Polygon', I got this error message.

(labelme) PS C:\Code\labelme> labelme examples/instance_segmentation/data_annotated
[INFO   ] __init__:get_config:70 - Loading config file from: C:\Users\Administrator\.labelmerc
..SegmentAnythingModel  __init__!!!!!!
Computing MD5: C:\Users\Administrator\.cache/gdown\https-COLON--SLASH--SLASH-github.aaakk.us.kg-SLASH-wkentaro-SLASH-labelme-SLASH-releases-SLASH-download-SLASH-sam-20230416-SLASH-sam_vit_l_0b3195.quantized.encoder.onnx
MD5 matches: C:\Users\Administrator\.cache/gdown\https-COLON--SLASH--SLASH-github.aaakk.us.kg-SLASH-wkentaro-SLASH-labelme-SLASH-releases-SLASH-download-SLASH-sam-20230416-SLASH-sam_vit_l_0b3195.quantized.encoder.onnx
Computing MD5: C:\Users\Administrator\.cache/gdown\https-COLON--SLASH--SLASH-github.aaakk.us.kg-SLASH-wkentaro-SLASH-labelme-SLASH-releases-SLASH-download-SLASH-sam-20230416-SLASH-sam_vit_l_0b3195.quantized.decoder.onnx
MD5 matches: C:\Users\Administrator\.cache/gdown\https-COLON--SLASH--SLASH-github.aaakk.us.kg-SLASH-wkentaro-SLASH-labelme-SLASH-releases-SLASH-download-SLASH-sam-20230416-SLASH-sam_vit_l_0b3195.quantized.decoder.onnx
Traceback (most recent call last):
  File "c:\code\labelme\labelme\app.py", line 372, in <lambda>
    lambda: self.toggleDrawMode(False, createMode="ai_polygon"),
  File "c:\code\labelme\labelme\app.py", line 969, in toggleDrawMode
    self.canvas.createMode = createMode
  File "c:\code\labelme\labelme\widgets\canvas.py", line 133, in createMode
    image=labelme.utils.img_qt_to_arr(self.pixmap.toImage())
  File "c:\code\labelme\labelme\utils\image.py", line 61, in img_qt_to_arr
    bytes_ = img_qt.bits().asstring(w * h * d // 8)
AttributeError: 'memoryview' object has no attribute 'asstring'

AI did not work.

The program crashed with an error message when I tried to select an area on the image after clicking "Create AI-Polygon" for the second time.

Traceback (most recent call last):
  File "c:\code\labelme\labelme\widgets\canvas.py", line 754, in paintEvent
    points = self._ai_model.predict_polygon_from_points(
  File "c:\code\labelme\labelme\ai\models\segment_anything.py", line 77, in predict_polygon_from_points
    image_embedding = self._get_image_embedding()
  File "c:\code\labelme\labelme\ai\models\segment_anything.py", line 70, in _get_image_embedding
    if self._thread is not None:
AttributeError: 'SegmentAnythingModel' object has no attribute '_thread'
QBackingStore::endPaint() called with active painter; did you forget to destroy it or call QPainter::end() on it?
QBackingStore::endPaint() called with active painter; did you forget to destroy it or call QPainter::end() on it?
QPainter::begin: Painter already active
(labelme) PS C:\Code\labelme> 

[wkentaro]

Now it supports all 3 SAM models.

[Rusab]

Where can I get the SAM integrated Labelme?

[VicencPB]

Hi, thank you very much for making this software available, it's really useful.

I have been using the 'standard' labelme until now and I would like to try the 'sam version', but when I try to install it there appears an error.

• I have already conda installed, with Python 3.8 (Ubuntu 20.04.5 LTS), so I didn't do the first part of the installation:

---

Linux/Mac: command to install anaconda3 and labelme

curl -L https://github.com/wkentaro/dotfiles/raw/main/local/bin/install_anaconda3.sh | bash -s .
source .anaconda3/bin/activate

Just did the 2nd part of the installation instructions:

git clone https://github.com/wkentaro/labelme.git
cd labelme

... and when I run 'git checkout sam'

I get the following error message:

error: pathspec 'sam' did not match any file(s) known to git

I would appreciate any help to install it!

Thank you in advance!

[OceanWong1991]

@VicencPB Actually, The author already merge sam to master, and delete sam branch.

[OceanWong1991]

Where can I get the SAM integrated Labelme?

@Rusab JUST Try the latest version!

[wkentaro]

@OceanWong1991 Thanks for answering the questions!

Yes, the latest version should include everything you need to use SAM.

@VicencPB @Rusab Please follow the installation guide. To get the latest version, you need either of them:

1. install via pip: pip install labelme --upgrade
2. download the pre-built binary from the releases.

[VicencPB]

Thank you!

Now it works!