MearcAll papers

arXiv 2304.02643 · cs.CV cs.AI cs.LG

Segment Anything

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, and 6 more

Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.

No reproduction recordedclm_726e101e
We introduce the Segment Anything (SA) project: a new task, model, and dataset for image segmentation.

Abstract · page 1 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_58c59017
Nevertheless, room for improvement remains, as we discuss in §8.

Introduction · page 2 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_25629b23
Average annotation time per mask decreased from 34 to 14 seconds as the model improved.

Segment Anything Data Engine · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8320f012
As SAM improved, the average number of masks per image increased from 20 to 44 masks.

Segment Anything Data Engine · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4ee636c8
To estimate mask quality, we randomly sampled 500 images ( ∼50k masks) and asked our professional annotators to improve the quality of all masks in these images.

Segment Anything Dataset · page 6 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_29683c9d
SA-1B has higher representation in Europe and Asia & Oceania as well as middle income countries.

Segment Anything RAI Analysis · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4e57c32e
SA-1B has a substantially higher percentage of images in Europe and Asia & Oceania as well as in middle income countries.

Segment Anything RAI Analysis · page 7 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_01532199
This subset includes both datasets for which SAM outperforms and underperforms RITM according to automatic metrics.

Zero-Shot Single Point Valid Mask Evaluation · page 8 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_dc6696fd
SAM significantly outperforms prior interactive segmenters with 1 point and is on par with more points.

Zero-Shot Single Point Valid Mask Evaluation · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_8c433fe2
SAM yields higher results on 16 of the 23 datasets, by as much as ∼47 IoU.

Zero-Shot Single Point Valid Mask Evaluation · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_032cac83
In particular, with the oracle to perform ambiguity resolution, SAM outperforms RITM on all datasets.

Zero-Shot Single Point Valid Mask Evaluation · page 9 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_d3521075
Notably, it outperforms ViTDet-H on medium and large objects, as well as rare and common objects.

Zero-Shot Object Proposals · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_53a28620
In Fig. 11 we observe that SAM consistently outperforms ViTDet in the human study.

Zero-Shot Instance Segmentation · page 10 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_98dcf1cc
The fullysupervised ViTDet outperforms SAM, but the gap shrinks on the higher-quality LVIS masks.

Zero-Shot Instance Segmentation · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_6221f209
Interestingly, SAM outperforms ViTDet according to human ratings (see Fig. 11).

Zero-Shot Instance Segmentation · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_4a9458de
With this data, SAM performs only marginally lower than using all data ( ∼0.5 mIoU).

Ablations · page 11 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_87946c50
(Left) Each data engine stage leads to improvements on our 23 dataset suite, and training with only the automatic data (our default) yields similar results to using data from all … (excerpt)

Ablations · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_55f7045e
In general, we expect dedicated interactive segmentation methods to outperform SAM when many points are provided, e.g. [67].

Discussion · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_c9887830
Finally, there are domain-specific tools, such as [7], that we expect to outperform SAM in their respective domains.

Discussion · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_99acf1ee
Whether SAM achieves the status of a foundation model remains to be seen by how it is used in the community, but regardless we expect the perspective of this work, … (excerpt)

Discussion · page 12 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_420b4d75
Additionally, we find that lightly filtering SA-1B masks to discard any that cover more than 90% of the image qualitatively improves results.

Segment Anything Model and Task Details · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_042c2770
All filtering thresholds were selected to achieve both a large number of masks and high mask quality as judged by professional annotators using the method described in §5.

Automatic Mask Generation Details · page 18 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_533f8c4d
We find that SAM is better at segmenting clothing on those who present predominantly masculine, with disjoint 95% confidence intervals.

RAI Additional Details · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2e6f0890
Our evaluation differs from the standard interactive segmentation evaluation protocol which measures the average number of points needed to achieveX% IoU, with up to 20 points.

Zero-Shot Single Point Valid Mask Evaluation · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_50f81b8a
Given our application focus, which requires real-time prompt processing, we expect the best interactive segmentation models to outperform SAM when using a large number of points.

Zero-Shot Single Point Valid Mask Evaluation · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5fc8d3e7
In our experiments, we observe that RITM outperforms other baselines on our 23 dataset suite with 1 point evaluation.

Zero-Shot Single Point Valid Mask Evaluation · page 19 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_5986c817
Since this model produces fewer masks, we further increase the number of points sampled and NMS threshold to 128×128 and 0.95, respectively, obtaining ∼950 masks per image on average.

Zero-Shot Object Proposals · page 21 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_2881744b
Additionally, this set includes datasets both where SAM outperforms RITM with IoU metrics and vice-versa.

Human Study Experimental Design · page 23 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_cd4d66af
For Table 8, the full 1000 ratings are used to run statistical tests, which show that SAM’s mask quality improvement over ViTDet is statistically significant.

Human Study Experimental Design · page 24 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_2f4df2d5
(2) Sharing objectives and key results and meeting weekly with annotators increased the likelihood that annotators improved annotation quality and quantity over time.

Data Annotation Card · page 27 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedclm_b6013766
We compared the masks generated by SAM to a baseline state-of-the-art interactive segmentation model, RITM [92], using a perceptual quality scale from 1 to 10. • AP: We used average … (excerpt)

If you have any aggregated socio-demographic statistics about your anno- · page 28 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_827b09a5
Examples will follow. – Does the mask correspond to an actual object? – Does the mask have a good boundary? – Does the mask correspond to the provided point or … (excerpt)

If you have any aggregated socio-demographic statistics about your anno- · page 29 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_29e13972
Judging Mask Quality (1 of 3) Does the mask have a good boundary? • Errors in the boundary can include: – Incorrect holes in the mask – Incorrect pixels included … (excerpt)

If you have any aggregated socio-demographic statistics about your anno- · page 29 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_887a2cc2
This includes the case that there is no mask visible at all. • A low score (2-4): The object is mostly identifiable, but the mask quality is extremely poor (e.g. … (excerpt)

Annotation Guidelines · page 30 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_1e27091a
Example of a mask with a medium score (5-6): The mask clearly corresponds to the plate, but the boundary with the waffle is quite poor.

Annotation Guidelines · page 30 · Automated extraction, not reviewed · 0 reviewed reproductions

No reproduction recordedres_9fa913ff
Example of a mask with a medium-to-high score (6-8): the mask has two small-ish regions of poor boundary, at the top of the mask and on the bottom right.

Annotation Guidelines · page 30 · Automated extraction, not reviewed · 0 reviewed reproductions

Reproduced one of these claims?

Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2304.02643) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.