clm_726e101eWe introduce the Segment Anything (SA) project: a new task, model, and dataset for image segmentation.
arXiv 2304.02643 · cs.CV cs.AI cs.LG
Read on arXiv · arXiv non-exclusive license
Claims were extracted automatically by reducer-1 on 2026-11-12. The paper's license does not allow republishing it, so long claims are shortened to a quotation. Read the full text on arXiv.
clm_726e101eWe introduce the Segment Anything (SA) project: a new task, model, and dataset for image segmentation.
res_58c59017Nevertheless, room for improvement remains, as we discuss in §8.
res_25629b23Average annotation time per mask decreased from 34 to 14 seconds as the model improved.
res_8320f012As SAM improved, the average number of masks per image increased from 20 to 44 masks.
res_4ee636c8To estimate mask quality, we randomly sampled 500 images ( ∼50k masks) and asked our professional annotators to improve the quality of all masks in these images.
res_29683c9dSA-1B has higher representation in Europe and Asia & Oceania as well as middle income countries.
res_4e57c32eSA-1B has a substantially higher percentage of images in Europe and Asia & Oceania as well as in middle income countries.
clm_01532199This subset includes both datasets for which SAM outperforms and underperforms RITM according to automatic metrics.
res_dc6696fdSAM significantly outperforms prior interactive segmenters with 1 point and is on par with more points.
res_8c433fe2SAM yields higher results on 16 of the 23 datasets, by as much as ∼47 IoU.
clm_032cac83In particular, with the oracle to perform ambiguity resolution, SAM outperforms RITM on all datasets.
clm_d3521075Notably, it outperforms ViTDet-H on medium and large objects, as well as rare and common objects.
res_53a28620In Fig. 11 we observe that SAM consistently outperforms ViTDet in the human study.
clm_98dcf1ccThe fullysupervised ViTDet outperforms SAM, but the gap shrinks on the higher-quality LVIS masks.
res_6221f209Interestingly, SAM outperforms ViTDet according to human ratings (see Fig. 11).
res_4a9458deWith this data, SAM performs only marginally lower than using all data ( ∼0.5 mIoU).
res_87946c50(Left) Each data engine stage leads to improvements on our 23 dataset suite, and training with only the automatic data (our default) yields similar results to using data from all … (excerpt)
res_55f7045eIn general, we expect dedicated interactive segmentation methods to outperform SAM when many points are provided, e.g. [67].
res_c9887830Finally, there are domain-specific tools, such as [7], that we expect to outperform SAM in their respective domains.
res_99acf1eeWhether SAM achieves the status of a foundation model remains to be seen by how it is used in the community, but regardless we expect the perspective of this work, … (excerpt)
res_420b4d75Additionally, we find that lightly filtering SA-1B masks to discard any that cover more than 90% of the image qualitatively improves results.
res_042c2770All filtering thresholds were selected to achieve both a large number of masks and high mask quality as judged by professional annotators using the method described in §5.
res_533f8c4dWe find that SAM is better at segmenting clothing on those who present predominantly masculine, with disjoint 95% confidence intervals.
res_2e6f0890Our evaluation differs from the standard interactive segmentation evaluation protocol which measures the average number of points needed to achieveX% IoU, with up to 20 points.
clm_50f81b8aGiven our application focus, which requires real-time prompt processing, we expect the best interactive segmentation models to outperform SAM when using a large number of points.
res_5fc8d3e7In our experiments, we observe that RITM outperforms other baselines on our 23 dataset suite with 1 point evaluation.
res_5986c817Since this model produces fewer masks, we further increase the number of points sampled and NMS threshold to 128×128 and 0.95, respectively, obtaining ∼950 masks per image on average.
clm_2881744bAdditionally, this set includes datasets both where SAM outperforms RITM with IoU metrics and vice-versa.
res_cd4d66afFor Table 8, the full 1000 ratings are used to run statistical tests, which show that SAM’s mask quality improvement over ViTDet is statistically significant.
res_2f4df2d5(2) Sharing objectives and key results and meeting weekly with annotators increased the likelihood that annotators improved annotation quality and quantity over time.
clm_b6013766We compared the masks generated by SAM to a baseline state-of-the-art interactive segmentation model, RITM [92], using a perceptual quality scale from 1 to 10. • AP: We used average … (excerpt)
res_827b09a5Examples will follow. – Does the mask correspond to an actual object? – Does the mask have a good boundary? – Does the mask correspond to the provided point or … (excerpt)
res_29e13972Judging Mask Quality (1 of 3) Does the mask have a good boundary? • Errors in the boundary can include: – Incorrect holes in the mask – Incorrect pixels included … (excerpt)
res_887a2cc2This includes the case that there is no mask visible at all. • A low score (2-4): The object is mostly identifiable, but the mask quality is extremely poor (e.g. … (excerpt)
res_1e27091aExample of a mask with a medium score (5-6): The mask clearly corresponds to the plate, but the boundary with the waffle is quite poor.
res_9fa913ffExample of a mask with a medium-to-high score (6-8): the mask has two small-ish regions of poor boundary, at the top of the mask and on the bottom right.
Record your result in the Mearc app on your own machine, then use Publish this result with this paper's id (2304.02643) and the claim id shown next to the claim. Mearc creates a file that holds only what you choose to publish. Add it to the public claim data by pull request; a maintainer other than you reviews it before it counts.