|
View: |
Part 1: Document Description
|
|
Citation |
|
|---|---|
|
Title: |
EdgeSAM |
|
Identification Number: |
doi:10.21979/N9/KF8798 |
|
Distributor: |
DR-NTU (Data) |
|
Date of Distribution: |
2024-09-09 |
|
Version: |
2 |
|
Bibliographic Citation: |
Loy, Chen Change, 2024, "EdgeSAM", https://doi.org/10.21979/N9/KF8798, DR-NTU (Data), V2 |
|
Citation |
|
|
Title: |
EdgeSAM |
|
Identification Number: |
doi:10.21979/N9/KF8798 |
|
Authoring Entity: |
Loy, Chen Change (Nanyang Technological University) |
|
Software used in Production: |
NA |
|
Distributor: |
DR-NTU (Data) |
|
Access Authority: |
Loy, Chen Change |
|
Depositor: |
Loy, Chen Change |
|
Date of Deposit: |
2024-09-09 |
|
Holdings Information: |
https://doi.org/10.21979/N9/KF8798 |
|
Study Scope |
|
|
Keywords: |
Computer and Information Science, Computer and Information Science, Instance segmentation, Segment anything model |
|
Abstract: |
We present EdgeSAM, an accelerated variant of the Segment Anything Model (SAM), optimized for efficient execution on edge devices with minimal compromise in performance. Our approach involves distilling the original ViT-based SAM image encoder into a purely CNN-based architecture, better suited for edge devices. We carefully benchmark various distillation strategies and demonstrate that taskagnostic encoder distillation fails to capture the full knowledge embodied in SAM. To overcome this bottleneck, we include both the prompt encoder and mask decoder in the distillation process, with box and point prompts in the loop, so that the distilled model can accurately capture the intricate dynamics between user input and mask generation. To mitigate dataset bias issues stemming from point prompt distillation, we incorporate a lightweight module within the encoder. As a result, EdgeSAM achieves a 37-fold speed increase compared to the original SAM, and it also outperforms MobileSAM/EfficientSAM, being over 7 times as fast when deployed on edge devices while enhancing the mIoUs on COCO and LVIS by 2.3/1.5 and 3.1/1.6, respectively. It is also the first SAM variant that can run at over 30 FPS on an iPhone 14. |
|
Kind of Data: |
Code and data |
|
Methodology and Processing |
|
|
Sources Statement |
|
|
Data Access |
|
|
Notes: |
S-Lab License 1.0 <br/> Copyright 2022 S-Lab <br/><br/> Redistribution and use for non-commercial purpose in source and binary forms, with or without modification, are permitted provided that the following conditions are met: <br/> 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. <br/> 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. <br/> 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. <br/><br/> THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. <br/><br/> In the event that redistribution and/or use for commercial purpose in source or binary forms, with or without modification is required, please contact the contributor(s) of the work. |
|
Other Study Description Materials |
|
|
Related Studies |
|
|
GitHub page: <a href="https://github.com/chongzhou96/EdgeSAM">Link</a> |
|
|
Project page: <a href="https://www.mmlab-ntu.com/project/edgesam/">Link</a> |
|
|
Hugging Face demo: <a href="https://huggingface.co/spaces/chongzhou/EdgeSAM">Link</a> |
|
|
CutCha Photo app: <a href="https://apps.apple.com/us/app/cutcha-photo/id6478521132">Link</a> |
|
|
Related Publications |
|
|
Citation |
|
|
Identification Number: |
10.48550/arXiv.2312.06660 |
|
Bibliographic Citation: |
Zhou, C., Li, X., Loy, C. C., & Dai, B. (2023). Edgesam: Prompt-in-the-loop distillation for on-device deployment of sam. arXiv preprint arXiv:2312.06660. |
|
Citation |
|
|
Identification Number: |
10356/180234 |
|
Bibliographic Citation: |
Zhou, C., Li, X., Loy, C. C., & Dai, B. (2023). Edgesam: Prompt-in-the-loop distillation for on-device deployment of sam. arXiv preprint arXiv:2312.06660. |
|
Label: |
EdgeSAM- Prompt-In-the-Loop Distillation for On-Device Deployment of SAM.pdf |
|
Notes: |
application/pdf |