Top Related Projects
Blender glTF 2.0 importer and exporter
ARCore SDK for Android Studio
2D skeletal animation runtimes for Spine.
Example content for Unity projects based on AR Foundation
Quick Overview
AR Cut & Paste is an experimental project that allows users to "cut" objects from their real-world environment using a smartphone camera and "paste" them into a desktop application. It uses augmented reality (AR) and machine learning techniques to segment objects from the camera feed and transfer them to a computer in real-time.
Pros
- Innovative concept that bridges the gap between physical and digital worlds
- Utilizes cutting-edge technologies like AR and machine learning
- Provides a user-friendly interface for capturing and transferring real-world objects
- Potential applications in various fields such as design, architecture, and education
Cons
- Experimental project, not yet ready for production use
- Requires specific hardware setup (smartphone and computer)
- Limited to 2D object transfer, not full 3D reconstruction
- May have accuracy issues depending on lighting conditions and object complexity
Getting Started
To set up and run the AR Cut & Paste project:
-
Clone the repository:
git clone https://github.com/cyrildiagne/ar-cutpaste.git cd ar-cutpaste
-
Install dependencies:
pip install -r requirements.txt
-
Run the server:
python server.py
-
Install the AR Cut & Paste app on your smartphone (iOS only)
-
Ensure both your smartphone and computer are on the same Wi-Fi network
-
Open the app on your smartphone and point it at an object you want to capture
-
Use the desktop application to receive and paste the captured object
Note: This project is experimental and may require additional setup or troubleshooting. Refer to the repository's README for more detailed instructions and requirements.
Competitor Comparisons
Blender glTF 2.0 importer and exporter
Pros of glTF-Blender-IO
- Robust integration with Blender for importing/exporting glTF files
- Supports advanced 3D features like animations, materials, and textures
- Actively maintained by Khronos Group, ensuring compatibility with glTF standards
Cons of glTF-Blender-IO
- Limited to Blender ecosystem, not applicable for AR/mobile applications
- Steeper learning curve for users unfamiliar with 3D modeling software
- Lacks real-time AR functionality found in ar-cutpaste
Code Comparison
glTF-Blender-IO (Python):
class glTF2ImportUserExtension:
def __init__(self):
self.properties = bpy.context.scene.glTF2UserExtensions
def gather_import_properties(self):
return self.properties.get_dict()
ar-cutpaste (JavaScript):
const video = document.getElementById('video');
const canvas = document.getElementById('canvas');
const ctx = canvas.getContext('2d');
video.addEventListener('play', () => {
function step() {
ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
requestAnimationFrame(step);
}
requestAnimationFrame(step);
});
The code snippets highlight the different focus areas of each project. glTF-Blender-IO deals with 3D data import/export in Blender, while ar-cutpaste handles real-time video processing for AR applications.
ARCore SDK for Android Studio
Pros of ARCore Android SDK
- Comprehensive AR development toolkit for Android
- Backed by Google, ensuring long-term support and updates
- Extensive documentation and community resources
Cons of ARCore Android SDK
- Steeper learning curve for beginners
- Limited to Android platform
- Requires more setup and configuration
Code Comparison
ARCore Android SDK:
Session session = new Session(this);
Config config = new Config(session);
config.setUpdateMode(Config.UpdateMode.LATEST_CAMERA_IMAGE);
session.configure(config);
frame = session.update();
AR Cut & Paste:
import cv2
import numpy as np
image = cv2.imread('input.jpg')
mask = np.zeros(image.shape[:2], np.uint8)
bgdModel = np.zeros((1,65), np.float64)
fgdModel = np.zeros((1,65), np.float64)
The ARCore Android SDK code snippet demonstrates session initialization and configuration in Java, while the AR Cut & Paste example shows image processing setup in Python. ARCore focuses on AR session management, whereas AR Cut & Paste emphasizes image manipulation for the cut-and-paste functionality.
2D skeletal animation runtimes for Spine.
Pros of spine-runtimes
- Comprehensive 2D animation toolset for game development
- Supports multiple platforms and programming languages
- Extensive documentation and community support
Cons of spine-runtimes
- Steeper learning curve for beginners
- Requires a commercial license for production use
- Limited to 2D skeletal animations
Code Comparison
spine-runtimes (Java):
SkeletonRenderer renderer = new SkeletonRenderer();
SkeletonData skeletonData = new SkeletonJson().readSkeletonData(Gdx.files.internal("spineboy.json"));
Skeleton skeleton = new Skeleton(skeletonData);
Animation animation = skeletonData.findAnimation("walk");
ar-cutpaste (Python):
import cv2
import numpy as np
def segment_image(image):
mask = np.zeros(image.shape[:2], np.uint8)
bgdModel = np.zeros((1, 65), np.float64)
fgdModel = np.zeros((1, 65), np.float64)
spine-runtimes focuses on 2D skeletal animations for games, offering a robust framework for creating complex animations across multiple platforms. It provides extensive features but requires a commercial license and has a steeper learning curve.
ar-cutpaste, on the other hand, is an open-source project for AR-based image manipulation, allowing users to "cut" objects from the real world and "paste" them into digital environments. It's more accessible for beginners but has a narrower focus compared to spine-runtimes.
The code snippets highlight the different purposes of these projects: spine-runtimes deals with skeletal animation data, while ar-cutpaste focuses on image segmentation for AR applications.
Example content for Unity projects based on AR Foundation
Pros of arfoundation-samples
- Comprehensive set of AR samples covering various AR functionalities
- Built on Unity's AR Foundation, offering cross-platform compatibility
- Regularly updated with new features and improvements
Cons of arfoundation-samples
- Steeper learning curve for developers new to Unity
- Requires Unity engine, which may be overkill for simple AR projects
- More complex setup process compared to ar-cutpaste
Code Comparison
ar-cutpaste (Python):
def segment(img):
inputs = tf.convert_to_tensor(img[tf.newaxis, ...], dtype=tf.float32)
outputs = model(inputs)
return outputs['output_0'][0]
arfoundation-samples (C#):
void OnARObjectPlaced(ARObjectPlacementEventArgs args)
{
var placedObject = args.placedObject;
placedObject.transform.localScale = Vector3.one * 0.1f;
placedObject.AddComponent<ARSelectionInteractable>();
}
The ar-cutpaste repository focuses on a specific AR use case (cut and paste objects), while arfoundation-samples provides a broader range of AR functionalities. ar-cutpaste is more accessible for Python developers, while arfoundation-samples caters to Unity developers and offers more extensive AR capabilities at the cost of increased complexity.
Convert
designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual CopilotREADME
AR Cut & Paste
An AR+ML prototype that allows cutting elements from your surroundings and pasting them in an image editing software.
Although only Photoshop is being handled currently, it may handle different outputs in the future.
Demo & more infos: Thread
â ï¸ This is a research prototype and not a consumer / photoshop user tool.
Update 2020.05.11: If you're looking for an easy to use app based on this research, head over to https://clipdrop.co
Modules
This prototype runs as 3 independent modules:
-
The mobile app
- Check out the /app folder for instructions on how to deploy the app to your mobile.
-
The local server
- The interface between the mobile app and Photoshop.
- It finds the position pointed on screen by the camera using screenpoint
- Check out the /server folder for instructions on configuring the local server
-
The object detection / background removal service
- For now, the salience detection and background removal are delegated to an external service
- It would be a lot simpler to use something like DeepLap directly within the mobile app. But that hasn't been implemented in this repo yet.
Usage
1 - Configure Photoshop
- Go to "Preferences > Plug-ins", enable "Remote Connection" and set a friendly password that you'll need later.
- Make sure that your PS document settings match those in
server/src/ps.py
, otherwise only an empty layer will be pasted. - Also make sure that your document has some sort of background. If the background is just blank, SIFT will probably not have enough feature to do a correct match.
2 - Setup the external salience object detection service
Option 1: Set up your own model service (requires a CUDA GPU)
-
As mentioned above, for the time being, you must deploy the BASNet model (Qin & al, CVPR 2019) as an external HTTP service using this BASNet-HTTP wrapper (requires a CUDA GPU)
-
You will need the deployed service URL to configure the local server
-
Make sure to configure a different port if you're running BASNet on the same computer as the local service
Option 2: Use a community provided endpoint
A public endpoint has been provided by members of the community. This is useful if you don't have your own CUDA GPU or do not want to go through the process of running the servce on your own.
Use this endpoint by launching the local server with --basnet_service_ip http://u2net-predictor.tenant-compass.global.coreweave.com
3 - Configure and run the local server
- Follow the instructions in /server to setup & run the local server.
4 - Configure and run the mobile app
- Follow the instructions in /app to setup & deploy the mobile app.
Thanks and Acknowledgements
- BASNet code for 'BASNet: Boundary-Aware Salient Object Detection code', Xuebin Qin, Zichen Zhang, Chenyang Huang, Chao Gao, Masood Dehghan and Martin Jagersand
- RunwayML for the Photoshop paste code
- CoreWeave for hosting the public U^2Net model endpoint on Tesla V100s
Top Related Projects
Blender glTF 2.0 importer and exporter
ARCore SDK for Android Studio
2D skeletal animation runtimes for Spine.
Example content for Unity projects based on AR Foundation
Convert
designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual Copilot