Top Related Projects
Fast, secure, efficient backup program
Deduplicating archiver with compression and authenticated encryption.
Store securely encrypted backups in the cloud!
Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
Open Source Continuous File Synchronization
Quick Overview
Duplicacy is an open-source, cross-platform cloud backup tool that supports various storage backends. It offers a unique lock-free deduplication algorithm, allowing multiple clients to back up to the same storage without interference. Duplicacy provides both a command-line interface and a web-based GUI for managing backups.
Pros
- Lock-free deduplication allows multiple clients to back up simultaneously
- Supports numerous storage backends, including cloud services and local storage
- Offers strong encryption for data security
- Provides both CLI and web-based GUI interfaces
Cons
- Steeper learning curve compared to some other backup solutions
- Limited documentation for advanced features
- Web GUI is not as feature-rich as the CLI version
Code Examples
# Initialize a repository
duplicacy init mybackup s3://my-bucket-name
# Perform a backup
duplicacy backup -threads 4
# Restore files
duplicacy restore -r 1234 -threads 4 /path/to/restore
# List snapshots
duplicacy list
# Prune old snapshots
duplicacy prune -keep 0:365 -keep 30:180 -keep 7:30 -keep 1:7
Getting Started
- Install Duplicacy from the releases page
- Initialize a repository:
duplicacy init <snapshot-id> <storage-url>
- Create a backup:
duplicacy backup -threads 4
- To restore files:
duplicacy restore -r <revision> -threads 4 <path>
For more detailed instructions and options, refer to the Duplicacy Wiki.
Competitor Comparisons
Fast, secure, efficient backup program
Pros of Restic
- Open-source with a larger community and more frequent updates
- Supports a wider range of storage backends (local, SFTP, S3, etc.)
- Built-in encryption and deduplication
Cons of Restic
- Generally slower performance, especially for large datasets
- Higher memory usage during backup and restore operations
- Less efficient handling of small file changes
Code Comparison
Restic:
func (be *b2Backend) Save(ctx context.Context, h restic.Handle, rd restic.RewindReader) error {
objName := be.Filename(h)
return be.b2.UploadFile(ctx, be.cfg.Bucket, objName, rd)
}
Duplicacy:
func (storage *B2Storage) UploadFile(filePath string, content []byte) (err error) {
bucket := storage.bucket
return bucket.UploadFile(filePath, content)
}
Both projects use similar approaches for file uploads, but Duplicacy's implementation is more straightforward. Restic's code shows more context handling and flexibility, while Duplicacy focuses on simplicity.
Deduplicating archiver with compression and authenticated encryption.
Pros of Borg
- Mature project with a large community and extensive documentation
- Supports compression and encryption out of the box
- Offers a flexible retention policy system for managing backups
Cons of Borg
- Requires Python installation, which may not be ideal for all systems
- Limited cross-platform support (primarily Linux and macOS)
- Can be more complex to set up and use for beginners
Code Comparison
Borg backup creation:
borg create -v --stats /path/to/repo::archive_name /path/to/source
Duplicacy backup creation:
duplicacy backup -threads 4
Borg focuses on a command-line interface with various options, while Duplicacy uses a more straightforward approach with configuration files.
Key Differences
- Duplicacy is written in Go, offering better cross-platform compatibility
- Borg uses a custom archive format, while Duplicacy supports multiple cloud storage providers
- Duplicacy emphasizes simplicity and ease of use, whereas Borg provides more advanced features and customization options
Both projects aim to provide efficient and secure backup solutions, but they cater to different user needs and preferences. Borg is more suitable for advanced users and Linux environments, while Duplicacy offers a more user-friendly experience across various platforms.
Store securely encrypted backups in the cloud!
Pros of Duplicati
- More user-friendly interface with a web-based GUI
- Supports a wider range of storage backends out-of-the-box
- Offers built-in scheduling and email notifications
Cons of Duplicati
- Generally slower performance, especially for large datasets
- Higher resource usage during backup operations
- Less efficient deduplication compared to Duplicacy
Code Comparison
Duplicacy uses a lock-free algorithm for concurrent backups:
func (storage *Storage) CreateDirectory(dir string) (err error) {
for i := 0; i < 3; i++ {
err = storage.client.MkdirAll(dir, 0744)
if err == nil {
return nil
}
time.Sleep(1 * time.Second)
}
return err
}
Duplicati uses a more traditional locking mechanism:
public void Lock()
{
if (m_lock == null)
m_lock = new InterprocessLock(m_lockfile);
m_lock.Acquire();
}
Both projects are open-source backup solutions, but they differ in their approach. Duplicacy focuses on performance and efficiency, using a unique lock-free algorithm for concurrent operations. Duplicati, on the other hand, prioritizes user-friendliness and feature richness, offering a web-based GUI and built-in scheduling. While Duplicati supports more storage backends out-of-the-box, it generally has slower performance and higher resource usage compared to Duplicacy, especially for large datasets.
Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.
Pros of Kopia
- Open-source with a more permissive license (Apache 2.0)
- Supports a wider range of storage backends, including local, network, and cloud options
- Offers a user-friendly GUI in addition to CLI
Cons of Kopia
- Relatively newer project with a smaller user base
- May have fewer advanced features compared to Duplicacy's mature codebase
Code Comparison
Kopia (Go):
func (r *Repository) CreateSnapshot(ctx context.Context, src SourceInfo) (*Snapshot, error) {
manifests, err := r.ListSnapshots(ctx, src)
if err != nil {
return nil, errors.Wrap(err, "error listing previous snapshots")
}
// ... (additional snapshot creation logic)
}
Duplicacy (Go):
func (storage *Storage) CreateSnapshot(top string, tag string, patterns []string) (snapshotID string, err error) {
if len(patterns) == 0 {
patterns = []string{"/"}
}
snapshot := CreateEmptySnapshot(top, tag)
// ... (additional snapshot creation logic)
}
Both projects use Go and implement snapshot functionality, but Kopia's approach appears more context-aware and error-handling focused, while Duplicacy's implementation seems more straightforward with default patterns.
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
Pros of rclone
- Supports a wider range of cloud storage providers and protocols
- More active development and larger community support
- Offers a command-line interface and a mounting feature for cloud storage
Cons of rclone
- Lacks built-in backup and versioning capabilities
- More complex configuration and setup process
- No native deduplication feature
Code Comparison
rclone:
func (f *Fs) Copy(ctx context.Context, src Object, remote string) (Object, error) {
do := f.Features().Copy
if do == nil {
return nil, errors.New("copy not supported")
}
return do(ctx, src, remote)
}
Duplicacy:
func (storage *Storage) UploadFile(threadIndex int, filePath string, content []byte) (err error) {
if len(content) > storage.config.ChunkSize {
return fmt.Errorf("Content size %d is too big", len(content))
}
return storage.client.UploadFile(threadIndex, filePath, content)
}
Both projects are open-source backup and sync tools, but they serve different purposes. rclone focuses on cloud storage synchronization and supports numerous providers, while Duplicacy is primarily a backup solution with built-in deduplication and versioning. rclone offers more flexibility in terms of supported services, while Duplicacy provides a more streamlined backup experience with advanced features like client-side encryption and chunk-level deduplication.
Open Source Continuous File Synchronization
Pros of Syncthing
- Decentralized peer-to-peer file synchronization
- Cross-platform support (Windows, macOS, Linux, Android)
- Open-source with a large community and active development
Cons of Syncthing
- Not designed for backup purposes, focuses on synchronization
- Requires devices to be online simultaneously for syncing
- May have higher resource usage due to continuous syncing
Code Comparison
Syncthing (Go):
func (m *Model) Index(deviceID protocol.DeviceID, folder string, files []protocol.FileInfo, flags uint32, options []protocol.Option) {
m.fmut.Lock()
defer m.fmut.Unlock()
if cfg, ok := m.folderCfgs[folder]; !ok || !cfg.SharedWith(deviceID) {
l.Infof("Index for folder %s from device %s; folder not shared with device", folder, deviceID)
return
}
}
Duplicacy (Go):
func (storage *StorageBase) DeleteFile(filePath string) (err error) {
if storage.DryRun {
return nil
}
return storage.deleteFile(filePath)
}
While both projects are written in Go, Syncthing focuses on file synchronization across devices, whereas Duplicacy is designed for backup and storage management. The code snippets reflect their different purposes, with Syncthing handling device indexing and Duplicacy managing file operations for backups.
Convert
designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual CopilotREADME
Duplicacy: A lock-free deduplication cloud backup tool
Duplicacy is a new generation cross-platform cloud backup tool based on the idea of Lock-Free Deduplication.
Our paper explaining the inner workings of Duplicacy has been accepted by IEEE Transactions on Cloud Computing and will appear in a future issue this year. The final draft version is available here for those who don't have IEEE subscriptions.
This repository hosts source code, design documents, and binary releases of the command line version of Duplicacy. There is also a Web GUI frontend built for Windows, macOS, and Linux, available from https://duplicacy.com.
There is a special edition of Duplicacy developed for VMware vSphere (ESXi) named Vertical Backup that can back up virtual machine files on ESXi to local drives, network or cloud storages.
Features
There are 3 core advantages of Duplicacy over any other open-source or commercial backup tools:
-
Duplicacy is the only cloud backup tool that allows multiple computers to back up to the same cloud storage, taking advantage of cross-computer deduplication whenever possible, without direct communication among them. This feature turns any cloud storage server supporting only a basic set of file operations into a sophisticated deduplication-aware server.
-
Unlike other chunk-based backup tools where chunks are grouped into pack files and a chunk database is used to track which chunks are stored inside each pack file, Duplicacy takes a database-less approach where every chunk is saved independently using its hash as the file name to facilitate quick lookups. The avoidance of a centralized chunk database not only produces a simpler and less error-prone implementation, but also makes it easier to develop advanced features, such as Asymmetric Encryption for stronger encryption and Erasure Coding for resilient data protection.
-
Duplicacy is fast. While the performance wasn't the top-priority design goal, Duplicacy has been shown to outperform other backup tools by a considerable margin, as indicated by the following results obtained from a benchmarking experiment backing up the Linux code base using Duplicacy and 3 other open-source backup tools.
Getting Started
Storages
Duplicacy currently provides the following storage backends:
- Local disk
- SFTP
- Dropbox
- Amazon S3
- Wasabi
- DigitalOcean Spaces
- Google Cloud Storage
- Microsoft Azure
- Backblaze B2
- Google Drive
- Microsoft OneDrive
- Hubic
- OpenStack Swift
- WebDAV (under beta testing)
- pcloud (via WebDAV)
- Box.com (via WebDAV)
- File Fabric by Storage Made Easy
Please consult the wiki page on how to set up Duplicacy to work with each cloud storage.
For reference, the following chart shows the running times (in seconds) of backing up the Linux code base to each of those supported storages:
For complete benchmark results please visit https://github.com/gilbertchen/cloud-storage-comparison.
Comparison with Other Backup Tools
duplicity works by applying the rsync algorithm (or more specific, the librsync library) to find the differences from previous backups and only then uploading the differences. It is the only existing backup tool with extensive cloud support -- the long list of storage backends covers almost every cloud provider one can think of. However, duplicity's biggest flaw lies in its incremental model -- a chain of dependent backups starts with a full backup followed by a number of incremental ones, and ends when another full backup is uploaded. Deleting one backup will render useless all the subsequent backups on the same chain. Periodic full backups are required, in order to make previous backups disposable.
bup also uses librsync to split files into chunks but save chunks in the git packfile format. It doesn't support any cloud storage, or deletion of old backups.
Duplicati is one of the first backup tools that adopt the chunk-based approach to split files into chunks which are then uploaded to the storage. The chunk-based approach got the incremental backup model right in the sense that every incremental backup is actually a full snapshot. As Duplicati splits files into fixed-size chunks, deletions or insertions of a few bytes will foil the deduplication. Cloud support is extensive, but multiple clients can't back up to the same storage location.
Attic has been acclaimed by some as the Holy Grail of backups. It follows the same incremental backup model like Duplicati but embraces the variable-size chunk algorithm for better performance and higher deduplication efficiency (not susceptible to byte insertion and deletion any more). Deletions of old backup are also supported. However, no cloud backends are implemented. Although concurrent backups from multiple clients to the same storage is in theory possible by the use of locking, it is not recommended by the developer due to chunk indices being kept in a local cache. Concurrent access is not only a convenience; it is a necessity for better deduplication. For instance, if multiple machines with the same OS installed can back up their entire drives to the same storage, only one copy of the system files needs to be stored, greatly reducing the storage space regardless of the number of machines. Attic still adopts the traditional approach of using a centralized indexing database to manage chunks and relies heavily on caching to improve performance. The presence of exclusive locking makes it hard to be extended to cloud storages.
restic is a more recent addition. It uses a format similar to the git packfile format. Multiple clients backing up to the same storage are still guarded by locks, and because a chunk database is used, deduplication isn't real-time (different clients sharing the same files will upload different copies of the same chunks). A prune operation will completely block all other clients connected to the storage from doing their regular backups. Moreover, since most cloud storage services do not provide a locking service, the best effort is to use some basic file operations to simulate a lock, but distributed locking is known to be a hard problem and it is unclear how reliable restic's lock implementation is. A faulty implementation may cause a prune operation to accidentally delete data still in use, resulting in unrecoverable data loss. This is the exact problem that we avoided by taking the lock-free approach.
The following table compares the feature lists of all these backup tools:
Feature/Tool | duplicity | bup | Duplicati | Attic | restic | Duplicacy |
---|---|---|---|---|---|---|
Incremental Backup | Yes | Yes | Yes | Yes | Yes | Yes |
Full Snapshot | No | Yes | Yes | Yes | Yes | Yes |
Compression | Yes | Yes | Yes | Yes | No | Yes |
Deduplication | Weak | Yes | Weak | Yes | Yes | Yes |
Encryption | Yes | Yes | Yes | Yes | Yes | Yes |
Deletion | No | No | Yes | Yes | No | Yes |
Concurrent Access | No | No | No | Not recommended | Exclusive locking | Lock-free |
Cloud Support | Extensive | No | Extensive | No | Limited | Extensive |
Snapshot Migration | No | No | No | No | No | Yes |
License
- Free for personal use or commercial trial
- Non-trial commercial use requires per-computer CLI licenses available from duplicacy.com at a cost of $50 per year
- The computer with a valid commercial license for the GUI version may run the CLI version without a CLI license
- CLI licenses are not required to restore or manage backups; only the backup command requires valid CLI licenses
- Modification and redistribution are permitted, but commercial use of derivative works is subject to the same requirements of this license
Top Related Projects
Fast, secure, efficient backup program
Deduplicating archiver with compression and authenticated encryption.
Store securely encrypted backups in the cloud!
Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
Open Source Continuous File Synchronization
Convert
designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual Copilot