s3proxy

Access other storage backends via the S3 API

1,720

223

1,720

127

View on GitHub

Top Related Projects

minio

46,602

MinIO is a high-performance, S3 compatible object store, open sourced under GNU AGPLv3 license.

aws-cli

15,322

Universal Command Line Interface for Amazon Web Services

aws-sdk-java

4,110

The official AWS SDK for Java 1.x. The AWS SDK for Java 2.x is available here: https://github.com/aws/aws-sdk-java-v2/

Quick Overview

s3proxy is a lightweight HTTP proxy that provides access to Amazon S3 buckets. It allows you to interact with S3 buckets using standard HTTP requests, without the need to directly use the AWS SDK or configure AWS credentials.

Pros

Simplifies S3 Access: s3proxy abstracts away the complexity of working with the AWS SDK, making it easier to interact with S3 buckets.
Language Agnostic: Since it exposes an HTTP interface, s3proxy can be used with any programming language that can make HTTP requests.
Caching: s3proxy can cache responses from S3, improving performance for repeated requests.
Authentication: s3proxy supports various authentication methods, including AWS Signature Version 4, making it easier to secure access to your S3 buckets.

Cons

Limited Functionality: s3proxy is a lightweight proxy and may not provide all the features and functionality of the full AWS SDK.
Dependency on s3proxy: Applications using s3proxy are dependent on the availability and correct configuration of the s3proxy service.
Performance Overhead: The additional layer of the proxy may introduce some performance overhead compared to direct S3 access.
Complexity of Setup: Setting up and configuring s3proxy may require additional effort, especially for complex S3 access scenarios.

Code Examples

N/A (s3proxy is not a code library)

Getting Started

To get started with s3proxy, follow these steps:

Download the latest release of s3proxy from the GitHub repository.
Extract the downloaded archive and navigate to the extracted directory.
Start the s3proxy server using the following command:

./s3proxy --properties s3proxy.properties

The s3proxy.properties file contains the configuration for the s3proxy server, such as the S3 bucket endpoints, authentication settings, and caching options. Here's an example s3proxy.properties file:

s3proxy.endpoint=http://s3.amazonaws.com
s3proxy.virtual-host-style=true
s3proxy.authorization=aws-v4
s3proxy.aws-access-key-id=your-access-key-id
s3proxy.aws-secret-key=your-secret-key
s3proxy.cache-control=max-age=3600

Once the s3proxy server is running, you can interact with your S3 buckets using standard HTTP requests. For example, to list the contents of an S3 bucket, you can use the following curl command:

curl http://localhost:8080/your-bucket-name

Refer to the s3proxy documentation for more information on configuring and using s3proxy.

Competitor Comparisons

minio

46,602

MinIO is a high-performance, S3 compatible object store, open sourced under GNU AGPLv3 license.

Pros of minio/minio

Supports a wide range of storage backends, including Amazon S3, Google Cloud Storage, and Azure Blob Storage, making it a more versatile solution.
Provides a comprehensive set of features, including object versioning, bucket lifecycle management, and server-side encryption.
Offers a user-friendly web-based console for managing the storage environment.

Cons of minio/minio

Requires more resources (CPU, memory, and storage) to run compared to gaul/s3proxy, which may be a consideration for smaller deployments.
Has a steeper learning curve due to its more extensive feature set and configuration options.
May have a higher maintenance overhead compared to the simpler gaul/s3proxy.

Code Comparison

minio/minio (main.go):

func main() {
    // Set global variables
    setGlobalConfig()

    // Handle signals
    handleSignals()

    // Initialize server config
    initServerConfig()

    // Initialize server
    initServer()

    // Wait for server to start
    waitForServerStart()
}

gaul/s3proxy (main.go):

func main() {
    // Parse command-line arguments
    parseFlags()

    // Initialize the S3 proxy
    initS3Proxy()

    // Start the S3 proxy
    startS3Proxy()
}

aws-cli

15,322

Universal Command Line Interface for Amazon Web Services

Pros of aws/aws-cli

Comprehensive coverage of AWS services and features
Actively maintained and updated by the AWS team
Provides a consistent command-line interface for interacting with AWS

Cons of aws/aws-cli

Larger codebase and dependencies compared to gaul/s3proxy
May have a steeper learning curve for users not familiar with AWS CLI
Requires installation of the full AWS CLI package, even if only using a subset of the functionality

Code Comparison

aws/aws-cli:

import boto3

session = boto3.Session(profile_name='default')
s3 = session.resource('s3')

bucket = s3.Bucket('my-bucket')
for obj in bucket.objects.all():
    print(obj.key)

gaul/s3proxy:

S3Proxy s3Proxy = new S3Proxy();
s3Proxy.setEndpoint("http://localhost:8080");
s3Proxy.setBucket("my-bucket");

List<String> objects = s3Proxy.listObjects();
for (String obj : objects) {
    System.out.println(obj);
}

aws-sdk-java

4,110

The official AWS SDK for Java 1.x. The AWS SDK for Java 2.x is available here: https://github.com/aws/aws-sdk-java-v2/

Pros of aws/aws-sdk-java

Comprehensive coverage of AWS services: The aws-sdk-java provides access to a wide range of AWS services, making it a versatile choice for building applications that interact with the AWS ecosystem.
Actively maintained and supported: The sdk is actively maintained by the AWS team, ensuring regular updates, bug fixes, and security patches.
Extensive documentation and community support: The sdk has extensive documentation and a large community of developers, providing ample resources for learning and troubleshooting.

Cons of aws/aws-sdk-java

Complexity and overhead: The sdk can be complex to set up and use, especially for simple use cases, as it includes a significant amount of functionality and dependencies.
Performance overhead: The sdk can introduce performance overhead due to its comprehensive nature, which may be a concern for applications with strict performance requirements.

Code Comparison

aws/aws-sdk-java:

AmazonS3 s3 = AmazonS3ClientBuilder.standard()
    .withCredentials(new ProfileCredentialsProvider())
    .withRegion(Regions.US_WEST_2)
    .build();

S3Object object = s3.getObject("my-bucket", "my-key");

gaul/s3proxy:

S3Proxy s3Proxy = new S3Proxy();
s3Proxy.setAwsAccessKey("your-access-key");
s3Proxy.setAwsSecretKey("your-secret-key");
s3Proxy.setAwsRegion("us-west-2");
s3Proxy.setPort(8080);
s3Proxy.start();

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

S3Proxy

S3Proxy implements the S3 API and proxies requests, enabling several use cases:

translation from S3 to Backblaze B2, EMC Atmos, Google Cloud, Microsoft Azure, and OpenStack Swift
testing without Amazon by using the local filesystem
extension via middlewares
embedding into Java applications

Usage with Docker

Docker Hub hosts a Docker image and has instructions on how to run it.

Usage without Docker

Users can download releases from GitHub. Developers can build the project by running mvn package which produces a binary at target/s3proxy. S3Proxy requires Java 11 or newer to run.

Configure S3Proxy via a properties file. An example using the local file system as the storage backend with anonymous access:

s3proxy.authorization=none
s3proxy.endpoint=http://127.0.0.1:8080
jclouds.provider=filesystem
jclouds.filesystem.basedir=/tmp/s3proxy

First create the filesystem basedir:

mkdir /tmp/s3proxy

Next run S3Proxy. Linux and Mac OS X users can run the executable jar:

chmod +x s3proxy
s3proxy --properties s3proxy.conf

Windows users must explicitly invoke java:

java -jar s3proxy --properties s3proxy.conf

Finally test by creating a bucket then listing all the buckets:

$ curl --request PUT http://localhost:8080/testbucket

$ curl http://localhost:8080/
<?xml version="1.0" ?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a</ID><DisplayName>CustomersName@amazon.com</DisplayName></Owner><Buckets><Bucket><Name>testbucket</Name><CreationDate>2015-08-05T22:16:24.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>

Usage with Java

Maven Central hosts S3Proxy artifacts and the wiki has instructions on Java use.

Supported storage backends

atmos
aws-s3 (Amazon-only)
azureblob
b2
filesystem (on-disk storage)
google-cloud-storage
openstack-swift
rackspace-cloudfiles-uk and rackspace-cloudfiles-us
s3 (all implementations)
transient (in-memory storage)

See the wiki for examples of configurations.

Assigning buckets to backends

S3Proxy can be configured to assign buckets to different backends with the same credentials. The configuration in the properties file is as follows:

s3proxy.bucket-locator.1=bucket
s3proxy.bucket-locator.2=another-bucket

In addition to the explicit names, glob syntax can be used to configure many buckets for a given backend.

A bucket (or a glob) cannot be assigned to multiple backends.

Middlewares

S3Proxy can modify its behavior based on middlewares:

SSL Support

S3Proxy can listen on HTTPS by setting the secure-endpoint and configuring a keystore. You can read more about how configure S3Proxy for SSL Support in the dedicated wiki page with Docker, Kubernetes or simply Java.

Limitations

S3Proxy has broad compatibility with the S3 API, however, it does not support:

ACLs other than private and public-read
BitTorrent hosting
bucket logging
bucket policies
CORS bucket operations like getting or setting the CORS configuration for a bucket. S3Proxy only supports a static configuration (see below).
hosting static websites
object server-side encryption
object tagging
object versioning, see #74
POST upload policies, see #73
requester pays buckets
select object content

S3Proxy emulates the following operations:

copy multi-part objects, see #76

S3Proxy has basic CORS preflight and actual request/response handling. It can be configured within the properties file (and corresponding ENV variables for Docker):

s3proxy.cors-allow-origins=https://example\.com https://.+\.example\.com https://example\.cloud
s3proxy.cors-allow-methods=GET PUT
s3proxy.cors-allow-headers=Accept Content-Type
s3proxy.cors-allow-credential=true

CORS cannot be configured per bucket. s3proxy.cors-allow-all=true will accept any origin and header. Actual CORS requests are supported for GET, PUT, POST, HEAD and DELETE methods.

The wiki collects compatibility notes for specific storage backends.

Support

References

Apache jclouds provides storage backend support for S3Proxy
Ceph s3-tests help maintain and improve compatibility with the S3 API
fake-s3, gofakes3, minio, S3 ninja, and s3rver provide functionality similar to S3Proxy when using the filesystem backend
GlacierProxy and SwiftProxy provide similar functionality for the Amazon Glacier and OpenStack Swift APIs
s3mock - Adobe's s3 mock implementation
sbt-s3 runs S3Proxy via the Scala Build Tool
swift3 provides an S3 middleware for OpenStack Swift
Zenko provide similar multi-cloud functionality

License

Licensed under the Apache License, Version 2.0

Top Related Projects

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot