Top Related Projects
Deep Learning for humans
Tensors and Dynamic neural networks in Python with strong GPU acceleration
ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more
Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more
scikit-learn: machine learning in Python
Quick Overview
TensorFlow Addons is an extension library for TensorFlow, providing additional functionality not available in the core TensorFlow package. It includes custom ops, layers, optimizers, and other components that enhance TensorFlow's capabilities for machine learning and deep learning tasks.
Pros
- Extends TensorFlow's functionality with additional tools and features
- Maintained by the TensorFlow community, ensuring compatibility and quality
- Provides implementations of cutting-edge research and algorithms
- Offers flexibility for advanced users and researchers
Cons
- May have compatibility issues with different TensorFlow versions
- Some features might be experimental or less stable than core TensorFlow
- Requires additional installation and setup
- Documentation can be less comprehensive than core TensorFlow
Code Examples
- Using the AdamW optimizer:
import tensorflow as tf
import tensorflow_addons as tfa
model = tf.keras.Sequential([...])
optimizer = tfa.optimizers.AdamW(learning_rate=1e-3, weight_decay=1e-4)
model.compile(optimizer=optimizer, loss='categorical_crossentropy')
- Applying the Mixup data augmentation technique:
import tensorflow as tf
import tensorflow_addons as tfa
@tf.function
def mixup(x, y, alpha=0.2):
mixup_layer = tfa.image.MixupLayer(alpha=alpha)
x, y = mixup_layer([x, y])
return x, y
# Use in a tf.data pipeline
dataset = dataset.map(mixup)
- Using the F1Score metric:
import tensorflow as tf
import tensorflow_addons as tfa
model = tf.keras.Sequential([...])
f1 = tfa.metrics.F1Score(num_classes=10, average='macro')
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=[f1])
Getting Started
To get started with TensorFlow Addons, follow these steps:
- Install TensorFlow Addons:
pip install tensorflow-addons
- Import TensorFlow Addons in your Python script:
import tensorflow as tf
import tensorflow_addons as tfa
- Use TensorFlow Addons components in your TensorFlow code:
# Example: Using LAMB optimizer
optimizer = tfa.optimizers.LAMB(learning_rate=1e-3)
# Example: Using GeLU activation
x = tfa.activations.gelu(x)
# Example: Using WeightNormalization layer
dense = tfa.layers.WeightNormalization(tf.keras.layers.Dense(64))
Now you can explore and utilize the various additional features provided by TensorFlow Addons in your TensorFlow projects.
Competitor Comparisons
Deep Learning for humans
Pros of Keras
- Higher-level API, making it more user-friendly and easier to learn
- More flexible and modular architecture, allowing for easier customization
- Supports multiple backend engines (TensorFlow, Theano, CNTK)
Cons of Keras
- May have slightly slower performance due to higher-level abstractions
- Less fine-grained control over low-level operations
- Smaller ecosystem of specialized tools and extensions
Code Comparison
Keras:
from keras.models import Sequential
from keras.layers import Dense
model = Sequential([
Dense(64, activation='relu', input_shape=(10,)),
Dense(1, activation='sigmoid')
])
TensorFlow Addons:
import tensorflow as tf
import tensorflow_addons as tfa
model = tf.keras.Sequential([
tf.keras.layers.Dense(64, activation='relu', input_shape=(10,)),
tf.keras.layers.Dense(1, activation='sigmoid')
])
optimizer = tfa.optimizers.AdamW(learning_rate=1e-3, weight_decay=1e-5)
The code comparison shows that while both libraries can be used to create neural networks, TensorFlow Addons provides additional functionality (like the AdamW optimizer) that extends TensorFlow's capabilities. Keras offers a more straightforward API for basic model creation.
Tensors and Dynamic neural networks in Python with strong GPU acceleration
Pros of PyTorch
- More intuitive and Pythonic API, easier for beginners to learn
- Dynamic computational graphs allow for more flexible model architectures
- Better support for debugging and easier to integrate with Python tools
Cons of PyTorch
- Smaller ecosystem compared to TensorFlow, fewer pre-built models and tools
- Less robust deployment options, especially for mobile and edge devices
- Steeper learning curve for production-level deployment
Code Comparison
PyTorch example:
import torch
x = torch.tensor([1, 2, 3])
y = torch.tensor([4, 5, 6])
z = torch.add(x, y)
TensorFlow Addons example:
import tensorflow as tf
import tensorflow_addons as tfa
x = tf.constant([1, 2, 3])
y = tf.constant([4, 5, 6])
z = tf.add(x, y)
Both examples demonstrate basic tensor operations, but PyTorch's syntax is generally considered more intuitive and closer to standard Python. TensorFlow Addons extends TensorFlow's functionality but maintains its core syntax and structure.
ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
Pros of ONNX Runtime
- Broader framework support, including TensorFlow, PyTorch, and more
- Optimized for cross-platform deployment and inference
- Extensive hardware acceleration support (CPU, GPU, TPU, etc.)
Cons of ONNX Runtime
- Less tightly integrated with TensorFlow ecosystem
- May require additional steps to convert TensorFlow models to ONNX format
Code Comparison
TensorFlow Addons example:
import tensorflow_addons as tfa
optimizer = tfa.optimizers.AdamW(
learning_rate=1e-3, weight_decay=1e-5
)
ONNX Runtime example:
import onnxruntime as ort
session = ort.InferenceSession("model.onnx")
input_name = session.get_inputs()[0].name
output = session.run(None, {input_name: input_data})
TensorFlow Addons focuses on extending TensorFlow functionality with additional layers, optimizers, and metrics. ONNX Runtime, on the other hand, is designed for efficient model inference across various frameworks and platforms. While TensorFlow Addons is more specialized for TensorFlow users, ONNX Runtime offers broader compatibility and optimization for deployment scenarios.
Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more
Pros of MXNet
- More flexible and supports multiple programming languages (Python, C++, R, Julia, etc.)
- Designed for distributed computing and scalability
- Offers a hybrid frontend for both imperative and symbolic programming
Cons of MXNet
- Smaller community and ecosystem compared to TensorFlow
- Less extensive documentation and tutorials
- Steeper learning curve for beginners
Code Comparison
MXNet:
import mxnet as mx
from mxnet import nd, autograd, gluon
x = nd.array([[1, 2], [3, 4]])
y = nd.array([[5, 6], [7, 8]])
z = nd.dot(x, y)
TensorFlow Addons:
import tensorflow as tf
import tensorflow_addons as tfa
x = tf.constant([[1, 2], [3, 4]])
y = tf.constant([[5, 6], [7, 8]])
z = tf.matmul(x, y)
Both libraries offer similar functionality for basic operations, but MXNet's syntax is more concise. TensorFlow Addons extends TensorFlow's capabilities with additional layers, optimizers, and metrics, while MXNet provides a more comprehensive set of built-in features. The choice between the two depends on specific project requirements, team expertise, and desired scalability.
Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more
Pros of JAX
- More flexible and composable, allowing for easier customization of models and training loops
- Better support for automatic differentiation and vectorization, leading to improved performance
- Simpler API with a focus on functional programming paradigms
Cons of JAX
- Smaller ecosystem and fewer pre-built models compared to TensorFlow Addons
- Steeper learning curve for developers coming from traditional imperative programming backgrounds
- Less extensive documentation and community support
Code Comparison
JAX example:
import jax.numpy as jnp
from jax import grad, jit
def loss(params, x, y):
return jnp.mean((params[0] * x + params[1] - y) ** 2)
grad_loss = jit(grad(loss))
TensorFlow Addons example:
import tensorflow as tf
import tensorflow_addons as tfa
model = tf.keras.Sequential([
tf.keras.layers.Dense(1)
])
optimizer = tfa.optimizers.AdamW(learning_rate=0.001, weight_decay=0.01)
model.compile(optimizer=optimizer, loss='mse')
Both JAX and TensorFlow Addons offer powerful tools for machine learning, but they cater to different needs. JAX provides a more flexible and performant foundation for custom implementations, while TensorFlow Addons extends the TensorFlow ecosystem with additional functionality and optimizations.
scikit-learn: machine learning in Python
Pros of scikit-learn
- Simpler API and easier to use for beginners
- Broader range of machine learning algorithms and tools
- Better documentation and community support
Cons of scikit-learn
- Less optimized for deep learning tasks
- Limited GPU acceleration support
- Slower performance for large-scale data processing
Code Comparison
scikit-learn:
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=1000, n_features=4)
clf = RandomForestClassifier()
clf.fit(X, y)
TensorFlow Addons:
import tensorflow as tf
import tensorflow_addons as tfa
model = tf.keras.Sequential([
tf.keras.layers.Dense(10, activation='relu'),
tf.keras.layers.Dense(1, activation='sigmoid')
])
model.compile(optimizer=tfa.optimizers.AdamW(learning_rate=1e-3, weight_decay=1e-4))
scikit-learn focuses on traditional machine learning algorithms with a simple API, while TensorFlow Addons extends TensorFlow's capabilities for deep learning tasks with additional layers, optimizers, and loss functions. scikit-learn is more accessible for beginners and offers a wider range of algorithms, but TensorFlow Addons provides better performance for complex neural networks and GPU acceleration.
Convert 
designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual CopilotREADME
:warning: :warning: :warning:
TensorFlow Addons (TFA) has ended development and introduction of new features.
TFA has entered a minimal maintenance and release mode until a planned end of life in May 2024. Please modify downstream libraries to take dependencies from other repositories in our TensorFlow community (e.g. Keras, Keras-CV, and Keras-NLP)
For more information see: https://github.com/tensorflow/addons/issues/2807
Continuous Build Status
| Build | Status |
|---|---|
| Ubuntu/macOS |  |
| Ubuntu GPU custom ops |  |
TensorFlow Addons is a repository of contributions that conform to well-established API patterns, but implement new functionality not available in core TensorFlow. TensorFlow natively supports a large number of operators, layers, metrics, losses, and optimizers. However, in a fast moving field like ML, there are many interesting new developments that cannot be integrated into core TensorFlow (because their broad applicability is not yet clear, or it is mostly used by a smaller subset of the community).
Addons Subpackages
- tfa.activations
- tfa.callbacks
- tfa.image
- tfa.layers
- tfa.losses
- tfa.metrics
- tfa.optimizers
- tfa.rnn
- tfa.seq2seq
- tfa.text
Maintainership
The maintainers of TensorFlow Addons can be found in the CODEOWNERS file of the repo. This file is parsed and pull requests will automatically tag the owners using a bot. If you would like to maintain something, please feel free to submit a PR. We encourage multiple owners for all submodules.
Installation
Stable Builds
TensorFlow Addons is available on PyPI for Linux & macOS (Windows support was dropped due to inconsistent TF2.15 whl packaging). To install the latest version, run the following:
pip install tensorflow-addons
To ensure you have a version of TensorFlow that is compatible with TensorFlow Addons,
you can specify the tensorflow extra requirement during install:
pip install tensorflow-addons[tensorflow]
Similar extras exist for the tensorflow-gpu and tensorflow-cpu packages.
To use TensorFlow Addons:
import tensorflow as tf
import tensorflow_addons as tfa
Python Op Compatility
TensorFlow Addons is actively working towards forward compatibility with TensorFlow 2.x.
However, there are still a few private API uses within the repository so at the moment
we can only guarantee compatibility with the TensorFlow versions which it was tested against.
Warnings will be emitted when importing tensorflow_addons if your TensorFlow version does not match
what it was tested against.
Python Op Compatibility Matrix
| TensorFlow Addons | TensorFlow | Python |
|---|---|---|
| tfa-nightly | 2.12, 2.13, 2.14 | 3.9, 3.10, 3.11 |
| tensorflow-addons-0.22.0 | 2.12, 2.13, 2.14 | 3.9, 3.10, 3.11 |
| tensorflow-addons-0.21.0 | 2.11, 2.12, 2.13 | 3.8, 3.9, 3.10, 3.11 |
| tensorflow-addons-0.20.0 | 2.10, 2.11, 2.12 | 3.8, 3.9, 3.10, 3.11 |
| tensorflow-addons-0.19.0 | 2.9, 2.10, 2.11 | 3.7, 3.8, 3.9, 3.10 |
| tensorflow-addons-0.18.0 | 2.8, 2.9, 2.10 | 3.7, 3.8, 3.9, 3.10 |
| tensorflow-addons-0.17.1 | 2.7, 2.8, 2.9 | 3.7, 3.8, 3.9, 3.10 |
| tensorflow-addons-0.16.1 | 2.6, 2.7, 2.8 | 3.7, 3.8, 3.9, 3.10 |
| tensorflow-addons-0.15.0 | 2.5, 2.6, 2.7 | 3.7, 3.8, 3.9 |
| tensorflow-addons-0.14.0 | 2.4, 2.5, 2.6 | 3.6, 3.7, 3.8, 3.9 |
| tensorflow-addons-0.13.0 | 2.3, 2.4, 2.5 | 3.6, 3.7, 3.8, 3.9 |
| tensorflow-addons-0.12.1 | 2.3, 2.4 | 3.6, 3.7, 3.8 |
| tensorflow-addons-0.11.2 | 2.2, 2.3 | 3.5, 3.6, 3.7, 3.8 |
| tensorflow-addons-0.10.0 | 2.2 | 3.5, 3.6, 3.7, 3.8 |
| tensorflow-addons-0.9.1 | 2.1, 2.2 | 3.5, 3.6, 3.7 |
| tensorflow-addons-0.8.3 | 2.1 | 3.5, 3.6, 3.7 |
| tensorflow-addons-0.7.1 | 2.1 | 2.7, 3.5, 3.6, 3.7 |
| tensorflow-addons-0.6.0 | 2.0 | 2.7, 3.5, 3.6, 3.7 |
C++ Custom Op Compatibility
TensorFlow C++ APIs are not stable and thus we can only guarantee compatibility with the version TensorFlow Addons was built against. It is possible custom ops will work with multiple versions of TensorFlow, but there is also a chance for segmentation faults or other problematic crashes. Warnings will be emitted when loading a custom op if your TensorFlow version does not match what it was built against.
Additionally, custom ops registration does not have a stable ABI interface so it is
required that users have a compatible installation of TensorFlow even if the versions
match what we had built against. A simplification of this is that TensorFlow Addons
custom ops will work with pip-installed TensorFlow but will have issues when TensorFlow
is compiled differently. A typical example of this would be conda-installed TensorFlow.
RFC #133 aims to fix this.
C++ Custom Op Compatibility Matrix
| TensorFlow Addons | TensorFlow | Compiler | cuDNN | CUDA |
|---|---|---|---|---|
| tfa-nightly | 2.14 | GCC 9.3.1 | 8.6 | 11.8 |
| tensorflow-addons-0.22.0 | 2.14 | GCC 9.3.1 | 8.6 | 11.8 |
| tensorflow-addons-0.21.0 | 2.13 | GCC 9.3.1 | 8.6 | 11.8 |
| tensorflow-addons-0.20.0 | 2.12 | GCC 9.3.1 | 8.6 | 11.8 |
| tensorflow-addons-0.19.0 | 2.11 | GCC 9.3.1 | 8.1 | 11.2 |
| tensorflow-addons-0.18.0 | 2.10 | GCC 9.3.1 | 8.1 | 11.2 |
| tensorflow-addons-0.17.1 | 2.9 | GCC 9.3.1 | 8.1 | 11.2 |
| tensorflow-addons-0.16.1 | 2.8 | GCC 7.3.1 | 8.1 | 11.2 |
| tensorflow-addons-0.15.0 | 2.7 | GCC 7.3.1 | 8.1 | 11.2 |
| tensorflow-addons-0.14.0 | 2.6 | GCC 7.3.1 | 8.1 | 11.2 |
| tensorflow-addons-0.13.0 | 2.5 | GCC 7.3.1 | 8.1 | 11.2 |
| tensorflow-addons-0.12.1 | 2.4 | GCC 7.3.1 | 8.0 | 11.0 |
| tensorflow-addons-0.11.2 | 2.3 | GCC 7.3.1 | 7.6 | 10.1 |
| tensorflow-addons-0.10.0 | 2.2 | GCC 7.3.1 | 7.6 | 10.1 |
| tensorflow-addons-0.9.1 | 2.1 | GCC 7.3.1 | 7.6 | 10.1 |
| tensorflow-addons-0.8.3 | 2.1 | GCC 7.3.1 | 7.6 | 10.1 |
| tensorflow-addons-0.7.1 | 2.1 | GCC 7.3.1 | 7.6 | 10.1 |
| tensorflow-addons-0.6.0 | 2.0 | GCC 7.3.1 | 7.4 | 10.0 |
Nightly Builds
There are also nightly builds of TensorFlow Addons under the pip package
tfa-nightly, which is built against the latest stable version of TensorFlow. Nightly builds
include newer features, but may be less stable than the versioned releases. Contrary to
what the name implies, nightly builds are not released every night, but at every commit
of the master branch. 0.9.0.dev20200306094440 means that the commit time was
2020/03/06 at 09:44:40 Coordinated Universal Time.
pip install tfa-nightly
Installing from Source
You can also install from source. This requires the Bazel build system (version >= 1.0.0).
CPU Custom Ops
git clone https://github.com/tensorflow/addons.git
cd addons
# This script links project with TensorFlow dependency
python3 ./configure.py
bazel build build_pip_pkg
bazel-bin/build_pip_pkg artifacts
pip install artifacts/tensorflow_addons-*.whl
GPU and CPU Custom Ops
git clone https://github.com/tensorflow/addons.git
cd addons
export TF_NEED_CUDA="1"
# Set these if the below defaults are different on your system
export TF_CUDA_VERSION="12"
export TF_CUDNN_VERSION="8"
export CUDA_TOOLKIT_PATH="/usr/local/cuda"
export CUDNN_INSTALL_PATH="/usr/lib/x86_64-linux-gnu"
# This script links project with TensorFlow dependency
python3 ./configure.py
bazel build build_pip_pkg
bazel-bin/build_pip_pkg artifacts
pip install artifacts/tensorflow_addons-*.whl
Tutorials
See docs/tutorials/
for end-to-end examples of various addons.
Core Concepts
Standardized API within Subpackages
User experience and project maintainability are core concepts in TensorFlow Addons. In order to achieve these we require that our additions conform to established API patterns seen in core TensorFlow.
GPU and CPU Custom Ops
TensorFlow Addons supports precompiled custom ops for CPU and GPU. However, GPU custom ops currently only work on Linux distributions. For this reason Windows and macOS will fallback to pure TensorFlow Python implementations whenever possible.
The order of priority on macOS/Windows is:
- Pure TensorFlow + Python implementation (works on CPU and GPU)
- C++ implementation for CPU
The order of priority on Linux is:
- CUDA implementation
- C++ implementation
- Pure TensorFlow + Python implementation (works on CPU and GPU)
If you want to change the default priority, "C++ and CUDA" VS "pure TensorFlow Python",
you can set the environment variable TF_ADDONS_PY_OPS=1 from the command line or
run tfa.options.disable_custom_kernel() in your code.
For example, if you are on Linux and you have compatibility problems with the compiled ops, you can give priority to the Python implementations:
From the command line:
export TF_ADDONS_PY_OPS=1
or in your code:
import tensorflow_addons as tfa
tfa.options.disable_custom_kernel()
This variable defaults to True on Windows and macOS, and False on Linux.
Proxy Maintainership
TensorFlow Addons has been designed to compartmentalize submodules so that they can be maintained by community users who have expertise, and a vested interest in that component. We heavily encourage users to submit sign up to maintain a submodule by submitting your username to the CODEOWNERS file.
Full write access will only be granted after substantial contribution has been made in order to limit the number of users with write permission. Contributions can come in the form of issue closings, bug fixes, documentation, new code, or optimizing existing code. Submodule maintainership can be granted with a lower barrier for entry as this will not include write permissions to the repo.
For more information see the RFC on this topic.
Periodic Evaluation of Subpackages
Given the nature of this repository, submodules may become less and less useful to the community as time goes on. In order to keep the repository sustainable, we'll be performing bi-annual reviews of our code to ensure everything still belongs within the repo. Contributing factors to this review will be:
- Number of active maintainers
- Amount of OSS use
- Amount of issues or bugs attributed to the code
- If a better solution is now available
Functionality within TensorFlow Addons can be categorized into three groups:
- Suggested: well-maintained API; use is encouraged.
- Discouraged: a better alternative is available; the API is kept for historic reasons; or the API requires maintenance and is the waiting period to be deprecated.
- Deprecated: use at your own risk; subject to be deleted.
The status change between these three groups is: Suggested <-> Discouraged -> Deprecated.
The period between an API being marked as deprecated and being deleted will be 90 days. The rationale being:
-
In the event that TensorFlow Addons releases monthly, there will be 2-3 releases before an API is deleted. The release notes could give user enough warning.
-
90 days gives maintainers ample time to fix their code.
Contributing
TensorFlow Addons is a community-led open source project (only a few maintainers work for Google!). As such, the project depends on public contributions, bug fixes, and documentation. This project adheres to TensorFlow's code of conduct. By participating, you are expected to uphold this code.
Do you want to contribute but are not sure of what? Here are a few suggestions:
- Add a new tutorial. Located in
docs/tutorials/, these are a great way to familiarize yourself and others with TensorFlow Addons. See the guidelines for more information on how to add examples. - Improve the docstrings. The docstrings are fetched and then displayed in the documentation. Do a change and hundreds of developers will see it and benefit from it. Maintainers are often focused on making APIs, fixing bugs and other code related changes. The documentation will never be loved enough!
- Solve an existing issue. These range from low-level software bugs to higher-level design problems. Check out the label help wanted. If you're a new contributor, the label good first issue can be a good place to start.
- Review a pull request. So you're not a software engineer but you know a lot about a certain field a research? That's awesome and we need your help! Many people are submitting pull requests to add layers/optimizers/functions taken from recent papers. Since TensorFlow Addons maintainers are not specialized in everything, you can imagine how hard it is to review. It takes very long to read the paper, understand it and check the math in the pull request. If you're specialized, look at the list of pull requests. If there is something from a paper you know, please comment on the pull request to check the math is ok. If you see that everything is good, say it! It will help the maintainers to sleep better at night knowing that he/she wasn't the only person to approve the pull request.
- You have an opinion and want to share it? The docs are not very helpful for a function or a class? You tried to open a pull request but you didn't manage to install or test anything and you think it's too complicated? You made a pull request but you didn't find the process good enough and it made no sense to you? Please say it! We want feedback. Maintainers are too much the head into the code to understand what it's like for someone new to open source to come to this project. If you don't understand something, be aware there are no people who are bad at understanding, there are just bad tutorials and bad guides.
Please see contribution guidelines to get started (and remember, if you don't understand something, open an issue, or even make a pull request to improve the guide!).
Community
- Public Mailing List
- SIG Monthly Meeting Notes
- Join our mailing list and receive calendar invites to the meeting
License
Top Related Projects
Deep Learning for humans
Tensors and Dynamic neural networks in Python with strong GPU acceleration
ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more
Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more
scikit-learn: machine learning in Python
Convert designs to code with AI
Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.
Try Visual Copilot