Convert Figma logo to code with AI

AI4Finance-Foundation logoFinRL

FinRL: Financial Reinforcement Learning. 🔥

9,620
2,332
9,620
245

Top Related Projects

15,147

Qlib is an AI-oriented quantitative investment platform that aims to realize the potential, empower research, and create value using AI technologies in quantitative investment, from exploring ideas to implementing productions. Qlib supports diverse machine learning modeling paradigms. including supervised learning, market dynamics modeling, and RL.

An open source reinforcement learning framework for training, evaluating, and deploying robust trading agents.

Python Backtesting library for trading strategies

Python library for backtesting trading strategies & analyzing financial markets (formerly pythalesians)

MlFinLab helps portfolio managers and traders who want to leverage the power of machine learning by providing reproducible, interpretable, and easy to use tools.

9,430

Lean Algorithmic Trading Engine by QuantConnect (Python, C#)

Quick Overview

FinRL is an open-source library for financial reinforcement learning. It provides a unified framework for various financial tasks, including portfolio management, algorithmic trading, and risk management. The library aims to bridge the gap between reinforcement learning and quantitative finance.

Pros

  • Comprehensive framework for applying RL to finance
  • Supports multiple market environments and RL algorithms
  • Includes pre-built datasets and data processing tools
  • Active community and regular updates

Cons

  • Steep learning curve for those new to RL or finance
  • Limited documentation for some advanced features
  • Performance can be slow for large-scale simulations
  • Requires careful hyperparameter tuning for optimal results

Code Examples

  1. Creating an environment:
from finrl.apps import stocks_trading
from finrl.config import TRAINED_MODEL_DIR, RESULTS_DIR

env = stocks_trading.StockTradingEnv(df=processed_df, 
                                     stock_dim=len(stock_list),
                                     hmax=100,
                                     initial_amount=1000000,
                                     transaction_cost_pct=0.001)
  1. Training an agent:
from finrl.agents.stablebaselines3 import DRLAgent

agent = DRLAgent(env=env)
model = agent.get_model("ppo")
trained_model = agent.train_model(model=model, 
                                  tb_log_name='ppo',
                                  total_timesteps=50000)
  1. Backtesting:
from finrl.plot import backtest_stats, backtest_plot

df_account_value, df_actions = DRLAgent.DRL_prediction(
    model=trained_model,
    environment=env)

perf_stats_all = backtest_stats(account_value=df_account_value)
backtest_plot(df_account_value, 
              baseline_ticker='SPY', 
              baseline_start=df_account_value.index[0], 
              baseline_end=df_account_value.index[-1])

Getting Started

  1. Install FinRL:
pip install finrl
  1. Import necessary modules:
from finrl.apps import stocks_trading
from finrl.agents.stablebaselines3 import DRLAgent
from finrl.config import TRAINED_MODEL_DIR, RESULTS_DIR
import pandas as pd
  1. Prepare data and create environment:
# Load and preprocess your data
df = pd.read_csv('your_data.csv')
processed_df = stocks_trading.preprocess_data(df)

# Create environment
env = stocks_trading.StockTradingEnv(df=processed_df, 
                                     stock_dim=len(stock_list),
                                     hmax=100,
                                     initial_amount=1000000,
                                     transaction_cost_pct=0.001)
  1. Train and evaluate a model:
agent = DRLAgent(env=env)
model = agent.get_model("ppo")
trained_model = agent.train_model(model=model, 
                                  tb_log_name='ppo',
                                  total_timesteps=50000)

# Backtest the trained model
df_account_value, df_actions = DRLAgent.DRL_prediction(
    model=trained_model,
    environment=env)

Competitor Comparisons

15,147

Qlib is an AI-oriented quantitative investment platform that aims to realize the potential, empower research, and create value using AI technologies in quantitative investment, from exploring ideas to implementing productions. Qlib supports diverse machine learning modeling paradigms. including supervised learning, market dynamics modeling, and RL.

Pros of qlib

  • More comprehensive and feature-rich, offering a wider range of tools and models
  • Better documentation and extensive examples for various use cases
  • Stronger industry backing and support from Microsoft

Cons of qlib

  • Steeper learning curve due to its complexity and extensive features
  • Less focus on reinforcement learning compared to FinRL
  • May be overkill for simpler financial modeling tasks

Code Comparison

FinRL example:

from finrl.apps import config
from finrl.finrl_meta.preprocessor.yahoodownloader import YahooDownloader
from finrl.finrl_meta.preprocessor.preprocessors import FeatureEngineer

df = YahooDownloader(start_date="2009-01-01", 
                     end_date="2021-01-01", 
                     ticker_list=config.DOW_30_TICKER).fetch_data()

qlib example:

import qlib
from qlib.data import D
from qlib.config import REG_CN

qlib.init(provider_uri='~/.qlib/qlib_data/cn_data', region=REG_CN)
instruments = ['SH000300']
fields = ['$close', '$volume', '$factor', '$paused']
df = D.features(instruments, fields, start_time='2010-01-01', end_time='2017-12-31')

Both libraries provide data fetching and preprocessing capabilities, but qlib offers more flexibility in data sources and feature engineering.

An open source reinforcement learning framework for training, evaluating, and deploying robust trading agents.

Pros of TensorTrade

  • More flexible and customizable trading environment
  • Supports a wider range of financial instruments and markets
  • Better documentation and examples for getting started

Cons of TensorTrade

  • Less focus on deep reinforcement learning algorithms
  • Smaller community and fewer contributions
  • Limited built-in data sources and preprocessing tools

Code Comparison

FinRL example:

from finrl.agents.stablebaselines3.models import DRLAgent
from finrl.finrl_meta.preprocessor.yahoodownloader import YahooDownloader

df = YahooDownloader(start_date="2009-01-01", end_date="2021-01-01", ticker_list=["AAPL"]).fetch_data()
agent = DRLAgent(env=env)
model = agent.get_model("ppo")

TensorTrade example:

from tensortrade.env import default
from tensortrade.feed.core import Stream, DataFeed
from tensortrade.oms.instruments import USD, BTC

env = default.create(
    instrument=USD/BTC,
    features=DataFeed([Stream.source(exchange.streams["USD-BTC"], dtype="float")]),
    action_scheme="managed-risk",
    reward_scheme="risk-adjusted"
)

Both libraries provide tools for reinforcement learning in financial markets, but TensorTrade offers more flexibility in environment creation and supports a broader range of financial instruments. FinRL, on the other hand, focuses more on deep reinforcement learning algorithms and provides better integration with popular data sources.

Python Backtesting library for trading strategies

Pros of backtrader

  • More mature and established project with a larger community
  • Comprehensive documentation and extensive examples
  • Flexible and customizable framework for backtesting trading strategies

Cons of backtrader

  • Primarily focused on backtesting, with limited support for live trading
  • Steeper learning curve for beginners due to its extensive features
  • Less emphasis on machine learning and AI-driven strategies

Code Comparison

backtrader:

class MyStrategy(bt.Strategy):
    def __init__(self):
        self.sma = bt.indicators.SimpleMovingAverage(self.data.close, period=20)

    def next(self):
        if self.data.close[0] > self.sma[0]:
            self.buy()
        elif self.data.close[0] < self.sma[0]:
            self.sell()

FinRL:

class DRLAgent(object):
    def __init__(self, env, price_array, tech_array, turbulence_array):
        self.env = env
        self.price_array = price_array
        self.tech_array = tech_array
        self.turbulence_array = turbulence_array

    def train(self):
        # Training logic here

Python library for backtesting trading strategies & analyzing financial markets (formerly pythalesians)

Pros of finmarketpy

  • More comprehensive financial market analysis tools, including backtesting and market data handling
  • Better documentation and examples for various financial market scenarios
  • Wider range of financial instruments and asset classes supported

Cons of finmarketpy

  • Less focus on reinforcement learning for financial applications
  • Not as actively maintained or updated as FinRL
  • Steeper learning curve for users new to financial market analysis

Code Comparison

FinRL example (portfolio optimization):

env = StockPortfolioEnv(df=df, 
                        stock_dim=stock_dimension,
                        hmax=100, 
                        initial_amount=1000000,
                        transaction_cost_pct=0.001)

model = PPO("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=50000)

finmarketpy example (backtesting):

backtest = Backtest(market_data, 
                    trading_model,
                    parameters,
                    date_start=start_date,
                    date_end=end_date)

backtest.run_backtest()
backtest.plot_results()

MlFinLab helps portfolio managers and traders who want to leverage the power of machine learning by providing reproducible, interpretable, and easy to use tools.

Pros of mlfinlab

  • More comprehensive library for financial machine learning tasks
  • Implements advanced techniques from academic research papers
  • Larger community and more frequent updates

Cons of mlfinlab

  • Steeper learning curve due to more complex implementations
  • Less focus on reinforcement learning for trading strategies
  • Requires more domain knowledge to use effectively

Code Comparison

mlfinlab:

from mlfinlab.data_structures import imbalance_data_structures as ds

tick_data = ds.get_tick_data(file_path, num_prev_ticks=10)
dollar_bars = ds.get_dollar_bars(tick_data, threshold=70000000)

FinRL:

from finrl.meta.preprocessor.yahoodownloader import YahooDownloader
from finrl.meta.env_stock_trading.env_stocktrading import StockTradingEnv

df = YahooDownloader(start_date, end_date, ticker_list).fetch_data()
env = StockTradingEnv(df, initial_amount=100000, hmax=1000, transaction_cost_pct=0.001)

Both libraries offer valuable tools for financial machine learning, but mlfinlab focuses more on data structures and advanced techniques, while FinRL emphasizes reinforcement learning for trading strategies. The choice between them depends on the specific requirements of your project and your level of expertise in financial machine learning.

9,430

Lean Algorithmic Trading Engine by QuantConnect (Python, C#)

Pros of Lean

  • More comprehensive and production-ready algorithmic trading platform
  • Larger community and extensive documentation
  • Supports multiple asset classes and brokers

Cons of Lean

  • Steeper learning curve for beginners
  • Less focus on reinforcement learning techniques
  • Requires more setup and configuration

Code Comparison

FinRL example (reinforcement learning approach):

env = StockTradingEnv(df=df, ...)
model = PPO("MlpPolicy", env, ...)
model.learn(total_timesteps=50000)

Lean example (traditional algorithmic approach):

public class MyAlgorithm : QCAlgorithm
{
    public override void Initialize()
    {
        SetStartDate(2020, 1, 1);
        SetCash(100000);
        AddEquity("SPY", Resolution.Daily);
    }
}

FinRL focuses on applying reinforcement learning to financial trading, making it easier to implement RL-based strategies. Lean, on the other hand, provides a more traditional algorithmic trading framework with a wider range of features and asset classes. FinRL is more suitable for researchers and those interested in RL applications in finance, while Lean is better suited for production-ready algorithmic trading systems.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

image

FinRL: Financial Reinforcement Learning twitter facebook google+ linkedin

Downloads Downloads Python 3.6 PyPI Documentation Status License

FinGPT: Open-source for open-finance! Revolutionize FinTech.

Check out our latest competition: ACM ICAIF 2023 FinRL Contest

Visitors

Financial reinforcement learning (FinRL) (Document website) is the first open-source framework for financial reinforcement learning. FinRL has evolved into an ecosystem

Dev RoadmapStageUsersProjectDesription
0.0 (Preparation)entrancepractitionersFinRL-Metagym-style market environments
1.0 (Proof-of-Concept)full-stackdevelopersthis repoautomatic pipeline
2.0 (Professional)professionexpertsElegantRLalgorithms
3.0 (Production)servicehedge fundsPodracercloud-native deployment

Outline

Overview

FinRL has three layers: market environments, agents, and applications. For a trading task (on the top), an agent (in the middle) interacts with a market environment (at the bottom), making sequential decisions.

A quick start: Stock_NeurIPS2018.ipynb. Videos FinRL at AI4Finance Youtube Channel.

File Structure

The main folder finrl has three subfolders applications, agents, meta. We employ a train-test-trade pipeline with three files: train.py, test.py, and trade.py.

FinRL
├── finrl (main folder)
│   ├── applications
│   	├── Stock_NeurIPS2018
│   	├── imitation_learning
│   	├── cryptocurrency_trading
│   	├── high_frequency_trading
│   	├── portfolio_allocation
│   	└── stock_trading
│   ├── agents
│   	├── elegantrl
│   	├── rllib
│   	└── stablebaseline3
│   ├── meta
│   	├── data_processors
│   	├── env_cryptocurrency_trading
│   	├── env_portfolio_allocation
│   	├── env_stock_trading
│   	├── preprocessor
│   	├── data_processor.py
│       ├── meta_config_tickers.py
│   	└── meta_config.py
│   ├── config.py
│   ├── config_tickers.py
│   ├── main.py
│   ├── plot.py
│   ├── train.py
│   ├── test.py
│   └── trade.py
│
├── examples
├── unit_tests (unit tests to verify codes on env & data)
│   ├── environments
│   	└── test_env_cashpenalty.py
│   └── downloaders
│   	├── test_yahoodownload.py
│   	└── test_alpaca_downloader.py
├── setup.py
├── requirements.txt
└── README.md

Supported Data Sources

Data SourceTypeRange and FrequencyRequest LimitsRaw DataPreprocessed Data
AkshareCN Securities2015-now, 1dayAccount-specificOHLCVPrices&Indicators
AlpacaUS Stocks, ETFs2015-now, 1minAccount-specificOHLCVPrices&Indicators
BaostockCN Securities1990-12-19-now, 5minAccount-specificOHLCVPrices&Indicators
BinanceCryptocurrencyAPI-specific, 1s, 1minAPI-specificTick-level daily aggegrated trades, OHLCVPrices&Indicators
CCXTCryptocurrencyAPI-specific, 1minAPI-specificOHLCVPrices&Indicators
EODhistoricaldataUS SecuritiesFrequency-specific, 1minAPI-specificOHLCVPrices&Indicators
IEXCloudNMS US securities1970-now, 1 day100 per second per IPOHLCVPrices&Indicators
JoinQuantCN Securities2005-now, 1min3 requests each timeOHLCVPrices&Indicators
QuantConnectUS Securities1998-now, 1sNAOHLCVPrices&Indicators
RiceQuantCN Securities2005-now, 1msAccount-specificOHLCVPrices&Indicators
SinopacTaiwan securities2023-04-13~now, 1minAccount-specificOHLCVPrices&Indicators
TushareCN Securities, A share-now, 1 minAccount-specificOHLCVPrices&Indicators
WRDSUS Securities2003-now, 1ms5 requests each timeIntraday TradesPrices&Indicators
YahooFinanceUS SecuritiesFrequency-specific, 1min2,000/hourOHLCVPrices&Indicators

OHLCV: open, high, low, and close prices; volume. adjusted_close: adjusted close price

Technical indicators: 'macd', 'boll_ub', 'boll_lb', 'rsi_30', 'dx_30', 'close_30_sma', 'close_60_sma'. Users also can add new features.

Installation

Status Update

Version History [click to expand]
  • 2022-06-25 0.3.5: Formal release of FinRL, neo_finrl is chenged to FinRL-Meta with related files in directory: meta.
  • 2021-08-25 0.3.1: pytorch version with a three-layer architecture, apps (financial tasks), drl_agents (drl algorithms), neo_finrl (gym env)
  • 2020-12-14 Upgraded to Pytorch with stable-baselines3; Remove tensorflow 1.0 at this moment, under development to support tensorflow 2.0
  • 2020-11-27 0.1: Beta version with tensorflow 1.5

Tutorials

A complete list at blogs

Publications

TitleConference/JournalLinkCitationsYear
Dynamic Datasets and Market Environments for Financial Reinforcement LearningMachine Learning - Springer Naturepaper code72024
FinRL-Meta: FinRL-Meta: Market Environments and Benchmarks for Data-Driven Financial Reinforcement LearningNeurIPS 2022paper code372022
FinRL: Deep reinforcement learning framework to automate trading in quantitative financeACM International Conference on AI in Finance (ICAIF)paper492021
FinRL: A deep reinforcement learning library for automated stock trading in quantitative financeNeurIPS 2020 Deep RL Workshoppaper872020
Deep reinforcement learning for automated stock trading: An ensemble strategyACM International Conference on AI in Finance (ICAIF)paper code1542020
Practical deep reinforcement learning approach for stock tradingNeurIPS 2018 Workshop on Challenges and Opportunities for AI in Financial Servicespaper code1642018

News

Citing FinRL

@article{dynamic_datasets,
    author = {Liu, Xiao-Yang and Xia, Ziyi and Yang, Hongyang and Gao, Jiechao and Zha, Daochen and Zhu, Ming and Wang, Christina Dan and Wang, Zhaoran and Guo, Jian},
    title = {Dynamic Datasets and Market Environments for Financial Reinforcement Learning},
    journal = {Machine Learning - Springer Nature},
    year = {2024}
}
@article{liu2022finrl_meta,
  title={FinRL-Meta: Market Environments and Benchmarks for Data-Driven Financial Reinforcement Learning},
  author={Liu, Xiao-Yang and Xia, Ziyi and Rui, Jingyang and Gao, Jiechao and Yang, Hongyang and Zhu, Ming and Wang, Christina Dan and Wang, Zhaoran and Guo, Jian},
  journal={NeurIPS},
  year={2022}
}
@article{liu2021finrl,
    author  = {Liu, Xiao-Yang and Yang, Hongyang and Gao, Jiechao and Wang, Christina Dan},
    title   = {{FinRL}: Deep reinforcement learning framework to automate trading in quantitative finance},
    journal = {ACM International Conference on AI in Finance (ICAIF)},
    year    = {2021}
}

@article{finrl2020,
    author  = {Liu, Xiao-Yang and Yang, Hongyang and Chen, Qian and Zhang, Runjia and Yang, Liuqing and Xiao, Bowen and Wang, Christina Dan},
    title   = {{FinRL}: A deep reinforcement learning library for automated stock trading in quantitative finance},
    journal = {Deep RL Workshop, NeurIPS 2020},
    year    = {2020}
}
@article{liu2018practical,
  title={Practical deep reinforcement learning approach for stock trading},
  author={Liu, Xiao-Yang and Xiong, Zhuoran and Zhong, Shan and Yang, Hongyang and Walid, Anwar},
  journal={NeurIPS Workshop on Deep Reinforcement Learning},
  year={2018}
}

We published FinRL papers that are listed at Google Scholar. Previous papers are given in the list.

Join and Contribute

Welcome to AI4Finance community!

Please check Contributing Guidances.

Contributors

Thank you!

LICENSE

MIT License

Disclaimer: We are sharing codes for academic purpose under the MIT education license. Nothing herein is financial advice, and NOT a recommendation to trade real money. Please use common sense and always first consult a professional before trading or investing.