Training an RL Agent (PPO/SAC/DQN) for a Trading Strategy

Training an RL Agent (PPO/SAC/DQN) for a Trading Strategy

Imagine: you've spent months training a DQN agent on historical data, only to have it lose capital on the live market due to unaccounted slippage. In one of our projects, a client came with a similar problem: they trained PPO on minute candles, achieving a Sharpe ratio of 1.8 in testing, but in live trading the drawdown reached 40%. The client was losing around $15,000 monthly because of these shortcomings. We discovered that the environment did not account for fees and liquidity. After calibrating the reward and adding walk-forward validation, the Sharpe ratio climbed back to 1.5 and the drawdown dropped to 12%. This saved $4,000 per month.

Designing an RL agent for crypto trading is not just about picking an algorithm. You face market non-stationarity, hidden fees, slippage, and the risk of overfitting. We take on the full cycle—from building the data pipeline to live trading. We use proven algorithms PPO, SAC, and DQN, adapting them to your strategy. Our experience: over five years in blockchain development, 15+ projects for DeFi and CEX, including integration with Binance API. Contact us for a detailed analysis of your strategy.

Three Working Algorithms: DQN, PPO, SAC

Each algorithm has its niche. Let's look at their strengths and typical use cases.

DQN (Deep Q-Network)

Suitable for discrete actions (buy/hold/sell) and simple strategies. DQN approximates the Q-function: Q(state, action) — the expected discounted reward for taking action in state.

import torch import torch.nn as nn from collections import deque import random class DQNNetwork(nn.Module): def __init__(self, state_dim, n_actions, hidden_dim=256): super().__init__() # Dueling architecture: separate Value and Advantage streams self.shared = nn.Sequential( nn.Linear(state_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, hidden_dim), nn.ReLU() ) self.value_stream = nn.Linear(hidden_dim, 1) self.advantage_stream = nn.Linear(hidden_dim, n_actions) def forward(self, x): shared = self.shared(x) value = self.value_stream(shared) advantage = self.advantage_stream(shared) # Dueling: Q = V + (A - mean(A)) q_values = value + (advantage - advantage.mean(dim=1, keepdim=True)) return q_values class PrioritizedReplayBuffer: """Prioritized Experience Replay — sample important transitions more often""" def __init__(self, capacity=50000, alpha=0.6): self.buffer = deque(maxlen=capacity) self.priorities = deque(maxlen=capacity) self.alpha = alpha def push(self, state, action, reward, next_state, done, td_error=1.0): priority = (abs(td_error) + 1e-5) ** self.alpha self.buffer.append((state, action, reward, next_state, done)) self.priorities.append(priority) def sample(self, batch_size, beta=0.4): probs = np.array(self.priorities) / sum(self.priorities) indices = np.random.choice(len(self.buffer), batch_size, p=probs) # Importance sampling weights weights = (len(self.buffer) * probs[indices]) ** (-beta) weights /= weights.max() batch = [self.buffer[i] for i in indices] return batch, indices, weights 

Double DQN eliminates Q-value overestimation: the online network selects the action, and the target network evaluates it.

# Double DQN target calculation with torch.no_grad(): next_actions = online_net(next_states).argmax(dim=1) # online net selects next_q = target_net(next_states).gather(1, next_actions.unsqueeze(1)) # target evaluates targets = rewards + gamma * next_q * (1 - dones) 

PPO (Proximal Policy Optimization)

Suitable for both discrete and continuous actions, on-policy, stable training. PPO limits the policy update size via clipping.

class PPOActor(nn.Module): def __init__(self, state_dim, action_dim, hidden_dim=256): super().__init__() self.network = nn.Sequential( nn.Linear(state_dim, hidden_dim), nn.Tanh(), nn.Linear(hidden_dim, hidden_dim), nn.Tanh() ) self.policy_head = nn.Linear(hidden_dim, action_dim) self.value_head = nn.Linear(hidden_dim, 1) def forward(self, x): features = self.network(x) logits = self.policy_head(features) value = self.value_head(features) return logits, value def ppo_update(model, optimizer, states, actions, old_log_probs, advantages, returns, clip_eps=0.2, n_epochs=4): for _ in range(n_epochs): logits, values = model(states) dist = torch.distributions.Categorical(logits=logits) new_log_probs = dist.log_prob(actions) entropy = dist.entropy() # PPO clipped objective ratio = (new_log_probs - old_log_probs).exp() surr1 = ratio * advantages surr2 = torch.clamp(ratio, 1 - clip_eps, 1 + clip_eps) * advantages actor_loss = -torch.min(surr1, surr2).mean() critic_loss = (returns - values.squeeze()).pow(2).mean() entropy_loss = -entropy.mean() total_loss = actor_loss + 0.5 * critic_loss + 0.01 * entropy_loss optimizer.zero_grad() total_loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5) optimizer.step() 

SAC (Soft Actor-Critic)

Suitable for continuous action space (positioning 0%–100% of capital), off-policy, maximum sample efficiency. SAC maximizes: J(π) = E[Σ γ^t (r_t + α H(π(·|s_t)))]. The entropy term H encourages exploration.

class SACActorContinuous(nn.Module): def __init__(self, state_dim, action_dim, hidden_dim=256): super().__init__() self.network = nn.Sequential( nn.Linear(state_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, hidden_dim), nn.ReLU() ) self.mean_head = nn.Linear(hidden_dim, action_dim) self.log_std_head = nn.Linear(hidden_dim, action_dim) def forward(self, x): features = self.network(x) mean = self.mean_head(features) log_std = self.log_std_head(features).clamp(-20, 2) std = log_std.exp() dist = torch.distributions.Normal(mean, std) action = dist.rsample() # reparameterization trick # Squash to [-1, 1] action_tanh = torch.tanh(action) log_prob = dist.log_prob(action) - torch.log(1 - action_tanh.pow(2) + 1e-6) return action_tanh, log_prob.sum(-1, keepdim=True) 

Which Algorithm to Choose for Your Strategy?

The choice depends on the action space and sample efficiency requirements. If your strategy uses only discrete signals (buy/sell/hold), DQN with dueling and PER will give stable results. For continuous capital management (e.g., portfolio percentage), SAC is unrivaled: it's 2–3 times more sample-efficient than PPO. PPO is a universal choice when you need reliability and ease of tuning.

We often combine algorithms in a multi-agent architecture: a macro-agent based on DQN determines the overall direction, and a micro-agent based on SAC executes trades. This reduces variance and improves the Sharpe ratio by 15–30%.

Why Is a Correct Reward Function Important?

Reward shaping is a key step that determines the agent's behavior. Typical mistakes: the agent learns to accumulate unrealized profit (without accounting for slippage) or starts trading very rarely to avoid fees. We use a multi-component reward: PnL, drawdown penalty, fees, and spread. For example, reward = ΔP&L - λ1 * fee - λ2 * max_drawdown. The λ coefficients are chosen to simulate realistic conditions.

In one DeFi project, an incorrect reward led the agent to open hundreds of micro-trades, generating a loss from fees. After redesigning the reward (penalty for the number of trades), the agent became profitable.

Algorithm Comparison for Crypto Trading

How to Set Up RL Agent Training: Step-by-Step Plan

1. Define the action space and state space. For discrete actions (buy/sell/hold), DQN is suitable; for continuous positioning, SAC. The state includes prices, volumes, indicators.
2. Design the reward function. Account for PnL, fees, slippage, drawdown penalty.
3. Choose the algorithm and neural network architecture. We use dueling DQN, PPO with clipping, SAC with automatic entropy tuning.
4. Train with validation. We apply walk-forward validation with 36 rolling windows and early stopping.
5. Test on out-of-sample data. Evaluate Sharpe ratio, max drawdown, reward stability.

Typical Challenges and How We Solve Them

Market non-stationarity — an agent trained on a calm market may fail in high volatility. As noted in reinforcement learning specification, distribution shift is a serious challenge. We use curriculum learning: gradually increase environment volatility, and in production — continuous fine-tuning with a drift detector.

Reward hacking — artificially inflated rewards. Protection via reward clipping and using a realistic simulator with market data (Level 2, historical candles).

Overfitting — agent memorization. We use walk-forward validation with 36 rolling windows and testing on fully excluded periods (out-of-sample).

What's Included in the Work

• Strategy analysis and data pipeline preparation.
• Custom environment design (gymnasium) accounting for fees, slippage, and drawdowns.
• Algorithm and neural network architecture selection.
• Training with hyperparameter tuning (grid/random search, Bayesian optimization).
• Walk-forward validation and robustness to market regime changes.
• Integration with broker API (Binance, Bybit, KuCoin).
• Documentation, training your team, 3-month support.

Contact us so we can analyze your strategy. We guarantee result quality and support at all stages.

Estimated Timelines and Stages

Order a consultation — we will select the algorithm and architecture for your task. We will evaluate the project for free within 2 business days.