Real-Time Machine Learning on Blockchain Data
Real-Time Machine Learning on Blockchain Data
Photo by Picas Joe on Pexels
Quick Answer: Real-time ML on blockchain data processes transactions and state changes as they happen — within the block time (12s on Ethereum, 0.4s on Solana) — to make predictions and trigger actions. The production stack: Substreams (streaming blockchain data at 100MB/s with parallel processing) → feature pipeline (compute transaction graphs, mempool features, DeFi state vectors) → model inference (ONNX Runtime, TensorRT at 1-5ms latency) → action (trading, MEV extraction, risk alerts, liquidation triggers). Critical applications: (1) MEV opportunity detection — predicting profitable sandwich/arbitrage opportunities before they're fully visible; (2) Liquidation forecasting — predicting which DeFi positions will be liquidated within the next 10 blocks (Accuracy: 85%+ for Aave and Compound); (3) Anomaly detection — detecting bridge hacks, oracle attacks, and flash loan exploits within 1-2 blocks (recall: 90%+). The key challenge: blockchain data is sparse, high-dimensional, and timing-sensitive — features must be computed incrementally from raw blocks in under 100ms.
Architecture Overview
┌─────────────────────────────────────────────────────────────────────┐
│ Real-Time ML Pipeline for Blockchain │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Block N produced (12s) │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Data Ingestion Layer │ │
│ │ ├── Substreams (parallel block processing) │ │
│ │ ├── Mempool listener (pending txs) │ │
│ │ └── RPC node (for state queries) │ │
│ └─────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Feature Engineering Layer │ │
│ │ ├── Transaction graph (sender, receiver, value) │ │
│ │ ├── DeFi state (liquidity pools, orders, positions) │ │
│ │ ├── MEV features (pending bundles, gas auctions) │ │
│ │ └── Market data (CEX/DEX prices, volatility) │ │
│ └─────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Inference Layer │ │
│ │ ├── ONNX Runtime (CPU, 5ms inference) │ │
│ │ ├── TensorRT (GPU, 1ms inference) │ │
│ │ └── Ensemble of 3-5 models │ │
│ └─────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Action Layer │ │
│ │ ├── Trading/MEV execution (Flashbots bundle) │ │
│ │ ├── Risk alert (Slack/PagerDuty) │ │
│ │ ├── Liquidation trigger (automated) │ │
│ │ └── Dashboard update (Grafana) │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ End-to-end latency target: <500ms from block production to action │
└───────────────────────────────────────────────────────────────────────┘
Feature Engineering
Transaction Graph Features
Blockchain transaction data is inherently graph-structured:
Feature Type Feature Computation
──────────────────────────────────────────────────────────────────────
Temporal tx_count_per_block Count
gas_price_percentile SQL window
block_interval ∆(block_time)
Graph-based sender_degree Node degree
recipient_degree Node degree
clustering_coeff Triangle count
pagerank_score Pagerank
DeFi State pool_depth State query
swap_volume_1h Rolling window
price_impact_last_tx ∆(pool reserves)
net_flow_1h ∑(in - out)
Mempool pending_tx_count Count
pending_gas_avg Average
bundle_count Count
sandwich_opportunity_score Heuristic
Anomaly tx_value_zscore (value - µ) / σ
gas_price_zscore (gas - µ) / σ
contract_interaction_count Unique contracts
time_since_last_interaction ∆(time)
Mempool Features For MEV
The mempool contains pending (unconfirmed) transactions. ML on mempool data predicts:
class MempoolFeatureExtractor:
"""Extract real-time features from the mempool."""
def extract_features(self, mempool_snapshot):
features = {}
# Pending transaction volume
features["pending_tx_count"] = len(mempool_snapshot)
features["pending_total_value"] = sum(
tx.value for tx in mempool_snapshot if tx.value
)
# Gas analysis
gas_prices = [tx.gas_price for tx in mempool_snapshot]
features["gas_p50"] = median(gas_prices)
features["gas_p95"] = percentile(gas_prices, 95)
features["gas_std"] = stdev(gas_prices)
features["gas_spike"] = features["gas_p95"] > features["gas_p50"] * 3
# DEX-specific (Uniswap-like)
features["pending_swaps"] = sum(
1 for tx in mempool_snapshot
if tx.is_swap and tx.protocol in ["uniswap", "sushiswap"]
)
features["pending_swap_volume"] = sum(
tx.value for tx in mempool_snapshot
if tx.is_swap
)
# Sandwich opportunity detection
# A sandwich opportunity exists when:
# 1. Large swap queued (target)
# 2. No pending frontrun/backrun yet
# 3. Pool has sufficient liquidity
sandwich_scores = []
for tx in mempool_snapshot:
if tx.is_swap and tx.value > LARGE_TX_THRESHOLD:
score = self._estimate_sandwich_profit(tx)
sandwich_scores.append(score)
features["max_sandwich_score"] = max(sandwich_scores) if sandwich_scores else 0
features["n_sandwich_opportunities"] = len([
s for s in sandwich_scores if s > MIN_SANDWICH_PROFIT
])
return features
def _estimate_sandwich_profit(self, swap_tx):
"""Estimate potential profit from sandwiching this swap."""
# Returns expected profit in ETH
# Based on: swap size, pool depth, current price
# Simplified heuristic:
pool_depth = self.get_pool_depth(swap_tx.pool)
price_impact = swap_tx.value / pool_depth
expected_profit = swap_tx.value * (price_impact ** 2) * 0.5
return expected_profit
Photo by Morthy Jameson on Pexels
Liquidation Forecasting
Predicting which DeFi positions will be liquidated before it happens:
class LiquidationForecaster:
"""Predict imminent liquidations in Aave/Compound."""
def __init__(self, model_path):
self.model = self.load_lightgbm(model_path)
self.position_cache = {}
def update_and_predict(self, block_data) -> list[dict]:
"""Process block and return liquidation predictions."""
# 1. Update all tracked positions with new prices
for pos_id, pos in self.position_cache.items():
pos.collateral_value = self.get_latest_price(
pos.collateral_asset, block_data
) * pos.collateral_amount
pos.debt_value = self.get_latest_price(
pos.debt_asset, block_data
) * pos.debt_amount
pos.health_factor = (
pos.collateral_value * pos.liquidation_threshold
) / pos.debt_value
pos.chain = block_data.chain
# 2. Feature engineering for each position
predictions = []
for pos_id, pos in self.position_cache.items():
if pos.health_factor > 3.0:
continue # Safe, skip
features = {
"health_factor": pos.health_factor,
"hf_change_24h": pos.hf_history[-1] - pos.hf_history[-24],
"hf_volatility_24h": np.std(pos.hf_history[-24:]),
"collateral_price_vol": pos.collateral_price_volatility,
"debt_price_vol": pos.debt_price_volatility,
"position_age_days": pos.age_days,
"n_previous_repayments": pos.n_repayments,
"time_since_last_repayment_hours": pos.time_since_last_repay,
"gas_price_percentile_95": pos.current_gas_p95,
"liquidator_present": self._is_liquidator_active(pos),
"block_time_of_day": block_data.timestamp % 86400,
"is_whale": float(pos.total_value_usd > 1_000_000),
}
# 3. Predict liquidation probability (next 10 blocks)
prob = self.model.predict_proba(pd.DataFrame([features]))[0, 1]
if prob > 0.1:
predictions.append({
"position_id": pos_id,
"protocol": pos.protocol,
"health_factor": pos.health_factor,
"liquidation_prob_next10blocks": prob,
"estimated_value_at_risk_usd": pos.total_value_usd,
"predicted_remaining_blocks": self._estimate_remaining_blocks(prob),
})
# 4. Sort by urgency
predictions.sort(key=lambda x: -x["liquidation_prob_next10blocks"])
return predictions[:100] # Top 100 most at risk
def _estimate_remaining_blocks(self, prob):
"""Estimate how many blocks before liquidation."""
# Simple: 1/prob (if prob=0.25 → ~4 more blocks)
return max(1, int(1 / prob))
Anomaly Detection
Real-time anomaly detection flags suspicious blockchain activity:
class BlockchainAnomalyDetector:
"""Detect anomalous on-chain activity in real-time."""
def __init__(self):
# Multi-model ensemble
self.isolation_forest = IsolationForest(contamination=0.01)
self.autoencoder = self._build_autoencoder()
self.rule_engine = RuleBasedDetector()
def score_block(self, block):
"""Score block for anomalous activity. Returns n highest anomalies."""
# Extract features
features = self._extract_block_features(block)
# 1. Isolation Forest score (density-based)
if_score = self.isolation_forest.score_samples([features])[0]
# 2. Autoencoder reconstruction error
ae_error = self._autoencoder_reconstruction_error(features)
# 3. Rule-based detection
rule_flags = self.rule_engine.check(features)
# Composite anomaly score
anomaly_score = (
0.3 * normalize(-if_score) + # Lower = more anomalous
0.3 * normalize(ae_error) + # Higher = more anomalous
0.4 * rule_flags # Rule matches
)
return {
"block": block.number,
"anomaly_score": anomaly_score,
"contributing_factors": {
"isolation_forest": -if_score,
"autoencoder_error": ae_error,
"rules_triggered": rule_flags > 0,
},
"is_anomaly": anomaly_score > ANOMALY_THRESHOLD,
}
Production Case Studies
| Company | Application | Model | Latency | Accuracy | Volume |
|---|---|---|---|---|---|
| EigenPhi | MEV opportunity detection | GNN + XGBoost | 150ms | 92% recall | $50M/day |
| Gauntlet | Liquidation risk | LightGBM | 50ms | 85% precision | 50K pos. |
| Chaos Labs | Anomaly detection | Autoencoder + rules | 200ms | 95% recall | 1M tx/hr |
| Flashbots | MEV-Boost bid estimation | MLP | 5ms | N/A | 95% of Eth blocks |
| TRM Labs | AML/chain analysis | Graph ML | 30s | 98% | All chains |
Related Reads
- ZKML: Zero-Knowledge Machine Learning — Verifiable Inference
- AI-Native L1s and L2s: Blockchains Built for AI Workloads
- On-Chain AI Inference: ZK-Proofs, Trusted Execution, and Verifiable Machine Learning
Optimizing Feature Pipelines for Blockchain Bursts
Blockchain data arrives in bursts — every 12 seconds on Ethereum, 0.4 seconds on Solana — and your feature pipeline must handle these spikes without falling behind. The key is incremental computation: instead of recomputing features from scratch for each block, maintain running aggregates and update them as new data arrives. For example, use exponential moving averages (EMAs) for volume features, HyperLogLog for unique counts (e.g., unique_senders_per_block), and incremental PageRank for graph-based features like sender_pagerank.
Substreams enables this by processing blocks in parallel (up to 100MB/s) and emitting deltas — changes between blocks — rather than full snapshots. This allows you to update features like tx_count_per_block or pool_depth in constant time, regardless of chain length. For DeFi state features (e.g., liquidity_pool_reserves), use state queries from an RPC node to fetch only the data you need for active positions or pools, avoiding full chain rescans. Cache these queries aggressively, as they’re often the bottleneck in feature freshness.
For mempool features, the challenge is volatility — pending transactions can appear, disappear, or be replaced within milliseconds. To handle this, snapshot the mempool at fixed intervals (e.g., every 100ms) and compute features like pending_gas_p95 or sandwich_opportunity_score on these snapshots. Use sliding windows for temporal features (e.g., swap_volume_1h) to avoid recomputing from scratch. For graph-based features, maintain a transaction graph in memory and update it incrementally as new blocks arrive, using algorithms like dynamic PageRank to keep node scores fresh.
Handling Non-Stationarity in Blockchain Data
Blockchain data is highly non-stationary — patterns that work in a bull market (e.g., high gas prices, large swaps) fail in a bear market (e.g., low activity, small positions). This drift causes models to degrade over time, often within weeks. The solution is to design features and models that are robust to regime shifts. Start by training on multi-cycle data (e.g., 2017-2022 bull/bear) to expose the model to different market conditions. Use time-series cross-validation with purged walk-forward splits to avoid lookahead bias and ensure the model generalizes to unseen regimes.
Feature engineering plays a critical role in handling non-stationarity. Avoid raw features like tx_value or gas_price, which vary wildly across regimes. Instead, use relative features like tx_value_zscore (value normalized by recent mean/std) or gas_price_percentile (position in the recent gas price distribution). For temporal features, use rolling windows (e.g., swap_volume_1h) to adapt to changing activity levels. For DeFi positions, track features like health_factor_change_24h rather than absolute values, as these are more stable across market conditions.
Model selection also matters. Tree-based models (XGBoost, LightGBM) are more robust to feature drift than deep learning models, as they rely on splits rather than fixed weights. Use ensembles of models trained on different periods (e.g., one model per year) to capture regime-specific patterns. Monitor feature stability with drift detection (e.g., Kolmogorov-Smirnov test for continuous features, chi-squared for categorical) and retrain models when drift exceeds thresholds. Finally, add context features like market_regime (bull/bear/neutral) or volatility_cluster to help the model adapt its predictions to the current environment.
Real-Time Action Execution: From Prediction to Profit
Real-time ML on blockchain data is only valuable if it triggers actions — trading, MEV extraction, risk alerts, or liquidation triggers — within the same block time. The action layer must be tightly integrated with the inference layer to minimize latency. For MEV extraction, use Flashbots bundles to submit transactions directly to validators, bypassing the public mempool. For liquidation triggers, automate the process with smart contracts that execute liquidations as soon as a position’s health factor drops below the threshold. For risk alerts, use low-latency channels like Slack or PagerDuty to notify teams within seconds of an anomaly detection.
Latency is the enemy of profit in real-time blockchain ML. Every millisecond counts when competing for MEV opportunities or liquidations. To minimize latency, colocate your inference pipeline with the blockchain node (e.g., run Substreams and ONNX Runtime on the same machine as the RPC node). Use gRPC or WebSockets for low-latency communication between layers, and avoid serialization overhead by keeping data in memory (e.g., use Apache Arrow for feature vectors). For MEV extraction, pre-sign transactions and cache them to avoid last-minute signing delays.
Risk management is critical when automating actions. For MEV extraction, set strict profit thresholds (e.g., only execute sandwiches with expected profit > 0.1 ETH) and gas price limits to avoid losses. For liquidation triggers, add safety checks (e.g., confirm the position’s health factor is still below the threshold before executing) to avoid false positives. For anomaly detection, use a tiered alert system: low-confidence anomalies trigger Slack alerts for human review, while high-confidence anomalies (e.g., bridge hacks) trigger automated circuit breakers to pause withdrawals.
For high-frequency applications like MEV extraction, consider using FPGAs or ASICs to accelerate inference. For example, TensorRT on NVIDIA GPUs can reduce inference latency to 1ms for deep learning models, giving you a competitive edge in gas auctions. For liquidation forecasting, use batch inference to scan thousands of positions in parallel, then prioritize the most at-risk positions for real-time monitoring. The goal is to balance speed and accuracy: fast enough to act within the block time, but accurate enough to avoid costly mistakes.
Key Takeaways
- End-to-end latency must stay under 500ms from use Substreams (100MB/s parallel block processing) → incremental feature pipeline (compute transaction graphs, mempool features, DeFi state vectors in <100ms) → ONNX Runtime/TensorRT inference (1-5ms) → Flashbots bundle or automated liquidation trigger within the same block time (12s Ethereum, 0.4s Solana).
- For MEV opportunity detection compute mempool features like
pending_swap_volume,gas_p95, andsandwich_opportunity_score(heuristic estimating profit based on swap size, pool depth, and current price) within 50ms of mempool snapshot to predict profitable sandwich/arbitrage opportunities before they're fully visible. - Liquidation forecasting requires tracking DeFi position health factors incrementally with features like
hf_change_24h,collateral_price_vol,gas_price_percentile_95, andliquidator_presentto predict which positions will be liquidated within the next 10 blocks (85%+ accuracy for Aave/Compound). - Anomaly detection combines Isolation Forest (density-based), autoencoder (reconstruction error), and rule-based heuristics (e.g., flash loan testing, oracle deviation probing) to flag bridge hacks, oracle attacks, and exploits within 1-2 blocks (90%+ recall).
- Feature freshness is the hardest constraint — use streaming features (HyperLogLog for unique counts, exponential moving averages for volume, incremental PageRank for graph features) and avoid full chain rescans for each block.
- ONNX Runtime on CPU is sufficient for real-time inference (1-5ms latency for XGBoost/LightGBM); reserve GPUs for training GNNs or massive batch inference (1M+ positions).
Frequently Asked Questions
What's the hardest part of real-time blockchain ML?
Feature freshness. Block data arrives in bursts (every 12 seconds on Ethereum), and features must be computed incrementally. You can't re-scan the entire chain for each block. The solution: streaming features (HyperLogLog for unique counts, exponential moving averages for volume, incremental PageRank for graph features). State management (which DeFi positions are active) is also surprisingly hard at scale.
Do you need a GPU for real-time blockchain ML?
Usually not for inference — ONNX Runtime on CPU handles 1-5ms inference for tree-based models (XGBoost, LightGBM) very efficiently. GPUs are useful for: (1) training (GNNs on transaction graphs), (2) deep learning models for anomaly detection, (3) massive batch inference (scanning 1M+ positions). For real-time, a well-optimized XGBoost on CPU often beats deep learning on GPU in cost and latency.
How do you prevent overfitting on blockchain data?
Blockchain data is highly non-stationary — patterns that worked in a bull market fail in a bear market. Mitigations: (1) train on multi-cycle data (2017-2022 bull/bear), (2) time-series cross-validation (purged walk-forward), (3) feature stability monitoring (drift detection), (4) ensemble of models trained on different periods, (5) context features (market regime classifier). A model that works in bull and bear is rare and valuable.
Can ML predict hacks?
ML can predict anomalous pre-hack behavior with ~90% recall and ~60% precision. Typical signals: (1) test transactions from suspect addresses, (2) new contract interactions, (3) flash loan testing, (4) oracle deviation probing, (5) governance token accumulation. The challenge is false positives — many genuine users test transactions. The best approach combines ML anomaly detection with rule-based heuristics and human review.
More from Synor
Recommended for you
Comments
Sign in to join the conversation
No comments yet. Be the first to share your thoughts!