Jekyll2024-07-06T14:47:42+00:00https://algoquant.github.io//feed.xmlalgoquant Homealgoquant's Home Page on GitHubBollinger Strategy Implementation2024-06-30T00:00:00+00:002024-06-30T00:00:00+00:00https://algoquant.github.io//2024/06/30/Bollinger-Strategy-Implementation Bollinger Strategy Implementation

Configuration of the Bollinger Strategy in MachineTrader

The Bollinger strategy trades a single stock during the market trading hours, using intraday stock prices. Each day it starts trading at the market open with no position and a clean slate (reset to its base state), and it stops trading at the market close. It liquidates all its positions just before the market close. Its state is reset overnight to its base state.

You can read more about the theory of the Bollinger strategy in this document.

To implement the Bollinger strategy in MachineTrader, you need to install the tab named Bollinger Strategy in your MachineTrader instance.
You can click this link, to download the file named Bollinger_Strategy.json from the MachineTrader Community share drive.
This file contains the JavaScript code for the tab named Bollinger Strategy. You can then import this JavaScript file into your MachineTrader instance, and it will create the tab named Bollinger Strategy.


Implementation of the Bollinger Strategy

The tab named Bollinger Strategy executes the Bollinger strategy in real time.

Let’s look at the different flows in the tab named Bollinger Strategy.

At the top of the tab, there are flows for downloading the documentation. You can click on the gray inject nodes to download the documentation. The second flow downloads the current file that you’re reading now.


The flows below create the flow variables in the tab Bollinger Strategy.
Flow variables are variables that are shared between the different nodes in the same tab. Once the variables are created, they can be used in other function nodes in the same tab.
You need to run (click) these flows once before the other flows are run. You don’t need to run these flows every day.

The function called Create flow variables creates the flow variables, such as the stock symbol to trade, the threshold level for the z-score, the position limit (the maximum number of shares the strategy can own at any time), etc.

The Bollinger strategy maintains two PostgreSQL tables, one for the submitted trade orders and one for the executed (filled) orders.
The functions called Create table tableSubmits** and Create table tableFills** create empty PostgreSQL tables (if they don’t already exist), for storing the submitted orders and for the executed (filled) orders.
The table called tableSubmits is filled with the orders submitted to the broker API, and the table called tableFills is filled with the orders that were confirmed by the broker.
These two tables are erased overnight, so they start empty at the beginning of each trading day.


The flow below initializes the flow variables just before the market open. The flow runs automatically - you don’t need to click on it.

The flow initializes the flow variables such as the open position queue, etc., each day just before the market open.
The function node called Initialize flow variables resets the flow variables to their initial values.


The flow below initializes the prices just after the market open. This flow runs automatically - you don’t need to click on it.

The flow initializes the flow variables such as the previous valid price, the EMA price, the price variance, etc.


The flow below updates the technical indicators with live (recent) prices. The flow runs automatically - you don’t need to click on it.

The first node in this flow is an inject node that runs every \(10\) seconds. That’s the frequency at which the strategy is run (updated). It can be changed by the user as needed.
The inject node activates the whole flow, which updates the technical indicators with the live (current) prices.

The first function in this flow is called Get stock price.
It extracts the live (current) price of the given (target) stock from the global array variable containing the last prices of many stocks. This array is updated every second in the tab called Alpaca Prices.

The function called Calculate EMA price and volatility obtains the stock price from the function Get stock price through a connector, and it calculates (updates) the EMA moving average price and volatility.
It also scrubs the prices from bad data using the dollar stock return (equal to the new price minus previous price). If the stock return is above the threshold, then the new price is ignored, and the previous valid price is used.

The function called Update tech indicators calculates the z-scores, using the the average prices and volatilities from the function Calculate EMA price and volatility.


The flow below runs the Bollinger strategy. The flow runs automatically - you don’t need to click on it.

The first function in this flow called Calculate PnL calculates the unrealized profits and losses (PnLs) of the open positions, and also the PnL drawdown from its maximum.
The Bollinger strategy maintains a queue of the strategy’s open positions throughout the day.
The function Calculate PnL matches the open positions in the queue with the current stock price, and calculates the unrealized PnL.

The function called Bollinger strategy executes the Bollinger strategy, using the z-scores from the function called Update tech indicators in the flow above it.
The function Bollinger strategy compares the z-scores with the threshold level \(\theta\), it then calculates the number of shares to buy or sell, and passes it into the function called Submit orders.
The function called Submit orders creates a string with the trade order, and passes it into the yellow Alpaca API node. It also writes the submitted trade order into the PostgreSQL table called tableSubmits.


The flow below liquidates all of the strategy’s stock positions at the end of the day. The flow runs automatically - you don’t need to click on it.

The Bollinger strategy maintains a queue of the strategy’s open positions throughout the day.
The function called Liquidate positions calculates the total open positions in the queue, and it submits a trade order to liquidate them, one minute before the market close. The function can also liquidate the total open positions reported by Alpaca.

The function called Liquidate positions creates a string with the trade order, and passes it into the yellow Alpaca API node. It also writes the submitted trade order into the PostgreSQL table called tableSubmits.
Pressing the gray inject node called Liquidate now liquidates all of the strategy’s stock positions at any time during the market trading hours.


The flow below obtains trade confirmations from Alpaca and inserts them into the fill trades table. The flow runs automatically - you don’t need to click on it.

The function called Confirm trades parses the trade confirmations from Alpaca and inserts them into the table tableFills.
It writes the time stamp, the stock symbol, the trade type (buy or sell), the number of shares filled, the fill price, etc.
This allows the user to track the executed trades and troubleshoot any issues.


The flows below execute trades manually.

The function called Buy 1 share creates a string with the trade order to buy one share of the stock, and passes it into the yellow Alpaca API node. It also writes the submitted trade order into the PostgreSQL table called tableSubmits.
Similarly the function called Sell 1 share.
These functions are activated by pressing the grey inject nodes to their left.

The flow below erases the trade tables overnight. The flow runs automatically - you don’t need to click on it.

The Bollinger strategy maintains two PostgreSQL tables, one for the submitted trade orders and one for the executed (filled) orders.
The function called Get trade tables returns a vector with the table names, which is then split by the split node.
The function called Erase or Delete trade tables erases (deletes) the table contents, so they start empty at the beginning of the next trading day.


The flows below create web links for downloading the trade tables to CSV files.

The two flows below create web links for downloading the trade tables to CSV files. You don’t need to click on them.
To download the CSV files to their computer, the user needs to run this shell script.

]]>Bollinger Strategy Theory2024-06-30T00:00:00+00:002024-06-30T00:00:00+00:00https://algoquant.github.io//2024/06/30/Bollinger-Strategy-Theory The Bollinger Strategy

The Bollinger Strategy

The Bollinger Strategy is designed to capture the reversion of prices to their equilibrium value, and it buys stocks when they’re cheap, and sells them when they’re expensive (rich).

The richness or cheapness of the prices is often determined by their z-scores.
The z-score \(z_t\) is equal to the difference between the current price \(p_t\) minus the equilibrium price \(p_e\), divided by the moving average volatility of the prices \(\sigma_t\): \[{\normalsize z_t = \frac{p_t - p_e}{\sigma_t}}\]

The equilibrium price \(p_e\) is often chosen to be the moving average price \(p_e = \bar{p}_t\).
You can read more about the moving average prices and volatilities below, and also in this document.

As an alternative, the equilibrium price can be chosen to be the last trade price \(p_f\) - the price at which the last trade was filled (executed). The advantage of this is that it prevents a subsequent trade at a loss. For example, a subsequent buy at a higher price than the previous sell, etc.

The strategy determines if the stock is cheap or expensive depending on the value of the z-score compared to the threshold level \(\theta\).

Below is a plot of the z-scores for intraday 1-minute SPY stock prices for May 31st, 2024.
The z-score equilibrium price \(p_e\) was chosen to be the moving average price \(p_e = \bar{p}_t\), with \(\lambda = 0.9\). See below for more details on the moving average prices.
In this case, the z-scores oscillate around zero, and they can be positive or negative.
The horizontal dashed lines are for the threshold levels \(\theta = 1\) and \(\theta = -1\).

The strategy buys \(1\) share if the stock is cheap (the z-score is less than minus the threshold level \(z_t < -\theta\)), and it sells \(1\) share if the stock is expensive (the z-score is greater than the threshold level \(z_t > \theta\)).

The strategy has several options for establishing positions depending on the value of the z-score.
It can maintain either a single long or a short position, and switch only when the z-score value crosses the threshold \(\theta\).
Or it can add positions as the z-score value continues increasing above the threshold \(\theta\) (double down). It then accumulates an inventory of positions, either long or short, up to the position limit.

You can read more about the implementation of the Bollinger strategy in this document.


The Bollinger Strategy Parameters

The performance of the Bollinger strategy depends on the values of its two parameters: the lambda decay parameter \(\lambda\), and the z-score threshold level \(\theta\).

The decay parameter \(\lambda\) is used for calculating the moving average prices and volatilities.

If \(\lambda\) is closer to \(1\) then the moving average price has a stronger dependence on the past prices, and it adjusts more slowly to the current prices.
In this case, the difference between the current price and the moving average price can be large.
The range of the z-score values is also large.

If \(\lambda\) is closer to \(0\) then the moving average price adjusts quickly to the current prices, and the difference between the current price and the moving average price is small.
The range of the z-score values is also small.

The parameter \(\theta\) is used to determine if the z-score value indicates that the stock is mispriced (its price is cheap or expensive).

Calibration of the Bollinger Strategy

For the purpose of calibration, the historical prices can be split into in-sample prices (past training set) and out-of-sample prices (future test set).

The Bollinger strategy is simulated on the training set prices, and the values of the \(\lambda\) and \(\theta\) parameters are chosen to maximize its performance. For example, to maximize its cumulative profits (PnLs), or to maximize its Sharpe ratio.
The Bollinger strategy is then simulated on the test set prices, to test the performance of the calibrated model.

For example, we could select the past 20 days of intraday stock prices for the training set data, and find the values of the \(\lambda\) and \(\theta\) parameters which maximize the cumulative PnL for that time interval.
We could then run the calibrated Bollinger strategy on live prices on the next day, to test its out-of-sample performance.

The values of the optimal \(\lambda\) and \(\theta\) parameters depend on the length of the training set data (the training set).
A smaller training set (say only 10 past days) results in a greater variance (variability) of the optimal parameters.
A larger training set (say 30 past days) results in a lower variance of the optimal parameters (thanks to diversification), but at the cost of a larger bias, because the optimal parameters adjust more slowly to the new prices.

This is an example of the bias-variance tradeoff with respect to the size of the training set.
A smaller training set has a smaller bias (it adjusts quickly to the new prices), but a larger variance (the optimal parameters change significantly over time).
A larger training set has a smaller variance, but it also has a larger bias (it adjusts more slowly to the new prices).
To achieve good out-of-sample performance, the optimal parameters should have both a small bias and a small variance.
Achieving the best bias-variance tradeoff is one of the major objectives of machine learning.

Choosing the right model parameters is difficult. It is the responsibility of the investor to make that choice.


The Moving Average Prices

The moving average prices can be calculated for a streaming time series of prices \(p_t\).

The Exponential Moving Average (EMA) price \(\bar{p}_t\) is calculated using a decay parameter \(\lambda\) and the recursive formula:

\[{\normalsize \bar{p}_t = \lambda \bar{p}_{t-1} + (1-\lambda) p_t}\]

The moving average price at time \(t\): \(p_t\) is equal to the decay parameter \(\lambda\) times the moving average price at time \(t-1\): \(p_{t-1}\), plus \((1-\lambda)\) times the price at time \(t-1\): \(\bar{p}_{t-1}\).

The decay parameter \(\lambda\) determines the persistence of data memory, i.e. how quickly the contribution of the past data to the average value fades over time.

If the decay parameter \(\lambda\) is closer to \(1\) then the effect of past data fades slowly over time. If \(\lambda\) is closer to \(0\) then the effect of past data fades quickly over time.

The above recursive formula can be expressed as a series:

\[{\normalsize \bar{p}_t = (1-\lambda) \sum\limits_{j=0}^{\infty} {\lambda}^j p_{t-j} = (1-\lambda) (p_t + \lambda p_{t-1} + \lambda^2 p_{t-2} + \ldots)} \]

The formula is called exponential because the decay parameter \(\lambda\) is raised to the power of \(j\), so the weights \(\lambda^j\) decrease exponentially as \(j\) increases.

The advantage of the EMA price is that it applies a greater weight to more recent prices than to past ones.

The EMA calculation is also faster because there’s less computation, and it doesn’t require maintaining a buffer (queue) of past prices.


The Moving Average Variance of Prices

The moving average variance of prices \(\sigma^2_t\) measures the dispersion of the streaming prices around the moving average price. It can be calculated for a streaming time series of prices \(p_t\).

The Exponential Moving Average (EMA) variance of prices \(\sigma^2_t\) is calculated using the two recursive formulas:

\[{\normalsize \sigma^2_t = \lambda \sigma^2_{t-1} + (1-\lambda) (p_t - \bar{p}_{t-1})^2}\] \[{\normalsize \bar{p}_t = \lambda \bar{p}_{t-1} + (1-\lambda) p_t}\]

The moving average variance at time \(t\): \(\sigma^2_t\) is equal to the decay parameter \(\lambda\) times the moving average variance at time \(t-1\): \(\sigma^2_{t-1}\), plus \((1-\lambda)\) times the squared difference between the current price \(p_t\) minus the moving average price at time \(t-1\): \(\bar{p}_{t-1}\).


Choosing The Decay parameter \(\lambda\)

The moving average prices are a smoothed version of the streaming prices.

The decay parameter \(\lambda\) determines the strength of the data smoothing.

If \(\lambda\) is closer to \(1\) then the moving average prices are smoother, but if it’s closer to \(0\) then the average prices are more variable and they follow the streaming prices more closely.

The effect of the decay parameter \(\lambda\) on the moving average prices is illustrated in the animated plot below of VTI stock prices. When \(\lambda = 0.99\) the average prices are very smooth, but as it decreases, they become more variable and they follow the streaming prices more closely.


The moving average prices are less volatile (smoother), but they also lag behind the streaming prices - they are biased.
Notice for example that the average prices (red line above) reach their minimum much later than the streaming prices (blue line).
And the smoother the average prices are, the more biased they are.
A lower variance is desirable, but it comes at the cost of a larger bias (time lag).
This is an example of the bias-variance tradeoff.
Achieving the best bias-variance tradeoff is one of the major objectives of machine learning.

The decay parameter \(\lambda\) is usually chosen to achieve the best bias-variance tradeoff.

The best tradeoff depends on the application. For example, if the streaming prices are very volatile, then a larger decay parameter \(\lambda\) is desirable, to reduce the volatility.

The optimal value of the decay parameter \(\lambda\) can be determined using a simulation called backtesting (cross-validation).

Backtesting is performed on historical data.
The data is split into in-sample (past training set) and out-of-sample (future test set) data sets.
Forecasting models often require the smoothing of the data.
The smoothing is applied to the training set data, and the forecasts are tested on the out-of-sample data.
The value of \(\lambda\) which minimizes the out-of-sample forecast errors is chosen as the optimal value.

The backtesting procedure risks overfitting to the historical data.
The optimal value of \(\lambda\) may not be the best choice in live data. So users should be cautious when selecting parameters using the backtesting procedure.

]]>Pair Strategy Implementation2024-06-30T00:00:00+00:002024-06-30T00:00:00+00:00https://algoquant.github.io//2024/06/30/Pair-Strategy-Implementation Pair Strategy Implementation

Configuration of the Pair Strategy in MachineTrader

The Pair strategy trades a portfolio of two stocks during the market trading hours, using intraday stock prices. Each day it starts trading at the market open with no position and a clean slate (reset to its base state), and it stops trading at the market close. It liquidates all its positions just before the market close. Its state is reset overnight to its base state.

You can read more about the theory of the Pair strategy in this document.

To implement the Pair strategy in MachineTrader, you need to install the tab named Pair Strategy in your MachineTrader instance.
You can download the file named Pair_Strategy.json from the MachineTrader Community share drive by clicking on this link.
This file contains the JavaScript code for the tab named Pair Strategy. You can then import this JavaScript file into your MachineTrader instance, and it will create the tab named Pair Strategy.


Implementation of the Pair Strategy

The tab named Pair Strategy executes the Pair strategy in real time.

Let’s look at the different flows in the tab named Pair Strategy.

At the top of the tab, there are flows for downloading the documentation. You can click on the gray inject nodes to download the documentation. The second flow downloads the current file that you’re reading now.


The flow below creates the flow variables in the tab Pair Strategy.
Flow variables are variables that are shared between the different nodes in the same tab. Once the variables are created, they can be used in other function nodes in the same tab.
You need to run (click) this flow once before the other flows are run. You don’t need to run this flow every day.

The function called Create flow variables creates the flow variables, such as the stock symbol to trade, the threshold level for the z-score, the position limit (the maximum number of shares the strategy can own at any time), etc.


The flow below creates the PostgreSQL tables.

The Pair strategy maintains four PostgreSQL tables for the trade orders. Two tables for the submitted trade orders (one for each stock), and two for the executed (filled) orders.

The functions called Create submits tables and Create fills tables create empty PostgreSQL tables (if they don’t already exist), for storing the submitted orders and for the executed (filled) orders. You need to run (click) these flows once before the other flows are run. You don’t need to run these flows every day.

The two tables called submitsPair* are filled with the orders submitted to the broker API, and the two tables called fillsPair* are filled with the orders that were confirmed by the broker.
These four tables are erased overnight, so they start empty at the beginning of each trading day.


The flow below initializes the flow variables just before the market open. The flow runs automatically - you don’t need to click on it.

The flow initializes the flow variables such as the open position queue, etc., each day just before the market open.
The function node called Initialize flow variables resets the flow variables to their initial values.


The flow below initializes the prices just after the market open. This flow runs automatically - you don’t need to click on it.

The flow initializes the flow variables such as the previous valid price, the EMA price, the price variance, etc.


The flow below updates the technical indicators with live (recent) prices. The flow runs automatically - you don’t need to click on it.

The first node in this flow is an inject node that runs every \(10\) seconds. That’s the frequency at which the strategy is run (updated). It can be changed by the user as needed.
The inject node activates the whole flow, which updates the technical indicators with the live (current) prices.

The first function in this flow is called Get stock price.
It extracts the live (current) prices of the stock pair from the global array variable containing the last prices of many stocks. This array is updated every second in the tab called Alpaca Prices.

The function called Calculate EMA price and volatility obtains the stock price from the function Get stock price through a connector, and it calculates (updates) the EMA moving average price and volatility.
It also scrubs the prices from bad data using the dollar stock return (equal to the new price minus previous price). If the stock return is above the threshold, then the new price is ignored, and the previous valid price is used.

The function called Update tech indicators calculates the z-scores, using the average prices and volatilities from the function Calculate EMA price and volatility through a connector.


The flow below runs the Pair strategy. The flow runs automatically - you don’t need to click on it.

The first function in this flow called Calculate PnL calculates the unrealized profits and losses (PnLs) of the open positions, and also the PnL drawdown from its maximum.
The Pair strategy maintains a queue of the strategy’s open positions throughout the day.
The function Calculate PnL matches the open positions in the queue with the current stock prices, and calculates the unrealized PnL.

The function called Pair strategy executes the Pair strategy, using the z-scores from the function called Update tech indicators in the flow above it.
The function Pair strategy compares the z-scores with the threshold level \(\theta\), and calculates the number of pairs to buy or sell, and passes it into the function called Submit orders.
The function called Submit orders creates strings with the trade orders, and passes them into the yellow Alpaca API node. It also writes the submitted trade orders into the PostgreSQL tables called submitsPair*.


The flow below liquidates all of the strategy’s stock positions at the end of the day. The flow runs automatically - you don’t need to click on it.

The Pair strategy maintains a queue of the strategy’s open positions throughout the day.
The function called Liquidate positions calculates the total open positions in the queue, and it submits a trade order to liquidate them, one minute before the market close. The function can also liquidate the total open positions reported by Alpaca.

The function called Liquidate positions creates a string with the trade order, and passes it into the yellow Alpaca API node. It also writes the submitted trade order into the PostgreSQL table called submitsPair*.
Pressing the gray inject node called Liquidate now liquidates all of the strategy’s stock positions at any time during the market trading hours.


The flow below obtains trade confirmations from Alpaca and inserts them into the fill trades table. The flow runs automatically - you don’t need to click on it.

The function called Confirm trades parses the trade confirmations from Alpaca and inserts them into the table fillsPair*.
It writes the time stamp, the stock symbol, the trade type (buy or sell), the number of shares filled, the fill price, etc.
This allows the user to track the executed trades and troubleshoot any issues.


The flows below execute trades manually.

The function called Buy pair creates a string with the trade order to buy one share of the stock, and passes it into the yellow Alpaca API node. It also writes the submitted trade order into the PostgreSQL table called submitsPair*.
Similarly the function called Sell pair.
These functions are activated by pressing the grey inject nodes to their left.

The flow below erases the trade tables overnight. The flow runs automatically - you don’t need to click on it.

The Pair strategy maintains two PostgreSQL tables, one for the submitted trade orders and one for the executed (filled) orders.
The function called Get trade tables returns a vector with the table names, which is then split by the split node.
The function called Erase or Delete trade tables erases (deletes) the table contents, so they start empty at the beginning of the next trading day.


The flows below create web links for downloading the trade tables to CSV files.

The two flows below create web links for downloading the trade tables to CSV files. You don’t need to click on them.
To download the CSV files to their computer, the user needs to run this shell script.

]]>Pair Strategy Theory2024-06-30T00:00:00+00:002024-06-30T00:00:00+00:00https://algoquant.github.io//2024/06/30/Pair-Strategy-Theory The Pair Strategy

The Pair Portfolio

The Pair strategy trades a portfolio of two stocks as if they were a single stock.
When the strategy buys the pair, it buys \(w_1\) shares of stock1 and \(w_2\) shares of stock2, and vice versa.

Typically, the weight \(w_1\) is positive and the weight \(w_2\) is negative. So if the strategy is long one pair, it’s long \(w_1\) shares of stock1 and short \(w_2\) shares of stock2, and vice versa.

The price of the stock pair is equal to the sum of the prices of the two stocks: \(p = w_1 p_1 + w_2 p_2\).

The stock pair and their weights \(w_1\) and \(w_2\) are often chosen so that the portfolio price is range-bound (stationary) around an equilibrium price.

Below is a plot of the prices of the pair portfolio consisting of 1 share of XLK long (\(w_1 = 1\)) vs \(0.6\) shares of QQQ short (\(w_2 = -0.6\)), for May 31st, 2024.
The stock pair XLK and QQQ was selected because they are ETFs, and their portfolios have a significant overlap among technology and growth stocks, so their prices are highly correlated.

The Z-scores of the Pair

The strategy determines if the portfolio is expensive (rich) or cheap depending on the value of the z-score compared to the threshold level \(\theta\).

The z-score \(z_t\) is equal to the difference between the current portfolio price \(p_t\) minus the equilibrium price \(p_e\), divided by the moving average volatility of the prices \(\sigma_t\): \[{\normalsize z_t = \frac{p_t - p_e}{\sigma_t}}\]

The equilibrium price \(p_e\) is often chosen to be the moving average price \(p_e = \bar{p}_t\).
You can read more about the moving average prices and volatilities below, and also in this document.

As an alternative, the equilibrium price can be chosen to be the last trade price \(p_f\) - the price at which the last trade was filled (executed). The advantage of this is that it prevents a subsequent trade at a loss. For example, a subsequent buy at a higher price than the previous sell, etc.

Below is a plot of the z-scores for intraday 1-minute XLK/QQQ pair prices for May 31st, 2024.
The z-score equilibrium price \(p_e\) was chosen to be the moving average price \(p_e = \bar{p}_t\), with \(\lambda = 0.9\). See below for more details on the moving average prices.
In this case, the z-scores oscillate around zero, and they can be positive or negative.
The horizontal dashed lines are for the threshold levels \(\theta = 1\) and \(\theta = -1\).

The Pair Strategy

The Pair strategy is designed to capture the reversion of the pair portfolio price to its equilibrium value, and it buys the pair when it’s cheap, and sells it when it’s expensive (rich).

The strategy buys \(1\) pair if it’s cheap (the z-score is less than minus the threshold level \(z_t < -\theta\)), and it sells \(1\) pair if it’s expensive (the z-score is greater than the threshold level \(z_t > \theta\)).

The strategy has several options for establishing positions depending on the value of the z-score.
It can maintain either a single long or a short position, and switch only when the z-score value crosses the threshold \(\theta\).
Or it can add positions as the z-score value continues increasing above the threshold \(\theta\) (double down). It then accumulates an inventory of positions, either long or short, up to the position limit.

You can read more about the implementation of the Pair strategy in MachineTrader in this document.

The Pair Strategy Performance

Below is the simulated performance of the Pair strategy for a single day, May 31st, 2024, using the above z-scores.
The Pair strategy maintains a single long or short position, and switches when the z-score value crosses the threshold \(\theta\).
It assumes the transaction cost of \(\$0.05\) for trading a pair. The strategy executes \(61\) trades during this day.
The Pair strategy achieves a cumulative profit of almost \(\$2.5\), while the static pair is almost flat.
The simulated performance is hypothetical, and assumes that the optimal values of the portfolio weights and the threshold level \(\theta\) are known. In reality, the live performance will often be much lower than this example.

You can read more about the implementation of the Pair strategy in this document.


The Pair Strategy Parameters

The performance of the Pair strategy depends on the choice of the stock pair, and on the values of its two parameters: the lambda decay parameter \(\lambda\), and the z-score threshold level \(\theta\).

The stock pairs are selected so that the price of the portfolio is range-bound (stationary) around an equilibrium price. This is often achieved by selecting ETFs with related or overlapping portfolios. Or by selecting stocks from the same industry or sector.

The decay parameter \(\lambda\) is used for calculating the moving average prices and volatilities.

If \(\lambda\) is closer to \(1\) then the moving average price has a stronger dependence on the past prices, and it adjusts more slowly to the current prices.
In this case, the difference between the current price and the moving average price can be large.
The range of the z-score values is also large.

If \(\lambda\) is closer to \(0\) then the moving average price adjusts quickly to the current prices, and the difference between the current price and the moving average price is small.
The range of the z-score values is also small.

The parameter \(\theta\) is used to determine if the z-score value indicates that the stock is mispriced (its price is cheap or expensive).

Calibration of the Pair Strategy

For the purpose of calibration, the historical prices can be split into in-sample prices (past training set) and out-of-sample prices (future test set).

The Pair strategy is simulated on the training set prices, and the values of the \(\lambda\) and \(\theta\) parameters are chosen to maximize its performance. For example, to maximize its cumulative profits (PnLs), or to maximize its Sharpe ratio.
The Pair strategy is then simulated on the test set prices, to test the performance of the calibrated model.

For example, we could select the past 20 days of intraday stock prices for the training set data, and find the values of the \(\lambda\) and \(\theta\) parameters which maximize the cumulative PnL for that time interval.
We could then run the calibrated Pair strategy on live prices on the next day, to test its out-of-sample performance.

The values of the optimal \(\lambda\) and \(\theta\) parameters depend on the length of the training set data (the training set).
A smaller training set (say only 10 past days) results in a greater variance (variability) of the optimal parameters.
A larger training set (say 30 past days) results in a lower variance of the optimal parameters (thanks to diversification), but at the cost of a larger bias, because the optimal parameters adjust more slowly to the new prices.

This is an example of the bias-variance tradeoff with respect to the size of the training set.
A smaller training set has a smaller bias (it adjusts quickly to the new prices), but a larger variance (the optimal parameters change significantly over time).
A larger training set has a smaller variance, but it also has a larger bias (it adjusts more slowly to the new prices).
To achieve good out-of-sample performance, the optimal parameters should have both a small bias and a small variance.
Achieving the best bias-variance tradeoff is one of the major objectives of machine learning.

Choosing the right model parameters is difficult. It is the responsibility of the investor to make that choice.


The Moving Average Prices

The moving average prices can be calculated for a streaming time series of prices \(p_t\).

The Exponential Moving Average (EMA) price \(\bar{p}_t\) is calculated using a decay parameter \(\lambda\) and the recursive formula:

\[{\normalsize \bar{p}_t = \lambda \bar{p}_{t-1} + (1-\lambda) p_t}\]

The moving average price at time \(t\): \(p_t\) is equal to the decay parameter \(\lambda\) times the moving average price at time \(t-1\): \(p_{t-1}\), plus \((1-\lambda)\) times the price at time \(t-1\): \(\bar{p}_{t-1}\).

The decay parameter \(\lambda\) determines the persistence of data memory, i.e. how quickly the contribution of the past data to the average value fades over time.

If the decay parameter \(\lambda\) is closer to \(1\) then the effect of past data fades slowly over time. If \(\lambda\) is closer to \(0\) then the effect of past data fades quickly over time.

The above recursive formula can be expressed as a series:

\[{\normalsize \bar{p}_t = (1-\lambda) \sum\limits_{j=0}^{\infty} {\lambda}^j p_{t-j} = (1-\lambda) (p_t + \lambda p_{t-1} + \lambda^2 p_{t-2} + \ldots)} \]

The formula is called exponential because the decay parameter \(\lambda\) is raised to the power of \(j\), so the weights \(\lambda^j\) decrease exponentially as \(j\) increases.

The advantage of the EMA price is that it applies a greater weight to more recent prices than to past ones.

The EMA calculation is also faster because there’s less computation, and it doesn’t require maintaining a buffer (queue) of past prices.


The Moving Average Variance of Prices

The moving average variance of prices \(\sigma^2_t\) measures the dispersion of the streaming prices around the moving average price. It can be calculated for a streaming time series of prices \(p_t\).

The Exponential Moving Average (EMA) variance of prices \(\sigma^2_t\) is calculated using the two recursive formulas:

\[{\normalsize \sigma^2_t = \lambda \sigma^2_{t-1} + (1-\lambda) (p_t - \bar{p}_{t-1})^2}\] \[{\normalsize \bar{p}_t = \lambda \bar{p}_{t-1} + (1-\lambda) p_t}\]

The moving average variance at time \(t\): \(\sigma^2_t\) is equal to the decay parameter \(\lambda\) times the moving average variance at time \(t-1\): \(\sigma^2_{t-1}\), plus \((1-\lambda)\) times the squared difference between the current price \(p_t\) minus the moving average price at time \(t-1\): \(\bar{p}_{t-1}\).


Choosing The Decay parameter \(\lambda\)

The moving average prices are a smoothed version of the streaming prices.

The decay parameter \(\lambda\) determines the strength of the data smoothing.

If \(\lambda\) is closer to \(1\) then the moving average prices are smoother, but if it’s closer to \(0\) then the average prices are more variable and they follow the streaming prices more closely.

The effect of the decay parameter \(\lambda\) on the moving average prices is illustrated in the animated plot below of VTI stock prices. When \(\lambda = 0.99\) the average prices are very smooth, but as it decreases, they become more variable and they follow the streaming prices more closely.


The moving average prices are less volatile (smoother), but they also lag behind the streaming prices - they are biased.
Notice for example that the average prices (red line above) reach their minimum much later than the streaming prices (blue line).
And the smoother the average prices are, the more biased they are.
A lower variance is desirable, but it comes at the cost of a larger bias (time lag).
This is an example of the bias-variance tradeoff.
Achieving the best bias-variance tradeoff is one of the major objectives of machine learning.

The decay parameter \(\lambda\) is usually chosen to achieve the best bias-variance tradeoff.

The best tradeoff depends on the application. For example, if the streaming prices are very volatile, then a larger decay parameter \(\lambda\) is desirable, to reduce the volatility.

The optimal value of the decay parameter \(\lambda\) can be determined using a simulation called backtesting (cross-validation).

Backtesting is performed on historical data.
The data is split into in-sample (past training set) and out-of-sample (future test set) data sets.
Forecasting models often require the smoothing of the data.
The smoothing is applied to the training set data, and the forecasts are tested on the out-of-sample data.
The value of \(\lambda\) which minimizes the out-of-sample forecast errors is chosen as the optimal value.

The backtesting procedure risks overfitting to the historical data.
The optimal value of \(\lambda\) may not be the best choice in live data. So users should be cautious when selecting parameters using the backtesting procedure.

]]>Moving Averages Implementation2024-06-17T00:00:00+00:002024-06-17T00:00:00+00:00https://algoquant.github.io//2024/06/17/Moving-Averages-Implementation Implementation of the Moving Average Prices And Volatilities

Implementation of the Moving Average Prices and Volatilities

The Node-RED tab named Tech Indicators contains the implementations of various technical indicators in MachineTrader, including the moving average prices and volatilities.

The tab named Tech Indicators calculates the moving average prices and volatilities from live streaming stock prices for a single selected stock.

You can download the tab named Tech Indicators from the MachineTrader-Community repository on GitHub by clicking on this link.

After you download the JavaScript file named Tech_Indicators.json to your computer, you should then install it in your MachineTrader instance, to create the tab named Tech Indicators.

The tab Tech Indicators receives the streaming stock prices from Alpaca, from the tab named Alpaca Prices, which is already included in your MachineTrader instance.
The tab Tech Indicators also relies on the global variables produced by the tab named Globals/Utilities.

After you install the tab Tech Indicators, you must connect the link-in node to the left of the function node named Get stock price, to the link-out node called Prices out in the tab Alpaca Prices.

You can read more about the moving average prices and volatilities in this document.

You can also watch an instructional video about Calculating Moving Average Prices And Volatilities on the MachineTrader YouTube channel.


Let’s take a look at the tab named Tech Indicators.

The function node named Initialize the parameters creates flow variables needed for the calculations. It should be run once before the other nodes are run, by pressing the inject node on the left.

The multiple stock prices stream in from the tab Alpaca Prices into the node named Get stock price, through the small link-in node to its left. The function node Get stock price selects the prices only for the ticker in the tab variable symboln. The single stock prices then stream out through the small link-out node to its right, to the nodes which calculate the moving averages.

The dashed lines are the connectors for passing the live streaming stock prices to the function nodes.
alt text


The function node named Initialize the parameters creates the flow variables needed for the calculations:
alt text

It should be run once before the other nodes are run, by pressing the inject node on the left. The inject node is on a timer, to run on weekdays at 9:30 AM, so that all the flow variables are initialized before the start of stock trading.

The function node named Initialize the parameters creates the following flow variables:

  • symboln = the ticker string of the selected stock.
  • lambdaf = the lambda decay factor which multiplies past estimates.
  • volf = the volatility floor.
  • pricec = the current stock price.
  • pricep = the previous stock price, initially set to NaN.
  • pricema = the moving average price, initially set to NaN.
  • lb = the look-back window, i.e. the number of elements in the data queue.
  • priceq = the data queue (buffer) for storing the recent stock prices.
  • endpq = the position of the end of the price queue.
  • retq = the data queue (buffer) for storing the recent stock returns.
  • endrq = the position of the end of the returns queue.


The data queue is used for calculating the simple moving averages of prices and volatilities.

The data queue is an array (vector) for storing the recent stock prices or returns. The data queue only stores the most recent number of elements equal to lb.
The data queue is a first in last out queue. Data is added one element at a time and it’s stored in the order it’s received. New data is written to the end of the queue, and the endq variable is shifted to the next to last element.
The element P14 at the end of the queue is replaced with new data and becomes the element P1 at the front of the queue.
The element P13 becomes the end of the queue P14. The data elements in the queue are just relabeled - not copied. This avoids copying the other data elements in the queue, which speeds up the code.

Below is an illustration of how an element is added to the data queue:

alt text
alt text


The function node named Get stock price obtains the streaming prices for the ticker symboln from the tab named Alpaca Prices.

alt text
alt text

The prices for all the stocks stream from the tab Alpaca Prices into the small link-in node on the left.
The function node Get stock price then extracts the prices only for the ticker symboln, and it passes them to the small link-out node on the right. The streaming stock prices from the link-out node can then be passed to the other nodes and strategy tabs.

For the live prices to stream, it’s necessary to download and install the tabs named Alpaca Prices and Globals. Then you must connect the link-in node to the left of the function node named Get stock price to the link-out node called Prices out in the tab Alpaca Prices.


The function node named Calculate the SMA prices calculates the simple moving average prices.

alt text
alt text

The stock prices flow from the node Get stock price into the small link-in node on the left. The function node Calculate the SMA prices then calculates the simple moving average prices, and it passes them to the small link-out node on the right. The streaming SMA prices from the link-out node can then be passed to the other nodes and strategy tabs.


The function node named Calculate the EMA prices calculates the exponential moving average prices.

alt text
alt text

The stock prices flow from the node Get stock price into the small link-in node on the left. The function node Calculate the EMA prices then calculates the exponential moving average prices, and it passes them to the small link-out node on the right. The streaming EMA prices from the link-out node can then be passed to the other nodes and strategy tabs.


The function node named Calculate the SMA return volatility calculates the simple moving average volatility of the stock returns.

alt text
alt text

The stock prices flow from the node Get stock price into the small link-in node on the left. The function node Calculate the SMA return volatility then calculates the SMA return volatility, and it passes it to the small link-out node on the right. The streaming SMA return volatility from the link-out node can then be passed to the other nodes and strategy tabs.


The function node named Calculate the EMA return volatility calculates the exponential moving average volatility of the stock returns.

alt text
alt text

The stock prices flow from the node Get stock price into the small link-in node on the left. The function node Calculate the EMA return volatility then calculates the EMA return volatility, and it passes it to the small link-out node on the right. The streaming EMA return volatility from the link-out node can then be passed to the other nodes and strategy tabs.


The function node named Calculate the SMA price volatility calculates the simple moving average volatility of the stock prices.

alt text
alt text

The stock prices flow from the node Get stock price into the small link-in node on the left. The function node Calculate the SMA price volatility then calculates the SMA price volatility, and it passes it to the small link-out node on the right. The streaming SMA price volatility from the link-out node can then be passed to the other nodes and strategy tabs.


The function node named Calculate the EMA price volatility calculates the exponential moving average volatility of the stock prices.

alt text
alt text

The stock prices flow from the node Get stock price into the small link-in node on the left. The function node Calculate the EMA price volatility then calculates the EMA price volatility, and it passes it to the small link-out node on the right. The streaming EMA price volatility from the link-out node can then be passed to the other nodes and strategy tabs.


]]>Moving Averages Theory2024-06-17T00:00:00+00:002024-06-17T00:00:00+00:00https://algoquant.github.io//2024/06/17/Moving-Averages-Theory The Moving Average Prices And Volatilities

The Moving Average Prices

The moving average prices can be calculated for a streaming time series of prices \(p_i\).

The Simple Moving Average (SMA) price \(p^{SMA}_i\) is the average of the past prices \(p_{i-j}\) over a look-back window of length \(L\):

\[{\normalsize p^{SMA}_i = \frac{1}{L} \sum\limits_{j=0}^{L-1} p_{i-j}}\] The parameter \(L\) is equal to the length of the look-back window.

The drawback of the SMA is that it applies the same weight to the recent and past prices. Its calculation also requires maintaining a buffer (queue) of past prices, with the length equal to the parameter \(L\).


The Exponential Moving Average (EMA) price \(p^{EMA}_i\) is calculated using a decay factor \(\lambda\) and the recursive formula:

\[{\normalsize p^{EMA}_i = \lambda p^{EMA}_{i-1} + (1-\lambda) p_i}\]

The above recursive formula can be expressed as a series:

\[{\normalsize p^{EMA}_i = (1-\lambda) \sum\limits_{j=0}^{\infty} {\lambda}^j p_{i-j} = (1-\lambda) (p_i + \lambda p_{i-1} + \lambda^2 p_{i-2} + \ldots)} \] The formula is called exponential because the decay factor \(\lambda\) is raised to the power of \(j\), so the weights \(\lambda^j\) decrease exponentially as \(j\) increases.

The advantage of the EMA is that it applies a greater weight to more recent prices than to past ones.

And it doesn’t forget the past prices completely - it only applies a smaller weight to them.
The EMA calculation is also faster because there’s less computation, and it doesn’t require maintaining a buffer (queue) of past prices.

The SMA and EMA averages are not the same, but they are related to each other, and they move in sympathy with each other.

The decay factor \(\lambda\) determines the persistence of data memory, i.e. how quickly the contribution of the past data to the average value fades over time.

If the decay factor \(\lambda\) is closer to \(1\) (one) then the effect of past data fades slowly over time. If \(\lambda\) is closer to \(0\) (zero) then the effect of past data fades quickly over time.

The length of the look-back window also determines the persistence of data memory.

In the Simple Moving Average (SMA), the data which is more than \(L\) time periods in the past is abruptly forgotten. If the window is small, then only very recent data is used in the calculation of the average. If the window is large, then older data is used in the calculation of the average.

The decay factor \(\lambda\) and the look-back window length \(L\) are related to each other.

If the decay factor \(\lambda\) is closer to \(1\) (one) then the EMA adjusts slowly to new prices, corresponding to a larger look-back window length \(L\). If \(\lambda\) is closer to \(0\) (zero) then the EMA adjusts quickly to new prices, corresponding to a smaller look-back window length \(L\).


Choosing The Decay Factor \(\lambda\) And The Look-back window length \(L\)

The moving average prices are smoothed versions of the streaming prices.

The decay factor \(\lambda\) and the look-back window length \(L\) determine the strength of the data smoothing.

If \(\lambda\) is closer to \(1\) (one) then the moving average prices are smoother, but if it’s closer to \(0\) (zero) then the average prices are more variable and they follow the streaming prices more closely.

If the look-back window length \(L\) is large, then more prices are used in the calculation of the average and it’s therefore smoother. If the look-back window length \(L\) is small, then fewer prices are used and the average follows the streaming prices more closely.

The effect of the decay factor \(\lambda\) on the moving average prices is illustrated in the animated plot below of VTI stock prices. When \(\lambda = 0.99\) the average prices are very smooth, and they lag behind the streaming prices. But as \(\lambda\) decreases, the average prices become more variable and they follow the streaming prices more closely.


The moving average prices are less volatile (smoother), but they also lag behind the streaming prices - they are biased.
Notice for example that the average prices (red line above) reach their minimum much later than the streaming prices (blue line).
And the smoother the average prices are, the more biased they are.
A lower variance is desirable, but it comes at the cost of a larger bias (time lag).
This is an example of the bias-variance tradeoff.
Achieving the best bias-variance tradeoff is one of the major objectives of machine learning.

The decay factor \(\lambda\) and the look-back window length \(L\) are usually chosen to achieve the best bias-variance tradeoff.

The best tradeoff depends on the application. For example, if the streaming prices are very volatile, then a larger decay factor \(\lambda\) is desirable (or a larger look-back window length \(L\)), to reduce the volatility.

The optimal value of the decay factor \(\lambda\) can be determined using a simulation called backtesting (cross-validation).

Backtesting is performed on historical data.
The data is split into in-sample and out-of-sample sets.
Forecasting models often require the smoothing of the data.
The smoothing is applied to the in-sample data, and the forecasts are tested on the out-of-sample data.
The value of \(\lambda\) which minimizes the out-of-sample forecast errors is chosen as the optimal value.

The backtesting procedure risks overfitting to the historical data.
The optimal value of \(\lambda\) may not be the best choice in live data. So users should be cautious when selecting parameters using the backtesting procedure.


The Moving Average Variance of Returns

The moving average variance of returns \(\sigma^2_i\) measures the dispersion of the streaming returns around the moving average return. It can be calculated for a streaming time series of prices \(p_i\).

The dollar return \(r_i\) is the difference between the current price \(p_i\) minus the previous price \(p_{i-1}\): \[{\normalsize r_i = p_i - p_{i-1}}\]

The Simple Moving Average (SMA) return \(r^{SMA}_i\) is equal to the average of the past returns:

\[{\normalsize r^{SMA}_i = \frac{1}{L} \sum\limits_{j=0}^{L-1} r_{i-j}}\]

The Simple Moving Average (SMA) variance \(\sigma^{2 SMA}_i\) is then given by the sum:

\[{\normalsize \sigma^{2 SMA}_i = \frac{1}{L-1} \sum\limits_{j=0}^{L-1} (r_{i-j} - r^{SMA}_{i-1})^2}\]

Note that the average return \(r^{SMA}_{i-1}\) is from the previous time step, to better capture the unexpected return in the current time step.


The Exponential Moving Average (EMA) variance \(\sigma^{2 EMA}_i\) is calculated using the two recursive formulas:

\[{\normalsize \sigma^{2 EMA}_i = \lambda \sigma^{2 EMA}_{i-1} + (1-\lambda) (r_i - r^{EMA}_{i-1})^2}\] \[{\normalsize r^{EMA}_i = \lambda r^{EMA}_{i-1} + (1-\lambda) r_i}\]

The SMA and EMA variances are not the same, but they are related to each other, and they move in sympathy with each other.

If the decay factor \(\lambda\) is closer to \(1\) (one) then the EMA variance adjusts slowly to new prices, corresponding to a larger look-back window length \(L\). If \(\lambda\) is closer to \(0\) (zero) then the EMA variance adjusts quickly to new prices, corresponding to a smaller look-back window length \(L\).


The Moving Average Variance of Prices

The moving average variance of prices \(\sigma^2_i\) measures the dispersion of the streaming prices around the moving average price. It can be calculated for a streaming time series of prices \(p_i\).

The Simple Moving Average (SMA) price \(p^{SMA}_i\) is equal to the average of the past prices:

\[{\normalsize p^{SMA}_i = \frac{1}{L} \sum\limits_{j=0}^{L-1} p_{i-j}}\]

The Simple Moving Average (SMA) variance \(\sigma^{2 SMA}_i\) is then given by the sum:

\[{\normalsize \sigma^{2 SMA}_i = \frac{1}{L-1} \sum\limits_{j=0}^{L-1} (p_{i-j} - p^{SMA}_{i-1})^2}\]

Note that the average price \(p^{SMA}_{i-1}\) is from the previous time step, to better capture the unexpected price in the current time step.


The Exponential Moving Average (EMA) variance of prices \(\sigma^{2 EMA}_i\) is calculated using the two recursive formulas:

\[{\normalsize \sigma^{2 EMA}_i = \lambda \sigma^{2 EMA}_{i-1} + (1-\lambda) (p_i - p^{EMA}_{i-1})^2}\] \[{\normalsize p^{EMA}_i = \lambda p^{EMA}_{i-1} + (1-\lambda) p_i}\]

The variance of the returns and of the prices measure different forms of dispersion.

Take for example the extreme case when the returns are constant \(r_i = r\).
Then the prices \(p_i = \sum {r_i}\) increase in a straight line. The variance of returns is zero, but the variance of prices is not zero. So if the prices exhibit a significant trend, then the variance of prices is large, even if the variance of returns is small.

Another example is Brownian motion. The returns are normally distributed, with a constant volatility. But the variance of prices is not constant, and it increases with the factor \(\lambda\), since then prices have more time to drift away from their past average value.

]]>Unit Testing Using Packages testthat and devtools2016-07-09T12:00:00+00:002016-07-09T12:00:00+00:00https://algoquant.github.io//development/testing/2016/07/09/unit-testingR unit testing using packages testthat and devtools

Unit testing in R can be performed using the R packages testthat and devtools. The packages testthat and devtools provide functions for implementing unit testing in R.

The package testthat provides functions for implementing unit tests for other R functions. The unit tests are stored in test files. It’s convenient to create separate test files for each tested R function, and name the test files test-function-name.R. The test files can be executed any time the user needs to verify that her R functions are working properly.

The package devtools provides functions for executing a whole set of unit tests stored in test files. This is especially convenient for testing a whole package of R functions.

You can read more about unit testing of R packages on Hadley Wickham’s website: unit testing of R packages.


Setting up a package for unit testing using devtools

The function use_testthat() from package devtools sets up a package for unit testing (in this example the package rutils stored in "C:/Develop/R/rutils"):

library(devtools)
devtools::use_testthat("C:/Develop/R/rutils")

The function use_testthat() from package devtools performs the following operations:

  • creates the directories ./tests and ./tests/testthat, (the directory ./tests/testthat is designed to store the test files named test*.R),

  • creates a file ./tests/testthat.R for running all the test files,

  • adds a field Suggests: testthat in the ./DESCRIPTION file.

After the package is set up for unit testing, the user can add test files to the directory ./tests/testthat, using the package testthat.


Writing unit tests using package testthat

The package testthat contains functions for implementing unit tests.

The functions expect_*() perform the actual unit tests by comparing the values returned by functions with the values expected by the user. The expect_*() functions use the base functions grep(), identical(), and all.equal() to determine equality.

The function test_that() is used as a wrapper around the functions expect_*(), to be able to handle the errors they produce when an unexpected value is returned.

The function context() defines a set of tests that test related functionality.

The package testthat contains functions for implementing unit tests:

  • expect_match(): tests whether its first argument matches a regular expression provided by the second argument,

  • expect_equal(): tests whether its first argument is equal to the second argument, using the function all.equal(),

  • expect_true() and expect_false(): test whether their first argument is TRUE or FALSE,

  • test_that(): performs one or more tests using calls to functions expect_*(),

  • context(): defines a set of tests that test related functionality,

Unit tests are organized as a set of calls to the functions expect_*(), each wrapped by the function test_that(). Unit tests related to a similar functionality or function can be collected in the same file under a single context().

Below are unit test code examples designed to demonstrate the functions from package testthat. The test code examples are collected under a single context().

  • The function expect_match() tests whether its first argument matches a regular expression provided by the second argument.

    If there is a match, then expect_match() invisibly returns its first argument, and if there is no match, then it produces an error without returning anything. For example, expect_match("character", "char") invisibly returns "character", while expect_match("integer", "char") produces an error without returning anything.

    The function expect_match(), together with functions typeof() and class(), can be used to test for the type or class of an object.

library(testthat)
# define context for demos of function expect_match()
context("demo of expect_match")
# expect_match() returns its first argument invisibly
test_result <- expect_match("character", "char")
# expect_match() produces an error
test_result <- expect_match("integer", "char")
# test for object type returns "character" invisibly
test_result <- expect_match(typeof("a"), "char")
# test for object type produces an error
test_result <- expect_match(typeof(11), "char")
# test for object class returns "matrix" invisibly
test_result <- expect_match(class(matrix(1:9, nr=3)), "matrix")
# test for object class produces an error
test_result <- expect_match(class(1:9), "matrix")


  • The function expect_equal() tests whether its first argument is equal to its second argument, by calling the function all.equal() to determine equality.

    If the arguments are equal, then expect_equal() invisibly returns its first argument, but if they are not equal, then it produces an error without returning anything.

library(testthat)
# define context for demos of function expect_equal()
context("demo of expect_equal")
# expect_equal() returns its first argument invisibly
test_result <- expect_equal(0.3/3, 0.1)
# expect_equal() produces an error
test_result <- expect_equal(0.3/3, 1.0)


  • The function test_that() performs one or more tests using calls to functions expect_*(), and handles any errors they produce when an unexpected value is returned.

    The first argument is a string representing the test name. The second argument is an expression containing one or more calls to functions expect_*().

    The function test_that() is used as a wrapper around the functions expect_*(). test_that() invisibly returns TRUE if its second argument doesn’t produce any errors, and produces an error without returning anything, if any of the functions expect_*() produce an error.

library(testthat)
# define context for demos of function test_that()
context("demo of test_that")
# produces error and handles it using test_that()
test_result <- test_that("first test", stop("whatever"))
# passes the test and invisibly returns TRUE
test_result <- test_that("second test", expect_true(1>0))
test_result <- test_that("third test", {
                           expect_true(1>0) 
                           expect_match(typeof("a"), "char")
                           })


  • We can now apply the above functions from the package testthat to write some unit test code.

    Below is unit test code for the function na_me() from package rutils:

library(testthat)
library(rutils)
# define context for tests of function na_me()
context("test na_me")
# test the type returned by na_me()
test_that("type must be character", {
  expect_match(typeof(na_me(env_etf$VTI)), "character")
})
# test the value returned by na_me()
test_that("value must be VTI", {
  expect_match(na_me(env_etf$VTI), "VTI")
})


  • Below is unit test code for the function end_points() from package rutils:
library(testthat)
library(rutils)
# define context for tests of function end_points()
context("test end_points")
# test the type returned by end_points()
test_that("end_points must be integer", {
  expect_match(typeof(end_points(env_etf$VTI)), "integer")
})
# test the value returned by end_points()
test_that("end_points value", expect_false(last(end_points(env_etf$VTI)) > NROW(env_etf$VTI)))


Applying unit tests using package devtools

Once the directory ./tests/testthat has been populated with test files named test*.R, the user can execute all the tests by calling the function test() from package devtools:

library(devtools)
library(rutils)
devtools::test("C:/Develop/R/rutils")
]]>Interactive Plots In R2016-07-05T00:00:00+00:002016-07-05T00:00:00+00:00https://algoquant.github.io//2016/07/05/Interactive-Plots-in-R Interactive Plots in R

Publishing R Markdown documents containing interactive plots

One of the advantages of writing R Markdown documents is that they can be compiled into HTML documents, which can incorporate interactive plots. This post is an R Markdown document that was compiled into an HTML document, and can be found in algoquant’s GitHub repository here: R interactive plots

Before creating interactive plots, we need to install the package rutils from GitHub:

# install.packages("devtools")
devtools::install_github(repo="algoquant/rutils")
library(rutils)

The rutils package contains a dataset of daily OHLC time series in xts format, for a portfolio of stock symbols. The time series are contained in an environment called env_etf. The data is set up for lazy loading, so it doesn’t require calling data(etf_data) to load it before being able to call it.

Now we can create interactive plots using the time series data from package rutils. Below is an example of an interactive time series plot produced using the dygraphs package. Left-click on the plot and drag your mouse, to select a date range, then double-click to return to the original range.

# load rutils which contains env_etf dataset
suppressMessages(suppressWarnings(library(rutils)))
suppressMessages(suppressWarnings(library(dygraphs)))
x_ts <- env_etf$price_s[, c("VTI", "IEF")]
# plot dygraph with date range selector
dygraph(x_ts, main="VTI and IEF prices") %>%
  dyOptions(colors=c("blue","green")) %>%
  dyRangeSelector()


The dygraphs package in R is an interface to the dygraphs JavaScript charting library. Interactive dygraphs plots require running JavaScript code, which can be embedded in HTML documents, and displayed by web browsers. But pdf documents can’t run JavaScript code, so they can’t display interactive dygraph() plots,

Publishing R Markdown documents with plotly interactive plots

Below is an example of an interactive time series plot produced using the plotly package. Left-click on the plot and drag your mouse, to select a date range, then double-click to return to the original range.

# load rutils which contains env_etf dataset
suppressMessages(suppressWarnings(library(rutils)))
suppressMessages(suppressWarnings(library(plotly)))
# create data frame of time series
data_frame <-
  data.frame(dates=index(env_etf$price_s),
    coredata(env_etf$price_s[, c("VTI", "IEF")]))
# plotly syntax using pipes
data_frame %>% 
  plot_ly(x=dates, y=VTI, fill="tozeroy", name="VTI") %>% 
  add_trace(x=dates, y=IEF, fill="tonexty", name="IEF") %>% 
  layout(title="VTI and IEF prices", 
         xaxis=list(title="Time"),
         yaxis=list(title="Stock Prices"),
         legend=list(x=0.1, y=0.9))
# standard plotly syntax - for reference
# p_lot <- plot_ly(data=data_frame, x=dates, y=VTI, fill="tozeroy", name="VTI")
# p_lot <- add_trace(p=p_lot, x=dates, y=IEF, fill="tonexty", name="IEF")
# p_lot <-   layout(title="VTI and IEF prices", xaxis=list(title="Time"), yaxis=list(title="Stock Prices"), legend=list(x=0.1, y=0.9))
# p_lot
]]>Publishing documents in R2016-07-02T12:00:00+00:002016-07-02T12:00:00+00:00https://algoquant.github.io//r,/markdown/2016/07/02/Publishing-documents-in-RR document publishing for reproducible research

An R document is a file containing:

  • plain text combined with markup code,
  • formulas,
  • and R code chunks.

Publishing an R document means compiling it into an easily readable pdf or HTML file, together with tables and plots produced from executing the R code chunks. The output document should be produced without having to copy and paste objects from different sources.

Compiling R documents into readable documents is part of a reproducible research framework, because it allows for easily reproducing the published documents from its source R file, together with tables and plots produced from executing the R code chunks embedded in the source.

Over the years, two different frameworks for publishing documents have emerged:

  • The first is the older PostScript framework, created by Adobe Systems for desktop publishing and printing. In the PostScript framework documents are published as pdf files, and they can’t be interactive.

  • The second is the newer HTML framework, created for web publishing. In the HTML framework documents are published as HTML files, and they can be interactive.

The publishing of R documents has followed the same pattern, and there are now two different frameworks for publishing R documents:

  • The first is the older Sweave framework for producing pdf files,

  • The second is the newer R Markdown framework for producing HTML and pdf files.

There are consequently two types of R documents: Sweave files and R Markdown files.

Publishing R documents using the Sweave framework

A Sweave document is a file (usually with extension Rnw) containing:

  • plain text combined with LaTeX markup code,
  • formulas written in LaTeX,
  • and R code chunks.

Publishing a Sweave document means compiling it into a pdf file, together with tables and plots produced from executing the R code chunks.

Sweave documents can be compiled into pdf documents using the knitr package. The knitr function knit2pdf() compiles an Rnw file into a pdf file, for example:
knitr::knit2pdf("statistics.Rnw")
Sweave documents can’t be compiled into HTML format, so they can’t incorporate interactive plots.

Many examples of Sweave documents can be found in algoquant’s GitHub repository of lecture slides: algoquant’s lecture slides on GitHub

Publishing R documents using the R Markdown framework

An R Markdown document is a file (with extension Rmd) containing:

  • a YAML header,
  • plain text combined with markdown code,
  • formulas written in LaTeX, nested between either ‘$’ or ‘$$’ symbols,
  • and R code chunks, nested between either single ‘' or quadruple '```’ backtick symbols.

Publishing an R Markdown document means compiling it into a pdf or HTML file, together with tables and plots produced from executing the R code chunks. R Markdown documents are compiled using the markdown package, and can be compiled into HTML documents, so they can incorporate interactive plots. The R Markdown framework is relatively easy to use because it relies on the simple markdown markup language, which is close to plain text and is therefore easy to read.

The function render() from package rmarkdown compiles an Rmd file into either a pdf or HTML file:

rmarkdown::render("R-publishing.Rmd")

The type of file produced by render() depends on the YAML header in the Rmd file. If the YAML header includes:
output: pdf_document then a pdf document is produced.
If the YAML header includes:
output: html_documentthen a HTML document is produced.

This post is an R Markdown document that was compiled into an HTML document, and can be found in algoquant’s GitHub repository here: R publishing

Resources for publishing R documents

Links to books, blogs, and tutorials about publishing R documents using the knitr package:


Links to books, blogs, and tutorials about publishing R documents using the rmarkdown package:


Links to books, blogs, and tutorials about publishing R documents and reproducible research:

]]>Hello World!2016-06-25T12:00:00+00:002016-06-25T12:00:00+00:00https://algoquant.github.io//websites/2016/06/25/hello-world!I finally took the time to set up my home page on GitHub

Unfortunately it forced me to learn more about coding in HTML, CSS, YAML, and liquid than I ever cared to know. Fortunately I discovered a decent amount of blogs and tutorials online to be able to set up the website without too much pain.

Selecting Jekyll for generating the website

There are many different website technologies for creating static websites. Since I chose to host my website on GitHub (where I already have a repository for other projects), I decided to explore website technologies used by other GitHub users.
As it happens, GitHub has developed a website generator written in Ruby, called Jekyll, which has become very popular, so I chose to use Jekyll for building my website as well.

Selecting a Jekyll theme

A Jekyll theme is a set of HTML and CSS files that determine the appearance of a website, like fonts, colors, shading, and page layouts. A Jekyll theme can be adapted to create websites with the same appearance as the theme.
I wanted to choose a visually attractive theme, but with additional features, like a sidebar containing icons and text, with links to my website pages and to other websites. I chose to adapt the Jekyll-Architect theme by Pietro Menna, and the Beautiful-Jekyll theme by Dean Attali. Here you can find more: Jekyll themes.
I also added Disqus comments to my blog posts.

You’re welcome to adapt my website for your own needs

You can download the website files using the buttons on the right sidebar.
You can read about how to set up this website in the README.md file. But be aware that this website isn’t a static website theme, but rather it’s a complete website, and its contents change over time. Please follow the same copyright rules as those for the original website themes that it’s based on.

Building websites using R Markdown and knitr

I use R for all my work, so in the future I may also try building my websites using R Markdown and knitr. Here are some references about building websites using R Markdown and knitr:

Building websites using R Markdown and knitr
Hacking htmlwidgets into website using rmarkdown
Hacking htmlwidgets into website using Jekyll

]]>