Estimator Help Docs
Estimator > Adjusting Data > Historic Data Adjustment

Historic Data Adjustment

Estimator Help Documentation

The historic data adjustment options provide a means to condition the risk and return estimations derived from the historic return series. The numerous statistical methods that can be applied are described below.

Return Estimation

Choose the appropriate option for estimating the historical returns from the Return Estimation drop down menu in the ribbon.

  • Choosing None makes no adjustment to the historic observations. Unless there is missing data, the Estimator simply calculates the sample means from the given return series. This is a traditional choice for estimating returns from historical data.

  • The Stein Classic option is a compromise between the asset means and their equally-weighted global mean. The sample means are "shrunk" towards the global mean (mean of means), which has the effect of reducing the total expected mean-square error. This shrinkage can be rationalized by the fact extreme performers are unlikely to repeat their extreme performance.  he total number of assets, the standard deviation of the asset, and the variability among asset means determine the amount that each asset's estimate is shrunk toward the mean. Noisy assets are shrunk the most, and extremely noisy assets may be shrunk all the way to the global mean. A noisy or volatile set of assets may have all the same result mean value because the Stein classic method is very aggressive about shrinking. Because of this possibility, this option should be used cautiously with noisy data.

  • The Stein-Michaud option is another variant of Stein estimation. In this case, the shrinkage target is the estimated Capital Asset Pricing Model (CAPM) expected return, calculated by multiplying the asset’s beta estimate by the return of the market portfolio. The amount of shrinkage depends on the total number of assets, the sum of squared deviations from the CAPM prediction, and the variability among the asset predictions. This option may be useful for joint estimation of the expected returns of multiple asset classes, e.g. stocks and bonds.

  • The Implied Returns option calculates the expected returns implied by the optimality of the benchmark portfolio, which appears as the market portfolio. If you are using Implied Returns, New Frontier recommends a cap-weighted portfolio that contains all of the assets. Implied Returns Estimation assumes that the market is in equilibrium, meaning that supply for assets equals the demand for assets, and that the market is represented by the market portfolio. The Estimator calculates the returns implied by these assumptions using unconstrained reverse optimization. The resulting expected returns are proportional to covariance matrix times the benchmark weights. This means that a high benchmark weight implies a high return to account for the large equilibrium holdings of the asset. Also, a higher non-diversifiable risk must also imply a higher return to compensate for that risk. Implied returns resemble fitting the expected returns of all assets to the value that will optimize (maximum Sharpe ratio without constraints) to the portfolio of the capitalization weights.  Implied Returns are used in the Black-Litterman forecasting method to estimate historical expected returns.

  • The Weighted HLM option (Hierarchical Linear Model) is an improvement to James-Stein using modern compute-intensive methods. HLM starts with a similar underlying model, and improves on James-Stein in several ways (detailed below). Since the last step involves Monte Carlo sampling, answers will differ slightly on each run unless a seed has been set (see below). Weighted HLM is a compute-intensive method taking longer than the other methods to calculate, so a progress meter appears when this option is selected and and an audio completion notice is given if your volume is on. Notable differences between this method and Stein Classic include the following:

    • By using market weights instead of equal weights, the estimates produced are more consistent with capital markets.

    • By using modern iterative fitting techniques instead of a simple formula to calculate the shrinkage levels, the assets' estimates never entirely shrink to the global mean. This means that some of the historical signal is retained, although as in James-Stein, riskier assets shrink more.

    • By using a parametric resampling procedure similar to the Optimizer, the estimates are further improved as they incorporate information from the market weights and correlations.

    • HLM Seed--When Weighted HLM is selected, you can set a seed for all pseudorandom numbers in Monte Carlo simulations, which fixes the results for all runs. With no seed, the results vary between runs, subject to the Monte Carlo error introduced by the random resampling. Because of the inevitability of sampling error, using a default seed is only recommended for testing purposes.  

For further information on return estimation, consult these publications:

Stein Estimation:

  • Michaud, R. and M. Carty. 1999. "Forecast from the Past."  Financial Planning. November. Available on New Frontier's website: https://newfrontieradvisors.com/media/1194/forecast-from-the-past-110199.pdf.

  • James, W. and C. Stein.1961. Estimation with Quadratic Loss, Proceeding of the 4th Berkeley Symposium on Probability and Statistics. Berkeley: University of California Press, 361-379.

Implied Returns:

  • Michaud, R., Esch, D., Michaud, R.. 2013. "Deconstructing Black-Litterman." Journal of Investment Management, 2013, first quarter. Working version available at  https://newfrontieradvisors.com/media/1158/deconstructing-black-litterman_2012_08_22.pdf

  • Grinold, R. and R. Kahn. 1994. Active Portfolio Management, Chicago: Irwin

Risk Estimation

Choose the appropriate option for calculating the historical standard deviations from the Risk Estimation drop down menu.

  • Choosing None makes no adjustment to the historic observations. It simply calculates the sample standard deviations and correlations from the given return series. This is the traditional choice for estimating risk from historical data.

  • The Ledoit Single Factor option computes the correlation estimates differently. The sample correlation matrix is shrunk toward the CAPM-predicted residual correlation matrix. This improves the forecast value of risk estimates from historical return value by reducing extreme values. The Ledoit process "estimates the covariance matrix of returns by an optimally weighted average of two existing estimators: the sample covariance matrix and single index covariance matrix." The Ledoit single factor estimation technique has the effect of shrinking the estimated correlations toward zero. This applies even to duplicated assets, which will have an estimated correlation less than 1.0, unlike any of the other options, which will produce correlations of 1.0 for assets with identical return series.

  • The Ledoit Nonlinear option is similar to the Ledoit Single Factor, but shrinks the assets differently from each other. It is likely best suited to larger investment universes as it was designed to be asymptotically correct as the universe size expands i. e. for large-dimensional covariance matrices. This option is new in the AAS version 6.6 and differs in two important ways from the estimator presented in the second reference below:

    • The output correlation matrix is rescaled so that the diagonal elements of the matrix are exactly equal to 1.0

    • Duplicated assets are first removed and reintroduced after the computation process — this will produce off-diagonal elements of 1.0 for any pair of duplicate estimates and identical entries for correlations between any duplicated assets and other assets.

  • The Semi-Covariance option, also known as shortfall risk estimation, assumes that downside volatility is of greater concern than upside for measuring risk. The Estimator calculates the correlations from all returns, but only the negative returns are used to find the standard deviations. Use this option only if there is enough data to support using only the downside returns and there is concern that the downside return distribution is not symmetrical to the upside, i.e. is not a reflection through the mean of the upside distribution.

For further information on risk estimation, consult ther following references:

  • Abstract, Ledoit and Wolf, “Improved Estimation of the Covariance Matrix of Stock Returns with an Application to Portfolio Selection”, November 2001.

  • Ledoit, Olivier and Wolf, Michael, Analytical Nonlinear Shrinkage of Large-Dimensional Covariance Matrices (November 2018). University of Zurich, Department of Economics, Working Paper No. 264, Revised version. Available at SSRN: https://ssrn.com/abstract=3047302 or http://dx.doi.org/10.2139/ssrn.3047302

Series Adjustment

Series adjustment provides options for altering the return series according to constants or a benchmark. In order to use the Benchmark Relative and Benchmark Average Options, you must select a benchmark return series. This is most often used for inflation-adjustment by setting the CPI or something similar (see importing from outside sources). Choose the series adjustment from the Series Adjustment drop down menu.

  • Choosing None results in no adjustment to the return series.

  • With the Benchmark Relative option, each period's benchmark return is subtracted from the corresponding asset returns. The resulting benchmark relative return series are then used to create estimates according to the selected historic adjustment options (above). Note that since the returns themselves change, all of the results will likely differ from unadjusted results, not just the returns. This option is particularly useful for liability-relative return optimization. It can also be used to determine the risk premium if you make the benchmark a risk-free asset.  If you use this option, keep the adjustment in mind when you make your forecasts.

  • The Benchmark Average option adjusts the historical returns according to the benchmark average. The Estimator subtracts the difference between the average of the benchmark return series and the entered Series Adjustment Value from the other return series. For example, if the benchmark return series consists of 8, 10, and 9, the average of the benchmark return series is 9. If you decide that the current average is 4 and enter that value in the Series Adjustment Value Field, the Estimator subtracts 5 from each of the returns in the other return series. So, if one of the return series had been 15, -5, 20, it would become 10, -10, and 15. When this option is enabled, the benchmark return series is often the Consumer Price Index or T-bill history.

  • The Adjust by Fixed Constant option resembles the Benchmark Average option. Here, however, the Estimator ignores the benchmark and simply subtracts the fixed constant (the Series Adjustment Value) from the historic return separately calculated for each asset.  

After making your choices, click on the Run Estimator button. The Historical Worksheet and the Historical Table on the Results Worksheet populate with the results according to the selected options. You can alter the data that results from historical data adjustment as you desire, but New Frontier recommends special training. In particular, if you wish to enter your own correlations, contact New Frontier for guidance. Proceed to the Forecasts Worksheet to input forecasts as desired.  

© 2024 New Frontier Advisors