Statistics

Estimation, error bars, tests and regression, and how to tell a real effect from luck.

1. 01Error bar on a meanEasyAbout 2 min

   A strategy has 400 daily returns with a sample standard deviation of 1.5%. What is the standard error of its mean daily return?

   Hint

   The standard error of a mean is the standard deviation divided by the square root of the sample size.

   Answer

   0.075% per day.

   Solution

   Standard error = 1.5% / √400 = 1.5% / 20 = 0.075%.

   A mean daily return of 0.05% would be smaller than one standard error, so on its own it is no evidence of an edge.

   Mark as done
2. 02Two stocks and one driverEasyAbout 3 min

   Two stocks are driven by the same market factor F and by their own noise: X = 0.8 F + 0.6 e1 and Y = 0.8 F + 0.6 e2, where F, e1 and e2 are independent with variance 1. What is the correlation between X and Y, and does X cause Y in this model?

   Hint

   Compute the covariance of X and Y, and check the variance of each one.

   Answer

   The correlation is 0.64, and there is no causal link between X and Y at all.

   Solution

   Each variance is 0.8^2 + 0.6^2 = 0.64 + 0.36 = 1.

   Only the shared factor contributes to the covariance: Cov(X, Y) = 0.8 × 0.8 × Var(F) = 0.64.

   So the correlation is 0.64 / (1 × 1) = 0.64.

   Neither stock appears in the formula of the other. The common driver alone creates the correlation, so changing X would leave Y untouched.

   Mark as done
3. 03What a significant result meansMediumAbout 5 min

   You test many trading rules at the 5% level. From experience, 10% of the rules you test have a real edge, and your test detects a real edge with probability 50%. A rule comes out significant. What is the probability that it has a real edge?

   Hint

   A p-value below 5% does not mean a 95% chance of a real effect. Count real and false hits in a large group of rules.

   Answer

   The probability is 10/19, about 53%.

   Solution

   Picture 1,000 rules. 100 have a real edge, and the test catches 50 of them.

   The other 900 have none, and 5% of them, which is 45, still look significant.

   So 95 rules are significant, and 50 of them are real: 50/95 = 10/19, about 53%.

   The 5% level controls false alarms among useless rules. It says nothing direct about the share of real effects among the significant ones.

   Mark as done
4. 04Two signals that move togetherMediumAbout 6 min

   You regress returns on two signals whose sample correlation is 0.8. Compared with the case where the two signals are uncorrelated (same sample size, same noise and same variance of each signal), by what factor does the variance of each slope estimate grow? By what factor does its standard error grow?

   Hint

   Ask how much of one signal is left over once you remove the part that the other signal explains. Only that leftover part identifies its slope.

   Answer

   The variance grows by 25/9, about 2.78, and the standard error by 5/3, about 1.67.

   Solution

   The slope of signal 1 is identified only by the part of signal 1 that signal 2 cannot explain. That part has a share 1 - r^2 of its variance.

   So the slope variance grows by the factor 1 / (1 - r^2). Here 1 - 0.8^2 = 0.36, so the factor is 1 / 0.36 = 25/9, about 2.78.

   The standard error is the square root of the variance, so it grows by √(25/9) = 5/3, about 1.67.

   The model can still predict well. It just cannot tell the two signals apart very precisely.

   Mark as done
5. 05Overlapping labelsHardAbout 12 min

   You train a model on daily data, and the label for day t is the sum of the next 20 daily returns. Assume daily returns are independent with equal variance. What is the correlation between the labels of two neighboring days? If you hold out a block of test days, how many training days must you drop on each side of the block so that no training label shares a daily return with any test label?

   Hint

   Count the daily returns that two labels k days apart have in common.

   Answer

   The correlation is 19/20 = 0.95, and you must drop 19 days on each side of the test block.

   Solution

   The labels of day t and day t + k share 20 - k of their daily returns when k is below 20, and none from k = 20 on.

   With independent returns of variance s^2, their covariance is (20 - k) s^2, and each label has variance 20 s^2.

   So the correlation is (20 - k) / 20, which is 19/20 = 0.95 for neighboring days.

   A training label overlaps a test label whenever the two days are fewer than 20 apart. So you drop the 19 training days directly before and after the block.

   Random shuffled folds ignore this overlap, so the test score then partly measures memory of almost identical labels.

   Mark as done
6. 06Which variance estimateHardAbout 15 min

   You have 5 independent observations from a normal distribution with unknown mean and variance σ^2. Let S be the sum of squared deviations from the sample mean. Compare S/4, S/5 and S/6 as estimates of σ^2: which one is unbiased, and which one has the smallest mean squared error?

   Hint

   For normal data, S / σ^2 follows a chi-square distribution with n - 1 = 4 degrees of freedom, with mean 4 and variance 8.

   Answer

   S/4 is unbiased. S/6 has the smallest mean squared error: 1/3 σ^4, against 9/25 σ^4 for S/5 and 1/2 σ^4 for S/4.

   Solution

   S = σ^2 × Q with Q chi-square with 4 degrees of freedom, so E[S] = 4 σ^2 and Var(S) = 8 σ^4.

   For the estimate S/c, the bias is (4/c - 1) σ^2 and the variance is 8/c^2 σ^4. The mean squared error is the variance plus the squared bias.

   c = 4: no bias, variance 8/16 = 1/2, so the error is 1/2 σ^4.

   c = 5: bias -1/5 σ^2, variance 8/25, so the error is 8/25 + 1/25 = 9/25 σ^4.

   c = 6: bias -1/3 σ^2, variance 8/36, so the error is 2/9 + 1/9 = 1/3 σ^4.

   Accepting a small bias buys a larger drop in variance. In general the divisor n + 1 minimizes the mean squared error for normal data.

   Mark as done

Compare your own answer with the worked solution.