Student's t-test
Template:Short description The t-test is any statistical hypothesis test in which the test statistic follows a Student's t-distribution under the null hypothesis.
A t-test is the most commonly applied when the test statistic would follow a normal distribution if the value of a scaling term in the test statistic were known. When the scaling term is unknown and is replaced by an estimate based on the data, the test statistics (under certain conditions) follow a Student's t distribution. The t-test can be used, for example, to determine if the means of two sets of data are significantly different from each other.
History
The term "t-statistic" is abbreviated from "hypothesis test statistic".Template:Citation needed In statistics, the t-distribution was first derived as a posterior distribution in 1876 by Helmert[1][2][3] and Lüroth.[4][5][6] The t-distribution also appeared in a more general form as Pearson Type IV distribution in Karl Pearson's 1895 paper.[7] However, the T-Distribution, also known as Student's T Distribution gets its name from William Sealy Gosset who first published it in English in 1908 in the scientific journal Biometrika using his pseudonym "Student"[8][9] because his employer preferred staff to use pen names when publishing scientific papers instead of their real name, so he used the name "Student" to hide his identity.[10] Gosset worked at the Guinness Brewery in Dublin, Ireland, and was interested in the problems of small samples – for example, the chemical properties of barley with small sample sizes. Hence a second version of the etymology of the term Student is that Guinness did not want their competitors to know that they were using the t-test to determine the quality of raw material. Although it was William Gosset after whom the term "Student" is penned, it was actually through the work of Ronald Fisher that the distribution became well known as "Student's distribution"[11] and "Student's t-test".
Gosset had been hired owing to Claude Guinness's policy of recruiting the best graduates from Oxford and Cambridge to apply biochemistry and statistics to Guinness's industrial processes.[12] Gosset devised the t-test as an economical way to monitor the quality of stout. The t-test work was submitted to and accepted in the journal Biometrika and published in 1908.[13] Company policy at Guinness forbade its chemists from publishing their findings, so Gosset published his statistical work under the pseudonym "Student" (see Student's t-distribution for a detailed history of this pseudonym, which is not to be confused with the literal term student).
Guinness had a policy of allowing technical staff leave for study (so-called "study leave"), which Gosset used during the first two terms of the 1906–1907 academic year in Professor Karl Pearson's Biometric Laboratory at University College London.[14] Gosset's identity was then known to fellow statisticians and to editor-in-chief Karl Pearson.[15]
Uses
Among the most frequently used t-tests are:
- A one-sample location test of whether the mean of a population has a value specified in a null hypothesis.
- A two-sample location test of the null hypothesis such that the means of two populations are equal. All such tests are usually called Student's t-tests, though strictly speaking that name should only be used if the variances of the two populations are also assumed to be equal; the form of the test used when this assumption is dropped is sometimes called Welch's t-test. These tests are often referred to as "unpaired" or "independent samples" t-tests, as they are typically applied when the statistical units underlying the two samples being compared are non-overlapping.[16]
Assumptions
Most test statistics have the form Template:Math, where Template:Math and Template:Math are functions of the data.
Template:Math may be sensitive to the alternative hypothesis (i.e., its magnitude tends to be larger when the alternative hypothesis is true), whereas Template:Math is a scaling parameter that allows the distribution of Template:Math to be determined.
As an example, in the one-sample t-test
- <math>t = \frac{Z} {s} = \frac{\bar{X} - \mu}{\widehat\sigma / \sqrt{n}}</math>
where Template:Math is the sample mean from a sample Template:Math, of size Template:Math, Template:Math is the standard error of the mean, <math display="inline"> \widehat\sigma</math> is the estimate of the standard deviation of the population, and Template:Math is the population mean.
The assumptions underlying a t-test in the simplest form above are that:
- Template:Math follows a normal distribution with mean Template:Math and variance Template:Math
- Template:Math follows a [[chi-squared distribution|Template:Math distribution]] with Template:Math degrees of freedom. This assumption is met when the observations used for estimating Template:Math come from a normal distribution (and i.i.d for each group).
- Template:Math and Template:Math are independent.
In the t-test comparing the means of two independent samples, the following assumptions should be met:
- The means of the two populations being compared should follow normal distributions. Under weak assumptions, this follows in large samples from the central limit theorem, even when the distribution of observations in each group is non-normal.[17]
- If using Student's original definition of the t-test, the two populations being compared should have the same variance (testable using F-test, Levene's test, Bartlett's test, or the Brown–Forsythe test; or assessable graphically using a Q–Q plot). If the sample sizes in the two groups being compared are equal, Student's original t-test is highly robust to the presence of unequal variances.[18] Welch's t-test is insensitive to equality of the variances regardless of whether the sample sizes are similar.
- The data used to carry out the test should either be sampled independently from the two populations being compared or be fully paired. This is in general not testable from the data, but if the data are known to be dependent (e.g. paired by test design), a dependent test has to be applied. For partially paired data, the classical independent t-tests may give invalid results as the test statistic might not follow a t distribution, while the dependent t-test is sub-optimal as it discards the unpaired data.[19]
Most two-sample t-tests are robust to all but large deviations from the assumptions.[20]
For exactness, the t-test and Z-test require normality of the sample means, and the t-test additionally requires that the sample variance follows a scaled [[Chi-squared distribution|χTemplate:Isup distribution]], and that the sample mean and sample variance be statistically independent. Normality of the individual data values is not required if these conditions are met. By the central limit theorem, sample means of moderately large samples are often well-approximated by a normal distribution even if the data are not normally distributed. For non-normal data, the distribution of the sample variance may deviate substantially from a χTemplate:Isup distribution. However, if the sample size is large, Slutsky's theorem implies that the distribution of the sample variance has little effect on the distribution of the test statistic.
Unpaired and paired two-sample t-tests
Two-sample t-tests for a difference in mean involve independent samples (unpaired samples) or paired samples. Paired t-tests are a form of blocking, and have greater power than unpaired tests when the paired units are similar with respect to "noise factors" that are independent of membership in the two groups being compared.[21] In a different context, paired t-tests can be used to reduce the effects of confounding factors in an observational study.
Independent (unpaired) samples
The independent samples t-test is used when two separate sets of independent and identically distributed samples are obtained, one from each of the two populations being compared. For example, suppose we are evaluating the effect of a medical treatment, and we enroll 100 subjects into our study, then randomly assign 50 subjects to the treatment group and 50 subjects to the control group. In this case, we have two independent samples and would use the unpaired form of the t-test.
Paired samples
Paired samples t-tests typically consist of a sample of matched pairs of similar units, or one group of units that has been tested twice (a "repeated measures" t-test).
A typical example of the repeated measures t-test would be where subjects are tested prior to a treatment, say for high blood pressure, and the same subjects are tested again after treatment with a blood-pressure-lowering medication. By comparing the same patient's numbers before and after treatment, we are effectively using each patient as their own control. That way the correct rejection of the null hypothesis (here: of no difference made by the treatment) can become much more likely, with statistical power increasing simply because the random interpatient variation has now been eliminated. However, an increase of statistical power comes at a price: more tests are required, each subject having to be tested twice. Because half of the sample now depends on the other half, the paired version of Student's t-test has only Template:Math degrees of freedom (with Template:Math being the total number of observations). Pairs become individual test units, and the sample has to be doubled to achieve the same number of degrees of freedom. Normally, there are Template:Math degrees of freedom (with Template:Math being the total number of observations).[22]
A paired samples t-test based on a "matched-pairs sample" results from an unpaired sample that is subsequently used to form a paired sample, by using additional variables that were measured along with the variable of interest.[23] The matching is carried out by identifying pairs of values consisting of one observation from each of the two samples, where the pair is similar in terms of other measured variables. This approach is sometimes used in observational studies to reduce or eliminate the effects of confounding factors.
Paired samples t-tests are often referred to as "dependent samples t-tests".
Calculations
Explicit expressions that can be used to carry out various t-tests are given below. In each case, the formula for a test statistic that either exactly follows or closely approximates a t-distribution under the null hypothesis is given. Also, the appropriate degrees of freedom are given in each case. Each of these statistics can be used to carry out either a one-tailed or two-tailed test.
Once the t value and degrees of freedom are determined, a p-value can be found using a table of values from Student's t-distribution. If the calculated p-value is below the threshold chosen for statistical significance (usually the 0.10, the 0.05, or 0.01 level), then the null hypothesis is rejected in favor of the alternative hypothesis.
One-sample t-test
In testing the null hypothesis that the population mean is equal to a specified value Template:Math, one uses the statistic
- <math> t = \frac{\bar{x} - \mu_0}{s/\sqrt{n}} </math>
where <math>\bar x</math> is the sample mean, Template:Math is the sample standard deviation and Template:Math is the sample size. The degrees of freedom used in this test are Template:Math. Although the parent population does not need to be normally distributed, the distribution of the population of sample means <math>\bar x</math> is assumed to be normal.
By the central limit theorem, if the observations are independent and the second moment exists, then <math>t</math> will be approximately normal N(0;1).
Slope of a regression line
Suppose one is fitting the model
- <math> Y = \alpha + \beta x + \varepsilon </math>
where Template:Math is known, Template:Math and Template:Math are unknown, Template:Math is a normally distributed random variable with mean 0 and unknown variance Template:Math, and Template:Math is the outcome of interest. We want to test the null hypothesis that the slope Template:Math is equal to some specified value Template:Math (often taken to be 0, in which case the null hypothesis is that Template:Math and Template:Math are uncorrelated).
Let
- <math>
\begin{align} \widehat\alpha, \widehat\beta & = \text{least-squares estimators}, \\ SE_{\widehat\alpha}, SE_{\widehat\beta} & = \text{the standard errors of least-squares estimators}. \end{align} </math>
Then
- <math>
t_\text{score} = \frac{\widehat\beta - \beta_0}{ SE_{\widehat\beta} }\sim\mathcal{T}_{n-2} </math>
has a t-distribution with Template:Math degrees of freedom if the null hypothesis is true. The standard error of the slope coefficient:
- <math>
SE_{\widehat\beta} = \frac{\sqrt{\dfrac{1}{n - 2}\displaystyle\sum_{i=1}^n \left(y_i - \widehat y_i\right)^2}}{\sqrt{ \displaystyle\sum_{i=1}^n \left(x_i - \bar{x}\right)^2 }} </math>
can be written in terms of the residuals. Let
- <math>
\begin{align} \widehat\varepsilon_i & = y_i - \widehat y_i = y_i - \left(\widehat\alpha + \widehat\beta x_i\right) = \text{residuals} = \text{estimated errors}, \\ \text{SSR} & = \sum_{i=1}^n {\widehat\varepsilon_i}^2 = \text{sum of squares of residuals}. \end{align} </math>
Then Template:Mathscore is given by:
- <math> t_\text{score} = \frac{\left(\widehat\beta - \beta_0\right)\sqrt{n-2}}{ \sqrt{\frac{SSR}{\sum_{i=1}^n \left(x_i - \bar{x}\right)^2}} }. </math>
Another way to determine the Template:Mathscore is:
- <math> t_\text{score} = \frac{r\sqrt{n-2}}{\sqrt{1-r^2}}, </math>
where r is the Pearson correlation coefficient.
The Template:Mathscore, intercept can be determined from the Template:Mathscore, slope:
- <math> t_\text{score,intercept} = \frac{\alpha}{\beta} \frac{t_\text{score,slope}}{\sqrt{s_\text{x}^2+\bar{x}^2}} </math>
where Template:Math is the sample variance.
Independent two-sample t-test
Equal sample sizes and variance
Given two groups (1, 2), this test is only applicable when:
- the two sample sizes (that is, the number Template:Math of participants of each group) are equal;
- it can be assumed that the two distributions have the same variance;
Violations of these assumptions are discussed below.
The Template:Math statistic to test whether the means are different can be calculated as follows:
- <math> t = \frac{\bar{X}_1 - \bar{X}_2}{s_p \sqrt\frac{2}{n}} </math>
where
- <math> s_p = \sqrt{\frac{s_{X_1}^2+s_{X_2}^2}{2}}.</math>
Here Template:Math is the pooled standard deviation for Template:Math and Template:Math and Template:Math are the unbiased estimators of the variances of the two samples. The denominator of Template:Math is the standard error of the difference between two means.
For significance testing, the degrees of freedom for this test is Template:Math where Template:Math is the number of participants in each group.
Equal or unequal sample sizes, similar variances (Template:Sfrac < Template:Sfrac < 2)
This test is used only when it can be assumed that the two distributions have the same variance. (When this assumption is violated, see below.) The previous formulae are a special case of the formulae below, one recovers them when both samples are equal in size: Template:Math.
The Template:Math statistic to test whether the means are different can be calculated as follows:
- <math>t = \frac{\bar {X}_1 - \bar{X}_2}{s_p \cdot \sqrt{\frac{1}{n_1}+\frac{1}{n_2}}}</math>
where
- <math> s_p = \sqrt{\frac{\left(n_1-1\right)s_{X_1}^2+\left(n_2-1\right)s_{X_2}^2}{n_1+n_2-2}}</math>
is an estimator of the pooled standard deviation of the two samples: it is defined in this way so that its square is an unbiased estimator of the common variance whether or not the population means are the same. In these formulae, Template:Math is the number of degrees of freedom for each group, and the total sample size minus two (that is, Template:Math) is the total number of degrees of freedom, which is used in significance testing.
Equal or unequal sample sizes, unequal variances (sX1 > 2sX2 or sX2 > 2sX1)
Template:Main This test, also known as Welch's t-test, is used only when the two population variances are not assumed to be equal (the two sample sizes may or may not be equal) and hence must be estimated separately. The Template:Math statistic to test whether the population means are different is calculated as:
- <math>t = \frac{\bar{X}_1 - \bar{X}_2}{s_{\bar\Delta}}</math>
where
- <math>s_{\bar\Delta} = \sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}.
</math>
Here Template:Math is the unbiased estimator of the variance of each of the two samples with Template:Math = number of participants in group Template:Math (1 or 2). In this case Template:Math is not a pooled variance. For use in significance testing, the distribution of the test statistic is approximated as an ordinary Student's t-distribution with the degrees of freedom calculated using
- <math> \mathrm{d.f.} = \frac{\left(\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}\right)^2}{\frac{\left(s_1^2/n_1\right)^2}{n_1-1} + \frac{\left(s_2^2/n_2\right)^2}{n_2-1}}.
</math>
This is known as the Welch–Satterthwaite equation. The true distribution of the test statistic actually depends (slightly) on the two unknown population variances (see Behrens–Fisher problem).
Dependent t-test for paired samples
This test is used when the samples are dependent; that is, when there is only one sample that has been tested twice (repeated measures) or when there are two samples that have been matched or "paired". This is an example of a paired difference test. The t statistic is calculated as
- <math>t = \frac{\bar{X}_D - \mu_0}{s_D/\sqrt n} </math>
where <math>\bar{X}_D</math> and <math>s_D</math> are the average and standard deviation of the differences between all pairs. The pairs are e.g. either one person's pre-test and post-test scores or between-pairs of persons matched into meaningful groups (for instance drawn from the same family or age group: see table). The constant Template:Math is zero if we want to test whether the average of the difference is significantly different. The degree of freedom used is Template:Math, where Template:Math represents the number of pairs.
Example of repeated measures Number Name Test 1 Test 2 1 Mike 35% 67% 2 Melanie 50% 46% 3 Melissa 90% 86% 4 Mitchell 78% 91%
Example of matched pairs Pair Name Age Test 1 John 35 250 1 Jane 36 340 2 Jimmy 22 460 2 Jessy 21 200
Worked examples
Template:Bad summary Let Template:Math denote a set obtained by drawing a random sample of six measurements:
- <math>A_1=\{30.02,\ 29.99,\ 30.11,\ 29.97,\ 30.01,\ 29.99\}</math>
and let Template:Math denote a second set obtained similarly:
- <math>A_2=\{29.89,\ 29.93,\ 29.72,\ 29.98,\ 30.02,\ 29.98\}</math>
These could be, for example, the weights of screws that were chosen out of a bucket.
We will carry out tests of the null hypothesis that the means of the populations from which the two samples were taken are equal.
The difference between the two sample means, each denoted by Template:Math, which appears in the numerator for all the two-sample testing approaches discussed above, is
- <math>\bar{X}_1 - \bar{X}_2 = 0.095.</math>
The sample standard deviations for the two samples are approximately 0.05 and 0.11, respectively. For such small samples, a test of equality between the two population variances would not be very powerful. Since the sample sizes are equal, the two forms of the two-sample t-test will perform similarly in this example.
Unequal variances
If the approach for unequal variances (discussed above) is followed, the results are
- <math>\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}} \approx 0.04849</math>
and the degrees of freedom
- <math>\text{d.f.} \approx 7.031.</math>
The test statistic is approximately 1.959, which gives a two-tailed test p-value of 0.09077.
Equal variances
If the approach for equal variances (discussed above) is followed, the results are
- <math>s_p \approx 0.08396</math>
and the degrees of freedom
- <math>\text{d.f.} = 10.</math>
The test statistic is approximately equal to 1.959, which gives a two-tailed p-value of 0.07857.
Related statistical tests
Alternatives to the t-test for location problems
The t-test provides an exact test for the equality of the means of two i.i.d. normal populations with unknown, but equal, variances. (Welch's t-test is a nearly exact test for the case where the data are normal but the variances may differ.) For moderately large samples and a one tailed test, the t-test is relatively robust to moderate violations of the normality assumption.[24] In large enough samples, the t-test asymptotically approaches the z-test, and becomes robust even to large deviations from normality.[17]
If the data are substantially non-normal and the sample size is small, the t-test can give misleading results. See Location test for Gaussian scale mixture distributions for some theory related to one particular family of non-normal distributions.
When the normality assumption does not hold, a non-parametric alternative to the t-test may have better statistical power. However, when data are non-normal with differing variances between groups, a t-test may have better type-1 error control than some non-parametric alternatives.[25] Furthermore, non-parametric methods, such as the Mann-Whitney U test discussed below, typically do not test for a difference of means, so should be used carefully if a difference of means is of primary scientific interest.[17] For example, Mann-Whitney U test will keep the type 1 error at the desired level alpha if both groups have the same distribution. It will also have power in detecting an alternative by which group B has the same distribution as A but after some shift by a constant (in which case there would indeed be a difference in the means of the two groups). However, there could be cases where group A and B will have different distributions but with the same means (such as two distributions, one with positive skewness and the other with a negative one, but shifted so to have the same means). In such cases, MW could have more than alpha level power in rejecting the Null hypothesis but attributing the interpretation of difference in means to such a result would be incorrect.
In the presence of an outlier, the t-test is not robust. For example, for two independent samples when the data distributions are asymmetric (that is, the distributions are skewed) or the distributions have large tails, then the Wilcoxon rank-sum test (also known as the Mann–Whitney U test) can have three to four times higher power than the t-test.[24][26][27] The nonparametric counterpart to the paired samples t-test is the Wilcoxon signed-rank test for paired samples. For a discussion on choosing between the t-test and nonparametric alternatives, see Lumley, et al. (2002).[17]
One-way analysis of variance (ANOVA) generalizes the two-sample t-test when the data belong to more than two groups.
A design which includes both paired observations and independent observations
When both paired observations and independent observations are present in the two sample design, assuming data are missing completely at random (MCAR), the paired observations or independent observations may be discarded in order to proceed with the standard tests above. Alternatively making use of all of the available data, assuming normality and MCAR, the generalized partially overlapping samples t-test could be used.[28]
Multivariate testing
Template:Main A generalization of Student's t statistic, called Hotelling's t-squared statistic, allows for the testing of hypotheses on multiple (often correlated) measures within the same sample. For instance, a researcher might submit a number of subjects to a personality test consisting of multiple personality scales (e.g. the Minnesota Multiphasic Personality Inventory). Because measures of this type are usually positively correlated, it is not advisable to conduct separate univariate t-tests to test hypotheses, as these would neglect the covariance among measures and inflate the chance of falsely rejecting at least one hypothesis (Type I error). In this case a single multivariate test is preferable for hypothesis testing. Fisher's Method for combining multiple tests with alpha reduced for positive correlation among tests is one. Another is Hotelling's TTemplate:Isup statistic follows a TTemplate:Isup distribution. However, in practice the distribution is rarely used, since tabulated values for TTemplate:Isup are hard to find. Usually, TTemplate:Isup is converted instead to an F statistic.
For a one-sample multivariate test, the hypothesis is that the mean vector (Template:Math) is equal to a given vector (Template:Math). The test statistic is [[Hotelling's t-squared statistic|Hotelling's tTemplate:Isup]]:
- <math>
t^2=n(\bar{\mathbf x}-{\boldsymbol\mu_0})'{\mathbf S}^{-1}(\bar{\mathbf x}-{\boldsymbol\mu_0}) </math>
where Template:Math is the sample size, Template:Math is the vector of column means and Template:Math is an Template:Math sample covariance matrix.
For a two-sample multivariate test, the hypothesis is that the mean vectors (Template:Math) of two samples are equal. The test statistic is [[Hotelling's two-sample t-squared statistic|Hotelling's two-sample tTemplate:Isup]]:
- <math>t^2 = \frac{n_1 n_2}{n_1+n_2}\left(\bar{\mathbf x}_1-\bar{\mathbf x}_2\right)'{\mathbf S_\text{pooled}}^{-1}\left(\bar{\mathbf x}_1-\bar{\mathbf x}_2\right).</math>
Software implementations
Many spreadsheet programs and statistics packages, such as QtiPlot, LibreOffice Calc, Microsoft Excel, SAS, SPSS, Stata, DAP, gretl, R, Python, PSPP, Matlab and Minitab, include implementations of Student's t-test.
| Language/Program | Function | Notes |
|---|---|---|
| Microsoft Excel pre 2010 | TTEST(array1, array2, tails, type) |
See [1] |
| Microsoft Excel 2010 and later | T.TEST(array1, array2, tails, type) |
See [2] |
| LibreOffice Calc | TTEST(Data1; Data2; Mode; Type) |
See [3] |
| Google Sheets | TTEST(range1, range2, tails, type) |
See [4] |
| Python | scipy.stats.ttest_ind(a, b, equal_var=True) |
See [5] |
| Matlab | ttest(data1, data2) |
See [6] |
| Mathematica | TTest[{data1,data2}] |
See [7] |
| R | t.test(data1, data2, var.equal=TRUE) |
See [8] |
| SAS | PROC TTEST |
See [9] |
| Java | tTest(sample1, sample2) |
See [10] |
| Julia | EqualVarianceTTest(sample1, sample2) |
See [11] |
| Stata | ttest data1 == data2 |
See [12] |
See also
Template:Portal Template:Div col
- Conditional change model
- F-test
- Noncentral t-distribution in power analysis
- Student's t-statistic
- Z-test
- Mann–Whitney U test
- Šidák correction for t-test
- Welch's t-test
- Analysis of variance (ANOVA)
References
Citations
| references-column-width
| references-column-count references-column-count-{{#if:1|{{{1}}}}} }}
| {{#if:
| references-column-width }} }}" style="{{#if:
| {{#iferror: {{#ifexpr: 1 > 1 }}
| Template:Column-width
| Template:Column-count }}
| {{#if:
| Template:Column-width }} }} list-style-type: {{#switch:
| upper-alpha
| upper-roman
| lower-alpha
| lower-greek
| lower-roman = {{{group}}}
| #default = decimal}};">
- ↑ Template:Citation
- ↑ Template:Cite journal
- ↑ Template:Cite journal
- ↑ Template:Cite journal
- ↑ Pfanzagl J, Sheynin O (1996). "Studies in the history of probability and statistics. XLIV. A forerunner of the t-distribution". Biometrika. 83 (4): 891–898. doi:10.1093/biomet/83.4.891. MR 1766040.
- ↑ Template:Cite journal
- ↑ Pearson, K. (1895-01-01). "Contributions to the Mathematical Theory of Evolution. II. Skew Variation in Homogeneous Material". Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences. 186: 343–414 (374). doi:10.1098/rsta.1895.0010. ISSN 1364-503X
- ↑ "Student" William Sealy Gosset (1908). "The probable error of a mean" (PDF). Biometrika. 6 (1): 1–25. doi:10.1093/biomet/6.1.1. hdl:10338.dmlcz/143545. JSTOR 2331554
- ↑ Template:Cite web
- ↑ Wendl MC (2016). "Pseudonymous fame". Science. 351 (6280): 1406. doi:10.1126/science.351.6280.1406. Template:PMID
- ↑ Template:Cite book
- ↑ Template:MacTutor Biography
- ↑ Template:Cite journal
- ↑ Template:Cite journal
- ↑ Template:Cite book
- ↑ Template:Cite book
- ↑ 17.0 17.1 17.2 17.3 Template:Cite journal
- ↑ Template:Cite journal
- ↑ Template:Cite journal
- ↑ Template:Cite book
- ↑ Template:Cite bookTemplate:ISBN missing
- ↑ Template:Cite web
- ↑ Template:Cite journal
- ↑ 24.0 24.1 Template:Cite journal
- ↑ Template:Cite journal
- ↑ Template:Cite journal
- ↑ Template:Cite journal
- ↑ Template:Cite journal