4 Deriving and checking rms errors

4.1 The error analysis: An overview

The error analysis may be described by referring to the stages of the zero-point analysis that are listed in Table 1. The principal steps of the error analysis are as follows.

• At stages 1-3, derive or adopt interim errors.
• Between stages 3 and 4, test those errors by comparing data bases.
• At stages 4-6, use further comparisons between data bases to derive an rms error for each added data base.

The quantity determined in this error analysis is the rms error per datum. The generic symbol $\sigma_0$ will be used here to refer to that quantity. Given values of $\sigma_0$ and numbers of contributing data, standard deviations of mean values may be calculated and included in the final catalog.

The scatter that appears when data sets are compared is regarded here as the prime source of information about $\sigma_0$. No use is made of tacit errors that are too small to explain such scatter (see Sect. 7.7 of Paper II). In addition, no assumption is made that $\sigma_0$ is related to high-dispersion analysis procedures. The reason for not making such an assumption will be given in Sect. 4.6.

4.2 The error analysis: Stages 1-3

At stage 1, rms errors of means for the standard stars are carried over from Table 5 of Paper II. At stage 2, it is found that there are a number of zero-point stars with more than one datum. From the scatter in the data for those stars, a value of $\sigma_0$ is calculated (see Appendix A). The resulting value of $\sigma_0$ is $0.103 \pm 0.009$ dex.

At stage 3, results added to the data base include photometric data and the McWilliam (1990) data (see Table 1). McWilliam quotes rms errors for each of his results. Those errors are adopted on an interim basis. For the photometric data, it is assumed that the observed scatter around Eq. (2) is contributed equally by the results of Gustafsson et al. (1974) and the rescaled results of Williams (1971, 1972). The resulting value of $\sigma_0$ is $0.097 \pm 0.009$ dex.

4.3 Testing the errors: The procedure

Note that there are now three sets of interim errors to be tested. One set is the value of $\sigma_0$ for the zero-point data. The second set of errors is for the McWilliam data, while the third set is the value of $\sigma_0$ for the photometric data. To test those errors, a data-comparison algorithm is used. The algorithm is derived mathematically in Eqs. (B1) through (B44) of Appendix B of T91, so the description of the algorithm given here will be limited to a conceptual summary.

Let two sets of N data each (say, set A and set B) be considered. Suppose that rms errors are known for set A, but not for set B. Let an estimate be made for the set B error, and let the N differences between data sets A and B be calculated. If accidental error affects both data sets, the N differences will scatter around some mean. Suppose that one can account for that scatter by combining the known errors for data set A and the estimated error for data set B. One can then conclude that the estimate for the latter error is correct.

This procedure can also be applied to the N residuals from some general relation between data sets A and B. Since the amount of scatter depends in some degree on the adopted relation, it is good procedure to determine the relation before settling on a final value for the set B error. In practice, trial versions of Eq. (1) and its counterparts are calculated. If S differs significantly from zero (recall Sect. 3.3), the scatter around the equation is used to obtain the set B error. If S does not differ significantly from zero, the error algorithm is applied while a simple zero-point offset between data sets A and B is derived.

4.4 Testing the errors: Results

Let the value of $\sigma_0$ derived for data set B be referred to as an "external'' error. This name is applied because the error is derived, in part, from data which are external to data set B. "Internal'' errors have been obtained for each data set during stages 1-3. The task now at hand is to compare internal and external values of $\sigma_0$. A selection of results from those comparisons is given in Table 2.

The first three lines of Table 2 imply, in brief, that if one compares McWilliam (1990) and zero-point data and then compares photometric and McWilliam data, the results are satisfactory. Note that this is true, in particular, if McWilliam data with small internal errors are used. To achieve closure, it is also necessary to compare photometric and zero-point data. Given the interim error for the zero-point data, one hopes to recover the internal error for the photometric data. What actually happens, however, is that a difference of $2.5\sigma$ appears between the internal and external photometric errors. (See the fourth line of Table 2, especially the entry set off by asterisks).

To evaluate this difference, a test is made in which the photometric data are compared to weighted means of the McWilliam and zero-point data. The weights are values of $\sigma_0^{-2}$ for the contributing data, and the means are formed after the two sets of data are reduced to a common zero point. This time, the difference between the external and internal errors is less than $2\sigma$ (see the fifth line of Table 2). The result in the fourth line of Table 2 is therefore regarded as a statistical fluctuation. In subsequent stages of the analysis, the internal errors for stages 1-3 are adopted without change.

  

Table 2: Comparison of internal and external rms errors

• All numbers in Cols. 3, 5, and 6 are in dex.
• $^{\mathrm{a}}$ "McW'' is McWilliam 1990, ``Zpt'' refers to the zero-point data, and "GKA/W'' refers to Gustafsson et al. 1974 and Williams 1971, 1972.
• $^{\mathrm{b}}$ "McW'' is McWilliam 1990.
• $^{\mathrm{c}}$ This is the number of stars used in the comparison.
• $^{\mathrm{d}}$ No stars used to derive Eq. (2) are used in this test. As a result, the quoted internal and external results are from completely independent determinations.

4.5 The error analysis: Stages 4-6

At stage 4 of the analysis, the data-comparison algorithm is applied to derive a value of $\sigma_0$ for the results of Brown et al. (1989). Using again the terminology of Sect. 4.3, let the Brown et al. data make up set B. For set A, an interim set of weighted averages is used. Contributing data for the averages are from stages 1-3 of the analysis. As before, the weights are values of $\sigma_0^{-2}$. The weights are used to assign rms errors to the set A averages. (The equations applied here are Eqs. (C3) and (C5) of Appendix C of T91.)

At stage 5, the above procedure is repeated for the "Luck et al.'' data (from Luck 1991; Luck & Challener 1995; and Luck & Wepfer 1995). Interim averages from stages 1-4 are employed, and the Luck et al. data are first corrected for the way in which a model-atmosphere grid was used to derive them (see Appendix B). At stage 6, interim averages from stages 1-5 are used. The data-comparison algorithm is now applied to derive values of $\sigma_0$ for data sets requiring special error analyses (note again the last line of Table 1). In a special preliminary reduction, the data of Helfer & Wallerstein (1968) are tested at this stage (see below).

For the Brown et al. and Luck et al. results, the derived values of $\sigma_0$are $0.11\ \pm\ 0.01$ dex and $0.12\ \pm\ 0.02$ dex, respectively. Note that those errors and the others given so far are quite similar. The quoted error for the Luck et al. data is for values of [Fe/H] which those authors obtain while calculating spectroscopic values of $\log g$. Luck et al. also give values of [Fe/H] derived by using so-called "physical'' gravities. For those latter data, $\sigma_0 = 0.07 \pm$ 0.02 dex, so their precision may be somewhat better than that of their "spectroscopic'' counterparts. Nevertheless, the "spectroscopic'' data are adopted when available. (This procedure is a response to the problem posed by the metallicity of $\mu$ Leo and will be discussed in a subsequent paper.)

At stage 6, unusually large values of $\sigma_0$ are obtained for some data sets. Even if those data sets require no systematic corrections, they are excluded from the zero-point data set. Identifying noisy data is another reason for iterating the analysis (recall Sect. 3.2).

4.6 Accidental errors and analysis procedures

As noted above, no use is made here of errors inferred from high-dispersion analysis procedures. In particular, such errors are not substituted for errors derived from data scatter. There is admittedly a certain surface plausibility in assuming that there is a link between small errors and reliable analysis procedures. However, it should be noted that such a link may be urged on the basis of supposition or even outright bias. To forestall such problems, an appeal to numerical demonstration is required.

To gain some insight into this issue, the data of Helfer & Wallerstein (1968) have been tested. Those data are from differential curve-of-growth (DCOG) analyses. The shortcomings of principle of such analyses seem to be beyond reasonable doubt, judging from comments made by Griffin (1969, 1975). It is of obvious interest to find out whether those shortcomings lead to an inflated accidental error.

In a test reduction, the Helfer-Wallerstein data were held out of the analysis until stage 6. An error was then derived for those data by using the data-comparison algorithm. The derived error for the Helfer-Wallerstein data was found to be $0.126 \pm 0.027$ dex. Recall that for the stage 2 zero-point data as a whole, $\sigma_0 = 0.103 \pm$ 0.009 dex. Plainly the two values of $\sigma_0$ are effectively the same.

Model-atmosphere procedures are likely to resemble each other more closely than any of them resemble DCOG analysis. For this reason, the result just given suggests that there should be no general correlation between error size and type of model-atmosphere analysis. A correlation of this sort may appear in specific instances; note that such a correlation may have been found for the two kinds of Luck et al. analysis (recall Sect. 4.5). However, it seems clear that one should insist on numerical demonstration in all cases before deciding that such a correlation exists.