What does "a least-squares approximation to a sequence of data is simply their averaged value (mathematical expectation)" mean?

Question

I am reading the paper "Multiscale coarse graining of liquid-state systems" (https://doi.org/10.1063/1.2038787). In the paper, right after equation (1), they say "a least-squares approximation to a sequence of data is simply their averaged value (mathematical expectation)".

They use it as a justification to convert their least-square force-matching problem, to an averaging problem. My question is, when did least squares mean "average value"?

Let's consider a super simple problem of trying to predict $y$ from $x$ using function $g$ with parameter $t$, so $$y=g(x,t)$$

If I collect $N$ data points, my LSE would be $$LSE = \sum_{i=1}^N (y_i-g(x_i,t))(y_i-g(x_i,t))$$ differentiating with respect to $t$, and equating to zero, you get: $$\sum_{i=1}^N g'(x_i,t)(y_i - g(x_i,t))=0$$

I don't see how they can simply make the claim that "a least-squares approximation to a sequence of data is simply their averaged value"?

Any advice you have regarding this would be appreciated.

Answer 1

The fit, which minimises the root mean square value, is a fit to the "average values". To see this let's consider an example. Suppose we have five different input parameters $x_i = \{x_1, x_2, \ldots, x_5\}$ and that we take for each setting ten data point $y_{ij} = \{y_{1,1}, \ldots, y_{1,10}, y_{2,1}, \ldots, y_{2,10}, \ldots y_{5,10}\}$. If we use the least-square fit to find the optimal model $\hat y(x) = c x + \epsilon$, where $\epsilon$ is a random error, we obtain the same result, for the two following procedure:

1. Fit the model to the all data $(x_i, y_{ij})$, or
2. first calculate the average value for each parameter, $\bar y_{.j} = \frac{1}{10}\sum_{j=1}^{10} y_{j}$ and then fit the model to the averages $(x_i, \bar y_{.j})$.

This is evident when you look at a fitted line, e.g.

Here the R-code used to produce the plot

# set a seed for random numbers:
set.seed(1121)

# set true values by random sampling:
slope     = rnorm(1,mean=0, sd=5)
intercept = rnorm(1,mean=0, sd=5)

# get input parameters:
x = seq(-3, 10, length.out=5)
X = rep(x, 10) # just copy data multiple time

# get measurement values
Y = slope*X + rnorm(10*5, mean=0, sd=3) # add a random error

# plot:
df = data.frame(x=X, y=Y)
library(ggplot2)
gg = ggplot(df, aes(x, y)) + 
    geom_point(color="blue", alpha=0.6) +
    labs(x="Input parameters", y="Measurements")
print(gg)

# fit:
gg2 = gg + geom_smooth(method='lm', formula="y~x", color="red", alpha=0.1)
print(gg2)

lm.out = lm(y ~ x, df)
print(lm.out)

# fit2:
dim(X) = c(10,5)
dim(Y) = c(10,5)
yMean  = mean.rowwise(Y)
df2    = data.frame(x, y=yMean)
lm.out2 = lm(y ~ x, df2)
print(lm.out2)

Answer 2

Updated response to provide more information.

Let X and Y be random variables, and x and y be specific values of these random variables. Is Y is completely dependent on X, then there is a function y(x) that gives a unique y for each x, such as $y(x) = ax^2$ or $y(x) = mx + b$. If a function exists, then y is completely dependent on x. If the function is linear in x, then y and x are linearly dependent (completely correlated). (Note: uncorrelated does not imply independent; for $y = ax^2$, Y and X are uncorrelated, but Y depends on X. Independent does imply uncorrelated.)

Suppose we have two random variables Y and X and we expect Y to be somewhat linearly dependent on X; for example, if X is the net worth for a person and Y is the value of the home for that person. We take a sample of X and record the value x, and record the y for that x. If we plot the y vs. x values and see that a straight-line fit is reasonable (or more rigorously run standard tests to determine if there is a significant correlation), the best straight-line fit is determined by regression (a least squares evaluation) and using the least-squares evaluation the y values along the line are the best estimate (mean) values of y for x. That is, for a given x, the distribution of y values has a mean that lies along the regression line. In that sense, the y values are means. (This is rigorously true if for every x value, there is a random variable Y having a normal distribution and all the Y distributions have the same variance. But, results are not seriously affected if departures from normal distributions and equal variances are not too extreme.)

If there is complete correlation based on the sample, the linear correlation coefficient is +1, and y is a linear function of x with no uncertainty.

A good elementary discussion of all this is given in the text Elementary Statistics by Triola. More details are found in advanced statistics tests.