Basic Idea of Factor Analysis as a Data Reduction Method
Let us look at the following example. Suppose we want to measure people's satisfaction with their lives. We design a satisfaction questionnaire with various items; among other things we ask our subjects how satisfied they are with their hobbies and how intensely they are pursuing a hobby. Most likely, the responses to the two items are highly correlated with each other. Given a high correlation between the two items, we can conclude that they are quite redundant.
Combining Two Variables into a Single Factor
One can summarize the correlation between two variables in a scatter plot. A regression line can then be fitted that represents the "best"; summary of the linear relationship between the variables. If we could define a variable that would approximate the regression line in such a plot, then that variable would capture most of the "essence" of the two items. Subjects' single scores on that new factor, represented by the regression line, could then be used in future data analyses to represent that essence of the two items. In a sense we have reduced the two variables to one factor. Note that the new factor is actually a linear combination of the two variables.
Principal Components Analysis
The example described above, combining two correlated variables into one factor, illustrates the basic idea of factor analysis or of principal components analysis to be precise. If we extend the two-variable example to multiple variables, then the computations become more involved, but the basic principle of expressing two or more variables by a single factor remains the same.