Daniel Dvorkin (danielmedic) wrote in stat_geeks,
Daniel Dvorkin

How many free parameters are there in a covariance matrix?

I'm trying to calculate BIC for a fairly complicated model which includes several covariance matrices, and it just occurred to me that I don't actually know how many free parameters this represents. If I have a D-by-D covariance matrix, the naive answer is that this represents D(D+1)/2 free parameters, because the matrix must be symmetric -- but the matrix must also be positive definite, which is a stronger condition, so I'm guessing that the actual number is something less than that. Any thoughts?

(x-posted to statisticians)
  • Post a new comment


    default userpic
I may be confused in this, so someone please correct me if I'm mistaken. However, I don't think I'd say there are D(D+1)/2 "FREE" parameters. I think the 'free parameters' are the ones that are still free to vary. In your model, you're constraining a number of them usually.

...I think
Fair enough; perhaps "number of parameters to be estimated" would be better than "free parameters." The point is that fitting the model includes estimating a number of covariance matrices, and I'd like to know how many parameters I'm actually estimating when I estimate such a matrix.
I'm reading the BIC the same way as AIC: That it's a value meant to rank several competing models against each other, not to evaluate one alone.

In BIC under certain assumptions which I'm assuming you'r using, the number of free parameter k turn up in this term: "+ k * ln(n)" where n is the number of data points. Assuming that all competing models have the same number of data points, this means that you can subtract "k1*ln(n)" from the BIC value of each model and still preserve their ranking.

Put another way, it does not matter how many free parameters your model REALLY has, you only need to figure out how many more or fewer the other models have.
True enough, but some of the models I'm comparing to each other have different-sized covariance matrices, so I need to know how many parameters I'm dealing with to calculate the difference (unless there's some very clever trick for comparing covariance matrices that doesn't require this.)
ah I see. If k = C * D(D+1) then the ranking would depend on the value of C. :-/

I don't know if a positive definite matrix is defined by a set of independent elements, or if the elements are correlated. (Correlations between covariances in this case.. :-}) If they, are the use of BIC is a lot more complicated, since it breaks the assumption of the usual implementations.

But as far as I can see from the literature, this is not the case. The possible values of an element may be restricted, but this is not the same as saying that the element is determined by the other elements. Nor can you from COV(X,Y) and COV(Y,Z) directly infer COV(X,Z) far as I can see