High-dimensional covariance estimation by minimizing $\ell_1$-penalized log-determinant divergence

Abstract

Given i.i.d. observations of a random vector $X \in \mathbb{R}^p$, we study the problem of estimating both its covariance matrix $Σ^*$, and its inverse covariance or concentration matrix {$Θ^* = (Σ^*)^{-1}$.} We estimate $Θ^*$ by minimizing an $\ell_1$-penalized log-determinant Bregman divergence; in the multivariate Gaussian case, this approach corresponds to $\ell_1$-penalized maximum likelihood, and the structure of $Θ^*$ is specified by the graph of an associated Gaussian Markov random field. We analyze the performance of this estimator under high-dimensional scaling, in which the number of nodes in the graph $p$, the number of edges $s$ and the maximum node degree $d$, are allowed to grow as a function of the sample size $n$. In addition to the parameters $(p,s,d)$, our analysis identifies other key quantities covariance matrix $Σ^*$; and (b) the $\ell_\infty$ operator norm of the sub-matrix $Γ^*_{S S}$, where $S$ indexes the graph edges, and $Γ^* = (Θ^*)^{-1} \otimes (Θ^*)^{-1}$; and (c) a mutual incoherence or irrepresentability measure on the matrix $Γ^*$ and (d) the rate of decay $1/f(n,δ)$ on the probabilities $ \{|\hatΣ^n_{ij}- Σ^*_{ij}| > δ\}$, where $\hatΣ^n$ is the sample covariance based on $n$ samples. Our first result establishes consistency of our estimate $\hatΘ$ in the elementwise maximum-norm. This in turn allows us to derive convergence rates in Frobenius and spectral norms, with improvements upon existing results for graphs with maximum node degrees $d = o(\sqrt{s})$. In our second result, we show that with probability converging to one, the estimate $\hatΘ$ correctly specifies the zero pattern of the concentration matrix $Θ^*$.

35 pages, 9 figures