Regularized Covariance Estimation for Weighted Maximum Likelihood Policy Search Methods

• Type: Article in International Conference Proceedings Book
• Date: 2015

Authors

Abdolmaleki, A (Author) Other; lau, n (Author) FCUP; reis, lp (Author) REIT; Neumann, G (Author) Other

Conference proceedings International

Title: IEEE-RAS International Conference on Humanoid Robots

Pages: 154-159

15th IEEE RAS International Conference on Humanoid Robots, Humanoids 2015

3 November 2015 through 5 November 2015

Indexing

ISI Web of Knowledge - 3 Citations

Scopus - 12 Citations

Other information

Authenticus ID: P-00K-AP3

DOI: 10.1109/HUMANOIDS.2015.7363529

Abstract (EN): Many episode-based (or direct) policy search algorithms, maintain a multivariate Gaussian distribution as search distribution over the parameter space of some objective function. One class of algorithms, such as episodic REPS, PoWER or PI2 uses, a weighted maximum likelihood estimate (WMLE) to update the mean and covariance matrix of this distribution in each iteration. However, due to high dimensionality of covariance matrices and limited number of samples, the WMLE is an unreliable estimator. The use of WMLE leads to overfitted covariance estimates, and, hence the variance/entropy of the search distribution decreases too quickly, which may cause premature convergence. In order to alleviate this problem, the estimated covariance matrix can be regularized in different ways, for example by using a convex combination of the diagonal covariance estimate and the sample covariance estimate. In this paper, we propose a new covariance matrix regularization technique for policy search methods that uses the convex combination of the sample covariance matrix and the old covariance matrix used in last iteration. The combination weighting is determined by specifying the desired entropy of the new search distribution. With this mechanism, the entropy of the search distribution can be gradually decreased without damage from the maximum likelihood estimate.

Language: English

Type (Professor's evaluation): Scientific

No. of pages: 6