PUB - Publikationen an der Universität Bielefeld

In this thesis, the generalized regression (GR) framework is introduced as an extension of conventional regression estimation to the case of unsupervised learning. This approach unifies many supervised and unsupervised techniques for machine learning within a common framework. With regard to unsupervised methods it also closes the gap between projection and generative models, which are both treated as latent variable models for unsupervised regression differing only by the probabilistic nature of the latent variables. Therefore all models of this thesis can be considered as variations of a basic regression model, which gives rise to a convenient projection-regression scheme for estimation of function parameters and deterministic latent variables. For random latent variables an analogous estimation scheme is given by the well-known EM-algorithm. For the case of nonlinear dependencies on the latent variables, a homotopy-based optimization scheme is proposed for finding good local optima with respect to the objective functions. The rich class of unsupervised methods which is considered in this work comprises several well-known techniques which are shown to be special cases of GR, such as principal component analysis (PCA), vector quantization, factor analysis, independent component analysis or conventional mixture modelling. In addition, also several new developments, which have been inspired by the GR framework, are introduced. First a general basis function model for principal curves is proposed, which also comprises the computationally attractive polygonal and discrete approximations. In addition, suitable penalty functions (regularizers) which provide a convenient estimation scheme are introduced, together with a homotopy-based optimization scheme for finding good local minima of the highly non-convex error functions. The importance of homotopy-based optimization is also been shown for the case of local PCA. In particular, topographic local PCA is proposed as a generalization of the well-known self-organizing map (SOM). Furthermore a generative principal component model which requires the estimation of an effective latent variable dimensionality is introduced. This model is also extended to a principal component mixture with local subspace dimensionalities and again homotopy-based optimization, now via a fixed noise-variance parameter, is important in order to achieve reliable estimates. For that purpose a two-phase deterministic annealing approach is proposed, which combines the estimation of spherical and principal component mixtures in a sequential manner. In addition, on several application examples the feasibility of the proposed methods and their utility in practical real world domains is demonstrated. First unsupervised regression techniques are applied to the domains of visualization and sonification. For the visualization of high-dimensional data, principal curve models are utilized to provide distance plots of the data, which are well-suited for the detection of clusters and outliers. As an acoustic counterpart to the distance plots, principal curve sonification is introduced to provide an auditory display for multivariate data. Finally unsupervised regression models for images of handwritten digits are proposed for pattern recognition. As suitable candidates, several principal component mixtures are compared for the performance of their associated plug-in classifiers. For a possible improvement of image data representations, the topographic local PCA map is proposed for learning of invariances with respect to geometric transformations of the images with one or two main degrees of freedom.