Learning from Data: Concepts, Theory, and Methods (2nd Edition)

This is the second edition, published by Wiley-IEEE Press in 2007, by Vladimir Cherkassky, professor in the Department of Electrical and Computer Engineering at the University of Minnesota, and Filip M. Mulier. The book seeks to establish a unified, interdisciplinary conceptual framework for "learning dependencies from data": it places statistical, neural-network, and pattern-recognition learning methods under one set of principles and shows that the many new methods of recent years share a small number of fundamental ideas. The book moves through several levels: it first defines the predictive learning problem (density estimation, classification, regression) and discusses classical and nonparametric methods under finite samples, along with stochastic approximation; it then introduces the regularization framework, covering the curse of dimensionality, penalty terms, model selection, and the bias-variance tradeoff; it goes on to develop statistical learning theory systematically, including the consistency conditions of ERM, growth functions and VC dimension, generalization bounds, and structural risk minimization, linking VC dimension to Occam's razor and Popper's falsifiability; it then discusses nonlinear optimization strategies (backpropagation, EM, greedy optimization, CART) and methods of data reduction and dimensionality reduction. The second edition completely rewrites the chapter on support vector machines, adds a chapter on non-inductive inference and alternative learning formulations, and discusses in greater depth the relation of VC theory to other paradigms. The book includes more than a hundred figures, case studies, and exercises, with summaries at the end of each chapter, balancing theoretical depth with readability, and is suitable for graduate courses.