In kernel ridge regression we have seen the final solution was not sparse in the variables ® .

We will now formulate a regression method that is sparse, i.e. it has the concept of support

vectors that determine the solution.

The thing to notice is that the sparseness arose from complementary slackness conditions

which in turn came from the fact that we had inequality constraints. In the SVM the penalty

that was paid for being on the wrong side of the support plane was given by C

P

i » k

i for

positive integers k, where » i is the orthogonal distance away from the support plane. Note

that the term jjwjj2 was there to penalize large w and hence to regularize the solution.

Importantly, there was no penalt