The motion within an image region is modeled using an affine transformation. Affine motion parameters are estimated using a coarse-to-fine gradient-based method. The approach is implemented using a robust M-estimation technique to make the computation insensitive to outliers which result from multiple motions. The approach estimates a dominant motion in the scene and detects points which are not moving consistently with the dominant motion. Additional motions are optionally estimated using these outliers.