Measuring per-pixel surface roughness is useful for computer vision and graphics applications such as visual inspection and photo-realistic image synthesis. In this paper, we propose a method for measuring per-pixel surface roughness from specular reflection components in two images. Our proposed method focuses on the fact that the observed specular reflection component depends on the light sources around the mirror-reflection direction seen from the viewing direction. Specifically, we use linear/non-linear patterns displayed on an LCD for estimating per-pixel correspondence/surface roughness. We conduct experiments using a display-polarization camera system, and study how our method behaves depending on the nonlinear patterns.