The explosive growth of uploaded photos requires service providers to improve the overall efficiency for analyzing and manipulating uploaded photos. Seeing the needs, we investigate the feasibility to speed up photo-related analysis (e.g., face detection) by leveraging the observations from large-scale consumer photos. To deal with the huge computations of face detection on the large-scale photos, we aim to reduce search hypotheses (i.e., subwindows) for face detectors by thresholding (or ranking) on the confidence of hypothesis that indicates the possibility to be a face. Unlike the traditional sliding window which views all the hypotheses equally, prior knowledge of face sizes and locations in images is learned from the large-scale consumer photos, and used to model the confidence of each hypothesis. By thresholding on the ranked hypotheses, less likely hypotheses can be reduced for better efficiency. Our experiment demonstrates that 40% subwindows can be skipped while maintaining 95% face recall, compared to traditional sliding-window-based methods.