Image Ranker is an open-source tool for efficiently ranking large collections of images through pairwise comparisons using a Bayesian TrueSkill-based ranking algorithm. It enables fast and scalable human-in-the-loop preference estimation without requiring exhaustive comparisons.

The system maintains probabilistic ratings for each image, modeling both estimated quality (μ) and uncertainty (σ), and updates rankings globally after each comparison. This allows accurate ranking from incomplete and noisy comparison data.

To further accelerate ranking on large datasets, Image Ranker includes several optimization strategies:

Sequential elimination, reducing ranking complexity from O(N²) to O(N) Uncertainty-driven sampling (smart shuffle) to prioritize informative comparisons Auto-shuffling for continuous ranking efficiency The tool provides a lightweight web-based interface that allows users to:

Compare images directly from local directories without uploading data Track and export rankings (CSV) with resume capability Annotate exclusions with structured reasons Attach contextual metadata to images Image Ranker is particularly useful for:

Human preference data collection for reinforcement learning from human feedback (RLHF) Evaluation and selection of generative model outputs (e.g. diffusion models) Dataset curation and ranking under limited labeling budgets By combining probabilistic ranking with efficient comparison strategies, Image Ranker enables scalable preference estimation workflows for modern machine learning and data-centric applications.