benchmark.md
March 3, 2026 · View on GitHub
🏆 Evaluation Results
FireRed-Image-Edit establishes a new state-of-the-art among open-source models on Imgedit, Gedit, and RedEdit, while surpassing our closed-source competitors in specific dimensions—a distinction further corroborated by human evaluations highlighting its superior prompt following and visual consistency.
Results on ImgEdit
| Model | Overall ↑ | Add | Adjust | Extract | Replace | Remove | BG | Style | Hybrid | Action |
|---|---|---|---|---|---|---|---|---|---|---|
| 🔹 Proprietary Models | ||||||||||
| Nano-Banana | 4.29 | 4.62 | 4.41 | 3.68 | 4.34 | 4.39 | 4.40 | 4.18 | 3.72 | 4.83 |
| Seedream4.0 | 4.30 | 4.33 | 4.38 | 3.89 | 4.65 | 4.57 | 4.35 | 4.22 | 3.71 | 4.61 |
| Seedream4.5 | 4.32 | 4.57 | 4.65 | 2.97 | 4.66 | 4.46 | 4.37 | 4.92 | 3.71 | 4.56 |
| Nano-Banana-Pro | 4.37 | 4.44 | 4.62 | 3.42 | 4.60 | 4.63 | 4.32 | 4.97 | 3.64 | 4.69 |
| 🔹 Open-source Models | ||||||||||
| FLUX.1 Kontext [Dev] | 3.71 | 3.99 | 3.88 | 2.19 | 4.27 | 3.13 | 3.98 | 4.51 | 3.23 | 4.18 |
| Step1X-Edit-v1.2 | 3.95 | 3.91 | 4.04 | 2.68 | 4.48 | 4.26 | 3.90 | 4.82 | 3.23 | 4.22 |
| Qwen-Image-Edit-2509 | 4.31 | 4.34 | 4.27 | 3.42 | 4.73 | 4.36 | 4.37 | 4.91 | 3.56 | 4.80 |
| FLUX.2 [Dev] | 4.35 | 4.50 | 4.18 | 3.83 | 4.65 | 4.65 | 4.31 | 4.88 | 3.46 | 4.70 |
| LongCat-Image-Edit | 4.45 | 4.44 | 4.53 | 3.83 | 4.80 | 4.60 | 4.33 | 4.92 | 3.75 | 4.82 |
| Qwen-Image-Edit-2511 | 4.51 | 4.54 | 4.57 | 4.13 | 4.70 | 4.46 | 4.36 | 4.89 | 4.16 | 4.81 |
| FireRed-Image-Edit | 4.56 | 4.55 | 4.66 | 4.34 | 4.75 | 4.58 | 4.45 | 4.97 | 4.07 | 4.71 |
Results on GEdit (official public benchmark)
| Model | G_SC ↑ (EN) | G_PQ ↑ (EN) | G_O ↑ (EN) | G_SC ↑ (CN) | G_PQ ↑ (CN) | G_O ↑ (CN) |
|---|---|---|---|---|---|---|
| 🔹 Proprietary Models | ||||||
| Nano-Banana | 7.396 | 8.454 | 7.291 | 7.540 | 8.424 | 7.399 |
| Seedream4.0 | 8.143 | 8.124 | 7.701 | 8.159 | 8.074 | 7.692 |
| Nano-Banana-Pro | 8.102 | 8.344 | 7.738 | 8.135 | 8.306 | 7.799 |
| Seedream4.5 | 8.268 | 8.167 | 7.820 | 8.254 | 8.167 | 7.800 |
| 🔹 Open-source Models | ||||||
| FLUX.2 [Dev] | 7.835 | 8.064 | 7.413 | 7.697 | 8.046 | 7.278 |
| Qwen-Image-Edit-2509 | 7.974 | 7.714 | 7.480 | 7.988 | 7.679 | 7.467 |
| Step1X-Edit-v1.2 | 7.974 | 7.714 | 7.480 | 7.988 | 7.679 | 7.467 |
| Longcat-Image-Edit | 8.128 | 8.177 | 7.748 | 8.141 | 8.117 | 7.731 |
| Qwen-Image-Edit-2511 | 8.297 | 8.202 | 7.877 | 8.252 | 8.134 | 7.819 |
| FireRed-Image-Edit | 8.363 | 8.245 | 7.943 | 8.287 | 8.227 | 7.887 |
Results on REDEdit-Bench-CN (General Dimensions)
| Model | Overall | Add | Adjust | BG | Beauty | Color | Compose | Extract | Portrait | Low-level | Motion | Remove | Replace | Stylize | Text | Viewpoint |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 🔹 Proprietary Models | ||||||||||||||||
| Seedream4.0 | 4.15 | 4.55 | 4.11 | 4.61 | 3.83 | 4.14 | 4.16 | 2.48 | 4.77 | 4.17 | 4.68 | 4.02 | 4.53 | 4.94 | 3.94 | 3.29 |
| Seedream4.5 | 4.18 | 4.58 | 4.09 | 4.57 | 3.97 | 4.12 | 4.05 | 2.56 | 4.80 | 3.99 | 4.78 | 4.12 | 4.53 | 4.94 | 4.07 | 3.53 |
| Nano-Banana | 4.13 | 4.66 | 4.26 | 4.63 | 4.37 | 4.13 | 3.94 | 3.17 | 4.83 | 4.05 | 4.75 | 4.07 | 4.74 | 3.63 | 3.69 | 3.09 |
| Nano-Banana-Pro | 4.48 | 4.66 | 4.41 | 4.58 | 4.35 | 4.58 | 4.36 | 3.42 | 4.86 | 4.46 | 4.91 | 4.54 | 4.79 | 4.85 | 4.69 | 3.75 |
| 🔹 Open-source Models | ||||||||||||||||
| Qwen-Image-Edit-2509 | 4.00 | 4.45 | 4.04 | 4.48 | 3.36 | 4.20 | 3.92 | 2.64 | 4.16 | 3.52 | 4.66 | 4.27 | 4.66 | 4.81 | 3.53 | 3.32 |
| FLUX.2 [Dev] | 4.05 | 4.31 | 3.88 | 4.57 | 3.80 | 3.91 | 3.85 | 2.47 | 4.50 | 4.43 | 4.68 | 3.50 | 4.47 | 4.95 | 3.53 | 3.88 |
| Longcat-Image-Edit | 4.12 | 4.34 | 4.25 | 4.54 | 3.72 | 4.12 | 3.92 | 2.48 | 4.49 | 4.31 | 4.67 | 4.27 | 4.61 | 4.94 | 3.83 | 3.30 |
| Qwen-Image-Edit-2511 | 4.18 | 4.50 | 4.23 | 4.52 | 3.61 | 4.09 | 4.00 | 3.22 | 4.31 | 4.19 | 4.66 | 4.41 | 4.68 | 4.83 | 4.08 | 3.51 |
| FireRed-Image-Edit | 4.33 | 4.57 | 4.37 | 4.64 | 3.69 | 4.45 | 4.29 | 3.49 | 4.50 | 4.56 | 4.65 | 4.47 | 4.81 | 4.93 | 4.49 | 3.14 |
Results on REDEdit-Bench-EN (General dimensions)
| Model | Overall | Add | Adjust | BG | Beauty | Color | Compose | Extract | Portrait | Low-level | Motion | Remove | Replace | Stylize | Text | Viewpoint |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 🔹 Proprietary Models | ||||||||||||||||
| Nano-Banana | 4.15 | 4.65 | 4.23 | 4.60 | 4.37 | 4.08 | 3.98 | 3.39 | 4.72 | 4.03 | 4.63 | 4.07 | 4.68 | 3.68 | 3.87 | 3.23 |
| Seedream4.0 | 4.18 | 4.59 | 4.12 | 4.63 | 3.89 | 4.10 | 4.14 | 2.28 | 4.77 | 4.12 | 4.73 | 4.23 | 4.56 | 4.98 | 4.21 | 3.42 |
| Seedream4.5 | 4.20 | 4.66 | 4.08 | 4.64 | 4.12 | 4.07 | 4.10 | 2.23 | 4.74 | 4.28 | 4.75 | 4.24 | 4.58 | 4.97 | 4.20 | 3.44 |
| Nano-Banana-Pro | 4.42 | 4.72 | 4.40 | 4.64 | 4.37 | 4.43 | 4.32 | 3.25 | 4.82 | 4.36 | 4.85 | 4.52 | 4.75 | 4.90 | 4.54 | 3.51 |
| 🔹 Open-source Models | ||||||||||||||||
| Qwen-Image-Edit-2509 | 3.99 | 4.47 | 4.06 | 4.49 | 3.13 | 3.98 | 3.85 | 2.91 | 4.30 | 3.71 | 4.58 | 4.40 | 4.67 | 4.77 | 3.77 | 2.85 |
| FLUX.2 [Dev] | 4.07 | 4.37 | 3.96 | 4.47 | 3.72 | 3.86 | 3.87 | 2.36 | 4.44 | 4.45 | 4.67 | 4.02 | 4.48 | 4.87 | 3.80 | 3.84 |
| LongCat-Image-Edit | 4.12 | 4.38 | 4.04 | 4.49 | 3.89 | 4.10 | 3.93 | 2.98 | 4.47 | 4.27 | 4.69 | 4.24 | 4.51 | 4.86 | 3.83 | 3.25 |
| Qwen-Image-Edit-2511 | 4.23 | 4.55 | 4.17 | 4.56 | 3.49 | 4.07 | 4.07 | 3.54 | 4.42 | 4.52 | 4.72 | 4.42 | 4.65 | 4.85 | 4.06 | 3.38 |
| FireRed-Image-Edit | 4.26 | 4.41 | 4.33 | 4.60 | 3.55 | 4.47 | 4.25 | 3.49 | 4.50 | 4.44 | 4.65 | 4.46 | 4.70 | 4.94 | 4.44 | 2.78 |