Anime Scene Search Engine (2021)
December 31, 2021 · View on GitHub
Anime Scene Search Engine (2021)
trace.moe 2021 Project Updates
by @soruly
All presentations slides: https://github.com/soruly/slides
Boring Stats
- 12 million total page views (GA) +7.5% YoY
- ? million search served by Telegram Bot
- ? million API search served for 3rd party apps/bots
- 11k Web Extension users (10k Chrome, Firefox+Opera+Edge: <2k)
- 130k monthly active users, 6k daily users (GA)
- 450k monthly unique visiters, 20k daily (Cloudflare)
- 22k daily search
Updates Summary of 2021
- New website design
- MS Edge Extension
- trace.moe API standardized and out of beta
- Rate limit and queue system
- Search quota
- User accounts for donator
- New backend infrastructure
Website redesign
- Nextjs
- SFW by default, video preview of all results, concise anime info
- Cloudflare Pages + Cloudflare Workers (for image proxy and title translation)
- Search traffic and search time
- Database statistics
- New status page
https://github.com/soruly/trace.moe-www
MS Edge Extension
Windows Explorer Context Menu
https://github.com/soruly/trace.moe-windows-menu
Telegram Bot
Linode => Vercel => Heroku / Railway
https://github.com/soruly/trace.moe-telegram-bot-vercel
data size
- 5162 => 5604 anime series
- 29256 => 31259 hours, 21TB = >23TB
- 2.7 billion frames analyzed, 772 million indexed (156GB in size)
Analyzing all anime takes 2x3950X+1x3700X 75 hours to complete (peak ~10000 hours/day)
API
https://github.com/soruly/trace.moe-api
New Backend Infrastructure
- Tidy up a lot 🥰
- fully containerized
- Updated LIRE, Solr, Java (8=>17)
- RabbitMQ => WebSocket
- CI tests
Previously Planned Works
- Migrate from PHP to nodejs ✅
- Search queue w/ priority ✅
- Test cases ✅
- Migrate to cloud partially 🥴
- New server ⌛
New Planned Works
- Prepare for Web Extension Manifest V3
- Index Youtube channels automatically
- Detect and store scene boundaries up-front (with Rust?)
- Better search image manipulation (like black border crop)
- Cross reference results with JCD if new server is available
- Match video to video
Project URLs
- https://github.com/soruly/trace.moe
- https://github.com/soruly/trace.moe-api
- https://github.com/soruly/trace.moe-www
- https://github.com/soruly/trace.moe-media
- https://github.com/soruly/trace.moe-worker
- https://github.com/soruly/trace.moe-WebExtension
- https://github.com/soruly/trace.moe-telegram-bot
- https://github.com/soruly/trace.moe-image-proxy (included in trace.moe-www)
- https://github.com/soruly/anilist-crawler (no longer a dependency)
- https://github.com/soruly/anilist-chinese (no longer a dependency)
- https://github.com/soruly/liresolr
- https://github.com/soruly/sola (no active development)
Official Channels
- Patron: https://www.patreon.com/soruly
- Discord: https://discord.gg/K9jn6Kj
- Telegram: https://telegram.me/whatanimeupdates
- Facebook Page (abandoned)
Credits
- Dr. Mathias Lux for LIRE Project and liresolr
- anilist.co for providing anime info via Anilist API
- Sponsors from Patreon, GitHub and Paypal
- fans who helped answering questions in various channels
- friends who reported bugs and issues to me