Lingua Motus ✨🤟

March 5, 2026 · View on GitHub

An on-device ASL fingerspelling coach — not a grader.

Lingua Motus (“language of motion” in Latin) is a Swift Playgrounds app that helps beginners practice ASL fingerspelling with real-time, supportive feedback.
Instead of giving a cold correct/incorrect, it tries to explain what to fix — curl/straighten, touch/separate, rotate/orient — so learning feels human, encouraging, and progress-driven.

“Simple can be harder than complex…” — Steve Jobs

🖼️ Screenshots

HomeLessonSplashCustom Fingerspelling
HomeLessonSplashCustom Fingerspelling

🎥 Demo Video: Coming soon

🎯 The problem

A lot of ASL apps stop at recognition:

  • ✅ “Correct”
  • ❌ “Wrong”

But that’s not how humans learn. Beginners need direction, not judgment.

Lingua Motus is designed as a coach:

  1. Detect the hand pose live
  2. Reduce jitter + unfair snap judgments
  3. Generate actionable corrective guidance that tells you what to do next

🌟 What makes it different

🧠 Coach-like corrective feedback (not pass/fail)

When a sign is off, the app tries to surface a high-level action you can actually apply:

  • Curl / Straighten
  • Touch / Separate
  • Rotate / Orient

To make it intuitive, feedback is visualized using SF Symbol arrows placed near the relevant joint(s), so users learn what to move — not just what they got wrong.

👻 Ghost Hand overlay (visual guidance that feels natural)

A toggleable ghost-hand reference overlay appears directly over the camera preview to guide beginners in real time.

  • ✅ Cleanly masked overlay
  • ✅ Controlled by app state
  • ✅ Designed to feel “built-in”, not patched on

🎉 Custom fingerspelling practice (practice what you actually use)

Users can practice their own words/phrases, not random drills.
The flow ends with a satisfying completion moment (yes — confetti ✨) to keep learning motivating.

🫳 Stability filtering (fair to shaky hands)

Beginners don’t hold perfect poses.
The pipeline smooths jitter and waits for more consistent frames before surfacing strong corrections — reducing frustration and false negatives.

📱 Responsive UI (iPhone + iPad)

Designed to adapt across device sizes and orientations, including iPad landscape.

✅ Key features

  • Real-time on-device recognition

    • Vision hand landmarks in real time
    • Core ML letter + number classifiers (bundled .mlmodelc resources)

  • Corrective hint generation

    • Geometric checks + rule-based hinting for actionable feedback

  • Ghost Hand overlay

    • Masked reference overlay on top of the live camera preview

  • Custom practice flow

    • Fingerspell user-entered words/phrases + completion moment

🧩 Tech stack

  • SwiftUI — UI, navigation, state-driven overlays
  • AVFoundation — camera pipeline
  • Vision — hand pose landmarks + masking
  • Core ML — letters & numbers classification
  • SIMD / vector math — fast geometric checks for hinting
  • JSON resources — feedback + error rules
    • Resources/Feedback.JSON
    • Resources/ErrorChecks.JSON

🏗️ Project structure (high-level)

  • MyApp.swift — entry point, injects shared view models
  • MainAppScreen.swift — routes onboarding/loading/home based on app state
  • Views/CameraView.swift — camera preview + ghost hand + feedback overlays
  • Models/CameraViewModel.swift + ML helpers — capture → landmarks → features → prediction → feedback
  • MLModel/ — compiled Core ML models (letter_classifier.mlmodelc, number_classifier.mlmodelc)

✅ Requirements

  • iOS 18.1+ (configured in Package.swift)
  • Camera permission (requested with an explicit purpose string)

▶️ How to run

  1. Open the .swiftpm project in Swift Playgrounds
  2. Run on iPhone/iPad (or supported simulator) and grant camera permission
  3. Start a lesson and practice

Option B — Xcode

  1. Open the .swiftpm package in Xcode
  2. Select a real device (best for camera + performance)
  3. Build & Run

🔒 Privacy

All recognition and analysis is fully on-device.
No video is recorded or saved.

🙌 Credits

  • Confetti effect adapted from JC’s tutorial:
    “SwiftUI Tutorials: Designing a Dynamic Confetti Effect” (May 8, 2024)

🤖 AI tooling disclosure

AI tools were used as assistants for:

  • writing refinement
  • design exploration
  • development support (debugging/refactoring/math organization)

All outputs were reviewed, tested, and integrated manually.

👤 Author

Youssef Ashraf

📄 License

Apache-2.0