yasbd-union
August 26, 2026 · View on GitHub
Sentence splitting for when you genuinely have no idea what language you're looking at.
What this is
yasbd-union is an experimental add-on for yasbd-lib.
It takes sentence-splitting rules from every built-in language in yasbd-lib and throws them into one shared space.
Sometimes it behaves nicely. Sometimes it makes bold assumptions. Sometimes it surprises even you.
That's basically the whole deal.
auto vs xx
auto tries to be smart about it.
It looks at your text, decides what language it is, and uses the right rules for the job. Clean and structured.
xx doesn't bother with that step.
It assumes your text is already a mix of everything and just applies all available rules at once.
auto | xx | |
|---|---|---|
| Language handling | Detects first | Doesn't care |
| Accuracy | Stable | Depends on bundled language rules |
| Mixed text | Not ideal | Basically its natural habitat |
| False splits | Rare | Happens sometimes |
| Personality | Careful | A bit chaotic, but trying its best |
| Best for | Clean text | Mixed-language messes |
Install
pip install yasbd-union
Then register it:
from yasbd import BoundaryDetector
detector = BoundaryDetector(lang="xx", external_lang_packs = ["yasbd_union"])
Example
sentences = list(detector.segment(
"Dr. Wang said 你好世界。Prof. Li replied 是的。"
))
print(sentences)
Output:
["Dr. Wang said 你好世界。", "Prof. Li replied 是的。"]
When to use xx
Use it when:
- You don't know what language your text is in
- Your input is messy, mixed, or unpredictable
- You're dealing with logs, chats, or scraped text
- You just want something that "tries its best"
When not to use xx
Avoid it when:
- You need strict, repeatable results
- Your text is single-language
- You don't want surprises in sentence boundaries
- You're trying to explain results to someone very literal
In those cases, auto or a specific language pack will behave better.
A few honest notes
- Some sentence splits will be slightly unexpected
- Results depend on what language rules are bundled in
yasbd-lib - It is not fully predictable by design
If that sounds like a problem, xx is probably not what you want.
License
MIT: If it breaks, it's still yours.