text-generation.md

💬 Text Generation

Below are some 🛠️ dynamic configuration settings related to Text Generation.

You may also wish to see:

• 🌟 Features / 💬 Text Generation for a higher-level introduction to the Text Generation features
• 📖 Usage / 💬 Text Generation section for more details on how to use the bot for Text Generation in a room

🗟 Prefix Requirement Type

In Direct Message rooms with the bot (1:1 rooms), it most usually makes sense for the bot to respond to all of your messages, as shown on this 🖼️ screenshot.

In group rooms (with multiple users), it may be more appropriate for the bot to only respond to messages that are prefixed with the command prefix (e.g. !bai) or which are mentioning the bot (e.g. @baibot), so that other chat exchange in the room will not trigger it. Such a setup is shown on the 🖼️ On-demand involvement in the room screenshot.

There are exceptions to these rules, and you can configure the bot to respond only to prefixed messages in a 1:1 room, or to respond to all messages even in a multi-user group room.

To support such use-cases, the bot has a text-generation prefix-requirement-type setting, which can be set to:

• (default) no: indicates that the bot would not require a prefix and would respond to all messages

• command_prefix: indicates that the bot would require that messages be prefixed with the command prefix (e.g. !bai) and would ignore all messages that are not prefixed

By default, the bot is auto-configured (upon joining a new room) to use the no setting in rooms that only include 2 users (you and the bot), and command_prefix in rooms with more than 2 users. To prevent surprises, the bot will not adjust this setting subsequently. You can manually adjust it via !bai config room text-generation set-prefix-requirement-type VALUE.

Example: !bai config room text-generation set-prefix-requirement-type command_prefix (this can also be set globally, see 🛠️ Room Settings)

Regardless of this configuration, the bot will also respond to messages by allowed 👥 Users which directly mention the bot (e.g. @baibot), even if they are not prefixed. An example of this can be seen on these screenshots:

• 🖼️ On-demand involvement in a thread
• 🖼️ On-demand involvement in a reply chain

🪄 Auto Usage

Text generation is enabled by default (the text-generation auto-usage setting being set to always), but can be set to:

• (default) always: generate text for all messages (also see 🗟 Prefix Requirement Type)

• never: never generate text for messages

• only_for_voice: only generate text when the original user message was a voice message, later transcribed via 🦻 Speech-to-Text

• only_for_text: only generate text when original user message was a text message

Example: !bai config room text-generation set-auto-usage only_for_voice (this can also be set globally, see 🛠️ Room Settings)

♻️ Context Management

The bot also supports ♻️ context management, which automatically trims the oldest messages once a conversation grows past the context window. It drops whole turns at a time, so a reply is never separated from the message it answered.

Counting tokens precisely needs the model's own tokenizer. For OpenAI models, the bot counts them with the tiktoken-rs library. For every other provider, including the recommended Venice, the bot falls back to a provider-neutral approximation that needs no per-model tokenizer: it counts ASCII text at about four characters per token and other scripts (Cyrillic, CJK, and so on) at about two. Treat it as rough, within roughly 10-20% of the real count for typical text, which is plenty for keeping a long conversation inside the context window.

This setting is disabled by default, but can be enabled via !bai config room text-generation set-context-management-enabled true (this can also be set globally, see 🛠️ Room Settings).

💭 Thinking Notice

The bot can post a 💭 "thinking…" notice while text generation is running, useful for slow models (for example, reasoning models that may run for minutes) where the response would otherwise look stuck.

When enabled, a placeholder message appears only after a short delay (so fast responses get no notice), updates periodically with varying status text, and is then edited in place to become the final answer.

This setting is disabled by default, but can be enabled via !bai config room text-generation set-thinking-notice-enabled true (this can also be set globally, see 🛠️ Room Settings).

👤 Sender Context Mode

In multi-user rooms, it may be useful for the model to know which participant sent each message in the conversation context.

To support this, the bot has a text-generation sender-context-mode setting, which can be set to:

• (default) disabled: do not attach sender metadata to messages before sending them to the model

• matrix_user_id: prefix text messages with the sender's Matrix user ID, for example: [sender=@alice:example.com] Hello bot

• matrix_user_id_and_timestamp: prefix text messages with the sender's Matrix user ID and the message timestamp, for example: [sender=@alice:example.com sent_at=2026-03-23T14:30:00Z] Hello bot

This sender metadata is attached to conversation messages before they are sent to the model provider. It applies to user and assistant text messages, but not to system prompts or non-text content.

⚠️ Enabling this sends Matrix user IDs, and optionally timestamps, to the model provider.

Example: !bai config room text-generation set-sender-context-mode matrix_user_id (this can also be set globally, see 🛠️ Room Settings)

⌨️ Prompt Override

You can override the system prompt configured at the 🤖 agent level.

Example (multi-line is supported):

!bai config room text-generation set-prompt-override You're a UI/UX expert. Everything you say needs to consider design and usability.

Where appropriate, you'll mention best practices and common pitfalls.

A prompt override can also be set globally, see 🛠️ Room Settings.

Prompts may contain the following placeholder variables which will be replaced every time the bot is interacted with:

Placeholder	Description	Example
{{ baibot_name }}	Name of the bot as configured in the user.name field in the Static configuration	Baibot
{{ baibot_model_id }}	Text-Generation model ID as configured in the 🤖 agent's configuration	gpt-4o
{{ baibot_now_utc }}	Current date and time in UTC (⚠️ usage may break prompt caching - see below)	2024-09-20 (Friday), 14:26:42 UTC
{{ baibot_conversation_start_time_utc }}	The date and time in UTC that the conversation started	2024-09-20 (Friday), 14:26:42 UTC

💡 {{ baibot_now_utc }} changes as time goes on, which prevents prompt caching from working. It's better to use {{ baibot_conversation_start_time_utc }} in prompts, as its value doesn't change yet still orients the bot to the current date/time.

💡 On the Venice provider, baibot derives the prompt-cache key from the system prompt and the conversation start time, both stable for the life of a conversation, and ships prompt_cache_retention: 24h by default. A stable system prompt then stays cached across the whole conversation instead of being reprocessed (and re-billed) on every turn.

Here's a prompt that combines some of the above variables:

You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."

🌡️ Temperature Override

You can override the temperature (randomness / creativity) parameter configured at the 🤖 agent level.

Example: !bai config room text-generation set-temperature-override 3.5 (this can also be set globally, see 🛠️ Room Settings)