Virbe Documentation
Tutorials

Language Detection and Switching

Automatically detect the user's language from speech and switch the virtual being's STT and TTS to match – in real time.

This tutorial covers language detection and switching in detail – how Virbe detects spoken languages, how to control when and whether a switch happens, and how to handle edge cases like unsupported languages or mixed-language sessions.

What you'll build: A pipeline that detects the user's language, selectively accepts or rejects language switches based on your rules, and handles the transition gracefully with an acknowledgement message.

Time: approximately 30 minutes

Prerequisites:

  • A Virbe profile with multiple languages configured (Configure → Profiles → Language tab)
  • An STT engine that supports your target languages (Azure STT supports the widest range)
  • TTS voices configured for each target language in the persona

How Language Detection Works

The STT engine continuously analyses the audio of each user turn. If it detects that the spoken language differs from the currently active language, it fires the internal conversation-language-change signal. This triggers the On Language Change handler.

Inside the handler, you have a choice:

  • Accept the change – the active STT and TTS switch to the detected language for subsequent turns
  • Reject the change – the active language stays the same; the user's next turn is re-processed in the current language

The detected language is available as a conversation variable that you can inspect before deciding.

Step 1 – Check the Available Language Variable

When the On Language Change handler fires, the detected language code is available via the Insert field picker inside the handler's nodes. Common codes: en-US, es-ES, fr-FR, de-DE, nl-NL.

You can use this value in an If-Else Router node to conditionally accept or reject the switch.

Step 2 – Build a Conditional Language Switch

A more controlled version of language switching:

On Language Change
    │
    └── Flow Entrypoint
              │
              └── If-Else Router
                    │
                    ├── [Detected language is "es-ES"] → Accept Language Change → Text ("Claro, cambiamos al español.")
                    ├── [Detected language is "fr-FR"] → Accept Language Change → Text ("Bien sûr, passons au français.")
                    └── [default / other language]     → Reject Language Change → Text ("I'm sorry, I only support English, Spanish, and French.")

This pattern:

  1. Accepts Spanish and French switches with an acknowledgement in the target language
  2. Rejects all other languages with an English explanation

Building the If-Else Router conditions

In the If-Else Router node:

  1. Add a branch for each supported language
  2. Set the condition: [Detected language variable] equals "es-ES" (use the Insert field picker to reference the detected language)
  3. Add a default branch for the reject case

Step 3 – Accept Language Change Node

The Accept Language Change node, when reached, immediately:

  1. Updates the active STT language to the detected language
  2. Updates the active TTS voice to the voice configured for that language in the persona
  3. Continues processing the current turn in the new language

Everything after the Accept node in this turn and all subsequent turns uses the new language – until another language change fires.

Step 4 – Reject Language Change Node

The Reject Language Change node, when reached:

  1. Discards the detected language change
  2. The current turn is re-processed in the previously active language
  3. Subsequent turns continue in the active language as before

Use Reject when: the detected language is not supported, the detection was incorrect (false positive), or your design keeps the conversation in a fixed language regardless of user language.

Step 5 – Handling False Positives

Language detection is generally reliable but can produce false positives – especially for:

  • Short utterances (single words are often ambiguous between languages)
  • Proper nouns that look like words in another language
  • Users with strong accents in a language different from their native one

Mitigations:

  • Add a confidence threshold check: If the detected language confidence is low (below a threshold), reject the change. This requires access to the confidence value, which depends on your STT engine's API. Check your STT engine's documentation for confidence score output.
  • Require two consecutive detections: Add a variable counter that increments each time a particular language is detected. Only accept the change after it has been detected twice in a row.
  • Limit to known languages: Only add languages to the profile that you genuinely want to support. If a language is not in the profile's language list, the STT engine will not trigger a language change event for it.

Step 6 – Resetting to the Default Language

If you want to reset the conversation to the default language when a user leaves and a new user starts:

  1. Configure the On conversation-stop handler to reset language-related variables
  2. Set the Defocus state behaviour → Create new conversation on focus toggle in the profile settings – this starts a fresh conversation for each new user, which automatically resets to the default language (the first language in the profile's language list)

Testing

  1. Open the Test conversation panel with Show system messages enabled
  2. Type a message in your default language – verify no language change event fires
  3. Type a message in a supported alternate language – verify you see a Flow Change event for the On Language Change handler, and that the Accept path is taken
  4. Verify subsequent responses are in the new language
  5. Type a message in a language that should be rejected – verify the Reject path is taken and the language stays the same

Further Reading

Use a Custom LLM Endpoint

Route your virtual being's conversations through your own LLM backend using a Custom Endpoint conversational engine.

Build a Multilingual Kiosk

Set up a Virbe Metahuman Kiosk that automatically detects and switches between multiple languages based on what the user speaks.

On this page

How Language Detection WorksStep 1 – Check the Available Language VariableStep 2 – Build a Conditional Language SwitchBuilding the If-Else Router conditionsStep 3 – Accept Language Change NodeStep 4 – Reject Language Change NodeStep 5 – Handling False PositivesStep 6 – Resetting to the Default LanguageTestingFurther Reading