Building AI That Speaks Real Spanish

The Problem: Spanish Isn't Just Spanish

When we started building Kleva, we quickly learned that "Spanish-speaking AI" isn't enough for Latin America. A Mexican from Monterrey speaks completely differently than someone from Buenos Aires or Santiago.

Most voice AI systems are trained on "neutral" Spanish—usually from Spain or dubbed movies. Try using that to collect debt in Guatemala, and you'll get laughed at... then hung up on.

The Dialect Challenge

Our Technical Solution

1. Multi-Model Architecture

Instead of one Spanish model, we built 7 country-specific models:

• Base Model: 100M parameters trained on 50,000 hours of LATAM audio
• Country Adapters: 10M parameters each, fine-tuned on local speech
• Context Switching: Real-time model swapping based on caller location

2. Slang and Regional Vocabulary

We built custom dictionaries with 15,000+ regional terms:

# Chilean Slang Mappings
"plata" → money
"lucas" → thousand pesos
"cachai" → understand
"po" → emphasis particle
"weon" → dude/person
"fome" → boring/bad

# Mexican Formality Levels
FORMAL: "Usted", "Le agradecería"
NEUTRAL: "Tú", "Por favor"
INFORMAL: "Tú", "Porfa"

3. Prosody and Intonation

Each country has unique speech patterns we had to model:

• Mexico: Melodic, rising intonation on questions
• Argentina: Italian-influenced rhythm, emphatic stress
• Chile: Fast pace, dropped syllables
• Colombia: Clear articulation, varied pitch

The Training Process

Data Collection Hell

We needed real conversations, not acted speeches. Our dataset:

• 2.5 million call recordings (anonymized)
• 450 native speakers from 23 countries
• 87 different Spanish dialects
• Age range: 18-75 (speech patterns vary by generation)

The Annotation Marathon

We hired 200 linguists across LATAM to annotate:

• Emotion (angry, frustrated, confused, willing)
• Formality level (1-5 scale)
• Regional markers (specific words/phrases)
• Code-switching (Spanish/English mix)

Unexpected Challenges

1. Background Noise Reality

Lab-quality audio doesn't exist in real collections. We deal with:

• Street vendors yelling
• Roosters (surprisingly common)
• Bad cellular connections
• Multiple people talking
• TV/Radio in background

Solution: Trained noise separation model on 10,000 hours of LATAM-specific background noise.

2. Code-Switching

Many debtors mix Spanish and English, especially in Mexico:

"No tengo cash ahorita, but I can hacer un payment next week, ¿está bien?"

Our model had to understand and respond appropriately to mixed-language input.

3. Cultural Taboos

What's polite in one country is rude in another:

• Never use "tú" with elderly Mexicans (always "usted")
• Don't be too direct with Colombians (considered aggressive)
• Argentinians appreciate directness (beating around bush is suspicious)
• Chileans hate overly formal speech (sounds condescending)

The Results

What's Next: Portuguese and Beyond

We're applying the same approach to Brazilian Portuguese (spoiler: São Paulo and Rio are like different languages). Next up: Colombian Caribbean coast vs Bogotá, and the eternal challenge of understanding Chileans when they're speaking fast.

The key lesson? There's no such thing as "Spanish AI" for LATAM. You need AI that speaks Mexican, Argentine, Chilean, Colombian... and knows when to switch between them.