The geeky details

How KindPhone works under the hood - and why

This is a bit technical. If you just want to know what KindPhone does, the front page covers it.

The short version: KindPhone answers the phone when a family carer cannot, and holds a warm, natural conversation with an elderly and sometimes confused caller. Your phone number forwards unanswered calls to Twilio (a phone company), which turns the caller's speech into text. My servers send the text to an AI (Claude) to generate a reply, and then safety check what comes back against a flowchart which defines what should/shouldn't happen within the call. The text goes back to Twilio to play as voice audio. After each call you get an email with a recording and the full transcript.

I've tried lots of ways of doing this. It's hard. There are five things we'd like to have: private, cheap, fast, realistic, and safe. So there were various trade-offs to explore. It turns out we can't have them all without spending more than I can afford at the moment.

Design goals

Here's what we'd like:

Cost. Affordable for ordinary families. £10-£20/month?
Speed. Replies within about two seconds, or the call feels broken.
Realism of conversation. It should feel like talking to a warm, patient person, not a machine.
Safety. A caller who believes what she is told must never be talked into harm.
Data privacy. Calls from someone's elderly mother are pretty personal data.

But these aims conflict with each other:

Speed needs either expensive hardware or expensive cloud services.
Realism needs the most capable AI models, which are run by third parties - not great for privacy.
Safety checks reject some replies and substitute scripted lines - so it's less realistic.
Cutting cost with smaller AI models raises the rate of dangerous mistakes: my measurements kept showing that cheaper models fail safety tests more often.
The privacy ideal, of everything on affordable servers I control, is really hard to do with adequate realism, speed and speech quality.

The story below is what happened as I worked out, mostly by measuring, which of these could be traded and which could not.

Why this is a hard problem

There are a few interesting and well-established problems in handling phone audio.

Telephone audio is awful. Landlines deliver narrow, crackly 8kHz audio. The five speech recognisers I tested on my own servers all scored around 40% word error rate on real elderly telephone speech; it is the audio as much as the model. The big cloud recognisers are better: the ElevenLabs version used one for seven months and it understood the caller noticeably better than anything I can run myself.
Pauses do not mean someone's finished. Someone searching for a word pauses mid-sentence. Standard voice systems hear silence and butt in. Deciding whether a pause means "finished" or "thinking" is still a problem the products I have tested have not solved perfectly.
The system hears itself. The system's own voice comes back down the phone line as echo, and it is not noise: it is clear speech that transcribes as real words.
Mistakes are not equal. If a shopping assistant says something odd, you shrug. If a reassuring voice tells a confused caller that a scalding bath sounds lovely, that is harm. The system has to be more cautious.

Round 1: let the AI just talk (July 2025)

The very first version, built in a weekend when I was at the end of my tether, did the obvious thing: connect the phone line to a cloud AI that listens and speaks in real time (OpenAI's live voice API, then ElevenLabs' agent platform).

The ElevenLabs version is genuinely good. It ran real calls for my moither for about seven months, structured as a flowchart of small AI prompts, one per topic, and it produced the recorded call history that most later experiments were tested against.

But it just costs too much for most people - about £200/month all in. Ouch. And it's a bit riskier than I'd like in terms of how it replies. I wouldn't want to offer that as a service to other people.

Round 2: nothing unscripted (February 2026)

So I swung to the opposite extreme: no AI ever speaks to the caller. Every word was written, approved, recorded in advance, and played back. AI still worked behind the scenes, transcribing the caller on my own servers and deciding which approved recording answered her, but it could never compose speech.

The machinery under this went through several generations. Speech recognition went from Vosk (fine at detecting speech, poor at transcribing it) to Whisper on a rented GPU, to Moonshine on a plain 4-core server after I decided the GPU's monthly cost had to go. Understanding the caller went from keyword matching (too brittle), to a local AI classifier (right answers, too slow), to sentence vector embeddings so that sentences with similar meanings get similar numbers.

I then spent three months building increasingly clever selection machinery to make the recordings feel less canned: libraries of phrasing variants, penalties for recently-played lines (in replays of past calls, the worst-case repetition fell by 75%), even a layer that chose what kind of response fitted the moment.

This was safe, affordable, and private. Nothing unapproved could ever be spoken, by construction. But it's just too stilted and unrealistic. It gets stuck playing the same recording, or similar ones, when you can hear the caller is getting frustrated. It's too close to the "press 1 to get annoyed" voice systems we all hate.

Round 3: small AI on my own hardware (spring 2026)

Could a small AI model on my own hardware write good replies with no cloud involved?

On affordable hardware, every model I tested missed the two-second budget, most by a lot. A rented GPU fixes the speed for roughly £100 or more a month. One GPU could probably serve several households, since calls would rarely overlap, but it is still the largest cost in the system by a distance. The genuinely cheap GPU market is mostly offshore, which conflicts with keeping call data in the UK.
I distilled the decisions of a big model into a tiny 2MB classifier that runs in under a millisecond. For deciding things (which topic is this, what kind of reply fits) it beat my hand-written rules, and the same technique (a small model trained to copy a bigger model's judgments) is how the safety check was built.

But writing warm sentences is a different job, and small models kept failing my safety checks too often. For composing safe, warm speech to a confused elderly caller, the models that write safely enough are the biggest ones, and the biggest ones only run in the cloud. So the question became where the AI runs and on what terms.

Round 4: the AI composes, my code enforces (July 2026)

The current design keeps the family-designed flowchart from Round 2 and the fluent language from Round 1, and adds rules my own code enforces on top of the instructions the AI is asked to follow:

The AI must follow the flowchart. Each turn, the model names which flowchart step it wants to move to. If that move is not allowed from where it stands, my code refuses it. I measured what happens without this: models navigating freely made illegal moves on roughly a third of turns. With enforcement an illegal move cannot happen: the model still asks for one on a few per cent of turns, and my code refuses each one and keeps the conversation where it was.
Every reply is checked as text before it becomes speech. Because the AI produces text and the reply is only turned into speech afterwards, there is always a moment where the words exist and have not been heard. Safety checks run at that point on my own server. One catches replies that endorse or smooth over something risky the caller just said. Another re-reads the last few turns, because the risky thing is often not in the latest sentence: when she mentions a worrying plan, chats about the weather, then says "so I'll do that then, shall I?", a bare "yes, that's a great idea" is the danger. In my tests it caught 94 to 100% of dangers of kinds it was never trained on, though those test sets are small. When a check trips, she hears a fallback bland response instead.
Actual for stand-ins. Real names are replaced before any text leaves my server and swapped back in the reply, mechanically, every turn. This is so easy to do that we might as well, though there are other things that might be said we are private. The bottom line here is that we are trusting the AI companies' policy on privacy. We'd prefer not to, but I can't achieve that yet.

Which AI, then?

I've tried various options and providers here, too many to mention. Open-weight models, frontier models, small providers, large providers. By the time you read this the landscape will have changed, but there were pros and cons, and for most, the cons won.

The best option I settled on was Anthropic's Claude, using AWS' European region. This has good privacy: nothing retained after the call, never used for training, and a design AWS documents as giving its own staff no technical path to read what passes through. On the same safety tests it beat every other model. Amazon and Anthropic are American companies, so this rests on contracts and technical architecture rather than European jurisdiction, and it costs more per call than European providers. But it's just about affordable.

Which voice?

I've experimented with different voices. I think having a familiar regional accent helps, and using a different gender from the person they're trying to reach reduces confusion. ElevenLabs has a great selection of voices - but it's too expensive.

Most providers have a basic selection of British female voices, which is what I was looking for.

Voice-cloning: some providers let you clone your own voice. This is morally dubious - you're fooling the caller - and gets a bit "uncanny valley". Not an option.

Round 5: sigh; just let Twilio handle the calls (August 2026)

There are several problems Twilio handles better than I managed to do.

Knowing when someone has finished speaking is genuinely hard, especially for a hesitant speaker who pauses mid-sentence to gather a thought. Wait politely through every pause and the whole conversation feels slow; jump in quickly and you cut her off mid-thought.
Add the quality of telephone audio and quiet elderly voices, and this gets harder.
When using Twilio to front the calls, I battled with some calls where the audio simply didn't arrive. Presumably this is my fault, but I never got to the bottom of it.

At this point, it's pretty clear I can't have everything I would like. But I've still got something that is excellent for me, and might help other people. So it's time to stop faffing, get it out there, and wait for more options to become possible in future.

So this is where we are:

What next?

I'd like to bring it all in house for privacy. That needs significantly better hardware than I'm likely to be able to afford, and better open-weight models. This might become possible in a couple of years.

Realistic voice conversations continue to develop, especially systems which both talk and listen at the same time. Those will probably become the norm, but hosting them yourself is likely to take quite a while.

If any of this is your kind of problem, or you think I have got something wrong, I would genuinely like to hear about it. Get in touch.