AI Phone Assistant: The Call Is the Easy Part

The misunderstanding

When companies think about an AI phone assistant, the discussion almost always revolves around the call itself: how natural the voice sounds, whether it understands dialect, what happens with follow-up questions. These are legitimate questions, but they are largely solved today – the assistant is a purchased product, and it does its job.

What is not solved: what happens to the concern once the call is over? At a manufacturer with two sites we had exactly this gap. The assistant answered calls, summarised them and sent the summary to a shared mailbox. There it sat. Someone had to decide by hand who was responsible – and that took time.

The architecture of “afterwards”

We built a small microservice that closes precisely this gap. The flow:

1. After every call the phone assistant sends a webhook with transcript and metadata.
2. The service analyses the concern with an LLM: what is it about, which department is affected, how urgent is it, is there a customer or order reference?
3. From the organisational structure – who owns which topic at which site, who deputises for whom – the responsible person is determined.
4. That person receives a Slack direct message with the summary, callback number and what the caller actually wanted.

The whole service is a FastAPI application in Docker Compose. None of it is exotic. The work is in the rules, not the code. Whoever looks at the message in Slack needs less than thirty seconds to decide whether to call back, delegate or file it – and that was the whole point.

The fallback is the real feature

The LLM is good at classifying a concern. It is not perfect, and it must not pretend to be. So there is a second path: if the analysis is uncertain – because the concern touches several departments, because the person cannot be identified unambiguously, or because the transcript does not give enough – the message does not go to a guessed person but into a fallback channel with interactive buttons. There, someone can take over or forward it with one click.

This decision made the difference. A system that is right nine times out of ten and misassigns the tenth will not be accepted. A system that is right nine times and honestly says “I don’t know” on the tenth will.

What we learned about the organisational structure

The responsibility logic was the part that required the most conversations – not with technology, but with people. Questions we had to settle:

• Who is responsible when the responsible person is absent?
• Does responsibility apply across sites or per site?
• What happens outside working hours?
• Who is allowed to take over in the fallback channel?

These rules previously existed only in people’s heads. Writing them down was valuable for the company in itself, independent of the phone assistant.

Hosting behind IIS

The service runs on a Windows Server behind an IIS reverse proxy with HTTPS. That too was a pragmatic choice: the server was there, the certificate was there, IT knew IIS. Docker Compose on Windows with IIS as a proxy in front is not a dream combination, but it works reliably – and a service that does not force IT to operate something new has better odds of survival.

A few things that mattered:

• The webhook endpoint verifies the sender’s signature before processing anything
• Every processing step is logged so misassignments can be traced
• The service is stateless; if it fails, only the current call is lost, not any history

Takeaway

The phone assistant is a product you buy. The routing behind it is the work you have to do yourself – and it consists more of organisational rules than of technology. Write down responsibilities cleanly and allow the system to admit uncertainty, and you get something employees actually use.