We tested AI session takeaways on 10 real talks

AI session summaries are suddenly everywhere. Cvent, Swapcard and a handful of specialist tools now offer them, and if you run events you have probably been pitched one this year. The pitch is always the same: every session, summarized, minutes after it ends.

The question I care about is simpler, and it's the one I'd ask as an organizer: can I put this in front of my attendees without checking every line? If not, the time saved disappears into review.

Gemlo drafts takeaways from session recordings, so I needed an honest answer for our own pipeline. We ran it on ten real talks and graded every line.

How we tested

We picked ten publicly available recorded talks that look like what you would actually see at an event:

• five keynotes, 18 to 40 minutes;
• two panels, one of them 51 minutes with four speakers;
• one interview;
• one 16-minute Q&A session with rapid back-and-forth;
• and one keynote played in a room and recorded on a phone from across the room, to see what a real room mic does to quality.

Each recording went through the same pipeline attendees get: the app records in five-minute segments, each segment is transcribed while the talk is still going, and when the session ends a model drafts the takeaways. We did not tell it how many to write. It decided per session, and produced between 11 and 20.

Then a separate, stronger model graded every takeaway against the full transcript, without seeing how they were made. For each one it checked:

• Grounded: is the supporting quote actually in the transcript?
• Faithful: does the takeaway say only what the speaker said, with no invented facts or numbers?
• Attribution: if it names a speaker, is it the right person?
• Specific: on a 1 to 5 scale, is it a concrete point the speaker made, or generic filler?

We also checked the quotes mechanically, by matching each one against the transcript text.

What held up

Check	Result
Takeaways graded	137 across 10 sessions
Supporting quote found in the transcript	137 of 137
Faithful to what was said	136 of 137
Credited to the wrong speaker	2 of 137
Specificity (1–5)	4.4 average; 129 scored 4 or 5
Coverage of each session's main ideas (1–5)	4 or 5 on all ten

Two results surprised me.

Every quote was real. Each takeaway carries a short line from the talk it is based on, and all 137 of those lines were in the transcript. That matters more than any summary score: it means an attendee, or you, can check any takeaway against what was actually said.

The phone across the room did fine. The room recording scored 4.75 out of 5 on specificity, against 4.85 for the same talk with clean audio, and every one of its quotes was grounded. The transcript was noisier, but the ideas came through, so you don't need a feed from the sound desk.

Where it went wrong

Three kinds of mistake showed up. You will see the same ones in any AI summary tool, so they're worth knowing.

1. Crediting the wrong person on a panel (2 of 137). Both mistakes came from sessions with several voices. In one, a point raised by an unnamed panelist was credited to a named one. In the other, an answer from a guest was credited to the host. Without speaker labels in the transcript, the model had to guess from context, and twice it guessed wrong.

Our instructions already told the model to leave a takeaway uncredited when the speaker wasn't clear. It wasn't enough. Next, we're adding a rule against merging several speakers' points under one name, and a check that drops the credit when the takeaway itself mentions a different speaker. An uncredited takeaway is better than a wrong one.

2. One small embellishment (1 of 137). A speaker gave a weekly number for her whole team. The takeaway added “per person”, which she never said. It's a small, plausible error, the kind that slips past a quick skim.

3. Generic wrap-ups. The lowest-scoring takeaways were almost all summaries of how a session ended: “The speaker closed with a call to action to…”. They were accurate, but nobody needs to keep them. One even summarized the host's wrap-up of the whole conference instead of the keynote.

The bigger lesson: length

The accuracy numbers were better than I expected. The writing was the weaker part.

The median takeaway was 315 characters, about three sentences, and the longest ran past 500. Many read like a report of the talk (“The speaker argues that…”) rather than an idea you would want on a card. Graders still scored them as specific, because they were. But a swipeable takeaway on a phone, or a line in your recap email, needs a headline first and detail second.

The next change is the format: one short sentence stating the idea, then at most two sentences of support. If you are evaluating any AI summary tool, read ten of its outputs on a phone. For each one, ask whether it's true, and then whether you'd want to keep it.

What to ask any vendor

If you are being pitched AI session summaries, these questions sort the good tools from the risky ones quickly:

1. Does every summary point show the line it came from? If not, neither you nor your attendees can check it.
2. How does it handle panels? Ask to see output from a multi-speaker session. Watch the speaker credits.
3. Can you review before attendees see it? And can you turn review off for sessions where speed matters more?
4. How soon after a session do attendees get it? Most forgetting happens in the first day. Summaries that land next week are fine as a recap, but they arrive after most of it is gone.
5. What does it need in the room? A sound-desk feed and a technician per track is a very different cost from a phone on a stand.
6. Who can see what attendees save? If attendees can keep and annotate takeaways, those notes should be theirs alone.

How Gemlo handles it

For the record, this is where Gemlo stands on those questions today. Every takeaway shows its source quote. A phone in the room is enough, and takeaways usually arrive within a couple of minutes of a session ending. Organizers choose per event whether takeaways release automatically or wait for review. Attendees keep the ones that matter and can add a private note; organizers and sponsors never see what an attendee kept or wrote.

We'll keep publishing test results like these as the pipeline changes, including the parts that went wrong.