ConvoKit AI Evaluation

ConvoKit AI Evaluation (the convokit.ai_evaluation package) provides transformers that evaluate mediated multi-party conversations, such as discussions moderated by an AI facilitator:

  • CIG (conversational information gain): an LLM rates each utterance from 1 to 4 for how much it adds to what the group already knew.

  • ClaimTracker: tracks the atomic claims each speaker holds through the conversation, then measures how far the participants converge and whether their final claims came from the mediator or from another participant.

  • MediatorRedirection: for each mediator turn, a language model estimates how much it changed the participant reply that followed, using Redirection.

Example usage: CIG demo, ClaimTracker demo, MediatorRedirection demo.

Installation

ConvoKit AI Evaluation is part of ConvoKit. CIG and ClaimTracker need ConvoKit’s GenAI dependencies:

pip install "convokit[genai]"

Then import the transformers from convokit.ai_evaluation:

from convokit.ai_evaluation import CIG, ClaimTracker, MediatorRedirection

CIG and ClaimTracker call an OpenAI or a Gemini model through convokit.genai, chosen by the model argument: gpt-* models (the default, gpt-5 for CIG and gpt-5-mini for ClaimTracker) need an OpenAI API key, and gemini-* models need a Gemini API key. Set the key with the GPT_API_KEY / GEMINI_API_KEY environment variable or with GenAIConfigManager (config_manager.set_api_key("gemini", "...")); Gemini can also run through Vertex AI (see GenAI). Scores from different models are not directly comparable: the prompts were written and checked with GPT-5 models, so use one model throughout a study. ClaimTracker also downloads a sentence embedding model and an NLI model from Hugging Face on first use. MediatorRedirection needs ConvoKit’s optional LLM dependencies (pip install convokit[llm]) and a Hugging Face token for the Gemma model.

Input format

The transformers read the AI fields of the corpus:

  • The mediator is the speaker whose ai_meta["role"] is "public assistant". All other speakers are participants.

  • Speaker names come from conversation.alias, so the prompts show messages under the names participants used in the chat.

  • The topic shown to the models is conversation.meta["topic"], followed by conversation.meta["statement"] if there is one.

  • Starting positions for ClaimTracker come from conversation.meta["pre_survey"], a dict from speaker ID to the text of that participant’s position before the conversation.

Corpora produced by ConvoKit AI Simulation already have the mediator role and names. For other corpora, each transformer takes functions that replace these defaults: is_mediator (Speaker → bool), name_func ((Conversation, Speaker) → str), topic_func (Conversation → str) and, for ClaimTracker, seed_text_func ((Conversation, Speaker) → str). For example, to treat both AI and human facilitators as mediators:

from convokit.ai_evaluation import CIG

cig = CIG(
    model="gpt-5",
    is_mediator=lambda speaker: speaker.meta.get("game_role") == "facilitator",
    topic_func=lambda convo: "Choosing a host city for a sporting event",
)
cig.transform(corpus)
cig.summarize(corpus)  # mean CIG per conversation, mediator vs participants

To use Gemini instead, pass a Gemini model name (the provider is inferred from it, or can be given with provider="gemini"):

from convokit.ai_evaluation import CIG, ClaimTracker

cig = CIG(model="gemini-2.5-flash")
tracker = ClaimTracker(model="gemini-2.5-flash")

CIG

Conversational Information Gain (CIG): a 1-4 score for each utterance indicating how much it added to what the group already knew (1 = nothing new, 4 = a new idea or reframing).

An LLM reads the whole conversation once and rates every utterance against what came before it. Two kinds of utterances are left unrated (score None):

  • the opening exchange, up to and including the first utterance by which every speaker has spoken at least once: there is nothing before it to add to, so it is shown to the LLM as context only;

  • fragments: 3 words or fewer, or 4-5 words without a final ., ! or ? (e.g. “ok!”, “yeah that makes sense”).

Reference: CIG: Measuring Conversational Information Gain in Deliberative Dialogues with Semantic Memory Dynamics, https://aclanthology.org/2026.acl-long.2203/

class convokit.ai_evaluation.cig.CIG(llm_client=None, model: str = 'gpt-5', provider: Optional[str] = None, config_manager: Optional[convokit.genai.genai_config.GenAIConfigManager] = None, reasoning_effort: str = 'minimal', custom_prompt: Optional[str] = None, cig_attribute_name: str = 'cig', is_mediator: Callable[[convokit.model.speaker.Speaker], bool] = <function default_is_mediator>, name_func: Callable[[convokit.model.conversation.Conversation, convokit.model.speaker.Speaker], str] = <function default_speaker_name>, topic_func: Callable[[convokit.model.conversation.Conversation], str] = <function default_topic>, n_workers: int = 1, verbosity: int = 10)

ConvoKit Transformer that rates every utterance of a conversation for conversational information gain (CIG) with an LLM, and stores the rating in utt.meta["cig"] (None where not rated).

Each utterance gets a score from 1 (adds nothing new) to 4 (a new idea or reframing). The opening exchange (up to and including the first utterance by which every speaker has spoken) is shown to the LLM as context only, and fragments (see is_fragment) are not rated.

Usage:

cig = CIG(model="gpt-5")
cig.transform(corpus)
cig.summarize(corpus)   # mean score per conversation, mediator vs participants

The corpus must identify the mediator and the name each speaker goes by in the chat. By default these come from the ConvoKit AI fields (speaker.ai_meta["role"] and conversation.alias); they can be replaced with is_mediator and name_func.

Reference: CIG: Measuring Conversational Information Gain in Deliberative Dialogues with Semantic Memory Dynamics, https://aclanthology.org/2026.acl-long.2203/

Parameters

  • llm_client – client with a generate_json(prompt) method; by default one for model built with get_eval_llm_client

  • model – model name for the default client: an OpenAI model (e.g. "gpt-5") or a Gemini model (e.g. "gemini-2.5-flash")

  • provider – "gpt" or "gemini" for the default client; inferred from model if not given

  • config_manager – GenAIConfigManager holding the API key for the default client

  • reasoning_effort – OpenAI reasoning effort for the default client (not used for Gemini)

  • custom_prompt – prompt template replacing prompts/cig_prompt.txt; must contain the {topic}, {context}, {target}, {start}, {end} and {total} placeholders

  • cig_attribute_name – name of the utterance metadata attribute to store the rating in

  • is_mediator – function from Speaker to whether it is the mediator (shown to the LLM as Mediator(mod))

  • name_func – function from (Conversation, Speaker) to the name shown for a participant

  • topic_func – function from Conversation to the topic line shown to the model

  • n_workers – number of conversations to rate concurrently

  • verbosity – print progress every verbosity conversations (0 to disable)

build_prompt(conversation: convokit.model.conversation.Conversation) → Tuple[Optional[str], List[int]]

Build the rating prompt for a conversation.

Parameters

conversation – the Conversation to rate

Returns

the prompt (None if the conversation has nothing to rate) and the positions of the utterances to rate, in chronological order

rating_targets(utts: List[convokit.model.utterance.Utterance]) → Tuple[List[bool], List[bool]]

Split a conversation’s utterances into the opening context batch and the rating targets.

Parameters

utts – the conversation’s utterances, in chronological order

Returns

two lists of booleans parallel to utts: whether each utterance is in the opening context batch, and whether it is to be rated (both empty if there are no utterances)

summarize(corpus: convokit.model.corpus.Corpus, selector: Callable[[convokit.model.conversation.Conversation], bool] = <function CIG.<lambda>>, speaker_selector: Callable[[convokit.model.speaker.Speaker], bool] = <function CIG.<lambda>>, min_mediator_utts: int = 2) → pandas.DataFrame

Compute the mean rating of the mediator’s utterances and of the participants’ utterances, per conversation.

Parameters

  • corpus – a Corpus that has been transformed

  • selector – function from Conversation to whether it should be included

  • speaker_selector – function from Speaker to whether its utterances count

  • min_mediator_utts – skip conversations where the mediator spoke fewer times than this (0 to keep all; conversations without a mediator are kept)

Returns

DataFrame with one row per (conversation, is_mediator) and columns conversation_id, is_mediator, n_rated and cig_mean

transform(corpus: convokit.model.corpus.Corpus, selector: Callable[[convokit.model.conversation.Conversation], bool] = <function CIG.<lambda>>)

Rate the utterances of the selected conversations and store each rating in utt.meta[cig_attribute_name] (None for utterances that are not rated).

Conversations whose rating fails are reported and left with None ratings.

Parameters

  • corpus – the Corpus to transform

  • selector – function from Conversation to whether it should be rated

Returns

the Corpus, with ratings added

convokit.ai_evaluation.cig.is_fragment(text: str) → bool

Check whether an utterance is too short to rate: at most 3 words, or at most 5 words without terminal punctuation (., ! or ?).

Parameters

text – the utterance text

Returns

True if the utterance is a fragment

ClaimTracker

Claim-level consensus: the propositions each participant holds before and after a mediated conversation, and the extent to which they come to be shared.

Pipeline

Each speaker, including the mediator, is assigned a memory of atomic claims (short, self-contained propositions, e.g. “Data centers use a lot of water.”). Utterances are processed in chronological order:

  1. Extraction. An LLM decomposes the utterance into atomic claims (claim_extraction_prompt), given the preceding context_window utterances for pronoun and reference resolution. Each claim is typed own (the speaker’s position), reported (attributed to someone else) or procedural (about the conversation itself). Claims extracted from the same utterance that mutually entail each other under a DeBERTa NLI cross-encoder (probability >= sibling_entail in both directions) are collapsed into one.

  2. Retrieval. Candidate entries are the speaker’s own active claims (all of them, or the nearest own_cap) and the top_k nearest active claims of every other speaker, by cosine similarity of sentence embeddings.

  3. Consolidation. An LLM (claim_consolidation_prompt) returns, for each new claim, its logical relation to one candidate (equivalent, forward_entail, backward_entail, contradiction or neutral), the candidate’s id, a merged_claim when the new claim refines the candidate, and supersedes when it contradicts the speaker’s own earlier claim.

  4. Memory update. The relation is mapped deterministically to an operation; the model never chooses the operation, and no entry is ever deleted:

    relation (to the chosen target)

    operation

    neutral, or no valid target

    ADD a new entry

    any relation, other speaker

    ADD a new entry with a link to the other speaker’s claim

    equivalent / backward_entail

    RESTATE: log the restatement on the existing entry

    forward_entail

    REFINE: replace the text with merged_claim, keep the previous text in the entry’s history

    contradiction, supersedes=true

    SUPERSEDE: mark the old entry retracted and link the new entry to it

    contradiction, supersedes=false

    ADD a new entry with a contradiction link (both held)

    Rows are checked from top to bottom; the other rows apply when the target is one of the speaker’s own claims. Every entry records first_turn, last_turn, history, restated, links and status; a speaker’s held claims are their active own entries.

Participants’ memories may be seeded from a pre-survey statement (seed_text_func) to establish their initial position.

Measurement

After the last utterance, for each pair of participants, an LLM (claim_agreement_prompt) labels pairs of claims across the two participants’ lists as agree or contradiction, once on the seed lists and once on the final held lists. A pair’s agreement is the mean, over its two participants, of the share of their claims with an agreeing counterpart; the conversation’s agreement is the mean over participant pairs. Per conversation:

  • agreement_gain = (final - initial) / (1 - initial): the fraction of the remaining disagreement closed during the conversation (negative if agreement decreased; undefined if the initial agreement is already 1).

Per participant, using the links recorded at consolidation time to find which speaker stated a claim first (origin_of):

  • share_from_mediator: share of the participant’s final claims first stated by the mediator;

  • share_from_partner: share first stated by another participant;

  • share_new_common_ground: of the claims newly shared with a partner at the end (not shared at the start), the share first stated by the mediator.

Reference: CIG: Measuring Conversational Information Gain in Deliberative Dialogues with Semantic Memory Dynamics, https://aclanthology.org/2026.acl-long.2203/

class convokit.ai_evaluation.claims.ClaimTracker(llm_client=None, model: str = 'gpt-5-mini', provider: Optional[str] = None, config_manager: Optional[convokit.genai.genai_config.GenAIConfigManager] = None, reasoning_effort: str = 'minimal', seed: int = 0, embedder=None, embedding_model: str = 'Qwen/Qwen3-Embedding-0.6B', nli=None, nli_model: str = 'cross-encoder/nli-deberta-v3-base', device: Optional[str] = None, top_k: int = 5, own_cap: int = 40, context_window: int = 3, sibling_entail: float = 0.5, is_mediator: Callable[[convokit.model.speaker.Speaker], bool] = <function default_is_mediator>, name_func: Callable[[convokit.model.conversation.Conversation, convokit.model.speaker.Speaker], str] = <function default_speaker_name>, topic_func: Callable[[convokit.model.conversation.Conversation], str] = <function default_topic>, seed_text_func: Optional[Callable[[convokit.model.conversation.Conversation, convokit.model.speaker.Speaker], Optional[str]]] = <function pre_survey_text>, custom_prompts: Optional[Dict[str, str]] = None, claims_attribute_name: str = 'claims', n_workers: int = 1, verbosity: int = 10)

ConvoKit Transformer that tracks the claims of every speaker in each conversation and measures the consensus the participants reach.

Each utterance is decomposed by an LLM into atomic claims, which are consolidated into a per-speaker claim memory (added, restated, refined or superseded). At the end, an LLM judges which claims the participants agree on before and after the conversation, and the claim links are used to find whether a participant’s claims first came from the mediator or from another participant. See the module docstring for the full method.

Usage:

tracker = ClaimTracker(model="gpt-5-mini")
tracker.transform(corpus)
tracker.summarize(corpus)   # one row per participant per conversation

What gets written to the corpus:

  • convo.meta["claims"]: everything the tracker produced for the conversation; the keys are initial and final (each participant’s claim list before and after), memory (every claim entry with its history and links), steps (what each utterance extracted and how it was recorded) and agreement (the judged pairs and the gain).

  • per participant, via corpus.get_speaker_convo_info(speaker_id, convo_id): claims_initial, claims_final, claims_share_from_mediator, claims_share_from_partner, claims_share_new_common_ground.

The corpus must identify the mediator and the name each speaker goes by in the chat. By default these come from the ConvoKit AI fields (speaker.ai_meta["role"] and conversation.alias); they can be replaced with is_mediator and name_func.

Reference: CIG: Measuring Conversational Information Gain in Deliberative Dialogues with Semantic Memory Dynamics, https://aclanthology.org/2026.acl-long.2203/

Parameters

  • llm_client – client with a generate_json(prompt) method; by default one for model in JSON mode with a fixed seed, built with get_eval_llm_client

  • model – model name for the default client: an OpenAI model (e.g. "gpt-5-mini") or a Gemini model (e.g. "gemini-2.5-flash")

  • provider – "gpt" or "gemini" for the default client; inferred from model if not given

  • config_manager – GenAIConfigManager holding the API key for the default client

  • reasoning_effort – OpenAI reasoning effort for the default client (not used for Gemini)

  • seed – seed for the default client

  • embedder – a SentenceTransformer used for retrieval (default: embedding_model)

  • embedding_model – name of the embedding model to load if embedder is not given (Qwen3 needs transformers >= 4.51)

  • nli – a sentence_transformers CrossEncoder whose label 1 is entailment, used to collapse near-paraphrases extracted from one utterance (default: nli_model)

  • nli_model – name of the NLI model to load if nli is not given

  • device – torch device for the default embedder and NLI model (default: "mps" if available, else "cpu")

  • top_k – number of nearest claims of each other speaker shown to the consolidation LLM

  • own_cap – if the speaker has more active claims than this, only roughly this many of the nearest ones are shown to the consolidation LLM

  • context_window – number of preceding utterances shown to the extraction LLM

  • sibling_entail – mutual entailment probability from which two claims extracted from one utterance count as the same claim

  • is_mediator – function from Speaker to whether it is the mediator

  • name_func – function from (Conversation, Speaker) to the name used for a participant

  • topic_func – function from Conversation to the topic line shown to the model

  • seed_text_func – function from (Conversation, Speaker) to the text a participant’s memory is seeded from (by default conversation.meta["pre_survey"][speaker.id]), or None for no seeding

  • custom_prompts – dict replacing any of the prompt files, with keys "extraction", "consolidation" and "agreement"

  • claims_attribute_name – name of the conversation metadata attribute to store results in

  • n_workers – number of conversations to track concurrently

  • verbosity – print progress every verbosity conversations (0 to disable)

static active(memory: List[dict], speaker: Optional[str] = None) → List[dict]

Get the active (not retracted) entries of the memory.

Parameters

  • memory – the memory

  • speaker – if given, only this speaker’s entries are returned

Returns

list of active entries

classmethod apply_updates(memory: List[dict], decisions: List[dict], new_claims: List[dict], turn) → List[dict]

Apply the operations to the memory, in place.

Parameters

  • memory – the memory

  • decisions – operations parallel to new_claims (see repair_update)

  • new_claims – the claims just extracted

  • turn – the turn at which the claims were made

Returns

the entries that were created or whose text changed

claim_agreement(claims_a: List[str], claims_b: List[str], name_a: str, name_b: str)

Judge with the LLM which claims of two lists agree or contradict each other.

Parameters

  • claims_a – the first participant’s claims

  • claims_b – the second participant’s claims

  • name_a – the first participant’s name

  • name_b – the second participant’s name

Returns

dict with pairs (the judged pairs, by index into each list) and, for agree and contradiction, the mean over both sides of the share of claims that have a counterpart with that relation (None if either list is empty)

collapse_siblings(claims: List[dict]) → List[dict]

Collapse claims of the same kind that entail each other (see sibling_entail) into the first of them, keeping the texts of the others in its siblings.

Parameters

claims – claims extracted from one utterance

Returns

the remaining claims, each with a siblings list

common_ground_shares(record: dict, agreement: dict) → Dict[str, dict]

Compute, per participant, the share of their newly shared claims that came from the mediator.

Newly shared claims are those the participant agrees on with a partner at the end but not at the start; a claim counts as coming from the mediator if the earliest origin over the claim and the partner claims paired with it is the mediator.

Parameters

  • record – the output of track

  • agreement – the output of judge_agreement

Returns

dict of participant name -> n_new_shared and share_new_common_ground (None if there are no newly shared claims)

consolidate(shown: List[dict], new_claims: List[dict]) → List[dict]

Ask the LLM for the relation of each new claim to one of the shown candidates.

Parameters

  • shown – the candidate entries

  • new_claims – the claims just extracted

Returns

the LLM’s list of memory updates (unvalidated)

classmethod decide(updates: List[dict], shown: Dict[str, dict], new_claims: List[dict], speaker: str) → List[dict]

Pair each new claim with the LLM’s output for it (by claim text, else by position) and turn it into a validated operation.

Parameters

  • updates – the LLM’s memory updates

  • shown – dict of id -> candidate entry shown to the LLM

  • new_claims – the claims just extracted

  • speaker – the name of the speaker of the new claims

Returns

list of operations parallel to new_claims (see repair_update)

extract_claims(context: str, target: str, speaker: str, turn) → List[dict]

Extract the atomic claims of one utterance with the LLM, with near-paraphrases collapsed into one claim (the others are kept as its siblings). The speaker is taken from the transcript, not from the LLM.

Parameters

  • context – the preceding utterances (or the topic line), as text

  • target – the utterance to extract claims from, as a transcript line

  • speaker – the name of the utterance’s speaker

  • turn – the utterance’s position in the conversation, or "pre-survey"

Returns

list of claim dicts with speaker, target_speaker, claim, kind, about, turn_id and siblings

classmethod held_entries(memory: List[dict], speaker: str, kinds=('own',)) → List[dict]

Get the claims a speaker holds: their active entries of the given kinds.

Parameters

  • memory – the memory

  • speaker – the speaker’s name

  • kinds – the claim kinds to include

Returns

list of entries

judge_agreement(record: dict) → dict

Compute the agreement between every pair of participants, before (seed claims) and after (held claims) the conversation.

Parameters

record – the output of track

Returns

dict with pairs (per participant pair, the claim_agreement results for initial and final), the means over pairs initial, final, contradiction_initial and contradiction_final, and gain, (final - initial) / (1 - initial) (None if undefined)

static new_entry(memory: List[dict], claim: dict, turn) → dict

Create a memory entry for a claim (without adding it to the memory).

Parameters

  • memory – the current memory, used to number the entry

  • claim – the claim dict

  • turn – the turn at which the claim was made

Returns

the new entry

classmethod origin_of(entry: dict, by_id: Dict[str, dict], names: List[str], seen=frozenset({}))

Find who held a claim first, following its equivalent and forward_entail links.

A claim the mediator reports on behalf of a participant is attributed to that participant.

Parameters

  • entry – the memory entry

  • by_id – dict of id -> entry for the whole memory

  • names – the participants’ names

  • seen – ids already visited (to avoid cycles)

Returns

(speaker name, turn) of the earliest origin, with the pre-survey as turn -1

static repair_update(update: dict, shown: Dict[str, dict], speaker: str) → dict

Turn one LLM output into a validated operation (see the table in the module docstring).

Parameters

  • update – one memory update from the LLM

  • shown – dict of id -> candidate entry shown to the LLM

  • speaker – the name of the speaker of the new claim

Returns

dict with op (ADD, RESTATE, REFINE or SUPERSEDE), relation, target (an entry id or None) and, for REFINE, text

retrieve_candidates(index: convokit.ai_evaluation.claims._VectorIndex, memory: List[dict], speaker: str, new_claims: List[dict]) → List[dict]

Retrieve the memory entries shown to the consolidation LLM: the speaker’s own active claims (all of them, or about own_cap of the nearest ones), followed by the top_k nearest active claims of every other speaker, for any of the new claims. Retracted claims are never shown.

Parameters

  • index – the embedding index of the memory

  • memory – the memory

  • speaker – the name of the speaker of the new claims

  • new_claims – the claims just extracted

Returns

list of candidate entries

source_shares(record: dict) → Dict[str, dict]

Count, per participant, how many of their held claims were first held by the mediator or by another participant.

Parameters

record – the output of track

Returns

dict of participant name -> n_final, n_from_mediator, n_from_partner, share_from_mediator and share_from_partner (shares are None if the participant holds no claims)

summarize(corpus: convokit.model.corpus.Corpus, selector: Callable[[convokit.model.conversation.Conversation], bool] = <function ClaimTracker.<lambda>>) → pandas.DataFrame

Collect the per-participant results in one table.

Parameters

  • corpus – a Corpus that has been transformed

  • selector – function from Conversation to whether it should be included

Returns

DataFrame with one row per (conversation, participant) and columns conversation_id, speaker (the speaker id), n_initial, n_final, agreement_gain (the conversation’s), share_from_mediator, share_from_partner and share_new_common_ground

track(conversation: convokit.model.conversation.Conversation) → dict

Build the claim memory of one conversation.

Parameters

conversation – the Conversation

Returns

dict with speakers (participant name -> speaker id), initial and final (name -> held claims), memory (all entries) and steps (what each utterance extracted, was shown, returned and decided)

transform(corpus: convokit.model.corpus.Corpus, selector: Callable[[convokit.model.conversation.Conversation], bool] = <function ClaimTracker.<lambda>>)

Track the claims of the selected conversations and store the results (see the class docstring).

Conversations whose tracking fails are reported and left without results.

Parameters

  • corpus – the Corpus to transform

  • selector – function from Conversation to whether it should be tracked

Returns

the Corpus, with results added

MediatorRedirection

Mediator redirection: for each mediator turn, did it change what the participants said next?

The score comes from a language model’s probabilities. Take a mediator turn T_k and the participant reply that followed it. Ask the model how likely that reply was given T_k, and how likely it was given a stand-in instead (the mediator’s previous turn, or nothing). The redirection score is the log-odds difference: positive means the reply was more expected after T_k than after the stand-in, i.e. the turn steered the conversation; near zero means the reply would have come anyway.

This is convokit.redirection (Nguyen et al., 2024) with context selectors for one mediator and several participants instead of two alternating speakers:

actual     [participant turn before T_k, T_k]      -> reply
reference  [participant turn before T_k, T_(k-1)]  -> same reply   (counterfactual "prev")
           [participant turn before T_k]           -> same reply   (counterfactual "delete")

Each participant line is prefixed with the participant’s name in the chat (e.g. “Goose: …”), the same name the mediator uses to address them; mediator lines are prefixed with “mediator: “.

Reference: Taking a turn for the better: Conversation redirection throughout the course of mental-health therapy, https://aclanthology.org/2024.findings-emnlp.555/

class convokit.ai_evaluation.redirection.MediatorRedirection(likelihood_model, counterfactual: str = 'prev', is_mediator: Callable[[convokit.model.speaker.Speaker], bool] = <function default_is_mediator>, name_func: Callable[[convokit.model.conversation.Conversation, convokit.model.speaker.Speaker], str] = <function default_speaker_name>, redirection_attribute_name: str = 'redirection')

ConvoKit Transformer that scores each mediator turn by how much it redirected the participant reply that followed it, and stores the score in utt.meta["redirection"].

The score compares how likely a language model finds the participant reply after the mediator turn versus after a counterfactual context (the mediator’s previous turn, or no mediator turn), as a log-odds difference: positive values mean the turn made the reply more expected. A mediator turn is scored only if it is followed by a participant reply, has a participant turn before it, does not directly follow another mediator turn, and (with the "prev" counterfactual) has an earlier mediator turn to compare against (see mediator_previous_context_selector).

Usage:

likelihood_model = GemmaLikelihoodModel(...)          # from convokit.redirection
redirection = MediatorRedirection(likelihood_model)
redirection.transform(corpus)
redirection.summarize(corpus)   # mean score per conversation

It also writes utt.meta["role"] ("mediator", or the participant’s name), the attribute convokit.redirection reads. fit fine-tunes the likelihood model (with convokit.redirection’s own “Speaker A/B” formatting); scoring works without it.

The corpus must identify the mediator and the name each speaker goes by in the chat. By default these come from the ConvoKit AI fields (speaker.ai_meta["role"] and conversation.alias); they can be replaced with is_mediator and name_func.

Reference: Taking a turn for the better: Conversation redirection throughout the course of mental-health therapy, https://aclanthology.org/2024.findings-emnlp.555/

Parameters

  • likelihood_model – a convokit.redirection LikelihoodModel (e.g. GemmaLikelihoodModel); scores can be computed with the off-the-shelf model, and fit fine-tunes it

  • counterfactual – "prev" or "delete" (see mediator_previous_context_selector)

  • is_mediator – function from Speaker to whether it is the mediator

  • name_func – function from (Conversation, Speaker) to a participant’s speaker prefix

  • redirection_attribute_name – name of the utterance metadata attribute to store scores in

fit(corpus: convokit.model.corpus.Corpus, y=None, train_selector: Callable[[convokit.model.conversation.Conversation], bool] = <function MediatorRedirection.<lambda>>, val_selector: Callable[[convokit.model.conversation.Conversation], bool] = <function MediatorRedirection.<lambda>>)

Fine-tune the likelihood model on the selected conversations (see convokit.redirection.Redirection.fit).

Parameters

  • corpus – the Corpus to train on

  • y – unused; accepted for compatibility with the Transformer interface

  • train_selector – function from Conversation to whether it is used for training

  • val_selector – function from Conversation to whether it is used for validation

Returns

this MediatorRedirection

label_roles(corpus: convokit.model.corpus.Corpus, selector: Callable[[convokit.model.conversation.Conversation], bool])

Write each utterance’s role into utt.meta["role"] in the selected conversations: "mediator" for the mediator, otherwise the participant’s name from name_func (with ” (participant)” appended if that name is itself "mediator").

Parameters

  • corpus – the Corpus to label

  • selector – function from Conversation to whether it should be labeled

summarize(corpus: convokit.model.corpus.Corpus, selector: Callable[[convokit.model.conversation.Conversation], bool] = <function MediatorRedirection.<lambda>>) → pandas.DataFrame

Compute the mean redirection score per conversation.

Parameters

  • corpus – a Corpus that has been transformed

  • selector – function from Conversation to whether it should be included

Returns

DataFrame with one row per conversation that has at least one score, and columns conversation_id, n_scored and redirection_mean

transform(corpus: convokit.model.corpus.Corpus, selector: Callable[[convokit.model.conversation.Conversation], bool] = <function MediatorRedirection.<lambda>>, verbosity: int = 5)

Label roles and store the redirection score of each scorable mediator turn in utt.meta[redirection_attribute_name].

Parameters

  • corpus – the Corpus to transform

  • selector – function from Conversation to whether it should be scored

  • verbosity – print progress every verbosity conversations

Returns

the Corpus, with scores added

convokit.ai_evaluation.redirection.mediator_future_context_selector(convo: convokit.model.conversation.Conversation, role_attribute_name: str = 'role') → Dict[str, List[str]]

Compute the future context of every mediator turn: the participant turn right after it, for use as the future_context_selector of convokit.redirection.Redirection.

Mediator turns followed by another mediator turn, by an empty utterance, or by nothing get no future context.

Parameters

  • convo – Conversation whose utterances carry meta[role_attribute_name]

  • role_attribute_name – utterance metadata attribute holding the role

Returns

dict of utterance id -> future context

convokit.ai_evaluation.redirection.mediator_previous_context_selector(convo: convokit.model.conversation.Conversation, counterfactual: str = 'prev', role_attribute_name: str = 'role') → Tuple[Dict[str, List[str]], Dict[str, List[str]]]

Compute the actual and reference contexts of the mediator turns, for use as the previous_context_selector of convokit.redirection.Redirection.

A context is computed for every mediator turn that has a participant turn before it and, with the "prev" counterfactual, an earlier mediator turn to stand in for it. Mediator turns directly following another mediator turn are skipped.

Parameters

  • convo – Conversation whose utterances carry meta[role_attribute_name], with the value "mediator" for the mediator

  • counterfactual – "prev" replaces the mediator turn with the mediator’s previous turn in the reference context; "delete" removes it

  • role_attribute_name – utterance metadata attribute holding the role

Returns

dicts of utterance id -> actual context and utterance id -> reference context

Raises

ValueError – if counterfactual is not "prev" or "delete"

convokit.ai_evaluation.redirection.mediator_speaker_prefixes(roles: List[str]) → Dict[str, str]

Build speaker prefixes that use each role name itself (e.g. “Goose: “), so that a participant’s prefix matches the name the mediator addresses them by.

Parameters

roles – the role names in a conversation

Returns

dict of role name -> prefix

LLM client

LLM access for the evaluation Transformers.

The Transformers call generate_json(prompt) on their client. get_eval_llm_client() builds one from a model name:

  • OpenAI models (gpt-*, o1/o3/o4): convokit.genai’s GPTClient always sends temperature and max_tokens, which OpenAI’s reasoning models (gpt-5, gpt-5-mini) reject, so GPT5Client is used instead. It keeps the convokit.genai LLMClient interface and adds reasoning_effort, an optional fixed seed and JSON mode.

  • Gemini models (gemini-*): convokit.genai’s GeminiClient, wrapped in JSONClient.

class convokit.ai_evaluation.llm.GPT5Client(model: str, config_manager: convokit.genai.genai_config.GenAIConfigManager = None, reasoning_effort: str = 'minimal', seed: Optional[int] = None, json_mode: bool = False, max_retries: int = 1)

Client for OpenAI reasoning models (the gpt-5 family) with the convokit.genai LLMClient interface.

Failed calls (and, in generate_json, unparseable replies) are retried after a 10-second wait.

Parameters

  • model – model name, e.g. "gpt-5" or "gpt-5-mini"

  • config_manager – GenAIConfigManager holding the "gpt" API key (default: a GenAIConfigManager reading ~/.convokit/config.yml)

  • reasoning_effort – OpenAI reasoning_effort ("minimal", "low", "medium" or "high")

  • seed – OpenAI seed for best-effort determinism (not sent if None)

  • json_mode – request response_format={"type": "json_object"}

  • max_retries – how many times to retry a failed call or an unparseable JSON reply

Raises

  • ImportError – if the openai package is not installed

  • ValueError – if no OpenAI API key is configured

generate(prompt: str, **kwargs) → convokit.genai.base.LLMResponse

Generate a reply to a prompt.

Parameters

  • prompt – the prompt

  • kwargs – ignored; accepted for compatibility with the LLMClient interface

Returns

an LLMResponse with the reply text, total tokens used (-1 if unknown), latency in seconds and the raw OpenAI response

generate_json(prompt: str)

Generate a reply to a prompt and parse it as JSON.

Parameters

prompt – the prompt

Returns

the parsed JSON (dict or list)

class convokit.ai_evaluation.llm.JSONClient(client: convokit.genai.base.LLMClient, max_retries: int = 1, **generate_kwargs)

Wraps a convokit.genai LLMClient with the generate_json method the evaluation Transformers call.

Unparseable replies are retried after a 10-second wait.

Parameters

  • client – the convokit.genai LLMClient, e.g. a GeminiClient

  • max_retries – how many times to retry an unparseable JSON reply

  • generate_kwargs – keyword arguments passed to every client.generate call, e.g. json_mode=True and seed=0 for a GeminiClient

generate(prompt: str, **kwargs) → convokit.genai.base.LLMResponse

Generate a reply to a prompt with the wrapped client.

Parameters

  • prompt – the prompt

  • kwargs – passed to the wrapped client’s generate, overriding generate_kwargs

Returns

the wrapped client’s LLMResponse

generate_json(prompt: str)

Generate a reply to a prompt and parse it as JSON.

Parameters

prompt – the prompt

Returns

the parsed JSON (dict or list)

convokit.ai_evaluation.llm.get_eval_llm_client(model: str, provider: Optional[str] = None, config_manager: convokit.genai.genai_config.GenAIConfigManager = None, reasoning_effort: str = 'minimal', seed: Optional[int] = None, json_mode: bool = False) → convokit.genai.base.LLMClient

Build the LLM client the evaluation Transformers use by default.

The API key is read by convokit.genai (GenAIConfigManager): the "gpt" or "gemini" key, from the GPT_API_KEY / GEMINI_API_KEY environment variable or ~/.convokit/config.yml. Gemini can also use Vertex AI (see convokit.genai’s GeminiClient).

Parameters

  • model – model name, e.g. "gpt-5" or "gemini-2.5-flash"

  • provider – "gpt" or "gemini"; inferred from model if not given

  • config_manager – GenAIConfigManager holding the API key (default: a GenAIConfigManager reading ~/.convokit/config.yml)

  • reasoning_effort – OpenAI reasoning_effort; not used for Gemini

  • seed – seed for best-effort determinism (not sent if None)

  • json_mode – for OpenAI, request response_format={"type": "json_object"} (which only allows a JSON object, not an array). Gemini clients always use JSON mode, since Gemini’s also allows arrays.

Returns

a client with generate and generate_json methods

Raises

ValueError – if the provider is not supported or cannot be inferred

convokit.ai_evaluation.llm.infer_provider(model: str) → str

Infer the convokit.genai provider of a model from its name.

Parameters

model – the model name

Returns

"gpt" for OpenAI models (gpt*, o1*, o3*, o4*) or "gemini" for Gemini models (gemini*)

Raises

ValueError – if the provider cannot be inferred

convokit.ai_evaluation.llm.parse_json(text: str)

Parse the JSON object or array in a model reply, tolerating Markdown code fences and control characters.

Parameters

text – the model reply

Returns

the parsed JSON (dict or list)

Raises

json.JSONDecodeError – if the reply is not valid JSON

Defaults for reading the ConvoKit AI format

Helpers shared by the evaluation Transformers.

They read the ConvoKit AI data format (see convokit/model/AI_FORMAT.md): the mediator is the speaker whose ai_meta["role"] is "public assistant", and conversation.alias maps speaker ids to the names used in the chat.

convokit.ai_evaluation.util.default_is_mediator(speaker: convokit.model.speaker.Speaker) → bool

Check whether a speaker is the mediator, i.e. its ConvoKit AI role (speaker.ai_meta["role"]) is "public assistant".

Parameters

speaker – the Speaker

Returns

True if the speaker is the mediator

convokit.ai_evaluation.util.default_speaker_name(conversation: convokit.model.conversation.Conversation, speaker: convokit.model.speaker.Speaker) → str

Get the name a speaker goes by in a conversation, from conversation.alias.

Parameters

  • conversation – the Conversation

  • speaker – the Speaker

Returns

the speaker’s alias in the conversation, or its id if it has none

convokit.ai_evaluation.util.default_topic(conversation: convokit.model.conversation.Conversation) → str

Get the topic line shown to the LLM: conversation.meta["topic"], followed by the debate statement conversation.meta["statement"] if there is one.

Parameters

conversation – the Conversation

Returns

the topic line (an empty string if neither field is set)

convokit.ai_evaluation.util.display_names(conversation: convokit.model.conversation.Conversation, is_mediator: Callable[[convokit.model.speaker.Speaker], bool], name_func: Callable[[convokit.model.conversation.Conversation, convokit.model.speaker.Speaker], str]) → Dict[str, str]

Map each non-mediator speaker id in a conversation to its name, in order of first utterance.

Parameters

  • conversation – the Conversation

  • is_mediator – function from Speaker to whether it is the mediator

  • name_func – function from (Conversation, Speaker) to the speaker’s name

Returns

dict of speaker id -> name

convokit.ai_evaluation.util.pre_survey_text(conversation: convokit.model.conversation.Conversation, speaker: convokit.model.speaker.Speaker) → Optional[str]

Get the reason a participant gave for their stance before the conversation.

Parameters

  • conversation – the Conversation; its meta["pre_survey"] maps speaker ids to text

  • speaker – the Speaker

Returns

the speaker’s pre-survey text, or None if there is none