Pivotal Moment Measure

The PivotalMomentMeasure transformer identifies pivotal moments in conversations as described in this paper.

We consider a moment in a conversation pivotal if the next response is expected to have a large impact on the conversation’s eventual outcome. Our method relies on two main components: an utteranceSimulatorModel for generating possible responses and a forecasterModel for forecasting the eventual outcome of the conversation.

PivotalMomentMeasure uses a temporally-ordered stream of conversational data in the form of “context tuples” to train and make predictions on. Context tuples are generated in chronological order for each utterance in a conversation. Each context tuple is defined as a NamedTuple with the following fields:

  • context: a chronological list of Utterances up to and including the most recent Utterance at the time this context was generated

  • current_utterance: the most recent utterance at the time this context tuple was generated

  • future_context: all Utterances chronologically after the current utterance (or an empty list if this Utterance is the last one)

  • conversation_id: the Conversation that this context-reply pair came from

We also provide a general utteranceSimulator interface to utteranceSimulatorModel models that abstracts away the implementation details into a standard fit-transform interface.

Example usage: pivotal moments demo in conversations gone awry

class convokit.pivotal_framework.pivotal.PivotalMomentMeasure(simulator_model: convokit.utterance_simulator.utteranceSimulatorModel.UtteranceSimulatorModel, forecaster_model: convokit.forecaster.forecasterModel.ForecasterModel, piv_attribute_name: str = 'PIV', simulated_reply_attribute_name: str = 'sim_replies', simulated_reply_forecast_attribute_name: str = 'sim_replies_forecasts', simulated_reply_forecast_prob_attribute_name: str = 'sim_replies_forecast_probs', forecast_attribute_name: str = 'forecast', forecast_prob_attribute_name: str = 'forecast_prob', labeler: str = 'has_removed_comment')

ConvoKit transformer to compute pivotal scores. The pivotal framework consists of two main components: (1) simulator_model to simulate potential next responses and (2) forecaster_model to predict the likelihood of the outcome based on these potential responses.

Parameters

  • simulator_model – UtteranceSimulatorModel to simulate next utterances in the conversation given a conversation context, ConvoKit provides implementation for models supported by Unsloth via unslothUtteranceSimulatorModel

  • forecaster_model – ForecasterModel to forecast the likelihood of the conversation’s outcome given a conversation context

  • piv_attribute_name – Name of metadata field to save pivotal scores

  • simulated_reply_attribute_name – Name of metadata field to save simulated replies generated by the simulator_model

  • simulated_reply_forecast_attribute_name – Name of metadata field to save the forecasts of the contexts with simulated replies

  • simulated_reply_forecast_prob_attribute_name – Name of metadata field to save the forecast probabilities of the contexts with simulated replies

  • forecast_attribute_name – Name of metadata field to save the forecasts of the contexts

  • forecast_attribute_name – Name of metadata field to save the forecast probabilties of the contexts

  • labeler – Name of metadata field to indicate the outcome or label of the conversation

fit(corpus: convokit.model.corpus.Corpus, forecaster_train_context_selector: Callable[[convokit.pivotal_framework.util.ContextTuple], bool] = <function PivotalMomentMeasure.<lambda>>, forecaster_val_context_selector: Optional[Callable[[convokit.pivotal_framework.util.ContextTuple], bool]] = None, forecaster_test_context_selector: Callable[[convokit.pivotal_framework.util.ContextTuple], bool] = <function PivotalMomentMeasure.<lambda>>, simulator_train_context_selector: Callable[[convokit.pivotal_framework.util.ContextTuple], bool] = None, simulator_val_context_selector: Optional[Callable[[convokit.pivotal_framework.util.ContextTuple], bool]] = None)

Fit the PivotalMomentMeasure transformer by fitting the underlying Forecaster and UtteranceSimulator components using the selected contexts for each component.

Parameters

  • corpus – Corpus containing the data

  • forecaster_train_context_selector – Function to select context tuples for forecaster training

  • forecaster_val_context_selector – Function to select context tuples for forecaster validation

  • forecaster_test_context_selector – Function to select context tuples for forecaster testing

  • simulator_train_context_selector – Function to select context tuples for simulator training

  • simulator_val_context_selector – Function to select context tuples for simulator validation

Returns

fitted PivotalMomentMeasure Transformer

fit_forecaster(corpus: convokit.model.corpus.Corpus, train_context_selector: Callable[[convokit.pivotal_framework.util.ContextTuple], bool] = <function PivotalMomentMeasure.<lambda>>, val_context_selector: Optional[Callable[[convokit.pivotal_framework.util.ContextTuple], bool]] = None, test_context_selector: Callable[[convokit.pivotal_framework.util.ContextTuple], bool] = <function PivotalMomentMeasure.<lambda>>)

Wrapper method for fitting and transforming the underlying forecaster model.

Parameters

  • corpus – Corpus containing the data to train and test on

  • train_context_selector – Function to select context tuples for training

  • val_context_selector – Function to select context tuples for validation

  • test_context_selector – Function to select context tuples for testing

fit_simulator(corpus: convokit.model.corpus.Corpus, train_context_selector: Callable[[convokit.pivotal_framework.util.ContextTuple], bool] = <function PivotalMomentMeasure.<lambda>>, val_context_selector: Optional[Callable[[convokit.pivotal_framework.util.ContextTuple], bool]] = None)

Wrapper method for fitting the underlying utterance simulator model.

Parameters

  • corpus – Corpus containing the data to train and validate on

  • train_context_selector – Function to select context tuples for training

  • val_context_selector – Function to select context tuples for validation

fit_transform(corpus: convokit.model.corpus.Corpus, forecaster_train_context_selector: Callable[[convokit.pivotal_framework.util.ContextTuple], bool] = <function PivotalMomentMeasure.<lambda>>, forecaster_val_context_selector: Optional[Callable[[convokit.pivotal_framework.util.ContextTuple], bool]] = None, forecaster_test_context_selector: Callable[[convokit.pivotal_framework.util.ContextTuple], bool] = <function PivotalMomentMeasure.<lambda>>, simulator_train_context_selector: Callable[[convokit.pivotal_framework.util.ContextTuple], bool] = <function PivotalMomentMeasure.<lambda>>, simulator_val_context_selector: Optional[Callable[[convokit.pivotal_framework.util.ContextTuple], bool]] = None, transform_context_selector: Callable[[convokit.pivotal_framework.util.ContextTuple], bool] = <function PivotalMomentMeasure.<lambda>>)

Fit and transform the PivotalMomentMeasure transformer.

Parameters

  • corpus – Corpus containing the data

  • forecaster_train_context_selector – Function to select context tuples for forecaster training

  • forecaster_val_context_selector – Function to select context tuples for forecaster validation

  • forecaster_test_context_selector – Function to select context tuples for forecaster testing

  • simulator_train_context_selector – Function to select context tuples for simulator training

  • simulator_val_context_selector – Function to select context tuples for simulator validation

  • transform_context_selector – Function to select context tuples to run over

Returns

annotated Corpus

summarize(corpus: convokit.model.corpus.Corpus)

Summarizes PivotalMomentMeasure transformer with distribution of pivotal scores.

Parameters

corpus – Corpus to analyze

transform(corpus: convokit.model.corpus.Corpus, context_selector: Callable[[convokit.pivotal_framework.util.ContextTuple], bool] = <function PivotalMomentMeasure.<lambda>>, annotate_high_low_scores: bool = False)

Apply the PivotalMomentMeasure transformer to the selected contexts and annotates the corpus with the pivotal scores.

Parameters

  • corpus – Corpus containing the data

  • context_selector – Function to select context tuples to run on

  • annotate_high_low_scores – Whether to annotate utterances with high and low pivotal scores based on top/bottom percentile

Returns

annotated Corpus

class convokit.utterance_simulator.utteranceSimulatorModel.UtteranceSimulatorModel

Abstract class representing a model to simulate the next utterance in the conversation. Different models can be supported by inheriting from this base class.

abstract fit(contexts, val_contexts)

Finetune this utterance simulator model on the given contexts and validates on val_contexts.

Parameters

  • contexts – Iterator over context tuples for training

  • val_contexts – Iterator over context tuples for validation

property name

Name of the simulator model.

abstract transform(contexts, simulated_reply_attribute_name)

Apply this model to the given data, and return its simulated responses in the form of a DataFrame indexed by (current) utterance ID

Parameters

contexts – Iterator over context tuples

Returns

a Pandas DataFrame, with one row for each context, indexed by the ID of that context’s current utterance. Contains list of simulated replies in the simulated_reply_attribute_name column.

class convokit.utterance_simulator.utteranceSimulator.UtteranceSimulator(simulator_model: convokit.utterance_simulator.utteranceSimulatorModel.UtteranceSimulatorModel, simulated_reply_attribute_name: str = 'sim_replies')

A wrapper class that provides a consistent interace to any UtteranceSimulatorModel instance. From a user perspective, this makes it easy to apply these models to ConvoKit corpora and swap between different models while maintaining the same interface. From a developer perspective, this provides a prebuilt foundation upon which new models can be easily developed.

Parameters

  • simulator_model – An instance of a UtteranceSimulatorModel subclass that implements the uttterance simulator model you want to use

  • simulated_reply_attribute_name – Name of metadata field to save simulated replies generated by the simulator_model

fit(corpus=<class 'convokit.model.corpus.Corpus'>, context_selector: Callable[[convokit.utterance_simulator.util.ContextTuple], bool] = <function UtteranceSimulator.<lambda>>, val_context_selector: Callable[[convokit.utterance_simulator.util.ContextTuple], bool] = <function UtteranceSimulator.<lambda>>)

Wrapper method for fine-tuning the underlying utterance simulator model. Handles the creation of context iterators which are passed to the underlying simulator_model to process accordingly.

Parameters

  • corpus – Corpus containing the data to train on

  • contexts – Function to select context tuples for training

  • val_contexts – Function to select context tuples for validation

Returns

fitted UtteranceSimulator Transformer

property name

Name of the simulator model.

transform(corpus: convokit.model.corpus.Corpus, context_selector: Callable[[convokit.utterance_simulator.util.ContextTuple], bool] = <function UtteranceSimulator.<lambda>>)

Wrapper method for applying the underlying utterance simulator model to generate replies over the conversation contexts. Handles the creation of context iterators which are passed to the underlying simulator_model to process accordingly.

Parameters

  • corpus – Corpus containing the data to run on

  • contexts – Function to select context tuples to transform

Returns

annotated Corpus