Paired Prediction¶
At a high level, Paired Prediction is a quasi-experimental method that controls for certain priors, see Cheng et al. 2014 for an illustrated example of PairedPrediction in research.
As an illustrative example, consider the Friends TV series, where we might want to examine how Rachel talks to Monica and Chandler differently. At one level, we might just look at the differences in the utterances where Rachel speaks to Monica and Rachel speaks to Chandler. But this inadvertently surfaces differences that might arise from Rachel interacting with Monica and Chandler separately in different settings and scenarios, and thus highlight only uninteresting differences in topics discussed.
Instead, we might want to look for subtler differences in speech, controlling for topic perhaps. One way we might to do this to look only at Conversations where Rachel, Monica, and Chandler are all present. We would then compare utterances where Rachel speaks to Monica and Rachel speaks to Chandler within that Conversation and look for differences between these paired sets of utterances.
Documentation for the two transformers that do paired prediction task is presented below. PairedPrediction transformer uses corpus object’s metadata features for predictions, while PairedVectorPrediction transformer utilizes vector data associated with the object. Also, see the documentation for Pairer transformer, which sets up the pairs needed in paired prediction analysis.
Example usage: Using Hyperconvo features to predict conversation growth on Reddit in a paired setting
-
class
convokit.paired_prediction.pairedPrediction.PairedPrediction(obj_type: str, pred_feats: List[str], clf=None, pair_id_attribute_name: str = 'pair_id', label_attribute_name: str = 'pair_obj_label', pair_orientation_attribute_name: str = 'pair_orientation')¶ At a high level, Paired Prediction is a quasi-experimental method that controls for certain priors, see Cheng et al. 2014 for an illustrated example of PairedPrediction in research. (https://cs.stanford.edu/people/jure/pubs/disqus-icwsm14.pdf)
See Pairer’s documentation for more information about pairing.
- Parameters
pred_feats – list of metadata attributes (i.e. predictive features) to be used in prediction. Features can either be values or a dictionary of key-value pairs.
clf – optional classifier to be used in the paired prediction
pair_id_attribute_name – metadata attribute name to use in annotating object with pair id, default: “pair_id”
label_attribute_name – metadata attribute name to use in annotating object with predicted label, default: “label”
pair_orientation_attribute_name – metadata attribute name to use in annotating object with pair orientation, default: “pair_orientation”
-
fit(corpus: convokit.model.corpus.Corpus, y=None, selector: Callable[[convokit.model.corpusComponent.CorpusComponent], bool] = <function PairedPrediction.<lambda>>)¶ Fit the internal classifier on the paired object features, with an optional selector selecting for which corpus objects to include in the analysis
- Parameters
corpus – target Corpus
selector – a (lambda) function that takes a Corpus object and returns a bool: True if the object is to be included in the paired prediction. By default, includes all objects.
- Returns
fitted PairedPrediction Transformer
-
get_coefs(feature_names: List[str], coef_func=None)¶ Get dataframe of classifier coefficients.
- Parameters
feature_names – list of feature names to get coefficients for
coef_func – function for accessing the list of coefficients from the classifier model; by default, assumes it is a pipeline with a logistic regression component
- Returns
DataFrame of features and coefficients, indexed by feature names
-
summarize(corpus: convokit.model.corpus.Corpus, selector: Callable[[convokit.model.corpusComponent.CorpusComponent], bool] = <function PairedPrediction.<lambda>>, cv=sklearn.model_selection.KFold)¶ Run PairedPrediction on the corpus with cross-validation and returns the mean cross-validation score.
- Parameters
corpus – target Corpus (must be annotated with pair information using PairedPrediction.transform())
selector – a (lambda) function that takes a Corpus object and returns a bool: True if the object is to be included in summary. By default, includes all objects.
cv – optional CV model: default is KFold(n_splits=5, shuffle=True)
- Returns
cross-validation accuracy score
-
transform(corpus: convokit.model.corpus.Corpus) → convokit.model.corpus.Corpus¶ PairedPrediction does not add any annotations to the Corpus.
-
class
convokit.paired_prediction.pairedVectorPrediction.PairedVectorPrediction(obj_type: str, vector_name: str, clf=None, pair_id_attribute_name: str = 'pair_id', label_attribute_name: str = 'pair_obj_label', pair_orientation_attribute_name: str = 'pair_orientation')¶ Transformer for doing a Paired Prediction with vectors.
- Parameters
obj_type – corpus component type being used for analysis: ‘utterance’, ‘speaker’, or ‘conversation’
vector_name – name of the vector matrix containing the bag-of-words vectors
clf – classifier to be used in the paired prediction; by default: standard-scaled logistic regression
pair_id_attribute_name – metadata attribute name to use in annotating object with pair id, default: “pair_id”
label_attribute_name – metadata attribute name to use in annotating object with predicted label, default: “label”
pair_orientation_attribute_name – metadata attribute name to use in annotating object with pair orientation, default: “pair_orientation”
-
fit(corpus: convokit.model.corpus.Corpus, y=None, selector: Callable[[convokit.model.corpusComponent.CorpusComponent], bool] = <function PairedVectorPrediction.<lambda>>)¶ Fit the internal classifier to the Corpus component objects.
- Parameters
corpus – the target Corpus
selector – selector (lambda) function for which objects should be included in the analysis
- Returns
this Transformer object with a fitted internal classifier
-
get_coefs(feature_names: List[str], coef_func=None)¶ Get dataframe of classifier coefficients. By default, assumes it is a pipeline with a logistic regression component. For other setups, the user should define a custom coef_func.
- Parameters
feature_names – list of feature names to get coefficients for
coef_func – (optional) function for accessing the list of coefficients from the classifier model
- Returns
DataFrame of features and coefficients, indexed by feature names
-
summarize(corpus: convokit.model.corpus.Corpus, selector: Callable[[convokit.model.corpusComponent.CorpusComponent], bool] = <function PairedVectorPrediction.<lambda>>, cv=sklearn.model_selection.KFold)¶ Run PairedPrediction on the corpus with cross-validation.
- Parameters
corpus – annoted Corpus (with pair information from PairedPrediction.transform())
selector – selector (lambda) function for which objects should be included in the analysis
cv – optional CV model: default is KFold(n_splits=5, shuffle=True)
- Returns
cross-validation accuracy score