Parliament Question Time Corpus

A collections of questions and answers from parliamentary question periods in the British House of Commons from May 1979 to December 2016 (433,787 utterances), scraped from They Work For You <https://www.theyworkforyou.com/>_.

Distributed together with: Asking Too Much? The Rhetorical Role of Questions in Political Discourse. Justine Zhang, Arthur Spirling, Cristian Danescu-Niculescu-Mizil. EMNLP 2017.

Dataset details

Speaker-level information

The speakers in the dataset are members of the Parliament (MP). For each MP, the dataset further includes the following metadata:

  • name: name of the MP

  • member_start: start date of the MP as the member of the Parliament

  • member_end: end date of the MP as the member of the Parliament (set to year 3020 if the MP was still in Parliament as of Dec 2016)

  • parties: a list of parties that the MP has belonged to in the past

  • first_govt: first government (by Prime Minister) in which the MP was in office

  • first_govt_coarse: first government in which the MP was in office. here, consecutive governments of the same party (e.g., thatcher+major) are grouped together.

Note that some of the metadata information may be missing, especially for MPs active before the Blair government.

Utterance-level information

Each question or answer is viewed as an utterance. For each utterance, we provide:

  • id: index of the utterance

  • speaker: the MP who spoke the utterance

  • conversation_id: id of the first utterance in the conversation this utterance belongs to

  • reply_to: id of the utterance to which this utterance replies to (None if the utterance is not a reply)

  • timestamp: time of the utterance

  • text: textual content of the utterance

Additional metadata include:

  • next_id: id of the utterance replying to this one (None if the utterance has no reply)

  • is_question: whether the utterance is a question

  • is_answer: whether the utterance is an answer to a question

  • pair_idx: index of the question-answer pair

  • is_incumbent: whether the MP is incumbent (i.e., a member of the government party)

  • is_minister: whether the MP is a Minister

  • is_oppn: whether the MP is from the official opposition party

  • party: party affiliation of the MP

  • tenure: the number of years that the MP has been in office at the time of the utterance

  • govt: current government (by Prime Minister) at the time of the utterance

  • govt_coarse: current government (by Prime Minister) at the time of the utterance. here, consecutive governments of the same party (e.g., thatcher+major) are grouped together.

  • pair_has_features: whether the pair to which the utterance belongs has a question that contains at least one q_arc term and an answer that contains at least one arcs term.

  • dept_name: the name of the department to which the answering minister for the question or answer belongs. inferred from the raw HTML (see dept_name_raw for an un-processed version of the same attribute).

  • dept_name_coarse: department name, listed as other for departments other than the 10 containing the most question-answer pairs.

Note that some of the metadata information may be missing, especially for utterances dating back to before the Blair government.

The dataset also comes with the following processed fields, which can be loaded separately via corpus.load_info(‘utterance’,[list of fields]):

  • parsed: SpaCy dependency parse

  • arcs: dependency parse arcs, without nouns

  • q_arcs: dependency parse arcs for questions only, without nouns

The latter two fields are used in the original publication to represent utterances.

Usage

To download directly with ConvoKit:

>>> from convokit import Corpus, download
>>> corpus = Corpus(filename=download("parliament-corpus"))

For some quick stats:

>>> corpus.print_summary_stats()
Number of Speakers: 1978
Number of Utterances: 433787
Number of Conversations: 216894

Additional note

See this example notebook for an example of how to group questions in this dataset according to their rhetorical roles.