Interpretation Protocol (Topic Model Interpretation Workflow)


Possible WE1S project workflow for interpreting topic models (Draft 1)

(including parallel, alternative, or conjoined human and machine processes):

 

 

Human Processes

(Typical procedure for steps requiring judgment will be to have a panel of three or more people perform the step and compare results. The hermeneutical rhythm will typically consist of iterative cycles of observations suggesting research questions, and research questions suggesting ways to sharpen observation.)

Machine Processes

(We may be able to automate some steps and sequences)

1
Assess topic models to determine appropriate number of topics. We may decide to generate one, two, or three numbers of topics for simultaneous interpretation.
(Questions: Can we define criteria for "best" topic model? Do we know any published theory or methods for choosing right number of topics? Cf. Scotts issues for discussion.pdf)

Generate topic models at many levels of granularity--e.g., 25, 50, 150, 200, 250, 300, 350, 400, 450, 500

2
Initial analysis of topic models.
  1. Assign labels for topics (assisted by automated process suggested at right).
  2. Identify and label any major clusters of topics. 
  3. Flag for attention any illegible topics.
Assemble materials to facilitate interpretation:
  1. Create sorted keys files in a spreadsheet.
  2. Create topic cloud vizualizations.
  3. Create clustering visualizations
    (Testing phase: compare a human-panel-only clustering with a machine clustering of topics)
  4. Assess "nearness" of topics (We don't yet have a method to do this; but cf. Goldstone DFR Browser "scaled view")
  5. If possible, auto-label topics with the top 2-4 most frequent words in a topic (based on an algorithm that establishes a minimum threshold of proportional frequency and decides what to do if there are one, two, three, or four top words that are, or are not, significantly more important than others.)
3

Detailed analysis of topic model (part I: total corpus, synchonic analysis).

  1. Study major topics and clusters of topics.
  2. Human panel reads sample articles and compares to the topic proportions found in the topic-counts.txt file created by Mallet. (This is a sanity check.)
  3. Human panel writes up analytical notes and observations, and compares.
  4. Members of the human write up report.

 

 
4

Detailed analysis of topic model (part II: comparative analysis).

  1. Study major correlations/differences between any two or three parts of our corpus of interest.
Create view of topic model that compares two or more parts of our corpora (e.g., NY Times vs. The Guardian) for the topics and topic weights they contain. We don't yet have an interface or method of using the composition.txt files produced by Mallet to do this. (cf. Goldstone DFR Browser "document view," which shows topics in a single document) (Alan's experiment)
5

Detailed analysis of topic model (part III: time-series analysis).

  1. Study trends in topics.
Create views of topic model that shows trend lines of topics (created by showing weights of topics in documents at time 1, followed by time 2, etc.). We don't yet have a method or tool for this, but cf. the following time-series views in the Goldstone DFR Browser: topics across years | topics within a single year. See also: demo vizualization of topics in State of Union addresses; the TOM code demos; Robert K. Nelson, "Mining the Dispatch") (Alan's experiment)
6

Write up results:

  1. Create key observations and data set and publish (with a catchy title like "Humanities in Public Discourse: The Manual").
  2. Co-author white paper with recommendations for humanities advocacy.
    1. Create subset of above as a brochure or infographic
  3. Disseminate research methods and conclusions in academic talks, panels, papers, articles.