Preliminaries

• Next Meeting Dates (options): WE1S project Calendar

• Friday Jan. 8th?
• Current Scraping Work (status)
• UCSB (Developer Tasks page)
• McGill
• Interim and Future Scraping Work & Other Tasks:
• What can we define and delegate as tasks between now and next meeting?
• Continued scraping, inspection
• Continued work on the scrubbing list and extra stopwords list
• Github training?
• Winter 2016 work:
• Finalization of corpus (for publications we have collected)
• Begin scraping of political discourse relating to humanities (see Austin Yack's collection spreadsheet)
  

   

Workshop for End-to-End Trial Rehearsal of Workflow for Topic Modeling WE1S Corpus (at sample scale)

       Step-by-step instructions for Workshop:

         (We'll run in parallel at our various locations. However, some steps will be pre-prepared.)

• Infrastructure for Workshop:

• Parallel installations on computers at the following locations:
• Transcriptions (UCSB) (Prep the workstation attached to the projector and Skype)
• Alan
• Lindsay
• Scott (if he is able to participate in the workshop)
• Installations should include: (see under spotlight at top of this page)

---

• Workshop Stage 1 -- Assemble a demo subcorpus of the WE1S corpus
  

• Access latest "flattened zipped" versions of the WE1S corpus on the Mirrormask server at the location: File Station > Archives  (screenshot of example). For this workshop, we will be using the flattened version of 25 October 2015 (corpus_2015-10-25_flat.zip). (Note: accessing the Mirrormask server requires working over the UCSB VPN and login permissions for the WE1S user)
• Download the zipped file, unzip it, and copy out part of the corpus for use as our workshop demo corpus. (We will be using New York Times articles, 2010-2014, mentioning "humanities" and the also New York Times articles, 2010-14 mentioning "liberal arts") (To save time, you can download the humanities files for these years here, and the liberal arts files here from the WE1S Google Drive, which you need to log into as the WE1S user.)
• Paste the demo corpus into your local workspace in folders: C:\workspace\1\articles-h and C:\workspace\1\articles-la

---

• Workshop Stage 2 - De-duplicate the demo subcorpus

• This step has been prepared in advance with the help of Jeremy (see corpus_compare.xlsx )
• Total number of New York Times 2010-14 "humanities" articles: 1,062
• Total number of New York Times 2010-14 "liberal arts" articles: 751
• Total articles consolidating above after de-duping: 1,623
• Jeremy will demonstrate the process in real time.
• De-duplication script:
  
  corpus_compare.py
  https://mirrormask.english.ucsb.edu:5001/fbsharing/CBkLLLlX
  
  The de-duplication python script generates a file listing of duplicate file pairs in spreadsheet format, along with a rating of their reasons for matching and degree of matching. It can be downloaded and run on your computer. It requires the installation of python 2.7 and scikit-learn. Alternately, it can be used form a virtual container (anaconda) running on our server with all requirements already installed.
  
      usage:
  
      corpus_compare.py [-h] [-i INPUTPATHS [INPUTPATHS …]]]
      [-f FILEPATTERN] [-o OUTPUTFILE] [-t THRESHOLD]
  
  Use Example 1:
  
      corpus_compare.py
  
  Scans all .txt files in the current directory and subdirectories; default settings and output.
  
  Use Example 2:
  
      corpus_compare.py -i /mytemp/doc-compare-test/2015-11-16-workshop/data/ -f “*.txt” -t 0.85 -o /mytemp/doc-compare-test/2015-11-16-workshop/corpus_compare-args.csv
  
  Scans with a specified input directory, file filter, threshold, and output file.
• The de-duplicated, consolidated demo sub-corpus (consisting of New York Times articles, 2010-14 mentioning "humanities" and/or "liberal arts") should be put on your local workspace in the folder C:\workspace\1\articles\

---

• Workshop Stage 3 - Scrub the demo subcorpus

• Apply to the subcorpus Scott's python scrubbing script (scrub.py) using the current version of the config.py file containing scrubbing information (for convenient view of content of file, see text version of config.py (2015-11-16).txt).
• Deposit results in a separate folder in the workspace: C:\workspace\articles-scrubbed\
  
• Download a copy of the current WE1S we1s_extra_stopwords.txt file from the WE1S Google Drive here; and put in in your workspace folder, C:/workspace/1/ (This will be used when running MALLET during the topic modeling stage next.)

---

• Workshop Stage 4 - Topic model the subcorpus using MALLET:

• (A) Open a command shell ("command line") window
1. Windows "command prompt" (using shell command language based on MS-DOS) -- quick guide -- tip on enabling copy/paste operations
2. Mac "terminal" (using shell command language based on Unix) -- cheat sheet 
3. Linux command line (also uses bash)
• (B) Navigate to the Mallet folder (directory)
1. Windows: type the following at the command line, followed by a return (helpful tip: the <F5> function key pastes in the previously used command)
•  cd c:\mallet
2. Mac: type the following at the command line
• cd /Users/yourusername/mallet
• As with the Windows command above, this will depend on where and how you have saved the MALLET folder you downloaded when you installed the package. The above command assumes you've saved MALLET (and titled the folder it's in "mallet") under your home directory. 

---

• (C) Input a folder of text files and process them into a .mallet data file
  Use the command below, varying the path and file names as desired (red italics indicate path/filenames you supply).  Use forward slashes in Windows, backward slashes on Macs. There must be no hidden returns in the command. Best practice is to set the job up in a text document (without "wordwrap" view turned on) and copy/paste the command into the command shell.
• General format (Windows): bin\mallet import-dir --input path to folder --output path and filename of desired output data file with .mallet extension --keep-sequence --remove-stopwords
• Command line for this workshop (Windows):
• bin\mallet import-dir --input C:\workspace\1\articles-scrubbed --output C:\workspace\1\topic-models\topics.mallet --keep-sequence --remove-stopwords  --extra-stopwords C:\workspace\resources\we1s_extra_stopwords.txt
• Command line for this workshop (Mac):
• ./bin/mallet import-dir --input /Users/[your-user-name]/workspace/1/articles-scrubbed --output /Users/[your-user-name]/workspace/1/topic-models/topics.mallet --keep-sequence --remove-stopwords  --extra-stopwords /Users/[your-user-name]/workspace/resources/we1s_extra_stopwords.txt

---

• (D) Create the topic model ("train" the topic model) -- Use the following command.
• General format (Windows): bin\mallet train-topics  --input path and filename of the previously created data file with .mallet extension --num-topics desired number of topics --optimize-interval 20 --output-state path to output folder\topic-state.gz --output-topic-keys path to output folder\keys.txt --output-doc-topics --path to output folder\composition.txt --word-topic-counts-file  path to output folder\topic_counts.txt
• Command line for this workshop (Windows):
• bin\mallet train-topics  --input C:\workspace\1\topic-models\topics.mallet --num-topics 100 --optimize-interval 20 --output-state C:\workspace\1\topic-models\topic-state.gz --output-topic-keys C:\workspace\1\topic-models\keys.txt --output-doc-topics C:\workspace\1\topic-models\composition.txt --word-topic-counts-file C:\workspace\1\topic-models\topic_counts.txt
• Command line for this workshop (Mac):
• ./bin/mallet train-topics  --input /Users/[your-user-name]/workspace/1/topic-models/topics.mallet --num-topics 100 --optimize-interval 20 --output-state /Users/[your-user-name]/workspace/1/topic-models/topic-state.gz --output-topic-keys /Users/[your-user-name]/workspace/1/topic-models/keys.txt --output-doc-topics /Users/[your-user-name]/workspace/1/topic-models/composition.txt --word-topic-counts-file /Users/[your-user-name]/workspace/1/topic-models/topic_counts.txt

---

• Workshop Stage 5 - Clustering the topics:

• Use Scott's topicsToDocs.py script on the topic-counts.txt file produced by MALLET to create "topic-documents" from the individual topics in the topic model. (Or use Lexos to do the same):
• Configure the settings at the top of the topicsToDocs.py script to the following (Windows; adapt path name for Mac)
• input_file_path = "C:/workspace/1/topic-models"
  input_file = "topic_counts.txt"
  num_topics = 100
• The script will deposit the resulting topic documents (named "Topic0.txt" Topic1.txt" etc.) in the same folder as its input (in this case C:/workspace/1/topic-models/.  Move the topic documents into C:/workspace/1/topic-models/topic-documents/ 
• Using the Anaconda distribution of Python, open your local copy of the iPython notebook titled dariah-de-tutorial-v3.ipynb (available from the WE1S Google Drive here). This is the the WE1S adaptation of a DARIAH-DE tutorial iPython notebook (edited by Scott and Alan). The notebook contains executable Python commands and scripts tht we will use to produce PCA clustering and dendogram visualizations of the topic-documents.
• Open the Anaconda "Launcher"
• From the options in the launcher, choose "ipython-notebook" and launch. From the folder list that opens, find and open the local copy of dariah-de-tutorial-v3.ipynb. This will start a local web service on your computer and open the iPython notebook as if it were a Web page in your browser. (Note: the local web service is started in a command or terminal window on your computer. Leave that window open.)
• In the iPython notebook, scroll down a bit and begin at the section titled "Getting Some Data."  You'll see paragraphs of explanatory text interspersed with executable cells containing live Python code that can be run. To run the code in a cell, place your cursor in the cell, and then press CTRL-return.
• Execute the code in each of the cells in sequence.
• Note: when you get to the set of cells under the heading "Preparing to Analyze Document Vectors," there is one cell (#4) to pay special attention to. It's the cell that follows immediately after the explanation, "We will use CountVectorizer both to load the texts and to construct a DTM from them. We'll start by constructing a list of filenames...." This cell contains code that begins with filenames = [  The paths and file names here have to be specific for where you put the 100 topic-documents on your local machine.
• For Windows machines, a copy of the code for all the filenames for this workshop are pre-filled in the iPython notebook.  The code for this cell is also contained here in this plain text file titled filenames-for-iPython-notebook.txt
• For Mac machines, you will need to adapt the code (using search and replace in a text processor)
• Example of results from using this iPython notebook:
• Alan's results from NY Times 2002-2006 "humanities": PCA | Dendogram

(d) For the Next Workshop: Interpret the Topic Model of the Sub-corpus

• Discuss the topic model of the sub-corpus based on inspecting the topic model and also the clustering dendogram (and other clustering experiments)
• Work out a systematic, documentable workflow for interpreting topic models

(e) For a Later Workshop: Improve

• Improve workflow
• Experiment with alternative workflows, e.g., Andrew Goldstone's DFRtopics R package?

(f) Manifest schema, Database system

• reports from Scott and Jeremy

Scott created a demo of webform access to a mongodb database, and I have build a system to serve it out of containers (virtual machines). An early form example and a more recent database-connected example are hosted here:

    1. WE1S flask+deform  

    http://mirrormask.english.ucsb.edu:8500/

    2. WE1S flask+alpaca (+pymongo)  

    http://mirrormask.english.ucsb.edu:8503/

(NOTE -- as always you may need to campus VPN in order to access these URLs)