New Zealand!
(a) Near-future and Far-future Scheduling
-
Next Meeting Dates (options in November): WE1S project Calendar
-
Future Research Outcomes (?)
-
Panel proposal for DH 2017 in Montreal.
- WE1S Panel would include several of us giving talks on the sub-topics:
- WE1S overview and corpus
- WE1S topic modeling methods, clustering methods, and results
- WE1S Manifest Schema
- WE1S infrastructure (Github
container: MongoDB) - Collaboration with Lexos
-
Project write-up for submission to the new DHCommons Journal for peer-review of DH projects.
- Later: individual or co-authored articles for journals on various aspects of the project.
(b) Scraping Work
-
Current status of UCSB scraping (Developer Tasks page)
- Main fixes and to-do's that need to be applied to the summer scraping work?
- Ongoing scraping in fall? (Ashley and possibly also Jamal serving as scrapers)
- Status of the WE1S Corpus
- on Google Drive
- mirrored on Mirrormask (Jeremy's server) at: mirrormask/4humwe1s-GDrive/ (screenshot)
- "flattened" versions archived at: mirrormask/4humwe1s-GDrive/archives/(screenshot)
- backup archive at: TimeBackup/ (screenshot)
-
Scraping of Globe and Mail
- Scraping to be done by Nathalie Popa (McGill U.)
- Current problems
-
Austin Yack's research on government and legislative documents on the humanities
- Whitehouse.gov research
- www.congress.gov research (via Sunlight Foundation's API) -- example query string | Web interface for API
(c) Next Meeting: Workshop for End-to-End Trial Rehearsal of Workflow for Topic Modeling WE1S Corpus (at sample scale)
Plan for Workshop:
(We'll run in parallel at our various locations. However, some steps may be pre-prepared; and some may be performed as a tutorial only from one location.)
-
Infrastructure for Workshop:
- Parallel installations on computers at the following locations:
- Transcriptions (UCSB) (Prep the workstation attached to the projector and Skype)
- Alan
- Lindsay
- Scott
- Installations should include (in addition to tools we all already have on our machines)
- Anaconda distro of Python.
- Relevant Python scripts:
- Relevant iPython notebooks:
-
Workshop Stage 1 -- Assemble a demo sub-corpus of the WE1S corpus
- Access latest "flattened" collections of files in which final file names have been assigned (e.g., no "File12.txt" names) (screenshot of example)
- Carve out a part of the corpus (e.g., NY Times, 2010-2014, "humanities" and "liberal arts") for our demo corpus
- Copy the demo corpus to a working folder (e.g., on a Windows machine: C:\workspace\nyt-2010-2014\articles\
-
Workshop Stage 2 - De-duplicate the sub-corpus
- Report from Jeremy (and discussion)
-
Workshop Stage 3 - Scrub the sub-corpus
- Run Scott's python scrubbing script (scrub.py) on the corpus, and deposit results in a separate folder in the workspace (on a Windows machine, for example, C:\workspace\nyt-2010-2014\articles-scrubbed\
- Note: the most current version of the config-py file used to add words and phrases to the scrub.py script is kept on Google Drive: we1s-2 > stopwords_and_scrubbing_list. This config-py reflects Lindsay and Alan's cumulative additions to the scrubbing list so far.)
- Note: there is an Extra Stopwords list for MALLET at the Google Drive location above. It is used during topic modeling with MALLET.
-
Workshop Stage 4 - Topic model the sub-corpus using MALLET:
- [Steps to be filled in here]
- Recent topic modeling experiments with WE1S corpus:
- Alan: topic models of NY Times 2002-6 "humanities", and NY times 2010-14 "humanities" (the five years before and after the Great Recession).
- Ashley & Zach: topic models of 1980s discourse about the humanities versus 1990s discourse about the humanities
-
Workshop Stage 5 - Clustering topics:
- Use Scott's topicsToDocs.py script on the topic-counts.txt file produced by MALLET to create "topic-documents" from the individual topics in the topic model. (Or use Lexos to do the same)
- Use Scott's adaptation of the DARIAH-DE tutorial iPython notebook to produce PCA clustering and dendogram visualizations of the topic-documents
(d) For the Next Workshop: Interpret the Topic Model of the Sub-corpus
- Discuss the topic model of the sub-corpus based on inspecting the topic model and also the clustering dendogram (and other clustering experiments)
- Work out a systematic, documentable workflow for interpreting topic models
(e) For a Later Workshop: Improve
- Improve workflow
- Experiment with alternative workflows, e.g., Andrew Goldstone's DFRtopics R package?
(f) Manifest schema, Database system
- reports from Scott and Jeremy
Scott created a demo of webform access to a mongodb database, and I have build a system to serve it out of containers (virtual machines). An early form example and a more recent database-connected example are hosted here:
1. WE1S flask+deform
http://mirrormask.english.ucsb.edu:8500/
2. WE1S flask+alpaca (+pymongo)
http://mirrormask.english.ucsb.edu:8501/
(NOTE -- as always you may need to campus VPN in order to access these URLs)