Operational Master Files

     The operational master file for WE1S scrubbing fixes is the config.py script (which configures the scrub.py script).  The current copy with the latest fixes is here on the WE1S Google drive. Fixes stored in config.py will be active when scrub.py runs (when both are stored in the same directory on a local workstation). (Config.py last revised 2 Jan. 2016 by Alan)

     The operational master file for WE1S stopwords is the spreadsheet we1s-stopwords-master-file.xslx (on the WE1S Google Drive). That spreadsheet file contains both extra stopwords added for the WE1S project (sheet 1) and the  standard Mallet stopwords list (sheet 2). In both cases, the lists exist in two forms: all-lower-case and "proper" case (first letter capitalized).  (There are also a few all-upper-case words in the lists.) The spreadsheet is used to generate plain-text stopwords files as needed, depending on whether we use the scrubbing script not only to apply fixes but also to delete stopwords from all articles before topic modeling (in which case we generate a stopword list combining the Mallet and WE1S lists) or we ask Mallet to ignore stopwords only at the topic-modeling stage (in which case we generate a stopword list consisting of only the WE1S list).  Note: if using the scrub.py to delete stopwords from the articles, both the lower-case and proper case lists of stopwords must be included in the stopwords file (to delete both versions of words). (we1s-stopwords-master-file.xlsx last revised 2 Jan. 2016 by Alan.)

Function of This Web Page

     This web page functions as a convenience to record the fixes and stopwords in the operational versions of the files described above.  Collaborators on WE1S can check the web page to see if a problem they discover is already take care of.  If not, they can add suggestions to the bottom of each category under the label "to be added" indicated in red)

     The web page also explains the principles we are using for identifying fixes and stopwords.

     Note:  We can include here also problems that only show up in some publications, so long as applying fixes to them has no deleterious impact on other publications. However, indicate with a note when a fix should be applied only to a specific publication. For example: we want to remove "New York Times" only from the NY Times files, not from other publications.

Explanation:

The general principle is that we want the WE1S scrubbing script (whose values are set in config.py) and the WE1S extra stopword list (whose values are set in we1s_extra_stopwords.txt) to work in tandem to prevent the Mallet topic modeling program from encountering frequent non-thematic lexical elements that detract from discovering meaningful topics. Consider the following hypothetical example of a cluster of words that Mallet suggests are part of a single topic because they frequently co-occur in a corpus: "whale harpoon of ship oil John the blubber etc. boat sea above seals captain Monday 's ocean". Here the words in red are thematically significant words.  The other words or word fragments are non-thematic because they are either too common or co-occur in non-meaningful ways with other terms. For example, the fact that a word like "of" or a name like "John" co-occurs with both terms A and B does not mean that terms A and B actually have anything in common as a theme.  (There could be many different "John's" in a corpus, for instance). Note: we only worry about problems that occur frequently, as determined by text analysis of topic models, for example. Infrequent problems will tend to drop out of sight in topic modeling.

Punctuation Fixes:
(applied in config.py)

• Punctuation fixes currently in config.py (for scrubbing script)
• . (replace period with period followed by space)
• : (replace colon with period followed by space)
• ? (replace ? with ? followed by space)

•  Punctuation fixes to be added to config.py
•  's (replace apostrophe s with nothing)

Spelling Variants

(applied in config.py)

• Spelling variants currently consolidated in config.py (for scrubbing list)
  Principles for scrubbing: (1) for important concepts in our context, we standardize to the American spelling.
• centre --> center
• labour --> labor
• organisations --> organization
• programme --> program

• Spelling variants to be added for consolidation in config.py
•  

Tokenizations:

(applied in config.py)

("Tokenize" in this context means change multi-word phrases into what text analysis program will treat as a single word.)
Principles: We consolidate: (1) important organization, nation, or other entity names for our context; (2) important acronyms in our context; (3) important names or phrases in which one part of the phrase would otherwise be stopped out; (4) phrases, titles, names that tend to be used as indicators of humanities works or issues (e.g., liberal_arts, Romeo_and_Juliet); (5) other important phrases functioning as compound words (e.g., social_sciences)

• Tokenizations currently in config.py (for scrubbing script)
• Abu_Dhabi
• Affordable_Care_Act
• American Association of University Professors --> AAUP
• American_Studies_Association
• Art_History
• Big_Ten
• Bryn_Mawr
• center-left, centre-left --> center_left
• Carnegie_Mellon
• Claremont_McKenna
• Chronicle_of_Higher_Education
• Cold_War
• Common_Core
• continuing_education
• Cooper_Union
• Department_of_Education 
• distance education, distance-education ---> distance_education
• distance_learning
• distant_reading
• East_Coast
• East_Asian
• e-mail, E-mail --> email
• Emily_Dickinson
• Ezra_Pound
• Fairleigh_Dickinson
• Ford_Foundation
• full_time
• [consolidations for school grades]
• first_grade (etc.)
• hard_headed
• hard_nosed
• hard_science
• hard_sciences
• hard_times
• hard_wired
• hard_work
• hard_working
• Harvard_University
• Harvey_Mudd
• high-paying --> high_paying
• high school, high-school, high schools, High School --> high_school
• high-skill --> high_skill
• high-skilled --> high_skilled
• high_technology
• H.M.O. --> HMO
• Ho_Chi_Minh
• Hong_Kong
• Hurricane_Katrina
• Ivy_League
• King_Lear
• Las_Vegas
• cum_laude
• [consolidations related to "left"]
• American_left
• British_left
• left_brain
• left_leaning
• left_wing
• the_left
• [consolidations related to "right"]
• [same with variation as for "left" above"]
• Letters_to_the_Editor
• liberal arts, liberal-arts, liberal art, liberal-art --> liberal_arts
• Long_Island
• long-term, long term --> long_term
• long-time, long time --> long_time
• los_angeles
• [consolidations related to humanities majors] -- both singular and plural of "major" need to be consolidated (e.g., "English major" and "English majors")
• English_major
• History_major
• Philosophy_major
• French_major
• Classics_major
• Art_major
• Language_major
• humanities_major
• arts_major
• art_major
• Art_History_major
• major_in_humanities
• [consolidations related to humanities minors]
• [same with variation as for humanities majors]
• Martin_Luther_King
• M.D., M.D.s --> M_D  
• Middle East, Middle Eastern --> Middle_East
• M.A. --> M_A
• MacArthur_Foundation
• Mellon_Foundation
• Modern Language Association --> MLA
• Mother_Teresa
• National Endowment for the Humanities, N. E. H. --> NEH
• National Endowment for the Arts, N. E. A. --> NEA
• National_Humanities_Center
• National_Commission_on_Excellence_in_Education 
• New_Haven
• New_Left
• New_Jersey
• New_York
• Nobel_laureate
• North America, North_American --> North_America
• Op Ed, Op ed, op ed --> Op_Ed
• part time, part-time --> part_time
• Pell grant, Pell grants --> Pell_grants
• Ph.D., Ph. D., PhD --> Ph_D
• Phi_Beta_Kappa
• poet laureate, Poet laureate, poets laureate --> poet_laureate
• political_correctness
• queer_studies
• queer_theory
• Rockefeller_Foundation
• Rome_and_Juliet
• Royal_Society
• SAT_scores
• SAT averages, SAT score averages --> SAT_averages
• SAT_verbal
• SAT_math
• SAT ACT, SAT/ACT, SAT / ACT, ACT SAT, ACT/SAT, ACT / SAT --> SAT_ACT
• SAT subject test, SAT subject tests --> SAT_subject_test
• SAT test, SAT tests --> SAT_tests
• SAT_results
• SAT_data
• Saul_Bellow
• sci-fi, sci fi, Sci-Fi, Sci Fi ---> sci_fi
• social_sciences
• social_science
• social_scientist
• social_studies
• South America, South American --> South_America
• Star_Trek
• Star_Wars
• startup, startups, Startup, Startups, start-ups, Start-ups, Start-up --> start-up
• status quo, status-quo --> status_quo
• Tel_Aviv
• The_National_Review
• Title_IX
• Uncle_Sam
• United Kingdom, U. K., U.K. --> United_Kingdom
• United_Nations
• University_of_California
• University_of_Chicago
• University_of_Bridgeport 
• US, USA, U.S.A., U. S. A., U.S., United States, United States of America --> U_S
• vice_chancellor
• vice_dean
• vice_president
• vice_presidential
• vice_provost
• vice_rector
• waiting list, waiting lists --> waiting_list
• Wall_Street_Journal
• Wi-fi, wifi, Wifi ---> wi-fi
• White_House
• White_Plains
• World_War_I
• World_War_II
• Yale_University

Tokenizations to be added to config.py

• Alternative Arts Council
• Alvin Ailey Dance School Theatre
• American Ballet Theatre
• American Federation of Teachers
• American Library Association
• American Psychiatric Association 
• American Revolution
• American Symphony Orchestra 
• Arts Council
• Arts Council of England
• Arts Factory
• Arts and Humanities Research Board
• Association of Medical Research Charities 
• Australian National University
• Bath Spa University 
• Beloit University
• Berkshire Theatre Festival
• Birmingham City Council
• Boilerhouse Theatre Company
• Brandeis University
• Bristol University
• British Council
• Cambridge University Library
• Cardiff University
• Columbia University
• Dairy Council
• Economic and Social Research Council
• Edinburgh University
• English National Opera
• Essex University
• Florida State University
• Film Council   
• Friedrich Wilhelm University
• Gateshead Council
• Higher Education Funding Council
• Howard University 
• Johns Hopkins University 
• Jubilee Arts
• Kingston University 
• Labour Government
• Leeds University
• London Arts Board 
• London County Council 
• Luton University
• Manchester University
•  Medical Research Council 
• Metropolitian Borough Council
• Middlesex University   Muslim Council of Britain  
• National Association for the Advancement of Colored People
• National Anthem
• National Autistic Society
• National Book Award
• National Bureau for Students with Disabilities
• National Campaign for the Arts
• National Centre for Popular Music
• National Council of La Raza
• National Endowment for Science and Technology
• National Environment Research Council 
• National Enquirer
• National Foundation for Educational Research
• National Foundation for Jewish Culture
• National Gallery
• National Gallery of Scotland
• National Gay and Lesbian Task Force
• National Geographic
• National Hispanic Foundation
• National Institution for Equal Opportunity
• National Institute for Research
• National Maritime Museum
• National Organization of Women (also spelled Organisation)
• National Philharmonic Orchestra of Russia
• National Poetry Day
• National Postgraduate Committee
• National Railway Museum
• National Religious Party
• National Rifle Association
• National Science Foundation
• National Society of Painters
• National Theatre
• National Trust
• National Trust for Scotland
• National Union of Teachers
• Native American 
• Northwestern University
• Northumbria University
• Ohio University
• Open University
• Piccadilly Theatre

• Poole Arts Centre 

• Princeton University
•  Queen’s University
• Refugee Council

• Research Defense Society

• Research Fortnight 

• Royal Court Theatre
•  Schiller International University
• Shadow Arts Council
• Sheffield Hallam University
• Stanford University
• Stella Adler Theatre School

• South Tyneside Arts Studio 

• Tory Government
• University College London
• University of East Anglia
• University of East London
• University of Melbourne
• University of North London
• University of Salford
• University of Texas at Austin
• Warwick University
• Wimbledon’s Polka Theatre
• York University

• o   American_Express
• o   Black_History_Month
• o   Board_of_Education
• o   Business_World
• o   City_College_of_New_York
• o   City_University_of_New_York
• o   Corporation_for_Public_Broadcasting
• o   D.C., D. C. à D_C
• o   D.C. Public Library, District of Columbia Public Library à D.C._Public_Library
• o   Department_of_Education
• o   Department_of_Public_Works
• o   District_of_Columbia
• o   Federal_City_College
• o   Fort_Washington
• o   George_Mason_University
• o   George_Washington
• o   George_Washington_University
• o   House_Intelligence_Committee
• o   House_Speaker
• o   Howard _University
• o   Kansas_City
• o   Kennedy_Center_Opera_House
• o   Latin_American
• o   Mexico_City
• o   National_Book_Award
• o   National_Endowment_for_the_Arts
• o   National_Endowment_for_the_Humanities
• o   National_Gallery
• o   National_Geographic
• o   National_Geographic_Society
• o   National_Guard
• o   National_Humanities_Center
• o   National_Humanities_Medal
• o   National_Museum_of_American_History
• o   National_Museum_of_Natural_History
• o   National_Park_Service
• o   National_Public_Radio
• o   New_York_City
• o   New_York_Public_Library
• o   North_American
• o   Ohio State, Ohio State University à Ohio_State_University
• o   Paul_Peck_Humanities_Institute
• o   Secretary_of_Education
• o   Secretary_of_State
• o   State_Department
• o   Third_World
• o   University_of_Maryland
• o   Washington, D.C. à Washington_D.C.
• o   Washington_Metropolitan_Area
• o   Washington_Performing_Arts_Society
• o   Washington_Post
• o   White_House
• o   World_War_I
• o   World_War_II

Phrases stopped out in config.py (for scrubbing script):

• Stop phrases currently in config.py (for scrubbing script)
• Associated Press
• Continue reading the main story
• Corrections & Amplifications
• Credit: By
• New York Times (this fix specific to NYT)
• njtowns@nytimes.com
• N.Y. / Region
• Published:
• Room for Debate (this fix specific to NYT)
• Special to the New York Times (this fix specific to NYT)
• Sunday New Jersey Section
• 620 Eighth Avenue, New York, N.Y. 10018-1405

• Stop phrases to add to config.py 
• Guardian (this fix specific to Guardian)
•    Washington_Post (this fix specific to WP)
•    Washington_Ways (this fix specific to WP)
•    Los_Angeles_Times (this fix specific to LATimes)
•    National_Public_Radio (this fix specific to NPR)

Single words to be stopped out in we1s_extra_stopwords.txt:

• Names currently stopped out in we1s_extra_stopwords.txt
  Principles for adding words : (1) common first names of people, but not first names that are also common last names or place names; (2) months, numbers, days of week and their abbreviations; (3) phrases such as "pages" that in newspapers have a primarily functional rather than thematic meaning; (3) common inadvertent OCR and word fragment problems
• ’ll
  ’ve
  ’ve
  ’ve
  “’ve
  á
  à
  ã
  a.'s
  aaron
  abdul
  abe
  abigail
  ABOUT
  abraham
  abram
  abrams
  adams
  additional
  adele
  adlai
  adler
  afternoon
  afternoons
  ain
  al
  alan
  alexander
  alexandra
  alfred
  ali
  alice
  alicia
  alison
  allan
  allen
  allison
  alvin
  amanda
  amir
  amy
  and/or
  anderson
  andré
  andrea
  andrew
  andrews
  andy
  angela
  angelina
  ann
  anna
  anne
  annette
  annie
  anthony
  appl
  apr
  april
  aren’
  aren’
  arnold
  ashley
  AT&
  aubrey
  audrey
  aug
  august
  avenue
  avery
  axelrod
  baldwin
  bam
  barry
  barton
  based
  becker
  ben
  benjamin
  benson
  bernard
  beth
  betsy
  betty
  beverley
  beverly
  bianca
  bob
  bonnie
  brad
  bradley
  brady
  brett
  brian
  bruce
  burke
  BY
  ç
  calvin
  cameron
  campbell
  can't
  canʼt
  carlos
  carlson
  carly
  carol
  caroline
  carolyn
  carolyn
  case
  catherine
  cathy
  celeste
  chad
  charles
  charlotte
  chelsea
  chen
  chester
  chris
  christine
  christopher
  chuck
  claim
  claimed
  claire
  clara
  clark
  cohen
  coleman
  collins
  comprises
  comprising
  connected
  connor
  constance
  continue
  continued
  continues
  cooper
  couldn
  craig
  crawford
  cummings
  d.'s
  daly
  dan
  daniel
  danny
  daphne
  dave
  david
  davidson
  davis
  de
  debbie
  deborah
  debra
  dec
  dec.
  december
  del
  dempsey
  denise
  dennis
  derek
  derrick
  diana
  diane
  dick
  didn
  didn'
  didn"t
  didn’
  didn’
  doesn
  doesn"t
  doesn't
  doesnʼt
  don
  don
  don"t
  don’
  don't
  donʼt
  dorothy
  doug
  douglas
  douglass
  dr
  dr
  du
  duncan
  dwight
  é
  ë
  e.g
  earl
  eddie
  edmund
  eduardo
  edward
  edwin
  eighth
  eileen
  el
  elena
  eli
  elisabeth
  elizabeth
  ellen
  elliott
  elliott
  emily
  en
  entitled
  eric
  eric
  erica
  erik
  erika
  erin
  esther
  ethan
  eugene
  evans
  evening
  everett
  evylyn
  ext
  fact
  feb
  feb
  february
  felicia
  ferdinand
  fernando
  fig
  figs
  filed
  FOR
  form
  francis
  françois
  frank
  fred
  frederick
  friday
  fridays
  gabriel
  gabrielle
  gary
  george
  GEORGE
  gerald
  geraldine
  glenn
  gloria
  goings-
  goldman
  goldman
  greg
  gregg
  gregory
  greta
  gretchen
  gustav
  gustavo
  hadn
  hannah
  hans
  harold
  harriet
  harry
  hasn
  hasn’
  hassan
  helen
  henry
  herman
  he's
  hester
  hour
  hours
  howard
  href
  htm
  html
  HTML
  http
  hundred
  hundreds
  í
  ian
  id
  ii
  iii
  iii
  il
  i'm
  iʼm
  IN
  include
  included
  includes
  including
  irving
  isaac
  isaiah
  isn
  isn’
  isn’
  it"s
  item
  items
  it's
  itʼs
  jack
  jake
  james
  jamie
  jan
  JAN
  jane
  janet
  january
  jason
  jay
  jean
  jeanette
  jeanne
  jeannette
  jeff
  jeffrey
  jenny
  jeremy
  jerome
  jerry
  jesse
  jessica
  jill
  jim
  jimmy
  joan
  joanie
  joanna
  joanne
  joe
  joel
  john
  johnny
  johnson
  jon
  jonah
  jonathan
  jones
  jose
  joseph
  josh
  joshua
  jr
  jul
  jul
  JUL
  Jules
  julia
  Juliá
  julie
  juliet
  july
  JULY
  jun
  june
  JUNE
  JUNE
  karen
  kari
  kate
  katharine
  kathleen
  kathryn
  kathy
  katie
  kay
  kaye
  keith
  kelly
  kelsey
  ken
  kenneth
  kenny
  kent
  kerry
  kevin
  kevin
  kim
  kimberly
  kirk
  kirsten
  konrad
  kristin
  kristina
  kurt
  la
  lauren
  laurence
  laurie
  lawrence
  lee
  leif
  leigh
  leila
  lena
  leo
  leonard
  lets
  lev
  li
  liam
  lila
  lillian
  lilly
  lin
  lindsay
  lindsey
  lionel
  liz
  liza
  lizzie
  ll
  llc
  lloyd
  lonni
  lonnie
  lori
  louie
  louis
  louis
  louisa
  lucas
  lucinda
  lucy
  ludwig
  luis
  luke
  lynn
  lynne
  madeleine
  madeline
  mar
  MAR
  marc
  march
  marcia
  marcos
  marcus
  marcy
  margaret
  maria
  marianne
  marie
  marie
  marilyn
  marilynne
  mario
  marion
  marissa
  mark
  mark
  martha
  martin
  marvin
  mary
  mary
  marybeth
  mathew
  matthew
  matthew
  maurice
  max
  maxwell
  maynard
  megan
  melanie
  melinda
  melissa
  meryl
  michael
  michael
  michel
  michele
  michelle
  miguel
  mike
  mikhail
  miller
  minute
  minutes
  mitchell
  Monday
  mondays
  month
  months
  morgan
  morning
  mornings
  mr
  MR
  mrs.
  Mrs.
  Ms.
  murray
  mville
  ñ
  nancy
  naomi
  natalie
  natasha
  nate
  nathan
  nathaniel
  neil
  nelson
  nicholas
  nicholas
  nick
  nicole
  nina
  ninth
  NOT
  nov
  NOV
  november
  nytimes
  ó
  ö
  oct
  OCT
  october
  oliver
  olivia
  omar
  ON
  ONE
  org
  org.
  org.;
  page
  pamela
  paolo
  parker
  part
  pat
  patricia
  patrick
  patrick
  patti
  paul
  paul
  paula
  pauline
  paulo
  pdf
  PDF
  PER
  pete
  peter
  peter
  pgs
  phil
  philip
  philippe
  phillip
  php
  PHP
  phyllida
  PLUS
  PM
  pp
  present
  provide
  provided
  providing
  pt.
  put
  puts
  putting
  rachel
  ralph
  ray
  rebecca
  reginald
  related
  rené
  result
  richard
  richard
  rick
  rick
  ricky
  robert
  robert
  roberto
  robin
  roger
  roland
  ron
  ronald
  rosie
  ross
  roy
  ruth
  ryan
  sachs
  said
  sally
  sam
  samantha
  samuel
  sandra
  sara
  sarah
  Saturday
  saturdays
  se
  sean
  sebastian
  selected
  sept
  SEPT
  september
  series
  set
  seth
  seymour
  sherry
  sheryl
  shouldn
  shouldn’
  shown
  sidney
  similar
  sixth
  smith
  sonia
  specific
  spoke
  st.
  st.
  stanley
  stella
  step
  stephanie
  stephen
  stephen
  steps
  steve
  steven
  street
  street
  streets
  stuart
  suggest
  suggested
  suggesting
  suggests
  sullivan
  Sunday
  sundays
  susan
  susan
  suzanne
  sylvia
  takes
  taking
  tamara
  tanya
  telling
  tells
  tens
  terms
  terry
  that"s
  that's
  thatʼs
  th-century
  THE
  theo
  there's
  thereʼs
  theresa
  therese
  THESE
  thirds
  THIS
  thom
  thomas
  thousand
  thousands
  Thursday
  thursdays
  tim
  timothy
  tm
  TO
  tobias
  told
  tom
  toni
  tony
  travis
  trevor
  Tuesday
  tuesdays
  tyler
  uspto
  valerie
  vanessa
  versus
  victoria
  vincent
  vivian
  vladimir
  vol
  walt
  walter
  wasn
  Wednesday
  wednesdays
  week
  weeks
  wendy
  weren
  weren'
  WHAT
  WHEN
  WHERE
  wherein
  WHY
  william
  william
  williams
  willie
  willis
  WITH
  wo
  WOULD
  wouldn
  www
  xi
  xia
  years
  york
  YOU
  you"re
  you're
  youʼre
  zachary

Other words and partial words to add to we1s_extra_stopwords.txt

•  Â
• EDT
• Five 
• four 
• Ô 
• twenty 
• Two 
• wll
•  Zillion
• Zooey 
• Π (pie symbol)