This week's NLP experimentation involves Project Gutenberg's plain-text edition of The Pickwick Papers. I parsed the text into individual chapters and calculated some summary statistics about each, then build a visualization of each chapter's new vocabulary.
Setup¶
A few imports to start with:
- First, I need
collections.OrderedDict--for now, I'll be using one to store individual book chapters. re, for some very simple data cleanup.requests.get()to grab the book
The rest of these imports are just NLTK, Seaborn and the standard Python data science toolkit.
from collections import OrderedDict
import re
import nltk
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from requests import get
import seaborn as sns
%matplotlib inline
First I grab Project Gutenberg's plain-text edition of The Pickwick Papers, then strip out (using split() and list indexing) Gutenberg's front- and backmatter and the table of contents to get just the book's text.
Also, Gutenberg's plain-text version uses underscores to indicate italics. I don't need those so I remove them here.
rq = get('https://www.gutenberg.org/files/580/580-0.txt')
#Not ascii, which requests assumes
rq.encoding='utf8'
pickwick = rq.text
pickwick = pickwick.split('THE POSTHUMOUS PAPERS OF THE PICKWICK CLUB')[2]
pickwick = pickwick.split('End of the Project Gutenberg EBook of The Pickwick Papers')[0]
pickwick = pickwick.replace('_', '')
First, I break the text into paragraphs, then use enumerate() and filter() to get indexes of the chapter headings (conveniently, they're the only "paragraphs" that start with the string "CHAPTER "). This gives me a list like this:
[(2, 'CHAPTER I. THE PICKWICKIANS'),
(26, 'CHAPTER II. THE FIRST DAY’S JOURNEY, AND THE FIRST EVENING’S ADVENTURES;\r\nWITH THEIR CONSEQUENCES'),
...]I then cycle through this list to locate the text of each chapter. The chapter texts are stored in an OrderedDict, with the chapter titles used as keys.
Then I delete my _chapterheads list, since I no longer need it and want to limit the number of copies of this book I have stored in memory.
paragraphs = pickwick.split('\r\n\r\n')
_chapterheads = list(filter(lambda x: x[1].startswith('CHAPTER '), enumerate(paragraphs)))
chapters = OrderedDict()
for i in range(len(_chapterheads)-1):
ch = _chapterheads[i]
nextch = _chapterheads[i+1]
chapters[ch[1]] = list(filter(lambda x: x.strip() != '', paragraphs[ch[0]+1:nextch[0]]))
ch = _chapterheads[i+1]
chapters[ch[1]] = list(filter(lambda x: x.strip() != '', paragraphs[ch[0]+1:]))
del(_chapterheads)
Next, I need lists of tokens:
- First, generate lists of entities, including both words and punctuation marks, one for each chapter.
- Then, a copy of that first list, but lowercased and with punctuation removed.
- Finally a third list with each chapter separated into sentences.
_chaptertexts = ['\n'.join(x) for x in chapters.values()]
_booktext = '\n\n'.join(_chaptertexts)
chaptertokens = [nltk.word_tokenize(chap) for chap in _chaptertexts]
chapterwords = [[w.lower() for w in chap if not re.match('\W', w)] for chap in chaptertokens]
chaptersentences = [nltk.sent_tokenize(chap) for chap in _chaptertexts]
del(_chaptertexts)
del(_booktext)
Next, I copy the lists from chapterwords into sets, giving me a list of the unique words in each chapter--I'll use that soon. Then I have a copy of this with stopwords removed, and a frequency distribution of the words in each chapter.
sw = nltk.corpus.stopwords.words('english')
chaptervocabsw = [set(chap) for chap in chapterwords]
chaptervocab = [set([word for word in chap if word not in sw]) for chap in chaptervocabsw]
#Remember we're not counting punctuation here.
chapterfreq = [nltk.FreqDist(chap) for chap in chapterwords]
newvocab = []
for c in range(57):
newvoc = chaptervocabsw[c].difference(*chaptervocabsw[:c])
newvocab.append(nltk.FreqDist(dict([x for x in chapterfreq[c].items() if x[0] in newvoc])))
#Make newvocab 1-indexed.
newvocab = dict(enumerate(newvocab, start=1))
Now I use Pandas to build a DataFrame of potentially-interesting statistics. This is done with a complex-looking list comprehension that generates an 8-tuple describing each chapter. The DataFrame constructor interprets this as a table.
chapterdata = pd.DataFrame([
(
len(chapterwords[x-1]),
np.mean([len(tok) for tok in chaptertokens[x-1] if not re.match('\W', tok)]),
len(chaptersentences[x-1]),
len(chapterwords[x-1]) / len(chaptersentences[x-1]),
len(chaptervocabsw[x-1]),
len(chapterwords[x-1])/len(chaptervocabsw[x-1]),
len(newvocab[x]),
len(newvocab[x])/len(chaptervocabsw[x-1])*100,
)
for x in range(1,58)],
index=pd.Index(range(1,58), name='chapter'),
columns=['word_count', 'avg_word_length', 'sentence_count', 'avg_sentence_length',\
'unique_words', 'lexical_diversity', 'new_vocabulary', 'pct_new_vocab'])
chapterdata.to_csv('~/data/nlp/pickwick_details.csv')
chapterdata
Skimming the data, it looks like there's not much variation in word length per chapter, but quite a bit more in sentence length and lexical diversity (the ratio of total word length to unique words in the chapter). We can quickly verify this with a simple calculation on the dataframe, normalizing the standard deviations of each column.
chapterdata.std()/chapterdata.mean()
I wonder if there happens to be any correlation between sentence length and lexical diversity:
print(np.corrcoef(chapterdata.avg_sentence_length, chapterdata.lexical_diversity))
No, there isn't. At least not a significant one. We can see this in a quick scatterplot:
sns.set(style="whitegrid")
sns.scatterplot('avg_sentence_length', 'lexical_diversity', data=chapterdata)
plt.ylabel('Lexical Diversity')
plt.xlabel('Average Sentence Length')
plt.show()
Finally, a serious graph. I went through several versions of this; this one's my favorite. I'm graphing the percent of each chapter's vocabulary that is new to that chapter. Next, I add labels to each bar, then annotate each with a list of the most common new words in that chapter.
MOST_COMMON_COUNT = 5
mycolor="#3E000C"
fig, ax = plt.subplots(figsize=(14, 25))
sns.set_style("whitegrid", {'axes.grid' : False})
sns.barplot('pct_new_vocab', chapterdata.index, data=chapterdata, orient='h', color=mycolor)
plt.grid(axis='x')
xax = ax.get_xaxis()
xax.set_label_position('top')
plt.xlabel("Percent New Vocabulary")
plt.ylabel('Chapter')
ax.xaxis.tick_top()
for chapnum in range(1, 58):
chaplbl = str(chapnum)
#barlbl = '%2.0f%% (%d)' % (chapterdata.pct_new_vocab[chapnum], chapterdata.new_vocabulary[chapnum])
barlbl = '%2.1f%%' % chapterdata.pct_new_vocab[chapnum]
plt.text(1, chapnum-1, barlbl,\
weight='bold', color='white', verticalalignment='center')
if chapnum >= 3:
newwords = ', '.join(['"%s" (%d)'%word for word in newvocab[chapnum].most_common(MOST_COMMON_COUNT)])
#the bbox param here gives the annotation a semitransparent white background to partially hide
# the gridlines behind it.
plt.text(chapterdata.pct_new_vocab[chapnum]+1, chapnum-1, newwords,\
verticalalignment='center', bbox=dict(facecolor='#ffffff99'))
plt.show()
