In [33]:
text3.dispersion_plot(["garden", "gate", "wings", "light", "god", "food"])
# this was not working on the online lab, but in the local one (anaconda) it works
# update: now it works on the online lab too
import nltk
from nltk.book import *
*** Introductory Examples for the NLTK Book *** Loading text1, ..., text9 and sent1, ..., sent9 Type the name of the text or sentence to view it. Type: 'texts()' or 'sents()' to list the materials. text1: Moby Dick by Herman Melville 1851 text2: Sense and Sensibility by Jane Austen 1811 text3: The Book of Genesis text4: Inaugural Address Corpus text5: Chat Corpus text6: Monty Python and the Holy Grail text7: Wall Street Journal text8: Personals Corpus text9: The Man Who Was Thursday by G . K . Chesterton 1908
text1
<Text: Moby Dick by Herman Melville 1851>
text1.concordance("pan")
Displaying 5 of 5 matches: aketh the seas to seethe like boiling pan ." -- LORD BACON ' S VERSION OF THE P e seductive god , bowing your head to Pan . But few thoughts of Pan stirred Aha our head to Pan . But few thoughts of Pan stirred Ahab ' s brain , as standing self , like a hot musket ' s powder - pan ." At last the shank , in one complet t see . Aye , aye ; and powder in the pan ;-- that ' s not good . Best spill it
text1.concordance("leg")
Displaying 25 of 89 matches: s much as to say -- I won ' t touch a leg of ye ." " Good night , landlord ," s not that lingo to me . Dost see that leg ?-- I ' ll take that leg away from th ost see that leg ?-- I ' ll take that leg away from thy stern , if ever thou ta d thou wilt find that he has only one leg ." " What do you mean , sir ? Was the w , too , that ever since he lost his leg last voyage by that accursed whale , , unless it was the cruel loss of his leg . And yet I also felt a strange awe o to ? And nothing about his losing his leg last voyage , according to the prophe mayhap , ye ' ve heard tell about the leg , and how he lost it ; aye , ye have t -- I mean they know he ' s only one leg ; and that a parmacetti took the othe that I know all about the loss of his leg ." " ALL about it , eh -- sure you do e Pequod ; and Captain Ahab ; and the leg he had lost ; and the Cape Horn fit ; n Peleg in the act of withdrawing his leg from my immediate vicinity . That was e windlass , here and there using his leg very freely , while imperturbable Bil mness was owing to the barbaric white leg upon which he partly stood . It had p previously come to me that this ivory leg had at sea been fashioned from the po nch or so , into the plank . His bone leg steadied in that hole ; one arm eleva had . You know the old man ' s ivory leg , well I dreamed he kicked me with it my soul , my little man , I kicked my leg right off ! And then , presto ! Ahab what ' s the row ? It ' s not a real leg , only a false leg .' And there ' s a It ' s not a real leg , only a false leg .' And there ' s a mighty difference s thinking to myself , ' what ' s his leg now , but a cane -- a whalebone cane good ,' says he --' he used his ivory leg , didn ' t he ?' ' Yes , he did ,' sa ill ? it wasn ' t a common pitch pine leg he kicked with , was it ? No , you we reat man , and with a beautiful ivory leg , Stubb . It ' s an honour ; I consid urpose on the upper part of his ivory leg . From his complete inattention to th
text1.concordance("bed")
Displaying 25 of 76 matches: feet on the hob quietly toasting for bed . " In judging of that tempestuous wi swer that his house was full -- not a bed unoccupied . " But avast ," he added that I never liked to sleep two in a bed ; that if I should ever do so , it wo gether , he must undress and get into bed before I did . Supper over , the comp en . No man prefers to sleep two in a bed . In fact , you would a good deal rat y I as a sailor should sleep two in a bed , more than anybody else ; for sailor ; for sailors no more sleep two in a bed at sea , than bachelor Kings do ashor vigorously set to planing away at my bed , the while grinning like an ape . Th im for heaven ' s sake to quit -- the bed was soft enough to suit me , and I di t his door inside , and jump into his bed , not to be wakened by the most viole night unless in some other person ' s bed , I began to think that after all I m es , twos , and threes , and going to bed , yet no sign of my harpooneer . " La lly he ' s an early bird -- airley to bed and airley to rise -- yes , he ' s th lay . I come to your house and want a bed ; you tell me you can only give me ha er be turning flukes -- it ' s a nice bed ; Sal and me slept in that ere bed th ce bed ; Sal and me slept in that ere bed the night we were spliced . There ' s of room for two to kick about in that bed ; it ' s an almighty big bed that . W in that bed ; it ' s an almighty big bed that . Why , afore we give it up , Sa hed , sure enough , with a prodigious bed , almost big enough indeed for any fo ye ." I turned round from eyeing the bed , but he had disappeared . Folding ba the counterpane , I stooped over the bed . Though none of the most elegant , i l harpoon standing at the head of the bed . But what is this on the chest ? I t neck . I sat down on the side of the bed , and commenced thinking about this h mat . After thinking some time on the bed - side , I got up and took off my mon en blowing out the light tumbled into bed , and commended myself to the care of
text1.concordance("harpoon")
Displaying 25 of 76 matches: hen they were nigh enough to risk a harpoon from the bowsprit ? Now having a ni n a sunrise and a sunset . And that harpoon -- so like a corkscrew now -- was f over the fire - place , and a tall harpoon standing at the head of the bed . B , when lo and behold , he takes the harpoon from the bed corner , slips out the ow of what fine steel the head of a harpoon is made , and how exceedingly sharp ot monkey jacket , and sporting his harpoon like a marshal ' s baton . CHAPTER ordially justified his bringing his harpoon into breakfast with him , and using topping to adjust the sheath on his harpoon barbs . I asked him why he carried a particular affection for his own harpoon , because it was of assured stuff , private reasons , preferred his own harpoon . Shifting the barrow from my hand our of doom was come . Dropping his harpoon , the brawny savage caught him in h e eel in your bowl ? Where ' s your harpoon ?" Fishiest of all fishy places was ed forth her arm , and demanded his harpoon ; she allowed no harpoon in her cha manded his harpoon ; she allowed no harpoon in her chambers . " Why not ? said every true whaleman sleeps with his harpoon -- but why not ?" " Because it ' s d in my first floor back , with his harpoon in his side ; ever since then I all . Now , art thou the man to pitch a harpoon down a live whale ' s throat , and ll the wooden shaft of Queequeg ' s harpoon , which the landlady the evening pr ght I ; but at any rate , since the harpoon stands yonder , and he seldom or ne turning , told me that Queequeg ' s harpoon was missing . " He ' s killed himse ds the ship , Queequeg carrying his harpoon , Captain Peleg in his gruff voice long . By the great anchor , what a harpoon he ' s got there ! looks like good ing his left knee , and poising his harpoon , cried out in some such way as thi at the close vicinity of the flying harpoon , had retreated towards the cabin g at whales of the sea ; the unerring harpoon of the son fitly replacing the infa
text1.concordance("supper")
Displaying 25 of 30 matches: ught so . All right ; take a seat . Supper ?-- you want supper ? Supper ' ll b ; take a seat . Supper ?-- you want supper ? Supper ' ll be ready directly ." seat . Supper ?-- you want supper ? Supper ' ll be ready directly ." I sat dow ings ; good heavens ! dumplings for supper ! One young fellow in a green box c ess and get into bed before I did . Supper over , the company went back to the e old rules would not apply . After supper , and another social chat and smoke ness that day , at least none but a supper and a bed . The landlord of the Spo Upon making known our desires for a supper and a bed , Mrs . Hussey , postponi Cod ?" she repeated . " A clam for supper ? a cold clam ; is THAT what you me do you think that we can make out a supper for us both on one clam ?" However howder for dinner , and chowder for supper , till you began to look for fish - ng very slip - shod , I assure ye . Supper concluded , we received a lamp , an very punctual then . I went down to supper . After sitting a long time listeni ake yourself ; get up and have some supper . You ' ll starve ; you ' ll kill y s day three years I ' ll have a hot supper smoking for ye in old Nantucket . H is safety , comfort , hearthstone , supper , warm blankets , friends , all tha he three soaked biscuits ye eat for supper turning over inside of ye -- nothin rily , hearts - alive . Pudding for supper , you know ;-- merry ' s the word . u would think there must be a brave supper cooking in the great bowels below . o be painted . CHAPTER 64 Stubb ' s Supper . Stubb ' s whale had been killed s stoutly stood up to his spermaceti supper at the capstan - head , as if that to form a more secure base for his supper ; and , at the same time darting hi on , Fleece , and I ' ll away to my supper ." Upon this , Fleece , holding bot , cook ," said Stubb , resuming his supper at the capstan ; " stand just where lled . " Cook , give me cutlets for supper to - morrow night in the mid - watc
text2.concordance("supper")
Displaying 4 of 4 matches: resolved to finish the basket after supper ." " You are very good , I hope it best for her . Let her name her own supper , and go to bed . Lord ! no wonder hold ten couple , and where can the supper be ?' I immediately saw that there nd other refreshments ; and let the supper be set out in the saloon .' Lady El
text5.concordance("soup")
Displaying 1 of 1 matches: ight now PART morning all it 's good soup U54 g' evening PART JOIN the time wh
text7.concordance("soup")
Displaying 7 of 7 matches: raised the ire of others . Campbell Soup , for one , is furious 0 its Souper fumes *T*-2 a spokesman for Campbell Soup . He says 0 Campbell was n't even co though , 0 it is n't one of Campbell Soup 's better products in terms of recyc in terms of recyclability . Campbell Soup , not surprisingly , does n't have a vern was forced *-37 out as Campbell Soup Co. 's president and chief executive e Associates , Fair Haven , Campbell Soup jumped 3 3\/8 to 47 1\/8 as the resi ank will make such a move . Campbell Soup forced out its president and chief e
text1.concordance("soup")
no matches
text1.concordance("food")
Displaying 14 of 14 matches: ueequeg seeing his favourite fishing food before him , and the chowder being s rning over the bottom of the sea for food . Charley Coffin said it was used fo jaws , the harpooneers chewed their food with such a relish that there was a is life on its summit , hoisting his food from the ground with a tackle ; in h the driftings of the sperm whale ' s food ; and , also , calling to mind the r ave every reason to believe that the food of the sperm whale -- squid or cuttl e to Moby Dick , they must also have food for their more common , daily appeti furnish to the sperm whale his only food . For though other species of whales h other species of whales find their food above water , and may be seen by man e spermaceti whale obtains his whole food in unknown zones below the surface ; can tell of what , precisely , that food consists . At times , when closely p s in water . But the Sperm Whale ' s food is far beneath the surface , and the . But the whaleman , as he seeks the food of light , so he lives in light . He not dawn upon that world . Here ' s food for thought , had Ahab time to think
text2.concordance("food")
Displaying 1 of 1 matches: from a long want of proper rest and food ; for it was many days since she had
text3.concordance("food")
Displaying 19 of 19 matches: pleasant to the sight , and good for food ; the tree of life also in the midst woman saw that the tree was good for food , and that it was pleasant to the ey ive . And take thou unto thee of all food that is eaten , and thou shalt gathe her it to thee ; and it shall be for food for thee , and for them . Thus did N years . And let them gather all the food of those good years that come , and hand of Pharaoh , and let them keep food in the cities . And that food shall m keep food in the cities . And that food shall be for store to the land again andfuls . And he gathered up all the food of the seven years , which were in t the land of Egypt , and laid up the food in the citi the food of the field , and laid up the food in the citi the food of the field , which was round about aid , From the land of Canaan to buy food . And Joseph knew his brethren , but nto him , Nay , my lord , but to buy food are thy servants come . We are all o our brethren here with me , and take food for the famine of your households , to them , Go again , buy us a little food . And Judah spake unto him , saying we brought down in our hands to buy food : we cannot tell who put our money i saying , Fill the men ' s sacks with food , as much as they can carry , and pu aid , Go again , and buy us a little food . And we said , We cannot go down : for seed of the field , and for your food , and for them of your households , or them of your households , and for food for your little ones . And they said
# melville
text1.similar('food')
head back bow hand mouth time crew sake body name eyes boat length work side stern voyages voyage coast names
# austen
text2.similar('food')
as this who wished hope marianne concern each return want gratitude fortunately esteem calmness respect warmth sometimes friendship debating chance
# genesis
text3.similar('food')
cattle us me god waters land days him field soul garden stone flesh father eyes thee hand way brother flock
text1.similar('monstrous')
true contemptible christian abundant few part mean careful puzzled mystifying passing curious loving wise doleful gamesome singular delightfully perilous fearless
text2.similar('monstrous')
very so exceedingly heartily a as good great extremely remarkably sweet vast amazingly
text2.common_contexts(["monstrous", "very"])
am_glad a_pretty a_lucky is_pretty be_glad
text2.common_contexts(["food", "light"])
No common contexts were found
text2.similar("light")
in was as other such however capable is well when fond little highly till eagerness excess replied you has mention
text2.common_contexts('well')
('The following word(s) were not found:', 'w e l l')
text2.similar('well')
much and little but it not that long to soon is now happy great only far right ever i pretty
text2.common_contexts(['light', 'well'])
very_of
from nltk.draw import *
text3.dispersion_plot(["garden", "gate", "wings", "light", "god", "food"])
# this was not working on the online lab, but in the local one (anaconda) it works
# update: now it works on the online lab too
pip install --user -U numpy
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (1.21.2) Note: you may need to restart the kernel to use updated packages.
import numpy
from numpy import *
from '/usr/local/lib/python3.7/dist-packages' import numpy
File "/tmp/ipykernel_13801/903367552.py", line 1 from '/usr/local/lib/python3.7/dist-packages' import numpy ^ SyntaxError: invalid syntax
# Install a pip package in the current Jupyter kernel
# https://jakevdp.github.io/blog/2017/12/05/installing-python-packages-from-jupyter/
import sys
!{sys.executable} -m pip install numpy
sys.path
--------------------------------------------------------------------------- NameError Traceback (most recent call last) /tmp/ipykernel_13801/3374087864.py in <module> ----> 1 sys.path NameError: name 'sys' is not defined
import numpy
numpy.__path__
# ohhh nevermind - - - this is where @manetta and @michael enter the stage
text1.generate()
Building ngram index...
long , from one to the top - mast , and no coffin and went out a sea captain -- this peaking of the whales . , so as to preserve all his might had in former years abounding with them , they toil with their lances , strange tales of Southern whaling . at once the bravest Indians he was , after in vain strove to pierce the profundity . ? then ?" a levelled flame of pale , And give no chance , watch him ; though the line , it is to be gainsaid . have been
'long , from one to the top - mast , and no coffin and went out a sea\ncaptain -- this peaking of the whales . , so as to preserve all his\nmight had in former years abounding with them , they toil with their\nlances , strange tales of Southern whaling . at once the bravest\nIndians he was , after in vain strove to pierce the profundity . ?\nthen ?" a levelled flame of pale , And give no chance , watch him ;\nthough the line , it is to be gainsaid . have been'
text3.generate()
Building ngram index...
laid by her , and said unto Cain , Where art thou , and said , Go to , I will not do it for ten ' s sons ; we dreamed each man according to their generatio the firstborn said unto Laban , Because I said , Nay , but Sarah shall her name be . , duke Elah , duke Shobal , and Akan . and looked upon my affliction . Bashemath Ishmael ' s blood , but Isra for as a prince hast thou found of all the cattle in the valley , and the wo The
"laid by her , and said unto Cain , Where art thou , and said , Go to ,\nI will not do it for ten ' s sons ; we dreamed each man according to\ntheir generatio the firstborn said unto Laban , Because I said , Nay ,\nbut Sarah shall her name be . , duke Elah , duke Shobal , and Akan .\nand looked upon my affliction . Bashemath Ishmael ' s blood , but Isra\nfor as a prince hast thou found of all the cattle in the valley , and\nthe wo The"
len(text3)
44764
len(text1)
260819
len(text6)
16967
set(text3)
{'strengthened',
'Mizraim',
'afterwards',
'seest',
'h',
'pitcher',
'servan',
'foals',
'four',
'Jegarsahadutha',
'mercies',
'meal',
'walked',
'Matred',
'sole',
'prosper',
'Canaanitish',
'heav',
'itself',
'Jamin',
'elder',
'peaceable',
'Pinon',
'Merari',
'Bela',
'kill',
'Methusael',
'Shuah',
'Happy',
'drank',
'Amraphel',
'declare',
'Medan',
'knead',
'The',
'Gomorrah',
'northward',
'We',
'least',
'How',
'help',
'whatsoever',
'spake',
'Obal',
'smooth',
'Eliphaz',
'weaned',
'yet',
'shoulders',
'buried',
'Asshur',
'Mizz',
'Naamah',
'ground',
'secretly',
'plenteous',
'Gershon',
'troop',
'Gihon',
'sure',
'likeness',
'Penuel',
'fifteen',
'prison',
'vowedst',
'Shelah',
'sakes',
'year',
'better',
'Nahor',
'Tamar',
'Japheth',
'when',
'Zeboiim',
'openly',
'images',
'river',
'Shuni',
'Girgashites',
'assembly',
'Euphrat',
'Seeing',
'serpent',
'knees',
'waited',
'serve',
'gone',
'Hagar',
'fury',
'worth',
'Beor',
'riv',
'hence',
'flock',
'youngest',
'gat',
'restrained',
'wherefore',
'husbandman',
'Moriah',
'beyond',
'go',
'sinew',
'Timna',
'Mam',
'choice',
'willing',
'instructor',
'wages',
'sworn',
'praise',
'fine',
'camel',
'maid',
'deceiver',
'cruelty',
'staff',
'lead',
'grow',
'without',
'Hadoram',
'Night',
'Kadesh',
'nigh',
'called',
'chode',
'occasion',
'dress',
'kindled',
'prayed',
'beast',
'Jahleel',
'aga',
'embalmed',
'goa',
'Reuel',
'Sidon',
'Leah',
'trade',
'therefore',
'shrank',
'could',
'oth',
'refrain',
'lieth',
'Karnaim',
'heap',
'Hadar',
'guiltiness',
'Zar',
'Egy',
'Edom',
'Kadmonites',
'send',
'laden',
'endure',
'Omar',
'chose',
'pleaseth',
'fruits',
'daughers',
'Alvan',
'fourteen',
'brink',
'whole',
'Kedemah',
'led',
'Elparan',
'Phuvah',
'hired',
'thigh',
'dignity',
'fall',
';)',
'abated',
'provender',
'meditate',
'Leummim',
'Zeboim',
'lingered',
'See',
'come',
'bak',
'interpreter',
'pla',
'Discern',
'surety',
'sod',
'Philistim',
'tempt',
'ark',
'draw',
'fo',
'prosperous',
'Birsha',
'Euphrates',
'done',
'Manahath',
'Mahanaim',
'pleasant',
'tru',
'verily',
'died',
'hou',
'this',
'Adah',
'hire',
'folly',
'deferred',
'hath',
'Arodi',
'till',
'found',
'Jerah',
'fulfilled',
'Achbor',
'wind',
'exceedingly',
'content',
'feebler',
'money',
'En',
'lien',
'Naphtuhim',
'day',
'Beeri',
'weight',
'Succoth',
'pleasure',
'uppermost',
'loveth',
'pillar',
'vail',
'Egyptian',
'rent',
'Emins',
'buy',
'Naaman',
'Edar',
')',
'great',
'filled',
'too',
'Say',
'judge',
'Almodad',
'searched',
'am',
'caused',
'then',
'overcome',
'O',
'void',
'thus',
'Kohath',
'Jacob',
'peradventure',
'concubi',
'broken',
'violence',
'Tema',
'generation',
'harlot',
'husba',
'thereof',
'Huppim',
'like',
'henceforth',
'little',
'can',
'Asshurim',
'shew',
'Can',
'numbering',
'beguiled',
'husband',
'lovest',
'Hezron',
'issue',
'Pharaoh',
'messengers',
'it',
'guard',
'garment',
'interpretations',
'mightier',
'deprived',
'hasted',
'Ammon',
'bought',
'brimstone',
'Admah',
'Ludim',
'seeing',
'themselv',
'righteousness',
'Kirjatharba',
'Until',
'become',
'ma',
'Isra',
'tenor',
'couch',
'Look',
'flocks',
'Abide',
'sheweth',
'loose',
'set',
'myself',
'marry',
'shortly',
'Akan',
'dread',
'pea',
'loss',
'breach',
'Sojourn',
'leanfleshed',
'iniquity',
'furnace',
',',
'da',
'lamb',
'Rephaims',
'gard',
'Bring',
'Hiddekel',
'Even',
'roughly',
'Wilt',
'Ard',
'time',
'rebuked',
'in',
'feeble',
'Ur',
"'",
'brick',
'meet',
'shepherd',
'angry',
'wit',
'knife',
'Moreh',
'Angel',
'current',
'bands',
'daughters',
'into',
'male',
'hand',
'budded',
'Goshen',
'seedtime',
'hairy',
'was',
'overtook',
'denied',
'borders',
'Only',
'scarlet',
'consume',
'saying',
'prevail',
'Magog',
'seventy',
'dealt',
'Whose',
'Gilead',
'torn',
'camels',
'sit',
'firmament',
'wise',
'EleloheIsrael',
'yea',
'presence',
'who',
'corrupt',
'few',
'eyed',
'Salem',
'Rameses',
'Moabites',
'escape',
'younge',
'dreadful',
'forget',
'Paran',
'his',
'magicians',
'mine',
'Bilhan',
'grown',
'pitch',
'Ishbak',
'Ebal',
'nations',
'bowels',
'messenger',
'wrong',
'covenant',
'dream',
'Ephah',
'Bow',
'yourselves',
'blesseth',
'changed',
'gifts',
'Ashbel',
'Elishah',
'stranger',
'Issachar',
'Joseph',
'refused',
'firstlings',
'sware',
'Should',
'sackcloth',
'By',
'beheld',
'cloud',
'dwelled',
'ours',
'despised',
'hollow',
'Pildash',
'offeri',
'age',
'Thou',
'married',
'Tebah',
'thi',
'troughs',
'reproach',
'Shinar',
'tongue',
'Now',
'Enmishpat',
'Stand',
'greater',
'between',
'skins',
'millions',
'neck',
'call',
'hastily',
'your',
'enlarge',
'Areli',
'morever',
'commandment',
'number',
'wrestled',
'earring',
'butler',
'spies',
'rider',
'Jobab',
'sheep',
'Oh',
'under',
'sevens',
'might',
'failed',
'keep',
'slayeth',
'Lot',
'stood',
'what',
'abroad',
'lesser',
'sacks',
'breed',
'nought',
'hear',
'speckl',
'pledge',
'ir',
'Perizzit',
'shore',
'n',
'dreamer',
'dungeon',
'Zaavan',
'we',
'waxen',
'full',
'Mezahab',
'they',
'wells',
'bear',
'fear',
'enemies',
'tarried',
'leaped',
'surely',
'continually',
'deliverance',
'grew',
'Beersheba',
'servants',
'knowing',
'knoweth',
'throughout',
'poured',
'separate',
'charged',
'laugh',
'eldest',
'our',
'wondering',
'Sephar',
'dry',
'butlers',
'speech',
'bread',
'nothing',
'certain',
'honey',
'Zidon',
'harvest',
'befall',
'carry',
'unawares',
'haven',
'Thorns',
'Siddim',
'Elah',
'peop',
'Sodom',
'fill',
'down',
'Shall',
'Onan',
'seas',
'should',
'Bashemath',
'white',
'foremost',
'ride',
'weapons',
'fema',
'Yet',
'longedst',
'tender',
'stink',
'rightly',
'foreskin',
'pit',
'So',
'repented',
'stayed',
'In',
'other',
'widow',
'seasons',
'solemnly',
'answer',
'Elbethel',
'Arvadite',
'sow',
'household',
'hearth',
'Bilhah',
'became',
'far',
'mi',
'Hamul',
'room',
'Ararat',
'confound',
'most',
'removing',
'whoredom',
'colts',
'truth',
'To',
'couching',
'sanctified',
'beautiful',
'destroyed',
'Tell',
'Let',
'entreated',
'Methusa',
'bitter',
'begin',
'Give',
'here',
'while',
'Abelmizraim',
'mocking',
'flee',
'comforted',
'Eri',
'know',
'finding',
'mean',
'last',
'Haran',
'until',
'harm',
'Becher',
'Send',
'inn',
'lights',
'waxed',
'Elon',
'seven',
'burdens',
'Potiphar',
'd',
'waters',
'seek',
'Tubalcain',
'neither',
'goods',
'compassed',
'roof',
'silv',
'mother',
'heir',
'duke',
'giving',
'brought',
'Timnath',
'were',
'deal',
'those',
'Israel',
'why',
'withered',
'sad',
'bad',
'enough',
'eleven',
'third',
'these',
'ended',
'accepted',
'wives',
'golden',
'soon',
'arose',
'Pass',
'lan',
'Ophir',
'fire',
'slimepits',
'multitude',
'Zebulun',
'signs',
'gavest',
'spent',
'work',
'ewes',
'bade',
'treasure',
'Who',
'Beware',
'seventh',
'southward',
'awaked',
'joined',
'goats',
'mead',
'Gatam',
'booths',
'off',
'Horite',
'week',
'committed',
'As',
'state',
'fair',
'Not',
'office',
'hearkened',
'overthrew',
'drought',
'handmaidens',
'organ',
'togeth',
'love',
'Shechem',
'received',
'Ahuzzath',
'of',
'joint',
'touch',
'ready',
'Hanoch',
'Both',
'see',
'sack',
'lo',
'corn',
'face',
'spread',
'wentest',
'seeth',
'city',
'statutes',
'strife',
'needeth',
'prince',
'Benam',
'saved',
'Amal',
'salvation',
'cannot',
'hate',
'land',
'purchased',
'Which',
'circumcise',
'shoelatchet',
'nostrils',
'thread',
'th',
'worship',
'Deborah',
'colours',
'open',
'beside',
'Heaven',
'thirteenth',
'night',
'each',
'speaking',
'smoke',
'hands',
'Hamathite',
'whose',
'Philistines',
'floor',
'horror',
'moon',
'handmaid',
'males',
'hundredfo',
'through',
'tell',
'Whereas',
'dea',
'Esau',
'Dothan',
'overspread',
'vow',
'changes',
'affliction',
'Zilpah',
'sac',
'Din',
'one',
'perpetual',
'That',
'spoil',
'fast',
'Forgive',
'moreover',
'Judith',
'Appoint',
'behold',
'sinners',
'sepulchre',
'fierce',
'Almighty',
'bough',
'vestures',
'Assyr',
'Kiriathaim',
'perform',
'none',
'felt',
'yielded',
'brown',
'Zibeon',
'two',
'earing',
'Whence',
'fatfleshed',
'visions',
'bondman',
'sist',
'hardly',
'Sinite',
'fourscore',
'sent',
'wot',
'company',
'double',
'wroth',
'vagabond',
'herds',
'Rebekah',
'flood',
'think',
'laid',
'he',
'Shimron',
'wood',
'mistress',
'purchase',
'either',
'poor',
'sawest',
'He',
'victuals',
'dipped',
'Rosh',
'wash',
'flo',
'years',
'heel',
'thereon',
'.',
'air',
'Salah',
'aileth',
'met',
'Behold',
'fathers',
'Mizpah',
'overtake',
'Laban',
'li',
'prisoners',
'ill',
'Tola',
'womenservants',
'fountains',
'discreet',
'rams',
'kings',
'Eber',
'frost',
'espied',
'bondwoman',
'songs',
'named',
'guilty',
'kine',
'clave',
'began',
'some',
'scarce',
'LO',
'reward',
'oak',
'earth',
'Feed',
'touched',
'weep',
'wandered',
'wilderness',
'wilt',
'Do',
'unit',
'steward',
'bow',
'blessings',
'comi',
'goest',
'art',
'tongues',
'five',
'mark',
'Elam',
'persons',
'honour',
'Cainan',
'observed',
'erected',
'Ephra',
'twel',
'stone',
'obtain',
'myrrh',
'Jabal',
'princes',
'maidservants',
'grass',
'hide',
'watering',
'Judge',
'citi',
'wall',
'eyes',
'na',
'name',
'obey',
'fruitful',
'hills',
'Malchiel',
'herself',
'springing',
'Me',
'gre',
'carried',
'words',
'savour',
'sand',
'deceitfully',
'six',
'heard',
'mourn',
'prospered',
'book',
'cried',
'lying',
'sat',
'whether',
'Male',
'tree',
'himself',
'Fill',
'portion',
'Naphtali',
'Kenaz',
'alone',
'Woman',
'Mehetabel',
'deeds',
'haste',
'travailed',
'builded',
'fifth',
'restore',
'Peleg',
'an',
'back',
'violently',
'tithes',
'Kenizzites',
'get',
'to',
'silver',
'dove',
'nig',
'handfuls',
'direct',
'run',
'up',
'health',
'trained',
'Horites',
'morsel',
'rained',
'thine',
'battle',
'priest',
'jud',
'sto',
'It',
'chi',
'Shem',
'gutters',
'Hebrews',
'Keturah',
'Eshcol',
'distress',
'so',
'twenty',
'Phara',
'Tidal',
',)',
'wearied',
'abide',
'strange',
'Reumah',
'Nineveh',
'established',
'devoured',
'mules',
'blame',
'on',
'saw',
'Serug',
'or',
'required',
'fish',
'lands',
'cometh',
'alive',
'Pathrusim',
'receive',
'just',
'ou',
'kn',
'right',
'foolishly',
'Unstable',
'Hitti',
'answered',
'slime',
'mouth',
'drove',
...}
sorted(set(text3))
['!',
"'",
'(',
')',
',',
',)',
'.',
'.)',
':',
';',
';)',
'?',
'?)',
'A',
'Abel',
'Abelmizraim',
'Abidah',
'Abide',
'Abimael',
'Abimelech',
'Abr',
'Abrah',
'Abraham',
'Abram',
'Accad',
'Achbor',
'Adah',
'Adam',
'Adbeel',
'Admah',
'Adullamite',
'After',
'Aholibamah',
'Ahuzzath',
'Ajah',
'Akan',
'All',
'Allonbachuth',
'Almighty',
'Almodad',
'Also',
'Alvah',
'Alvan',
'Am',
'Amal',
'Amalek',
'Amalekites',
'Ammon',
'Amorite',
'Amorites',
'Amraphel',
'An',
'Anah',
'Anamim',
'And',
'Aner',
'Angel',
'Appoint',
'Aram',
'Aran',
'Ararat',
'Arbah',
'Ard',
'Are',
'Areli',
'Arioch',
'Arise',
'Arkite',
'Arodi',
'Arphaxad',
'Art',
'Arvadite',
'As',
'Asenath',
'Ashbel',
'Asher',
'Ashkenaz',
'Ashteroth',
'Ask',
'Asshur',
'Asshurim',
'Assyr',
'Assyria',
'At',
'Atad',
'Avith',
'Baalhanan',
'Babel',
'Bashemath',
'Be',
'Because',
'Becher',
'Bedad',
'Beeri',
'Beerlahairoi',
'Beersheba',
'Behold',
'Bela',
'Belah',
'Benam',
'Benjamin',
'Beno',
'Beor',
'Bera',
'Bered',
'Beriah',
'Bethel',
'Bethlehem',
'Bethuel',
'Beware',
'Bilhah',
'Bilhan',
'Binding',
'Birsha',
'Bless',
'Blessed',
'Both',
'Bow',
'Bozrah',
'Bring',
'But',
'Buz',
'By',
'Cain',
'Cainan',
'Calah',
'Calneh',
'Can',
'Cana',
'Canaan',
'Canaanite',
'Canaanites',
'Canaanitish',
'Caphtorim',
'Carmi',
'Casluhim',
'Cast',
'Cause',
'Chaldees',
'Chedorlaomer',
'Cheran',
'Cherubims',
'Chesed',
'Chezib',
'Come',
'Cursed',
'Cush',
'Damascus',
'Dan',
'Day',
'Deborah',
'Dedan',
'Deliver',
'Diklah',
'Din',
'Dinah',
'Dinhabah',
'Discern',
'Dishan',
'Dishon',
'Do',
'Dodanim',
'Dothan',
'Drink',
'Duke',
'Dumah',
'Earth',
'Ebal',
'Eber',
'Edar',
'Eden',
'Edom',
'Edomites',
'Egy',
'Egypt',
'Egyptia',
'Egyptian',
'Egyptians',
'Ehi',
'Elah',
'Elam',
'Elbethel',
'Eldaah',
'EleloheIsrael',
'Eliezer',
'Eliphaz',
'Elishah',
'Ellasar',
'Elon',
'Elparan',
'Emins',
'En',
'Enmishpat',
'Eno',
'Enoch',
'Enos',
'Ephah',
'Epher',
'Ephra',
'Ephraim',
'Ephrath',
'Ephron',
'Er',
'Erech',
'Eri',
'Es',
'Esau',
'Escape',
'Esek',
'Eshban',
'Eshcol',
'Ethiopia',
'Euphrat',
'Euphrates',
'Eve',
'Even',
'Every',
'Except',
'Ezbon',
'Ezer',
'Fear',
'Feed',
'Fifteen',
'Fill',
'For',
'Forasmuch',
'Forgive',
'From',
'Fulfil',
'G',
'Gad',
'Gaham',
'Galeed',
'Gatam',
'Gather',
'Gaza',
'Gentiles',
'Gera',
'Gerar',
'Gershon',
'Get',
'Gether',
'Gihon',
'Gilead',
'Girgashites',
'Girgasite',
'Give',
'Go',
'God',
'Gomer',
'Gomorrah',
'Goshen',
'Guni',
'Hadad',
'Hadar',
'Hadoram',
'Hagar',
'Haggi',
'Hai',
'Ham',
'Hamathite',
'Hamor',
'Hamul',
'Hanoch',
'Happy',
'Haran',
'Hast',
'Haste',
'Have',
'Havilah',
'Hazarmaveth',
'Hazezontamar',
'Hazo',
'He',
'Hear',
'Heaven',
'Heber',
'Hebrew',
'Hebrews',
'Hebron',
'Hemam',
'Hemdan',
'Here',
'Hereby',
'Heth',
'Hezron',
'Hiddekel',
'Hinder',
'Hirah',
'His',
'Hitti',
'Hittite',
'Hittites',
'Hivite',
'Hobah',
'Hori',
'Horite',
'Horites',
'How',
'Hul',
'Huppim',
'Husham',
'Hushim',
'Huz',
'I',
'If',
'In',
'Irad',
'Iram',
'Is',
'Isa',
'Isaac',
'Iscah',
'Ishbak',
'Ishmael',
'Ishmeelites',
'Ishuah',
'Isra',
'Israel',
'Issachar',
'Isui',
'It',
'Ithran',
'Jaalam',
'Jabal',
'Jabbok',
'Jac',
'Jachin',
'Jacob',
'Jahleel',
'Jahzeel',
'Jamin',
'Japhe',
'Japheth',
'Jared',
'Javan',
'Jebusite',
'Jebusites',
'Jegarsahadutha',
'Jehovahjireh',
'Jemuel',
'Jerah',
'Jetheth',
'Jetur',
'Jeush',
'Jezer',
'Jidlaph',
'Jimnah',
'Job',
'Jobab',
'Jokshan',
'Joktan',
'Jordan',
'Joseph',
'Jubal',
'Judah',
'Judge',
'Judith',
'Kadesh',
'Kadmonites',
'Karnaim',
'Kedar',
'Kedemah',
'Kemuel',
'Kenaz',
'Kenites',
'Kenizzites',
'Keturah',
'Kiriathaim',
'Kirjatharba',
'Kittim',
'Know',
'Kohath',
'Kor',
'Korah',
'LO',
'LORD',
'Laban',
'Lahairoi',
'Lamech',
'Lasha',
'Lay',
'Leah',
'Lehabim',
'Lest',
'Let',
'Letushim',
'Leummim',
'Levi',
'Lie',
'Lift',
'Lo',
'Look',
'Lot',
'Lotan',
'Lud',
'Ludim',
'Luz',
'Maachah',
'Machir',
'Machpelah',
'Madai',
'Magdiel',
'Magog',
'Mahalaleel',
'Mahalath',
'Mahanaim',
'Make',
'Malchiel',
'Male',
'Mam',
'Mamre',
'Man',
'Manahath',
'Manass',
'Manasseh',
'Mash',
'Masrekah',
'Massa',
'Matred',
'Me',
'Medan',
'Mehetabel',
'Mehujael',
'Melchizedek',
'Merari',
'Mesha',
'Meshech',
'Mesopotamia',
'Methusa',
'Methusael',
'Methuselah',
'Mezahab',
'Mibsam',
'Mibzar',
'Midian',
'Midianites',
'Milcah',
'Mishma',
'Mizpah',
'Mizraim',
'Mizz',
'Moab',
'Moabites',
'Moreh',
'Moreover',
'Moriah',
'Muppim',
'My',
'Naamah',
'Naaman',
'Nahath',
'Nahor',
'Naphish',
'Naphtali',
'Naphtuhim',
'Nay',
'Nebajoth',
'Neither',
'Night',
'Nimrod',
'Nineveh',
'Noah',
'Nod',
'Not',
'Now',
'O',
'Obal',
'Of',
'Oh',
'Ohad',
'Omar',
'On',
'Onam',
'Onan',
'Only',
'Ophir',
'Our',
'Out',
'Padan',
'Padanaram',
'Paran',
'Pass',
'Pathrusim',
'Pau',
'Peace',
'Peleg',
'Peniel',
'Penuel',
'Peradventure',
'Perizzit',
'Perizzite',
'Perizzites',
'Phallu',
'Phara',
'Pharaoh',
'Pharez',
'Phichol',
'Philistim',
'Philistines',
'Phut',
'Phuvah',
'Pildash',
'Pinon',
'Pison',
'Potiphar',
'Potipherah',
'Put',
'Raamah',
'Rachel',
'Rameses',
'Rebek',
'Rebekah',
'Rehoboth',
'Remain',
'Rephaims',
'Resen',
'Return',
'Reu',
'Reub',
'Reuben',
'Reuel',
'Reumah',
'Riphath',
'Rosh',
'Sabtah',
'Sabtech',
'Said',
'Salah',
'Salem',
'Samlah',
'Sarah',
'Sarai',
'Saul',
'Save',
'Say',
'Se',
'Seba',
'See',
'Seeing',
'Seir',
'Sell',
'Send',
'Sephar',
'Serah',
'Sered',
'Serug',
'Set',
'Seth',
'Shalem',
'Shall',
'Shalt',
'Shammah',
'Shaul',
'Shaveh',
'She',
'Sheba',
'Shebah',
'Shechem',
'Shed',
'Shel',
'Shelah',
'Sheleph',
'Shem',
'Shemeber',
'Shepho',
'Shillem',
'Shiloh',
'Shimron',
'Shinab',
'Shinar',
'Shobal',
'Should',
'Shuah',
'Shuni',
'Shur',
'Sichem',
'Siddim',
'Sidon',
'Simeon',
'Sinite',
'Sitnah',
'Slay',
'So',
'Sod',
'Sodom',
'Sojourn',
'Some',
'Spake',
'Speak',
'Spirit',
'Stand',
'Succoth',
'Surely',
'Swear',
'Syrian',
'Take',
'Tamar',
'Tarshish',
'Tebah',
'Tell',
'Tema',
'Teman',
'Temani',
'Terah',
'Thahash',
'That',
'The',
'Then',
'There',
'Therefore',
'These',
'They',
'Thirty',
'This',
'Thorns',
'Thou',
'Thus',
'Thy',
'Tidal',
'Timna',
'Timnah',
'Timnath',
'Tiras',
'To',
'Togarmah',
'Tola',
'Tubal',
'Tubalcain',
'Twelve',
'Two',
'Unstable',
'Until',
'Unto',
'Up',
'Upon',
'Ur',
'Uz',
'Uzal',
'We',
'What',
'When',
'Whence',
'Where',
'Whereas',
'Wherefore',
'Which',
'While',
'Who',
'Whose',
'Whoso',
'Why',
'Wilt',
'With',
'Woman',
'Ye',
'Yea',
'Yet',
'Zaavan',
'Zaphnathpaaneah',
'Zar',
'Zarah',
'Zeboiim',
'Zeboim',
'Zebul',
'Zebulun',
'Zemarite',
'Zepho',
'Zerah',
'Zibeon',
'Zidon',
'Zillah',
'Zilpah',
'Zimran',
'Ziphion',
'Zo',
'Zoar',
'Zohar',
'Zuzims',
'a',
'abated',
'abide',
'able',
'abode',
'abomination',
'about',
'above',
'abroad',
'absent',
'abundantly',
'accept',
'accepted',
'according',
'acknowledged',
'activity',
'add',
'adder',
'afar',
'afflict',
'affliction',
'afraid',
'after',
'afterward',
'afterwards',
'aga',
'again',
'against',
'age',
'aileth',
'air',
'al',
'alive',
'all',
'almon',
'alo',
'alone',
'aloud',
'also',
'altar',
'altogether',
'always',
'am',
'among',
'amongst',
'an',
'and',
'angel',
'angels',
'anger',
'angry',
'anguish',
'anointedst',
'anoth',
'another',
'answer',
'answered',
'any',
'anything',
'appe',
'appear',
'appeared',
'appease',
'appoint',
'appointed',
'aprons',
'archer',
'archers',
'are',
'arise',
'ark',
'armed',
'arms',
'army',
'arose',
'arrayed',
'art',
'artificer',
'as',
'ascending',
'ash',
'ashamed',
'ask',
'asked',
'asketh',
'ass',
'assembly',
'asses',
'assigned',
'asswaged',
'at',
'attained',
'audience',
'avenged',
'aw',
'awaked',
'away',
'awoke',
'back',
'backward',
'bad',
'bade',
'badest',
'badne',
'bak',
'bake',
'bakemeats',
'baker',
'bakers',
'balm',
'bands',
'bank',
'bare',
'barr',
'barren',
'basket',
'baskets',
'battle',
'bdellium',
'be',
'bear',
'beari',
'bearing',
'beast',
'beasts',
'beautiful',
'became',
'because',
'become',
'bed',
'been',
'befall',
'befell',
'before',
'began',
'begat',
'beget',
'begettest',
'begin',
'beginning',
'begotten',
'beguiled',
'beheld',
'behind',
'behold',
'being',
'believed',
'belly',
'belong',
'beneath',
'bereaved',
'beside',
'besides',
'besought',
'best',
'betimes',
'better',
'between',
'betwixt',
'beyond',
'binding',
'bird',
'birds',
'birthday',
'birthright',
'biteth',
'bitter',
'blame',
'blameless',
'blasted',
'bless',
'blessed',
'blesseth',
'blessi',
'blessing',
'blessings',
'blindness',
'blood',
'blossoms',
'bodies',
'boldly',
'bondman',
'bondmen',
'bondwoman',
'bone',
'bones',
'book',
'booths',
'border',
'borders',
'born',
'bosom',
'both',
'bottle',
'bou',
'boug',
'bough',
'bought',
'bound',
'bow',
'bowed',
'bowels',
'bowing',
'boys',
'bracelets',
'branches',
'brass',
'bre',
'breach',
'bread',
'breadth',
'break',
'breaketh',
'breaking',
'breasts',
'breath',
'breathed',
'breed',
'brethren',
'brick',
'brimstone',
'bring',
'brink',
'broken',
'brook',
'broth',
'brother',
'brought',
'brown',
'bruise',
'budded',
'build',
'builded',
'built',
'bulls',
'bundle',
'bundles',
'burdens',
'buried',
'burn',
'burning',
'burnt',
'bury',
'buryingplace',
'business',
'but',
'butler',
'butlers',
'butlership',
'butter',
'buy',
'by',
'cakes',
'calf',
'call',
'called',
'came',
'camel',
'camels',
'camest',
'can',
'cannot',
'canst',
'captain',
'captive',
'captives',
'carcases',
'carried',
'carry',
'cast',
'castles',
'catt',
'cattle',
'caught',
'cause',
'caused',
'cave',
'cease',
'ceased',
'certain',
'certainly',
'chain',
'chamber',
'change',
'changed',
'changes',
'charge',
'charged',
'chariot',
'chariots',
'chesnut',
'chi',
'chief',
'child',
'childless',
'childr',
'children',
'chode',
'choice',
'chose',
'circumcis',
'circumcise',
'circumcised',
'citi',
'cities',
'city',
'clave',
'clean',
'clear',
'cleave',
'clo',
'closed',
'clothed',
'clothes',
'cloud',
'clusters',
'co',
'coat',
'coats',
'coffin',
'cold',
...]
len(set(text3))
2789
len(set(text1))
19317
# lexical richness
len(set(text3)) / len(text3)
0.06230453042623537
len(set(text2)) / len(text2)
0.04826383002768831
text2.count("light")
11
text2.concordance('halo')
no matches
text2.concordance('food')
Displaying 1 of 1 matches: from a long want of proper rest and food ; for it was many days since she had
text2.count('food')
1
text2.concordance('sheath')
Displaying 1 of 1 matches: while spoiling both them and their sheath by cutting the latter to pieces as
text3.count('food')
19
text1.count('food')
14
def lexical_diversity(text):
return len(set(text)) / len(text)
def percentage(count, total):
return 100 * count / total
lexical_diversity(text1)
0.07406285585022564
lexical_diversity(text2)
0.04826383002768831
percentage(text3.count('garden'), len(text3))
0.029041193816459657
sent1
['Call', 'me', 'Ishmael', '.']
sent2
['The', 'family', 'of', 'Dashwood', 'had', 'long', 'been', 'settled', 'in', 'Sussex', '.']
sent3
['In', 'the', 'beginning', 'God', 'created', 'the', 'heaven', 'and', 'the', 'earth', '.']
sent4
['Fellow', '-', 'Citizens', 'of', 'the', 'Senate', 'and', 'of', 'the', 'House', 'of', 'Representatives', ':']
sent5
['I', 'have', 'a', 'problem', 'with', 'people', 'PMing', 'me', 'to', 'lol', 'JOIN']
sent1.append('soup')
sent1
['Call', 'me', 'Ishmael', '.', 'soup']
text3[124]
'waters'
text3[123]
'the'
text2[420:666]
['old', 'gentleman', 'died', ':', 'his', 'will', 'was', 'read', ',', 'and', 'like', 'almost', 'every', 'other', 'will', ',', 'gave', 'as', 'much', 'disappointment', 'as', 'pleasure', '.', 'He', 'was', 'neither', 'so', 'unjust', ',', 'nor', 'so', 'ungrateful', ',', 'as', 'to', 'leave', 'his', 'estate', 'from', 'his', 'nephew', ';--', 'but', 'he', 'left', 'it', 'to', 'him', 'on', 'such', 'terms', 'as', 'destroyed', 'half', 'the', 'value', 'of', 'the', 'bequest', '.', 'Mr', '.', 'Dashwood', 'had', 'wished', 'for', 'it', 'more', 'for', 'the', 'sake', 'of', 'his', 'wife', 'and', 'daughters', 'than', 'for', 'himself', 'or', 'his', 'son', ';--', 'but', 'to', 'his', 'son', ',', 'and', 'his', 'son', "'", 's', 'son', ',', 'a', 'child', 'of', 'four', 'years', 'old', ',', 'it', 'was', 'secured', ',', 'in', 'such', 'a', 'way', ',', 'as', 'to', 'leave', 'to', 'himself', 'no', 'power', 'of', 'providing', 'for', 'those', 'who', 'were', 'most', 'dear', 'to', 'him', ',', 'and', 'who', 'most', 'needed', 'a', 'provision', 'by', 'any', 'charge', 'on', 'the', 'estate', ',', 'or', 'by', 'any', 'sale', 'of', 'its', 'valuable', 'woods', '.', 'The', 'whole', 'was', 'tied', 'up', 'for', 'the', 'benefit', 'of', 'this', 'child', ',', 'who', ',', 'in', 'occasional', 'visits', 'with', 'his', 'father', 'and', 'mother', 'at', 'Norland', ',', 'had', 'so', 'far', 'gained', 'on', 'the', 'affections', 'of', 'his', 'uncle', ',', 'by', 'such', 'attractions', 'as', 'are', 'by', 'no', 'means', 'unusual', 'in', 'children', 'of', 'two', 'or', 'three', 'years', 'old', ';', 'an', 'imperfect', 'articulation', ',', 'an', 'earnest', 'desire', 'of', 'having', 'his', 'own', 'way', ',', 'many', 'cunning', 'tricks', ',', 'and', 'a', 'great', 'deal', 'of', 'noise', ',', 'as', 'to', 'outweigh', 'all', 'the', 'value', 'of', 'all', 'the', 'attention', 'which', ',', 'for', 'years', ',', 'he', 'had']
# 3.1 Frequency Distributions
fdist1 = FreqDist(text1)
print(fdist1)
<FreqDist with 19317 samples and 260819 outcomes>
fdist1.most_common(50)
[(',', 18713),
('the', 13721),
('.', 6862),
('of', 6536),
('and', 6024),
('a', 4569),
('to', 4542),
(';', 4072),
('in', 3916),
('that', 2982),
("'", 2684),
('-', 2552),
('his', 2459),
('it', 2209),
('I', 2124),
('s', 1739),
('is', 1695),
('he', 1661),
('with', 1659),
('was', 1632),
('as', 1620),
('"', 1478),
('all', 1462),
('for', 1414),
('this', 1280),
('!', 1269),
('at', 1231),
('by', 1137),
('but', 1113),
('not', 1103),
('--', 1070),
('him', 1058),
('from', 1052),
('be', 1030),
('on', 1005),
('so', 918),
('whale', 906),
('one', 889),
('you', 841),
('had', 767),
('have', 760),
('there', 715),
('But', 705),
('or', 697),
('were', 680),
('now', 646),
('which', 640),
('?', 637),
('me', 627),
('like', 624)]
fdist1.plot(50, cumulative=True)
<AxesSubplot:xlabel='Samples', ylabel='Cumulative Counts'>
fdist2 = FreqDist(text2)
fdist3 = FreqDist(text3)
fdist2.hapaxes()
fdist3.hapaxes()
['form', 'void', 'Day', 'Night', 'firmame', 'Heaven', 'appe', 'Earth', 'signs', 'seasons', 'lesser', 'nig', 'darkne', 'fly', 'whales', 'winged', 'seas', 'likene', 'subdue', 'finished', 'sanctified', 'plant', 'gr', 'mist', 'breathed', 'parted', 'Pison', 'bdellium', 'onyx', 'Gihon', 'Ethiopia', 'Hiddekel', 'Assyria', 'Euphrates', 'freely', 'eatest', 'sle', 'ribs', 'rib', 'Woman', 'Man', 'cleave', 'ashamed', 'subtil', 'gard', 'knowing', 'desired', 'sewed', 'fig', 'leaves', 'aprons', 'walking', 'cool', 'whereof', 'gavest', 'belly', 'enmity', 'conception', 'Thorns', 'thistles', 'sweat', 'tak', 'coats', 'clothed', 'Cherubims', 'flaming', 'tiller', 'firstlings', 'fallen', 'crieth', 'tillest', 'henceforth', 'punishment', 'driven', 'findeth', 'whosoever', 'slayeth', 'vengeance', 'mark', 'finding', 'Nod', 'Methusa', 'Methusael', 'Jabal', 'Jubal', 'handle', 'organ', 'instructor', 'artificer', 'brass', 'ir', 'Naamah', 'spee', 'wounding', 'avenged', 'En', 'book', 'Male', 'begotten', 'Eno', 'always', 'giants', 'renown', 'thoughts', 'repented', 'repenteth', 'just', 'corrupted', 'gopher', 'rooms', 'fashion', 'height', 'cubit', 'finish', 'lower', 'stories', 'fema', 'cause', 'bird', 'hills', 'Fifteen', 'upward', 'asswaged', 'Ararat', 'decreased', 'tops', 'ma', 'raven', 'fro', 'sole', 'olive', 'leaf', 'pluckt', 'twentieth', 'breed', 'kinds', 'offerings', 'sweet', 'savour', 'While', 'remaineth', 'seedtime', 'cold', 'summer', 'winter', 'cease', 'dread', 'fishes', 'liveth', 'Whoso', 'sheddeth', 'shed', 'perpetual', 'clo', 'overspread', 'husbandman', 'vineyard', 'drunken', 'uncovered', 'shoulders', 'enlarge', 'Magog', 'Madai', 'Tubal', 'Meshech', 'Tiras', 'Ashkenaz', 'Riphath', 'Togarmah', 'Elishah', 'Tarshish', 'Kittim', 'Dodanim', 'isles', 'Gentiles', 'tongue', 'Phut', 'Seba', 'Sabtah', 'Sabtech', 'Erech', 'Accad', 'Calneh', 'Resen', 'Ludim', 'Anamim', 'Lehabim', 'Naphtuhim', 'Pathrusim', 'Casluhim', 'Philistim', ',)', 'Caphtorim', 'Jebusite', 'Girgasite', 'Arkite', 'Sinite', 'Arvadite', 'Zemarite', 'Hamathite', 'Gaza', 'Zeboim', 'Lasha', 'Lud', 'Hul', 'Gether', 'Mash', 'Almodad', 'Sheleph', 'Hazarmaveth', 'Jerah', 'Hadoram', 'Uzal', 'Diklah', 'Obal', 'Abimael', 'Ophir', 'Mesha', 'Sephar', 'natio', 'thoroughly', 'slime', 'morter', 'reach', 'begin', 'imagined', 'flo', 'nineteen', 'nativity', 'Iscah', 'blessi', 'Sichem', 'Moreh', 'having', 'enter', 'commended', 'entreated', 'plagued', 'plagues', 'rich', 'journeys', 'fir', 'Perizzite', 'sinners', 'northward', 'southward', 'westwa', 'war', 'Bera', 'Birsha', 'Shinab', 'Shemeber', 'Twelve', 'thirteenth', 'rebelled', 'fourteenth', 'Ashteroth', 'Karnaim', 'Zuzims', 'Emins', 'Kiriathaim', 'Elparan', 'Enmishpat', 'Amalekites', 'Hazezontamar', ';)', 'battle', 'slimepits', 'victuals', 'escaped', 'An', 'confederate', 'armed', 'trained', 'eighteen', 'Hobah', 'slaughter', 'dale', 'Melchizedek', 'Salem', 'tithes', 'shoelatchet', 'ri', 'Save', 'Aner', 'vision', 'Abr', 'shield', 'reward', 'childless', 'Eliezer', 'Look', 'heifer', 'goat', 'turtledove', 'pigeon', 'piece', 'carcases', 'horror', 'dark', 'smoking', 'burning', 'lamp', 'Euphrat', 'Kenites', 'Kenizzites', 'Kadmonites', 'Hittites', 'Perizzites', 'Girgashites', 'Jebusites', 'beari', 'obtain', 'wrong', 'bosom', 'hardly', 'submit', 'wild', 'Beerlahairoi', 'Bered', 'Neither', 'circumcise', 'circumcis', 'beget', 'next', 'talking', 'thirteen', 'plains', 'tr', 'morsel', 'hearts', 'measures', 'meal', 'knead', 'cakes', 'hearth', 'herd', 'fetcht', 'butter', 'dressed', 'ceased', 'pleasure', 'too', 'At', 'denied', 'Seeing', 'justice', 'judgment', 'altogether', 'wick', 'Judge', 'sakes', 'ash', 'spe', 'communing', 'lords', 'ways', 'street', 'bake', 'unleavened', 'compassed', 'quart', 'wickedly', 'shadow', 'roof', 'Stand', 'fellow', 'jud', 'worse', 'blindness', 'wearied', 'waxen', 'married', 'mocked', 'merciful', 'Escape', 'stay', 'magnified', 'saving', '?)', 'anything', 'risen', 'rained', 'brimstone', 'gat', 'Zo', 'Our', 'morrow', 'Moabites', 'Benam', 'Ammon', 'Said', 'innocency', 'sinning', 'prophet', 'deeds', 'sawest', 'wander', 'thousand', 'oth', 'prayed', 'healed', 'fast', 'wombs', 'visited', 'suck', 'mocking', 'Cast', 'putting', 'wandered', 'shrubs', 'sh', 'aileth', 'archer', 'Paran', 'doe', 'falsely', 'violently', 'mean', 'grove', 'tempt', 'lovest', 'Moriah', 'offer', 'saddled', 'Abide', 'yonder', 'worship', 'order', 'Lay', 'fearest', 'thicket', 'hor', 'Jehovahjireh', 'sworn', 'multiplying', 'shore', 'Huz', 'Buz', 'Kemuel', 'Chesed', 'Hazo', 'Pildash', 'Jidlaph', 'Reumah', 'Tebah', 'Gaham', 'Thahash', 'Maachah', 'Kirjatharba', 'mourn', 'sojourner', 'sepulchres', 'withhold', 'sepulchre', 'intreat', 'weighed', 'current', 'merchant', 'Mam', 'ruled', 'Beware', 'oa', 'Mesopotamia', 'kneel', 'speed', 'thereby', 'giving', 'trough', 'wondering', 'wit', 'golden', 'half', 'shekel', 'destitute', 'tru', 'standest', 'ungirded', 'errand', 'Speak', 'Both', 'proceedeth', 'bowing', 'Rebek', 'precious', 'Hinder', 'prospered', 'thousands', 'millions', 'damsels', 'rode', 'meditate', 'eventide', 'coming', 'walketh', 'mast', 'Zimran', 'Medan', 'Ishbak', 'Asshurim', 'Letushim', 'Leummim', 'Ephah', 'Epher', 'Abidah', 'Eldaah', 'concubines', 'gifts', 'purchased', 'Kedar', 'Adbeel', 'Mibsam', 'Mishma', 'Dumah', 'Massa', 'Tema', 'Jetur', 'Naphish', 'Kedemah', 'towns', 'castles', 'Assyr', 'barr', 'struggled', 'boys', 'cunning', 'sod', 'Feed', 'Sell', 'point', 'lentiles', 'Sojourn', 'perform', 'commandments', 'statutes', 'laws', 'sporting', 'lightly', 'lien', 'guiltiness', 'toucheth', 'sowed', 'received', 'hundredfo', 'forward', 'mightier', 'springing', 'ou', 'Esek', 'Sitnah', 'Ahuzzath', 'friends', 'army', 'pea', 'betimes', 'Shebah', 'Judith', 'Beeri', 'Hitti', 'Which', 'grief', 'dea', 'weapons', 'quiver', 'hunt', 'seem', 'deceiver', 'Upon', 'ne', 'badest', 'sit', 'felt', 'discerned', 'han', 'Art', 'wi', 'blesseth', 'scarce', 'hunting', 'trembled', 'bitter', 'Bless', 'subtilty', 'rightly', 'supplanted', 'tim', 'reserved', 'sustained', 'yoke', 'wherewith', 'touching', 'purposing', 'fury', 'Until', 'deprived', 'weary', 'Mahalath', 'ladder', 'reached', 'heav', 'ascending', 'descending', 'whereon', 'liest', 'north', 'awaked', 'dreadful', 'vowed', 'floc', 'roll', 'tidings', 'nought', 'eyed', 'beautiful', 'better', 'Fulfil', 'Reub', 'judged', 'wrestlings', 'Happy', 'wheat', 'hired', 'maiden', 'husba', 'endued', 'afterwards', 'reproa', 'add', 'tar', 'learned', 'experience', 'Appoint', 'comi', 'removing', 'goa', 'poplar', 'hazel', 'chesnut', 'strakes', 'appear', 'watering', 'troughs', 'whensoever', 'feeble', 'feebler', 'deceived', 'speckl', 'leaped', 'grisled', 'leap', 'grisl', 'doeth', 'anointedst', 'vowedst', 'Are', 'quite', 'ours', 'getting', 'stole', 'mou', 'captives', 'secretly', 'mirth', 'songs', 'tabret', 'foolishly', 'longedst', 'force', 'findest', 'discern', 'furniture', 'displease', 'custom', 'chode', 'hotly', 'Whereas', 'loss', 'drought', 'frost', 'mi', 'Jegarsahadutha', 'Mizpah', 'watch', 'absent', 'harm', 'sacrifice', 'Mahanaim', 'womenservan', 'distressed', 'worthy', 'mercies', 'Deliver', 'Thirty', 'milch', 'colts', 'bulls', 'foals', 'Pass', 'meeteth', 'asketh', 'droves', 'appease', 'accept', 'ford', 'Jabbok', 'brook', 'breaking', 'joint', 'breaketh', 'Tell', 'ask', 'Peniel', 'preserved', 'Penuel', 'halted', 'hindermost', 'handmaidens', 'meanest', 'urged', 'knoweth', 'overdrive', 'lead', 'softly', 'endure', 'folk', 'needeth', 'booths', 'Shalem', 'parcel', 'erected', 'EleloheIsrael', 'commune', 'wrought', 'folly', 'longeth', 'marriages', 'Ask', 'gift', 'deceitfully', 'reproach', 'deferred', 'delight', 'honourable', 'peaceable', 'large', 'herein', 'boldly', 'males', 'edge', 'wealth', 'stink', 'Perizzit', 'Should', 'fleddest', 'change', 'garmen', 'earrings', 'oak', 'terror', 'pursue', 'Elbethel', 'Deborah', 'beneath', 'Allonbachuth', 'na', 'sto', 'departing', 'Beno', 'Edar', 'concubi', 'twel', 'Zebul', 'Arbah', 'Amal', 'Amalek', 'Kor', 'Horite', 'inhabited', 'Hemam', 'Alvan', 'Manahath', 'Ebal', 'Shepho', 'Onam', 'Ajah', 'mules', 'Hemdan', 'Eshban', 'Ithran', 'Cheran', 'Bilhan', 'Zaavan', 'Akan', 'Aran', 'Beor', 'Dinhabah', 'Bozrah', 'Temani', 'Bedad', 'Avith', 'Masrekah', 'Pau', 'Mehetabel', 'Matred', 'Mezahab', 'Timnah', 'Alvah', 'Jetheth', 'Elah', 'Pinon', 'Mibzar', 'Magdiel', 'Iram', 'possessi', 'feeding', 'wiv', 'report', 'peaceably', 'binding', 'upright', 'Shalt', 'reign', 'moon', 'observed', 'wandering', 'seekest', 'conspired', 'dreamer', 'Some', 'Shed', 'rid', 'stript', 'spicery', 'conceal', 'sell', 'content', 'merchantmen', 'killed', 'dipped', 'fou', 'doubt', 'sackcloth', 'Shel', 'Chezib', 'marry', 'raise', 'spilled', 'Remain', 'gro', 'sheepshearers', 'wrapped', 'grown', '.)', 'widowhood', 'openly', 'shamed', 'months', 'played', 'whoredom', 'chi', 'Discern', 'acknowledged', 'travail', 'breach', 'Zarah', 'person', 'wotteth', 'business', 'loud', 'voi', 'bou', 'doer', 'butlers', 'bakers', 'continued', 'season', 'sad', 'sadly', 'interpretations', 'belong', 'budded', 'blossoms', 'clusters', 'ripe', 'grapes', 'former', 'think', 'mention', 'dungeon', 'uppermost', 'bakemeats', 'hang', 'birthday', 'butlership', 'forgat', 'meadow', 'brink', 'slept', 'faults', 'bak', 'office', 'hastily', 'dunge', 'shaved', 'canst', 'bank', 'riv', 'mead', 'poor', 'badne', 'lean', 'withered', 'declare', 'sheweth', 'forgotten', 'consume', 'following', 'doubled', 'twice', 'shortly', 'appoint', 'perish', 'Can', 'Forasmuch', 'rul', 'throne', 'ring', 'arrayed', 'vestures', 'linen', 'chain', 'ride', 'Bow', 'kn', 'Zaphnathpaaneah', 'handfuls', 'citi', 'numbering', 'plenteousness', 'famished', 'storehouses', 'Whence', 'From', 'spi', 'y', 'pris', 'hous', 'verified', 'verily', 'guilty', 'anguish', 'besought', 'Spake', 'required', 'understood', 'espied', 'befell', 'traffick', 'bundle', 'bundles', 'Me', 'childr', 'Slay', 'alo', 'solemnly', 'protest', 'straitly', 'state', 'tenor', 'wor', 'fruits', 'vessels', 'honey', 'spices', 'nuts', 'almon', 'oversig', 'dine', 'bade', 'occasion', 'sir', 'weig', 'Peace', 'treasure', 'sac', 'welfare', 'health', 'gracious', 'yearn', 'sought', 'chamber', 'refrained', 'Set', 'marvelled', 'messes', 'mess', 'merry', 'Fill', 'overtake', 'rewarded', ...]
# 3.2 Fine-grained selection of wordssss
V = set(text4)
long_words = [w for w in V if len(w) > 15]
sorted(long_words)
['RESPONSIBILITIES', 'antiphilosophists', 'constitutionally', 'contradistinction', 'discountenancing', 'disqualification', 'enthusiastically', 'instrumentalities', 'internationality', 'irresponsibility', 'misappropriation', 'misrepresentation', 'misunderstanding', 'responsibilities', 'sentimentalizing', 'transcontinental', 'uncharitableness', 'unconstitutional']
text = text3
fdist = FreqDist(text)
sorted(w for w in set(text) if len(w) > 7 and fdist[w] > 15)
['Abimelech', 'Benjamin', 'according', 'answered', 'brethren', 'children', 'commanded', 'conceived', 'covenant', 'daughter', 'daughters', 'firstborn', 'generations', 'multiply', 'returned', 'servants', 'therefore', 'together']
list(bigrams(['more', 'is', 'said', 'than', 'done']))
[('more', 'is'), ('is', 'said'), ('said', 'than'), ('than', 'done')]
text3.collocations()
said unto; pray thee; thou shalt; thou hast; thy seed; years old; spake unto; thou art; LORD God; every living; God hath; begat sons; seven years; shalt thou; little ones; living creature; creeping thing; savoury meat; thirty years; every beast
text7.collocations()
million *U*; New York; billion *U*; Wall Street; program trading; Mrs. Yeargin; vice president; Stock Exchange; Big Board; Georgia Gulf; chief executive; Dow Jones; S&P 500; says *T*-1; York Stock; last year; Sea Containers; South Korea; American Express; San Francisco
text6.collocations()
BLACK KNIGHT; clop clop; HEAD KNIGHT; mumble mumble; Holy Grail; squeak squeak; FRENCH GUARD; saw saw; Sir Robin; Run away; CARTOON CHARACTER; King Arthur; Iesu domine; Pie Iesu; DEAD PERSON; Round Table; clap clap; OLD MAN; dramatic chord; dona eis
# ahahah
| Example | Description |
|---|---|
| fdist = FreqDist(samples) | create a frequency distribution containing the given samples |
| fdist[sample] += 1 | increment the count for this sample |
| fdist['monstrous'] | count of the number of times a given sample occurred |
| fdist.freq('monstrous') | frequency of a given sample |
| fdist.N() | total number of samples |
| fdist.most_common(n) | the n most common samples and their frequencies |
| for sample in fdist: | iterate over the samples |
| fdist.max() | sample with the greatest count |
| fdist.tabulate() | tabulate the frequency distribution |
| fdist.plot() | graphical plot of the frequency distribution |
| fdist.plot(cumulative=True) | cumulative plot of the frequency distribution |
| fdist1 |= fdist2 | update fdist1 with counts from fdist2 |
| fdist1 < fdist2 | test if samples in fdist1 occur less frequently than in fdist2 |