fU VddlmZmZmZddlZddlZddlZddlmZddl m Z ddl m Z m Z m Z GddZGddee jZGd d ee j ZGd d ee j"ZGd dee j$ZGddee j&Zd-dZdZej,dej.ej0zej2zZd-dZdZej,ej;ddej.ej0zej2zZej,ej;ddej.ej0zej2zZd-dZ d-dZ!dZ"d.dZ#d/dZ$d.dZ%d0d Z&d!Z'd.d"Z(d.d#Z)d.d$Z*dd%l+m,Z,e,ejZfd&Z.e/d'k(rhgd(Z0e0D]Z1e2d)e1d*e3ee1e0D]Z1e2d+e1d*e3e!e1e0D]Z1e2d,e1d*e3e"e1yy)1)divisionprint_functionunicode_literalsN)log)overlaylinint loglinintcVeZdZdZdZdZd dZdZdZdZ dZ fdZ d Z d Z xZS) _BaseModelFcd|j|D]}|j|d|d|_y)z- Extract n-grams from tokens and count them. TN)_extract_ngrams count_ngrammodified)selftokensallow_new_wordsngrams >/usr/lib/python3/dist-packages/Onboard/pypredict/lm_wrapper.py learn_tokensz_BaseModel.learn_tokens$s7))&1 8E   UA 7 8 c#HKg}t|d}|D]}|jt|dd |D]h}|}t|D]V\}}t|jD]9}||zdzt |ks||||zdz} |t | dz k(sJ| ;Xjyw)a/ Extract n-grams from tokens. Doctests: >>> m = DynamicModel(3) >>> list(m._extract_ngrams(["word1", "word2", "", "word3"])) [['word1'], ['word1', 'word2'], ['word2'], ['word3']] >>> list(m._extract_ngrams(["word1", "word2", "", "word3"])) [['word1'], ['word1', 'word2'], ['word2'], [''], ['', 'word3'], ['word3']] >>> list(m._extract_ngrams(["a", "b", "c", "d"])) [['a'], ['a', 'b'], ['a', 'b', 'c'], ['b'], ['b', 'c'], ['b', 'c', 'd'], ['c'], ['c', 'd'], ['d']] TrN) split_tokensextend enumeraterangeorderlen) rrtoken_sections unk_sectionssection token_sectionitokennrs rrz_BaseModel._extract_ngrams+s$FG4 # FG  ! !,wt"D E F , $M#G$W- $%tzz*$As1uG , '!A#a% 0 CJqL010# $ $ $s A6B"9)B"cdg|jz}dg|jz}|jD]=}|t|ddz xxdz cc<|t|ddz xx|dz cc<?||fS)zc Return number of n-gram types and total occurances for each n-gram level. rr)r iter_ngramsr!)rcountstotalsngs r get_countsz_BaseModel.get_countsLs TZZTZZ""$ *B 3r!u:a< Q & 3r!u:a< BqE )  *v~rct|dr|j|_|jD]}|d}|d}|j|| |S)zg Copy contents of self to model. The order of the destination stays unchanged. smoothingrr)hasattrr0r*r)rmodelitrcounts rcopyz_BaseModel.copyXsZ 4 %"nnEO""$ ,BqEEqEE   eU + ,  rc|j}t|D]}|dk7rn|dz}t|d}|j|}t |dr|j |_|j D]R}|d}|d}t|}tt||dz } || }||kDs;|dk7sA|j||T|S)a Return a copy of self with all ngrams removed whose count is less or equal to . prune_count==-1 # prune all frequencies prune_count=0 # prune nothing prune_count>0 # prune frequencies below or equal prune_count rr0r) r reversedmax __class__r1r0r*r!minr) r prune_countsr prune_countr2r3rr4levelks rprunez_BaseModel.prunegs #L1 Kb  QJE  E1 u% 4 %"nnEO""$ 0BqEEqEEJEC %u-1A&q/K{" r(9!!%/ 0 rcd|_d|_d|_ tt||y#t $r}d|_|d}~wwxYw)NFr T) load_errorload_error_msgrsuperr loadIOError)rfilenameer;s rrFz_BaseModel.loadsI    *d ( 2 "DOG s, A >Ac|j|}|r1|jD]\}}|j||d|_|S)z Remove word context[-1] where it appears after history context[:-1] from the model. If the history is empty all n-grams containing word will be removed. T)get_remove_context_changesitemsrr)rcontextchangesrr4s rremove_contextz_BaseModel.remove_contextsN 11':  '  / u  . /!DMrc  i}|jD]x}|d}|d}tt|D]U}ttt||dzD]}|||z || dz k7s<t|dz k(sO| ||<xz|S)z} Simulate removal of context. Returns a dict of affected n-grams and their count changes (negative). rr)r*rr!r<)rrMrNr3rr4r&js rrKz%_BaseModel.get_remove_context_changess ""$ BqEEqEE3u:& s3w<156AQqSzWaRT]2CL1,,*/  r)T)__name__ __module__ __qualname__rrCrDrrr.r5rArFrOrK __classcell__)r;s@rr r s<HJN$B  !F rr ceZdZdZdZy) LanguageModelzp Abstract class representing the base class of all models. Keep this for access to class constants. ctN)NotImplementedError)rs r__init__zLanguageModel.__init__s !##rN)rRrSrT__doc__r[rrrWrWs $rrWc eZdZy) UnigramModelNrRrSrTr]rrr_r_rr_c eZdZy) DynamicModelNr`r]rrrcrcrarrcc eZdZy)DynamicModelKNNr`r]rrrererarrec eZdZy)CachedDynamicModelNr`r]rrrgrgrarrgcg}g}|D]4}||k(r|r|j||r|g}!g}$|j|6t|dkDs |r|d|k7r|j||S)a Split list of tokens at separator token. Doctests: # excluding separator >>> split_tokens(["", "word1", "word2", "word3"], "") [['word1', 'word2', 'word3']] >>> split_tokens(["word1", "", "word2", "word3"], "") [['word1'], ['word2', 'word3']] >>> split_tokens(["word1", "word2", "word3", ""], "") [['word1', 'word2', 'word3']] # including separator >>> split_tokens(["", "word1", "word2", "word3"], "", True) [['', 'word1', 'word2', 'word3']] >>> split_tokens(["word1", "", "word2", "word3"], "", True) [['word1'], ['', 'word2', 'word3']] >>> split_tokens(["word1", "word2", "word3", ""], "", True) [['word1', 'word2', 'word3']] rr)appendr!)r separatorkeep_separatorr"r%r's rrrs*NM ( I %%m4!* "   ' ( =A-*i7m, rcg}d}|D]}|||}|r|j||dz}!||d}|r|j||S)a$ Patition tokens with splits at the given indices. split_indices must be sorted in ascending order. Doctests: >>> test = split_tokens_at >>> test(["word0", "word1", "word2"], []) [['word0', 'word1', 'word2']] >>> test(["word0", "word1", "word2"], [0]) [['word1', 'word2']] >>> test(["word0", "word1", "word2"], [1]) [['word0'], ['word2']] >>> test(["word0", "word1", "word2"], [2]) [['word0', 'word1']] >>> test(["word0", "word1", "word2"], [0, 2]) [['word1']] >>> test(["word0", "word1", "word2"], [0, 1, 2]) [] >>> test(["word0", "word1", "word2", "word3", "word4"], [0, 2, 4]) [['word1'], ['word3']] # out of range indices >>> test(["word0", "word1", "word2"], [100, 1000]) [['word0', 'word1', 'word2']] rrN)ri)r split_indicesr" remainingr&r$s rsplit_tokens_atrosiBNI 1%   ! !' *aC  YZ Gg& ra- .*? (?: (?:[.;:!?](?:(?=[\s]) | \")) # punctuation | (?:\\s*\\n\\s*)+(?=[\\n]) # multiples newlines | # sentence end mark ) | .+$ # last sentence fragment c|jdd}tj|}g}g}|D]K}|j} |j }|j } t |} |j}|| t |z z }t |} |j}| | t |z z} tjdd|}t |} |j}| | t |z z} t |} |j}|| t |z z }|r*tjd|tjs|dz }|j||j|| gN||fS)z Split text into sentences.   rz z [.;:!?]"?$z )replaceSENTENCE_PATTERNfinditergroupstartendr!lstriprstripresubsearchUNICODEri) text disambiguatefilteredmatches sentencesspansmatchsentencebeginrxls rsplit_sentencesr7sm ||D%H''1GI E'';;= KKMEIIKCH A(H QX& &EH A(H 1s8}$ $CvveUH5H H A(H 1s8}$ $CH A(H QX& &E yy"**E&H   X & LL% &O''R e ru ( # (?:^|(?<=\s)) \S*(\S)\\2{{3,}}\S* # char repeated more than 3 times | [-]{{3}} # dash repeated more than 2 times (?=\s|$) | :[^\s:@]+?@ # password in URL ) | ( # (?:[-+]?\d+(?:[.,]\d+)*) # anything numeric looking | (?:[.,]\d+) ) | ( # word (?:[-]{{0,2}} # allow command line options [^\W\d]\w*(?:[-'´΄][\w]+)* # word, not starting with a digit [{trailing_characters}'´΄]?) | | | | # pass through control words | # pass through begin of text merkers | (?:^|(?<=\s)) (?: \| {standalone_operators} # common space delimited operators ) (?=\s|$) ) r )trailing_charactersstandalone_operators-z | [-]{1,2}c|rtj|}ntj|}g}g}|D]}|j}|dr4|j |d|j |j L|dr1|j d|j |j |ds|j d|j |j ||fS)Nr8zrr)CONTEXT_PATTERNru TEXT_PATTERNgroupsrispan)r is_contextrrrrrs rtokenize_sentencers!**84''1 F E ' !9 MM&) $ LL & AY MM' " LL & AY MM' " LL & ' 5=rc\g}g}t|\}}t|D]\}}t||\}} ||d} | D cgc]} | d| z| d| zg} } |dkDr$|jd|j| | g|j ||j | ||fScc} w)a~ Split text into word tokens. The result is ready for use in learn_tokens(). Sentence begins, if detected, are marked with "". Numbers are replaced with the number marker . Other tokens that could confuse the prediction are replaced with the unknown word marker "". Examples, text -> tokens: "We saw whales" -> ["We", "saw", "whales"] "We saw whales " -> ["We", "saw", "whales"] "Hello there! We saw 5 whales " -> ["Hello", "there", "", "We", "saw", "", "whales"] rrr)rrrrir) rrrrrsentence_spansr&rtssssbeginss r tokenize_textrs"F E / 5I~ +  8"8Z8B"1%24 5QqtF{AaDK( 5 5 q5 MM% LL&&) * b R  5=6sB)ct|d\}}tjd|tjtjztj zs/|j dt|}|j ||g||fS)zh Split text into word tokens + completion prefix. The result is ready for use in predict(). T)ru0 ^$ # empty string? | .*[-'´΄\w]$ # word at the end? | (?:^|.*\s)[|]=?$ # recognized operator? | .*(\S)\\1{3,}$ # anything repeated > 3 times? r )rr{rr~DOTALLVERBOSErir!)rrrtends rtokenize_contextrss"$T:MFE 88 2::bii/ :  <  b4y dD\" 5=rcd}tjdk\rt}nt} t ||d}|j d}d}|D]o}|j drd}|stjd|} | r*|d }t|t| jd }|j d sn|S|S#|$r }Yd}~yd}~wwxYw) zc Read the order from the header of the given file. Encoding may be 'utf-8', 'latin-1'. N)rr Fz\data\Tzngram (\d+)=\d+r\) sys version_infoFileNotFoundErrorrG read_corpussplit startswithr{r}r:intr) rHencodingr ex_classrexlinesdatalineresults r read_orderrs E 6!$8Xr2 JJt E D  ??: &D  YY148F=EE3v}}q'9#:;t$ L  L) s B::C C c|r|g}nddg}t|D]{\}} |&tj||j}nKd}tj||5}t |D]}|j }|sn||z }ddd|SS#1swYxYw#t $r} |t|dz k(r| Yd} ~ d} ~ wwxYw)z2 Read corpus, encoding may be 'utf-8', 'latin-1'. zutf-8zlatin-1N)rr r)rcodecsopenreadrreadlineUnicodeDecodeErrorr!) rHr num_lines encodingsr&encrfterrs rrrsJ i( 9%#  {{8c:??A[[C8"A"9-"JJL !  ""  K%$ K"" " C N1$$  s0AB"*BB"B B"" C+CCczt||}|jd}dD]}||vs|j||S)zk Read vocabulary with one word per line. Encoding may be 'utf-8', 'latin-1', like read_corpus. r)rrzz)rrri)rHrr vocabulary ctrl_words rread_vocabularyrsK x *DD!J7) J&   i () rci}|D]}|j|ddz||<t|jDcgc] }|d|k\s |}}t|dd}|r|d|S|Scc}w)z< Extract the most frequent words from . rrc |dS)Nrr])xs rz$extract_vocabulary..4s !rT)keyreverseN)getlistrLsorted)r min_count max_wordsmrrrLs rextract_vocabularyr.s} A uuQ{Q!QWWY =11Q49+s A) A)cNt|}|Dcgc] }||vr|nd c}Scc}w)Nr)set)rrvrs r filter_tokensr:s* JA.4 5aAW $ 55 5s"c|s |j}d}d}t|}tt|D]}t||dz z d}t ||dz z |zt|}|||} t| dk7sG|j | } | dk(r t || | | r t| dn td}||z }|dz }|r| |z nd} d|z} || fS#d} Y|| fSxYw)Nrrr8infinity) r r!rr:r<get_probabilityprintrfloat) r2rr ngram_countentropy word_countr&brIrp perplexitys rrr>s  KGVJ3v;   57 A  57 E!3v; /q  u:?%%e,AAvj%*Aq E*$5A qLG 1 K &0whz!QG'\  J  J s CCcHt|||||\}}||z }|r|dz|z SdS)z9 Calculate keystroke savings rate from simulated typing. gY@r)simulate_typing) query_model learn_modelrlimitprogress total_chars pressed_keyssaved_keystrokess rksrr]s> / [)UZ\d eK"\15@ e #k 1GaGrcd}d}t|D]2\}}d} d} | t|krtd| z\} } | r| t| dz nd} |t| t| z d}tjd|tj j }|j| |} ||vrt|t| z }|dk(rd}nd}t|D]}| || z } | dz } |dz }|dz }| t|kr|rt|\}} |j||s||t|||5||fS)Nrr . rz ^([\w]|[-'])*) rr!rr{r}r~rvpredictrinputrr)rrrrrrrr&r inputlinecursorrMrprefix prefix_to_end target_wordchoices added_charsr@rs rrrcs{KL *+C ( s8}$-dY.>?NGU07WS\!^,RF$S^CK%?%@AM))$4mRZZPVVXK!))'59Gg%!+.V< !#"#K ;' !Xf-- ! q  ! A L=s8}$B ,X6MFE  $ $V ,  QI \ BW+CZ  $$r)contextmanagerc#VKddl}ddl}|r|j|j|j|j}|r|nd}|jd|z|j d|jd|j|z dzzydyw)Nrtimeitz%-15s z %10.3fms i)timegccollectwriteflush)routrrrrs rrrs~    IIKq8 (T/"   ,499;?D"89: sB'B)__main__).rz . za. za. bzsplit_sentences('z'): ztokenize_text('ztokenize_context(')FrY)NN)rr)4 __future__rrrrr{rmathr pypredict.lmlmrrr r rWr_rcrergrrocompiler~rrrtrtokenize_patternformatrrrrrrrrrrrrr contextlibrstdoutrrRarrreprr]rrr sS(BA 33ZZz$J 0 0$ :r  :r  Z!2!2  R%:%: 'R.b2:: BII bjj (*6r4rzz*11021325**RYY.rzz9; "**-44031=5?**RYY.rzz9; .!F""H8  6>H 2%j&**( z)AQ T48M3NOPQM 4mD6I1JKLMS dD9I$9O4PQRSr