fuzzy_search package

Subpackages

• fuzzy_search.analysis package
  • Submodules
  • fuzzy_search.analysis.freq module
    • Ngram
      • Ngram.tokens
      • Ngram.size
      • Ngram.n
      • Ngram.string
      • Ngram.head_string
      • Ngram.tail_string
      • Ngram.__init__()
    • NgramFreq
      • NgramFreq.ngram_freq
      • NgramFreq.max_ngram_size
      • NgramFreq.total_ngram_tokens
      • NgramFreq.total_ngram_types
      • NgramFreq.num_docs
      • NgramFreq.start_tokens
      • NgramFreq.end_tokens
      • NgramFreq.__init__()
      • NgramFreq.count_ngrams()
      • NgramFreq.has_conditional_prob()
      • NgramFreq.has_freq()
      • NgramFreq.has_prob()
      • NgramFreq.total_tokens()
      • NgramFreq.total_types()
      • NgramFreq.vocab_size
    • check_lambdas()
    • compute_expected()
    • compute_llr()
    • compute_llr_from_observed()
    • compute_percentage_diff()
    • doc_to_string_tokens()
    • get_observed_from_counter()
    • make_token_ngrams()
  • fuzzy_search.analysis.similarity module
    • KeywordList
      • KeywordList.len_keys
      • KeywordList.max_length_diff
      • KeywordList.len_order
      • KeywordList.__init__()
      • KeywordList.find_close_distance_keywords()
      • KeywordList.find_closer_terms()
      • KeywordList.iterate_candidate_pairs()
    • SkipCooccurrence
      • SkipCooccurrence.cooc_freq
      • SkipCooccurrence.vocabulary
      • SkipCooccurrence.skip_size
      • SkipCooccurrence.__init__()
      • SkipCooccurrence.calculate_skip_cooccurrences()
      • SkipCooccurrence.get_term_coocs()
    • SkipgramSimilarity
      • SkipgramSimilarity.ngram_length
      • SkipgramSimilarity.skip_length
      • SkipgramSimilarity.vocabulary
      • SkipgramSimilarity.skipgram_vocabulary
      • SkipgramSimilarity.max_length_diff
      • SkipgramSimilarity.__init__()
      • SkipgramSimilarity.index_terms()
      • SkipgramSimilarity.rank_similar()
    • get_begin_sim()
    • get_end_sim()
    • get_min_length()
    • get_skip_coocs()
    • is_close_distance_keyword_pair()
    • vector_length()
  • fuzzy_search.analysis.spelling_compare module
    • Change
      • Change.decrease_word
      • Change.decrease_type
      • Change.increase_word
      • Change.increase_type
      • Change.sim_score
      • Change.__init__()
    • Diff
      • Diff.direction
      • Diff.frac1
      • Diff.frac2
      • Diff.freq1
      • Diff.freq2
      • Diff.llr
      • Diff.perc_diff1
      • Diff.perc_diff2
      • Diff.word
    • SpellingCompare
      • SpellingCompare.word_freq1
      • SpellingCompare.word_freq2
      • SpellingCompare.embeddings
      • SpellingCompare.total1
      • SpellingCompare.total2
      • SpellingCompare.word_frac1
      • SpellingCompare.word_frac2
      • SpellingCompare.__init__()
      • SpellingCompare.compute_percentage_diff()
      • SpellingCompare.get_frequency_change_words()
      • SpellingCompare.get_high_frequency_words()
    • sort_drop_jump()
  • fuzzy_search.analysis.subtoken module
    • BPEToken
      • BPEToken.token
      • BPEToken.symbols
      • BPEToken.__init__()
      • BPEToken.symbol_pairs
    • FrequencyTracker
      • FrequencyTracker.freq_buckets
      • FrequencyTracker.symbol_pair_freq
      • FrequencyTracker.max_freq
      • FrequencyTracker.all_with_max_frequency()
      • FrequencyTracker.frequency_of()
      • FrequencyTracker.most_frequent()
      • FrequencyTracker.most_frequent_shortest()
      • FrequencyTracker.update()
    • compare_token_symbol_pairs()
    • find_new_symbol_pairs()
    • generate_corpus_symbol_pairs()
    • generate_symbol_pairs()
    • generate_vocab()
    • index_symbol_pair()
    • make_byte_pair_encoding()
    • make_symbol_pair_freq()
    • merge_symbols_in_token()
    • merge_symbols_in_tokens()
    • prune_symbol_pair_freq()
    • string_tokens_to_corpus()
  • Module contents
• fuzzy_search.match package
  • Submodules
  • fuzzy_search.match.candidate_match module
    • Candidate
      • Candidate.__init__()
    • CandidatePartial
      • CandidatePartial.__init__()
    • add_skip_match()
    • candidate_from_partial()
    • get_match_start_offset()
    • get_match_string()
    • get_skip_count_overlap()
    • get_skip_match_length()
    • get_skip_set_overlap()
    • is_match()
    • remove_first_skip()
    • same_candidate()
    • shift_start_skip()
  • fuzzy_search.match.exact_match module
    • add_exact_match_score()
    • get_exact_match_ranges()
    • get_known_word_offsets()
    • index_known_word_offsets()
    • search_exact()
    • search_exact_phrases()
    • search_exact_phrases_with_word_boundaries()
    • search_exact_phrases_without_word_boundaries()
  • fuzzy_search.match.match_offsets module
    • adjust_match_end_offset()
    • adjust_match_offsets()
    • adjust_match_start_offset()
    • calculate_end_shift()
    • candidates_to_matches()
    • filter_matches_by_overlap()
    • map_string()
    • validate_match_props()
  • fuzzy_search.match.phrase_match module
    • MatchType
      • MatchType.FULL
      • MatchType.NONE
      • MatchType.PARTIAL_OF_PHRASE_TOKEN
      • MatchType.PARTIAL_OF_TEXT_TOKEN
    • PhraseMatch
      • PhraseMatch.__init__()
      • PhraseMatch.add_scores()
      • PhraseMatch.as_web_anno()
      • PhraseMatch.character_overlap
      • PhraseMatch.from_json()
      • PhraseMatch.has_label()
      • PhraseMatch.json()
      • PhraseMatch.label_list
      • PhraseMatch.levenshtein_similarity
      • PhraseMatch.ngram_overlap
      • PhraseMatch.overlaps()
      • PhraseMatch.score_character_overlap()
      • PhraseMatch.score_levenshtein_similarity()
      • PhraseMatch.score_ngram_overlap()
      • PhraseMatch.skipgram_overlap
    • PhraseMatchInContext
      • PhraseMatchInContext.as_web_anno()
      • PhraseMatchInContext.json()
    • phrase_match_from_json()
    • validate_match_props()
  • fuzzy_search.match.skip_match module
    • SkipMatches
      • SkipMatches.__init__()
      • SkipMatches.add_skip_match()
      • SkipMatches.remove_phrase()
    • filter_overlapping_phrase_candidates()
    • filter_skipgram_threshold()
    • get_skipmatch_candidates()
    • get_skipmatch_phrase_candidates()
    • get_skipset_overlap()
  • Module contents
• fuzzy_search.pattern package
  • Submodules
  • fuzzy_search.pattern.fuzzy_patterns module
    • context_before_pattern()
    • context_then_pattern()
    • escape_string()
    • get_context_patterns()
    • get_search_patterns()
    • list_context_pattern_types()
    • list_pattern_definitions()
    • list_pattern_names()
    • make_search_context_patterns()
    • pattern_before_context()
    • pattern_comma_then_context()
  • fuzzy_search.pattern.fuzzy_template module
    • FuzzyTemplate
      • FuzzyTemplate.__init__()
      • FuzzyTemplate.get_element()
      • FuzzyTemplate.get_elements_by_cardinality()
      • FuzzyTemplate.get_label_phrases()
      • FuzzyTemplate.get_labels_by_cardinality()
      • FuzzyTemplate.get_required_elements()
      • FuzzyTemplate.get_required_labels()
      • FuzzyTemplate.has_group()
      • FuzzyTemplate.has_label()
      • FuzzyTemplate.parse_group_element()
      • FuzzyTemplate.parse_label_element()
      • FuzzyTemplate.register_template()
    • FuzzyTemplateElement
      • FuzzyTemplateElement.__init__()
    • FuzzyTemplateGroupElement
      • FuzzyTemplateGroupElement.__init__()
    • FuzzyTemplateLabelElement
      • FuzzyTemplateLabelElement.__init__()
    • generate_group_from_json()
    • generate_label_from_json()
    • validate_element_properties()
  • Module contents
• fuzzy_search.phrase package
  • Submodules
  • fuzzy_search.phrase.phrase module
    • Phrase
      • Phrase.__init__()
      • Phrase.add_max_end_offset()
      • Phrase.add_max_start_offset()
      • Phrase.add_metadata()
      • Phrase.has_label()
      • Phrase.has_max_end_offset()
      • Phrase.has_max_start_offset()
      • Phrase.has_skipgram()
      • Phrase.is_early_skipgram()
      • Phrase.set_label()
      • Phrase.skipgram_offsets()
      • Phrase.within_range()
    • is_valid_label()
  • fuzzy_search.phrase.phrase_model module
    • PhraseModel
      • PhraseModel.__init__()
      • PhraseModel.add_custom()
      • PhraseModel.add_distractor()
      • PhraseModel.add_distractors()
      • PhraseModel.add_labels()
      • PhraseModel.add_model()
      • PhraseModel.add_phrase()
      • PhraseModel.add_phrases()
      • PhraseModel.add_variant()
      • PhraseModel.add_variants()
      • PhraseModel.get()
      • PhraseModel.get_labels()
      • PhraseModel.get_phrase()
      • PhraseModel.get_phrases()
      • PhraseModel.get_phrases_by_max_length()
      • PhraseModel.get_variants()
      • PhraseModel.has_custom()
      • PhraseModel.has_label()
      • PhraseModel.has_phrase()
      • PhraseModel.has_token()
      • PhraseModel.is_label()
      • PhraseModel.json
      • PhraseModel.remove_custom()
      • PhraseModel.remove_distractor()
      • PhraseModel.remove_distractors()
      • PhraseModel.remove_labels()
      • PhraseModel.remove_phrase()
      • PhraseModel.remove_phrases()
      • PhraseModel.remove_variant()
      • PhraseModel.remove_variants()
      • PhraseModel.set_phrase_token_max_end_offsets()
      • PhraseModel.set_phrase_token_max_start_offsets()
      • PhraseModel.variant_of()
      • PhraseModel.variants()
    • as_phrase_object()
    • is_phrase_dict()
  • Module contents
• fuzzy_search.search package
  • Submodules
  • fuzzy_search.search.config module
  • fuzzy_search.search.context_searcher module
    • FuzzyContextSearcher
      • FuzzyContextSearcher.add_match_context()
      • FuzzyContextSearcher.configure_context()
      • FuzzyContextSearcher.find_matches()
      • FuzzyContextSearcher.find_matches_in_context()
  • fuzzy_search.search.phrase_searcher module
    • FuzzyPhraseSearcher
      • FuzzyPhraseSearcher.__init__()
      • FuzzyPhraseSearcher.filter_matches_by_distractors()
      • FuzzyPhraseSearcher.filter_matches_by_threshold()
      • FuzzyPhraseSearcher.filter_phrase_candidates()
      • FuzzyPhraseSearcher.find_candidates()
      • FuzzyPhraseSearcher.find_exact_matches()
      • FuzzyPhraseSearcher.find_matches()
    • combine_fuzzy_and_exact_matches()
    • get_text_dict()
    • make_step_timer()
  • fuzzy_search.search.searcher module
    • FuzzySearcher
      • FuzzySearcher.__init__()
      • FuzzySearcher.configure()
      • FuzzySearcher.filter_matches_by_offset_threshold()
      • FuzzySearcher.find_skipgram_matches()
      • FuzzySearcher.index_distractors()
      • FuzzySearcher.index_phrase_model()
      • FuzzySearcher.index_phrases()
      • FuzzySearcher.index_variants()
  • fuzzy_search.search.template_searcher module
    • FuzzyTemplateSearcher
      • FuzzyTemplateSearcher.__init__()
      • FuzzyTemplateSearcher.filter_phrase_matches()
      • FuzzyTemplateSearcher.find_template_matches()
      • FuzzyTemplateSearcher.search_text()
      • FuzzyTemplateSearcher.set_template()
    • TemplateMatch
      • TemplateMatch.__init__()
    • find_next_element_end_index()
    • find_next_element_start_index()
    • find_next_group_match_sequence()
    • find_next_ordered_group_match_sequence()
    • find_next_unordered_group_match_sequence()
    • get_phrase_match_list_labels()
    • get_sequence_label_element_matches()
    • has_required_matches()
    • initialize_sequence()
    • share_label()
  • fuzzy_search.search.token_searcher module
    • FuzzyTokenSearcher
      • FuzzyTokenSearcher.__init__()
      • FuzzyTokenSearcher.add_vocabulary()
      • FuzzyTokenSearcher.add_vocabulary_skipgram_matches()
      • FuzzyTokenSearcher.configure()
      • FuzzyTokenSearcher.find_matches()
      • FuzzyTokenSearcher.find_skipgram_token_matches_for_token()
      • FuzzyTokenSearcher.find_skipgram_token_matches_in_text()
      • FuzzyTokenSearcher.find_vocabulary_text_phrase_term_pairs()
      • FuzzyTokenSearcher.has_distractor_pair()
      • FuzzyTokenSearcher.has_match_pair()
      • FuzzyTokenSearcher.has_text_phrase_term_pair()
      • FuzzyTokenSearcher.index_distractor_pair()
      • FuzzyTokenSearcher.index_phrase_token_skipgrams()
      • FuzzyTokenSearcher.index_text_phrase_term_pair()
      • FuzzyTokenSearcher.index_text_phrase_term_pairs()
      • FuzzyTokenSearcher.terms_to_id_tuple()
      • FuzzyTokenSearcher.terms_to_string()
      • FuzzyTokenSearcher.terms_to_tuple()
    • PartialPhraseMatch
      • PartialPhraseMatch.__init__()
      • PartialPhraseMatch.add_tokens()
      • PartialPhraseMatch.pop()
      • PartialPhraseMatch.push()
    • TokenMatch
      • TokenMatch.__init__()
    • copy_partial_match()
    • get_partial_phrases()
    • get_text_string()
    • get_text_tokens()
    • get_token_skip_match_type()
    • get_token_skip_match_types()
    • get_token_skipgram_matches()
    • get_tokenized_doc()
    • get_vocabulary_skipgram_matches()
    • has_max_end_offset()
    • has_max_start_offset()
    • is_distractor()
    • map_text_tokens_to_phrase_tokens()
    • token_is_out_of_phrase_range()
    • token_within_phrase_offset()
  • Module contents
• fuzzy_search.tokenization package
  • Submodules
  • fuzzy_search.tokenization.string module
    • SkipGram
      • SkipGram.string
      • SkipGram.start_offset
      • SkipGram.end_offset
      • SkipGram.length
      • SkipGram.__init__()
    • get_non_word_prefix()
    • get_non_word_suffix()
    • insert_skips()
    • make_ngrams()
    • score_char_overlap()
    • score_char_overlap_ratio()
    • score_levenshtein_distance()
    • score_levenshtein_similarity_ratio()
    • score_ngram_overlap()
    • score_ngram_overlap_ratio()
    • strip_prefix()
    • strip_suffix()
    • text2skipgrams()
    • token2skipgrams()
  • fuzzy_search.tokenization.token module
    • Annotation
      • Annotation.tag_type
      • Annotation.text
      • Annotation.offset
      • Annotation.end
      • Annotation.doc_id
      • Annotation.__init__()
    • CustomTokenizer
      • CustomTokenizer.tokenizer_func
      • CustomTokenizer.__init__()
      • CustomTokenizer.tokenize()
    • Doc
      • Doc.text
      • Doc.id
      • Doc.tokens
      • Doc.token_orig_set
      • Doc.token_norm_set
      • Doc.label_token_index
      • Doc.metadata
      • Doc.annotations
      • Doc.__init__()
      • Doc.add_annotations()
      • Doc.get_token()
      • Doc.has_token()
      • Doc.normalized
      • Doc.original
    • RegExTokenizer
      • RegExTokenizer.split_pattern
      • RegExTokenizer.__init__()
    • Tag
      • Tag.type
      • Tag.text
      • Tag.offset
      • Tag.tag_string
      • Tag.end
      • Tag.doc_id
      • Tag.__init__()
    • Token
      • Token.string
      • Token.index
      • Token.char_index
      • Token.doc_id
      • Token.metadata
      • Token.normalised_string
      • Token.label
      • Token.__init__()
      • Token.has_label()
      • Token.i
      • Token.l
      • Token.lower()
      • Token.n
      • Token.t
      • Token.update()
    • Tokenizer
      • Tokenizer.ignorecase
      • Tokenizer.include_boundary_tokens
      • Tokenizer.remove_punctuation
      • Tokenizer.__init__()
      • Tokenizer.nltk_wp_tokenizer
      • Tokenizer.tokenize()
      • Tokenizer.tokenize_doc()
    • tokens2string()
    • update_token()
  • fuzzy_search.tokenization.vocabulary module
    • Vocabulary
      • Vocabulary.term_id
      • Vocabulary.id_term
      • Vocabulary.term_freq
      • Vocabulary.ignorecase
      • Vocabulary.__init__()
      • Vocabulary.add_terms()
      • Vocabulary.has_term()
      • Vocabulary.id2term()
      • Vocabulary.reset_index()
      • Vocabulary.term2id()
  • Module contents

Module contents

fuzzy_search: a library for fuzzy phrase search in noisy and historical text.

Re-exports the most commonly used public classes (FuzzyPhraseSearcher, FuzzyTokenSearcher, PhraseMatch, PhraseModel, default_config) and provides the make_searcher convenience function for building a phrase searcher from a list of phrases and a config.

fuzzy_search.make_searcher(phrases: any, config)

Builds a FuzzyPhraseSearcher for a given list of phrases and configuration.

Parameters:

• phrases (any) – The phrases to search for, as accepted by PhraseModel.

• config – A configuration object/dict for the phrase model and searcher.

Returns:

A searcher configured with the built phrase model.

Return type:

FuzzyPhraseSearcher