fuzzy_search.search package

Submodules

fuzzy_search.search.config module

Default configuration values for fuzzy searchers.

Exposes default_config, the dictionary of default thresholds and behavior flags used by FuzzySearcher and its subclasses. Pass a partial dictionary with overrides to a searcher’s constructor or configure method to change only the desired settings.

fuzzy_search.search.context_searcher module

Context-aware fuzzy phrase searcher.

Defines FuzzyContextSearcher, which extends FuzzyPhraseSearcher to attach a window of surrounding text to each match, and to search within that context window for additional phrase matches.

class fuzzy_search.search.context_searcher.FuzzyContextSearcher(config: dict | None = None)[source]

Bases: FuzzyPhraseSearcher

Fuzzy phrase searcher that attaches surrounding text context to each match.

Extends FuzzyPhraseSearcher’s matching with a configurable prefix/suffix context window around each match, and supports re-searching that context window for further matches.

Attributes

context_sizeint: default size (in characters) of the prefix and suffix context window added around each match.

add_match_context(match: PhraseMatch, text: str | dict, context_size: None | int = None, prefix_size: None | int = None, suffix_size: None | int = None) → PhraseMatchInContext[source]

Add context to a given match and its corresponding text document.

Parameters:

match (PhraseMatch) – a phrase match object
text (Union[str, dict]) – the text that the match was taken from
context_size (int) – the size of the pre- and suffix window
prefix_size (Union[None, int]) – size of the prefix context
suffix_size (Union[None, int]) – size of the suffix context

Returns:

the phrase match object with context

Return type:

PhraseMatchInContext

configure_context(config: dict) → None[source]

Configure the context searcher.

Parameters:: config (dict) – a dictionary with configuration parameters to override the defaults

find_matches(text: str | dict, use_word_boundaries: None | bool = None, allow_overlapping_matches: bool = True, include_variants: bool = None, filter_distractors: bool = None, prefix_size: None | int = None, suffix_size: None | int = None, skip_exact_matching: bool = None) → List[PhraseMatchInContext][source]

Find fuzzy matches for registered phrases and add context around match string. This extends the find_matches function of the FuzzyPhraseSearcher by adding local context to each match.

Parameters:

text (Union[str, Dict[str, str]]) – the text (string or dictionary with ‘text’ property) to find fuzzy matching phrases in.
use_word_boundaries (bool) – use word boundaries in determining match boundaries
allow_overlapping_matches (bool) – boolean flag for whether to allow matches to overlap in their text ranges
include_variants (bool) – boolean flag for whether to include phrase variants for finding matches
filter_distractors (bool) – boolean flag for whether to remove phrase matches that better match distractors
prefix_size (Union[None, int]) – the size of the prefix context window
suffix_size (Union[None, int]) – the size of the suffix context window
skip_exact_matching (Union[None, bool]) – boolean flag whether to skip the exact matching step

Returns:

a list of phrases matches with text surrounding the match string

Return type:

PhraseMatchInContext

find_matches_in_context(match_in_context: PhraseMatchInContext, use_word_boundaries: None | bool = None, include_variants: None | bool = None, filter_distractors: None | bool = None) → List[PhraseMatch][source]

Use a MatchInContext object to find other phrases in the context of that match.

Parameters:

match_in_context (PhraseMatchInContext) – a match phrase with context from the text that the match was taken from
use_word_boundaries (bool) – boolean whether to adjust match strings to word boundaries
include_variants (bool) – boolean whether to include variants of phrases in matching
filter_distractors (bool) – boolean whether to remove matches that are closer to distractors

Returns:

a list of match objects

Return type:

List[PhraseMatch]

fuzzy_search.search.phrase_searcher module

Phrase-level fuzzy searcher.

Defines FuzzyPhraseSearcher, which combines an exact-matching pass with the skipgram-based fuzzy matching from FuzzySearcher to find whole-phrase matches in text, applying word-boundary, threshold, distractor, and overlap filtering.

class fuzzy_search.search.phrase_searcher.FuzzyPhraseSearcher(phrase_list: List[any] = None, phrase_model: Dict[str, any] | List[Dict[str, any]] | PhraseModel = None, config: None | Dict[str, str | int | float] = None, tokenizer: Tokenizer = None, token_searcher: FuzzyTokenSearcher = None)[source]

Bases: FuzzySearcher

Fuzzy searcher for finding whole-phrase matches in text.

Extends FuzzySearcher with an exact-matching pass (used to speed up and disambiguate fuzzy matching), candidate generation and filtering, and distractor/threshold/overlap-based filtering of the resulting matches.

This class represents the basic fuzzy searcher. You can pass a list of phrases or a phrase model and configuration dictionary that overrides the default configuration values. The default config dictionary is available via fuzzy_search.default_config.

To set e.g. the character ngram_size to 3 and the skip_size to 1 use the following dictionary:

config = {
    'ngram_size': 3,
    'skip_size': 1
}

Parameters:

phrase_list (list) – a list of phrases (a list of strings or more complex dictionaries with phrases and variants)
phrase_model (PhraseModel) – a phrase model
config (dict) – a configuration dictionary to override default configuration properties. Only the properties present in the config dictionary are updated.
tokenizer (Tokenizer) – a tokenizer instance
token_searcher (FuzzyTokenSearcher) – a fuzzy token searcher instance (using the same phrase model and config)

filter_matches_by_distractors(matches: List[PhraseMatch]) → List[PhraseMatch][source]

Remove matches that are a worse fit for their phrase than for one of that phrase’s known distractor phrases.

Parameters:: matches (List[PhraseMatch]) – phrase matches to filter
Returns:: matches that are not better explained by a distractor
Return type:: List[PhraseMatch]

filter_matches_by_threshold(matches: List[PhraseMatch]) → List[PhraseMatch][source]

Remove matches whose character overlap, ngram overlap, or Levenshtein similarity score falls below the searcher’s configured thresholds.

Parameters:: matches (List[PhraseMatch]) – phrase matches to filter
Returns:: matches that satisfy all configured similarity thresholds
Return type:: List[PhraseMatch]

filter_phrase_candidates(phrase_candidates: Dict[str, List[Candidate]], text: Dict[str, any], use_word_boundaries: bool, debug: int = 0) → List[Candidate][source]

Filter per-phrase candidate matches, optionally snapping their boundaries to word boundaries and removing candidates that overlap with a better candidate for the same phrase.

Parameters:

phrase_candidates (Dict[str, List[Candidate]]) – candidate matches grouped by phrase string
text (Dict[str, any]) – the text object the candidates were found in
use_word_boundaries (bool) – whether to adjust candidate boundaries to word boundaries
debug (int) – level to show debug information

Returns:

the filtered list of candidates across all phrases

Return type:

List[Candidate]

find_candidates(text: dict, use_word_boundaries: bool, include_variants: None | bool = None, known_word_start_offset: Dict[int, Dict[str, any]] = None, debug: int = 0) → List[Candidate][source]

Find candidate fuzzy matches for a given text.

Parameters:

text (dict) – the text object to match with phrases
use_word_boundaries (bool) – use word boundaries in determining match boundaries
include_variants (bool) – boolean flag for whether to include phrase variants for finding matches
known_word_start_offset (Dict[int, Dict[str, any]]) – a dictionary of known words and their text offsets based on exact matches
debug (int) – level to show debug information

Returns:

a list of candidate matches

Return type:

List[Candidate]

find_exact_matches(text: str | Dict[str, str], use_word_boundaries: None | bool = None, include_variants: None | bool = None, debug: int = 0) → List[PhraseMatch][source]

Find all fuzzy matching phrases for a given text.

Parameters:

text (Union[str, Dict[str, str]]) – the text (string or dictionary with ‘text’ property) to find fuzzy matching phrases in.
use_word_boundaries (Union[None, bool]) – use word boundaries in determining match boundaries
include_variants (Union[None, bool]) – boolean flag for whether to include phrase variants for finding matches
debug (int) – level to show debug information

Returns:

a list of phrases matches

Return type:

PhraseMatch

find_matches(text: str | Dict[str, str] | Doc, use_word_boundaries: None | bool = None, allow_overlapping_matches: None | bool = None, include_variants: None | bool = None, filter_distractors: None | bool = None, skip_exact_matching: bool = None, first_best: bool = False, debug: int = 0) → List[PhraseMatch][source]

Find all fuzzy matching phrases for a given text. By default, a first pass of exact matching is conducted to find exact occurrences of phrases. This is to speed up the fuzzy matching pass

Parameters:

text (Union[str, Dict[str, str]]) – the text (string or dictionary with ‘text’ property) to find fuzzy matching phrases in.
use_word_boundaries (bool) – use word boundaries in determining match boundaries
allow_overlapping_matches (bool) – boolean flag for whether to allow matches to overlap in their text ranges
include_variants (bool) – boolean flag for whether to include phrase variants for finding matches
filter_distractors (bool) – boolean flag for whether to remove phrase matches that better match distractors
skip_exact_matching (bool) – boolean flag whether to skip the exact matching step
debug (int) – level to show debug information

Returns:

a list of phrases matches

Returns:

whether to return only the first match with the best score, or all matches

Return type:

PhraseMatch

fuzzy_search.search.phrase_searcher.combine_fuzzy_and_exact_matches(filtered_matches: List[PhraseMatch], exact_matches: List[PhraseMatch], debug: int = 0)[source]

Merge fuzzy and exact phrase matches, preferring the exact match when both exist for the same phrase at the same offset.

Parameters:

filtered_matches (List[PhraseMatch]) – fuzzy matches that have passed filtering
exact_matches (List[PhraseMatch]) – exact phrase matches
debug (int) – level to show debug information

Returns:

the combined list of matches, sorted by offset

Return type:

List[PhraseMatch]

fuzzy_search.search.phrase_searcher.get_text_dict(text: str | dict | Doc, ignorecase: bool = False) → dict[source]

Check that text is in a dictionary with an id property, so that passing a long text goes by reference instead of copying the long text string.

Parameters:

text (Union[str, dict]) – a text string or text dictionary
ignorecase (bool) – boolean flag for whether to ignore case

Returns:

a text dictionary with an id property

Return type:

dict

fuzzy_search.search.phrase_searcher.make_step_timer()[source]: Create a closure that, on each call, prints and returns the elapsed time since the previous call (and the total time since the timer was created). Used for debug timing.

fuzzy_search.search.searcher module

Base fuzzy searcher class.

Defines FuzzySearcher, which indexes phrases (and their variants and distractors) using character skipgrams and provides the core skipgram-based matching logic that other, more specialized searchers in this package build upon.

class fuzzy_search.search.searcher.FuzzySearcher(phrase_list: List[any] = None, phrase_model: Dict[str, any] | PhraseModel = None, config: None | Dict[str, str | int | float] = None, tokenizer: Tokenizer = None)[source]

Bases: object

__init__(phrase_list: List[any] = None, phrase_model: Dict[str, any] | PhraseModel = None, config: None | Dict[str, str | int | float] = None, tokenizer: Tokenizer = None)[source]

This class represents the basic fuzzy searcher. You can pass a list of phrases or a phrase model and configuration dictionary that overrides the default configuration values. The default config dictionary is available via fuzzy_search.default_config.

To set e.g. the character ngram_size to 3 and the skip_size to 1 use the following dictionary:

config = {
    'ngram_size': 3,
    'skip_size': 1
}

Parameters:

phrase_list (list) – a list of phrases (a list of strings or more complex dictionaries with phrases and variants)
phrase_model (PhraseModel) – a phrase model
config (dict) – a configuration dictionary to override default configuration properties. Only the properties present in the config dictionary are updated.
tokenizer (Tokenizer) – a tokenizer instance (default tokenizer splits on whitespace)

configure(config: Dict[str, any]) → None[source]

Configure the fuzzy searcher with a given config object.

Parameters:: config (Dict[str, Union[str, int, float]]) – a config dictionary

static filter_matches_by_offset_threshold(matches: List[PhraseMatch], debug: int = 0)[source]

Filter out matches whose start offset exceeds their phrase’s configured maximum start offset.

Matches whose phrase has no max_start_offset restriction (None or -1) are always kept.

Parameters:

matches (List[PhraseMatch]) – a list of phrase matches to filter
debug (int) – level to show debug information

Returns:

the matches that satisfy the max_start_offset constraint

Return type:

List[PhraseMatch]

find_skipgram_matches(text: Dict[str, str | int | float | list], include_variants: None | bool = None, known_word_start_offset: Dict[int, Dict[str, any]] = None) → SkipMatches[source]

Find all skipgram matches between text and phrases.

Parameters:

text (Dict[str, Union[str, int, float, list]]) – the text object to match with phrases
include_variants (bool) – boolean flag for whether to include phrase variants for finding matches
known_word_start_offset (Dict[int, Dict[str, any]]) – a dictionary of known words and their text start_offsets based on exact matches

Returns:

a SkipMatches object contain all skipgram matches

Return type:

SkipMatches

index_distractors(distractors: List[str | Phrase]) → None[source]

Add a list of distractor phrases to filter out likely incorrect phrase matches.

Parameters:: distractors (List[Union[str, Phrase]]) – a list of distractors, either as string or as Phrase objects

index_phrase_model(phrase_model: List[Dict[str, str | int | float | list]] | PhraseModel, debug: int = 0)[source]

Add a phrase model to search for phrases in texts.

Parameters:

phrase_model (Union[List[Dict[str, Union[str, int, float, list]]], PhraseModel]) – a phrase model, either as dictionary or as PhraseModel object
debug (int) – level to show debug information

index_phrases(phrases: List[str | Phrase]) → None[source]

Add a list of phrases to search for in texts.

Parameters:: phrases (List[Union[str, Phrase]]) – a list of phrases, either as string or as Phrase objects

index_variants(variants: List[str | Phrase]) → None[source]

Add a list of variant phrases to search for in texts.

Parameters:: variants (List[Union[str, Phrase]]) – a list of variants, either as string or as Phrase objects

fuzzy_search.search.template_searcher module

Template-based fuzzy searcher.

Defines FuzzyTemplateSearcher, which searches text for phrase matches and then checks whether sequences of those matches fit a FuzzyTemplate made up of ordered and unordered, required and optional, label and group elements. Also includes the helper functions used to find phrase-match sequences that satisfy template group elements, and TemplateMatch, which represents a successful template match.

class fuzzy_search.search.template_searcher.FuzzyTemplateSearcher(template: None | FuzzyTemplate = None, config: None | dict = None)[source]

Bases: FuzzyContextSearcher

Fuzzy searcher that finds phrase matches in text and groups them into TemplateMatch objects wherever a sequence of matches satisfies a FuzzyTemplate.

__init__(template: None | FuzzyTemplate = None, config: None | dict = None)[source]

A fuzzy searcher for finding fuzzy matches in texts and checking if the matches fit a given template. The FuzzyTemplateSearcher incorporates a FuzzyContextSearcher for searching phrase matches in texts. The phrases are taken from the phrase model that is part of the template. The FuzzyContextSearcher uses the default configuration unless a searcher_config is specified that overrides specific properties.

Parameters:

template (FuzzyTemplate) – a fuzzy template to use for searching
searcher_config – an optional configuration dictionary to configure the FuzzyTemplateSearcher

filter_phrase_matches(phrase_matches: List[PhraseMatch]) → List[PhraseMatch][source]

Filter a list of phrase matches to only include phrase matches that have at least one label in common with the template.

Parameters:: phrase_matches (List[PhraseMatch]) – a list of phrase matches
Returns:: a filtered list of phrases matches
Return type:: List[PhraseMatch]

find_template_matches(phrase_matches: List[PhraseMatch]) → List[TemplateMatch][source]

Find all the matches that fit a template. The method returns a list of template matches, where each template match contains the phrase match that fit the template. There can be multiple template matches, if the phrase matches fit a template multiple times.

Parameters:: phrase_matches (List[PhraseMatch]) – a list of phrase matches
Returns:: a list of template matches
Return type:: List[TemplateMatch]

search_text(text: str | Dict[str, str]) → List[TemplateMatch][source]

Search phrases from the registered template’s phrase model in the text and check if the resulting matches together match the template. This method returns a dictionary including the individual phrase matches and any template matches.

Parameters:: text (Union[str, Dict[str, str]]) – a text to search in, either as a string or a dictionary with text and an identifier
Returns:: a dictionary with all phrase matches and template matches
Return type:: Dict[str, Union[List[PhraseMatch], List[TemplateMatch]]]

set_template(template: FuzzyTemplate) → None[source]

Set a new template for the searcher and index the corresponding phrase model.

Parameters:: template (FuzzyTemplate) – a fuzzy template to use for searching

class fuzzy_search.search.template_searcher.TemplateMatch(template: FuzzyTemplate, phrase_matches: List[PhraseMatch], template_sequence: Dict[str, any])[source]

Bases: object

A successful match of a FuzzyTemplate against a sequence of phrase matches found in a text.

__init__(template: FuzzyTemplate, phrase_matches: List[PhraseMatch], template_sequence: Dict[str, any])[source]

A match object for a given template, with a list of phrase matches that fill the template elements.

Parameters:

template (FuzzyTemplate) – the template that is matched
phrase_matches (List[PhraseMatch]) – the phrase matches that correspond to the template elements
template_sequence (Dict[str, any]) – a template sequence mapping each phrase match to the corresponding template labels

fuzzy_search.search.template_searcher.find_next_element_end_index(phrase_matches: List[PhraseMatch], template_element: FuzzyTemplateElement, element_start_index: int) → int[source]

Find the next phrase match that doesn’t match a template element, from a given starting point in a list of phrase matches.

Parameters:

phrase_matches (List[PhraseMatch]) – a list of phrase matches to be tested against a template element
template_element (FuzzyTemplateElement) – a template element to test the phrase matches against
element_start_index (int) – the index in the phrase list where the template elements first matches the template

Returns:

the index in the phrase list where the template element stops matching

Return type:

int

fuzzy_search.search.template_searcher.find_next_element_start_index(phrase_matches: List[PhraseMatch], template_element: FuzzyTemplateElement, template_start_index: int) → int[source]

Find the next phrase match that matches a template element, from a given starting point in a list of phrase matches.

Parameters:

phrase_matches (List[PhraseMatch]) – a list of phrase matches to be tested against a template element
template_element (FuzzyTemplateElement) – a template element to test the phrase matches against
template_start_index (int) – the index in the phrase list to start the matching process

Returns:

the index in the phrase list where the template element matches

Return type:

int

fuzzy_search.search.template_searcher.find_next_group_match_sequence(phrase_matches: List[PhraseMatch], template_group: FuzzyTemplateGroupElement, template_start_index: int) → None | Dict[str, any][source]

Find the next sequence of phrase matches that match a template group element, from a given starting point in the list of phrase matches. This function returns None if the template doesn’t match.

Parameters:

phrase_matches (List[PhraseMatch]) – a list of phrase matches to be tested against a template element
template_group (FuzzyTemplateGroupElement) – a template group element to test the phrase matches against
template_start_index (int) – the index in the phrase list to start the matching process

Returns:

a sequence with start and end indexes in the list of phrase matches that match the template group

Return type:

Union[None, Dict[str, any]]

fuzzy_search.search.template_searcher.find_next_ordered_group_match_sequence(phrase_matches: List[PhraseMatch], template_group: FuzzyTemplateGroupElement, template_start_index: int) → None | Dict[str, any][source]

Find the next sequence of phrase matches that match an ordered template group element, from a given starting point in the list of phrase matches. This function returns None if the template doesn’t match.

Parameters:

phrase_matches (List[PhraseMatch]) – a list of phrase matches to be tested against a template element
template_group (FuzzyTemplateGroupElement) – a template group element to test the phrase matches against
template_start_index (int) – the index in the phrase list to start the matching process

Returns:

a sequence with start and end indexes in the list of phrase matches that match the template group

Return type:

Union[None, Dict[str, any]]

fuzzy_search.search.template_searcher.find_next_unordered_group_match_sequence(phrase_matches: List[PhraseMatch], template_group: FuzzyTemplateGroupElement, template_start_index: int) → None | Dict[str, any][source]

Find the next sequence of phrase matches that match an unordered template group element, from a given starting point in the list of phrase matches. This function returns None if the template doesn’t match.

Parameters:

phrase_matches (List[PhraseMatch]) – a list of phrase matches to be tested against a template element
template_group (FuzzyTemplateGroupElement) – a template group element to test the phrase matches against
template_start_index (int) – the index in the phrase list to start the matching process

Returns:

a sequence with start and end indexes in the list of phrase matches that match the template group

Return type:

Union[None, Dict[str, any]]

fuzzy_search.search.template_searcher.get_phrase_match_list_labels(phrase_matches: List[PhraseMatch]) → List[str][source]

Return a list of all the labels of a list of phrase matches.

Parameters:: phrase_matches (List[PhraseMatch]) – a list of phrase matches
Returns:: a list of phrase match labels
Return type:: List[str]

fuzzy_search.search.template_searcher.get_sequence_label_element_matches(template_sequence: Dict[str, any]) → List[Dict[str, any]][source]

Recursively flatten a (possibly nested) template match sequence into a list of per-label phrase match dictionaries, annotating each with the group labels it belongs to.

Parameters:: template_sequence (Dict[str, any]) – a sequence dictionary as produced by initialize_sequence() and the find_next_*_group_match_sequence functions
Returns:: a list of dictionaries with keys “label”, “phrase_matches” and, for matches nested inside groups, “label_groups”
Return type:: List[Dict[str, any]]

fuzzy_search.search.template_searcher.has_required_matches(phrase_matches: List[PhraseMatch], template: FuzzyTemplate) → bool[source]

Check if list of phrase matches contain all required labels of a template.

Parameters:

phrase_matches (List[PhraseMatch]) – a list of phrase matches
template (FuzzyTemplate) – a fuzzy template to use for searching

Returns:

a True value only if all required labels have at least one match

fuzzy_search.search.template_searcher.initialize_sequence(element: FuzzyTemplateElement, start_index: int, end_index: int) → Dict[str, any][source]

Create an empty match-sequence dictionary for a template element, covering the given start/end indices into the list of phrase matches and with no phrase matches assigned yet.

Parameters:

element (FuzzyTemplateElement) – the template element (label or group) the sequence is for
start_index (int) – the start index in the phrase match list
end_index (int) – the end index in the phrase match list

Returns:

a sequence dictionary with keys element_label, element_type, element, start, end, phrase_matches, contains_required and element_sequences

Return type:

Dict[str, any]

fuzzy_search.search.template_searcher.share_label(object1: PhraseMatch | FuzzyTemplateElement, object2: PhraseMatch | FuzzyTemplateElement) → bool[source]

Check if two fuzzy objects (phrase matches of template elements) share at least one label.

Parameters:

object1 (Union[PhraseMatch, FuzzyTemplateElement]) – the first object to compare
object2 (Union[PhraseMatch, FuzzyTemplateElement]) – the second object to compare

Returns:

boolean value indicating that the two objects share a label

Return type:

bool

fuzzy_search.search.token_searcher module

Token-level fuzzy searcher.

Defines FuzzyTokenSearcher, which tokenizes both phrases and target text and uses character skipgrams at the token level (rather than over the whole phrase string) to find candidate token matches. These token matches are then chained into partial and full phrase matches. This is generally faster than whole-phrase skipgram matching (as done by FuzzySearcher), at the cost of being slightly less exhaustive. The module also defines the helper functions used to build a vocabulary of distractor/match term pairs and to turn token matches into phrase matches.

class fuzzy_search.search.token_searcher.FuzzyTokenSearcher(phrase_list: List[any] = None, phrase_model: Dict[str, any] | ~typing.List[~typing.Dict[str, any]] | ~fuzzy_search.phrase.phrase_model.PhraseModel=None, config: None | ~typing.Dict[str, str | int | float]=None, tokenizer: Tokenizer = None, vocabulary: [<class 'fuzzy_search.tokenization.vocabulary.Vocabulary'>, typing.List[str]]=None, index_vocabulary_pairs: bool = True, max_char_gap: int = 20, max_token_gap: int = 1, debug: int = 0)[source]

Bases: FuzzySearcher

Fuzzy searcher that matches phrases against text at the token level.

Tokenizes phrases and text and indexes phrase tokens by character skipgram, so that candidate token matches can be found per text token. Token matches are then chained into partial and full phrase matches, taking into account a vocabulary of known terms and known match/distractor term pairs to speed up and disambiguate matching.

__init__(phrase_list: List[any] = None, phrase_model: Dict[str, any] | ~typing.List[~typing.Dict[str, any]] | ~fuzzy_search.phrase.phrase_model.PhraseModel=None, config: None | ~typing.Dict[str, str | int | float]=None, tokenizer: Tokenizer = None, vocabulary: [<class 'fuzzy_search.tokenization.vocabulary.Vocabulary'>, typing.List[str]]=None, index_vocabulary_pairs: bool = True, max_char_gap: int = 20, max_token_gap: int = 1, debug: int = 0)[source]

This class represents the basic fuzzy searcher. You can pass a list of phrases or a phrase model and configuration dictionary that overrides the default configuration values. The default config dictionary is available via fuzzy_search.default_config.

To set e.g. the character ngram_size to 3 and the skip_size to 1 use the following dictionary:

config = {
    'ngram_size': 3,
    'skip_size': 1
}

Parameters:

phrase_list (list) – a list of phrases (a list of strings or more complex dictionaries with phrases and variants)
phrase_model (PhraseModel) – a phrase model
config (dict) – a configuration dictionary to override default configuration properties. Only the properties present in the config dictionary are updated.
tokenizer (Tokenizer) – a tokenizer instance

add_vocabulary(vocab: List[str] | Vocabulary)[source]

Add terms to the searcher’s vocabulary.

Parameters:: vocab (Union[List[str], Vocabulary]) – a list of term strings, or a Vocabulary instance to merge in

add_vocabulary_skipgram_matches()[source]: Precompute and cache, for every term in the vocabulary, its skipgram matches against phrase tokens, removing any matches that are registered distractor pairs or that have match type MatchType.NONE.

configure(config: Dict[str, any])[source]

Update any existing instance attributes of this searcher from a config dictionary.

Parameters:: config (Dict[str, any]) – a dictionary of configuration property names and values

find_matches(text: Doc | str | Dict[str, any], debug: int = 0) → List[PhraseMatch][source]

Find all fuzzy matching phrases for a given text using token-based searching. The FuzzyTokenSearcher turns the phrases and the target text into lists of word tokens (the tokenizer is configurable) and uses character skip grams to identify candidate phrase tokens matching tokens in the text. It then uses token sequences to identify fuzzy matches.

This speeds up the search (especially for the default settings ngram_size=2 and skip_size=2) at the cost of slightly less exhaustive search.

Parameters:

text (Union[str, Dict[str, str]]) – A tokenized text.
debug (int) – level to show debug information

Returns:

a list of phrase matches

Return type:

List[PartialPhraseMatch]

find_skipgram_token_matches_for_token(text_token: Token, partial_matches: Dict[str, List[Token]], token_matches: List[TokenMatch], debug: int = 0)[source]

Find all token matches between text tokens and phrase tokens using skipgrams.

Parameters:

text_token (Token) – a single text token to match with phrase tokens
partial_matches (Dict[str, List[Token]]) – a dictionary of phrase token strings an their partial text token matches
token_matches (List[TokenMatch]) – a list of matches between text tokens and phrase tokens
debug (int) – level to show debug information

find_skipgram_token_matches_in_text(text: Doc | str | Dict[str, any], debug: int = 0) → List[TokenMatch][source]

Find all token matches between text tokens and phrase tokens using skipgrams.

Parameters:

text (Dict[str, Union[str, int, float, list]]) – the text object to match with tokens
debug (int) – level to show debug information

Returns:

a list of matches between text tokens and phrase tokens

Return type:

List[TokenMatch]

find_vocabulary_text_phrase_term_pairs()[source]

Find, for every term in the vocabulary, the phrase tokens it has skipgram overlap with, and classify each (term, phrase_token) pair as a match or a distractor pair based on is_distractor().

Returns:: a tuple (match_pairs, distractor_pairs), each a set of ((term,), (phrase_token,)) tuples
Return type:: tuple

has_distractor_pair(text_terms: str | tuple, phrase_terms: str | tuple)[source]: Check whether (text_terms, phrase_terms) is registered as a known distractor pair.

has_match_pair(text_terms: str | tuple, phrase_terms: str | tuple)[source]: Check whether (text_terms, phrase_terms) is registered as a known match pair.

has_text_phrase_term_pair(text_terms: str | tuple, phrase_terms: str | tuple, pair_type: str)[source]: Check whether (text_terms, phrase_terms) is registered as a pair of the given type (‘match’ or ‘distractor’).

index_distractor_pair(text_terms: str | tuple, phrase_terms: str | tuple)[source]: Register text_terms as a distractor pair for phrase_terms (i.e. similar enough to be a skipgram candidate, but not an actual match).

index_phrase_token_skipgrams(debug: int = 0)[source]

Index the character skipgrams of every token that occurs in the registered phrase model, so that text tokens can later be looked up by skipgram.

Parameters:: debug (int) – level to show debug information

index_text_phrase_term_pair(text_terms: str | tuple, phrase_terms: str | tuple, pair_type: str)[source]

Index a single (text_terms, phrase_terms) pair as either a ‘match’ or ‘distractor’ pair.

For distractor pairs, also registers the text term’s skipgram matches against this phrase term as MatchType.NONE so it will be excluded from later matching.

Parameters:

text_terms (Union[str, tuple]) – the text term(s) of the pair
phrase_terms (Union[str, tuple]) – the phrase term(s) of the pair
pair_type (str) – either ‘match’ or ‘distractor’

index_text_phrase_term_pairs(text_phrase_term_pairs, pair_type: str)[source]

Index a collection of (text_terms, phrase_terms) pairs as either ‘match’ or ‘distractor’ pairs.

Parameters:

text_phrase_term_pairs – an iterable of (text_terms, phrase_terms) tuples
pair_type (str) – either ‘match’ or ‘distractor’

terms_to_id_tuple(terms: str | tuple)[source]: Convert terms to a tuple of their vocabulary term ids, or None if any term is not in the vocabulary.

static terms_to_string(terms: str | List[str] | tuple)[source]: Convert a string, list of strings, or tuple of terms into a single space-joined string.

static terms_to_tuple(terms: str | List[str] | tuple)[source]: Convert a string, list of strings, or tuple of terms into a tuple of terms.

class fuzzy_search.search.token_searcher.PartialPhraseMatch(phrase: Phrase, token_matches: List[TokenMatch] = None, max_char_gap: int = 20, max_token_gap: int = 1)[source]

Bases: object

A growing token-based match between a phrase and a sequence of text tokens, built up incrementally from individual TokenMatch objects while tracking missing and redundant phrase tokens and enforcing maximum character/token gaps between matched tokens.

__init__(phrase: Phrase, token_matches: List[TokenMatch] = None, max_char_gap: int = 20, max_token_gap: int = 1)[source]

Create a PartialPhraseMatch for a given phrase.

Parameters:

phrase (Phrase) – the phrase being matched
token_matches (List[TokenMatch]) – an optional initial list of token matches to add
max_char_gap (int) – the maximum allowed character gap between consecutive matched tokens
max_token_gap (int) – the maximum allowed token index gap between consecutive matched tokens

add_tokens(token_matches: List[TokenMatch] | TokenMatch)[source]

Add one or more token matches to this partial match and update its derived state.

Parameters:: token_matches (Union[List[TokenMatch], TokenMatch]) – a single token match or a list of token matches to add

pop()[source]: Remove the first (earliest) token match from this partial match and update its derived state (text/phrase tokens, offsets, length).

push(token_match: TokenMatch)[source]

Add a new token match to the end of this partial match, resetting the match if the gap to the new token match exceeds the configured maximum character/token gap, and updating which phrase tokens are missing or redundant.

Parameters:: token_match (TokenMatch) – the token match to add

class fuzzy_search.search.token_searcher.TokenMatch(text_tokens: Token | List[Token], phrase_tokens: str | List[str], match_type: MatchType)[source]

Bases: object

A match between one or more text tokens and one or more phrase tokens, with the type of match (full or partial) between them.

__init__(text_tokens: Token | List[Token], phrase_tokens: str | List[str], match_type: MatchType)[source]

Create a TokenMatch.

Parameters:

text_tokens (Union[Token, List[Token]]) – one or more text tokens involved in the match
phrase_tokens (Union[str, List[str]]) – one or more phrase tokens involved in the match
match_type (MatchType) – the type of match between the text and phrase tokens

fuzzy_search.search.token_searcher.copy_partial_match(partial_match: PartialPhraseMatch)[source]

Create a deep-ish copy of a PartialPhraseMatch, copying its tracked token and offset state.

Parameters:: partial_match (PartialPhraseMatch) – the partial match to copy
Returns:: a new, independent PartialPhraseMatch with the same state
Return type:: PartialPhraseMatch

fuzzy_search.search.token_searcher.get_partial_phrases(token_matches: List[TokenMatch], token_searcher: FuzzyTokenSearcher, max_char_gap: int = 20, debug: int = 0)[source]

Chain a sequence of token matches into candidate (partial) phrase matches.

Walks through the token matches in order, extending open partial matches for a phrase when a token match continues them, starting new partial matches when appropriate, and moving partial matches to the candidate list once they are too far (more than max_char_gap characters) from the next token match or once all tokens have been processed. Candidates that are incomplete (when a complete candidate exists for the same phrase) or whose length deviates too much from the phrase length are discarded.

Parameters:

token_matches (List[TokenMatch]) – a list of token matches between text tokens and phrase tokens, in text order
token_searcher (FuzzyTokenSearcher) – the token searcher holding phrase model and configuration
max_char_gap (int) – maximum character gap allowed between two token matches that belong to the same phrase match
debug (int) – level to show debug information

Returns:

candidate partial phrase matches grouped by phrase

Return type:

Dict[Phrase, List[PartialPhraseMatch]]

fuzzy_search.search.token_searcher.get_text_string(text: str | Dict[str, any] | Doc) → str[source]

Return the plain text string for the given text, regardless of its representation.

Parameters:: text (Union[str, Dict[str, any], Doc]) – a text string, a dictionary with a ‘text’ property, a Doc, or a list of tokens
Returns:: the underlying text string
Return type:: str

fuzzy_search.search.token_searcher.get_text_tokens(text: str | Dict[str, any] | Doc, tokenizer: Tokenizer = None)[source]

Return the list of tokens for the given text, tokenizing it if necessary.

Parameters:

text (Union[str, Dict[str, any], Doc]) – a text string, a dictionary with ‘text’ and ‘id’ properties, a Doc, or a list of Token objects (returned unchanged)
tokenizer (Tokenizer) – the tokenizer to use if text is not already tokenized

Returns:

a list of tokens

Return type:

List[Token]

fuzzy_search.search.token_searcher.get_token_skip_match_type(text_token_string: str, text_token_num_skips: int, skip_matches: SkipMatches, phrase_token_match: str, token_searcher: FuzzyTokenSearcher, debug: int = 0) → MatchType[source]

Determine the match type between a text token and a candidate phrase token, based on the proportion of skipgrams they share and the difference in their lengths.

Returns MatchType.NONE if the skipgram overlap is below the configured threshold for both tokens, or if the length difference (accounting for overlap) exceeds the configured maximum token length variance. Otherwise, returns MatchType.FULL if the tokens are about the same length, MatchType.PARTIAL_OF_PHRASE_TOKEN if the text token is shorter than the phrase token (i.e. the text token may be one of several tokens making up the phrase token), or MatchType.PARTIAL_OF_TEXT_TOKEN if the text token is longer.

Parameters:

text_token_string (str) – the normalised string of the text token
text_token_num_skips (int) – the number of skipgrams generated for the text token
skip_matches (SkipMatches) – the skip matches data for the candidate phrase token
phrase_token_match (str) – the candidate phrase token string
token_searcher (FuzzyTokenSearcher) – the token searcher holding configuration thresholds
debug (int) – level to show debug information

Returns:

the determined match type

Return type:

MatchType

fuzzy_search.search.token_searcher.get_token_skip_match_types(token_searcher: FuzzyTokenSearcher, text_token: Token, token_skip_matches: SkipMatches, text_token_skips, debug: int = 0)[source]

Classify the match type of every phrase token found in token_skip_matches and store the result in token_skip_matches.match_type.

Parameters:

token_searcher (FuzzyTokenSearcher) – the token searcher holding configuration thresholds
text_token (Token) – the text token that was matched
token_skip_matches (SkipMatches) – the skip matches to classify, updated in place
text_token_skips – the list of skipgrams generated for the text token
debug (int) – level to show debug information

fuzzy_search.search.token_searcher.get_token_skipgram_matches(text_token: Token, token_searcher: FuzzyTokenSearcher, debug: int = 0)[source]

Find all phrase tokens that share character skipgrams with the given text token (skipping any pairs registered as distractors or outside the phrase token’s allowed offset range), and classify each match’s type.

Parameters:

text_token (Token) – the text token to match against indexed phrase tokens
token_searcher (FuzzyTokenSearcher) – the token searcher holding the phrase token skipgram index
debug (int) – level to show debug information

Returns:

a SkipMatches object with the matching phrase tokens and their match types

Return type:

SkipMatches

fuzzy_search.search.token_searcher.get_tokenized_doc(text: str | Dict[str, any] | Doc, tokenizer: Tokenizer) → Doc[source]

Return text as a tokenized Doc, tokenizing it with the given tokenizer if needed.

Parameters:

text (Union[str, Dict[str, any], Doc]) – a text string, a dictionary with ‘text’ and ‘id’ properties, or a Doc
tokenizer (Tokenizer) – the tokenizer to use if text is not already a Doc

Returns:

the tokenized document

Return type:

Doc

fuzzy_search.search.token_searcher.get_vocabulary_skipgram_matches(text_token: Token, token_searcher: FuzzyTokenSearcher, debug: int = 0)[source]

Look up the precomputed skipgram matches for a vocabulary text token and shift their offsets to the token’s actual position in the text.

Parameters:

text_token (Token) – a text token that is part of the searcher’s vocabulary
token_searcher (FuzzyTokenSearcher) – the token searcher holding cached vocabulary skipgram matches
debug (int) – level to show debug information

Returns:

a SkipMatches object with offsets relative to the text

Return type:

SkipMatches

fuzzy_search.search.token_searcher.has_max_end_offset(phrase: Phrase)[source]: Return whether the phrase has a configured maximum end offset restriction.

fuzzy_search.search.token_searcher.has_max_start_offset(phrase: Phrase)[source]: Return whether the phrase has a configured maximum start offset restriction.

fuzzy_search.search.token_searcher.is_distractor(text_token: str, phrase_token: str, dist_threshold: int = 2, debug: int = 0)[source]: Check if a text token is a distractor for a phrase token.

fuzzy_search.search.token_searcher.map_text_tokens_to_phrase_tokens(partial_match: PartialPhraseMatch) → Dict[str, List[str]] | None[source]

Build a mapping from text token strings to the phrase token strings they matched, for a partial phrase match.

Parameters:: partial_match (PartialPhraseMatch) – a partial phrase match with its token matches
Returns:: a dictionary mapping text token strings to lists of matched phrase token strings, or None if the partial match does not cover all of the phrase’s tokens
Return type:: Union[Dict[str, List[str]], None]

fuzzy_search.search.token_searcher.token_is_out_of_phrase_range(token: Token, phrase: Phrase, token_searcher: FuzzyTokenSearcher)[source]

Check whether a text token’s character position falls outside the phrase’s configured max_start_offset / max_end_offset range, accounting for the token’s expected offset within the phrase.

Parameters:

token (Token) – a text token
phrase (Phrase) – the phrase to check the token’s position against
token_searcher (FuzzyTokenSearcher) – the token searcher holding the phrase model

Returns:

True if the token falls outside the phrase’s allowed offset range

Return type:

bool

fuzzy_search.search.token_searcher.token_within_phrase_offset(token_searcher: FuzzyTokenSearcher, text_token: Token, phrase_token: str, debug: int = 0)[source]

Check whether a text token’s character position is within the configured maximum start/end offset for the given phrase token.

Parameters:

token_searcher (FuzzyTokenSearcher) – the token searcher holding the phrase model
text_token (Token) – the text token to check
phrase_token (str) – the phrase token string being matched against
debug (int) – level to show debug information

Returns:

True if the text token is within the allowed offset range

Return type:

bool

Module contents

Search components for fuzzy matching phrases, tokens, and templates in text.