fuzzy_search.phrase package

Submodules

fuzzy_search.phrase.phrase module

The Phrase class, representing a single phrase to be matched against text, along with its skipgram index and other metadata used by the fuzzy matching algorithms.

class fuzzy_search.phrase.phrase.Phrase(phrase: str | Dict[str, str], ngram_size: int = 2, skip_size: int = 2, early_threshold: int = 3, late_threshold: int = 3, within_range_threshold: int = 3, ignorecase: bool = False, tokens: List[Token] = None, tokenizer: Tokenizer = None)[source]

Bases: object

A phrase to be matched against text. Computes and indexes the phrase’s skipgrams (and an early/late subset of them) on construction, which are used by the fuzzy matching algorithms to score candidate matches in noisy/historical text.

__init__(phrase: str | Dict[str, str], ngram_size: int = 2, skip_size: int = 2, early_threshold: int = 3, late_threshold: int = 3, within_range_threshold: int = 3, ignorecase: bool = False, tokens: List[Token] = None, tokenizer: Tokenizer = None)[source]

Create a Phrase from a string or a phrase dictionary.

Parameters:

phrase (Union[str, Dict[str, str]]) – a phrase string, or a dict with at least a ‘phrase’ key and optional ‘label’, ‘metadata’ and other keys
ngram_size (int) – the size of the ngrams used to compute skipgrams
skip_size (int) – the maximum number of characters skipped between ngram parts
early_threshold (int) – the character offset below which a skipgram is considered “early”
late_threshold (int) – the number of characters from the end below which a skipgram is considered “late”
within_range_threshold (int) – the maximum offset distance for two skipgrams to be considered within range
ignorecase (bool) – whether to ignore case when matching
tokens (List[Token]) – an optional pre-computed list of tokens for the phrase
tokenizer (Tokenizer) – an optional tokenizer to tokenize the phrase string if tokens is not given

add_max_end_offset(max_end_offset: int) → None[source]

Add a maximum offset from the end for matching a phrase in a text.

Parameters:: max_end_offset (int) – the maximum offset from the end to allow a phrase to match

add_max_start_offset(max_start_offset: int) → None[source]

Add a maximum offset from the start for matching a phrase in a text.

Parameters:: max_start_offset (int) – the maximum offset from the start to allow a phrase to match

add_metadata(metadata_dict: Dict[str, any]) → None[source]

Add key/value pairs as metadata for this phrase.

Parameters:: metadata_dict (Dict[str, any]) – a dictionary of key/value pairs as metadata
Returns:: None
Return type:: None

has_label(label_string: str) → bool[source]

Check if a given label belongs to at least one phrase in the phrase model.

Parameters:: label_string (str) – a label string
Returns:: a boolean whether the label is part of the phrase model
Return type:: bool

has_max_end_offset() → bool[source]: Return whether this phrase has a maximum end offset configured.

has_max_start_offset() → bool[source]: Return whether this phrase has a maximum start offset configured.

has_skipgram(skipgram: str) → bool[source]

For a given skipgram, return boolean whether it is in the index

Parameters:: skipgram (str) – an skipgram string
Returns:: A boolean whether skipgram is in the index
Return type:: bool

is_early_skipgram(skipgram: str) → bool[source]

For a given skipgram, return boolean whether it appears early in the phrase.

Parameters:: skipgram (str) – an skipgram string
Returns:: A boolean whether skipgram appears early in the phrase
Return type:: bool

set_label(label: str | List[str]) → None[source]

Set the label(s) of a phrase. Labels must be string and can be a single string or a list.

Parameters:: label (Union[str, List[str]]) – the label(s) of a phrase

skipgram_offsets(skipgram_string: str) → None | List[int][source]

For a given skipgram return the list of offsets at which it appears.

Parameters:: skipgram_string (str) – an skipgram string
Returns:: A list of string offsets at which the skipgram appears
Return type:: Union[None, List[int]]

within_range(skipgram1, skipgram2)[source]

Check whether two skipgrams occur close enough together in the phrase (within within_range_threshold characters) to be considered within range of each other.

Parameters:

skipgram1 – the first skipgram string
skipgram2 – the second skipgram string

Returns:

whether the two skipgrams are within range of each other

Return type:

bool

fuzzy_search.phrase.phrase.is_valid_label(label: str | List[str]) → bool[source]

Test whether label has a valid value.

Parameters:: label (Union[str, List[str]]) – a phrase label (either a string or a list of strings)
Returns:: whether the label is valid
Return type:: bool

fuzzy_search.phrase.phrase_model module

The PhraseModel class, which holds a collection of phrases together with their spelling variants, distractors, labels, and custom metadata, and provides the indexes used to search for them in text.

Bases: object

A collection of phrases for fuzzy searching, along with their registered spelling variants, distractors, labels and custom metadata. Maintains the indexes (by word, token, length, label, etc.) used by the search routines to find matching phrases efficiently.

Create a PhraseModel, optionally pre-populated with phrases, variants, distractors, labels, an entire model, and/or custom metadata.

Parameters:

phrases (Union[None, List[Union[str, Dict[str, Union[str, list]], Phrase]]]) – a list of phrases (strings, phrase dictionaries, or Phrase objects)
variants (Union[None, List[Union[Dict[str, List[str]], Phrase]]]) – a list of phrase dictionaries with a ‘variants’ property
phrase_labels (Union[None, List[Dict[str, str]]]) – a list of phrase/label dictionaries
distractors (Union[None, List[Union[Dict[str, List[str]], Phrase]]]) – a list of phrase dictionaries with a ‘distractors’ property
model (Union[None, List[Dict[str, Union[str, list]]]]) – a list of phrase dictionaries combining phrases, variants, distractors, labels and custom data
custom (Union[None, List[Dict[str, Union[str, int, float, list]]]]) – a list of phrase dictionaries with custom metadata properties
config (dict) – a configuration dictionary, supporting ‘ngram_size’ and ‘skip_size’ keys
tokenizer (Tokenizer) – an optional tokenizer used to tokenize phrases

add_custom(custom: List[Phrase | Dict[str, str | int | float | list]]) → None[source]

Add custom key/value pairs to the entry as phrase metadata.

param entry: an Array of phrase dictionaries, each with a ‘phrase’ property and additional key/value pairs type entry: Dict[str, Union[str, int, float, list]]

add_distractor(distractor_phrase: Phrase, main_phrase: Phrase)[source]

Add a phrase to the model as distractor of a given main phrase.

Parameters:

distractor_phrase (Phrase) – a distractor phrase to be added as distractor of main_phrase
main_phrase (Phrase) – a main phrase that the distractor phrase is a distractor of

add_distractors(distractors: List[Phrase | Dict[str, str | List[str]]], add_new_phrases: bool = True) → None[source]

Add distractors of a phrase. If the phrase is not registered, add it to the set. - input is a list of dictionaries: distractors = [{‘phrase’: ‘some phrase’, ‘distractors’: [‘some distractor’, ‘some other distractor’]}]

Parameters:

distractors (List[Dict[str, Union[str, List[str]]]]) – a list of phrase dictionaries with ‘distractor’ property
add_new_phrases (bool) – a Boolean to indicate if unknown phrases should be added

add_labels(phrase_labels: List[Phrase | Dict[str, str | List[str]]])[source]: Add a label to a phrase. This can be used to group phrases under the same label. - input is a list of phrase/label pair dictionaries: labels = [{‘phrase’: ‘some phrase’, ‘label’: ‘some label’}]

add_model(model: List[str | Dict[str, str | list]]) → None[source]

Add an entire model with list of phrase dictionaries.

Parameters:: model (List[Union[str, Dict[str, Union[str list]]]]) – a list of phrase dictionaries
Returns:: None
Return type:: None

add_phrase(phrase: Phrase) → None[source]

Add a phrase to the model as main phrase.

Parameters:: phrase (Phrase) – a phrase to be added

add_phrases(phrases: List[str | Dict[str, str | List[str]] | Phrase]) → None[source]

Add a list of phrases to the phrase model. Phrases must be either: - a list of strings - a list of dictionaries with property ‘phrase’ and the phrase as a string value - a list of Phrase objects

Parameters:: phrases (List[Union[str, Dict[str, Union[str, List[str]]]]]) – a list of phrases

add_variant(variant_phrase: Phrase, main_phrase: Phrase)[source]

Add a phrase to the model as variant of a given main phrase.

Parameters:

variant_phrase (Phrase) – a variant phrase to be added as variant of main_phrase
main_phrase (Phrase) – a main phrase that the variant phrase is a variant of

add_variants(variants: List[Phrase | Dict[str, str | List[str]]], add_new_phrases: bool = True) → None[source]

Add variants of a phrase. If the phrase is not registered, add it to the set. - input is a list of dictionaries: variants = [{‘phrase’: ‘some phrase’, ‘variants’: [‘some variant’, ‘some other variant’]}]

Parameters:

variants (List[Dict[str, Union[str, List[str]]]]) – a list of phrases or phrase dictionaries with ‘variant’ property
add_new_phrases (bool) – a Boolean to indicate if unknown phrases should be added

get(phrase_string: str, custom_property: str) → any[source]

Get the value of a custom property for a given phrase.

Parameters:

phrase_string (str) – a phrase string of a registered phrase.
custom_property (str) – the name of a custom property of the registered phrase

Returns:

the custom property of a given phrase

Return type:

any

get_labels(phrase: str | Phrase) → Set[str][source]

Return the label(s) of a registered phrase.

Parameters:: phrase (Union[str, Phrase]) – a phrase string or object
Returns:: a set of labels
Return type:: List[str]

get_phrase(phrase_string: str)[source]

Return the indexed phrase object for a given phrase string, or None if no phrase has that phrase string.

Parameters:: phrase_string (str) – a string representation of an indexed phrase
Returns:: the phrase object that has the given phrase
Return type:: Union[Phrase, None]

get_phrases() → List[Phrase][source]

Return a list of all registered phrases.

Returns:: a list of all registered phrases
Return type:: List[Phrase]

get_phrases_by_max_length(max_length: int, include_variants: bool = False) → Generator[Phrase, None, None][source]

Return all phrase in the phrase model that are no longer than a given length.

Parameters:

max_length (int) – the maximum length of phrases to be returned
include_variants – whether to include variants

Returns:

a generator that yield phrases

Return type:

Generator[Phrase, None, None]

get_variants(phrases: List[str] = None) → List[Dict[str, str | List[str]]][source]

Return registered variants of a specific list of phrases or of all registered phrases (when no list of phrases is given).

Parameters:: phrases (List[str]) – a list of registered phrase strings
Returns:: a list of dictionaries of phrases and their variants
Return type:: List[Dict[str, Union[str, List[str]]]]

has_custom(phrase_string: str, custom_property: str) → bool[source]

Check if a phrase has a given custom property.

Parameters:

phrase_string (str) – a phrase string of a registered phrase.
custom_property (str) – the name of a custom property of the registered phrase

Returns:

a boolean to indicate whether the phrase has a custom property of the given property name

Return type:

bool

has_label(phrase_string: str) → bool[source]

Check if a registered phrase has a label.

Parameters:: phrase_string (str) – a phrase string of a registered phrase
Returns:: a boolean indicating if the registered phrase has a label

has_phrase(phrase: str | Dict[str, any] | Phrase) → bool[source]

Check if phrase is registered in phrase_model.

Parameters:: phrase (Union[str, Dict[str, any], Phrase]) – a phrase string
Returns:: a boolean indicating whether phrase is registered
Return type:: bool

has_token(token: str | Token)[source]

Check if a given token occurs in any registered phrase.

Parameters:: token (Union[str, Token]) – a token object whose normalized form is checked against the token index
Returns:: whether the token occurs in any registered phrase
Return type:: bool

is_label(label: str) → bool[source]

Check if label is registered as label of any known phrase.

Parameters:: label (str) – a label string to be checked
Returns:: a boolean whether the label belongs to a known phrase
Return type:: bool

property json: List[Dict[str, str | List[str]]]

Return a JSON representation of the phrase model.

Returns:: a JSON respresentation of the phrase model
Return type:: List[Dict[str, Union[str, List[str]]]]

remove_custom(custom: List[Dict[str, any]]) → None[source]

Remove custom properties for a list of phrases.

Parameters:: custom (List[Dict[str, any]]) – a list of phrase dictionaries with custom properties to remove

remove_distractor(distractor_phrase: Phrase) → None[source]

Remove a distractor phrase from the model, including its connection to the phrase it is a distractor of.

Parameters:: distractor_phrase (Phrase) – a phrase that is registered as a distractor of one or more main phrases

remove_distractors(distractors: List[str | Phrase] | None = None, distractors_of_phrase: str | None = None)[source]

Remove a list of distractors of a phrase. - distractors: a list of dictionaries with phrases as key and the list of distractors to be removed as values distractors = [{‘phrase’: ‘some phrase’, ‘distractors’: [‘distractor to remove’, ‘some other distractor’]}] - phrase: remove all distractors of a given phrase

Parameters:

distractors (Union[List[Union[str, Phrase]], None]) – an optional list of phrase dictionaries with ‘distractors’ property
distractors_of_phrase (Union[str, None]) – an optional string of a registered phrase for which all distractors are removed

remove_labels(phrases: List[Phrase] | List[str]) → None[source]

Remove labels for known phrases.

Parameters:: phrases (Union[List[Phrase], List[str]]) – is a list of known phrases (either as Phrase objects or strings)

remove_phrase(phrase: Phrase)[source]

Remove a main phrase from the model, including its connections to any variant and distractor phrases.

Parameters:: phrase (Phrase) – a phrase that is registered as a main phrase

remove_phrases(phrases: List[str | Dict[str, str | List[str]] | Phrase])[source]

Remove a list of phrases from the phrase model. If it has any registered spelling variants, remove those as well.

Parameters:: phrases (List[Union[str, Dict[str, Union[str, List[str]]]]]) – a list of phrases/keyphrases

remove_variant(variant_phrase: Phrase) → None[source]

Remove a variant phrase from the model, including its connection to the phrase it is a variant of.

Parameters:: variant_phrase (Phrase) – a phrase that is registered as a variant of one or more main phrases

remove_variants(variants: List[str | Phrase] | None = None, variants_of_phrase: str | Phrase | None = None)[source]

Remove a list of spelling variants of a phrase.

Parameters:

variants (Union[List[str, Phrase]], None]) – a list of variant strings or variant phrase objects to remove
variants_of_phrase (Union[str, Phrase, None]) – an optional phrase string or phrase object for which all variants are to be removed

set_phrase_token_max_end_offsets()[source]: Check if a token only occurs in phrases with a max end offset, and if so set its max.

set_phrase_token_max_start_offsets()[source]: Check if a token only occurs in phrases with a max start offset, and if so set its max.

variant_of(variant: str | Phrase) → None | Phrase[source]

Return the main phrase that a given variant is registered as a variant of.

Parameters:: variant (Union[str, Phrase]) – a variant phrase string or phrase object
Returns:: the main phrase the variant belongs to, or None if it is not a registered variant
Return type:: Union[None, Phrase]

variants(phrase: str | Phrase) → None | List[Phrase][source]

Return all variants of a given phrase.

Parameters:: phrase (Union[str, Phrase]) – a phrase string or phrase object
Returns:: a list of variants of the phrase or None if it doesn’t have any
Type:: Union[None, List[Phrase]]

fuzzy_search.phrase.phrase_model.as_phrase_object(phrase: str | dict | Phrase, ngram_size: int = 2, skip_size: int = 2, tokenizer: Tokenizer = None) → Phrase[source]

Coerce a phrase given as a string, dictionary, or Phrase object into a Phrase object.

Parameters:

phrase (Union[str, dict, Phrase]) – a phrase string, phrase dictionary, or Phrase object
ngram_size (int) – the ngram size to use when constructing a new Phrase
skip_size (int) – the skip size to use when constructing a new Phrase
tokenizer (Tokenizer) – an optional tokenizer to use when constructing a new Phrase

Returns:

a Phrase object

Return type:

Phrase

fuzzy_search.phrase.phrase_model.is_phrase_dict(phrase_dict: Dict[str, str | List[str]]) → bool[source]

Check whether a dictionary is a valid phrase dictionary, i.e. it has a ‘phrase’ string value, and any ‘variants’, ‘distractors’ and ‘labels’ values are valid strings/lists of strings.

Parameters:: phrase_dict (Dict[str, Union[str, List[str]]]) – a dictionary to validate as a phrase dictionary
Returns:: whether the dictionary is a valid phrase dictionary
Return type:: bool

Module contents

Phrase and phrase model classes for representing fuzzy-searchable phrases.