fuzzy_search.tokenization package

Submodules

fuzzy_search.tokenization.string module

String-level helper functions for fuzzy term comparison.

Provides character ngram generation, ngram/character overlap scoring, Levenshtein-based similarity scoring, skipgram generation for strings and tokens, and small utilities for stripping non-word prefixes/suffixes.

class fuzzy_search.tokenization.string.SkipGram(skipgram_string: str, start_offset: int, end_offset: int, skipgram_length: int)

Bases: object

Represents a single skipgram extracted from a string.

string

The skipgram’s character string.

Type:

str

start_offset

The character offset where the skipgram starts.

Type:

int

end_offset

The character offset where the skipgram ends.

Type:

int

length

The span (in characters) covered by the skipgram in the source string.

Type:

int

__init__(skipgram_string: str, start_offset: int, end_offset: int, skipgram_length: int)

Initializes the SkipGram instance.

Parameters:

• skipgram_string (str) – The skipgram’s character string.

• start_offset (int) – The character offset where the skipgram starts.

• end_offset (int) – The character offset where the skipgram ends.

• skipgram_length (int) – The span (in characters) covered by the skipgram.

fuzzy_search.tokenization.string.get_non_word_prefix(string: str) → str

Check if a string has a non-word prefix and return it.

Parameters:

string (str) – the string from which the prefix is to be return

Returns:

the non-word prefix

Return type:

str

fuzzy_search.tokenization.string.get_non_word_suffix(string: str) → str

Check if a string has a non-word suffix and return it.

Parameters:

string (str) – the string from which the suffix is to be return

Returns:

the non-word suffix

Return type:

str

fuzzy_search.tokenization.string.insert_skips(window: str, skipgram_combinations: List[Tuple[int]])

For a given skip gram window, return all skip grams for a given configuration.

Parameters:

• window (str) – A substring (the sliding window) from which skipgrams are built. The first character of the window is always included.

• skipgram_combinations (List[Tuple[int]]) – Index combinations (relative to window, excluding index 0) specifying which characters after the first to combine.

Yields:

Tuple[str, int, Tuple[int, …]] – The skipgram string, the (1-based) length spanned within the window, and the index combination (including the leading 0) used to build it. Combinations whose indexes fall outside window are silently skipped.

fuzzy_search.tokenization.string.make_ngrams(text: str, n: int) → List[str]

Turn a term string into a list of ngrams of size n

Parameters:

• text (str) – a text string

• n (int) – the ngram size

Returns:

a list of ngrams

Return type:

List[str]

fuzzy_search.tokenization.string.score_char_overlap(term1: str, term2: str) → int

Count the number of overlapping character tokens in two strings.

Parameters:

• term1 (str) – a term string

• term2 (str) – a term string

Returns:

the number of overlapping ngrams

Return type:

int

fuzzy_search.tokenization.string.score_char_overlap_ratio(term1, term2)

Score the number of overlapping characters between two terms as proportion of the length of the first term

Parameters:

• term1 (str) – a term string

• term2 (str) – a term string

Returns:

the number of overlapping ngrams

Return type:

int

fuzzy_search.tokenization.string.score_levenshtein_distance(term1: str, term2: str) → int

Calculate Levenshtein distance between two string.

Parameters:

• term1 (str) – a term string

• term2 (str) – a term string

Returns:

the number of overlapping ngrams

Return type:

int

fuzzy_search.tokenization.string.score_levenshtein_similarity_ratio(term1, term2, score_cutoff: int = None)

Score the levenshtein similarity between two terms

Parameters:

• term1 (str) – a term string

• term2 (str) – a term string

• score_cutoff (int) – the maximum distance beyond which distance calculation should be cut off

Returns:

the number of overlapping ngrams

Return type:

int

fuzzy_search.tokenization.string.score_ngram_overlap(term1: str, term2: str, ngram_size: int)

Score the number of overlapping ngrams between two terms

Parameters:

• term1 (str) – a first term string

• term2 (str) – a second term string

• ngram_size (int) – the character ngram size

Returns:

the number of overlapping ngrams

Return type:

int

fuzzy_search.tokenization.string.score_ngram_overlap_ratio(term1, term2, ngram_size)

Score the number of overlapping ngrams between two terms as proportion of the length of the first term

Parameters:

• term1 (str) – a term string

• term2 (str) – a term string

• ngram_size (int) – the character ngram size

Returns:

the number of overlapping ngrams

Return type:

int

fuzzy_search.tokenization.string.strip_prefix(string: str) → str

Strip non-word prefix from string ending.

Parameters:

string (str) – the string from which the prefix is to be stripped

Returns:

the stripped string

Return type:

str

fuzzy_search.tokenization.string.strip_suffix(string: str) → str

Strip non-word suffix from string ending.

Parameters:

string (str) – the string from which the suffix is to be stripped

Returns:

the stripped string

Return type:

str

fuzzy_search.tokenization.string.text2skipgrams(text: str, ngram_size: int = 2, skip_size: int = 2) → Generator[SkipGram, None, None]

Turn a text string into a list of skipgrams.

Parameters:

• text (str) – an text string

• ngram_size (int) – an integer indicating the number of characters in the ngram

• skip_size (int) – an integer indicating how many skip characters in the ngrams

Returns:

An iterator returning tuples of skip_gram and offset

Return type:

Generator[tuple]

Algorithm:

A sliding window of size ngram_size + skip_size is moved one character at a time across text. For each window position, insert_skips enumerates all ways of picking ngram_size - 1 characters (in order, after the first) from the window to combine with the window’s first character, producing all skipgrams of ngram_size characters that allow up to skip_size skipped characters between them. ngram_size == 1 and very short texts are handled as special cases that bypass the windowing logic.

fuzzy_search.tokenization.string.token2skipgrams(token: str, ngram_size: int = 2, skip_size: int = 2, pad_token: bool = True) → Generator[SkipGram, None, None]

Turn a (padded) token string into a list of skipgrams.

Parameters:

• token (str) – a token string

• ngram_size (int) – an integer indicating the number of characters in the ngram

• skip_size (int) – an integer indicating how many skip characters in the ngrams

• pad_token (bool) – a boolean flag to indicate whether padding should be included at the boundaries of the token

Returns:

An iterator returning tuples of skip_gram and offset

Return type:

Generator[tuple]

Algorithm:

Like text2skipgrams(), but operates on a single token instead of running text, and optionally pads the token with # characters on both sides (using ngram_size - 1 padding characters) so that skipgrams near the token boundaries are generated consistently with skipgrams in the middle of the token. The padded token is scanned with the same sliding-window/insert_skips approach, after which offsets and combination indexes are corrected back to the un-padded token’s coordinate space, and combination indexes that fall in the padding are dropped.

fuzzy_search.tokenization.token module

Core data structures for tokens, documents, annotations and tokenizers.

This module defines the basic building blocks used throughout the fuzzy_search library to represent tokenized text: Token and Doc for the tokenized representation of a document, Annotation and Tag for spans of text with metadata, and several Tokenizer implementations for turning raw text into Token objects.

class fuzzy_search.tokenization.token.Annotation(tag_type: str, text: str, offset: int, doc_id: str = None)

Bases: object

Represents an annotation in a document.

tag_type

The type of the tag.

Type:

str

text

The text of the annotation.

Type:

str

offset

The starting index of the annotation.

Type:

int

end

The ending index of the annotation (offset + length of the text).

Type:

int

doc_id

The ID of the document the annotation belongs to.

Type:

str, optional

__init__(tag_type: str, text: str, offset: int, doc_id: str = None)

Initializes the Annotation instance.

Parameters:

• tag_type (str) – The type of the tag.

• text (str) – The text of the annotation.

• offset (int) – The starting index of the annotation.

• doc_id (str, optional) – The ID of the document the annotation belongs to.

class fuzzy_search.tokenization.token.CustomTokenizer(tokenizer_func: Callable, **kwargs)

Bases: Tokenizer

A tokenizer that uses a custom tokenizer function provided by the user.

tokenizer_func

A user-defined function for tokenizing text.

Type:

Callable

__init__(tokenizer_func: Callable, **kwargs)

Initializes the CustomTokenizer instance.

Parameters:

• tokenizer_func (Callable) – The custom function to use for tokenizing the text.

• **kwargs – Additional arguments passed to the parent Tokenizer class.

tokenize(doc_text: str, doc_id: str = None) → List[Token]

Tokenizes the input document text and returns a list of documents.

Parameters:

• doc_text (str) – The text of the document to tokenize.

• doc_id (str, optional) – The ID of the document. Defaults to None.

Returns:

A Doc object containing the tokenized text.

Return type:

Doc

class fuzzy_search.tokenization.token.Doc(text: str, doc_id: str, tokens: List[Token], metadata: Dict[str, any] = None)

Bases: object

Represents a document containing a list of tokens.

text

The text of the document.

Type:

str

id

The ID of the document.

Type:

str

tokens

A list of Token objects representing the document’s tokens.

Type:

list

token_orig_set

A dictionary mapping original token strings to lists of tokens.

Type:

dict

token_norm_set

A dictionary mapping normalized token strings to lists of tokens.

Type:

dict

label_token_index

A dictionary mapping labels to sets of tokens.

Type:

defaultdict

metadata

Metadata associated with the document.

Type:

dict

annotations

A list of annotations associated with the document.

Type:

list

__init__(text: str, doc_id: str, tokens: List[Token], metadata: Dict[str, any] = None)

Initializes a Document instance.

Parameters:

• text (str) – The text content of the document.

• doc_id (str) – The ID of the document.

• tokens (List[Token]) – A list of Token objects representing the document’s tokens.

• metadata (dict, optional) – Metadata associated with the document.

add_annotations(annotations: List[Annotation], replace: bool = False)

Adds annotations to the document.

Parameters:

• annotations (List[Annotation]) – A list of Annotation objects to add.

• replace (bool) – whether to replace existing annotation or add (default is False).

get_token(token_string: str) → Token | List[Token] | None

Retrieves a token (or tokens) from the document based on the token string.

Parameters:

token_string (str) – The token string to look up.

Returns:

A Token object or a list of Token objects that match the token string.

Return type:

Union[Token, List[Token]]

has_token(token: str | Token) → bool

Checks if the document contains a specific token.

Parameters:

token (Union[str, Token]) – A token string or a Token object.

Returns:

True if the token exists in the document, False otherwise.

Return type:

bool

property normalized

Retrieves a list of normalized tokens in the document.

Returns:

A list of the normalized tokens.

Return type:

List[str]

property original

Retrieves a list of original tokens in the document.

Returns:

A list of the original tokens.

Return type:

List[str]

class fuzzy_search.tokenization.token.RegExTokenizer(ignorecase: bool = False, include_boundary_tokens: bool = False, split_pattern: str = '\\s+', token_pattern: str = None)

Bases: Tokenizer

A tokenizer that splits text into tokens using a regular expression pattern.

split_pattern

The regular expression pattern used to split the text into tokens.

Type:

str

__init__(ignorecase: bool = False, include_boundary_tokens: bool = False, split_pattern: str = '\\s+', token_pattern: str = None)

Initializes the RegExTokenizer instance.

Parameters:

• ignorecase (bool, optional) – Whether to ignore case when tokenizing. Defaults to False.

• include_boundary_tokens (bool, optional) – Whether to include boundary tokens. Defaults to False.

• split_pattern (str, optional) – The regular expression pattern used to split the text. Defaults to r’’.

class fuzzy_search.tokenization.token.Tag(tag_type: str, text: str, offset: int, doc_id: str = None)

Bases: object

Represents a tag in a document, containing both the tag’s type and text.

type

The type of the tag.

Type:

str

text

The text content of the tag.

Type:

str

offset

The starting index of the tag in the document.

Type:

int

tag_string

The full tag string, including opening and closing tags.

Type:

str

end

The ending index of the tag (offset + length of the tag string).

Type:

int

doc_id

The ID of the document the tag belongs to.

Type:

str, optional

__init__(tag_type: str, text: str, offset: int, doc_id: str = None)

Initializes the Tag instance.

Parameters:

• tag_type (str) – The type of the tag.

• text (str) – The text content of the tag.

• offset (int) – The starting index of the tag.

• doc_id (str, optional) – The ID of the document the tag belongs to.

class fuzzy_search.tokenization.token.Token(string: str, index: int, char_index: int, doc_id: str = None, normalised_string: str = None, label: str | Set[str] = None, char_end_index: int = None, metadata: Dict[str, any] = None)

Bases: object

Represents a token in a document.

string

The original string of the token.

Type:

str

index

The index of the token in the document.

Type:

int

char_index

The character index of the token in the document.

Type:

int

doc_id

The ID of the document the token belongs to.

Type:

str, optional

metadata

Additional metadata associated with the token.

Type:

dict, optional

normalised_string

The normalized version of the token.

Type:

str

label

A set of labels assigned to the token.

Type:

set

__init__(string: str, index: int, char_index: int, doc_id: str = None, normalised_string: str = None, label: str | Set[str] = None, char_end_index: int = None, metadata: Dict[str, any] = None)

Initializes the Token instance.

Parameters:

• string (str) – The original string of the token.

• index (int) – The index of the token in the document.

• char_index (int) – The character index of the token in the document.

• doc_id (str, optional) – The ID of the document the token belongs to.

• normalised_string (str, optional) – The normalized version of the token.

• label (str, set, list, optional) – Labels associated with the token.

• char_index – The character index of the token from the end of the document.

• metadata (dict, optional) – Additional metadata associated with the token.

has_label(label: str)

Checks if the token has a specific label.

Parameters:

label (str) – The label to check for.

Returns:

True if the token has the label, False otherwise.

Return type:

bool

property i

Gets the index of the token.

Returns:

The index of the token.

Return type:

int

property l

Gets the labels associated with the token.

Returns:

A set of labels associated with the token.

Return type:

set

lower()

Converts the normalized string of the token to lowercase.

property n

Gets the normalized string of the token.

Returns:

The normalized string of the token.

Return type:

str

property t

Gets the original string of the token.

Returns:

The original string of the token.

Return type:

str

update(normalised_string: str) → Token

Creates a new token with an updated normalized string.

Parameters:

normalised_string (str) – The updated normalized string.

Returns:

A new token with the updated normalized string.

Return type:

Token

class fuzzy_search.tokenization.token.Tokenizer(ignorecase: bool = False, include_boundary_tokens: bool = False, remove_punctuation: bool = False, split_pattern='(\\S+)')

Bases: object

A base class for tokenizing a document into tokens.

ignorecase

Flag indicating whether to ignore case when tokenizing.

Type:

bool

include_boundary_tokens

Flag indicating whether to include boundary tokens.

Type:

bool

remove_punctuation

Flag indicating whether to remove punctuation from tokens.

Type:

bool

__init__(ignorecase: bool = False, include_boundary_tokens: bool = False, remove_punctuation: bool = False, split_pattern='(\\S+)')

Initializes the Tokenizer instance.

Parameters:

• ignorecase (bool, optional) – Whether to ignore case when tokenizing. Defaults to False.

• include_boundary_tokens (bool, optional) – Whether to include boundary tokens. Defaults to False.

• remove_punctuation (bool, optional) – Whether to remove punctuation. Defaults to False.

property nltk_wp_tokenizer

The NLTK WordPunctTokenizer used by the base _string_tokenizer().

Constructed lazily (and NLTK imported lazily) on first access, since NLTK is a relatively expensive import and subclasses such as RegExTokenizer and CustomTokenizer override _string_tokenizer() and never need it.

tokenize(doc_text: str, doc_id: str = None) → List[Token]

Tokenizes the input document text and returns a list of documents.

Parameters:

• doc_text (str) – The text of the document to tokenize.

• doc_id (str, optional) – The ID of the document. Defaults to None.

Returns:

A Doc object containing the tokenized text.

Return type:

Doc

tokenize_doc(doc_text: str, doc_id: str = None)

Tokenizes the input document text and returns a Doc object.

Parameters:

• doc_text (str) – The text of the document to tokenize.

• doc_id (str, optional) – The ID of the document. Defaults to None.

Returns:

A Doc object containing the tokenized text.

Return type:

Doc

fuzzy_search.tokenization.token.tokens2string(tokens: List[Token]) → str

Reconstruct an approximate original string from a list of tokens.

Tokens are joined using their original character indices, padding with spaces to align each token’s text at its char_index position.

Parameters:

tokens (List[Token]) – The tokens to join, assumed to be in order.

Returns:

The reconstructed string.

Return type:

str

fuzzy_search.tokenization.token.update_token(token: Token, new_normalised: str) → Token

Creates a new Token instance by updating the normalization string of an existing token.

Parameters:

• token (Token) – The original Token object to be updated.

• new_normalised (str) – The new normalized string for the token.

Returns:

A new Token object with the updated normalization string.

Return type:

Token

fuzzy_search.tokenization.vocabulary module

A simple bidirectional term-to-identifier mapping used for indexing terms and skipgrams.

class fuzzy_search.tokenization.vocabulary.Vocabulary(terms: List[str | Token] = None, ignorecase: bool = False)

Bases: object

Maps terms (strings or Tokens) to integer identifiers and back.

Terms are assigned identifiers in the order they are first added, starting from 0.

term_id

Maps each known term to its identifier.

Type:

Dict[str, int]

id_term

Maps each identifier back to its term.

Type:

Dict[int, str]

term_freq

Reserved for term frequency tracking (currently unused).

Type:

Dict[str, int]

ignorecase

Whether terms are lowercased before being indexed/looked up.

Type:

bool

__init__(terms: List[str | Token] = None, ignorecase: bool = False)

Initializes the Vocabulary, optionally indexing an initial list of terms.

Parameters:

• terms (List[Union[str, Token]], optional) – Terms to add to the vocabulary on creation.

• ignorecase (bool, optional) – Whether to lowercase terms before indexing and lookup. Defaults to False.

add_terms(terms: str | Token | List[str | Token], reset_index: bool = False)

Add a list of terms to the vocabulary. Use ‘reset_index=True’ to reset the vocabulary before adding the terms.

Parameters:

• terms (List[str]) – a list of terms to add to the vocabulary

• reset_index (bool) – a flag to indicate whether to empty the vocabulary before adding terms

has_term(term: str | Token, ignorecase: bool = None)

Checks whether a term is present in the vocabulary.

Parameters:

• term (Union[str, Token]) – The term (or Token, whose normalised string is used) to look up.

• ignorecase (bool, optional) – Whether to lowercase the term before lookup. Defaults to the vocabulary’s ignorecase setting.

Returns:

True if the term is in the vocabulary, False otherwise.

Return type:

bool

id2term(term_id: int)

Return the term for a given term ID.

Parameters:

term_id (int) – The identifier to look up.

Returns:

The corresponding term, or None if the identifier is unknown.

Return type:

Optional[str]

reset_index()

Clears all terms, identifiers and frequencies from the vocabulary.

term2id(term: str)

Return the term ID for a given term.

Parameters:

term (str) – The term to look up.

Returns:

The term’s identifier, or None if the term is not indexed.

Return type:

Optional[int]

Module contents