Benjamin Lee 20a54aacd6 factor search tokenization out into a function ... This ensures that the tokenization algorithm will remain in sync between querying, indexing, and deindexing. The existing code had slightly different behavior for querying, because it did not discard words with >50 bytes. This was inconsequential, because >50 byte tokens are never present in the index. Signed-off-by: strawberry <strawberry@puppygock.gay>		2024-06-12 14:04:47 -04:00
..
data.rs	factor search tokenization out into a function	2024-06-12 14:04:47 -04:00
mod.rs	dissolve key_value/*	2024-05-27 18:16:23 -04:00