Refactor README and Vicinity class to support any serializable item type by davidberenstein1957 · Pull Request #56 · MinishLab/vicinity

davidberenstein1957 · 2025-01-20T16:22:58Z

I was playing around and realised the items don't necessarily need to be strings.

Update

Updated README.md to clarify that items can be strings or other serialisable objects.
Modified the Vicinity class to accept a broader range of item types by changing type hints from str to Any in several methods.
Enhanced the insert and delete methods to handle non-string tokens appropriately, ensuring that items can be checked and managed regardless of their type.

Question
Also, why do we use tokens as argument names for items in the insert and delete methods?

Example
Something like the following works now. Slower during insert and deletion but I would say the flexibility is worth it.

import numpy as np

from vicinity import Backend, Metric, Vicinity

# Create some dummy data
items = [
    {"name": "triforce", "id": 0},
    {"name": "master sword", "id": 1},
    {"name": "hylian shield", "id": 2},
    {"name": "boomerang", "id": 3},
    {"name": "hookshot", "id": 4},
]


vectors = np.random.rand(len(items), 128)

# Initialize the Vicinity instance (using the basic backend and cosine metric)
vicinity = Vicinity.from_vectors_and_items(
    vectors=vectors, items=items, backend_type=Backend.BASIC, metric=Metric.COSINE
)

# Create a query vector
query_vector = np.random.rand(128)

# Query for nearest neighbors with a top-k search
results = vicinity.query(query_vector, k=3)

# Query for nearest neighbors with a threshold search
results = vicinity.query_threshold(query_vector, threshold=0.9)

vicinity.delete([items[0], items[1], items[2], items[3], items[4]])
vicinity.insert(items, vectors)
# Query with a list of query vectors
query_vectors = np.random.rand(5, 128)
results = vicinity.query(query_vectors, k=3)
print(results)

vicinity.save("my_vector_store", overwrite=True)
vicinity = Vicinity.load("my_vector_store")

results = vicinity.query(query_vectors, k=3)
print(results)

- Updated README.md to clarify that items can be strings or other serializable objects. - Modified the Vicinity class to accept a broader range of item types by changing type hints from `str` to `Any` in several methods. - Enhanced the insert and delete methods to handle non-string tokens appropriately, ensuring that items can be checked and managed regardless of their type.

… and evaluating backends

codecov-commenter · 2025-01-20T16:38:20Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines	Coverage Δ
tests/conftest.py	`100.00% <100.00%> (ø)`
tests/test_vicinity.py	`100.00% <100.00%> (ø)`
vicinity/vicinity.py	`98.62% <100.00%> (+0.06%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- Simplified the logic for checking and appending tokens in the insert method, ensuring that duplicate tokens are properly managed.

- Updated the `items` fixture to return a mix of dictionaries and strings based on index parity. - Modified `test_vicinity_insert_duplicate` to use the updated `items` fixture for inserting items. - Adjusted `test_vicinity_delete_and_query` to reference items by their indices instead of hardcoded values. - Enhanced the Vicinity class to streamline token management, ensuring proper handling of duplicates and improving error messaging for token deletions.

stephantul

This looks good, thanks! One improvement I see is that we can explicitly throw a ValueError on trying to save if the items are not JSON serializable. e.g., by catching a JsonEncodeError and giving a more informative warning.

Another improvement is the typing, but I'll do that in a follow-up because I actually don't know what the best way is 💀

Co-authored-by: Stephan Tulkens <stephantul@gmail.com>

…ling - Replaced the nested loop for checking duplicates with a single extend operation for tokens. - Improved efficiency by directly appending tokens to the items list, ensuring proper management of duplicates.

…ling - Replaced the nested loop for token matching with a more efficient list comprehension. - Enhanced error messaging to specify which tokens were not found in the vector space.

- Added a try-except block around the JSON serialization process to catch JSONEncodeError.

- Introduced a new pytest fixture `non_serializable_items` that generates a list of non-serializable objects for testing. - Added a test case `test_vicinity_save_and_load_non_serializable_items` to verify that saving a Vicinity instance with non-serializable items raises a JSONEncodeError. - Updated the Vicinity class documentation to specify that JSONEncodeError may be raised if items are not serializable.

davidberenstein1957 added 2 commits January 20, 2025 17:22

Update README.md to include examples for saving/loading vector stores…

9ffb491

… and evaluating backends

davidberenstein1957 added 2 commits January 20, 2025 17:40

Refactor Vicinity class to streamline token handling

7b2bb53

- Simplified the logic for checking and appending tokens in the insert method, ensuring that duplicate tokens are properly managed.

stephantul approved these changes Jan 20, 2025

View reviewed changes

Comment thread tests/test_vicinity.py Outdated

Comment thread tests/test_vicinity.py Outdated

Comment thread tests/test_vicinity.py Outdated

Comment thread vicinity/vicinity.py Outdated

Comment thread vicinity/vicinity.py

davidberenstein1957 and others added 3 commits January 20, 2025 19:41

Apply suggestions from code review

022c7b1

Co-authored-by: Stephan Tulkens <stephantul@gmail.com>

Refactor token insertion in Vicinity class to simplify duplicate hand…

eaabbfa

…ling - Replaced the nested loop for checking duplicates with a single extend operation for tokens. - Improved efficiency by directly appending tokens to the items list, ensuring proper management of duplicates.

Refactor token deletion logic in Vicinity class to improve error hand…

031c136

…ling - Replaced the nested loop for token matching with a more efficient list comprehension. - Enhanced error messaging to specify which tokens were not found in the vector space.

davidberenstein1957 requested a review from stephantul January 20, 2025 18:54

davidberenstein1957 added 2 commits January 20, 2025 19:57

Enhance error handling in Vicinity class for JSON serialization

26e7ed6

- Added a try-except block around the JSON serialization process to catch JSONEncodeError.

stephantul approved these changes Jan 24, 2025

View reviewed changes

stephantul merged commit 2a8272c into MinishLab:main Jan 24, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Refactor README and Vicinity class to support any serializable item type#56

Refactor README and Vicinity class to support any serializable item type#56
stephantul merged 9 commits into
MinishLab:mainfrom
davidberenstein1957:main

davidberenstein1957 commented Jan 20, 2025 •

edited

Loading

Uh oh!

codecov-commenter commented Jan 20, 2025 •

edited

Loading

Uh oh!

stephantul left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

davidberenstein1957 commented Jan 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Jan 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

stephantul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

davidberenstein1957 commented Jan 20, 2025 •

edited

Loading

codecov-commenter commented Jan 20, 2025 •

edited

Loading