Add zvec as a dense ANN backend

[morgan-coded]

Dense embeddings can now use zvec as an embedded, path-based ANN index. This keeps the first pass dense-only, with sparse indexing and the current default backend unchanged. The shared harness covers save/load, append, missing-ID deletes, and search with positional IDs; I also ran the full dense and sparse ANN suites (31 tests with 3 existing macOS skips), the optional-dependency test, and the changed-file lint hooks. In a seeded local 10,000 × 240 run, zvec matched the exact NumPy top result for all 100 queries. Closes #1040

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[davidmezzetti]

Thanks for the PR. At first glance this looks good with just a couple thoughts.

[morgan-coded]

Thanks, that makes sense. Dropped the extra metadata and config restore since dimensions and offset already persist through the embeddings config. zvec 0.5.1 writes a directory-backed collection and only reopens it by path, so I kept the uncompressed TAR to fit ANN.save(path)'s single-file contract. Happy to use a different convention if you have one.

[davidmezzetti]

Thanks for clarifying it requires multiple files.

TAR files have had security issues I've had to deal with in the past. Please model this after this class: https://github.com/neuml/txtai/blob/master/src/python/txtai/graph/networkx.py using ArchiveFactory.

[morgan-coded]

Thanks, that makes sense. Switched the zvec save/load to ArchiveFactory with the TAR handling modeled on networkx.py, so packing, path/symlink validation, and the data filter now come from txtai's archive code. Could you take another look?

[davidmezzetti]

Thanks, can you sync your fork with the latest? I had to debug a number of recent build errors unrelated to this PR.

[morgan-coded]

Thanks, can you sync your fork with the latest? I had to debug a number of recent build errors unrelated to this PR.

Synced the branch with the latest master and reran the ANN tests, optional-dependency test, and lint hooks. Everything passed locally. It should be good now

[davidmezzetti]

Please see the build error. It looks like we need to skip copying at least the LOCK file if not more. I can't find a ton on the file format but if there are other files to skip that would be good too.

[morgan-coded]

Excluded both zvec runtime LOCK files from the saved tar and recreated the required empty root lock before opening a loaded collection. Kept the rest of the zvec files in the archive; local tests are passing.

[davidmezzetti]

I'll run the build here but this is starting to get messy.

[davidmezzetti]

Can you call collection.close -> save -> then reopen instead of all this additional code to work around lockfiles? The last change just seems messy.

[davidmezzetti]

Are you sure you don't need to call self.backend.close vs just setting null? I don't know what platform you're developing on but Windows is just very finicky with locks.

[morgan-coded]

zvec's Collection (0.5.1) exposes no close() — the only teardown is destroy, which drops the data. Setting self.backend to None releases the collection and its file locks, the same release the existing close() performs before it removes the directory. Windows is genuinely finicky here, so the CI run on this commit is the real test; the save/reopen round-trips locally, and if Windows still trips on the lock I'll follow up.

[morgan-coded]

Developed this on macOS, so I ran testZvec and testZvecCustom on a Windows 11 machine (Python 3.12, zvec 0.5.1) to check the lock directly: both pass, save/load round-trips with no LOCK error — dropping the backend reference does release the lock on Windows. CI should confirm the same.

[morgan-coded]

The Windows job gets through the save with no LOCK error, so that part's confirmed. The red build status is unrelated to this change — pylint's name-length check flags a method in testworkflow.py, plus a testscoring.testsparse multiprocessing flake.

[davidmezzetti]

Thank you for the hard work on this. I'll merge it in and debug anything further, if anything.

[morgan-coded]

Thanks for merging this. If anything surfaces when you dig in further, let me know and I'll pick it back up.