docs: make server-side batching the default import method#466
Merged
Conversation
The "Server-side batching" section of manage-objects/import only had a Python example (plus a placeholder C# reference). Fill in the clients that support server-side batching (added in Weaviate v1.36): - TypeScript: enable the data.ingest() example and wire the tab - Java v6: implement the test using collection.batch.start() -> BatchContext - C#: implement the test using collection.Batch.InsertMany() Go has no server-side batching client API, so its tab stays "coming soon". Verified: all three snippets compile and run against Weaviate 1.38 with the CI-pinned client versions (TS v3.13.1, Java 6.2.0, C# 1.1.1).
The TypeScript server-side batching example passed bare objects ({ title })
to data.ingest(). Unlike data.insertMany(), data.ingest() does not treat a
bare object as properties on an untyped collection, so under auto-schema it
silently imported 5 objects with EMPTY properties (title dropped). Use the
DataObject wrapper ({ properties: { title } }) so the property persists, and
add a post-example fetchObjects() assertion that fails if the objects or
their titles are missing.
The java-v6 examples module depended on client6:6.2.1-SNAPSHOT (which no release publishes, and the pom has no snapshot repository), while docs CI built and installed client tag 6.2.0. That version lacks text2vecDigitalOcean and the tokenize APIs, so ModelProvidersTest and TokenizationTest failed to compile — breaking the whole module, including the new server-side batching test. Pin both the dependency and the CI-built client to the released 6.2.1.
![orca-security-eu[bot]](https://avatars.githubusercontent.com/in/276244?s=60&v=4){kind=small}
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
 Passed |
Infrastructure as Code |  0  0  0  0 |
View in Orca |
| Passed |
SAST | 0 0 0 0 |
View in Orca |
| Passed |
Secrets | 0 0 0 0 |
View in Orca |
| Passed |
Vulnerabilities | 0 0 0 0 |
View in Orca |
Restructure the batch-import guide so server-side (automatic) batching is the first and recommended way to import data, instead of a separate feature section: - Lead with server-side batching; present client-side batching as "Manual batching", the alternative for manual control or for the Go client. - The Go tab points to manual batching (the Go client has no server-side batching API) instead of "coming soon". - Note that server-side batching uses the gRPC API (enabled by default). - concepts/data-import.mdx: correct the now-stale "only the Python client supports server-side batch imports" (now Python, TypeScript, Java, and C#; Go does not yet), and fix a broken sentence.
6.3.0 is the current release; it keeps the java-v6 examples module compiling (text2vec-digitalocean + tokenize APIs) and passes the server-side batching test. Supersedes the 6.2.1 pin from the previous commit.
g-despot
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes server-side batching (SSB) the default, recommended way to import data on the batch-import how-to page, and fills in the SSB code snippets for every client that supports it. SSB shipped in Weaviate v1.36; the concepts page already called it "the recommended approach," but the how-to page still led with client-side batching and only Python had an SSB snippet.
Import page restructure (
manage-objects/import.mdx)SSB snippets added (were "coming soon")
collection.batch.stream()(already documented)collection.data.ingest()collection.batch.start()→BatchContextcollection.Batch.InsertMany()Fixes found while making the snippets actually run
data.ingest()silently dropped properties when passed bare objects on an untyped collection (imported empty objects). Switched to the{ properties: { title } }DataObject wrapper + afetchObjects()assertion. (The underlying client inconsistency is tracked separately for the TS client.)java-v6examples module depended on an unpublished6.2.1-SNAPSHOTwhile CI built6.2.0(which lackstext2vec-digitalocean+ tokenize APIs), so the whole module failed to compile. Pinned the pom + CI to the released6.3.0.Verification
All three new snippets were compiled and run end-to-end against a live Weaviate 1.38 (auto-schema on, matching CI) with the CI-pinned clients (TS
3.13.1, Javaclient6 6.3.0, C#1.1.1):data.ingest→ 5 objects, titles persisted (assertion passes; fails on the old bare shape).6.3.0→ wholejava-v6module compiles (all 44 test classes),testServerSideBatchImportpasses.Batch.InsertMany→ 5 objects, title persisted.yarn build-devsucceeds;validate-links-devshows 0 new broken links (new#manual-batching/#use-the-grpc-apianchors resolve; the preserved lower anchors still serve their inbound links).Notes
batch.dynamic()/fixed_size()without mentioningstream(); the tutorial and Python best-practices pages already lead SSB-first.