git:20260811.948d74b to git:20260903.36c28af

+ description: Create, ingest into, and query a Pinecone full-text-search (FTS) document index using the graduated document-schema API (Python SDK 10.0.0, API version 2026-07). Use when the user or agent asks to build a text search index on Pinecone, add dense or sparse vector fields, ingest documents, construct score_by clauses (text / query_string / dense_vector / sparse_vector), or compose with text-match filters ($match_phrase / $match_all / $match_any). Ships `scripts/ingest.py` for safe bulk ingestion (batch_upsert + error inspection + readiness polling); query construction is documented inline in this skill — write `documents.search(...)` calls directly, validated against `pc.indexes.describe(...)` output.

+ > **Requires `pinecone` Python SDK ≥ 10.0.0** (`pip install pinecone>=10.0.0`). The document-schema API graduated out of `pinecone.preview` in 10.0.0 — it is now a first-class, SemVer-covered part of the SDK, reachable directly off `pc` (`pc.indexes`, `pc.index(...)`). If you land on this skill from an older habit of importing `pinecone.preview`, stop: that package is deleted outright in 10.0.0 (`ModuleNotFoundError`, no shim). The packaged helper script pins `pinecone==10.0.0` via PEP 723 inline metadata; if you're writing your own code against this skill, pin at least that version. The wire API version is `2026-07`.

+ > **Authoritative reference (last resort).** If you hit a question this skill and its `references/*.md` files don't answer, the official Pinecone FTS docs are at <https://docs.pinecone.io/guides/search/full-text-search>. Prefer this skill's content for anything covered here — the docs may describe surfaces (e.g. classic vector API, or the older `pinecone.preview` shape) that don't apply to the graduated document-schema path. Consult the link only when you're genuinely stuck.

+ A workflow skill for building a Pinecone full-text-search index with the graduated document-schema API (`pc.indexes`, `pc.index(name)`, API version `2026-07`). Covers schema design (text, dense vector, sparse vector, filterable metadata), ingestion (including async indexing and polling), and query construction (`text` / `query_string` / `dense_vector` / `sparse_vector` scoring; `$match_phrase` / `$match_all` / `$match_any` text-match filters; `$eq` / `$in` / `$gte` / `$exists` / `$and` / `$or` / `$not` metadata filters).

+ This skill covers `pc.indexes.create(..., schema=...)`, `pc.index(name)`, `idx.documents.upsert(...)` / `idx.documents.batch_upsert(...)` / `idx.documents.search(...)`. If you find yourself reaching for any of the following, **stop** — those are different Pinecone APIs and this skill's guidance and helpers won't apply:

+ - **Classic vector / records API**: `pc.Index(name)`, `index.upsert(vectors=[...])`, `index.query(vector=..., sparse_vector=...)`, `pc.create_index(dimension=..., metric=..., spec=ServerlessSpec(...))`. This is the *deprecated sugar* path in 10.0.0 — it still runs, but it creates a schemaless index served by the vector data plane, addressing the vector by the reserved `_values` field. It cannot hold `full_text_search` fields.

+ - **Integrated-embedding / records indexes**: `pc.create_index_for_model(...)` / `pc.indexes.create_for_model(...)` with `embed={...}`. Pinecone vectorizes text server-side, and the resulting `semantic_text` field is served by the **records** API (`upsert_records` / `search_records`), not the documents API. Different upsert/search shapes. A `semantic_text` field cannot be combined with `full_text_search` fields in the same index.

+ If the user already has a non-document-schema index, they can stand up a separate document-schema index alongside it — the two are independent — but you can't add FTS fields to a classic or integrated-embedding index after the fact, and a document-schema index only ever serves reads and writes through `index.documents.*` — never `index.upsert` / `index.query` / `index.upsert_records` (those calls are refused with "This index has a document schema, so writes must go through the documents API").

+ 1. **Discover the schema.** Call `pc.indexes.describe(<index>)` and read the `schema.fields` dict. Each field's class indicates its type (`StringField`, `FloatField`, `DenseVectorField`, etc.); attributes tell you whether it's FTS-enabled (`full_text_search`), filterable, or carries a `dimension`. Skip this step only if you've already seen the schema in this conversation.

+ 3. **Execute** with `idx = pc.index(name=<index>); resp = idx.documents.search(...)` and read `resp.matches`.

+ - `filter` keys are field names (must exist in schema, or be an auto-indexed metadata field from upserted documents — see **Filterable metadata isn't declared in the schema** below) OR logical operators (`$and`, `$or`, `$not`). Field values are operator dicts (`{"$gt": 5}`, NOT bare values).

+ - `include_fields` is required on every call. Pass `["*"]` for all stored fields, `[]` for ids+score only, or a list of names. Omitting it on some SDK/backend builds 400s.

+ | filterable metadata (string / auto-indexed) | `$eq`, `$ne`, `$in`, `$nin`, `$exists` |

+ | `string_list` filterable (auto-indexed, not schema-declared) | `$in`, `$nin`, `$exists` |

+ | `float` filterable (auto-indexed, not schema-declared) | `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$exists` |

+ | `boolean` filterable (auto-indexed, not schema-declared) | `$eq`, `$exists` |

+ m._score # match score (NOT `score`)

+ | `--batch-size` | `-b` | no | Default 50 (matches the SDK's own `batch_upsert` default). **Reduce for large dense vectors.** A 50-doc batch with 3072-dim float vectors lands ~5-10 MB and can be rejected; drop to `--batch-size 25` (or lower) at high dimensions. |

+ | `--max-concurrency` | — | no | Default 4. Parallel HTTP connections used to upload batches. |

+ Upserting in batches of 50 ...

+ batch @ 0: 50 docs in 0.31s (total: 50/5000)

+ batch @ 50: 50 docs in 0.29s (total: 100/5000)

+ - The user is doing per-doc patch updates. Use `documents.update(...)` for partial field updates (see **Updating documents** in `references/ingestion.md`) — the script is for bulk loads, not per-record operations.

+ 2. Pick analyzer settings on each text field — `language`, `stemming`, `stop_words`. Stemming on for long prose, off for proper nouns / identifiers. **Decide which fields are FTS and which are filterable-only metadata** — see **Filterable metadata isn't declared in the schema** below; filterable-only metadata is a documents-side decision, not a schema field.

+ 3. Assemble the schema with `SchemaBuilder` and **confirm it with the user before calling `indexes.create`** — schemas are immutable in `2026-07`, so a wrong call costs a re-ingest.

+ 4. Create the index. `pc.indexes.create(...)` polls until the index is ready by default — no separate wait loop needed unless you passed `timeout=-1`.

+ 6. (The script polls automatically — by the time it exits cleanly, the index is searchable. If you skip the script and roll your own, you must poll `documents.search` with a sentinel query and a deadline; `batch_upsert` returning ≠ searchable — this is a *document*-indexing wait, separate from and in addition to the index-creation wait in step 4.)

+ 2. Because schemas are immutable, **plan a new index, not a migration**. Get user confirmation before recreating. A hybrid index must declare its `sparse_vector` field explicitly at creation — there is no way to add one later.

+ 4. Schema → create (blocks until ready by default) → ingest with embeddings inline or pre-cached.

+ 1. **(Optional) Discover the schema** by calling `pc.indexes.describe(<NAME>)` and reading `schema.fields`. Skip if you already know the field types from earlier in the conversation.

+ - Omitting `include_fields` (some SDK/backend builds 400). Always pass it explicitly.

+ | Category / tag / list membership ("category = fiction", "tagged X") | `filter={"<field>": {"$in": [...]}}` (works on plain filterable metadata and `string_list` filterable fields) |

+ ## Filterable metadata isn't declared in the schema at all

+

+ This is the single biggest shape change from the old `pinecone.preview` API, and it's easy to get only half right.

+

+ On a **managed** index (the deployment type every example in this skill uses — `{"deployment_type": "managed", "cloud": ..., "region": ...}`, which is also the default when `deployment=` is omitted), the schema may **only** declare fields that participate in **search**: `dense_vector`, `sparse_vector`, and `string` fields with `full_text_search` enabled. **Every other field type — `string` (filterable, no FTS), `string_list`, `float`, `boolean` — is rejected at create time with a 400 if it appears in the schema.** This is confirmed live, not just documented: the server's own error names all four types explicitly — *"The schema only accepts fields used for search (field types `dense_vector`, `sparse_vector`, and `string` with `full_text_search` configuration). To use field '&lt;name&gt;' for filtering (field types `boolean`, `float`, `string`, or `string_list`), omit it from the schema and include it in documents."* (That restriction is specific to managed/BYOC deployments — schema-declared filterable metadata is only legal on **pod** deployments, which this skill doesn't cover.) The `SchemaBuilder` methods `add_float_field`, `add_boolean_field`, and `add_string_list_field` still exist and still work correctly for a pod deployment; for the managed deployments this skill always uses, don't call any of them.

+

+ Instead: **don't declare any filterable-only field in the schema, of any type.** Just include the field in the documents you upsert — Pinecone indexes whatever's present on an upserted document for filtering automatically (exact-match on strings and numbers/booleans, membership on lists), whether or not it appears in the schema, with no configuration needed.

+

+ ```python

+ # WRONG on a managed index — the server 400s on EVERY one of these, not just category:

+ schema = (

+ SchemaBuilder()

+ .add_string_field("body", full_text_search={"language": "en"})

+ .add_string_field("category", filterable=True) # <-- rejected

+ .add_float_field("year", filterable=True) # <-- also rejected

+ .add_string_list_field("tags", filterable=True) # <-- also rejected

+ .build()

+ )

+

+ # RIGHT — the schema declares only search fields. category/year/tags are

+ # simply included on upserted documents and auto-index for filtering.

+ schema = (

+ SchemaBuilder()

+ .add_string_field("body", full_text_search={"language": "en"})

+ .build()

+ )

+

+ idx.documents.upsert(namespace=NAMESPACE, documents=[{

+ "_id": "doc-1",

+ "body": "...",

+ "year": 2025.0, # not in the schema — still filterable via $gt / $gte / $eq

+ "tags": ["classic"], # not in the schema — still filterable via $in / $nin

+ "category": "fiction", # not in the schema — still filterable via $eq / $in / $exists

+ }])

+ ```

+

+ The **forward-looking note** in `references/schema-design.md` from the old preview docs — "declare metadata fields today to be future-proof" — no longer applies; declaring any filterable-only field is now actively wrong, not just unnecessary.

+

+ 1. **Design the schema.** Decide which string fields are full-text-searchable (declared in the schema), whether you need a `dense_vector` field (and whether it earns its place), and whether you also need a `sparse_vector` field — those are the *only* things that ever go in the schema. Every filterable metadata field — string, string_list, float, boolean alike — is NOT declared; it's just included on upserted documents. Schemas are **fixed at index creation** in `2026-07` — plan carefully. → `references/schema-design.md`

+ 2. **Ingest documents.** For bulk loads from a prepared JSONL, run the bundled `scripts/ingest.py` helper (it does `batch_upsert` + error inspection + readiness polling correctly by construction — see the **Ingesting — use the packaged helper** section above). For per-doc patch updates, use `documents.update(...)`. Either way, documents are indexed asynchronously after the HTTP call returns; `batch_upsert` returning 202 ≠ searchable. → `references/ingestion.md` for the canonical pattern in detail.

+ from pinecone import Pinecone, SchemaBuilder

+ INDEX_NAME = "my-fts-index" # TODO: name your index (lowercase alphanumeric + hyphens, 1-45 chars)

+ # 1. Schema — one FTS string field. That's the only kind of field that goes

+ # here: `category` and `year` below are deliberately NOT in the schema —

+ # see "Filterable metadata isn't declared in the schema at all" in

+ # SKILL.md. Field names must NOT start with `_` (reserved for `_id` /

+ # `_score`) or `$` (reserved for filter operators), and are limited to 64

+ # bytes.

+ # 2. Create the index. Polls until ready by default (pass timeout=-1 to

+ # return immediately instead). Deployment defaults to managed/aws/us-east-1

+ # when omitted; pass `deployment=` explicitly to pick a different region.

+ # read_capacity defaults to {"mode": "OnDemand"}; pass

+ if not pc.indexes.exists(INDEX_NAME):

+ pc.indexes.create(

+ name=INDEX_NAME,

+ schema=schema,

+ deployment={"deployment_type": "managed", "cloud": "aws", "region": "us-east-1"},

+ )

+ idx = pc.index(name=INDEX_NAME)

+ # 3. Upsert a single document. `_id` is required, every other field is optional.

+ # `category` and `year` aren't in the schema but are still filterable —

+ # see above.

+ # upsert REPLACES the document on conflict; use documents.update(...) for

+ # per-field patches (references/ingestion.md).

+ # 4. Poll until the FTS side is searchable (upsert returns BEFORE docs are indexed).

+ # 5. Search — text scoring composed with metadata filter.

+ print(m._id, m._score, m.to_dict())

+ - **No filterable metadata field goes in the schema on managed indexes — string, string_list, float, and boolean alike.** Only `dense_vector`, `sparse_vector`, and FTS-enabled `string` fields are legal in `schema=`; every other field type is rejected with `400`. Confirmed live against the real API, not just documented. Omit it and let it auto-index from upserted documents instead. See **Filterable metadata isn't declared in the schema at all** above — this is the change most likely to break code carried over from the old `pinecone.preview` API, where such fields were declarable.

+ - **`include_fields` is required on every `documents.search(...)` call.** Pass `["*"]` for all stored fields or a list of names to project. Omitting it on some SDK/backend builds yields `400` instead of a sane default; always pass it explicitly to avoid surprises.

+ - **Match score is `_score`; doc id is `_id`.** The system match score is always on the `_score` field so a user metadata field literally named `score` can coexist. Always read `m._score`, never `m.score`.

+ - **Vector-field cardinality: at most one `dense_vector` and at most one `sparse_vector` per index** in `2026-07`. Multiple text fields are fine.

+ - **A hybrid index must declare its `sparse_vector` field at create time — there's no adding one later.** `metric="dotproduct"` on the dense field is NOT a hybrid declaration by itself in `2026-07` (it was in the old preview API). The create call succeeds either way; if the `sparse_vector` field is missing, only the *sparse writes* are refused, later, often from a different part of the codebase. If you're porting an old preview schema, audit every `metric="dotproduct"` dense field for a missing sparse field before recreating it.

+ - **Dense-vector payload size matters at batch time.** A 50-doc batch with 3072-dim float vectors lands around 5–10 MB and can be rejected. If every batch fails, try reducing the embedding dimension via your provider's truncation knob (e.g. Gemini's `output_dimensionality=768`) before debugging schema.

+ - **Async indexing: `batch_upsert` returning ≠ searchable.** The server builds inverted indexes in the background after the HTTP call returns. If you query immediately you'll see empty result sets. Always poll `documents.search` with a sentinel query and a deadline (pattern in `references/ingestion.md`). This is separate from — and in addition to — `pc.indexes.create()`'s own default polling for the *index* becoming ready.

+ - **String FTS field shape is `full_text_search={...}` (dict), or `True` for server defaults.** **User-settable sub-fields:** `language`, `stemming`, `stop_words`, `ngram`. **Server-applied** (visible in `describe()` responses but NOT settable at index creation): `lowercase` (default `true`) and `max_term_len`. Stemming is opt-in (default `false`) and required if `stop_words=True` is set. A string field is *either* FTS-enabled *or* filterable, never both on a managed index — passing `filterable=True` alongside `full_text_search` makes the server silently keep the filter and drop the search config.

+ - **Schemas are fixed at index creation in `2026-07`.** Adding, removing, or retyping fields after creation is not supported. Changing dimension or metric on an existing vector field requires a new index. Plan the schema once.

+ - **Per-field updates are supported: `documents.update(...)`.** Pass `documents=[{"_id": ..., "set_fields": {...}}]`-style records, or `filter=` + `set_fields=`/`remove_fields=` to patch many documents at once by metadata match. `documents.upsert` still fully replaces a document on conflicting `_id` if that's what you want instead. See `references/ingestion.md` → "Updating documents".

+ - **Document operations: search, fetch, and delete all support `filter` now.** `documents.fetch` and `documents.delete` both gained a `filter` parameter — pass exactly one of `ids`, `filter`, or (for delete) `delete_all`. A filtered `fetch` is paginated (up to 10,000 docs per page via `pagination_token`); an ID-based fetch is never paginated. `documents.delete` returns a response object with `matched_records` (a point-in-time count for filtered deletes; `None` for ID-list or `delete_all` deletes — the delete itself is applied asynchronously).

+ - **`documents.list(...)` enumerates document IDs in a namespace, with no equivalent in the old preview API.** Lazily-paginated, sorted by ID, optionally filtered by `prefix`. See `references/querying.md` → "`documents.list` — enumerate document IDs".

+ - **Namespaces auto-create on first upsert.** Pass any namespace string to `documents.upsert` / `batch_upsert` and the namespace is created on the fly; documents from different namespaces are fully isolated. Use `"__default__"` if you don't need partitioning.

+ - **Namespace management and `describe_index_stats` now work on document-schema indexes.** `idx.create_namespace(name=...)`, `idx.list_namespaces()`, `idx.describe_namespace(name=...)`, `idx.delete_namespace(name=...)`, and `idx.describe_index_stats()` are all available and confirmed working — see `references/ingestion.md` → "Namespace management" for signatures and examples.

+ - **Document and request size limits**: per-document max **2 MB**; per-request max **2 MB and 1000 documents**; per FTS-enabled `string` field max **100 KB and 10,000 tokens** (tokens > 256 bytes are truncated by the analyzer); per-document filterable metadata (everything *not* in an FTS field) max **40 KB**. A schema can declare up to **100 FTS string fields**. For long-prose corpora, chunk before ingest — see `references/ingestion.md`.

+ - **Indexes can't be created in CMEK-enabled projects alongside any `full_text_search` field, no backup/restore, no fuzzy or regex search, no S3 bulk import** for document-shaped indexes in `2026-07`. If any of these are hard requirements, the document-schema FTS surface isn't yet ready.