---
name: embed-catalog
description: Convert a user's catalog (products, artworks, articles, listings; CSV, TSV, JSON, Excel, Shopify/WooCommerce export, Google Merchant feed, database dump) into the BehaviorGPT catalog parquet and embed it with the UnboxAI API. Use when the user wants to bring, upload, embed, or use their own catalog, products or items with BehaviorGPT, or asks to build a catalog parquet.
---

# Embed your own catalog

Goal: take whatever item data the user has, produce one parquet that passes `check_catalog_schema`, upload it with `client.embed`, and hand back the `catalog_id`.

`docs/catalog-format.md` is the source of truth for columns, types and failure modes. Read it before starting. `examples/sample_catalog.csv` shows realistic values for every column. `src/behaviorgpt/resources/catalogs.py` holds `CATALOG_SCHEMA` and `check_catalog_schema`.

## Environment

- Run Python through the project env: `uv sync` once, then `uv run python <script>`. pandas is in the dev group, which `uv sync` installs.
- The API key comes from `UNBOXAI_API_KEY`, set in `.env` (copied from `.env.example`). Never print, log, or echo the key. If it is missing, ask the user to add it; do not ask them to paste it in chat.
- Put conversion scripts and the output parquet in `.untracked/` (gitignored). Never commit the user's catalog.

## Workflow

### 1. Inspect the source

Load a sample (first ~20 rows) and report: row count, column names, a few example values per column, and which columns look like lists, prices, dates or image links. Do not guess types from the file extension alone.

### 2. Propose a column mapping and confirm it

Map source columns to the 15 target columns. Show the mapping as a table and wait for the user to confirm before converting. Flag every target column that will be all-null.

Common sources:

| Target | Shopify export | WooCommerce export | Google Merchant feed |
|---|---|---|---|
| `id` | `Handle` | `ID` or `SKU` | `id` |
| `name` | `Title` | `Name` | `title` |
| `brand` | `Vendor` | brand attribute | `brand` |
| `categories` | `Product Category` or `Type` | `Categories` (`>` separators become `, `) | `product_type` or `google_product_category` (`>` separators become `, `) |
| `image_url` | `Image Src` | first URL in `Images` | `image_link` |
| `price` | `Variant Price` | `Regular price` | number part of `price` |
| `currency` | ask the user | ask the user | currency part of `price`, e.g. `299.00 SEK` |

For a catalog that isn't products for sale, map by role: the title to `name`, the creator or publisher to `brand`, the taxonomy to `categories`, tags to `keywords`, and leave `price` and `currency` null. [Curate my wall](https://github.com/Unbox-AI/aic-artworks) does this for artworks: artist as `brand`, department, medium and style as `categories`.

Source-specific traps:

- Shopify exports one row per variant and per extra image. Group by `Handle` and keep the first row that has a `Title`.
- Variant-level catalogs: decide with the user whether a product is the parent or each variant. Default to the parent; variants mostly differ by size and add near-duplicate vectors.
- `categories` is one comma-separated string, general to specific, e.g. `Clothing, Trousers`.
- Set `event_type` and `group` to `"product"` on every row, whatever the catalog holds.

### 3. Fill the popularity columns when order data exists

Ask whether the user has order or sales history, or for other catalogs any interaction counts such as views, plays or saves. Without it, leave `sales_since`, `timestamp`, `frequency` null and say that every item will rank as equally popular.

With order lines (item id + date + quantity), compute per item, relative to today:

- `sales_since`: 12 int64 values. Units sold in the trailing 30, 60, 90, ..., 330 days (11 cumulative windows, so values never decrease), then the all-time total.
- `timestamp`: epoch seconds of the first sale.
- `frequency`: total interactions. Use the all-time total if nothing better exists.

### 4. Enforce the limits

- At most 100 000 items. If the source is larger, ask the user how to select, and suggest the top 100 000 by `frequency` or recent sales.
- Unique `id`. Report how many duplicates were dropped; the server would silently keep the first.
- `id` and `name` must be non-null. Drop rows missing either and report the count.

### 5. Build and validate the parquet

Write the mapped data as a CSV with the target column names to `.untracked/<catalog_name>.csv`, then convert it with `catalog_from_csv` (see "From CSV to parquet" in `docs/catalog-format.md`). It reads every cell as text, parses list columns, casts `timestamp` and `frequency` to int64, drops duplicate ids, writes `.untracked/<catalog_name>.parquet` and runs `check_catalog_schema` on it. If it raises `UnboxAIError`, the message names the column; fix the source and rerun until it passes.

```python
from behaviorgpt import catalog_from_csv

path = catalog_from_csv(".untracked/my_catalog.csv")  # .untracked/my_catalog.parquet
```

### 6. Check images before uploading

`image_url` carries most of the signal, and the job fails if none of the first 500 fetched images is usable. Fetch a random sample of ~20 URLs with a plain HTTPS GET, no cookies, no browser headers, and report the success rate and status codes per host.

- Relative paths: prefix the store's domain.
- `gs://`, `s3://`, login-gated or bot-protected hosts (403, 429, challenge pages): these will not be fetched. Tell the user they need a public image host or CDN URL.
- Protocol-relative `//cdn...`: prefix `https:`.

### 7. Upload, only after explicit confirmation

Uploading sends the catalog to UnboxAI. State the file path, row count, and image success rate, and ask before running:

```python
from behaviorgpt import Search, UnboxAIClient

client = UnboxAIClient(market="SE")  # market code of the user's store
job = client.embed(".untracked/my_catalog.parquet", wait=True, timeout=1800.0)
print(job.catalog_id)
```

The file name (without `.parquet`) is the catalog's name. Uploading the same name again refreshes that catalog and keeps its `catalog_id`; a new name takes one of the key's catalog slots (2 by default). Ask the user whether this is a new catalog or a refresh of one they already embedded, and name the file to match. If the upload is refused because the key is at its limit, tell the user; nothing was replaced, and they can refresh an existing catalog under its name or contact alexander@unboxai.com to remove one or raise the limit.

If the user only wants to add a few items to a catalog they already embedded, or change some of its items, build a parquet of just those items and upload it with `client.add_items(path, catalog_id=..., wait=True)` instead: only those items are embedded, an `id` already in the catalog is replaced, and the catalog keeps serving. A later full `embed` of the catalog replaces everything, so their main catalog file should get the same rows.

Load `.env` first (`from dotenv import load_dotenv; load_dotenv()`) if the key is not already in the environment. Embedding takes several minutes; run it in the background if the tool allows.

On failure, `UnboxAIError` carries the server's reason. `client.job_status(job.job_id).error` holds the same text. Map it back to a fix from `docs/catalog-format.md`, most often the image host.

### 8. Smoke test and hand over

Run one query and one similarity lookup against the new catalog, and show the top names:

```python
res = client.complete(history=[Search("<something in the catalog>")], catalog_id=job.catalog_id, limit=5)
print(res.names)

pick = client.random_item(job.catalog_id)
print(pick.data["name"], "->", client.similar_items(pick.id, catalog_id=job.catalog_id).names[:5])
```

Finish by giving the user the `catalog_id` (private to their API key), and point them to steps 6 to 8 of `notebooks/showcase.ipynb` and the demo at https://behaviorgpt.unboxai.com/ ("Bring your own catalog").