Example: Wikinews

This page indexes a real publication end to end, with real data you can download yourself. The source is Wikinews, the free news wiki. Its full archive is a public dump under a Creative Commons licence, so you can run every step here without a content deal.

The English archive holds about 20,900 published articles, from 2004 to 2026. The median article is 249 words.

By the end you will have parsed the dump, sent it to Miso, confirmed it landed, and asked a question that cites one of the articles.

Every step on this page was run for real, against the live dump and a live Miso app, so the numbers and the responses are the ones you get.

What you need

A secret API key. Keep it on a server. See Authentication.
Python 3, with mwxml and mwparserfromhell.

export MISO_SECRET_KEY="YOUR_SECRET_KEY"

Step 1 — Get the dump

Wikimedia publishes a new dump every few weeks. The articles file is about 47 MB compressed:

curl -O https://dumps.wikimedia.org/enwikinews/latest/\
enwikinews-latest-pages-articles.xml.bz2

You do not need to decompress it. Python reads it as it goes.

To try the flow on a handful of articles first, pull them from the MediaWiki API instead, with action=query&generator=categorymembers&gcmtitle=Category:Published.

Step 2 — Map the fields

Wikinews articles are wikitext, so the parser reads templates and link markup rather than columns. Each piece maps to a Miso field:

In the wikitext	Miso field	Note
`<id>` from the dump	`product_id`	Stable. The title is not — articles get renamed.
`<title>`	`title`
`{{date\|…}}`, `{{byline\|date=…}}`	`published_at`	The publication date. The revision timestamp is the last edit, which can be years later.
Body text	`description`	After the templates and link markup are stripped.
`[[File:…]]`	`cover_image`	The lead image. Resolve the file name to a URL.
`[[Category:…]]`	`tags`	Drop the maintenance categories.

Step 3 — Parse the dump

Two maintained libraries do the heavy lifting. mwxml reads the dump, and mwparserfromhell parses wikitext into a tree, so templates and links come out correctly nested instead of half-matched by a regular expression.

pip install mwxml mwparserfromhell

Stream the file: 20,900 articles will not fit in memory as one tree. Keep only namespace 0, and only articles that carry {{publish}} — the rest are drafts, talk pages, and templates.

import bz2, re
from datetime import datetime, timezone
from urllib.parse import quote

import mwparserfromhell
import mwxml

DUMP = "enwikinews-latest-pages-articles.xml.bz2"
DROP_SECTIONS = {"sources", "external links", "related news"}
SKIP_CATEGORY = re.compile(
    r"^(Published|Archived|Articles? |Pages |Translated news|"
    r"Original reporting|Corrected|[A-Z][a-z]+ \d{1,2}, \d{4}$)", re.I)
IMAGE_EXT = re.compile(r"\.(jpe?g|png|gif|svg)$", re.I)
NOT_A_PHOTO = re.compile(r"(logo|icon|stub|wikinews|commons)", re.I)
DATE_FORMATS = ("%B %d, %Y", "%d %B %Y", "%B %Y")

def _to_iso(raw):
    for fmt in DATE_FORMATS:
        try:
            d = datetime.strptime(raw.strip(), fmt)
            return d.replace(tzinfo=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
        except ValueError:
            continue
    return None

def parse_date(code):
    """{{date}}, then {{byline|date=}}, then the date category."""
    for tpl in code.filter_templates():
        name = tpl.name.strip().lower()
        if name == "date" and tpl.params:
            iso = _to_iso(str(tpl.params[0]))
            if iso:
                return iso
        if name == "byline" and tpl.has("date"):
            iso = _to_iso(str(tpl.get("date").value))
            if iso:
                return iso
    for link in code.filter_wikilinks():
        title = str(link.title)
        if title.lower().startswith("category:"):
            iso = _to_iso(title.split(":", 1)[1])
            if iso:
                return iso
    return None

def body_text(code):
    keep = []
    for section in code.get_sections(include_lead=True, flat=True):
        heads = section.filter_headings()
        name = heads[0].title.strip().lower() if heads else ""
        if name in DROP_SECTIONS:
            continue
        for h in heads:
            section.remove(h)
        keep.append(section.strip_code(normalize=True, collapse=True))
    text = "\n\n".join(k.strip() for k in keep if k.strip())
    return re.sub(r"\n{3,}", "\n\n", text)

def topics(code):
    out = []
    for link in code.filter_wikilinks():
        title = str(link.title)
        if not title.lower().startswith("category:"):
            continue
        name = title.split(":", 1)[1].strip()
        if name and not SKIP_CATEGORY.match(name) and name not in out:
            out.append(name)
    return out

def cover_image(code, width=1200):
    for link in code.filter_wikilinks():
        title = str(link.title)
        if not re.match(r"(File|Image):", title, re.I):
            continue
        name = title.split(":", 1)[1].strip()
        if not IMAGE_EXT.search(name) or NOT_A_PHOTO.search(name):
            continue
        slug = quote(name.replace(" ", "_"))
        return (f"https://en.wikinews.org/wiki/Special:FilePath/"
                f"{slug}?width={width}")
    return None

def is_published(code):
    return any(t.name.strip().lower() == "publish"
               for t in code.filter_templates())

def to_product(page, revision, code):
    body = body_text(code)
    slug = quote(page.title.replace(" ", "_"))
    record = {
        "product_id": f"wikinews-{page.id}",
        "title": page.title,
        "url": f"https://en.wikinews.org/wiki/{slug}",
        "description": body,
        "published_at": parse_date(code),
        "updated_at": revision.timestamp.strftime("%Y-%m-%dT%H:%M:%SZ"),
        "cover_image": cover_image(code),
        "tags": topics(code)[:12],
        "custom_attributes": {
            "source": "wikinews",
            "license": "CC BY 2.5",
            "word_count": len(body.split()),
        },
    }
    return {k: v for k, v in record.items() if v}

def articles(path=DUMP):
    dump = mwxml.Dump.from_file(bz2.open(path, "rt", encoding="utf-8"))
    for page in dump:
        if page.namespace != 0 or page.redirect:
            continue
        for revision in page:
            if not revision.text:
                break
            code = mwparserfromhell.parse(revision.text)
            if is_published(code):
                yield to_product(page, revision, code)
            break

The parse takes about two minutes for the whole archive, and yields 20,903 articles. strip_code() is the call worth having: it removes templates, links, and markup in one pass, and it does not trip over a template inside a template.

One record comes out like this:

{
  "product_id": "wikinews-820",
  "title": "Big Linux Beta 3 released",
  "url": "https://en.wikinews.org/wiki/Big_Linux_Beta_3_released",
  "cover_image": "https://en.wikinews.org/wiki/Special:FilePath/Biglinux1.png?width=1200",
  "description": "Big Linux 2.0 Beta 3 has been released. Big Linux is a Live Compact Disc Brazilian Linux distribution, based on Knoppix…",
  "published_at": "2004-11-15T00:00:00Z",
  "updated_at": "2024-12-17T15:53:33Z",
  "tags": [
    "Brazil",
    "Computing",
    "Science and technology",
    "FLOSS",
    "Linux",
    "Software"
  ],
  "custom_attributes": {
    "source": "wikinews",
    "license": "CC BY 2.5",
    "word_count": 280
  }
}

Should you send the HTML?

Send html when you have it, because Miso reads the tags to understand the structure of the document. The dump carries wikitext only, so this example sends description, which is enough for a first index. For a production load, render each page with the MediaWiki API (action=parse) and send html instead. That is one request per article, so cache the result.

The image is a file name, not a URL

Wikitext names the file — [[File:Gross-Powell.jpg|thumb|Colin Powell]] — and gives no address. Special:FilePath resolves it, whether the file sits on Wikinews or on Commons:

https://en.wikinews.org/wiki/Special:FilePath/Gross-Powell.jpg?width=1200

Take photo extensions only, because .ogv files also appear as [[File:…]], and drop logos and icons. 55% of the archive has a usable image. Omit cover_image for the rest, because an empty string tells your front end nothing.

Get the date right, or ruin recency

Here published_at is 2009 and updated_at is 2024, because a maintenance edit touched thousands of old articles. Never fall back to the revision timestamp. It puts 2004 news at the top of every question about recent events.

The parser reads {{date}}, then {{byline|date=}}, then the date category. That covers every article but 369 of them (1.8%), which get no published_at at all.

Step 4 — Validate before you write

Send one record with dry_run=1. Nothing is written, and the schema is checked:

import json, os, requests
from wikinews import articles

API = "https://api.askmiso.com"
HEADERS = {
    "X-Api-Key": os.environ["MISO_SECRET_KEY"],
    "Content-Type": "application/json",
}

first = next(articles())
res = requests.post(f"{API}/v1/products?dry_run=1",
                    headers=HEADERS, json={"data": [first]})
print(res.status_code, res.json())

A pass returns Dry run pass. A 422 names every problem in the batch. Fix the mapping now, while the batch is one record.

Step 5 — Upload the archive

Send about 100 records per call. One bad record fails a whole batch, so keep batches small enough to debug. A product upload is asynchronous: it returns a task id, and you poll until the records land.

import time

CHUNK = 100

def send(products):
    res = requests.post(f"{API}/v1/products", headers=HEADERS,
                        json={"data": products})

    if res.status_code == 429:            # over the rate limit
        time.sleep(5)
        return send(products)

    if res.status_code == 422:            # nothing was written
        for problem in res.json().get("data", []):
            print("invalid:", problem)

    res.raise_for_status()
    return res.json()["data"]

def wait_for(task_id, timeout=300):
    url = f"{API}/v1/products/_status/{task_id}"
    deadline = time.time() + timeout
    while time.time() < deadline:
        body = requests.get(url, headers=HEADERS).json()
        if body.get("message") != "pending":
            return body["data"]["items"]
        time.sleep(2)
    raise TimeoutError(task_id)

def index_all():
    batch, done = [], 0

    for product in articles():
        batch.append(product)
        if len(batch) < CHUNK:
            continue

        accepted = send(batch)
        # Most apps are asynchronous. Some return the result at once.
        if "task_id" in accepted:
            items = wait_for(accepted["task_id"])
        else:
            items = accepted.get("items", [])

        done += len(items)
        print(f"{done} articles indexed")
        batch = []

    if batch:
        send(batch)

Each finished batch reports one entry per record:

{
  "message": "success",
  "data": {
    "items": [
      {
        "id": "wikinews-100811",
        "status": "created"
      }
    ]
  }
}

The archive runs oldest first, which is what you want: history lands before recent news is written on top of it.

The full load: 20,901 records in 210 batches, about 23 minutes, with no failed batch. Four articles are skipped, because the wikitext cleaner leaves them with an empty body.

The index trails the upload. At the halfway point Miso accepted 11,000 records, and 8,501 of them were searchable. Both numbers meet at the end, so count the catalog when the load finishes, not while it runs.

Step 6 — Ask a question

def ask(question, reader="wikinews-demo"):
    res = requests.post(f"{API}/v1/ask/questions", headers=HEADERS,
                        json={"anonymous_id": reader,
                              "question": question})
    qid = res.json()["data"]["question_id"]

    url = f"{API}/v1/ask/questions/{qid}/answer"
    while True:
        answer = requests.get(url, headers=HEADERS).json()["data"]
        if answer["finished"]:
            return answer
        time.sleep(1.5)

answer = ask("What did Wikinews report about the Linux operating system?")
print(answer["answer"])
for source in answer["sources"]:
    print("-", source["title"], source["url"])

That query answers from the archive you uploaded, about 23 seconds after the question:

Wikinews reported extensively on the major shift in the CentOS ecosystem
following Red Hat's December 2020 announcement to move focus away from CentOS
Linux in favor of CentOS Stream. This transition effectively moved the
end-of-life (EOL) date for CentOS 8 from May 2029 to December 31, 2021,
causing significant concern within the community [1][2].

sources carried three articles. Each one looks like this:

{
  "product_id": "wikinews-2911067",
  "title": "Red Hat to move focus away from CentOS in favour of Stream…",
  "url": "https://en.wikinews.org/wiki/Red_Hat_to_move_focus_away_from_CentOS…",
  "date": "2020-12-14T00:00:00+00:00",
  "snippet": "CentOS 8's End of Life (EOL) has been moved up from May 2029…",
  "highlight_text": "CentOS 8's End of Life (EOL) has been moved up from May 2029…",
  "boosted": false
}

The [1] markers in the answer point at those entries, in order. Miso also returns six follow-up questions, such as "What is the primary goal of the Rocky Linux project?".

If the answer cites a Wikinews article and the link works, the integration is done.

An empty or refused answer usually means the batch is not indexed yet, or the question is outside the archive. A refusal is not an error: finish_reason is still success, and sources is empty.

Keep it in sync

The Wikinews archive is fixed: the wiki stopped publishing new articles, so one load is enough. Your own newsroom is not fixed. Miso never compares your source against its index, so send every change as it happens.

Event	What to send
An article is published	`POST /v1/products` with the new record.
An article is edited	`POST /v1/products` with the same `product_id`. It overwrites.
An article is withdrawn	`POST /v1/products/_delete`.

curl -X POST "https://api.askmiso.com/v1/products/_delete" \
  -H "X-Api-Key: $MISO_SECRET_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "data": { "product_ids": ["wikinews-100811"] } }'

The body takes product_ids and nothing else, as a flat array. A nested array returns 422. Delete is asynchronous, the same as an upload, so it returns a task id that you poll.

Ask Miso what it holds

GET /v1/products/_ids returns every product_id in your index, in one response. There is no paging.

curl "https://api.askmiso.com/v1/products/_ids" \
  -H "X-Api-Key: $MISO_SECRET_KEY"

{
  "message": "success",
  "data": {
    "ids": ["wikinews-100811", "wikinews-129125"]
  }
}

Add type=article to count one content type only.

Reconcile after each dump

This is the repair for a missed delete. Compare what Miso holds against what your source holds now, and remove the difference.

def indexed_ids():
    res = requests.get(f"{API}/v1/products/_ids", headers=HEADERS)
    res.raise_for_status()
    return set(res.json()["data"]["ids"])

def reconcile(source_ids, apply=False):
    """Delete records Miso still holds that the source no longer has."""
    indexed = indexed_ids()
    orphans = sorted(indexed - set(source_ids))

    print(f"{len(indexed)} indexed, {len(source_ids)} at source, "
          f"{len(orphans)} to delete")

    # Guard: a broken extract looks like "everything was deleted".
    if orphans and len(orphans) > 0.10 * len(indexed):
        raise SystemExit("Refusing: more than 10% would be deleted. "
                         "Check the source extract first.")

    url = f"{API}/v1/products/_delete"
    if not apply:
        url += "?dry_run=1"

    for i in range(0, len(orphans), CHUNK):
        res = requests.post(
            url, headers=HEADERS,
            json={"data": {"product_ids": orphans[i:i + CHUNK]}})
        res.raise_for_status()

    return orphans

# Wikinews: product_id is the page id, so build the set from the new dump.
source_ids = {p["product_id"] for p in articles()}
reconcile(source_ids)                 # dry run, prints the count
# reconcile(source_ids, apply=True)   # then delete for real

Run it after each new dump. Three things make it safe:

dry_run=1 first, every time. It reports without deleting.
The percentage guard. If your extract fails and returns 40 ids, the difference is your whole catalog. The guard stops that.
product_id is the page id, not the title. A renamed article keeps its id, so a rename never looks like a deletion.

A missed delete never expires. If an article comes down at the source and no delete reaches Miso, the old version stays in the index and keeps appearing in answers. Wire the delete to the same event that unpublishes the page.

For a wiki, action=query&list=recentchanges&rcnamespace=0 gives you the changed pages since your last run. For a CMS, use its publish webhook.

Without the libraries

If you cannot add dependencies, the same parse works with the standard library and regular expressions: xml.etree.ElementTree.iterparse over bz2.open, then a pass of re.sub to strip templates and links. Both versions were run on the same dump:

Version	Articles	Images	Undated	Time
`mwxml` + `mwparserfromhell`	20,903	11,423	365	118 s
Standard library, regular expressions	20,905	11,519	369	27 s

The regular expressions are four times faster and agree to within 0.01%. They are also the version that breaks first: a template inside a template, or a link inside a caption, and the body text silently keeps its markup. Use them for a one-off. Use the libraries for anything you run twice.

The two-article difference is redirects, which mwxml filters for you.

Licence

Wikinews text is CC BY 2.5 for the archived articles, and each page states its own terms. Attribution is required, so keep url on every record.

Images carry their own licence, which is not the article's licence. Some are fair use. Check the file before you publish a cover image outside a test.

Before you call it done

One article maps correctly, and a dry run passes.
published_at is the publication date, not the last edit.
Every batch reported its records.
A question about a known article cites that article, with a working link.
A change at the source reaches Miso within your sync window.

Integrating Your Data — every field, and the other upload methods.
Getting Started — the SDK, to put this on a site.

Getting Started

Integrating Your Data