Scrape

The engine's lead lane. Say it in prose — "scrap facebook for hogwarts", "who should we contact", "contact gap", "اسحب" — and the lead agent plus the /scrape runbook activate with the product scope already resolved.

kun holds none of the scraping code. It holds the routing: the keywords, the skill, the agent, two hooks, one workflow, and this page. The scrapers live in the product repos, where they belong.

The number that decides everything

Measured 2026-08-17 by hogwarts/scripts/crm/contact-gap.ts over the live hogwarts workspace — 3,316 school leads:

So: automated enrichment caps at +93 rows, and a full scrape run adds +15 (505 of the scraper's 817 names were already in the CRM). Meanwhile 186 tier-A/B schools are contactable and completely unworked.

The free OSM lane has been run, and it is finished

On 2026-08-17 hogwarts/scripts/crm/osm-refetch.ts re-read all 3,145 OSM elements the original import had reduced to a name and a coordinate. The projection going in was ~210 new contacts. The measured result was one.

The reason is the useful part, and it is the kind of thing only running it can tell you: the CRM already held OSM's contact tags. 101 phones, 50 emails and 36 websites came back byte-identical to what was already stored. The import did take contact where OpenStreetMap had it — what it dropped were the attribute tags. A second run now plans zero writes.

What the lane did buy is real, just not reachability: +2,528 coordinates (3,136 of 3,156 rows carry one now), +1,106 English names, +58 private/public classifications, and grades, ISCED level, gender, operator, street, city and postcode. Coordinates are infrastructure — they are what lets a directory or a paid lookup match the right school instead of a same-named one two countries away.

Government open data is the answer to the other 93%

Not a better scraper. As of 2026-08-17 that is measured rather than argued:

48 schools from one emirate, against the entire free OSM lane's 1. Caveats worth carrying: ADEK's emails are mostly regulator-routed (9338@adek.gov.ae) — deliverable, but not a direct line; phones are local-format and must go through normalize-contacts.ts; and a public register is not automatically redistributable, so check the licence before republishing it (enriching our own CRM is ordinary use). Dedup on adek:<SCH_ESIS_ID>.

That is why this lane is configured the way it is. Every spell makes contact-gap analysis, enrichment and outreach the easy path, and makes raw discovery something you choose out loud with the number in front of you. Closing the 93% needs a real school directory — Saudi MoE, ADEK, Egypt MoE — or the network. Not a better scraper.

mkan reached the identical conclusion independently for its own market: "Coverage is not the constraint on this business; inventory is."

Two more findings that reorder the channel plan:

• Only 45 of the 176 contacts are mobile. The rest are switchboards. A landline in a WhatsApp campaign is a silent non-delivery that reads as disinterest, so reach is labelled, never guessed.
• 119 rows carry an email. For this MENA-wide list (SA 1001 · EG 764 · SD 609 · AE 603 · QA 173) email is the larger channel, not WhatsApp. WhatsApp stays right for the Sudan slice.

Keywords

The Owlery — school 20 in the spellbook.

Abdout types scrap. Both spellings trigger, in the spellbook and in the skill's when_to_use.

"scrap whatsapp" is not a thing, and the spell says so

WhatsApp has no scrapeable surface — no directory, no public profile index, nothing to crawl. The spell exists because the phrase gets said, and it routes to the two real capabilities behind it:

1. Harvest numbers from the public web and Facebook Page About tabs (that is the FB and website lanes), then normalize-contacts.ts to E.164, labelling mobile vs landline.
2. Send through hogwarts/src/lib/whatsapp/ — the Evolution API sender that already exists with retry, rate limiter, templates and dispatch. Never build a second sender.

Configuring a capability we do not have would be worse than not configuring it at all.

Where the code lives

twenty-rest.ts is vendored, not shared: fix bugs in both copies until it is extracted to a package, and prefer mkan's if they ever disagree.

import-to-twenty.js is forbidden. It mints a fresh crypto.randomUUID() per row with ON CONFLICT (id) DO NOTHING — no dedup key — and writes raw SQL into every workspace, so Sudan schools land in mkan, sijillee and moallimee. Use twenty-upsert with a stable external id (fb:<pageId> / osm:<nodeId>).

Reaching Twenty

Copyexport TWENTY_API_URL=http://localhost:3100   # NEVER 3000 — that is hogwarts' Next dev server
export TWENTY_API_KEY=$(security find-generic-password -s databayt-twenty -a hogwarts -w)
npx tsx scripts/crm/contact-gap.ts

Keychain service databayt-twenty, account = hogwarts | mkan | sijillee | moallimee. Off this Mac, the backend answers at the Tailscale Funnel https://twenty-api-2.tail42a5c4.ts.net. The backend is Docker on Abdout's laptop, so the CRM is down whenever the laptop is — every scheduled CRM job is a Mac job. See CRM.

Defaulting to port 3000 is not hypothetical: it is why Hermes' CRM crons silently returned nothing for weeks while reporting last_status: ok.

Readers and backends — Agent Reach

Every reading lane has a backend, and a lane that returns nothing is ambiguous until you know which one failed: the target had no contact, or the reader was broken. A broken reader reported as "low yield" is how a good lane gets abandoned, so the runbook now demands a one-known-good-row pre-flight before any batch, and asks every yield number to name the reader that produced it.

Two tools cover this, and the line between them is the account-risk line:

Scrapling covers everything that needs no login, which is most of the remaining yield. Agent Reach is a capability layer over readers — one CLI routing Facebook, Instagram, web, RSS and Exa semantic search across multiple backends, with agent-reach doctor --json reporting which backend serves each platform right now. Its premise is that access methods break and get swapped underneath you, which is precisely this lane's failure mode.

Status: both installed 2026-08-19 — Scrapling 0.4.14, Agent Reach v1.5.0, each in its own uv tool environment. An earlier version of this page said Python 3.9.6 blocked them; that was wrong and is corrected rather than quietly deleted — python3 is 3.9.6, but python3.11 and uv are both present.

Copyuv tool install --with markdownify 'scrapling[fetchers]' && scrapling install
uv tool install 'https://github.com/Panniantong/agent-reach/archive/main.zip'

Two install traps, both hit for real. uv tool install agent-reach fetches the wrong package — PyPI's agent-reach is 0.1.0 by a different author pointing at github.com/jgalea/agent-reach, a name collision rather than the ~73k★ project; the version mismatch is the tell, and agent-reach --version must print v1.5.0. And scrapling[fetchers] omits markdownify, so scrapling extract renders the page and then dies at the markdown step.

The payoff, measured the same day on a real queue row. tpsdxb.com/contact-us — 5,652 bytes of JS shell with zero contact under plain curl — yielded info@tpsdxb.com, registrar@tpsdxb.com and +971-428-444-65 through scrapling extract fetch. One unreachable row became contactable.

agent-reach doctor is a config check, not a liveness check. It reports 4/15 channels up and marks "any web page via Jina Reader" ✅ available, while r.jina.ai returns 451 for every URL from this machine. Doctor verifies that a backend is configured, not that it answers — which is precisely why this lane's pre-flight is one known-good row, not a health command. Facebook, Instagram and Exa remain unlocked pending OpenCLI + the dedicated account, and mcporter.

What was measured before adopting anything, 2026-08-19:

Agent Reach's headline free win is the piece that does not work here, and the fix for the gap it left came from the second tool rather than the first. That is the whole argument for measuring a dependency before designing around its README.

Where Scrapling must not go. Its StealthyFetcher bypasses Cloudflare and spoofs fingerprints — ordinary against an anonymous school website, and the worst possible pairing against a platform we hold an irreplaceable logged-in account on, because evasion raises the stakes of detection on the one account we cannot re-buy. The guard blocks that shape and the --cdp-url shape (which would attach to the vault) while leaving every anonymous run free, which is the whole reason to adopt it. We honour robots.txt on the website lane; Scrapling makes that optional and we do not.

Its MCP server is a separate decision from the library. Useful for inspecting one page interactively; wrong for the queue, because this lane's rule is zero tokens per lead — 600+ pages through an MCP is 600+ pages of model context for work a local extractor does free.

The account rule widens rather than relaxes. Agent Reach does not log in for you — its OpenCLI backend drives the browser session you already have, which on this machine is the session vault holding Abdout's personal Facebook. So scrape-guard was extended to match opencli facebook, opencli instagram and Agent Reach's Facebook/Instagram subcommands before any install rather than after; agent-reach doctor, install and check-update stay unblocked because a guard that cries wolf gets routed around. Because Agent Reach's design is that backends get swapped, the guard carries a maintenance note: when doctor reports a new active_backend for those channels, add that binary — it cannot discover the swap on its own.

Its skill is deliberately not installed. Agent Reach ships a Claude Code SKILL.md whose description is "MUST USE when the user wants to research/search anything, or mentions any platform or URL" — a fleet-wide dispatch collision, and the listing budget has roughly 80 characters of headroom. The runbook calls its CLI instead.

MCP — REST only, deliberately

There is no Twenty MCP configured, and we are not adding one. Twenty does ship an MCP server in-tree (twenty-server/src/engine/api/mcp/), so this was a real choice:

• The REST + metadata API is the surface that survives. The packages/twenty-api rewrite (Slices 0–11, ~10–15% to parity) preserves REST. A raw-SQL or MCP-specific integration is migration debt against a backend that is actively being replaced.
• Adding it means changing the pinned backend. The Docker image is pinned at v2.31.1 after a restore-verified backup; enabling a new endpoint on it is config risk for no capability we lack.
• Scrapers cost zero tokens, and should stay that way. REST scripts keep 3,156 CRM rows out of model context entirely. An MCP would pull them in per query.

Same reasoning rules out the postgres MCP for Twenty: never psql into a workspace schema. Raw SQL skips search vectors, timeline and activity, and it is exactly what the rewrite does not preserve.

What the lane does use: chrome-devtools / browser (user-level) for the CDP Facebook lane, and keychain for the workspace token.

The two hooks

scrape-guard (PreToolUse, Bash) — blocks a Facebook scrape that would drive the wrong Chrome.

On this machine, ~/.claude/bin/chrome-debug.sh runs one persistent session vault Chrome at ~/.claude/chrome-debug-profile on port 9222 — the browser Abdout logs into by hand and the browser agents attach to. Pointing the scraper at it means scraping Facebook as him, and losing that account administers the business portfolio. Page tokens are no longer at stake — publishing moved to a System User on 2026-08-19 — and the scrape account is kept cheap to lose by holding zero roles.

A shell hook cannot ask Chrome which Facebook user is signed in — CDP exposes the browser build, never the profile. So the guard checks the thing that is checkable and is the actual failure mode: it reads --user-data-dir straight off the live Chrome process serving the port the run will attach to. Session vault → blocked. It also warns, without blocking, when a run declares no throttle.

Copyexport FB_SCRAPE_PORT=9333
export FB_SCRAPE_PROFILE="$HOME/.claude/chrome-fbscrape-profile"
export FB_SCRAPE_DELAY_MS=4000

Read-only commands (contact-gap.ts, normalize-contacts.ts) are never blocked.

It also covers Agent Reach and Scrapling — added before either tool was installed. Agent Reach's social subcommands are blocked outright because its OpenCLI backend drives the existing browser session by design. Scrapling is deliberately not blocked wholesale: its anonymous lane is the one we want, so only two shapes are caught — aimed at a logged-in social platform, or attaching to an existing CDP endpoint. agent-reach doctor, install and check-update stay unblocked, as does every ordinary scrapling extract against a school website or a government register.

opencli, agent-reach and scrapling also had to join the execution-verb list: they are invoked as bare binaries, so without that the new patterns would have matched the command text and then been discarded by the verb gate — coverage that looks present and does nothing. Nineteen smoke cases cover the matrix, including the four that must stay allowed.

scrape-yield (PostToolUse, Bash, async) — appends the contact-gap delta to .claude/logs/scrape-runs.log after every lead-lane run, so yield is measured rather than remembered. It never calls Twenty; it diffs the artifact contact-gap.ts already writes. If that file was not refreshed, it logs UNMEASURED rather than reporting a zero delta as though zero were a measurement.

Every earlier read of this funnel was optimistic and each was corrected only because someone measured. That is what the ledger is for.

The workflow

Workflow({ name: "scrape", args: { product: "hogwarts" } }) — gap → enrich → tier → report, dry-run by default.

Discovery is off unless you ask for it (stages: ["gap","discover","enrich","tier","report"]), and when it runs, the stage states the +15 measured yield before doing anything.

Every stage must return a dropped[] array — a top-N cap, a sampling limit, a rate-limit stop, a checkpoint bail, an auth wall. Silent truncation reads as "we covered everything" when it didn't, so the schema requires the field and the workflow logs each entry before the summary. The return value carries a recommendation, because a lead pass that reports counts without naming the next move is how "scrape more" wins by default.

Guardrails

• Dry-run first, --apply second. Run --apply twice; the company count must be identical after the second run. That is the dedup test.
• Fill-empty-never-replace-populated. A conflict becomes a dated note, not an overwrite.
• Zero tokens per lead. Claude writes the extractor; the extractor runs on all 3,156 rows free. Bulk classification → Gemini free tier (~20 req/day/model). Judgment → the local claude -p drain on the Max pool. No ANTHROPIC_API_KEY — it is dead in prod (401).
• Facebook: dedicated account, throttled, never Abdout's.
• WhatsApp: dedicated number, a separate Evolution instance from school notifications, warm-up ramp 10 → 20 → 30/day, randomized 40–180s gaps, stop-on-reply, OUTREACH_SEND=off kill switch.

Related

• Keywords — The Owlery, school 20
• CRM — the Twenty workspaces
• Agents · Hooks · Workflows