LinkedIn posts pipeline

LinkedIn posts pipeline

Keeps docs/linkedin/posts/ in sync with Rafael’s LinkedIn activity.

LinkedIn (logged-in browser tab)
   │  1. crawl_recent_activity.js  (paste in console, or via Playwright)
   ▼
tmp/chunks/chunk_*.json            (raw crawl data — gitignored)
   │  2. generate_posts.py
   ▼
docs/linkedin/posts/{N}.md + README.md

LinkedIn has no public API for profile activity, so step 1 runs inside your logged-in session against the same internal endpoint the “Recent activity” page uses. It only reads your own feed data. Re-run whenever you posted something new.

Step 1 — Crawl

  1. Open https://www.linkedin.com/in/rafaelpbelchior/recent-activity/all/ in a browser where you are logged in.
  2. Open DevTools → Console, paste the contents of crawl_recent_activity.js, press Enter.

  3. Run:

    await __liCrawlRun({ untilYear: 2024, chunkSize: 40 })

    untilYear stops the crawl once a post older than Jan 1 of that year is seen (default 2020). Use the current year minus a bit to keep runs fast.

  4. Save the chunks. Two options:
    • Console dump: run __liCrawlDumpAll(40) and copy each __LICHUNK__{...} line into tmp/chunks/chunk_<start>-<end>.json.
    • Playwright/agent: retrieve with __liCrawlChunk(0, 40), __liCrawlChunk(40, 80), … and write each result to tmp/chunks/chunk_<start>-<end>.json. (Tool-output wrapping like Result: "…" is fine — generate_posts.py unwraps it.)

Chunk file names must match chunk_*.json.

Step 2 — Generate markdown

python3 scripts/linkedin/generate_posts.py               # since 2020-01-01 (default)
python3 scripts/linkedin/generate_posts.py --since 2024-01-01

Behavior:

  • merges all chunks, dedupes by activity id
  • keeps posts on/after --since (dates are exact: decoded from the URN)
  • rewrites docs/linkedin/posts/ (removes existing *.md first)
  • newest post = 1.md; front matter: date (UTC), urn, type (post/reshare), media, url, and reshare_of for reshares
  • regenerates README.md index table

Step 3 — Update the work log

Update docs/linkedin/work.log.md (“Latest crawl” table) with: crawl date, API pages, activity count, coverage range, and the new newest post (always posts/1.md — note its date/URN there).

Notes / caveats

  • queryId drift: if the crawl fails with HTTP 400/404, LinkedIn bumped the GraphQL query id. Open the Recent-activity page → Network tab → find a voyagerFeedDashProfileUpdates request and copy its queryId into crawl_recent_activity.js (QUERY_ID const) — or in a pinch run a crawl ad hoc and paste the URL from the network tab.
  • For reshares, commentary.text holds the shared post’s text; Rafael’s own commentary (if any) is what appeared above it. The initial DOM scrape (tmp/linkedin-posts.json, gitignored) captured his own comments for the 5 newest activities and is overlaid by the generator when present.
  • tmp/ is gitignored (see .gitignore / _config.yml exclude:) — raw crawl data never reaches the repo or the published site.
  • AGENTS.md rules respected: nothing is pushed, nothing leaves the workspace.