Scrollmark

Local-first X/Twitter research archive, search workbench, media viewer, and portable bundle system.

Built for researchers, designers, engineers, and high-signal collectors who need to preserve, search, inspect, and share what flows through the X web app without sending private archives to a cloud service.

Issues · 简体中文 · Bundle Format · QC Runbook · Performance Gates · Store Listing Draft · Publishing Runbook · Store Sync Artifact

Scrollmark is maintained by Kyle McCleary. It began as an MIT-licensed fork of prinsss/twitter-web-exporter, but has since been rebuilt and overhauled across capture, search, storage, UI, bundle import/export, diagnostics, performance, branding, and release workflows. Original copyright and license notices are preserved.

• What Scrollmark does
• Screenshots
• Feature matrix
• Installation
• Core workflow
• Search language
• Portable bundles
• Exports
• Privacy and security model
• Performance model
• Project map
• Development
• Testing and release gates
• Limitations
• FAQ
• Roadmap
• Attribution and license

Scrollmark runs as a userscript on x.com, twitter.com, and mobile.x.com. It observes the same GraphQL/API responses that the X web app loads while you browse, parses useful structures out of those responses, stores them locally in IndexedDB, and gives you a fast explorer for search, review, export, and sharing.

It is intentionally not a cloud product, not a bot, and not a Twitter/X developer API client. The core idea is simple: if the web app loads useful research material into your browser, Scrollmark can help you preserve and query it locally.

Visual research archive Switch the bookmark explorer into fullscreen masonry mode to scan papers, diagrams, design references, screenshots, article cards, and videos as a spatial archive. The masonry view keeps deterministic ordering and folder-aware filtering, so it behaves like an infinite visual feed rather than a static export preview.
	Table search and inspection Use the table view for dense metadata work: exact snippets, author operators, folders, metrics, dates, media cells, selected-row export, and raw IDs stay visible in one place. The search bar supports natural language, exact phrases, boosted phrase windows, boolean logic, exclusions, author shorthand, folders, dates, domains, and dotted metadata fields.
Portable Bundle Viewer Imported bundles reuse the same explorer spine as live bookmarks instead of falling back to a raw JSON scroller. A collaborator can open a shared archive, search it, filter it, inspect it in masonry mode, and export derived subsets without touching their own live X account data.
	Data and bundle export Export selected rows, all current results, or a canonical bundle ZIP for sharing. Bundle export is separate from raw JSON/CSV/HTML export so portable archives can carry manifest metadata and normalized record files. Search history export is available when ranking, phrase matching, or repro-quality diagnostics matter.
Fast media export Download image and video attachments in bulk, tune concurrency and pacing, include metadata sidecars, or copy URL manifests for external tooling. Media export is intentionally separate from canonical bundle export: bundle ZIPs are for portable data archives; media export is for large binary downloads.
	Live capture widget The floating widget tracks module counters while you browse, exposes monitors and Bundle Viewer, and keeps diagnostic controls close without forcing a separate backend or dashboard. Scrollmark observes the same browser-loaded GraphQL/API responses the X web app is already using.
Settings and localization Configure hook mode, safe mode, repair mode, module monitors, database actions, bundle library, language, theme, and diagnostics from the in-page settings panel. The UI has a localization layer rather than hard-coded English-only strings.

1. Install a userscript manager.
   • Firefox: Violentmonkey or Tampermonkey.
   • Chrome: Tampermonkey with browser user scripts enabled.
2. Install the latest Scrollmark userscript:

https://github.com/kmccleary3301/scrollmark/releases/latest/download/scrollmark.user.js

3. Open or hard-reload https://x.com/home.
4. Confirm the floating Scrollmark launcher appears on the page.
5. Open the widget and verify the header reads:

Scrollmark
By Kyle McCleary

Recent Chrome builds require explicit user-script permission for userscript managers.

The production build emits dist/scrollmark.user.js.

npm install
npm run build

For the local e2e install endpoint used during development, serve the parent workspace so the historical compatibility path resolves:

cd /home/skra/projects/twitter_scraping
python3 -m http.server 8123

Then install the generated e2e build from the compatibility URL used by vite.config.ts:

http://localhost:8123/greasemonkey_project/twitter-web-exporter/dist/twitter-web-exporter-e2e.user.js

1. Browse X normally.
   • Open home timeline, bookmarks, bookmark folders, user profiles, tweet threads, likes, followers/following pages, retweet panels, quote surfaces, lists, communities, or search timelines.
   • Scroll enough for the X web app to load the data you want preserved.
2. Watch the Scrollmark widget counters.
   • Counters represent parsed local captures, not all possible remote records.
   • If a page has not loaded a response in your browser, Scrollmark cannot parse it yet.
3. Open a module explorer.
   • Bookmarks is the primary research surface.
   • Tweet Details, User Tweets, Likes, User Details, and relationship modules expose other parsed shapes.
   • Bundle Viewer opens imported portable archives rather than live captures.
4. Search and filter.
   • Use plain text for broad recall.
   • Quote phrases for exact constraints.
   • Add operators for folders, authors, dates, media, domains, and numeric thresholds.
5. Choose a view.
   • Table view is best for metadata inspection, selection, and exports.
   • Masonry view is best for visual scanning of images/videos from tweets and article previews.
6. Export what you need.
   • Export Data for JSON/CSV/HTML or canonical bundle ZIP.
   • Export Media for large binary media downloads.
   • Export Search History when debugging search quality or ranking behavior.
7. Share safely.
   • Export a canonical bundle ZIP and send it to a collaborator.
   • The recipient imports it into the Bundle Library, where it remains isolated from their live X account data.

Scrollmark search is intentionally closer to a compact research query language than a plain browser find box. Unstructured text is expanded into content-term matches plus boosted adjacent phrase windows; quoted phrases are enforced as phrases; operators add hard constraints.

• Empty search defaults to newest-first ordering, preferably bookmark/save time where available and post time as fallback.
• Plain multi-term searches rank term hits and boosted adjacent phrase windows.
• Quoted phrases are strict phrase constraints.
• Folder filters and other metadata operators constrain the candidate set before display.
• Stale worker responses are ignored so slower older searches do not overwrite newer results.

Canonical bundles are ZIP files for sharing Scrollmark records without mutating anyone's account.

bundle.zip
├─ manifest.json
├─ records/
│  └─ records.jsonl
└─ media/
   └─ media-urls.txt

Imported bundles are stored in isolated IndexedDB tables:

imported_bundles
imported_bundle_collections
imported_bundle_items
imported_entity_snapshots
imported_bundle_import_reports

That isolation is deliberate. Importing a bundle does not create X bookmarks, follow users, like posts, write to live tweet tables, or modify the recipient's account state. See docs/bundles/canonical-bundle-v1.md for the v1 format boundary.

For very large binary media exports, browser memory limits still matter. Canonical data bundles are designed to be much lighter than media ZIPs because they primarily contain structured records and optional media URLs rather than the media bytes themselves.

Scrollmark is designed around local control.

Userscript permissions are intentionally narrow for this architecture:

@match    *://twitter.com/*
@match    *://x.com/*
@match    *://mobile.x.com/*
@grant    unsafeWindow
@grant    GM_xmlhttpRequest
@connect  cdn.syndication.twimg.com

unsafeWindow is used to observe web-app runtime/network behavior from the userscript environment. GM_xmlhttpRequest is used for controlled media/export support where normal page fetch semantics are insufficient.

The final release work focused on making large archives usable without forcing the whole corpus through the DOM or main thread.

Current local performance gates are documented in docs/release/final-hill-performance-gates.md. They include search latency, phrase-quality checks, browser-driven table/masonry scroll behavior, bundle ZIP latency, bundle roundtrip validation, and Chrome CDP metric collection smoke tests.

scrollmark/
├─ README.md
├─ package.json
├─ vite.config.ts
├─ docs/
│  ├─ bundles/
│  │  └─ canonical-bundle-v1.md
│  ├─ db-backed-table-baseline.md
│  ├─ db-backed-table-controller-audit.md
│  ├─ db-backed-table-developer-guide.md
│  ├─ release/
│  │  ├─ final-hill-performance-gates.md
│  │  ├─ final-release-checklist.md
│  │  ├─ store-listing-draft.md
│  │  └─ unified-qc-session-runbook.md
│  └─ screenshots/
│     ├─ README.md
│     ├─ hero-bookmarks-masonry-research.png
│     ├─ bookmarks-masonry-search-fullscreen.png
│     ├─ search-table-from-operator-fixed.png
│     ├─ bundle-viewer-fullscreen-clean.png
│     ├─ export-data-bundle-modal.png
│     └─ export-media-modal.png
├─ e2e/
│  ├─ bundles/
│  │  └─ canonical_bundle_roundtrip_harness.ts
│  ├─ fixtures/
│  │  └─ bundles/
│  └─ perf/
│     ├─ browser_viewer_scroll_harness.mjs
│     ├─ bundle_export_latency_benchmark.ts
│     ├─ run_final_hill_perf_suite.sh
│     ├─ search_engine_latency_benchmark.ts
│     └─ search_phrase_quality_harness.ts
└─ src/
   ├─ components/
   │  ├─ bundles/          # Bundle Library UI
   │  ├─ modals/           # Export/settings/dialog surfaces
   │  └─ table/            # Shared explorer, table, masonry, virtualization
   ├─ contracts/           # Versioned search contracts and knobs
   ├─ core/
   │  ├─ bundles/          # Canonical ZIP import/export machinery
   │  ├─ database/         # IndexedDB/Dexie schema and storage helpers
   │  ├─ options/          # Runtime options and persistence
   │  ├─ perf/             # Diagnostics/performance counters
   │  └─ search/           # Worker client/contracts/search worker
   ├─ i18n/
   │  └─ locales/          # UI translations
   ├─ modules/
   │  ├─ bookmarks/
   │  ├─ followers/
   │  ├─ following/
   │  ├─ interaction-events/
   │  ├─ likes/
   │  ├─ local-search/
   │  ├─ quotes/
   │  ├─ raw-capture/
   │  ├─ retweeters/
   │  ├─ search-timeline/
   │  ├─ tweet-detail/
   │  ├─ user-detail/
   │  ├─ user-media/
   │  └─ user-tweets/
   ├─ types/
   └─ utils/               # API parsing, search parsing/ranking, media/export helpers

Install dependencies:

npm install

Install Playwright browsers when running browser harnesses:

npx playwright install chromium firefox

Some internal filenames and IndexedDB discovery strings intentionally retain twitter-web-exporter compatibility names. The user-facing product identity is Scrollmark.

Run the baseline gates before opening a release PR or publishing a userscript artifact:

npm run lint
npm run build
npm run build:e2e
TWE_BUILD_VARIANT=chrome-e2e npx vite build
./e2e/perf/run_final_hill_perf_suite.sh

Manual release QC is intentionally backloaded and documented in docs/release/unified-qc-session-runbook.md. The short version: verify install, capture, search, table scrolling, masonry scrolling, bundle export/import, media export, diagnostics export, Firefox behavior, and Chrome behavior against real X sessions.

Scrollmark is powerful, but the boundary is explicit.

• It can only parse data the X web app actually loads in your browser.
• X can change route names, GraphQL response shapes, or timeline structures; parser updates may be required.
• It does not bypass visibility limits in the web app.
• It does not automate scrolling or account actions by itself.
• Imported bundles are local archives, not instructions to recreate bookmarks/folders in an X account.
• Browser storage quotas and memory limits still apply, especially for very large media exports.
• The project is currently optimized for desktop Firefox and Chrome userscript-manager installs.

The near-term release posture is stabilization rather than feature sprawl.

Scrollmark is maintained by Kyle McCleary and published at kmccleary3301/scrollmark. The project remains licensed under MIT.