You can do it yourself with requests + readability + markdownify. Then you add crawling, robots.txt, retries, chunking and dedup, and soon you're maintaining a crawler. This tutorial shows a shortcut: URL to Markdown for LLM & RAG, an Apify Actor I built for this job.

What it does

• Fetches public pages over plain HTTP (no headless browser), about 0.5 s per page in our tests.
• Extracts the main content and drops menus, footers, ads and cookie banners. Or use full mode for the whole page.
• Outputs Markdown or plain text, plus metadata: title, description, language, published date, word and character counts.
• Optionally pre-chunks the text with overlap, ready for embeddings.
• Optionally crawls same-site links or seeds from sitemap.xml, so you can grab an entire docs section.
• Can generate llms.txt and llms-full.txt for the pages it converts.

Step 1: Convert a single page

The simplest input is a list of URLs:

{
  "urls": ["https://docs.python.org/3/tutorial/index.html"],
  "extractMode": "main",
  "outputFormat": "markdown"
}

urls is the only required field. It accepts full URLs, bare domains (example.com/page) or several URLs on one line.

Step 2: Crawl a docs section and chunk it

Here's the input we used in testing to pull the Python tutorial for a RAG index:

{
  "urls": ["https://docs.python.org/3/tutorial/index.html"],
  "extractMode": "main",
  "outputFormat": "markdown",
  "crawlDepth": 1,
  "maxPages": 50,
  "includeUrlRegex": ["/tutorial/"],
  "chunkSize": 2000,
  "chunkOverlap": 200
}

What the important fields do:

• crawlDepth: 0 converts only the listed URLs. 1 or more also follows links on the same website.
• includeUrlRegex / excludeUrlRegex: keep the crawl inside /tutorial/ or /docs/ and skip changelogs or tag pages.
• useSitemap: seed the crawl from sitemap.xml (found via robots.txt) instead of following links.
• chunkSize / chunkOverlap: split each page into overlapping chunks, measured in characters.
• removeSelectors / keepSelectors: fine-tune extraction, for example remove .cookie-banner or keep only article.
• maxResults: hard stop after N converted pages. This is your cost cap.

Step 3: Run it from code

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("liora-tools/url-to-markdown").call(run_input={
    "urls": ["https://docs.python.org/3/tutorial/index.html"],
    "crawlDepth": 1,
    "includeUrlRegex": ["/tutorial/"],
    "chunkSize": 2000,
    "chunkOverlap": 200,
    "maxResults": 50,
})

docs = []
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    if page["status"] != "ok":
        continue
    for chunk in page.get("chunks") or [{"text": page["content"]}]:
        docs.append({"text": chunk["text"], "source": page["url"], "title": page["title"]})

print(len(docs), "chunks ready for embedding")

From there, pass docs to your embedding model and vector store: LangChain, LlamaIndex, pgvector or anything else. Each chunk keeps its source URL, so your RAG answers can cite pages.

Prefer one HTTP call? Use the sync endpoint:

curl -X POST "https://api.apify.com/v2/acts/liora-tools~url-to-markdown/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://docs.python.org/3/tutorial/errors.html"]}'

Step 4: Read the output

One item per page. This is a real record from our test crawl of the Python tutorial (the content field is shortened here):

{
  "url": "https://docs.python.org/3/tutorial/errors.html",
  "depth": 1,
  "status": "ok",
  "error": null,
  "statusCode": 200,
  "finalUrl": "https://docs.python.org/3/tutorial/errors.html",
  "title": "8. Errors and Exceptions",
  "description": "Until now error messages haven’t been more than mentioned, but if you have tried out the examples you have probably seen some. …",
  "language": "en",
  "publishedDate": "2026-10-21",
  "content": "# 8. Errors and Exceptions\n\nUntil now error messages haven’t been more than mentioned …\n\n## 8.1. Syntax Errors\n\n…",
  "wordCount": 2888,
  "charCount": 22581,
  "chunks": null,
  "links": null,
  "durationMs": 443,
  "scrapedAt": "2026-10-21T16:27:17.018135+00:00"
}

Headings, code blocks and links survive as real Markdown, which LLMs handle well. With chunking on, chunks becomes a list like [{"index": 0, "charStart": 0, "text": "…"}].

The status field tells you what happened on every URL:

• ok: converted.
• needs_javascript: the page is a client-side app shell with no server-rendered text.
• no_text: no readable text found.
• skipped: disallowed by robots.txt.
• error: failed after retries.

After each run, a RUN_SUMMARY record in the key-value store lists counts, failedUrls and a retryInput you can paste back as input to retry only the failures.

Measured results

From our own tests on 2026-10-21:

• Wikipedia "Large language model": 13,058 words, 52 chunks, 1.4 s.
• BBC News front page: 1,298 words, 0.8 s.
• Python tutorial crawl at depth 1: 6 pages in about 4 s total.

Keeping a knowledge base fresh

For a scheduled job, turn on "onlyChanged": true with a stateKey. Pages whose HTML didn't change since the last run with that key are skipped, and skipped pages are free. Put it on an Apify Schedule (say, weekly) and re-embed only what changed.

What it costs

Pay per event: $0.001 per converted page ($1 per 1,000 pages), plus Apify's $0.00005 start fee per run. A 300-page documentation site costs about $0.30. Errors, robots.txt skips, empty pages and JavaScript-only shells aren't charged. maxConsecutiveFailures (default 25) also stops a run early if a site keeps failing.

Honest limits

• No JavaScript rendering. Single-page apps that render everything in the browser come back as needs_javascript. You aren't charged, but you won't get the text either. If such a page embeds its article as schema.org JSON-LD, the Actor uses that. Otherwise, use a browser-based crawler.
• Public pages only. No proxies, stealth browsers, CAPTCHA bypass or logins. robots.txt is respected by default.
• No PDFs or Office files. This Actor handles HTML pages only.
• Main-content extraction is heuristic. On unusual layouts it can keep a sidebar or miss a section. keepSelectors / removeSelectors fix most cases.
• You're responsible for your right to use the content you convert.

Wrap-up

If your LLM app needs web content, cleaning it is the cheapest quality win you'll get: fewer tokens, better retrieval, citable sources. Try it on one page of your own docs first, then add crawlDepth and chunking once the output looks right.

Actor: https://apify.com/liora-tools/url-to-markdown

Disclosure: written with AI assistance and reviewed by the Liora Tools team; I build these tools.

Here's the part many people miss: most of these ATS platforms already publish official public JSON APIs for their job boards. That's how companies embed job listings on their own sites. Greenhouse, Lever, Ashby, Workable, SmartRecruiters and Recruitee all have them. You don't need a headless browser, proxies or a login. You need to know the endpoints and normalize the fields.

This tutorial shows how to do that with an Apify Actor I built, Career Site Jobs: Greenhouse, Lever, Ashby & more. It wraps those public APIs and returns one consistent schema.

Why not call the APIs yourself?

You can, and for one company it's easy. The work shows up when you scale:

• Each ATS names fields differently (location.name vs categories.location vs locationName).
• Salary shows up in a structured field on some boards and only inside the description on others.
• "Remote" can be a boolean, a workplace type or just the word in the location string.
• Tracking new postings means storing what you saw last time.

The Actor handles those parts so you get a flat table you can drop into a spreadsheet, a database or an LLM prompt.

Step 1: Collect your company list

The input accepts three formats, one per line:

• A careers board URL: https://boards.greenhouse.io/stripe, https://jobs.lever.co/palantir, https://jobs.ashbyhq.com/openai
• An explicit ats:slug: greenhouse:stripe, ashby:openai
• A bare slug like stripe. The Actor then tries each ATS automatically.

It also reads Workable (apply.workable.com/...), SmartRecruiters, Recruitee, Personio (acme.jobs.personio.de) and Workday (pass the full myworkdayjobs.com career-site URL).

To find a company's ATS, open its careers page and hover over a job link. The domain usually tells you (boards.greenhouse.io, jobs.lever.co, jobs.ashbyhq.com).

If you just want to try it, there are two preset lists: ai-companies and big-tech-startups.

Step 2: Configure filters

Here's an input that pulls remote engineering roles that mention Python from three boards:

{
  "companies": [
    "https://boards.greenhouse.io/stripe",
    "https://jobs.lever.co/palantir",
    "ashby:openai"
  ],
  "titleKeywords": ["engineer"],
  "excludeTitleKeywords": ["intern"],
  "descriptionKeywords": ["Python"],
  "remoteOnly": true,
  "includeDescription": true,
  "descriptionFormat": "text",
  "maxItems": 100
}

Useful filters from the input schema:

• titleKeywords / excludeTitleKeywords: case-insensitive title matching.
• descriptionKeywords: searches the full job description, not only the title.
• locations, departments, employmentTypes: substring matches.
• workplaceTypes: any of remote, hybrid, onsite.
• seniorityLevels: detected from the title (junior, senior, principal/staff, ...).
• minSalary and onlyWithSalary: work only where a salary is published.
• onlyVisaSponsorship: a rule-based check of the description text.
• postedWithinDays: uses the publish date from the ATS. Not every ATS provides one.
• maxItems / maxJobsPerCompany: hard caps that also cap your cost.

Step 3: Run it

You can run it from the Apify Console (paste the JSON, click Start) or from code. With the Python client:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("liora-tools/ats-career-site-jobs").call(run_input={
    "companies": ["https://boards.greenhouse.io/stripe", "ashby:openai"],
    "titleKeywords": ["engineer"],
    "maxItems": 50,
})

for job in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(job["company"], "|", job["title"], "|", job["location"], "|", job["url"])

Or with plain curl, which returns the dataset items directly:

curl -X POST "https://api.apify.com/v2/acts/liora-tools~ats-career-site-jobs/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"companies": ["huggingface"], "maxItems": 10}'

Step 4: Read the output

Each item uses the same schema no matter which ATS it came from. This record is from one of our test runs against Hugging Face's Workable board (description shortened):

{
  "ats": "workable",
  "companySlug": "huggingface",
  "company": "Hugging Face",
  "jobId": "81B46579FE",
  "title": "Open-Source Machine Learning Engineer - EMEA Remote",
  "department": "Open Source",
  "team": "Engineering",
  "location": "Paris, ÃŽle-de-France, France",
  "offices": ["Paris, ÃŽle-de-France, France"],
  "isRemote": true,
  "workplaceType": "remote",
  "employmentType": "Full-time",
  "publishedAt": "2026-05-28T16:00:00+00:00",
  "url": "https://apply.workable.com/j/81B46579FE",
  "applyUrl": "https://apply.workable.com/j/81B46579FE/apply",
  "compensation": null,
  "descriptionText": "At Hugging Face, we're on a journey to democratize good AI. …",
  "isNew": null,
  "scrapedAt": "2026-10-21T09:36:53.996612+00:00"
}

When a board publishes pay, compensation is filled in. One record from the same test set had {"summary": "$257K – $335K • Offers Equity"}. When there's no salary, it's null. The Actor doesn't guess.

Export the dataset as CSV, JSON or Excel from the Console, or pull it through the API.

Step 5: Turn it into a monitor

This is the feature I use most. Set "onlyNewJobs": true and give the watchlist a stateKey:

{
  "companies": ["greenhouse:stripe", "ashby:openai"],
  "titleKeywords": ["data engineer"],
  "onlyNewJobs": true,
  "stateKey": "data-eng-watchlist"
}

Then add an Apify Schedule (for example, daily at 08:00). Each run returns only postings that weren't there on the last run with the same key. Connect a webhook or an integration (Slack, email, Google Sheets, Zapier, Make) and you have a job alert that only fires on new roles.

Use different stateKey values for different watchlists so they don't overwrite each other.

What it costs

Pricing is pay per event: $0.002 per job saved ($2 per 1,000 jobs), plus Apify's default start fee of $0.00005 per run. A daily monitor on 20 companies that returns a handful of new jobs costs fractions of a cent per day. Set maxItems if you want a hard ceiling.

Honest limits

• Only supported ATS platforms. Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Recruitee, Personio and Workday. iCIMS, Taleo and SuccessFactors aren't supported yet. Companies with a fully custom careers page won't work either.
• Workday is polite by design. It's read through the same public JSON endpoint the career site uses, and only when the site's robots.txt allows it. Disallowed sites are skipped.
• Data is only as good as the board. Salary, publish date and department show up only if the company fills them in. postedWithinDays can't filter on boards with no publish date.
• Seniority and visa sponsorship are rule-based, read from titles and description text. Treat them as a helpful filter, not ground truth.
• Public postings only. No logins, no proxies, no CAPTCHA bypass. Personal data such as emails and phone numbers in descriptions is masked.

Ideas for what to build

• A niche job board ("remote Rust jobs at AI startups") rebuilt every morning.
• Hiring-signal tracking for sales: which target accounts just opened ten engineering roles?
• Salary benchmarking from boards that publish pay ranges.
• Feeding plain-text descriptions into an LLM to tag tech stack or seniority.

If you try it and a board you need isn't covered, open an issue on the Actor page. That's how the support list grows.

Actor: https://apify.com/liora-tools/ats-career-site-jobs

Disclosure: written with AI assistance and reviewed by the Liora Tools team; I build these tools.