Apify Actors / Medium and Substack articles in one dataset
Apify ActorMedium and Substack articles in one dataset
Medium and Substack serve posts in different ways. Medium's public feeds hold the 10 newest stories of an author, publication or tag. Substack's archive API gives anonymous clients about 20 to 25 newest posts, with the rest listed in a sitemap. This Actor reads both and saves every post with the same fields, so a newsletter and a Medium publication fit in one table.
Give it Medium authors, publications or tags, Substack publications or authors, or Substack search terms. Free posts come with the whole text in contentText; paid and member-only posts get isPaywalled: true and the free preview. Substack rows also carry likes, comments and restacks. Duplicates, posts outside your dates and paid posts you choose to skip aren't charged.
$0.002 per article saved · pay per result, no subscription
Price
| Event | Price | Charged when |
|---|---|---|
article | $0.002 | Once per row saved to the dataset, the same for a free post with full text and a paid post with its preview |
| Example run | Units | Cost |
|---|---|---|
| The input below (1 Substack publication, 5 posts) | 5 × $0.002 | $0.01 |
| Empty form (two example sources, 10 each) | up to 20 × $0.002 | up to $0.04 |
| 5 Substack newsletters, last 50 posts each | 250 × $0.002 | $0.50 |
| 20 Medium tags, 10 stories each | 200 × $0.002 | $0.40 |
What you get
- The same fields on both platforms, with
platform,url,title,author,publication,publishedAtandtags. contentTextwith the whole text of free posts, andcontentHtmltoo when "Include HTML" is on.isPaywalledandcontentStatus(full,previewornone), so you can tell a full post from a preview.likes,commentsandrestackson Substack rows, andwordCountandcoverImageon every row.
Input
Set the run options in the Apify console, or send the same object through the API.
{
"substackPublications": [
"astralcodexten"
],
"maxArticlesPerSource": 5
}The five newest posts of one Substack publication. Each source field takes up to 50 entries (Substack search takes up to 20 terms), and maxArticlesPerSource goes from 1 to 1,000. With every source field empty, the run reads two example sources.
Output
{
"platform": "medium",
"id": "6b0811a1c015",
"url": "https://netflixtechblog.com/making-failure-diagnosis-reliable-with-pensive-ai-6b0811a1c015",
"title": "Making Failure Diagnosis Reliable with Pensive AI",
"subtitle": null,
"author": "Netflix Technology Blog",
"authorHandle": null,
"publication": "Netflix TechBlog",
"publishedAt": "2026-10-26T17:04:57.000Z",
"updatedAt": "2026-10-26T17:04:58.324Z",
"tags": ["pensive", "data-engineering", "maestros", "orchestration", "data-science"],
"language": null,
"postType": "story",
"isPaywalled": false,
"contentStatus": "full",
"contentText": "By Praneeth Yenugutala, Victor Dusa, Anastasia Samoilova, Megan Ren, Justin Wei, Marcelo Mayworm\n\nAt Netflix, data and machine learning practitioners run millio...",
"contentHtml": null,
"previewText": null,
"wordCount": 3531,
"likes": null,
"comments": null,
"restacks": null,
"coverImage": "https://cdn-images-1.medium.com/max/1024/1*2TSVpCdPYj4fMMlFPuheCg.png",
"sourceType": "medium-publication",
"sourceInput": "netflix-techblog",
"sourceId": "medium:publication:netflix-techblog",
"scrapedAt": "2026-11-05T15:15:42.560Z"
}One real row from a test run on Apify on 2026-11-05 with the empty input, contentText shortened. On Medium rows likes, comments, restacks and language are null; Substack rows fill them.
Run it from your own code
The Actor takes the same input as a JSON body. This call starts a run, waits for it and returns the dataset items.
curl -X POST "https://api.apify.com/v2/acts/zovo-one~medium-substack-articles/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{"substackPublications": ["astralcodexten"], "maxArticlesPerSource": 5}'A sync call waits at most 300 seconds. For longer runs, start a run with POST /v2/acts/zovo-one~medium-substack-articles/runs, wait for it, then read /v2/datasets/<defaultDatasetId>/items.
Limits
Medium's public feeds hold the 10 newest stories of each author, publication or tag, so Medium sources return at most 10 articles each, there's no Medium keyword search and clap counts aren't available. Substack's archive API gives anonymous clients only its first page; older posts are read through the publication's sitemap.xml, one request per post, so big pulls take longer. Substack search results come by relevance, not date, and each term reads at most 40 result pages. A Substack author is read from their own primary publication, so guest posts elsewhere aren't included. A run sends at most 5,000 requests; for bigger jobs, split the input over several runs.
Questions
How do I scrape a Substack newsletter's back catalogue?
Put the publication in "Substack publications" (a name, name.substack.com or a custom domain) and raise "Max articles per source", up to 1,000. The newest posts come from the archive's first page and older ones from the sitemap, one request each. A "Published to" date far in the past is slow on a big publication, because the sitemap has no dates and every newer post is fetched once and skipped.
What happens with paid and member-only posts?
The Actor never logs in and never gets around a paywall. A paid post gets isPaywalled: true and contentStatus: "preview", and on Substack previewText holds the free part shown above the paywall, cut at 2,000 characters. Turn on "Skip paywalled posts" to leave paid posts out; they aren't saved or charged.
Why does a Medium source stop at 10?
Medium's public feed for an author, publication or tag holds 10 stories. A higher "Max articles per source" still applies to Substack sources, and the run log tells you Medium is capped.
Can I search Substack by keyword?
Yes. Put up to 20 terms in "Substack search", one per line. Results come by relevance, not date, so a date filter skips results outside the range but doesn't stop the search early. Each term reads at most 40 result pages.
What happens when a site blocks the run?
The Actor uses a direct connection by default. If a site refuses it on the Apify platform (403, 429 or a bot check page), the run switches once to Apify Proxy and tries that source again. You can also pick a proxy in the input. When every source is blocked and nothing was saved, the run fails with a message that says so, and you pay nothing per article.
Can I republish the text?
Articles are still their authors' work. The Actor reads public feeds and public JSON endpoints without a login and doesn't save paywalled text. Check the copyright and terms that apply before you republish, and use the data in line with Medium's and Substack's terms.
Support this work
If this Actor saves you time, you can support the work on GitHub Sponsors. Monthly tiers start at $5, and one-time tiers start at $15. Sponsorship is optional, and the tool works the same without it.
Related tools
Built and maintained by Zovo. Actor on Apify · All Actors.