newSummBot
The crawler behind NewSumm — what it reads, how to control it, and how to have your publication removed.
Last updated 31 August 2026
How to identify it
Every request we make carries this User-Agent header. We use one stable agent, we never present as a browser, and we never vary the agent to get around a block.
newSummBot/1.0 (+https://newsumm.com/bot)What newSummBot does
NewSumm is a news aggregator. newSummBot collects article metadata from publishers so that related coverage can be grouped into a single story and summarised, with every article credited and linked back to the publisher's own site.
-
Reads RSS and Atom feeds only. The bot requests the feed URL a publisher has chosen to publish. It does not crawl your site, follow links into article pages, or fetch anything you have not put in a feed.
-
It does not bypass paywalls, logins or metering. It has no credentials and makes no attempt to obtain any.
-
Fetches once an hour at most. One request per feed, and only for sources that are active in our index.
-
Stores only feed-level metadata. The headline, article URL, the description or content your feed itself supplies, the author, the publication date and the URL of the feed's image.
-
It does not copy your images. We store the image URL and load it from your servers, so removing or changing an image on your side changes it on ours.
-
Article text supplied in a feed is deleted after 7 days. It is used to generate a summary and then cleared automatically.
What we publish
Our pages show a short AI-generated summary of a group of related articles, alongside the headline, the publication name and a link to each original article. We do not republish full articles, and we do not present publisher content as our own. The same content appears in the NewSumm mobile apps for iOS and Android, under the same rules described on this page.
Summaries are generated automatically and can contain errors. They are commentary on the coverage, not a substitute for it — every story points readers back to the original reporting.
Controlling newSummBot with robots.txt
We check your robots.txt
before every feed fetch and honour it. Address us with the token
newSummBot.
Block us from your entire site
User-agent: newSummBot
Disallow: /Block one section, leave the rest available
User-agent: newSummBot
Disallow: /premium/robots.txt for up to 24 hours.
A rule that blocks all crawlers (User-agent: *)
applies to us too, unless a newSummBot
group overrides it. If a robots.txt
is missing or unreachable, we treat that as no restrictions, which is the standard convention.
We read robots.txt
as a machine-readable reservation of rights, including for text and data mining. If you would rather not
rely on it, use the removal form below and we will act on it directly.
Removing your publication
Publishers can have their sources taken out of NewSumm themselves, without waiting on us, using the source removal form.
-
1
Search for your domain and select the sources that belong to you.
-
2
Confirm with an email address on that same domain.
-
3
Click the link we email you — the request is only acted on once you do, and it expires after 7 days.
The email-on-the-same-domain requirement is there to stop a third party from removing someone else's publication. If you cannot use an address on the domain, or the form does not find your source, email support@newsumm.com and we will handle it manually.
Removal stops future fetching and takes existing articles from that source out of our index.
Copyright and personal data
Copyright
Headlines, article text and images remain the property of the publisher. We rely on short extracts, attribution and links back to the source. If you believe material on NewSumm exceeds that, contact support@newsumm.com with the URL and we will remove it. See our Terms of Service for more.
Personal data
News articles can contain personal data, such as an author's name. We process it to run a news aggregation service and to keep attribution accurate. If you are named in content on NewSumm and want it corrected or removed, write to support@newsumm.com. Our Privacy Policy covers how we handle data about readers of the site and app.
Reporting a problem with the bot
If newSummBot is requesting a feed too often, ignoring a rule you have set, or reaching something it should not, tell us at support@newsumm.com. Include the feed URL and, if you have them, log lines showing the requests. Traffic identifying itself as newSummBot from anywhere other than our systems is not ours — we would like to hear about that too.
Want your sources removed?
Verify with an email address on your domain and we will stop fetching your feeds and remove your articles from the index.