Join us

Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives

In observing Perplexity's stealth crawling behavior, it was noted that they obscure their crawling identity in an attempt to circumvent website preferences by modifying user agents and changing ASNs. Despite being blocked, Perplexity's crawlers continued to try to access restricted content, using undisclosed IPs and user agents.


Let's keep in touch!

Stay updated with my latest posts and news. I share insights, updates, and exclusive content.

By subscribing, you share your email with @faun and accept our Terms & Privacy. Unsubscribe anytime.

Give a Pawfive to this post!


Only registered users can post comments. Please, login or signup.

Start blogging about your favorite technologies, reach more readers and earn rewards!

Join other developers and claim your FAUN.dev account now!

Avatar

The FAUN

@faun
A worldwide community of developers and DevOps enthusiasts!
Developer Influence
3k

Influence

302k

Total Hits

1

Posts