I’m not quite sure what’s going on there exactly, but I block JS in my browser (via NoScript). When I downloaded the link you provided with wget in the terminal, it returns what looks like a bot block page to me. (It includes the text “This question is for testing whether you are a human visitor and to prevent automated spam submission.” with an embedded CAPTCHA image.) If I load the link in Firefox though, it provides a PDF even with JS disabled in my browser. Usually that means a site is doing something like User-Agent sniffing or running a cookie check to block automated scrapers, but if I download the link with wget again after loading it once in my browser, it provides the PDF directly – so presumably the site has some middleware that allows requests by IP after you’ve passed an initial not-bot approval? (Maybe time limited? Haven’t experimented to find out.)
You might be able to get around this by setting User-Agent and other headers in an initial request to impersonate the browser? (Check copy as cURL for the URL in FF’s network dev tools to see how to emulate the request exactly as your browser would do it.)
I just did a slightly more reliable test: disabled js in the FF settings (about:config→javascript.enabled=false). And the PDF was fetched and rendered with some delay.
It is clear from wget that the server is at least willing to push html,js,garbage that masquerades as a “pdf”. But I believe you are correct; that w/this sample URL, Mozilla is getting a true PDF based on some opaque judgment by the server.
So that URL turns out to be a bad example. I see this all the time though. I will have to start collecting more PDF URLs that push shenanigans until one reproduces Mozilla’s anti-user behavior.
I’m not quite sure what’s going on there exactly, but I block JS in my browser (via NoScript). When I downloaded the link you provided with wget in the terminal, it returns what looks like a bot block page to me. (It includes the text “This question is for testing whether you are a human visitor and to prevent automated spam submission.” with an embedded CAPTCHA image.) If I load the link in Firefox though, it provides a PDF even with JS disabled in my browser. Usually that means a site is doing something like User-Agent sniffing or running a cookie check to block automated scrapers, but if I download the link with wget again after loading it once in my browser, it provides the PDF directly – so presumably the site has some middleware that allows requests by IP after you’ve passed an initial not-bot approval? (Maybe time limited? Haven’t experimented to find out.)
You might be able to get around this by setting User-Agent and other headers in an initial request to impersonate the browser? (Check copy as cURL for the URL in FF’s network dev tools to see how to emulate the request exactly as your browser would do it.)
I just did a slightly more reliable test: disabled js in the FF settings (
about:config→javascript.enabled=false). And the PDF was fetched and rendered with some delay.It is clear from wget that the server is at least willing to push html,js,garbage that masquerades as a “pdf”. But I believe you are correct; that w/this sample URL, Mozilla is getting a true PDF based on some opaque judgment by the server.
So that URL turns out to be a bad example. I see this all the time though. I will have to start collecting more PDF URLs that push shenanigans until one reproduces Mozilla’s anti-user behavior.