> the publishers never gave permission to use their content
Yes they did, yes they did. The publisher explicitly granted access by putting up the news content in an public subdirectory of a public webserver. News items don't just magically `pop up' in the public subdirectory of a webserver. Usually at least two employees (writer, reviewer) have to log into, and explicitly perform several steps in an CMS to publish a news item.
The decision of publisher is encoded as HTTP server configuration (including CMS, if any). The server obviously provides access only to some resources, as the operator wishes -- the ones meant for public consumption, while other resources are password-protected, only accessible from certain network addresses, or not accessible over HTTP at all.
Just as much as an image resource can be, and often is, protected from deep-linking based on HTTP `Referer' header, a news item can be protected from being indexed by Google News bot and displayed as a snipped in Google News based on `User-Agent' header.
In more details:
A typical User Agent that HTTP GETs some resources from the server describes itself with a `User-Agent' header. Google is honest here, and openly indicates its user agent as a spider bot, with explicit purpose of indexing content for web search.
Even better, Google uses a /separate/ spider bot for Google News service, and it is entirely reasonable to serve the general websearch bot with content -- so the website itself remains indexed -- and deny the news bot access, to prevent Google News from displaying news snippets.
Upon HTTP request, the server reads through request, including resource URL, headers (including User-Agent and Cookies, if any), compares that info to configuration. If, and only if, it matches configuration for unrestricted access, the resource is served. Otherwise, access denied condition is indicated and optionally authentication requested.
Let's assume for a moment the law goes through as requested by the press. The most probable implementation would be, Google includes a particular header in HTTP requests -- indicating itself as a `news-bot-that-has-subscription', and the server would, in turn, compare the header with configuration and either reply with news contents, or perhaps indicate access denied, should subscription expire, or for some other reason. The mechanics is pretty much same as it would be today to restrict Google News bot from gathering, and Google News from displaying (some or all) news snippets.
In short, the publishers have already configured some resources as protected from /some/ User Agents (based on request URL, headers, cookies, network address or anything else), and it is entirely their own choice they serve the news to both Google bots (websearch and news).
That's a strawman; the bots never indicated intention to read. They indicated (via `User-Agent' header) intention to index content and got HTTP 200 OK response to that.
Nothing in the HTTP protocol specs (RFC 2616) even suggests it is only meant for direct human consumption. To the contrary, it states,
user agent
The client which initiates a request. These are often browsers,
editors, spiders (web-traversing robots), or other end user tools.
...and the Google bots clearly identify themselves as indexing bots.
Please don't be obtuse. The whole matter is not about Google doing something strange or different than any other search engine. It's just about collective bargaining of one business group against another business. Some german press companies stood up as a (self-appointed and thus somewhat suspect) representation of whole german press industry and are trying to get a bigger slice of the cake.
If I don't lock my house, that doesn't give you the right to break in. You're absolutely right that publishers could make more use of the tools at their disposal to protect their content in whatever way they wish, but they have rights regardless of whether they do.
This analogy only muddles the picture. We are talking about private property, but not a home, more like a shop or pub.
Its only raison d'etre is to serve certain guests. We are talking about a commercial server that is managed by a professional admin, and is explicitly open to only some of the public, and it already screens and filters guests, based on various criteria. For example, the server may disallow hotlinking of images, serve some content only upon receipt of a HTTP cookie confirming payment and serve different content for mobile devices, some movies blocked in certain countries, based on IP geolocation. The decision is based on HTTP request data and will of the owner, encoded as server's configuration. Ask yourself, how much different is Google News bot?
How about this analogy instead: ``if I hire a professional bouncer for my high-street shop, and the bouncer stops some people from entering and lets others in, that doesn't mean those let in can indeed walk in''?
Which would be crazy, and press would have a field day with such a shop.
Some access control is really easy to implement, other types are not, or they're not cost-effective. It's perfectly reasonable for a content provider to combine technological measures (like e.g. IP geolocation) with legal ones (terms of use, copyright).
That said, you're absolutely right: how hard can it be to block the Google News bot for a news organization? Not very. My point is merely that you can't draw any legal or moral conclusions from a 200 OK response, that's absurd.
To riff on your example, if my high-street shop has a sign that says "only people over 18" but doesn't have a bouncer, does that mean people under 18 should feel free to enter because nobody's physically stopping them?
"That's a strawman; the bots never indicated intention to read. They indicated (via `User-Agent' header) intention to index content and got HTTP 200 OK response to that."
The publishers aren't objecting to being indexed. They are objecting because snippets of their articles are being copied, placed on Google News, and used to sell ads without their permission, and they're not being compensated.
Yes they did, yes they did. The publisher explicitly granted access by putting up the news content in an public subdirectory of a public webserver. News items don't just magically `pop up' in the public subdirectory of a webserver. Usually at least two employees (writer, reviewer) have to log into, and explicitly perform several steps in an CMS to publish a news item.
The decision of publisher is encoded as HTTP server configuration (including CMS, if any). The server obviously provides access only to some resources, as the operator wishes -- the ones meant for public consumption, while other resources are password-protected, only accessible from certain network addresses, or not accessible over HTTP at all.
Just as much as an image resource can be, and often is, protected from deep-linking based on HTTP `Referer' header, a news item can be protected from being indexed by Google News bot and displayed as a snipped in Google News based on `User-Agent' header.
In more details:
A typical User Agent that HTTP GETs some resources from the server describes itself with a `User-Agent' header. Google is honest here, and openly indicates its user agent as a spider bot, with explicit purpose of indexing content for web search.
Even better, Google uses a /separate/ spider bot for Google News service, and it is entirely reasonable to serve the general websearch bot with content -- so the website itself remains indexed -- and deny the news bot access, to prevent Google News from displaying news snippets.
Upon HTTP request, the server reads through request, including resource URL, headers (including User-Agent and Cookies, if any), compares that info to configuration. If, and only if, it matches configuration for unrestricted access, the resource is served. Otherwise, access denied condition is indicated and optionally authentication requested.
Let's assume for a moment the law goes through as requested by the press. The most probable implementation would be, Google includes a particular header in HTTP requests -- indicating itself as a `news-bot-that-has-subscription', and the server would, in turn, compare the header with configuration and either reply with news contents, or perhaps indicate access denied, should subscription expire, or for some other reason. The mechanics is pretty much same as it would be today to restrict Google News bot from gathering, and Google News from displaying (some or all) news snippets.
In short, the publishers have already configured some resources as protected from /some/ User Agents (based on request URL, headers, cookies, network address or anything else), and it is entirely their own choice they serve the news to both Google bots (websearch and news).