Reducing spam

[adamamyl]

As an administrator of CKAN, I don't want my public site to be polluted with spam. Nor do I want to re-invent the wheel and maintain something, that others are doing.

I would like something like Mollom in Drupal, or Akismet with Wordpress, or StopForumSpam with MediaWiki (and if I'm using one of those pieces of software already, on the same site), to integrate with the existing keys).

I don't believe captchas are useful, and make a very poor user experience.

I would like an extension that, once I've registered and got an API key, I can essentially install, update my config file with the secrets, and it just works.

I might also want to have an aging factor, where new users posting a lot (from an external address) in a short timescale suddenly get rate limited/blocked, until an admin unlocks them.

[davidread]

Absolutely! The prospect of spam is is very much limiting CKAN's ability to take user input all-over the site. The vision of datahub.io being like Wikipedia where anyone can add datasets is curtailed. Simple commenting is outsourced to problematic external services like Disqus. Nearly all government users of CKAN automatically lock-down the site to avoid user input, which could be problematic.

TBH our experience at DGU suggests you need most of the tools in the box, including captchas, Bayesian filters, nofollow, email registration, flagging high activity, buttons for users to flag spam (or other unwanted content) etc. And no automatic method is going to defeat spam farms employing human operators, so you are going to need manual moderation as a back-stop and oversight.

I suggest you need the UI for a moderation queue, plus a list of content that was not moderated (to check for false positives) and another for content that was banned (to check for false negatives). You might also want to differentiate between blatant spam posts and genuine posts that break guidelines - the latter you might want to tell the user and leave a place-holder saying the post was removed.

I would argue that with so many methods to identify spam you need some configurability. e.g. if Mollom is undecided if it is spam, do you show it before you moderate it or not? what about if a user marks it as spam, do you hide it straight away? What about it two different users mark it as spam?

Some of those things are in ckanext-dgu InventoryPlugin feature for adding comments to datasets, such as Mollom integration, basic moderation queue and the "this is spam" button, but needs generalizing and putting into core or a new module. Or maybe it's simpler to just start again.