Merge pull request #986 from jmerle/update-checker

Automatically check for documentation updates

.github/CONTRIBUTING.md

@@ -61,7 +61,7 @@ In addition to the [guidelines for contributing code](#contributing-code-and-fea
 
 Please don't submit a pull request updating the version number of a documentation, unless a change is required in the scraper and you've verified that it works.
 
-To ask that an existing documentation be updated, please use the [Trello board](https://trello.com/c/2B0hmW7M/52-request-updates-here).
+To ask that an existing documentation be updated, first check the last two [Documentation versions reports](https://github.com/freeCodeCamp/devdocs/issues?utf8=%E2%9C%93&q=Documentation+versions+report+is%3Aissue+author%3Adevdocs-bot+sort%3Aupdated-desc). Only create an issue if the documentation has been wrongly marked as up-to-date.
 
 ## Coding conventions
 

.travis.yml

@@ -6,6 +6,10 @@ before_script:
   - gem update --system
   - gem install bundler
 
+script:
+  - if [ "$TRAVIS_EVENT_TYPE" != "cron" ]; then bundle exec rake; fi
+  - if [ "$TRAVIS_EVENT_TYPE" = "cron" ]; then bundle exec thor updates:check --github-token $GH_TOKEN --upload; fi
+
 deploy:
   provider: heroku
   app: devdocs

Gemfile

@@ -42,6 +42,7 @@ group :docs do
   gem 'unix_utils', require: false
   gem 'tty-pager', require: false
   gem 'net-sftp', '>= 2.1.3.rc2', require: false
+  gem 'terminal-table', require: false
 end
 
 group :test do

Gemfile.lock

@@ -104,6 +104,8 @@ GEM
       unicode-display_width (~> 1.4.0)
       unicode_utils (~> 1.4.0)
     strings-ansi (0.1.0)
+    terminal-table (1.8.0)
+      unicode-display_width (~> 1.1, >= 1.1.1)
     thin (1.7.2)
       daemons (~> 1.0, >= 1.0.9)
       eventmachine (~> 1.0, >= 1.0.4)
@@ -157,6 +159,7 @@ DEPENDENCIES
   sinatra-contrib
   sprockets
   sprockets-helpers
+  terminal-table
   sprockets-sass
   thin
   thor

docs/adding-docs.md

@@ -17,6 +17,7 @@ Adding a documentation may look like a daunting task but once you get the hang o
 10. To add syntax highlighting or execute custom JavaScript on the pages, create a file in the `assets/javascripts/views/pages/` directory (take a look at the other files to see how it works).
 11. Add the documentation's icon in the `public/icons/docs/[my_doc]/` directory, in both 16x16 and 32x32-pixels formats. It'll be added to the icon spritesheet after your pull request is merged.
 12. Add the documentation's copyright details to the list in `assets/javascripts/templates/pages/about_tmpl.coffee`. This is the data shown in the table on the [about](https://devdocs.io/about) page, and is ordered alphabetically. Simply copying an existing item, placing it in the right slot and updating the values to match the new scraper will do the job.
+13. Ensure `thor updates:check [my_doc]` shows the correct latest version.
 
 If the documentation includes more than a few hundreds pages and is available for download, try to scrape it locally (e.g. using `FileScraper`). It'll make the development process much faster and avoids putting too much load on the source site. (It's not a problem if your scraper is coupled to your local setup, just explain how it works in your pull request.)
 

docs/scraper-reference.md

@@ -184,3 +184,44 @@ More information about how filters work is available on the [Filter Reference](.
     Overrides the `:title` option for the root page only.
 
   _Note: this filter is disabled by default._
+
+## Keeping scrapers up-to-date
+
+In order to keep scrapers up-to-date the `get_latest_version(opts)` method should be overridden. If `self.release` is defined, this should return the latest version of the documentation. If `self.release` is not defined, it should return the Epoch time when the documentation was last modified. If the documentation will never change, simply return `1.0.0`. The result of this method is periodically reported in a "Documentation versions report" issue which helps maintainers keep track of outdated documentations.
+
+To make life easier, there are a few utility methods that you can use in `get_latest_version`:
+* `fetch(url, opts)`
+
+  Makes a GET request to the url and returns the response body.
+
+  Example: [lib/docs/scrapers/bash.rb](../lib/docs/scrapers/bash.rb)
+* `fetch_doc(url, opts)`
+
+  Makes a GET request to the url and returns the HTML body converted to a Nokogiri document.
+
+  Example: [lib/docs/scrapers/git.rb](../lib/docs/scrapers/git.rb)
+* `fetch_json(url, opts)`
+
+  Makes a GET request to the url and returns the JSON body converted to a dictionary.
+
+  Example: [lib/docs/scrapers/mdn/mdn.rb](../lib/docs/scrapers/mdn/mdn.rb)
+* `get_npm_version(package, opts)`
+
+  Returns the latest version of the given npm package.
+
+  Example: [lib/docs/scrapers/bower.rb](../lib/docs/scrapers/bower.rb)
+* `get_latest_github_release(owner, repo, opts)`
+
+  Returns the tag name of the latest GitHub release of the given repository. If the tag name is preceded by a "v", the "v" will be removed.
+
+  Example: [lib/docs/scrapers/jsdoc.rb](../lib/docs/scrapers/jsdoc.rb)
+* `get_github_tags(owner, repo, opts)`
+
+  Returns the list of tags on the given repository ([format](https://developer.github.com/v3/repos/#list-tags)).
+
+  Example: [lib/docs/scrapers/liquid.rb](../lib/docs/scrapers/liquid.rb)
+* `get_github_file_contents(owner, repo, path, opts)`
+
+  Returns the contents of the requested file in the default branch of the given repository.
+
+  Example: [lib/docs/scrapers/minitest.rb](../lib/docs/scrapers/minitest.rb)

lib/docs/core/doc.rb

@@ -152,7 +152,6 @@ module Docs
       end
     end
 
-
     def initialize
       raise NotImplementedError, "#{self.class} is an abstract class and cannot be instantiated." if self.class.abstract
     end
@@ -164,5 +163,104 @@ module Docs
     def build_pages(&block)
       raise NotImplementedError
     end
+
+    def get_scraper_version(opts)
+      if self.class.method_defined?(:options) and !options[:release].nil?
+        options[:release]
+      else
+        # If options[:release] does not exist, we return the Epoch timestamp of when the doc was last modified in DevDocs production
+        json = fetch_json('https://devdocs.io/docs.json', opts)
+        items = json.select {|item| item['name'] == self.class.name}
+        items = items.map {|item| item['mtime']}
+        items.max
+      end
+    end
+
+    # Should return the latest version of this documentation
+    # If options[:release] is defined, it should be in the same format
+    # If options[:release] is not defined, it should return the Epoch timestamp of when the documentation was last updated
+    # If the docs will never change, simply return '1.0.0'
+    def get_latest_version(opts)
+      raise NotImplementedError
+    end
+
+    # Returns whether or not this scraper is outdated.
+    #
+    # The default implementation assumes the documentation uses a semver(-like) approach when it comes to versions.
+    # Patch updates are ignored because there are usually little to no documentation changes in bug-fix-only releases.
+    #
+    # Scrapers of documentations that do not use this versioning approach should override this method.
+    #
+    # Examples of the default implementation:
+    # 1 -> 2 = outdated
+    # 1.1 -> 1.2 = outdated
+    # 1.1.1 -> 1.1.2 = not outdated
+    def is_outdated(scraper_version, latest_version)
+      scraper_parts = scraper_version.to_s.split(/\./).map(&:to_i)
+      latest_parts = latest_version.to_s.split(/\./).map(&:to_i)
+
+      # Only check the first two parts, the third part is for patch updates
+      [0, 1].each do |i|
+        break if i >= scraper_parts.length or i >= latest_parts.length
+        return true if latest_parts[i] > scraper_parts[i]
+        return false if latest_parts[i] < scraper_parts[i]
+      end
+
+      false
+    end
+
+    private
+
+    #
+    # Utility methods for get_latest_version
+    #
+
+    def fetch(url, opts)
+      headers = {}
+
+      if opts.key?(:github_token) and url.start_with?('https://api.github.com/')
+        headers['Authorization'] = "token #{opts[:github_token]}"
+      end
+
+      opts[:logger].debug("Fetching #{url}")
+      response = Request.run(url, { connecttimeout: 15, headers: headers })
+
+      if response.success?
+        response.body
+      else
+        reason = response.timed_out? ? "Timed out while connecting to #{url}" : "Couldn't fetch #{url} (response code #{response.code})"
+        opts[:logger].error(reason)
+        raise reason
+      end
+    end
+
+    def fetch_doc(url, opts)
+      body = fetch(url, opts)
+      Nokogiri::HTML.parse(body, nil, 'UTF-8')
+    end
+
+    def fetch_json(url, opts)
+      JSON.parse fetch(url, opts)
+    end
+
+    def get_npm_version(package, opts)
+      json = fetch_json("https://registry.npmjs.com/#{package}", opts)
+      json['dist-tags']['latest']
+    end
+
+    def get_latest_github_release(owner, repo, opts)
+      release = fetch_json("https://api.github.com/repos/#{owner}/#{repo}/releases/latest", opts)
+      tag_name = release['tag_name']
+      tag_name.start_with?('v') ? tag_name[1..-1] : tag_name
+    end
+
+    def get_github_tags(owner, repo, opts)
+      fetch_json("https://api.github.com/repos/#{owner}/#{repo}/tags", opts)
+    end
+
+    def get_github_file_contents(owner, repo, path, opts)
+      json = fetch_json("https://api.github.com/repos/#{owner}/#{repo}/contents/#{path}", opts)
+      Base64.decode64(json['content'])
+    end
   end
 end

lib/docs/scrapers/angular.rb

@@ -155,6 +155,10 @@ module Docs
       end
     end
 
+    def get_latest_version(opts)
+      get_npm_version('@angular/core', opts)
+    end
+
     private
 
     def parse(response)

lib/docs/scrapers/angularjs.rb

@@ -69,5 +69,9 @@ module Docs
       self.release = '1.2.32'
       self.base_url = "https://code.angularjs.org/#{release}/docs/partials/"
     end
+
+    def get_latest_version(opts)
+      get_npm_version('angular', opts)
+    end
   end
 end

lib/docs/scrapers/ansible.rb

@@ -87,5 +87,10 @@ module Docs
         quickstart.html
         list_of_all_modules.html)
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://docs.ansible.com/ansible/latest/index.html', opts)
+      doc.at_css('.DocSiteProduct-CurrentVersion').content.strip
+    end
   end
 end

lib/docs/scrapers/apache.rb

@@ -33,5 +33,10 @@ module Docs
       &copy; 2018 The Apache Software Foundation<br>
       Licensed under the Apache License, Version 2.0.
     HTML
+
+    def get_latest_version(opts)
+      doc = fetch_doc('http://httpd.apache.org/docs/', opts)
+      doc.at_css('#apcontents > ul a')['href'][0...-1]
+    end
   end
 end

lib/docs/scrapers/apache_pig.rb

@@ -43,5 +43,10 @@ module Docs
       self.base_url = "https://pig.apache.org/docs/r#{release}/"
     end
 
+    def get_latest_version(opts)
+      doc = fetch_doc('https://pig.apache.org/', opts)
+      item = doc.at_css('div[id="menu_1.2"] > .menuitem:last-child')
+      item.content.strip.sub(/Release /, '')
+    end
   end
 end

lib/docs/scrapers/async.rb

@@ -17,5 +17,10 @@ module Docs
       &copy; 2010&ndash;2018 Caolan McMahon<br>
       Licensed under the MIT License.
     HTML
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://caolan.github.io/async/v3/', opts)
+      doc.at_css('#version-dropdown > a').content.strip[1..-1]
+    end
   end
 end

lib/docs/scrapers/babel.rb

@@ -22,5 +22,10 @@ module Docs
     stub '' do
       '<div></div>'
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://babeljs.io/docs/en/', opts)
+      doc.at_css('a[href="/versions"] > h3').content
+    end
   end
 end

lib/docs/scrapers/backbone.rb

@@ -20,5 +20,10 @@ module Docs
       &copy; 2010&ndash;2016 Jeremy Ashkenas, DocumentCloud<br>
       Licensed under the MIT License.
     HTML
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://backbonejs.org/', opts)
+      doc.at_css('.version').content[1...-1]
+    end
   end
 end

lib/docs/scrapers/bash.rb

@@ -17,5 +17,10 @@ module Docs
       Copyright &copy; 2000, 2001, 2002, 2007, 2008 Free Software Foundation, Inc.<br>
       Licensed under the GNU Free Documentation License.
     HTML
+
+    def get_latest_version(opts)
+      body = fetch('https://www.gnu.org/software/bash/manual/html_node/index.html', opts)
+      body.scan(/, Version ([0-9.]+)/)[0][0][0...-1]
+    end
   end
 end

lib/docs/scrapers/bluebird.rb

@@ -18,5 +18,9 @@ module Docs
       &copy; 2013&ndash;2017 Petka Antonov<br>
       Licensed under the MIT License.
     HTML
+
+    def get_latest_version(opts)
+      get_npm_version('bluebird', opts)
+    end
   end
 end

lib/docs/scrapers/bootstrap.rb

@@ -34,5 +34,10 @@ module Docs
 
       options[:only] = %w(getting-started/ css/ components/ javascript/)
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://getbootstrap.com/', opts)
+      doc.at_css('#bd-versions').content.strip[1..-1]
+    end
   end
 end

lib/docs/scrapers/bottle.rb

@@ -27,5 +27,11 @@ module Docs
       self.release = '0.11.7'
       self.base_url = "https://bottlepy.org/docs/#{self.version}/"
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://bottlepy.org/docs/stable/', opts)
+      label = doc.at_css('.sphinxsidebarwrapper > ul > li > b')
+      label.content.sub(/Bottle /, '')
+    end
   end
 end

lib/docs/scrapers/bower.rb

@@ -19,5 +19,9 @@ module Docs
       &copy; 2018 Bower contributors<br>
       Licensed under the MIT License.
     HTML
+
+    def get_latest_version(opts)
+      get_npm_version('bower', opts)
+    end
   end
 end

lib/docs/scrapers/c.rb

@@ -26,6 +26,13 @@ module Docs
       Licensed under the Creative Commons Attribution-ShareAlike Unported License v3.0.
     HTML
 
+    def get_latest_version(opts)
+      doc = fetch_doc('https://en.cppreference.com/w/Cppreference:Archives', opts)
+      link = doc.at_css('a[title^="File:"]')
+      date = link.content.scan(/(\d+)\./)[0][0]
+      DateTime.strptime(date, '%Y%m%d').to_time.to_i
+    end
+
     private
 
     def file_path_for(*)

lib/docs/scrapers/cakephp.rb

@@ -71,6 +71,11 @@ module Docs
       self.base_url = 'https://api.cakephp.org/2.7/'
     end
 
+    def get_latest_version(opts)
+      doc = fetch_doc('https://api.cakephp.org/3.7/', opts)
+      doc.at_css('.version-picker .dropdown-toggle').content.strip
+    end
+
     private
 
     def parse(response)

lib/docs/scrapers/chai.rb

@@ -23,5 +23,9 @@ module Docs
       &copy; 2016 Chai.js Assertion Library<br>
       Licensed under the MIT License.
     HTML
+
+    def get_latest_version(opts)
+      get_npm_version('chai', opts)
+    end
   end
 end

lib/docs/scrapers/chef.rb

@@ -47,5 +47,10 @@ module Docs
 
       options[:only_patterns] = [/\A#{client_path}\//, /\A#{server_path}\//]
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://downloads.chef.io/chef', opts)
+      doc.at_css('h1.product-heading > span').content.strip
+    end
   end
 end

lib/docs/scrapers/clojure.rb

@@ -27,5 +27,10 @@ module Docs
       self.release = '1.7'
       self.base_url = 'https://clojure.github.io/clojure/branch-clojure-1.7.0/'
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('http://clojure.github.io/clojure/index.html', opts)
+      doc.at_css('#header-version').content[1..-1]
+    end
   end
 end

lib/docs/scrapers/cmake.rb

@@ -59,5 +59,11 @@ module Docs
       self.release = '3.5.2'
       self.base_url = 'https://cmake.org/cmake/help/v3.5/'
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://cmake.org/documentation/', opts)
+      link = doc.at_css('.entry-content ul > li > strong > a > big')
+      link.content.scan(/([0-9.]+)/)[0][0]
+    end
   end
 end

lib/docs/scrapers/codeception.rb

@@ -18,5 +18,10 @@ module Docs
       &copy; 2011 Michael Bodnarchuk and contributors<br>
       Licensed under the MIT License.
     HTML
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://codeception.com/changelog', opts)
+      doc.at_css('#page > h4').content
+    end
   end
 end

lib/docs/scrapers/codeceptjs.rb

@@ -21,5 +21,9 @@ module Docs
       &copy; 2015 DavertMik &lt;davert@codegyre.com&gt; (http://codegyre.com)<br>
       Licensed under the MIT License.
     HTML
+
+    def get_latest_version(opts)
+      get_npm_version('codeceptjs', opts)
+    end
   end
 end

lib/docs/scrapers/codeigniter.rb

@@ -38,5 +38,11 @@ module Docs
     version '3' do
       self.release = '3.1.8'
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://codeigniter.com/user_guide/changelog.html', opts)
+      header = doc.at_css('#change-log h2')
+      header.content.scan(/([0-9.]+)/)[0][0]
+    end
   end
 end

lib/docs/scrapers/coffeescript.rb

@@ -30,5 +30,9 @@ module Docs
 
       options[:container] = '.container'
     end
+
+    def get_latest_version(opts)
+      get_npm_version('coffeescript', opts)
+    end
   end
 end

lib/docs/scrapers/cordova.rb

@@ -42,5 +42,15 @@ module Docs
       self.release = '6.5.0'
       self.base_url = 'https://cordova.apache.org/docs/en/6.x/'
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://cordova.apache.org/docs/en/latest/', opts)
+
+      label = doc.at_css('#versionDropdown').content.strip
+      version = label.scan(/([0-9.]+)/)[0][0]
+      version = version[0...-1] if version.end_with?('.')
+
+      version
+    end
   end
 end

lib/docs/scrapers/cpp.rb

@@ -34,6 +34,14 @@ module Docs
       Licensed under the Creative Commons Attribution-ShareAlike Unported License v3.0.
     HTML
 
+    # Same as get_latest_version in lib/docs/scrapers/c.rb
+    def get_latest_version(opts)
+      doc = fetch_doc('https://en.cppreference.com/w/Cppreference:Archives', opts)
+      link = doc.at_css('a[title^="File:"]')
+      date = link.content.scan(/(\d+)\./)[0][0]
+      DateTime.strptime(date, '%Y%m%d').to_time.to_i
+    end
+
     private
 
     def file_path_for(*)

lib/docs/scrapers/crystal.rb

@@ -34,5 +34,10 @@ module Docs
         HTML
       end
     }
+
+    def get_latest_version(opts)
+      body = fetch('https://crystal-lang.org/api', opts)
+      body.scan(/Crystal Docs ([0-9.]+)/)[0][0]
+    end
   end
 end

lib/docs/scrapers/d.rb

@@ -26,5 +26,10 @@ module Docs
     def initial_urls
       %w(https://dlang.org/phobos/index.html https://dlang.org/spec/intro.html)
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://dlang.org/changelog/', opts)
+      doc.at_css('#content > ul > li:nth-child(2) > a')['id']
+    end
   end
 end

lib/docs/scrapers/d3.rb

@@ -58,5 +58,9 @@ module Docs
       options[:root_title] = 'D3.js'
       options[:only_patterns] = [/\.md\z/]
     end
+
+    def get_latest_version(opts)
+      get_npm_version('d3', opts)
+    end
   end
 end

lib/docs/scrapers/dart.rb

@@ -31,5 +31,11 @@ module Docs
       self.release = '1.24.3'
       self.base_url = "https://api.dartlang.org/stable/#{release}/"
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://api.dartlang.org/', opts)
+      label = doc.at_css('footer > span').content.strip
+      label.sub(/Dart /, '')
+    end
   end
 end

lib/docs/scrapers/django.rb

@@ -63,5 +63,10 @@ module Docs
       self.release = '1.8.18'
       self.base_url = 'https://docs.djangoproject.com/en/1.8/'
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://docs.djangoproject.com/', opts)
+      doc.at_css('#doc-versions > li.current > span > strong').content
+    end
   end
 end

lib/docs/scrapers/docker.rb

@@ -137,5 +137,11 @@ module Docs
       options[:container] = '#docs'
       options[:only_patterns] << /\Aswarm\//
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://docs.docker.com/', opts)
+      label = doc.at_css('.nav-container button.dropdown-toggle').content.strip
+      label.scan(/([0-9.]+)/)[0][0]
+    end
   end
 end

lib/docs/scrapers/dojo.rb

@@ -36,6 +36,11 @@ module Docs
       urls.map { |url| "<a href='#{url}'>#{url}</a>" }.join
     end
 
+    def get_latest_version(opts)
+      doc = fetch_doc('https://dojotoolkit.org/api/', opts)
+      doc.at_css('#versionSelector > option[selected]').content
+    end
+
     private
 
     def get_url_list(json, set = Set.new)

lib/docs/scrapers/drupal.rb

@@ -98,5 +98,10 @@ module Docs
         /\A[\w\-\.]+\.php\/7\.x\z/
       ]
     end
+
+    def get_latest_version(opts)
+      json = fetch_json('https://packagist.org/packages/drupal/drupal.json', opts)
+      json['package']['versions'].keys.find {|version| !version.end_with?('-dev')}
+    end
   end
 end

lib/docs/scrapers/electron.rb

@@ -22,5 +22,10 @@ module Docs
       &copy; 2013&ndash;2018 GitHub Inc.<br>
       Licensed under the MIT license.
     HTML
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://electronjs.org/docs', opts)
+      doc.at_css('.docs-version').content
+    end
   end
 end

lib/docs/scrapers/elixir.rb

@@ -97,5 +97,10 @@ module Docs
         'https://elixir-lang.org/getting-started/'
       ]
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://hexdocs.pm/elixir/api-reference.html', opts)
+      doc.at_css('h2.sidebar-projectVersion').content.strip[1..-1]
+    end
   end
 end

lib/docs/scrapers/ember.rb

@@ -56,5 +56,10 @@ module Docs
         https://emberjs.com/api/ember-data/2.14/classes/DS
       )
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://emberjs.com/api/ember/release', opts)
+      doc.at_css('.sidebar > .select-container .ember-power-select-selected-item').content.strip
+    end
   end
 end

lib/docs/scrapers/erlang.rb

@@ -55,5 +55,10 @@ module Docs
     version '18' do
       self.release = '18.3'
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://www.erlang.org/downloads', opts)
+      doc.at_css('.col-lg-3 > ul > li').content.strip.sub(/OTP /, '')
+    end
   end
 end

lib/docs/scrapers/eslint.rb

@@ -20,5 +20,9 @@ module Docs
       &copy; JS Foundation and other contributors<br>
       Licensed under the MIT License.
     HTML
+
+    def get_latest_version(opts)
+      get_npm_version('eslint', opts)
+    end
   end
 end

lib/docs/scrapers/express.rb

@@ -28,5 +28,9 @@ module Docs
       &copy; 2017 StrongLoop, IBM, and other expressjs.com contributors.<br>
       Licensed under the Creative Commons Attribution-ShareAlike License v3.0.
     HTML
+
+    def get_latest_version(opts)
+      get_npm_version('express', opts)
+    end
   end
 end

lib/docs/scrapers/falcon.rb

@@ -33,5 +33,10 @@ module Docs
       self.release = '1.2.0'
       self.base_url = "https://falcon.readthedocs.io/en/#{self.release}/"
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://falcon.readthedocs.io/en/stable/changes/index.html', opts)
+      doc.at_css('#changelogs ul > li > a').content
+    end
   end
 end

lib/docs/scrapers/fish.rb

@@ -46,5 +46,10 @@ module Docs
       self.release = '2.2.0'
       self.base_url = "https://fishshell.com/docs/#{version}/"
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('http://fishshell.com/docs/current/index.html', opts)
+      doc.at_css('#toc-index').content.scan(/([0-9.]+)/)[0][0]
+    end
   end
 end

lib/docs/scrapers/flow.rb

@@ -18,5 +18,9 @@ module Docs
       &copy; 2013&ndash;present Facebook Inc.<br>
       Licensed under the MIT License.
     HTML
+
+    def get_latest_version(opts)
+      get_npm_version('flow-bin', opts)
+    end
   end
 end

lib/docs/scrapers/git.rb

@@ -19,5 +19,10 @@ module Docs
       &copy; 2005&ndash;2018 Linus Torvalds and others<br>
       Licensed under the GNU General Public License version 2.
     HTML
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://git-scm.com/', opts)
+      doc.at_css('.version').content.strip
+    end
   end
 end

lib/docs/scrapers/gnu/gcc.rb

@@ -99,5 +99,11 @@ module Docs
 
       options[:replace_paths] = CPP_PATHS
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://gcc.gnu.org/onlinedocs/', opts)
+      label = doc.at_css('ul > li > ul > li > a').content.strip
+      label.scan(/([0-9.]+)/)[0][0]
+    end
   end
 end

lib/docs/scrapers/gnu/gnu_fortran.rb

@@ -25,5 +25,11 @@ module Docs
       self.release = '4.9.3'
       self.base_url = "https://gcc.gnu.org/onlinedocs/gcc-#{release}/gfortran/"
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://gcc.gnu.org/onlinedocs/', opts)
+      label = doc.at_css('ul > li > ul > li > a').content.strip
+      label.scan(/([0-9.]+)/)[0][0]
+    end
   end
 end

lib/docs/scrapers/go.rb

@@ -24,6 +24,11 @@ module Docs
       Licensed under the Creative Commons Attribution License 3.0.
     HTML
 
+    def get_latest_version(opts)
+      doc = fetch_doc('https://golang.org/project/', opts)
+      doc.at_css('#page ul > li > a').text[3..-1]
+    end
+
     private
 
     def parse(response) # Hook here because Nokogori removes whitespace from textareas

lib/docs/scrapers/godot.rb

@@ -37,5 +37,10 @@ module Docs
       self.release = '2.1'
       self.base_url = "http://docs.godotengine.org/en/#{self.version}/"
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://docs.godotengine.org/', opts)
+      doc.at_css('.version').content.strip
+    end
   end
 end

lib/docs/scrapers/graphite.rb

@@ -17,5 +17,10 @@ module Docs
       &copy; 2011&ndash;2016 The Graphite Project<br>
       Licensed under the Apache License, Version 2.0.
     HTML
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://graphite.readthedocs.io/en/latest/releases.html', opts)
+      doc.at_css('#release-notes li > a').content
+    end
   end
 end

lib/docs/scrapers/grunt.rb

@@ -26,5 +26,9 @@ module Docs
       &copy; GruntJS Team<br>
       Licensed under the MIT License.
     HTML
+
+    def get_latest_version(opts)
+      get_npm_version('grunt-cli', opts)
+    end
   end
 end

lib/docs/scrapers/handlebars.rb

@@ -19,5 +19,9 @@ module Docs
       &copy; 2011&ndash;2017 by Yehuda Katz<br>
       Licensed under the MIT License.
     HTML
+
+    def get_latest_version(opts)
+      get_npm_version('handlebars', opts)
+    end
   end
 end

lib/docs/scrapers/haskell.rb

@@ -10,7 +10,7 @@ module Docs
 
     html_filters.push 'haskell/entries', 'haskell/clean_html'
 
-    options[:container] = ->(filter) { filter.subpath.start_with?('users_guide') ? '.body' : '#content' }
+    options[:container] = ->(filter) {filter.subpath.start_with?('users_guide') ? '.body' : '#content'}
 
     options[:only_patterns] = [/\Alibraries\//, /\Ausers_guide\//]
     options[:skip_patterns] = [
@@ -68,5 +68,12 @@ module Docs
 
       options[:only_patterns] = [/\Alibraries\//]
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://downloads.haskell.org/~ghc/latest/docs/html/', opts)
+      links = doc.css('a').to_a
+      versions = links.map {|link| link['href'].scan(/ghc-([0-9.]+)/)}
+      versions.find {|version| !version.empty?}[0][0]
+    end
   end
 end

lib/docs/scrapers/haxe.rb

@@ -66,5 +66,11 @@ module Docs
     version 'Python' do
       self.base_url = 'https://api.haxe.org/python/'
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://api.haxe.org/', opts)
+      label = doc.at_css('.container.main-content h1 > small').content
+      label.sub(/version /, '')
+    end
   end
 end

lib/docs/scrapers/homebrew.rb

@@ -19,5 +19,9 @@ module Docs
       &copy; 2009&ndash;present Homebrew contributors<br>
       Licensed under the BSD 2-Clause License.
     HTML
+
+    def get_latest_version(opts)
+      get_latest_github_release('Homebrew', 'brew', opts)
+    end
   end
 end

lib/docs/scrapers/http.rb

@@ -7,6 +7,8 @@ module Docs
 
     html_filters.push 'http/clean_html', 'http/entries', 'title'
 
+    options[:mdn_tag] = 'HTTP'
+
     options[:root_title] = 'HTTP'
     options[:title] = ->(filter) { filter.current_url.host == 'tools.ietf.org' ? false : filter.default_title }
     options[:container] = ->(filter) { filter.current_url.host == 'tools.ietf.org' ? '.content' : nil }

lib/docs/scrapers/immutable.rb

@@ -54,5 +54,9 @@ module Docs
       JS
       capybara.html
     end
+
+    def get_latest_version(opts)
+      get_npm_version('immutable', opts)
+    end
   end
 end

lib/docs/scrapers/influxdata.rb

@@ -46,5 +46,11 @@ module Docs
       &copy; 2015 InfluxData, Inc.<br>
       Licensed under the MIT license.
     HTML
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://docs.influxdata.com/influxdb/', opts)
+      label = doc.at_css('.navbar--current-product').content.strip
+      label.scan(/([0-9.]+)/)[0][0]
+    end
   end
 end

lib/docs/scrapers/jasmine.rb

@@ -17,5 +17,9 @@ module Docs
       &copy; 2008&ndash;2017 Pivotal Labs<br>
       Licensed under the MIT License.
     HTML
+
+    def get_latest_version(opts)
+      get_latest_github_release('jasmine', 'jasmine', opts)
+    end
   end
 end

lib/docs/scrapers/jekyll.rb

@@ -28,5 +28,10 @@ module Docs
       &copy; 2008&ndash;2018 Tom Preston-Werner and Jekyll contributors<br>
       Licensed under the MIT license.
     HTML
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://jekyllrb.com/docs/', opts)
+      doc.at_css('.meta a').content[1..-1]
+    end
   end
 end

lib/docs/scrapers/jest.rb

@@ -17,5 +17,10 @@ module Docs
       &copy; 2014&ndash;present Facebook Inc.<br>
       Licensed under the BSD License.
     HTML
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://jestjs.io/docs/en/getting-started', opts)
+      doc.at_css('header > a > h3').content
+    end
   end
 end

lib/docs/scrapers/jquery/jquery_core.rb

@@ -22,5 +22,9 @@ module Docs
       /Selectors\/odd/i,
       /index/i
     ]
+
+    def get_latest_version(opts)
+      get_npm_version('jquery', opts)
+    end
   end
 end

lib/docs/scrapers/jquery/jquery_mobile.rb

@@ -16,5 +16,10 @@ module Docs
     options[:fix_urls] = ->(url) do
       url.sub! 'http://api.jquerymobile.com/', 'https://api.jquerymobile.com/'
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://jquerymobile.com/', opts)
+      doc.at_css('.download-box > .download-option:last-child > span').content.sub(/Version /, '')
+    end
   end
 end

lib/docs/scrapers/jquery/jquery_ui.rb

@@ -15,5 +15,9 @@ module Docs
     options[:fix_urls] = ->(url) do
       url.sub! 'http://api.jqueryui.com/', 'https://api.jqueryui.com/'
     end
+
+    def get_latest_version(opts)
+      get_npm_version('jquery-ui', opts)
+    end
   end
 end

lib/docs/scrapers/jsdoc.rb

@@ -21,5 +21,9 @@ module Docs
       &copy; 2011&ndash;2017 the contributors to the JSDoc 3 documentation project<br>
       Licensed under the Creative Commons Attribution-ShareAlike Unported License v3.0.
     HTML
+
+    def get_latest_version(opts)
+      get_latest_github_release('jsdoc3', 'jsdoc', opts)
+    end
   end
 end

lib/docs/scrapers/julia.rb

@@ -49,5 +49,9 @@ module Docs
 
       html_filters.push 'julia/entries_sphinx', 'julia/clean_html_sphinx', 'sphinx/clean_html'
     end
+
+    def get_latest_version(opts)
+      get_latest_github_release('JuliaLang', 'julia', opts)
+    end
   end
 end

lib/docs/scrapers/knockout.rb

@@ -33,5 +33,9 @@ module Docs
       &copy; Steven Sanderson, the Knockout.js team, and other contributors<br>
       Licensed under the MIT License.
     HTML
+
+    def get_latest_version(opts)
+      get_latest_github_release('knockout', 'knockout', opts)
+    end
   end
 end

lib/docs/scrapers/koa.rb

@@ -34,5 +34,9 @@ module Docs
       &copy; 2018 Koa contributors<br>
       Licensed under the MIT License.
     HTML
+
+    def get_latest_version(opts)
+      get_npm_version('koa', opts)
+    end
   end
 end

lib/docs/scrapers/kotlin.rb

@@ -28,5 +28,9 @@ module Docs
       &copy; 2010&ndash;2018 JetBrains s.r.o.<br>
       Licensed under the Apache License, Version 2.0.
     HTML
+
+    def get_latest_version(opts)
+      get_latest_github_release('JetBrains', 'kotlin', opts)
+    end
   end
 end

lib/docs/scrapers/laravel.rb

@@ -133,5 +133,9 @@ module Docs
         url
       end
     end
+
+    def get_latest_version(opts)
+      get_latest_github_release('laravel', 'laravel', opts)
+    end
   end
 end

lib/docs/scrapers/leaflet.rb

@@ -39,5 +39,10 @@ module Docs
       self.base_url = "https://leafletjs.com/reference-#{release}.html"
     end
 
+    def get_latest_version(opts)
+      doc = fetch_doc('https://leafletjs.com/index.html', opts)
+      link = doc.css('ul > li > a').to_a.select {|node| node.content == 'Docs'}.first
+      link['href'].scan(/reference-([0-9.]+)\.html/)[0][0]
+    end
   end
 end

lib/docs/scrapers/less.rb

@@ -21,5 +21,11 @@ module Docs
       &copy; 2009&ndash;2016 The Core Less Team<br>
       Licensed under the Creative Commons Attribution License 3.0.
     HTML
+
+    def get_latest_version(opts)
+      doc = fetch_doc('http://lesscss.org/features/', opts)
+      label = doc.at_css('.footer-links > li').content
+      label.scan(/([0-9.]+)/)[0][0]
+    end
   end
 end

lib/docs/scrapers/liquid.rb

@@ -19,5 +19,10 @@ module Docs
       &copy; 2005, 2006 Tobias Luetke<br>
       Licensed under the MIT License.
     HTML
+
+    def get_latest_version(opts)
+      tags = get_github_tags('Shopify', 'liquid', opts)
+      tags[0]['name'][1..-1]
+    end
   end
 end

lib/docs/scrapers/lodash.rb

@@ -32,5 +32,10 @@ module Docs
       self.release = '2.4.2'
       self.base_url = "https://lodash.com/docs/#{release}"
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://lodash.com/docs/', opts)
+      doc.at_css('#version > option[selected]').content
+    end
   end
 end

lib/docs/scrapers/love.rb

@@ -39,5 +39,10 @@ module Docs
       &copy; 2006&ndash;2016 L&Ouml;VE Development Team<br>
       Licensed under the GNU Free Documentation License, Version 1.3.
     HTML
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://love2d.org/wiki/Version_History', opts)
+      doc.at_css('#mw-content-text table a').content
+    end
   end
 end

lib/docs/scrapers/lua.rb

@@ -26,5 +26,10 @@ module Docs
       self.release = '5.1.5'
       self.base_url = 'https://www.lua.org/manual/5.1/'
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://www.lua.org/manual/', opts)
+      doc.at_css('p.menubar > a').content
+    end
   end
 end

lib/docs/scrapers/marionette.rb

@@ -38,5 +38,9 @@ module Docs
 
       html_filters.push 'marionette/entries_v2'
     end
+
+    def get_latest_version(opts)
+      get_npm_version('backbone.marionette', opts)
+    end
   end
 end

lib/docs/scrapers/markdown.rb

@@ -13,5 +13,9 @@ module Docs
       &copy; 2004 John Gruber<br>
       Licensed under the BSD License.
     HTML
+
+    def get_latest_version(opts)
+      '1.0.0'
+    end
   end
 end

lib/docs/scrapers/matplotlib.rb

@@ -64,5 +64,9 @@ module Docs
         "https://matplotlib.org/#{release}/mpl_toolkits/axes_grid/api/"
       ]
     end
+
+    def get_latest_version(opts)
+      get_latest_github_release('matplotlib', 'matplotlib', opts)
+    end
   end
 end

lib/docs/scrapers/mdn/css.rb

@@ -6,6 +6,8 @@ module Docs
 
     html_filters.push 'css/clean_html', 'css/entries', 'title'
 
+    options[:mdn_tag] = 'CSS'
+
     options[:root_title] = 'CSS'
 
     options[:skip] = %w(/CSS3 /Media/Visual /paged_media /Media/TV /Media/Tactile)

lib/docs/scrapers/mdn/dom.rb

@@ -8,6 +8,8 @@ module Docs
 
     html_filters.push 'dom/clean_html', 'dom/entries', 'title'
 
+    options[:mdn_tag] = 'XSLT_Reference'
+
     options[:root_title] = 'DOM'
 
     options[:skip] = %w(

lib/docs/scrapers/mdn/dom_events.rb

@@ -9,6 +9,8 @@ module Docs
     html_filters.insert_after 'clean_html', 'dom_events/clean_html'
     html_filters.push 'dom_events/entries', 'title'
 
+    options[:mdn_tag] = 'events'
+
     options[:root_title] = 'DOM Events'
 
     options[:skip] = %w(/MozOrientation)

lib/docs/scrapers/mdn/html.rb

@@ -7,6 +7,8 @@ module Docs
 
     html_filters.push 'html/clean_html', 'html/entries', 'title'
 
+    options[:mdn_tag] = 'HTML'
+
     options[:root_title] = 'HTML'
 
     options[:title] = ->(filter) do

lib/docs/scrapers/mdn/javascript.rb

@@ -8,6 +8,8 @@ module Docs
 
     html_filters.push 'javascript/clean_html', 'javascript/entries', 'title'
 
+    options[:mdn_tag] = 'JavaScript'
+
     options[:root_title] = 'JavaScript'
 
     # Don't want

lib/docs/scrapers/mdn/mdn.rb

@@ -21,6 +21,11 @@ module Docs
       Licensed under the Creative Commons Attribution-ShareAlike License v2.5 or later.
     HTML
 
+    def get_latest_version(opts)
+      json = fetch_json("https://developer.mozilla.org/en-US/docs/feeds/json/tag/#{options[:mdn_tag]}", opts)
+      DateTime.parse(json[0]['pubdate']).to_time.to_i
+    end
+
     private
 
     def process_response?(response)

lib/docs/scrapers/mdn/svg.rb

@@ -8,6 +8,8 @@ module Docs
 
     html_filters.push 'svg/clean_html', 'svg/entries', 'title'
 
+    options[:mdn_tag] = 'XSLT_Reference'
+
     options[:root_title] = 'SVG'
 
     options[:title] = ->(filter) do

lib/docs/scrapers/mdn/xslt_xpath.rb

@@ -8,6 +8,8 @@ module Docs
 
     html_filters.push 'xslt_xpath/clean_html', 'xslt_xpath/entries', 'title'
 
+    options[:mdn_tag] = 'XSLT_Reference'
+
     options[:root_title] = 'XSLT'
 
     options[:only_patterns] = [/\A\/XSLT/, /\A\/XPath/]

lib/docs/scrapers/meteor.rb

@@ -45,5 +45,10 @@ module Docs
       self.base_urls = ['https://guide.meteor.com/v1.3/', "https://docs.meteor.com/v#{self.release}/"]
       options[:fix_urls] = nil
     end
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://docs.meteor.com/#/full/', opts)
+      doc.at_css('select.version-select > option').content
+    end
   end
 end

lib/docs/scrapers/mocha.rb

@@ -18,5 +18,9 @@ module Docs
       &copy; 2011&ndash;2018 JS Foundation and contributors<br>
       Licensed under the Creative Commons Attribution 4.0 International License.
     HTML
+
+    def get_latest_version(opts)
+      get_npm_version('mocha', opts)
+    end
   end
 end

lib/docs/scrapers/modernizr.rb

@@ -15,5 +15,9 @@ module Docs
       &copy; 2009&ndash;2017 The Modernizr team<br>
       Licensed under the MIT License.
     HTML
+
+    def get_latest_version(opts)
+      get_npm_version('modernizr', opts)
+    end
   end
 end

lib/docs/scrapers/moment.rb

@@ -22,5 +22,10 @@ module Docs
       &copy; JS Foundation and other contributors<br>
       Licensed under the MIT License.
     HTML
+
+    def get_latest_version(opts)
+      doc = fetch_doc('http://momentjs.com/', opts)
+      doc.at_css('.hero-title > h1 > span').content
+    end
   end
 end

lib/docs/scrapers/mongoose.rb

@@ -26,5 +26,11 @@ module Docs
       &copy; 2010 LearnBoost<br>
       Licensed under the MIT License.
     HTML
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://mongoosejs.com/docs/', opts)
+      label = doc.at_css('.pure-menu-link').content.strip
+      label.sub(/Version /, '')
+    end
   end
 end

lib/docs/scrapers/nginx.rb

@@ -25,5 +25,11 @@ module Docs
       &copy; 2011-2018 Nginx, Inc.<br>
       Licensed under the BSD License.
     HTML
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://nginx.org/en/download.html', opts)
+      table = doc.at_css('#content > table').inner_html
+      table.scan(/nginx-([0-9.]+)</)[0][0]
+    end
   end
 end

lib/docs/scrapers/nginx_lua_module.rb

@@ -4,6 +4,9 @@ module Docs
     self.slug = 'nginx_lua_module'
     self.release = '0.10.13'
     self.base_url = "https://github.com/openresty/lua-nginx-module/tree/v#{self.release}/"
+    self.links = {
+      code: 'https://github.com/openresty/lua-nginx-module'
+    }
 
     html_filters.push 'nginx_lua_module/clean_html', 'nginx_lua_module/entries', 'title'
 
@@ -15,5 +18,11 @@ module Docs
       &copy; 2009&ndash;2018 Yichun "agentzh" Zhang (章亦春), OpenResty Inc.<br>
       Licensed under the BSD License.
     HTML
+
+    def get_latest_version(opts)
+      tags = get_github_tags('openresty', 'lua-nginx-module', opts)
+      tag = tags.find {|tag| !tag['name'].include?('rc')}
+      tag['name'][1..-1]
+    end
   end
 end

lib/docs/scrapers/nim.rb

@@ -17,5 +17,10 @@ module Docs
       &copy; 2006&ndash;2018 Andreas Rumpf<br>
       Licensed under the MIT License.
     HTML
+
+    def get_latest_version(opts)
+      doc = fetch_doc('https://nim-lang.org/docs/overview.html', opts)
+      doc.at_css('.container > .docinfo > tbody > tr:last-child > td').content.strip
+    end
   end
 end