Underneath the already known

We wanted our Google Analytics charts to be displayed in one of the sections of Inshaker. GA is an excellent service for complex auditorium analysis. Unfortunately, it has no public API (though, I think it will be available in future). Fortunately, it is able to export reports to several formats, including XML.
There are several existing tools for drawing charts on-the fly. Among them are the excellent flash-based amCharts and url-based Google Charts. I chose amCharts because they are more nice-looking and interactive. AmCharts rely on their own XML data files, which have rather straightforward formats.


So the first problem was to download and convert GA reports into amCharts-compatible formats. The second was to be able to do it automatically, so that our charts always reflect actual statistics.
What I came up to was writing a Ruby script to do all the download-and-convert job and launching it with cron each N minutes.

I think the code can speak for itself (since it’s Ruby). Some notes on it: I used the Curb library to download the reports. It’s not mature yet to handle cookies, so I had to do cookie handling myself (fetching them from HTTP response and adding them to request). Why care about cookies? In order to authorize in Google Analytics.
The REXML part is rather simple.
Before using the script make sure to create directories specified in Config module (and provide correct login data and site id)
Grab the script here or continue reading the code:

require 'rubygems'
require 'activesupport'
require 'rexml/document'
require 'unicode'
require 'curb'
require 'cgi'
include REXML
$KCODE = 'u'

# Downloading and conversion of
# Google Analytics reports into amCharts data.xml files

class String
  def merge_i
    self.gsub(" ", "").to_i
  end
end

module Config
  SITE_ID = "9038802" # your site id
  PERIOD = ARGV[0] ? ARGV[0].to_i : 30 # period (days)

  REPORTS_PATH = "reports/" # directory to store downloaded GA reports

  VISITORS_REPORT = "VisitorsOverviewReport"
  CONTENT_REPORT  = "ContentReport"
  GEO_REPORT      = "GeoMapReport"

  OUT_VISITS = "./stat/visitors/data.xml" # amCharts data.xml file (visitors stats)
  OUT_CITIES = "./stat/cities/data.xml" # (cities report xml file)

  EMAIL    = "my_google_analytics@mail.com" # your GA login email
  PASSWORD = "my_password" # your GA password
  PIE_COLORS = ["#a2bcda", "#c1b76f", "#f28358", "#edef00", "#24cbe5", "#64e572"]
end

def process_visits
  path_label = "AnalyticsReport/Report/Graph/Serie/Point/Label"
  path_value = "AnalyticsReport/Report/Graph/Serie/Point/Value"

  vxml = Document.new File.new(Config::REPORTS_PATH + Config::VISITORS_REPORT + ".xml")
  dates    = vxml.elements.to_a(path_label).map {|d| d = d.text}
  visitors = vxml.elements.to_a(path_value).map {|v| v = v.text.merge_i}

  cxml = Document.new File.new(Config::REPORTS_PATH + Config::CONTENT_REPORT + ".xml")
  views = cxml.elements.to_a(path_value).map {|v| v = v.text.merge_i}

  data_file = Document.new

  chart = Element.new("chart")
  data_file.add chart

  # X-Axis - dates
  series = Element.new("series")
  dates.each_with_index { |date, i|
    value = Element.new("value")
    value.attributes["xid"] = i.to_s
    value.text = date
    series.add value
  }
  chart.add_element series

  graphs = Element.new("graphs")
  chart.add_element graphs

  # Y-Axis - numbers
  [views, visitors].each_with_index { |set, i|
    graph = Element.new("graph")
    graph.attributes["gid"] = (i+1).to_s
    set.each_with_index { |num, j|
      value = Element.new("value")
      value.attributes["xid"] = j.to_s
      value.text = num
      graph.add value
    }
    graphs.add graph
  }

  # Output
  File.open(Config::OUT_VISITS, "w+") {|charts| data_file.write(charts) }
  puts "Built visits/visitors chart..."
end

def process_cities
  path_name  = "AnalyticsReport/Report/GeoMap/Region[position()<7]/Name"
  path_value = "AnalyticsReport/Report/GeoMap/Region[position()<7]/Value"

  cxml = Document.new File.new(Config::REPORTS_PATH + Config::GEO_REPORT + ".xml")
  cities = cxml.elements.to_a(path_name).map {|c| c = c.text}
  visitors =  cxml.elements.to_a(path_value).map {|v| v = v.text.merge_i}

  data_file = Document.new
  pie = Element.new("pie")

  data_file.add pie

  cities.each_with_index {|city, i|
    slice = Element.new("slice")
    slice.attributes["title"] = city
    if Config::PIE_COLORS[i] then slice.attributes["color"] = Config::PIE_COLORS[i] end
    slice.text = visitors[i].to_s
    pie.add slice
  }

  # Output
  File.open(Config::OUT_CITIES, "w+") {|pie| data_file.write(pie) }
  puts "Built cities chart..."
end

def get_cookies(header_str)
  cookies = {}
  headers = header_str.split("\n")
  headers.each {|h|
    if h =~ /Set-Cookie: (.+)/
      arr = $1.split("=")
      name = arr[0]
      cookies[name] = arr[1].split(";")[0]
    end
  }
  cstr = ""
  cookies.each {|name, val| cstr += name + "=" + CGI.escape(val) + ";" }
  cstr
end

def download_reports
  statuses = {} # statuses of downloading of reports (true - success, false - failed)

  dates = [Config::PERIOD.days.ago, Time.now].map {|d| d.strftime("%Y%m%d")}

  auth_url   = "https://www.google.com/accounts/ServiceLoginBoxAuth"
  report_pfx = "https://www.google.com/analytics/reporting/export?fmt=1&id=#{Config::SITE_ID}&pdr=#{dates[0]}-#{dates[1]}&segkey=city&cmp=average&&rpt="

  serv  = Curl::PostField.content("service", "analytics")
  hl    = Curl::PostField.content("hl", "ru-RU")
  email = Curl::PostField.content("Email", Config::EMAIL)
  passw = Curl::PostField.content("Passwd", Config::PASSWORD)

  # Authorization in Google Analytics
  c = Curl::Easy.new(auth_url)
  c.http_post(serv, hl, email, passw)
  cstr = get_cookies(c.header_str)

  # Downloading of reports
  [Config::VISITORS_REPORT, Config::CONTENT_REPORT, Config::GEO_REPORT].each { |rpt|
    c = Curl::Easy.new(report_pfx + rpt)
    c.headers["Cookie"] = cstr
    c.http_get
    if c.body_str =~ /<\/AnalyticsReport>/
      puts "Downloaded "+ rpt + "..."
      File.open(Config::REPORTS_PATH + rpt + ".xml", "w+") {|out| out.write(c.body_str) }
      statuses[rpt] = true
    else
      puts "Failed to download " + rpt + "..."
      statuses[rpt] = false
    end
  }
  statuses
end

dates = [Config::PERIOD.days.ago, Time.now].map {|d| d.strftime("%d %B %Y")}
puts "Trying to download reports from #{dates[0]} to #{dates[1]}"
statuses = download_reports
if(statuses[Config::VISITORS_REPORT] && statuses[Config::CONTENT_REPORT]) then process_visits end
if(statuses[Config::GEO_REPORT]) then process_cities end

The crontab (crontab -e) is like this

# m h  dom mon dow   command
0,30 * * * * /www/mysite/update_stats.sh

update_stats.sh executes analytic.rb and does some other work.