Google Search Console — OAuth Client, Callback URI & Pulling Query Data

⚠️ Cookbook example — not live code. (KEEP THIS CALLOUT.) Every code block below is an example snippet, not part of the llamapress.ai codebase, and not running on this server. This is a reference recipe for a Leo instance (an AI coding agent) to implement in its own app — read it to understand the pattern, then recreate it there.

Your app needs real Google Search data — the actual search terms people typed to find the site, with impressions, clicks, CTR, and average position, per page and per day, up to 16 months back. That data lives in Google Search Console (GSC), and this recipe is the complete integration: creating the OAuth client ID (Application type: Web application) in Google Cloud, setting the Authorized redirect URI (the callback your app handles), storing the credentials in .env, minting a durable refresh token, and a zero-gem Net::HTTP client that queries the Search Analytics API. This pattern is proven in production (it powers a weekly SEO loop).

When to use: an SEO dashboard, a "which queries rank just off page 1" report, feeding real query data into content planning, showing a client their search performance inside the app you built them. When not to: you only need to look at the data occasionally — the Search Console web UI (search.google.com/search-console) is free and needs no code. Also not for Google Analytics (different API, different scope entirely).

The 80/20 in one breath

In Google Cloud Console (console.cloud.google.com): pick/create a project → APIs & Services → Library → enable the Google Search Console API → configure the OAuth consent screen (External, then Publish to production — this is load-bearing, see Gotchas) → Credentials → Create Credentials → OAuth client ID → Application type Web application → under Authorized redirect URIs add your callback, e.g. https://yourapp.example.com/gsc_auth/callback.
Put the client's credentials in .env as GOOGLE_CLIENT_ID / GOOGLE_CLIENT_SECRET, then recreate the web container (docker compose up -d --force-recreate web) — a plain restart does NOT reload .env.
Verify the site as a property in Search Console (Domain property via a DNS TXT record is best: it covers all subdomains and both http/https, and its site URL is sc-domain:example.com).
Add the controller + routes below and visit /gsc_auth/new. Sign in with a Google account that has access to the property. The callback exchanges the code and shows you the refresh token — put it in .env as GSC_REFRESH_TOKEN.
Copy GscSearchAnalytics into app/services/gsc_search_analytics.rb and run: GscSearchAnalytics.new.top_queries — real query data comes back.

Layer 0 — Google Cloud setup (the part outside your codebase)

This happens in the browser, once, by a human (or by you walking the human through it):

Project: console.cloud.google.com → select or create a project.
Enable the API: APIs & Services → Library → search "Google Search Console API" → Enable. (Skipping this yields a 403 later even with perfect OAuth.)
Consent screen: APIs & Services → OAuth consent screen. User type External (unless everyone signing in is on your own Google Workspace). Add the .../auth/webmasters.readonly scope. Then Publish to production. A consent screen left in "Testing" expires every refresh token after 7 days — the #1 cause of "it worked last week."
Create the OAuth client ID: APIs & Services → Credentials → Create Credentials → OAuth client ID:
- Application type: Web application ("For use with requests from a web server" — this is the right type for a Rails app; not Desktop, not iOS/Android).
- Name: anything (e.g. MyApp Web Client) — only shown in the console, never to end users. If your app runs on multiple platforms, each platform needs its own client ID.
- Authorized redirect URIs: add the EXACT callback URL your app will handle:
```
https://yourapp.example.com/gsc_auth/callback
```
  Add http://localhost:3000/gsc_auth/callback too if you'll test locally. Google matches this string exactly — scheme, host, port, path, no trailing slash difference allowed. A mismatch fails with Error 400: redirect_uri_mismatch.
Copy the Client ID and Client secret into .env:

# .env
GOOGLE_CLIENT_ID=1234567890-abc123.apps.googleusercontent.com
GOOGLE_CLIENT_SECRET=GOCSPX-xxxxxxxxxxxxxxxx
# Filled in after the connect flow (Layer 1):
GSC_REFRESH_TOKEN=
# The Search Console property (domain property syntax shown):
GSC_SITE_URL=sc-domain:example.com

Then recreate the container so the new .env loads (docker compose up -d --force-recreate web — restart alone keeps the old environment).

Verify the property in Search Console (search.google.com/search-console → Add property). Prefer a Domain property: Google gives you a google-site-verification=... TXT record to add at the domain apex in your DNS. Domain properties cover every subdomain and both http/https, and their API site URL is sc-domain:example.com. A URL-prefix property (https://example.com/) also works — its API site URL is the full prefix URL.

Layer 1 — Controller: the OAuth connect flow (mints the refresh token)

One-time flow: start at /gsc_auth/new, Google redirects back to the callback with a code, the callback exchanges the code for tokens and displays the refresh token for you to put in .env. Only the refresh token is durable — access tokens die in about an hour and the client below re-mints them on demand.

# app/controllers/gsc_auth_controller.rb
require 'net/http'
require 'json'
require 'securerandom'

class GscAuthController < ApplicationController
  before_action :authenticate_user!
  before_action :ensure_admin

  SCOPE     = 'https://www.googleapis.com/auth/webmasters.readonly'.freeze
  AUTH_URL  = 'https://accounts.google.com/o/oauth2/v2/auth'.freeze
  TOKEN_URL = 'https://oauth2.googleapis.com/token'.freeze

  # GET /gsc_auth/new — start the OAuth flow
  def new
    state = SecureRandom.hex(24)
    session[:gsc_oauth_state] = state

    query = {
      client_id: ENV.fetch('GOOGLE_CLIENT_ID'),
      redirect_uri: gsc_auth_callback_url,   # must EXACTLY match an Authorized redirect URI
      response_type: 'code',
      scope: SCOPE,
      state: state,
      access_type: 'offline',   # request a refresh token
      prompt: 'consent'         # force consent so the refresh token is actually returned
    }
    redirect_to "#{AUTH_URL}?#{query.to_query}", allow_other_host: true
  end

  # GET /gsc_auth/callback — Google redirects here with ?code=...&state=...
  def callback
    if params[:state].blank? || params[:state] != session.delete(:gsc_oauth_state)
      return render plain: "Invalid state parameter — start over at /gsc_auth/new", status: :forbidden
    end
    if params[:error].present?
      return render plain: "Google authorization failed: #{params[:error]}", status: :bad_request
    end

    resp = Net::HTTP.post_form(URI(TOKEN_URL), {
      'client_id' => ENV.fetch('GOOGLE_CLIENT_ID'),
      'client_secret' => ENV.fetch('GOOGLE_CLIENT_SECRET'),
      'code' => params[:code],
      'grant_type' => 'authorization_code',
      'redirect_uri' => gsc_auth_callback_url   # must match the one used in #new
    })
    data = JSON.parse(resp.body)
    unless resp.is_a?(Net::HTTPSuccess)
      return render plain: "Token exchange failed: #{data['error_description'] || data['error']}", status: :bad_request
    end

    # Show the refresh token ONCE for the operator to copy into .env.
    # (Alternative: persist it encrypted on a settings/organization model instead.)
    render plain: <<~MSG
      Connected. Add this line to .env, then recreate the web container:

      GSC_REFRESH_TOKEN=#{data['refresh_token'] || '(no refresh_token returned — see the guide gotchas: re-run with prompt=consent)'}
    MSG
  end

  private

  def ensure_admin
    redirect_to root_path, alert: "Admin access required." unless current_user.admin?
  end
end

# config/routes.rb (add inside the draw block)
namespace :gsc_auth do
  get :new
  get :callback
end

If you add a "Connect Search Console" button instead of typing the URL, the form/link MUST carry data: { turbo: false } — otherwise Turbo intercepts the submit and the cross-origin redirect to accounts.google.com dies on CORS with no useful error.

No-code alternative — the OAuth Playground. If you don't want the controller at all (single-tenant app, one-time mint), add https://developers.google.com/oauthplayground as an extra Authorized redirect URI, open the Playground, click the gear → "Use your own OAuth credentials" → paste your client ID/secret, put EXACTLY https://www.googleapis.com/auth/webmasters.readonly (only the URL, nothing else) in the "Input your own scopes" box, Authorize APIs, sign in, then "Exchange authorization code for tokens" and copy the refresh token. You can remove the Playground redirect URI afterward. (Playground-specific traps are in the Gotchas.)

Layer 2 — The client (zero gems, Net::HTTP)

Token refresh + the Search Analytics query endpoint + the three reports that matter.

# app/services/gsc_search_analytics.rb
require 'net/http'
require 'json'
require 'erb'

# Read-only Google Search Console (Search Analytics API) client.
# Zero gems: refresh-token exchange and the REST endpoint via Net::HTTP.
class GscSearchAnalytics
  SCOPE     = 'https://www.googleapis.com/auth/webmasters.readonly'.freeze
  TOKEN_URL = 'https://oauth2.googleapis.com/token'.freeze
  API_BASE  = 'https://searchconsole.googleapis.com/webmasters/v3'.freeze

  class ConfigError < StandardError; end
  class ApiError < StandardError; end

  def initialize(site: ENV['GSC_SITE_URL'], refresh_token: ENV['GSC_REFRESH_TOKEN'])
    @site = site.presence or raise ConfigError, "GSC_SITE_URL is not set (e.g. sc-domain:example.com)"
    @refresh_token = refresh_token.presence or raise ConfigError, "GSC_REFRESH_TOKEN is not set — run the connect flow first"
  end

  # Sanity check: lists the properties this token can see. Your site must appear.
  def sites
    get("/sites").fetch('siteEntry', [])
  end

  # Raw Search Analytics query. Each row:
  # { "keys" => [..per dimension..], "clicks", "impressions", "ctr", "position" }
  def query(dimensions:, start_date:, end_date:, row_limit: 1000, dimension_filter_groups: nil)
    body = {
      startDate: start_date.to_s,
      endDate:   end_date.to_s,
      dimensions: Array(dimensions),
      rowLimit: row_limit
    }
    body[:dimensionFilterGroups] = dimension_filter_groups if dimension_filter_groups
    post("/sites/#{ERB::Util.url_encode(@site)}/searchAnalytics/query", body)['rows'] || []
  end

  # Top search terms in the last N days.
  def top_queries(days: 28, limit: 200)
    query(dimensions: ['query'], start_date: since(days), end_date: today, row_limit: limit)
      .map { |r| { query: r['keys'].first, **metrics(r) } }
  end

  # Top landing pages in the last N days.
  def top_pages(days: 28, limit: 200)
    query(dimensions: ['page'], start_date: since(days), end_date: today, row_limit: limit)
      .map { |r| { page: r['keys'].first, **metrics(r) } }
  end

  # LOW-HANGING FRUIT: page+query pairs ranking just off page 1 (positions 8-20)
  # with real impressions — the fastest SEO wins, sorted by opportunity size.
  def low_hanging_fruit(days: 28, min_position: 8.0, max_position: 20.0, min_impressions: 10)
    query(dimensions: %w[page query], start_date: since(days), end_date: today, row_limit: 5000)
      .map { |r| { page: r['keys'][0], query: r['keys'][1], **metrics(r) } }
      .select { |h| h[:position].between?(min_position, max_position) && h[:impressions] >= min_impressions }
      .sort_by { |h| -h[:impressions] }
  end

  # Every query one specific page ranks for.
  def queries_for_page(page_url, days: 28, limit: 500)
    filter = [{ filters: [{ dimension: 'page', operator: 'equals', expression: page_url }] }]
    query(dimensions: ['query'], start_date: since(days), end_date: today,
          row_limit: limit, dimension_filter_groups: filter)
      .map { |r| { query: r['keys'].first, **metrics(r) } }
      .sort_by { |h| -h[:impressions] }
  end

  private

  def metrics(row)
    {
      clicks:      row['clicks'].to_i,
      impressions: row['impressions'].to_i,
      ctr:         (row['ctr'].to_f * 100).round(2),  # as a percentage
      position:    row['position'].to_f.round(1)
    }
  end

  def today = Date.today
  def since(days) = Date.today - days.to_i

  def access_token
    return @access_token if @access_token && @access_token_expires_at > Time.current

    resp = Net::HTTP.post_form(URI(TOKEN_URL), {
      'client_id' => ENV.fetch('GOOGLE_CLIENT_ID'),
      'client_secret' => ENV.fetch('GOOGLE_CLIENT_SECRET'),
      'refresh_token' => @refresh_token,
      'grant_type' => 'refresh_token'
    })
    data = JSON.parse(resp.body)
    unless resp.is_a?(Net::HTTPSuccess)
      raise ConfigError, "GSC token refresh failed (revoked? consent screen in Testing? " \
                         "wrong client?): #{data['error']}: #{data['error_description']}"
    end
    @access_token_expires_at = Time.current + data.fetch('expires_in', 3600).to_i - 60
    @access_token = data.fetch('access_token')
  end

  def get(path)  = request(Net::HTTP::Get, path)
  def post(path, body) = request(Net::HTTP::Post, path, body)

  def request(klass, path, body = nil)
    uri = URI("#{API_BASE}#{path}")
    req = klass.new(uri)
    req['Authorization'] = "Bearer #{access_token}"
    req['Content-Type']  = 'application/json'
    req.body = body.to_json if body
    resp = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true, read_timeout: 60) { |http| http.request(req) }
    raise ApiError, "GSC API #{resp.code} on #{path}: #{resp.body&.slice(0, 500)}" unless resp.is_a?(Net::HTTPSuccess)
    JSON.parse(resp.body)
  end
end

Layer 3 — Using it

# Anywhere in the app (or bin/rails runner):
gsc = GscSearchAnalytics.new

# 1) Verify auth + property access — your site must be in this list:
gsc.sites
# => [{"siteUrl"=>"sc-domain:example.com", "permissionLevel"=>"siteOwner"}]

# 2) The reports:
gsc.top_queries(days: 28)
gsc.top_pages(days: 28)
gsc.low_hanging_fruit                        # positions 8-20 → your fastest SEO wins
gsc.queries_for_page("https://example.com/pricing")
# Each row: { query:/page:, clicks:, impressions:, ctr:, position: }

Wire the results into a dashboard, a weekly report job, or a CSV export — the data shape is a flat array of hashes, so a plain table view is enough to start.

Gotchas (the hard-won stuff)

Consent screen in "Testing" = refresh tokens die after 7 days. The integration works perfectly for a week, then every call fails with invalid_grant. Publish the consent screen to production before minting the token you intend to keep. Tell: if the token exchange response includes refresh_token_expires_in, you are in Testing mode; a production-mode exchange has no expiry field.
The redirect URI must match EXACTLY. redirect_uri_mismatch means the redirect_uri your app sent differs from every entry under Authorized redirect URIs — check scheme (http vs https), host, port, path, and trailing slash. Behind a proxy, Rails may generate http:// URLs; force https in the URL helper or config.
Google omits the refresh token on re-consent. A second OAuth pass for a user who already approved returns NO refresh_token at all. Always send access_type: 'offline' + prompt: 'consent', and never overwrite a stored refresh token with a blank.
.env changes need a container recreate, not a restart. Docker bakes env vars in at container create time. docker compose restart keeps the old values — use docker compose up -d --force-recreate web (or down + up -d).
Turbo breaks the OAuth redirect. Any "Connect" form/button must carry data: { turbo: false }, or the cross-origin redirect to accounts.google.com fails silently on CORS.
Scope box in the OAuth Playground: paste ONLY the URL. Error 400: invalid_scope ... invalid=[console, search] means words like "Search Console" got into the scope box and were space-split into bogus scopes.
Playground: "Use your own OAuth credentials" must actually take. After enabling it you must RE-run the Authorize step — the tell is the exchange request showing YOUR client_id. A token minted under Google's default Playground client cannot be refreshed by your app (unauthorized_client) and is auto-revoked within 24 hours. And use "Exchange authorization code for tokens", not "Refresh access token" — refreshing just re-uses whatever (possibly wrong-client) token you already had.
403 with working auth = API not enabled, or no property access. Enable the Search Console API in the SAME Google Cloud project as the OAuth client, and make sure the Google account that approved the consent has at least "Full" (or restricted read) access to the property in Search Console → Settings → Users.
Domain property site URL is sc-domain:example.com — not a normal URL. Only URL-prefix properties use the https://example.com/ form. Passing the wrong form 404s/403s the query call. gsc.sites shows the exact strings your token can see.
Data starts at verification — Google does not backfill. A property verified today has no history; "before vs after" comparisons against pre-verification dates compare against empty data. Also: the most recent 1–2 days are always incomplete (finalization lag), and a brand-new property shows nothing for the first ~2–3 days.
The refresh token is durable, but not immortal. It survives restarts and deploys in .env. It dies if: the OAuth client is rotated (this also kills every OTHER token minted under that client — Gmail integrations included, if they share the client), the user revokes the app in their Google Account security settings, or the token sits completely unused for 6 months. ConfigError: token refresh failed is the re-mint signal.

Files this pattern touches

app/controllers/gsc_auth_controller.rb      (one-time connect flow; removable after minting)
app/services/gsc_search_analytics.rb
config/routes.rb
.env                                        (GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET,
                                             GSC_REFRESH_TOKEN, GSC_SITE_URL)

How to adapt to your schema

Multi-tenant (each org connects its own Search Console)? Move the refresh token off .env onto your tenant model (organizations.gsc_refresh_token, encrypts :gsc_refresh_token), persist it in the callback instead of rendering it, and pass it to GscSearchAnalytics.new(refresh_token: org.gsc_refresh_token, site: org.gsc_site_url). The Google Ads cookbook guide shows this exact per-organization credential pattern.
Single-tenant / no controller wanted? Skip Layer 1 entirely and mint via the OAuth Playground path (Layer 1, last paragraph) — zero code changes, just the .env entries.
URL-prefix property instead of a domain property? Set GSC_SITE_URL=https://example.com/ (the exact verified prefix, trailing slash included).
Need write access (submit sitemaps, inspect URLs)? Swap the scope to https://www.googleapis.com/auth/webmasters and re-mint — the readonly token cannot be upgraded in place.
Safe to drop: the low_hanging_fruit / queries_for_page helpers if you only want totals; the whole auth controller once the token is minted and stored.