Google Search Console Bulk Data Integration via BigQuery

Google Search Console Bulk Data Integration via BigQuery

Google Search Console Bulk Data Integration via BigQuery

Google now allows Google Search Console (GSC) users to export all of their data via an API to BigQuery set up in Google Cloud Services (Google announcement). You can take the additional step of integrating that data export directly into a profile in seoClarity's Search Analytics. Note: Integration may incur an additional cost depending on the contract agreement.

Why integrate GSC Bulk Data with seoClarity?

  • Bring ALL your search console data into seoClarity.
  • No Historical Data Loss: The GSC export will only have data starting from when you first activate it, but using the merge profiles filter option in Search Analytics allows you to bring it together with your historical data already in our platform.
  • Conduct Deeper Analysis: Go beyond basic metrics. Use our powerful filters to segment data by Content Types, Page Tags, and other custom dimensions. Slice, dice, and analyze your search performance to uncover actionable opportunities.
  • Report with Ease: Access all your data in one platform to build comprehensive reports seamlessly, without toggling between systems.
  • Cost-Effective Scaling: Enjoy full access to your data without worrying about fees. We support integrations up to 1,000,000 rows at no extra cost.
Info
Need to integrate historical data?
If you have historical data in BigQuery, our team can help import up to 16 months of it into seoClarity. Email us at support@seoclarity.net to get started.

Background & Requirements for GSC Bulk Data Integration via BigQuery

  1. Google Search Console and Google BigQuery access is needed. 
  2. For instances with more than 500k rows, there is a cost associated. The number of rows can be identified by integrating data for a single day.
Notes
Note: At this time, there is no way to limit the integration to just 500k rows

How to Integrate Bulk Data Export with Search Analytics

Follow the steps in our video guide to setting up the Bulk Data Export below, but add one additional principal in the BigQuery setup section. In addition to the principal shown in the video, add search-console-data-export@seoclarity.seoclarity.net.iam.gserviceaccount.com with the exact same permissions. And provide us with the IDs of all three tables located in your Google Cloud BigQuery dataset:
  1. ExportLog
  2. searchdata_site_impression
  3. searchdata_url_impression
\

Search Console BigQuery Integration Steps

1) Navigate to the Google Cloud Console and login. Once you log in, click the Select a project dropdown. 


2) On the project selection popup, select the New Project button.


3) Give the project a name and associate a billing account. Data storage has a free usage and storage level, but if that is exceeded, your Google billing account will be charged. 


If you don't have a billing account for Google Cloud, you can set one up via the Billing tab.


4) Once the Google Cloud Project is created, select the project and copy the Project ID. Save this as it will be needed in Google Search Console.
 

5) On the Google Cloud menu, navigate to APIs and Services, and select Enabled APIs and Services.

6) Select the + Enable APIs And Services button.


7) From the API library, search for BigQuery API and select it.


8) Select the BigQuery API option.

9) If this is the first project with BigQuery, the button will say Activate (otherwise it will say Manage). Select it to activate BigQuery API for the project. 

10) To enable this, we need to add a new user to the project with the correct access privileges. On the IAM and Admin menu, select IAM. 


11) Select the Grant Access button.


12) Add the user 'search-console-data-export@system.gserviceaccount.com' in the New principals box. This is the email provided by Google in their Search Console documentation. Assign the roles of 'BigQuery Job User' and 'BigQuery Data Viewer'. Repeat the steps for the user search-console-data-export@seoclarity.seoclarity.net.iam.gserviceaccount.com with the exact same roles and provide support@seoclarity.net with the IDs of all three tables located in the Google Cloud BigQuery dataset:
  1. ExportLog
  2. searchdata_site_impression
  3. searchdata_url_impression



13) From the main menu, select BigQuery and select Analysis to see the database created once configuration is complete.


14) Once the BigQuery project is setup, navigate to Google Search Console and select Bulk data export in the settings. A message should appear stating Setup completed successfully. If a Setup failed message occurs, navigate back to Google Cloud and confirm the IAM permissions are correct. 


15) Take the table ID information from step 12 and navigate to the Settings page of the domain to be integrated. From the integrations tab select the Google Search Console via BigQuery option and input the corresponding table IDs and select the Save button. The integration process should now be complete and data should begin to populate in Search Analytics within 24-48 hours. 

    • Related Articles

    • Google Search Console Integration

      Google Search Console Integration Overview The objective of this integration is to authenticate a Google Search Console account within seoClarity. This integration takes the place of the previous Google Webmaster Tools integration. Background & ...

    • Integration settings

      Integrations Overview The Integrations tab in Settings provides a variety of options for data integrations with the platform. To access Integration Settings go to the gear icon in the top right corner and then click on Settings. Next click on the ...

    • Search Analytics (Google)

      Search Analytics (Google) Overview Search Analytics provides data based on Search Console producing a great insight into your SEO data and efforts. Syncing your GSC profile with seoClarity keeps all of your data in one spot for ease of use and ...

    • Google Analytics Historical Data Retrieval via Flat File

      Overview The standard Google Analytics integration will retrieve daily historical data for the last 13 months prior to the date of authentication. The integration requires that the Google Analytics view that is integrated does not contain a ...

    • Google BigQuery Extract

      Overview Integrating seoClarity into BigQuery allows for data to be connected to additional data sources such as Looker Studio (formerly Google Data Studio), Tableau, and Domo among others. Requirements A Google BigQuery account is required to ...