General

SEO Analytics for free – It combines Google Search with the Moz API

ADVERTISEMENT

Searching using Google Custom Search Engine

In the beginning, we must conduct a search on Google and get some results documented. To make sure that we’re in compliance with Google’s Terms of Service we’ll not use scraping Google.com in the first place, but instead make use of search engine’s Custom Search feature. This is the Google’s Custom Search is designed mainly to let webmasters add the possibility of an Google type search engine to their website. It is built-in REST Google Search API it is free. You can search Google and provide results in the well-known JSON formats. There are some limitations on quotas, however the API can be modified and extended to provide the most relevant information you can use.

If you have it properly set up for search online it is possible to make requests to Your Custom Search Engine, in our case, using PHP and consider them to be Google results, with certain limitations. The major drawbacks with the usage for Custom Search Engine Custom Search Engine are: (i) it does not use Google Web Search features, such as personal results; (ii) it doesn’t use Google Web Search capabilities, such as personal results as well as (ii) it may offer a subset from its Google index in the event that you have more than 10 websites.

However, despite the limitations there is numerous options for search results that you can forward for Custom Search Engine. Custom Search Engine to proxy the results you’d expect Google.com would offer. In our instance, we used the following details when making an phone call:

What is the location of:

  • https://www.googleapis.com/customsearch/v1 – is the URL for the Google Custom Search API
  • keykey – Your Google Developer API Key
  • userIp= It’s the IP address of the machine calling.
  • Cx= – Your Google Custom Search Engine ID
  • Q=iPhone+X Google search string (‘+’ replaces ‘)
  • cr=countryUS countryUS – Restrictions on countries (from the Goolge’s Names of Collections in the List of Country names)
  • begin=1 It is an index of the search result which will be displayed starting with one. e.g. SERP page 1. Each subsequent request will increase the number of pages until you have two pages or more.

Google has made it clear that its Google Custom Search engine differs from Google .com however, when I made my own testing and compared the results of both, I was amazed by the similarities that I continued to study. Be aware that the results and information provided below are the result of Google Custom Search (using ‘whole web search queries’) rather than Google.com.

Utilizing the free Moz API Account

Moz provide an (API). To use it, you need to sign up to use Mozscape API. You can register for Mozscape API code for free However, it is restricted to a quota of 2,500 rows each month, along with 1 request per 10 seconds. The current plans which are paid include more allowances, beginning at $250/month. If you sign up for a free account and API Key, you’ll be able gain access to the Links API links API and take a look at these indicators

What is the location of:

  • http://lsapi.seomoz.com/linkscape/url-metrics/” class=”redactor-autoparser-object”>http://lsapi.seomoz.com/linksc… – Is the URL for the Moz API
  • http%3A%2F%2Fwww.apple.com%2F – An encoded URL that we want to get data on
  • Cols=103616137253– – The amount number of Moz API codes that are listed on the above table.
  • AccessID=MOZ_ACCESS_ID – An encoded version of the Moz Access ID (found in your API account)
  • Expires=1560586149. Timeout to query has been set to several minutes in the near future.
  • Signature is A cryptographic variant from your Moz Access Identification (found inside the account of your API accounts)

The extraction of data with PHP along with MySQL

We’ve got an API for the Google Custom Search Engine as well as our Moz API, and are ready to gather information. Google as along with Moz respond to requests via their JSON format and can be utilized by numerous popular language programs. Alongside my favorite programming language PHP, I recorded my findings from both from Google as well as Moz into

A database, and chose MySQL Community Edition to use the. Other databases are also available to use, e.g. Postgres, Oracle, Microsoft SQL Server etc. It is a way to store data and ad-hoc analysis using SQL (Structured SQL Query Language) and other languages (like R, which I will cover in a subsequent). After creating tables for databases to contain Google outcomes (with fields for rank, URLs , and so on.) and a table to store Moz field information (ueid Ueid, upa, UDA etc. ) and a table for an data collection strategy.

Google provide a large amount of space for its Custom Search Engine (up to 100 million requests every day, using that identical Google Developer Console key) But it’s not accessible for free. Moz’s API accessible to users for free is limited to a limit of 2500. Even though the API for free is accessible to Moz the paid-for option offers between 120k and 40 million rows per month , depending on the plan, and the price range from $250 to $10,000 per month. Since I’m just exploring the options that are free for my site, I designed my own program to gather more than the 125 Google queries that are spread across two pages of search results (10 results per page) which lets me stay within the Moz’s 2500 row limit. If you are looking for a specific type of search that I want to target Google there is a wide range of options available. I decided to use Mondovo as they offer a range of lists that are organized into categories, and up to 500 words per listing, enough for the purpose of the test.

I also included a few of PHP helper classes the code I wrote to manage database I/O and HTTP.

In summary the most important PHP elements and resources used included:

  • Google Custom Search Engine Google Custom Search Engine Ash Kiswany wrote a great piece with Jacob Fogg’s PHP interface for Google Custom Search;
  • Mozscape API as previously mentioned the PHP-based implementation that allows you for connecting Moz Moz via Github was an excellent starting point;
  • The Web Crawler and HTTP web crawler as well as HTTP purple Toolz The crawler we have developed is called PurpleToolzBot which utilizes the Curl method as a part of HTTP, along with an HTML-based parser to the DOM.
  • Database I/O Database II/O PHP provides a great interface for MySQL that I have incorporated into instructional videos.

Another thing to be aware of is the 10 second interval between each Moz API calls. This will prevent Moz becoming overwhelmed by API users who are not in the free API users. To control this, within the software, I designed a “query throttler” which blocked access for the Moz API between successive calls within a predetermined time. However, even though it was functioning perfectly however, the ability to make calls to Moz repeatedly. It took just seven hours to complete.

Analyzing data using SQL and R

Data harvested. Now it’s time to have fun!

It’s time to into consideration the options available to us. This is sometimes called the data-wrangling. I’m using a no-cost statistical programming language referred to as R as well as an environment to develop (editor) called R Studio. There are other languages like Stata and a range of tools for data science graphics like Tableau however , they’re expensive while the finance director at Purple Toolz isn’t someone to put your money on!

I’ve been using R for a number of years because it is open source, and it includes numerous third-party libraries. This makes it very adaptable and perfect for this type of work.

Let’s get our hands dirty.

There are two database that contain the data I collected from searches of 125 keywords that are on 2 pages of results from searches (i.e. 20 URLs ranked according to word). Two tables in the database hold results from Google results, and a third table holds data from the Moz data result. To access theseresults, we’ll need perform an INNER JOIN in the database that will allow us to access the Moz information results.

can easily accomplish by using the RMySQL package with R. This is loaded by typing “install.packages(‘RMySQL’)” into R’s console and including the line “library(RMySQL)” at the top of our R script.

Then follow the steps below to connect your data and convert it into an R Data frame Variable referred to by the name of “theResults.”

We can now make use of R’s complete collection of statistical tools to start creating an mess.

Let’s review a few short overviews to help you better understand the details. The method I use is similar for all the fields. Let’s look at Moz’s ‘UEID field (the number of equity external hyperlinks that link to that URL). In the field R I get the following:

If you look at the graph, you can see that the data is deformed (a substantial amount) due to the relationship of media and mean that is affected by values that fall within the upper half in the spectrum (values greater than 75% observations). It’s possible to illustrate this graph as an R graph consisting of box-like whiskers in R where the value is an estimation of UEIDs in relation to rank starting at Google Custom Search position 1-20.

We’re using log scales along the y-axis to display the whole range of values, as they differ greatly!

box and whisker graph within R Moz’s UID according to Google ranking (note that it’s a logarithmic)

Box plots or whiskers are fantastic because they can provide an abundance of information (see Geom_boxplot, a function inside R). The purple box is the Inter-Quartile Range (IQR) which is the spectrum of values which is between 25 and 75% of the observations. The horizontal line within every box can be referred to as the median (the one located in the middle when you’re placing an order) While the lines that radiate out of within the box (called whiskers) are 1.5x the IQR. Whiskers with dots that aren’t whiskers are called “outliers,” and they represent the extent of what the rank’s data. Alongside the size log, we can see an impressive rise in rank from #10 from #1 on median value, suggesting that the amount of links coming from equity may be one Google rank factors. We can further investigate this using density graphs.

Density plots resemble distributions (histograms) however they show straight lines and instead of bars to represent data. Like a histogram the highest point on a density plot reveals the areas in which the data’s value is concentrated. It can be useful in trying to compare two kinds of. In the graph below I’ve separated the data points into two categories. (i) results that are displayed on the top of page 1 of SERPs and have the rank 1-10 are highlighted in pink. (ii) outcomes that show on the second page are presented with blue. I’ve added the medians of both categories to highlight the differences between results on Page 1 as well as Page 2.

The conclusion that can be drawn from these two Density graphs indicates that Page 1 SERP results had more External Equity backlinks (UEIDs) on the results than the page 2 results. Additionally, you can take a look at the medians for each category below. This clearly shows that Page 1 (38) are significantly higher than Pages 2. (11 ). We now have numbers can be used to inform our SEO strategy for building backlinks.

On this basis, one can be able to determine an equity-related backlink (UEID) are essential and, if I was to recommend a customer based on this data, I would recommend that they aim at a higher than 38 backlinks based on equity that will allow them to reach the top of the SERPs. Of of course it’s not a huge amount of data and further analysis more of the sample and other ranking factors have to be considered but you’ll get the idea.

Let’s examine an alternative measurement with less coverage than UEID. We will also look at what is the Moz UPA score that determines the chance that a site will be ranked top within search result pages.

UPA can be defined as a numerical number that is assigned to an URL that is an interval of 100-100. The data performs better when compared with the prior UEID unbounded variable , having the median and average being close to one another, which causes an overall distribution that is more regular as can be seen when creating a histogram using R.

We’ll be using the identical Page 1: Divide of Page 2 as well as the density plot that we previously used we look at the UPA scores as we break these UPA result into two distinct groups.

Simply put, two different distributions were created by two Moz API variables. Both displayed different variations on the SERPs they were displayed on. We also provide specific indicators (medians) to be used to help clients understand or integrate to your SEO.

However, it is only only a tiny portion of the truth and shouldn’t be considered a complete and accurate representation. With free tools available from Google and Moz There are many ways to start building analytical skills that can be based on your own theories on instead of relying on the conventional approach. SEO ranking factors change frequently therefore having the ability to utilize your own analysis tools to conduct your own tests will build credibility and possibly provide insight into aspects that were previously undiscovered.

 

Next Post