Google Scholar as a Data Source for Research Assessment in the Social Sciences

Research assessment has become a cornerstone of academic life, influencing funding decisions, hiring processes, and tenure promotions. In the realm of the Social Sciences, evaluating research output presents unique challenges compared to the hard sciences. While citation metrics have historically relied on proprietary databases like Web of Science (WoS) and Scopus, Google Scholar (GS) has emerged as a potent alternative data source. This discussion explores the viability, advantages, and limitations of using Google Scholar for research assessment in the Social Sciences.

The Landscape of Bibliometrics in Social Sciences

Traditionally, bibliometric analysis in the Social Sciences has been criticized for its heavy reliance on journal impact factors and citation counts derived from selective databases. WoS and Scopus, often referred to as "bibliometric data sources," employ rigorous selection criteria for journal inclusion. While this ensures high quality, it often excludes significant portions of Social Science scholarship, particularly non-English literature, books, and conference proceedings.

Unlike the natural sciences, where the journal article is the primary communication vehicle, Social Scientists frequently publish books, book chapters, and reports. Consequently, relying solely on WoS or Scopus can provide a skewed picture of research impact, marginalizing scholars who publish through non-journal channels. This gap has necessitated the search for more comprehensive data sources, bringing Google Scholar into the spotlight.

Why Google Scholar?

Google Scholar is a freely accessible web search engine that indexes the metadata of scholarly literature across an array of publishing formats and disciplines. It covers a vast spectrum of sources, including peer-reviewed online journals, books, theses, dissertations, preprints, abstracts, and technical reports from academic publishers, universities, and other scholarly organizations.

Comprehensive Coverage of Books and Chapters

For the Social Sciences, the most significant advantage of GS is its coverage of books. Monographs remain a vital outlet for research in fields such as History, Political Science, Sociology, and Anthropology. Citations to books are frequently tracked by GS from publisher websites, university repositories, and digitization projects like Google Books. When assessing the impact of a Social Science scholar, ignoring book citations can result in a severe underestimation of their total influence.

Visibility of Non-Journal Literature

Research assessment in the Social Sciences often needs to account for "grey literature"working papers, policy reports, and conference papers. These documents often influence practice and policy significantly before they appear in peer-reviewed journals. Google Scholar excels at indexing these items, providing a more immediate reflection of research activity and utility.

Global Reach and Language Diversity

Proprietary databases are often biased towards journals published in English and Western Europe. Google Scholar, with its automated crawling capabilities, indexes regional journals and non-English publications with greater frequency. This is crucial for assessing research in Area Studies or Sociology, where local language scholarship addresses specific cultural and societal contexts that international journals might overlook.

Metric Capabilities: The h-index and Beyond

Google Scholar automates the calculation of bibliometric indicators for individual authors through the "My Profile" feature. The most widely used metric is the h-index, proposed by physicist J.E. Hirsch. The h-index attempts to measure both productivity and citation impact. A scholar with an h-index of 10 has published at least 10 papers that have each been cited at least 10 times.

In the Social Sciences, where citation cycles are slower than in Physics or Biology, the h-index provided by Google Scholar is often higher and arguably more representative than those derived from WoS. Other metrics available through GS profiles include the i10-index (the number of publications with at least 10 citations) and the total citation count. These tools allow researchers to curate their own public profiles, offering transparency in assessment processes.

Limitations and Challenges

Despite its breadth, Google Scholar is not without flaws. When used for formal research assessment, several limitations must be acknowledged to ensure data integrity.

Data Quality and Cleaning

The "black box" nature of Google Scholar's bot is a primary concern. Unlike Scopus or WoS, which have clear lists of indexed titles, GS coverage is volatile and algorithmic. Duplicate records are common (e.g., a preprint and the final published version counted separately). Furthermore, GS does not discriminate rigorously between scholarly sources and non-scholarly documents that simply look scholarly, potentially capturing presentations or course materials that artificially inflate citation counts. Manual cleaning of data is often required before GS data can be used in high-stakes assessment.

Citation Inflation and "Predatory" Journals

The inclusiveness of Google Scholar extends to low-quality and predatory journals that engage in self-citation schemes. Since GS does not filter for journal quality based on peer-review standards in the same way subscription databases do, citations from these sources are counted equally. Critics argue that this can distort metrics, rewarding those who publish in venues that inflate citation counts rather than those producing rigorous scholarship.

Instability of Metrics

Citation counts in Google Scholar are not static. They can fluctuate as the crawler updates its index or as the algorithm changes. For a research assessment exercise, reproducibility is key. Two queries performed a week apart might yield slightly different citation counts, making it difficult to standardize assessments over time compared to the frozen datasets often provided by proprietary aggregators.

Comparison in the Context of Research Assessment

The debate between using Web of Science/Scopus versus Google Scholar is not necessarily binary. In the Social Sciences, a "comprehensive approach" is increasingly advocated.

• Web of Science/Scopus: Best for measuring "elite" international impact and high-quality journal visibility. They offer stable, verifiable data and are useful for benchmarking against top-tier international standards.
• Google Scholar: Best for measuring total societal and academic visibility, including the impact of books and grey literature. It captures the broader influence of a scholars work across the web.

Studies comparing citation counts between these sources consistently show that GS citation counts are significantly higher than those in WoS or Scopus, sometimes by a factor of two or three in Social Science disciplines. This discrepancy highlights that using subscription databases alone omits a vast portion of the citation network that Social Scientists inhabit.

Best Practices for Using Google Scholar

To effectively utilize Google Scholar as a data source for assessment, institutions and evaluators should adopt specific best practices:

• Triangulation: Use Google Scholar data alongside Web of Science or Scopus data. A profile that is strong in both indicates true impact within both the core journals and the broader literature.
• Manual Verification: Assessors should spot-check the publication lists and citations to ensure that duplicates and non-peer-reviewed items are minimized.
• Contextualization: Metrics should be normalized by field. An h-index of 20 in Economics may carry different weight than an h-index of 20 in Sociology. GS does not always disambiguate fields perfectly, requiring expert human interpretation.
• Use of Tools: Third-party tools like Publish or Perish, Harzing's AWPS, or R Packages (such as scholar) can extract data from Google Scholar more robustly, allowing for longitudinal analysis and easier data cleaning than the web interface alone.

Conclusion

Google Scholar has democratized access to bibliometric data, providing a lens through which the impact of Social Science research can be viewed in its totality. By including books, chapters, conference papers, and regional journals, it addresses the structural limitations of traditional citation databases. While it lacks the rigorous quality control of Web of Science or Scopus and presents challenges regarding data cleaning and inflation, its coverage makes it an indispensable tool for modern research assessment.

For the Social Sciences, where the monograph and the diverse dissemination of knowledge are paramount, ignoring Google Scholar means ignoring a significant portion of academic influence. A robust research assessment framework should therefore embrace Google Scholar not as a replacement for traditional metrics, but as a complementary data source that offers a more nuanced, inclusive, and realistic portrait of scholarly contribution.