Skip to main content
Hong Kong Baptist University Library Hong Kong Baptist University
 

Research Impact: Why care?: Home

What is impact?

What is Research Impact?

Citation counts are traditional key measures of scholarly works. Impact measurements have however, expanded significantly in recent years to include influences beyond academia. Research impact has a much broader meaning nowadays and can be wide ranging, varied and project-specific

For Hong Kong's upcoming Research Assessment Exercise RAE 2020, the University Grants Committee (UGC) defines impact as demonstrable contributions, effects, changes or benefits that research qualitatively makes to 6 areas:

Economy  |  Society  |  Culture 
Public Policy/Services  |  Health  |  Environment or Quality of Life

In preparation for RAE2020, HKBU's Graduate School has compiled a list of Impact resources with links to Impact Case Studies drawn from the Experience of UK Research Excellence Framework (login required).

In summary, research impact is a multi-dimensional indicator and it is important to be aware of the positive and negative effects of each measure. This Guide aims to provide you with basic information about some the most common measurement tools.

Why care?

Why care about Impact?

Research supported by public money is seen as needing justification. More and more, researchers are required to demonstrate their contribution to the economy and wider society, as well as to culture and the environment.

Funders around the world now expect researchers to demonstrate research impact. The HK RGC has assigned a 15% weighting for impact in the next RAE 2020, requiring researchers to demonstrate the influences of their work according to the six factors listed above.

Measuring Impact

Measuring Impact

Impact measurement is a controversial topic as there are no agreed standards. Criticisms include metric maximising practices, diverting attention away from actual content, and ignoring the reasons for citations. Various initiatives such as the San Francisco Declaration on Research Assessment (DORA) and Leiden Manifesto aim to educate and address these pitfalls.

Nevertheless, metrics can be useful tools for research assessment when applied appropriately. The following explains some of the traditional ways of measuring impact:

Journal Impact

Journal metrics rate the importance of a journal based on the number of articles published and the number of citations received. One aim is to help researchers decide which journal to publish in. Each system has its strengths and weaknesses and so judgment must be exercised when interpreting and applying the figures. Common journal metrics includes:

Journal Impact Factor (JIF)
Published annually in Web of Science's InCites Journal Citation Reports, JIF measures how frequently an "average article" is cited over a defined period and is thus an indication of how often a journal is cited by other journals in a field. However, journal citation distributions are highly skewed due to a small percentage accounting for a large majority of total citations. As such, a great percentage of papers even in high JIF journals will have relatively few citations.

JOURNAL IMPACT FACTOR = 

                     Average number of cited articles    
_________________________________________________

                      Number of citable items in previous two years

A JIF of 2.5 means that on the average, articles published one or two years ago have been cited 2.5 times.

SCImago Journal Rank (SJR)
Based on Scopus data, SJR was inspired by Google's PageRank algorithm and has visual presentation capabilities. It is a free online tool that ranks publications by weighted citation per document. A citation from an important journal will count as more than one and likewise, those in a less prestigious journal will have a lower weight.

Source Normalized Impact per Paper (SNIP)
Scopus-based SNIP measures contextual citation impact to allow direct comparisons between subject fields. For example, due to the usual higher value for STEM-related journals, Arts and Humanities journal titles are given a higher rating to balance out the value difference. 

CiteScore 
Freely accessible via Elsevier's Journal Metrics, CiteScore is based on citations received in one year to articles published in the previous 3 years, divided by the number of articles indexed in Scopus published in the same 3 years.

 

You may find Professor Anne-Wil Harzing's Where to submit your paper? Compare journals by impact helpful when trying to decide where to publish.

Author Impact

Author impact metrics are based on the number of publications by an author and the number of times they have been cited. Key author impact measurements include:

h-index
The h-index attempts to balance scholarly productivity with citation impact. A scholar with an index of h has published h papers that have been cited by other papers at least h times. Thus an author with an h-index of 50 has published 50 articles each with 50 or more citations. It is sometimes seen as unfair to early-career researchers.

i10-index
Used only in Google Scholar in its My Citations feature, i10-index is a basic and straightforward way of measuring a scholar's productivity. It simply counts only the number of publications with at least 10 citations.

Article Impact

Article metrics are from a variety of different data sources, such as the traditional Citation Counts, Field-Weighted Citation Impact (FWCI), and the more recent altmetrics. Scopus, Web of Science and Google Scholars all provide easy ways for authors to find their article impact. 

Citation Counts
The assumption of Citation Count is that the more important or influential the work, the higher the number of Citation Count. However, the argument is that citations don't measure quality and are purely based on subjective judgment. Citation Counts also favour mainstream research and may reveal negative instead of positive attention.

Field-Weighted Citation Impact (FWCI)
Field-Weighted Citation Impact metrics are a useful benchmark regardless of the differences across disciplines. Sourced from SciVal, FWCI compares the number of citations received by a researcher against the average number of citations received by all other publications that have the same publication year, type and discipline, calculated as follows:

FIELD-WEIGHTED CITATION IMPACT = 

           Number of citations received by an article
_______________________________________________

      Expected number of citations for similar articles

  • 1.00 means article was cited at world average for similar publications 
  • 1.45 means article was cited 44% more than expected
  • 0.75 means article was cited 25% less than average 

In addition to the above, altmetrics have become increasingly popular as measures of article level impact in recent years. See the next section for more details.

 

Why Altmetrics?

Why Altmetrics?

Short for alternative metrics, altmetrics complement traditional impact measures but unlike citation counts which may take years to emerge, they are available almost immediately. The birth of altmetrics in recent years has been a benefit to many scholars, in particular, those from Humanities and Social Sciences, providing them with an alternative way to assess their work apart from traditional journal metrics. Altmetrics have gained widespread attention, giving researchers total new perspectives to see how their works are influencing not only scholarly communities, but also beyond.

Altmetrics

Examples of altmetrics include: mainstream media coverage, citations on Wikipedia, discussions on research blogs, mentions on Facebook or Twitter, or exports to citation management systems like Mendeley. Main ways to access and make use of altmetrics data include: 

scopus.com | Found on the right hand sidebar of the Scopus Document details page, the 5 categories of PlumX metrics track various altmetrics data. This has become the primary source of article level metrics on Scopus alongside citation count and field-weighted citation impact.  

impactstory.org | Funded by the National Science Foundation and the Alfred P. Sloan Foundation, ImpactStory allows users to create profiles that gather usage data from various online research sharing platforms.

altmetric.com | Not to be confused with altmetrics.org, altmetric.com is a product offered by Digital Science that uses a doughnut symbol to visualise the reach and impact of articles. altmetric.com widgets can be found on many publisher article metrics pages or repositories such as Figshare.

Altmetrics are still in their infancy. Just like traditional metrics, they have limitations and can also be easily manipulated. To learn more, visit: altmetrics.org.

Watch this!

Why do they care?

 Watch this 2-min video on why these University of Leicester researchers care about impact.

Top tips

Top tips
for articulating impact

 Plan impact early as part of your research design.
 Organise engagement workshops, marketing and PR activities, etc.
 Ensure you get credit for your work - get an ORCID ID

Read Taylor & Francis' 10 tips on how to maximise the impact of your research.

Glossary

Altmetrics

Short for alternative metrics which includes Mendeley captures, Wikipedia mentions, and social media shares. Publishers such as PubMed Central, PLOS and Elsevier have all incorporated altmetrics into their workflow. 

Read more...

Bibliometrics
Statistical methods that measure, track, and analyse scholarly publications for impact assessment. Common bibliometrics include citation counts and Journal Impact Factor. 

Read more...

Captures 
1 of 5 Plum metrics tracking readers; bookmarks, favourites, readers, watchers that indicate readers interest to return to a work. Seen as early indicators of citation counts.

Read more...

Embargo
Term in academic publishing to describe period of time publishers require authors and users to wait before publications are available free and open access.

Read more...

Field-weighted Citation Impact (FWCI)
Ratio of citations received, compared with expected world average for subject field, publication type and year. 1 means publication is cited at world average for similar publications and so on. Available in SciVal.

Read more...

H-index
Author-level metrics based on quantity and quality of a researcher's publications. H-index measures both researcher productivity and citation impact.

Read more...

i10-index
Number of publications with at least 10 citations from others. Simple measure used by Google Scholar in its My Citations feature to gauge productivity of a scholar. 

Watch more...

Journal Impact Factor JIF
Average number of citations received per paper published in particular journal in preceding 2 years. 1 means article published 1 or 2 years ago has been cited once.  

Read more...

Institutional Repository IR
Archive for collecting, preserving, and disseminating an institution's research output. Works in IRs are said to benefit from an OA Citation Advantage. 

Read more...

Journal Citation Report JCR
Sourced from Web of Science Core Collection. JCR is published annually with information about academic journals in natural and social sciences, including impact factors. 

Read more...

Mentions
1 of 5 Plum metrics that tracks mentions of work such as blog posts, comments, reviews, and Wikipedia links as measures of users' interaction and engagements.

Read more...

Open Access Citation Advantage (OACA) 
Benefits researchers get from depositing work online outside playwall. OACA has been confirmed with definitive results as demonstrated in numerous research. 

Read more...

Open Access OA
Peer-reviewed research that is free for all or for reuse under licenses such as Creative Commons. Depositing your work into institutional repositories is green OA. Publishing in OA journals such as PLOS is Gold OA.

Read more...

ORCID
Short for Open Researcher and Contributor ID. 16-digit code for unambiguous and identification to resolve author identity confusions. Many publishers now require ORCID IDs during submissions. 

Read more...

PlumX Metrics
Metrics including Usage, Captures, Mentions, Social Media, and Citations that tracks user interactions. Particularly useful to Humanities & Social Sciences researchers who rely less on citation counts to assess impact.

Read more...

Plum Print
Star-shaped symbol containing PlumX metrics of Usage, Captures, Mentions, Social Media and Citations. Available in Scopus, Mendeley and Figshare, etc.

Read more...

Research Impact 
Contribution research makes beyond academia, including economy, society, culture, public policy or services, health, and environment or quality of life, as defined by HK RGC. Impact has 15% weighting for RAE 2020.

Read more...

Research Visibility 
Based on idea that research should be shared to create new insights. Methods include using ORCID ID to avoid losing work, sharing via repositories, and social media engagements.

Read more...

SCImago Journal Rank (SJR) 
Weighted citations depending on journal prestige. Citations from more important journals are given higher value and vice versa. Available in Scopus.

Read more...

Split Author Profiles
Wide-spread confusion in academia due to similar or multiple formats of same author names. Author profiling tools such as ORCID ID will help resolve author identity crisis.

Read more...

Usage
1 of 5 Plum Metrics that tracks usage such as reading or watching of a researcher's work. Regarded as the most popular statistics after citation counts.

Read more...

Find out more

Find out more about Impact!

Feel free to contact me if you have questions regarding Research Impact, or any other scholarly communication issues.

Pauline Lam
Scholarly Communications Librarian
lamlhp@hkbu.edu.hk