Chief Scientist renews call for innovation metrics overhaul

By James Riley

The data and systems used to measure innovation and commercialisation in Australia are no longer working for us, according to chief scientist Cathy Foley, who says our innovation metrics are in need of an overhaul.

The same can be said of the data we use to measure university rankings and the success of individual researchers, she says.

The shortcomings are standing in the way of improved innovation and commercialisation outcomes and leading to perverse outcomes in the allocation of resources for public research.

You can’t solve problems in the system unless you can measure those problems and you get those measurements right, Dr Foley says.

Speaking at the Australian Academy of Technology and Engineering’s Malcolm Chaikin Oration on Thursday, Dr Foley says the shortcomings in innovation metrics was getting in the way of meeting government objectives – whether that is in research translation and commercialisation or attracting more students into science, engineering and maths vocations.

“Innovation systems are dynamic and interconnected. Many metrics have focussed on measuring R&D. But the R&D data doesn’t capture all innovation – since not every innovation is the result of a research breakthrough,” Dr Foley said.

“For example, it is increasingly important to measure digital activities in the economy. But digital innovation and take-up is not adequately captured in measures of innovation.

“There is significant focus on commercialisation of public research and counting startups. But this, too, fails to recognise that most commercialisation happens within businesses – what’s called ‘intrapreneurship’.

“Internationally, the inputs to various commercialisation indices differ between countries, which makes comparisons less than enlightening,” she said.

The way that universities are measured runs into the same kinds of complexities.

Universities are extraordinary institutions that have many different roles and there are different ways of measuring what happens within them, she said, while the measurements for understanding ‘industry engagement’ are fairly blunt.

“International rankings drive behaviour and priorities in the university sector, but the parameters used can be narrow.

“Even within research institutions, the incentives operate to encourage a kind of scientific version of the Hunger Games! Citations, publication numbers, grants, bonuses for publishing in Nature and Science.

“These are good journals, but they’re not the only ones, and they’re not for every discipline.”

Dr Foley’s push on improving the metrics in the innovation system picks up where her predecessor Dr Alan Finkel left off.

The Office of the Chief Scientist and the Industry department’s Office of the Chief Economist had jointly conducted a substantial Innovation Metrics Review under Dr Finkel that was due to published in May 2019 but has never seen the light of day.

The Industry department’s Innovation Metrics Review web page is still live, and three years on the report and its recommendations are still being considered by the Minister’s office.

Dr Foley did not refer to the Innovation Metrics Review, but she was expansive about the role that a fit-for-purpose metrics systems can play.

“When we think about metrics we need to think about what purpose they are serving and whether they meet our needs now and for the future,” she said.

“To answer that question, we need to ask another one. What do we want to achieve?

“From where I stand, the answers to those questions are:

• Support and grow our research sector
• Kick-start high-tech manufacturing here in Australia
• Solve some big technology problems, including in climate and low-emissions technologies
• Become world leaders in emerging sectors such as quantum and space
• And to make sure we are funding the best and brightest to undertake fundamental research, coming up with answers to questions we haven’t even asked yet.

“This is what we want to achieve.”

Picture: blog.csiro.au

This article originally appeared at InnovationAus.

Subscribe to our free @AuManufacturing newsletter here.