Breaking the Curse on Science
Open data can help us avoid inherent biases in our work, says Ayushi Sood
Better Science through Better Data writing competition winner Ayushi Sood
Recently, an economist friend told me that “scientific inquiry is inherently cursed.” At first I was offended. But I had to agree after he elaborated further – science today suffers from something economists enigmatically call the “winner’s curse”.
The first scientific journals were print editions — something akin to a printed memo — circulated among researchers to update them of the findings of others in the field. To submit a paper for publication, only the observations required to prove results needed to be included in a manuscript, and rightly so: if every paper included every shred of data, journals would run into thousands of pages. This means, though, that what was communicated to the scientific community was only a fraction of what could have been communicated: only the observations that were ‘winners’ – the ones which best supported a result – would be presented, and the others would be effectively relegated to obscurity. Although we’re not limited by paper and page counts today, the same pattern of data use continues. This leads us to the problem of the winner’s curse: by the process of selection, the ‘winning’ observation oversells itself.
In economics, the winner’s curse refers to situations in auctions where the winner tends to overpay, because the actual value of the product is the average of the bids, not the highest bid. In scientific research, the curse takes hold in scientists who aim for publication in the most selective journals, with the most impressive results being favored. This ignores all the other results — the ones which weren’t so impressive — while giving disproportionate importance to the ‘winning result’.
The problem with this phenomenon isn’t immediately evident — isn’t the result what actually matters? The data is, after all, just a tool, necessary only to prove what’s important — the conclusion. In looking for conclusions in data, however, researchers can forget to ask: “does the conclusion effectively justify my repeated sampling of the real world?” In other words, is reality accurately reflected by the dataset presented? All the observations we take, whether they are inconclusive, negative, or ‘winners’, represent an analysis of the natural world. By only reporting the ones that work, the other sampling efforts cannot be used by anyone else. This process confers on a small, selected number of observations the authority to predict an unpredictable future! Back in the auction house, this would mean the value of the product is set only by the winning bid. When we report only the best set of data, we are relegating the less impressive observations to obscurity, even though these also represent an analysis of the real world, with real potential to inform.
So what does this mean for us? How should scientists avoid falling into the trap of the winner’s curse? One way would be to save, store and share all data — not just positive results. We are only human. By making our data openly accessible, we can avoid internal inconsistencies. The smallest of mistakes would be corrected by fresh eyes poring over the very same data.
More importantly, open data could prove to be a shot in the arm for scientific inquiry as a whole. What data I find important may be perfect for my study, yet a small cluster of ignored numbers in my dataset could lead to a breakthrough for someone else, possibly in a way that I could never have imagined! Gene expression data in cancer cells could provide insights into cell signaling pathways in neurodegenerative disorders. Algal bloom observations in polluted lakes could help in effective biomass production for algal biofuel. The analysis and application of open data could usher in a new age of scientific connectivity, with the available knowledge transcending traditional discipline boundaries in way never seen before.
Well, if it’s so good, why hasn’t open data been the norm since science began? We come back to the thousand-page journal here — the question wasn’t of why not, but of how. Transmitting every single byte of data through papers and talks was impossible before the advent of computers and the emergence of the internet in the 1990s. In 2017, however, we have the tools at our disposal to store, parse, organize and retrieve every single digit. The burgeoning field of data science and analysis is ours to exploit, just waiting to script the next scientific success story.
So, I have to hand it to the economists on this one — the winner’s curse is alive and kicking in science. But, like any good scientist, I’m thinking of solutions, and every clue suggests that open data, accessibility and collaboration could be just the spell that breaks this curse.
Ayushi Sood is an undergraduate microbiology student at Amity University, India. Her interest in what makes life tick made her fall in love with bacteria and astrobiology, and her passion for making scientific research more efficient and accessible led her to explore bioinformatics. She has been a part of research projects investigating nanoparticle-plant interactions, transgenic algae, and bacteria-algae associations. Ayushi enjoys dance, writing, and functional DIY craft. You can follow her work on Bitesize Bio and connect with her on LinkedInor Facebook.
Ageism in Academic Jobs in India
Farah Ishtiaq shares her experience on how age and success are linked in acquiring faculty positions in India India has recently been portrayed as a land of abundant opportunity in academia, investing seriously in research and development to attract skilled scientists. The government has introduced several attractive funding opportunities, with the aim of bringing back scientists working abroad to establish a long-term career here, and improving the overall research infrastructure. Wellcome Trust/DBT India Alliance (WT/DBT) fellowships, for example, have no age or nationality restrictions, relying on qualifications, research experience, career trajectory and the candidate’s determination to establish their own independent research. The WT/DBT India alliance was initiated to develop a large cohort of internationally competitive researchers and help in developing scientific excellence and leadership; a model recently adopted and launched by the Alliance for Accelerating Excellence in Science in Africa (AESA) as well. Since Africa shares a similar burden of healthcare with India, as well as many workplace challenges, Indian scientists are perceived as key collaborators in this mission. There’s a problem here though: age limits on academic positions. Prospective candidates for assistant professor posts in India are preferred by academic institutions to be younger than 35. Although funding bodies are not hiring agencies, the age limit imposed on faculty positions by academic institutions sabotages the driving principle behind these new funding opportunities — the current system is unable to absorb enough competent, experienced scientists. The dilemma for early-career researchers is serious; many fellows are facing this harsh reality and an uncertain future. Every research position has a maximum age limit in India; from a junior research fellow (JRF), with a cut off at 28 years old, to postdoctoral researchers where it is 35 (or 40 years for women). These limits put the Indian academic landscape in stark contrast with many other countries that also follow a tenure-track system. Overall, this makes India a viable option and destination only for scientists of a selected age class. And it certainly gets more complicated for women who want to pursue a career in science and raise a family, with very little allowance made for taking time out for such. I am now in my 40s, which prevents me from being offered an entry-level faculty position. I am also a recipient of a WT/DBT India Alliance fellowship. My funding allowed me to establish an independent laboratory to study the ecology and evolution of emerging infectious diseases in wild bird populations, but despite this incredible research opportunity, I feel my career clock is ticking faster than ever. Getting funding to do science is no longer a problem, but academic policies that prevent competent scientists becoming established are preventing me from succeeding. I feel redundant — surely the quality of my science should be the focus rather than my age? What I would have done differently Hopefully, this should give others like me some insight into avoiding some of the mistakes I made. First, I should have found a trustworthy mentor who could have helped me to navigate my career path. Never put all your eggs in one basket. I should have pitched my grant idea to multiple host institutions to maximise my likelihood of finding an institution that would guarantee a more permanent position by this stage. Do your homework in understanding the system and host academic institution. Even though I deferred my fellowship for a year as my daughter was too little to be left in a crèche, I realise I should have used that time to negotiate or better understand the policy or the institute’s vision for a research fellow like me. Many academic institutions have no guidelines on the role, involvement and career development of academic fellows. Many academics fail to understand the role and potential of fellows like me and often consider them just as an extended postdoc — not as a long-term prospect or potential collaborator. Hence, I did not get enough of an opportunity to teach and to mentor PhD students. Having my own PhD students would have bolstered my career at this stage, and allowed me to evolve as a mentor. Local advice & mindset I’ve received various pieces of advice for improving my faculty application and to enhance my chances of a secure job — this was to publish my current research: i) without foreign authors; and ii) as senior or first author in more prestigious journals such as Science, Nature or PNAS. Whilst the first is possible, the second is easier said than done. Apparently, that was the only thing my CV was missing — a clearer demonstration of my calibre and merit as an independent researcher. Interestingly, for established faculty struggling to earn tenure in India, the culture emphasises quantity of publications rather than quality. And, whilst the second piece of advice (primarily from engineers, cell or molecular biologists) is sound, what was missing was perspective on research in the field of ecology — collecting and publishing groundbreaking ecology data in top journals can take years longer than other disciplines. The journals I have been publishing in are not familiar to some of the members of recruitment panels I’ve met with. I have even been asked if ‘Ibis’ and ‘Parasites & Vectors’ were proper journals. Should India be a role model for developing nations? We are struggling to keep our skilled workers, despite the spending per researcher being equivalent to a developed nation like the UK. I hope other developing countries don’t replicate the above policies, as they certainly don’t help to address some of the major longer-term developmental challenges, including a shortage of researchers. In India, with only 200,000 full time researchers (and only 14% of them women) from a population of 1.3 billion, new research institutes currently being developed end up short of skilled workers and blinkered to new research areas. This all said, I am still very excited. I have a competitive edge and enthusiasm for research where I can play a leading role in high-quality research. Perhaps, it’s time to explore science career options elsewhere, maybe in Africa, and hope no one will question my age?
India debates a nationwide tenure system
Academic staff disagree on the merits, and the downsides, of scrapping a common year-long probation scheme. Academic researchers and administrators in India are debating the benefits of adopting a tenure-track system similar to that of the United States in Indian research institutes and universities. A few, including the Indian Institute of Science in Bengaluru, are already using the system, whereas others, such as the Indian Institutes of Technology, have a probationary-period process. Under that scheme, the performance of new faculty members is assessed after one year by a review committee, often comprised of department heads and institutional administrators. Some scientists are calling for the nationwide adoption of a five-year tenure-track review structure. After around five years, research faculty members are reviewed on the basis of their publications and funding received. Teaching ability and service to the institution usually have a supporting role. If the candidate is granted tenure, they receive a permanent appointment. If they are not, the appointment is terminated. Under the probationary system in India, research faculty members who receive a positive assessment at the end of their first year are given permanent positions as assistant professors. After another five years, they can apply to become associate professors — a position with higher rank and pay. If they are unsuccessful, however, their appointments are not terminated. Faculty members can stay at their institutions as assistant professors until they retire. Those who endorse the tenure-track system say that the probationary system allows low-performing researchers to remain in their posts. “How do we ensure that quick appointments to a very well paid, highly privileged and permanent position does not encourage complacency?” asks ecologist Vishwesha Guttal, who was awarded tenure in 2016 at the Indian Institute of Science, five years after he was hired. The issue of tenure-track versus one-year probation has sparked discussion and debate among academic scientists in India, partly in response to an interview published in June in the newspaper Hindustan Times with Jayant Udgaonkar, director of the Indian Institute of Science Education and Research in Pune. In the interview, he advocated adopting the tenure-track system nationally. His comments followed the release of a draft policy in May by the Ministry of Human Resource Development, which oversees higher education in India, recommending a gradual national adoption of the tenure-track system. The ministry could not be reached for comment. Udgaonkar, a biochemist, says that it is difficult to properly assess a researcher’s progress in a single year. He thinks that the tenure-track system provides scientific accountability and allows a candidate who has been given strong support and regular feedback to receive a comprehensive assessment at the end of five or sometimes seven years. But many do not agree. Theoretical physicist Arvind, director at the Indian Institute of Science Education and Research campus in Mohali, says that a five-year tenure track will increase job insecurity and put pressure on new faculty members to pursue only short-term research goals during that period. “Academia requires stability,” he says, adding that there is a paucity of fallback options for candidates who don’t make the cut. India has few second- or third-tier research institutions where a scientist whose bid for tenure is rejected elsewhere can seek another appointment, and few commensurate industry positions. Institutional support, easy access to equipment and resources, and timely disbursal of government grant funds have long been sore points in Indian academia; they have also been a talking point in discussions about adopting a tenure system. Gagandeep Kang, an academic gastroenterologist and executive director of the Translational Health Science and Technology Institute in Faridabad, says that institutions and government need to improve access to funding and resources to level the playing field for researchers who are up for tenure and allow for a more-rigorous review process. Ramaswamy Subramanian, a structural biologist and director of the Bindley Bioscience Center at Purdue University in West Lafayette, Indiana, says that if tenure is adopted, the process will need to be uniformly objective and fair. Subramanian, who has held tenured positions in Sweden and the United States, points out that tenure-review committee members in India, usually senior scientists and administrators, are likely to lack personal experience of the process. A nationwide system is unlikely to be adopted soon, predicts Arvind. “Each institution is autonomous,” he says. “There may, at best, be suggestions that the governing boards of individual institutions can then consider.” If you have a career story that you'd like to share, then please complete this form, or send your outline by email.
Q&A: Navigating academia and industry in India
What do you do? Do you enjoy it? Why? I am a doctoral candidate at the National Institute of Technology Karnataka. My dissertation focuses on intelligent systems for predictive modelling in financial applications. Aside from my PhD research topic, I enjoy working on problems that have the potential to make an impact through different aspects of research such as social, business and technology. Through my PhD programme, I have moulded research skills to work on funded projects or serve as independent consultant. What were your early career ambitions? After not performing well at school, my parents suggested that I take up a course at an ITI (Industrial Training Institute). This meant that I would be a skilled technician in an organisation after graduating. However, I decided to study for a Diploma in Electronics and Communication Engineering, which is technically a higher degree, as I did not want to limit my career prospects. The initial semesters were tough, but later I picked up with the help of learning in peer student groups. How did you make the decision between pursuing a career in academia or industry? Once I completed my diploma, thoughts of applying for jobs surfaced. However, at that time in India, a new scheme was introduced whereby diploma holders could apply and join the second year of an undergraduate programme. I stayed at the same institute where I had studied for my diploma and was awarded a BTech in Information Technology in 2007. Around this time, the economic recession was prevailing, the dot-com bubble had mostly subsided and the industry in India was changing rapidly. I felt that my place was not in industry, as I did not believe I had the excellent coding skills it required. I took up my first job as a lecturer on a contractual basis at Cochin University Engineering College at Kuttanad, which is a state-funded public university. After my first stint as a lecturer, I personally felt that academia provided a better comfort zone and space for professional growth. What were some of the challenges on your journey? Most of colleges where I worked already started to regulate for more qualified (PG/PhD) teaching staff. I was interviewed and received a job offer from Amrita University in Quilon in early 2008. But in that year, a breakthrough occurred while I was trying to qualify for GATE (Graduate Aptitude for Engineering), a national exam that provide chances to pursue Master’s and PhD programmes. More recently, this exam has been a criterion for selection for some positions public sector companies. I received a strong grade and rank, which I don’t think I would have got without the exposure and subject knowledge I’d acquired during my undergraduate training. What did you do next? After unsuccessful applications to graduate schools for Master’s programmes, I worked for a short period as technical staff at an institute for a government funded project on digitisation of a library with nationwide reach. Later in 2009, I applied for an MTech (Master of Technology) programme at a state public university and was selected for a teaching assistance scholarship. As part of the dissertation project work, I applied for an internship at ISRO (Indian Space Research Organization). Even without support from the university or a scholarship from ISRO, I worked on a two-semester project on the development of a software tool prototype for space research applications, which resulted in a related IEEE publication. How does being based in India affect the way you work? There have not been many drastic changes in India, from academic point of view, in recent years. There are constant checks and performance reviews either in government posts or private institutions. To an extent, although private institutions offer higher salaries, they also demand a higher workload as part of accreditations that may actually work positively in long run. What advice would you have for others trying to work in a similar sort of environment? There can be a sense of lethargy and inertia certain positions. The best policy is to keep searching for grants for funded projects, extend your professional skills, such as research reviewing and talking at conference and workshops. Undertake student support programmes like mentoring and community initiatives for spreading knowledge. What do you love about living and working in India? In my case, the government funded my research and hence I feel a sense of moral duty to give back to my nation. India has potential for growth both scientifically and economically; at least historically that has been evident. What’s your top career tip to younger colleagues? Stay focused and keep your eyes open for higher education and research opportunities. Reach out to your seniors, teachers and peers for advice. What else would you say to others trying to build a scientific career in India? From my experience, joining the best-ranked institute does not necessarily mean you will receive top training or skills, unless you have a true passion for your research. Smart work and motivation can instil students with the confidence to perform well and be recognised in academia. Make use of generous government scholarships as well as privately funded schemes.