Rule out conflicts of interest in psychology awards

Unclear regulations on conflicts of interest (COI) might lead to nepotism in the handing out of academic prizes and awards awarded by professional societies and associations. To assess to availability of statements on COIs, we systematically searched the websites of 58 psychological societies. We found that 27.6% had no COI statement available on their website. 27.6% included a statement on COIs on their website, but did not mention awards or prizes specifically. Only 44.8% of the included societies had an online COI statement that included a section on awards. Therefore, the majority of the included societies did not feature a findable COI statement pertaining to awards on their website.

appears that COI statements and policies around COIs are less common within psychology than, for example, the medical sciences (Chivers, 2019). Therefore, we will study if any information on COIs can be obtained from websites of psychological societies.
As COIs seem to be a less salient issue in psychology than in medicine (Chivers, 2019), we hypothesized that the majority of the psychological societies (greater than or equal to 50%) will not have a detailed COI statement concerning the distribution of awards or prizes.

Method
To assess the availability of COI statements concerning the distribution of awards, we developed a pre-registered observational study. The pre-registration contains our hypothesis, research design and analysis plan, and can be found at https://osf.io/6txmj.

Sampled Societies
For the current study, we included all 65 psychological societies listed on the website of the Association for Psychological Science (retrieved on the 5th of July 2019; Association for Psychological Science, 2019). It is not clear what the specific inclusion/exclusion criteria were when they compiled this list, but on the website it states: "The list below includes organizations from around the world that are focused on psychological science." The full list of societies can be found at https://osf.io/7hcds/. Due to language restrictions, we were unable to code two societies whose websites were in French and Spanish 1 , respectively. We deviated from the preregistered protocol by excluding these societies.

Coders
We recruited the coders at a hackathon during the conference of the Society for the Improvement of Psychological Science (SIPS) 2019, in Rotterdam, the Netherlands. The coders volunteered their time. Due to the nature of the session, the coders were not blind to the hypotheses nor to which website they were coding.
The order of the societies was randomized in R (R Core Team, 2019), and participants in our SIPS session were asked to code the societies in order of appearance on this list. We deviated here from the pre-registered protocol (see Appendix 1). According to the protocol, we would assign societies to coders, but this manner appeared to be too time-intensive and unpractical during the session. Instead, coders assigned themselves to societies in a Google Spreadsheet by choosing the first uncoded society of the randomized list. One coder deviated from the protocol and did not choose the next society on the list. To ensure objective coding, a second coder recoded all these societies (N = 3). Furthermore, during the session it appeared that not all the websites of the societies were in English. We asked coders to skip websites that were in a language they did not speak 2 . Later on, other coders who did speak the language scored these societies 3 . Finally, one coder skipped the Australian College of Psychologists, as they could not find their website via a Google search 4 .
We completed the pre-registration before any of the authors or the coders observed the data. We only inspected the website of the 'Deutsche Gesellschaft für Psychologie (German Psychological Society)' before pre-registration, as part of a small pilot to assess the protocol. We did include this society in our final sample.
To assess the accuracy of the coding, we selected a random sample of 15 societies in R (https://osf.io/gtv2b/; excluding societies with non-English or non-German websites 5 , and societies coded by the coder who deviated from protocol 6 ) to be reevaluate by a second coder.
We recoded the societies coded by the coder who deviated from protocol. In total, we recoded 18 societies. 66.67% of the assigned ratings coincided. From the six societies with deviating ratings, two societies did not hand out awards and we excluded them from our sample. Excluding these 2 Excluding the Czech-Moravian Psychological Society, which was randomly assigned to a coder who spoke Czech. 3  societies, 4 out of 16, or 75% of the ratings coincided. One society was coded as having a COI statement by coder 1 whereas coder 2 could not retrieve this statement. For one society, coder 1 did not find a statement on COI pertaining to rewards, but coder 2 did. For two societies, coder 1 did find statements on COI relating to awards, whereas coder 2 did not find these statements. All conflicts were resolved in a discussion between the second coders.

Procedure
At the session (for details, see https://osf.io/6txmj ), all coders were asked to rate the societies by following our pre-registered protocol (https://osf.io/j6ycx/ ). They searched and recorded COI statements on the websites of the sampled societies. Coders first inspected the websites of the societies themselves, by looking into the suggested sections: About Us, Governance, Goals, Guidelines, Code of Conduct, Jury/Committee, Conflict of Interest (COI). If this visual inspection yielded no results (i.e., the coder did not find a statement), coders Googled 7 specific search terms within the domain of the website to minimize the chance that the statement was missed on accident. The following search terms were used: Committee, Jury, Award, Guideline, Code of conduct, Conflict of interest/COI. After the visual inspection and the Google search, the coders rated the statements of the societies on a scale from one to three: 1. No statement on conflicts of interest is findable on the website. 2. A conflict of interest statement is mentioned on the website, but awards are not named.
3. Statement of conflicts of interest in relation to awards or prizes is available on the website. 7 In the pre-registerd protocol, we wrote: 'Search engine such as Google'. However, to keep the data collection as standardized as possible, we asked the coders to use Google only, and, by this, deviated from protocol.
We did not ask coders to assess the quality of the statements, just their presence, but we plan to do this in a follow-up study.
It is possible that the protocol left some variability among the coders. The protocol did not explicitly state whether grants (such as travel grants) should be considered 'awards'.
Moreover, under the current protocol a society received a score of 3 if there was at least one statement on COIs addressing awards, even though multiple awards may be awarded by the society. Lastly, some societies included statements on how awards should be distributed, but did not mention COIs specifically. We coded these societies with 3, as a rating 1 would not reflect the societies' guidelines.

Data Analysis
We calculated the percentages of societies falling in category 1, 2, or 3 (https://osf.io/zjy4s/). We did not perform any statistical hypothesis test, because our sample is non-random (it is unclear how the Association for Psychological Science compiled the list of psychological societies). Our conclusions only pertain to the 63 societies we coded. Considering the deviations from protocol, we decided to include an exploratory sensitivity analysis were we exclude all societies were associated with deviations from protocol.
Missing data. We predicted we would not obtain any missing data. However, we had to exclude two societies due to language issues, and one society that appeared not to have a website 8 . Moreover, four societies did not hand out awards 9 . We excluded these seven (10.8%) societies from our analyses.

Confirmatory Analyses
We computed what percentage of the 58 included psychological societies had either no conflict of interest statement (1), a conflict of interest statement but not pertaining to awards (2), or a conflict of interest statement pertaining to awards (3). Table 1 shows the results of this analysis. Overall, the majority of the websites, 72.4%, did not have a findable statement on COIs concerning the distribution of awards on their website, which is in line with our hypothesis.

Exploratory Analyses
We investigated if any of the deviations from our protocol caused any bias in the scoring of the societies. To that end, we conducted an exploratory sensitivity analysis in which we excluded all societies that were a) in a language other than English 10 or b) coded not according to protocol 11 . In total, we excluded six additional societies (10.3%). Based on the remaining 53 societies, we re-calculated the three percentages (see Table 2). When comparing Tables 1 and 2, it appears that including societies that were not scored according to the protocol did not greatly influence the results. Therefore, we will interpret the results of our confirmatory analysis.

Discussion
For 44.8% of the 58 psychological societies we could not find any statement on avoiding conflicts of interest on their website. 27.6% did include a COI statement on their website, but did not explicitly include guidelines towards the distribution of awards in this statement. 27.6% society websites included statement on conflicts of interest related to awards.
We conclude that the majority of the psychological societies did not include a findable statement on COIs concerning awards on their website. This is an alarming number, considering the importance of academic awards to researchers' careers.
It should be noted that societies might have statements, but that we were unable to retrieve them. However, considering the search areas and terms in our protocol, we deem it likely that we would have obtained the statement if it were published on a society's website (especially given the fact that we were also able to find COI statements that were buried deep in a website's archive of emails sent out years ago). Moreover, one can question the usefulness of a COI statement if the statement is not readily available. Lastly, we should note that we did not contact societies to investigate whether a COI statement was available on their non-public domain. I.e., it may be that the statement is only available to members. We will address this caveat in a followup project. We do believe that COI statements should be public, such that anyone, including nonmembers, considering its importance not only to awards, but also to all aspects of a society.
The current study only focused on a relatively small sample of psychological societies. It may be that the current sample was not representative of all psychological societies (e.g., some appeared not to exist anymore). Therefore, we limit our conclusions to the included societies. We aim to address these issues in a follow-up study.
As addressed, we only investigated the presence of COI statements and not their quality.
Therefore, it may well be that societies with available but vague COI statements received high ratings. A follow-up project may assess these quality concerns.
In sum, we showed that in our subset 55.2% of the societies featured some conflict of interest statement. Only half of these societies (27.6%) included a statement on how conflicts of interest should be handled in relation to academic awards. Therefore, we conclude that the majority of the investigated societies does not have COIs statements pertaining to awards, which is worrisome considering the impact of awards on academic careers.
With this protocol we aim to systematically search through the website of a psychological association of interest to find any relevant information regarding conflict of interest (COI) statements that relate to the members of the jury or committee that is responsible for selecting the awardee.
The search procedure consists of the following steps. Using the parameter "site:sitename.xyz" to limit the search to the website, e.g.
"site:www.psychology.org.au" would limit the search to the specified website (www.psychology.org.au , the website of the Australian psychological society in this case) and can be combined with any number of the search terms mentioned above.
The filtered content is then scanned for relevant information. Relevant information refers to any statements regarding Jury/Committee members responsible for awards, which are in any way related to conflicts of interests.
A society is considered to have a sufficient conflict of interest policy with respect to awards when it can reasonably be assumed that its jury/committee members are expected to avoid any situation that could potentially be considered a conflict of interest. A conflict of interest in this context is defined as follows: "A conflict of interest (COI) is a situation in which a person or organization is involved in multiple interests, financial interest, or otherwise, one of which could possibly corrupt the motivation of the individual or organization. Statements of societies will be categorised into different groups: 1. No findable conflict of interest statement on website 2. Conflict of interest statement is mentioned on website, but awards are not mentioned To make data collection as easy and as reproducible as possible, we would like to ask you to follow this workflow.
1. Request to be a contributor to our OSF page.
2. To avoid any biases in coding, we will assign you a society, just ask Andrea 3. Enter your name in the Google Spreadsheet behind the assigned society.
a. https://docs.google.com/spreadsheets/d/1TRn4As4PVjTsEGIcza5KYfYHYjZTe4 DKjyKGYTkWHg8/edit?usp=sharing 4. Go to the society's website and try to find the COI statement, following on the protocol on the OSF page.
5. Code the statement in the Google Spreadsheet, and leave a comment to explain the coding 6. Please take screenshots of the website where you found the information and upload it to the OSF. We will create a folder per society where you can upload the images.
a. Please save the screenshots as COI_NAMESOCIETY_XX (xx indicates the number of the screenshot, if you made multiple) b. If possible, download the website for offline use (on the webpage, right click, save as, full webpage). Please also upload these files to the OSF folder of the society.
If you are finished and feel like coding another society, please come and see Andrea If you made screenshots but failed to upload them to the OSF, you can email them to: a.h.stoevenbelt@tilburguniversity.edu