Ameliorating MELIBEA’s Open Access Policy Evaluator



SUMMARY: The MELIBEA evaluator of Open Access policies could prove useful in shaping OA mandates — but it still needs a good deal of work. Currently it conflates institutional and funder policies and criteria, mixes green and gold OA criteria, color-codes in an arbitrary and confusing way, and needs to validate its weights (e.g., against policy success criteria such as the percentage and growth rate of annual output deposited since the policy was adopted).

The MELIBEA Open Access policy validator is timely and promising. It has the potential to become very useful and even influential in shaping OA mandates — but that makes it all the more important to get it right, rather than releasing MELIBEA prematurely, when it still risks increasing confusion rather than providing clarity and direction in OA policy-making.

Remedios Melero is right to point out that — unlike the CSIC Cybermetrics Lab‘s ‘s University Rankings and Repository Rankings — the MELIBEA policy validator is not really meant to be a ranking. Yet MELIBEA has set up its composite algorithm and its graphics to make it a ranking just the same. 

It is further pointed out, correctly, that MELIBEA’s policy criteria for institutions and funders are not (and should not be) the same. Yet, with the coding as well as the algorithm, they are treated the same way (and funder policy is taken to be the generic template, institutional policy merely an ill-fitting special case). 

It is also pointed out, rightly, that a gold OA publishing policy is not central to institutional OA policy making — yet there it is, contributing sizeable components to the MELIBEA algorithm.

It is also pointed out that MELIBEA’s green color code has nothing to do with the “green OA” coding — yet there it is — red, green yellow — competing with the widespread use of green to designate OA self-archiving, and thereby inducing confusion, both overt and covert.

MELIBEA could be a useful and natural complement to the ROARMAP registry of OA policies. I (and no doubt other OA advocates) would be more than happy to give MELIBEA feedback on every aspect of its design and rationale.

But as it is designed now, I can only agree with Steve Hitchcock’s points and conclude that consulting MELIBEA today would be likely to create and compound confusion rather than helping to bring the all-important focus and direction to OA policy-making that I am sure CSIC, too, seeks, and seeks to help realize.

Here are just a few prima facie points:

(1) Since MELIBEA is not, and should not be construed as a ranking of OA policies — especially because it includes both institutional and funder policies — it is important not to plug it into an algorithm until and unless the algorithm has first been carefully tested, with consultation, to make sure it weights policy criteria in a way that optimizes OA progress and guides policy-makers in the right direction.

(2) For this reason, it is more important to allow users to generate separate flat lists of institutions or funders on the various policy criteria, considered and compared independently, rather than on the basis of a prematurely and arbitrarily weighted joint algorithm.

(3) This is all the more important since the data are based on less then 200 institutions, whereas the CSIC University Rankings are based on thousands. Since the population is still so small, MELIBEA risks having a disproportionate effect on initial conditions and hence direction-setting; all the more reason not to amplify noise and indirection by assigning untested initial weights without carefully thinking through and weighing the consequences.

(4) A potential internal cross-validator of some of the criteria would be a reliable measure of outcome — but that requires much more attention to estimating the annual size and growth-rate of each repository (in terms of OA’s target contents, which are full-text articles), normalized for institution size, annual total target output (an especially tricky denominator problem in the case of multi-institutional funder repositories) and the age of the policy. Policy criteria (such as request/require or immediate/delayed) should be cross-validated against these outcome measures (such as percentage and growth rate of annual target output) in determining the weights in the algorithm.

(5) The MELIBEA color coding needs to be revised — and revised quickly, if there is to be an algorithm at all. All those arbitrary colors in the display of single repositories as ranked by the algorithm are both unnecessary and confusing, and the validator is not comprehensibly labelled. The objective should be to order and focus clearly and intuitively. Whatever is correlated with more green OA output (such as a higher level or faster growth rate in OA’s target content, normalized) should be coded as darker or bigger shades of green. The same should be true for the policy criteria, separately and jointly: in each case, request/require, delayed/immediate, etc., the greenward polarity is obvious and intuitive. This should be reflected in the graphics as well as in any comparative rankings.

(6) If it includes repositories with no OA policy at all (i.e., just a repository and an open invitation to deposit) then all MELIBEA is doing is duplicating ROAR and ROARMAP, whereas its purpose, presumably, is to highlight, weigh and compare specific policy differences among (the very few) repositories that DO have policies.

(7) The sign-up data are also rather confusing; the criteria are not always consistent, relevant or applicable. The sign-up seems to be designed to make a funder-mandate the generic option, whereas this is quite the opposite of reality. There are far more institutions and institutional repositories and policies than there are funders, many of the funder criteria do not apply to institutions, and many of the institutional criteria make no sense for funders. There should be separate criterial lists for institutional policies and for funder policies; they are not the same sort of thing. There is also far too much focus and weight on gold OA policy and payment. If included at all, they should only be at the end, as an addendum, not the focus at the beginning, and on a par with green OA policy. 

(8) There is also potential confusion on the matter of “waivers” or “opt-outs”: There are two aspects of a mandate. One concerns whether or not deposit is required (and if so, whether that requirement can be waived) and the other concerns whether or not rights-reservation is required (and if so, whether that requirement can be waived). These two distinct and independent requirements/waivers are completely conflated in the current version of MELIBEA.

I hope there will be substantive consultation and conscientious redesign of these and other aspects of MELIBEA before it can be recommended for serious consideration and use.

Stevan Harnad
American Scientist Open Access Forum

The role of the research library in an emerging global public sphere (LIBER keynote)

The text and powerpoint of my keynote at the 39th annual LIBER (Association of European Research Libraries) Conference is now available.

Abstract

Presents a vision of a potential future global public sphere, why it is needed and signs of emergence, and the role of the research library in this global public sphere, as provider of a distributed knowledge commons, preserver of scholarly information, and source of specialized expertise. Key short-term transitional steps are covered, particularly transition to a fully open access scholarly publishing system.

Note: I’ll post a link to the webcast once this is available.

Funders Should Mandate Institutional Deposit (and, if desired, central harvest)

SUMMARY: The most effective and natural way to ensure that all institutions — the universal providers of all research, funded and unfunded, in all fields — provide open access (OA) to all of their peer-reviewed research (funded and unfunded, in all fields) is for both funders and institutions to mandate cooperative, convergent deposit, by the author, in the author’s own institutional repository, rather than competitive, divergent institutional-and/or-institution-external deposit by authors-and/or-publishers.


1. It is important for OA advocates to understand that it is not PubMed Central (PMC) that is making biomedical articles open access (OA) — it is the depositors of those articles. In the case of PMC, those depositors are authors (and publishers). PMC is serving both as a locus of deposit (i.e., a central, subject-based repository) and as a locus of search and use (like google).

2. The reason PMC has as much OA content as it does is the NIH OA mandate, requiring deposit — not the fact that PMC is a central locus of deposit.

3. The NIH mandate is a very good thing, and if OA is desired in other funded fields, the funders should mandate OA, as NIH and other funders have done (and as FRPAA is proposing to do).

4. But the fact that NIH requires direct deposit in PMC is not a good thing, even though it is being mindlessly imitated by some other funders and fields.

5. The universal providers of all research, funded and unfunded, in all fields, are institutions (the world’s universities and research institutes).

6. Hence, in order to ensure that all research, funded and unfunded, in all fields is made OA, it is essential that the universal providers, the institutions, mandate providing OA to their own research output.

7. Most research universities already have their own institutional repositories (IRs), but most do not yet require their researchers to deposit their research output in them; for universal OA, the world’s universities and research institutes must all mandate OA.

8. But if funder mandates stipulate that deposit must be institution-external (e.g. in PMC, or other institution-external repositories) then funder mandates not only fail to encourage and reinforce the incentive of institutions (the universal providers) to mandate OA (for all the rest of their research output, funded and unfunded, in all fields), but they compete with and discourage it, by needlessly requiring direct deposit elsewhere (and hence the deterrent prospect of authors having to do duplicate or multiple deposit, should their institution consider mandating OA for all of its research output).

9. For funders to split the task of fulfilling their OA mandates arbitrarily between authors and publishers is even more counterproductive, making it ambiguous who must deposit what, and ceding responsibility and control over when the deposit is done to publishers, whose own interests are not aligned with doing but with delaying deposit.

10. For these reasons, funders should unequivocally assign full responsibility for complying with their OA mandates to their fundees (and their institutions); those are in any case the only parties actually bound by or beholden to the funder’s mandate — and their own interests are aligned with immediate deposit rather than delay. 

11. The most reliable, effective and natural way to ensure that all institutions — the universal providers of all research, funded and unfunded, in all fields — provide OA to all of their peer-reviewed research articles (funded and unfunded, in all fields) is accordingly for both funders and institutions to mandate cooperative, convergent deposit, by the authoronce — in the author’s own own institutional repository, rather than competitive, divergent institutional-and/or-institution-external deposit by authors-and/or-publishers.

12. Central collections like PMC can then be harvested (or automatically exported, e.g. via SWORD) from the distributed, mandated IRs, rather than being deposited (or double-deposited) into directly.  (General web content providers do not deposit directly in Google either; they host their own content locally and then Google and others harvest it!)

13. Of course, if an author or institution does not yet have an IR, it is fine to deposit in a central IR like DEPOT that has been created specifically for that purpose.

14. It has been a historic (but correctable) strategic mistake (by NIH and its imitators) to cause funder and institutional mandates to diverge and compete, because funders needlessly insist upon direct institution-external deposit.

15. There is no need to imitate this initial counterproductive trend, by founding still more central repositories and encouraging  funders to require direct deposit in them: 

16. This gratuitously dysfunctional trend should be countered, by encouraging funders to mandate institutional deposit, and by ensuring that central collections are harvested from distributed IRs, rather than being designated as the loci of direct deposit.

See:

Optimizing OA Self-Archiving Mandates: What? Where? When? Why? How?” 

How To Integrate University and Funder Open Access Mandates” 

Which Green OA Mandate Is Optimal?

Conflating OA Repository-Content, Deposit-Locus, and Central-Service Issues

Institutional vs. Central Repositories: 2 (of 2)

Institutional vs. Central Repositories: 1 (of 2)

Swan, A., Needham, P., Probets, S., Muir, A., Oppenheim, C., O?Brien, A., Hardy, R., Rowland, F. and Brown, S. (2005) Developing a model for e-prints and open access journal content in UK further and higher education. Learned Publishing 18 (1). pp. 25-40.

Stevan Harnad
American Scientist Open Access Forum

Good works (social housing) in progress in Vancouver

Observed in a completely non-scientific way in my last foray into the heart of the City of Vancouver: construction work underway on 3 sizable housing projects targeted to people in need of support as well as housing. Kudos to the Province of BC and the City of Vancouver for this work, especially during tough fiscal times.

Those of us who advocate for the public good should try to remember to notice and say thank you when our politicians are doing things right, not just critique when they aren’t. It is not likely that these few projects will solve the problems of homelessness or lack of support for the mentally ill in Canada, but they are a step in the right direction. If this is what my tax dollars are going to support, that’s a good thing, and I for one would like to see more along these lines.