Coding Qualitative Data

A friend of mine recently pointed me towards MAXQDA for coding and parsing qualitative research. Too bad I just wrote a post on how garden-variety relational databases could be hornswoggled for the task. I was so proud of my handwritten beta, too….

A couple of quick web searches turned up NVivo and XSight, by QSR, QDA Miner by Provalis, and Atlas.ti. TAMS for Mac OSX may be the most honestly titleds: text analysis markup system.

And sure enough someone has been on the free and open source (FOSS) track. Weft QDA. Dexter. Transana.

• UPDATE: The CDC (United States) publishes AnSWR at zero cost.

And a review site or two for multi-methods CAQDAS research tools. Clearly I have some reading to do.

• UPDATE: There are a multitude of review sites, often hosted at university social science departments (e.g., sociology, ethnography, psychology), too many to list here and I’m not sure how to categorize them.

Please comment if you have worked with these packages and can recommend a way of organizing them by functionality and quality. There does not seem to be a single standard for what the packages ought to do, and how to do it well.

Tracking research interactions

How much information should the researcher keep about each site? Each interview? The answer, of course, is “all of it.” This can be an enormously time-consuming task, depending on the richness of the information the interviewer needs to collect about the site, the subject, and the interview instrument to be used.

Relational databases are purpose built for this sort of task. In a relational database, the user enters all the relevant information about each entity once, and only once. Whenever it is needed in the future, the database query looks up all the relevant bits of information from as many places as necessary for the task at hand. Many vendors are out there (Access, Filemaker, SAS, Oracle), but some of them are free and open source (MySQL, SugarCRM) and do not require years of study to become competent (OpenOffice).

To reiterate, you don’t need any money, and you don’t need a computer science degree to track your interviews in a relational database, but you can save yourself a ton of time.

Hardly anybody [I know] reads…

Bill Easterly, one of the most accomplished and iconoclastic development economists and commentators, may have overreached a bit. He has picked a bone with the Global Forum for Health over their failure to win publications and citations in journals worthy of consideration at tenure review. Isn’t that a strange criterion for judging the success of the Global Forum for Health? They are insufficiently concerned with winning faculty jobs at US universities?

The evaluation has lots of other things to say about them, both positive (they have a lot of meetings!) and negative (“absence of an agreed results framework”).

So the World Bank seems to be following a disturbing trend. It is financing economic research publications that hardly anybody reads, and financing health awareness efforts that hardly anybody is aware of.

This also creates a new challenge for aid watchers – how can we hold accountable an aid agency we don’t know exists? What other dark matter in the aid world is awaiting discovery?

Global Forum for Health is a self-described advocacy organization. If anything, Bill Easterly should be taking the development economics profession to task for its solipsistic criteria for success. In manufacturing industries, the gold standard of achievement is not publications, it is patents. Unfortunately, the economics profession has no equivalent to the patent.

My problem with Easterly’s criticism is the standard for judgment: “…publications that hardly anybody reads.” An astute reader will insert brackets, so it reads “publications that hardly anybody [I know] reads.” Tenure review boards matter primarily to people whose jobs hinge on tenure decisions, and in places that read the same publications that Easterly does.

Academics overseas may have a very different idea of where to publish their research, and what research is credible, and what research is relevant to their careers. Tbe Economist recently stirred up a hornet’s nest with its report on the views of top economists on the profession, who accuse the discipline collectively of dogmatism and sloppy analysis (Barro, Buiter, DeLong, Eichengreen, Lucas, and Krugman). To the extent that publication in the top journals is both extremely time-consuming and something of a beauty contest, perhaps it isn’t reasonable to discount any and all research being done outside the top American journals.

IRB is incompatible with open access to data

Mathematician Arvind Narayanan at his blog 33 Bits writes a compelling post on the failure of efforts to protect the identities of individuals in public domain, scientific data sets. De-anonymization of data is not new in the field of electronic data privacy. This is the first discussion that I have read of the specific conflict between IRB approval and scientific peer review.

This thesis of this blog that the amount of entropy required to de-anonymize an individual — 33 bits — is low enough that it doesn’t offer meaningful protection in most circumstances. Obviously, the argument applies even more strongly to the anonymity of a well-defined group of people.

Let’s be clear: the paper is from 1994; who slept with whom in high school is not a huge deal a decade and a half later. However, the problem is systemic, and IRBs (Institutional Review Boards) keep blithely approving releases of data with such nominal de-identification applied. The re-identification of the institutional affiliation of an entire population of a study is of more concern from the privacy perspective than the de-anonymization of individual identities: it needs to be done only once, and affects hundreds or thousands of individuals.

Recently, a group of researchers from the Berkman Center released a dataset of Facebook profile information from an entire cohort (the class of 2009) of college students from “an anonymous, northeastern American university.” It was promptly de-anonymized by Michael Zimmer, who revealed that it was Harvard College:….

What makes a science wiki tick?

Nature has a great article discussing various business strategies for making a collaborative scientific resource work. What makes some grow and others wither? What returns does a scientific organization get on its investment in public, collaborative technologies? Is there a way to see that contributors and the host organization all reap the rewards of their work?

From the article in Nature.

The science wikis face a tougher challenge in building critical mass, if only because they’re aiming at a much smaller audience. One obvious strategy is to avoid fragmenting that audience. As Evelo points out, “biologists aren’t going to work on a dozen wikis to see which will survive”. They are going to want the various wikis to be interoperable and mutually supporting, so that the data they enter in one can be easily ported to another — or will even flow to all the appropriate sites automatically.