{"id":307,"date":"2014-10-08T05:17:03","date_gmt":"2014-10-08T05:17:03","guid":{"rendered":"https:\/\/www.lianeallen.com\/home\/?p=307"},"modified":"2014-10-08T05:17:03","modified_gmt":"2014-10-08T05:17:03","slug":"male-vs-female-biographies-on-wikipedia","status":"publish","type":"post","link":"https:\/\/www.lianeallen.com\/home\/2014\/10\/male-vs-female-biographies-on-wikipedia\/","title":{"rendered":"Male vs Female Biographies on Wikipedia"},"content":{"rendered":"

I attended the Boston Girl Geeks Dinner<\/a> last week, and left with a mission – to extract the names from all the biographies on wikipedia, and analyze them for gender. <\/p>\n

There’s a gem that will compare a first name to a 40,000 name dictionary, and determine whether that name is female, male, mostly female, mostly male, or androgynous\/unknown. <\/p>\n

Wikipedia has an api, but it’s a long, convoluted process to get at just the names of biographical subjects. However, they also have a set of pages that organize biographies according to the quality of the bio. Each of those pages lists … the *names* of the biography subjects. Bingo!<\/p>\n

So, I got to learn how to extract xpath data from a page, and do a bit of work to extract exactly what I need from the data (the names are not the only text in the resulting string, so I had to strip the extra content out), then run it through the gender tool.<\/p>\n

I’ve made the tool as gentle on wikipedia as possible – there’s no automated running of the whole extraction, each page has to be run separately, by hand, so it won’t create a big load and thus accidentally get my IP address banned. I used a really small page during coding, so the incessant testing would only be grabbing a single page and a few lines of data.<\/p>\n

And it works! <\/p>\n

Next up: enabling people to edit records, to handle cases where the tool couldn’t identify the gender, and adding statistical analysis.<\/p>\n","protected":false},"excerpt":{"rendered":"

I attended the Boston Girl Geeks Dinner last week, and left with a mission – to extract the names from all the biographies on wikipedia, and analyze them for gender. There’s a gem that will compare a first name to a 40,000 name dictionary, and determine whether that name is female, male, mostly female, mostly … <\/p>\n

Continue reading “Male vs Female Biographies on Wikipedia”<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":308,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-307","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/www.lianeallen.com\/home\/wp-json\/wp\/v2\/posts\/307","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.lianeallen.com\/home\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.lianeallen.com\/home\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.lianeallen.com\/home\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lianeallen.com\/home\/wp-json\/wp\/v2\/comments?post=307"}],"version-history":[{"count":1,"href":"https:\/\/www.lianeallen.com\/home\/wp-json\/wp\/v2\/posts\/307\/revisions"}],"predecessor-version":[{"id":309,"href":"https:\/\/www.lianeallen.com\/home\/wp-json\/wp\/v2\/posts\/307\/revisions\/309"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.lianeallen.com\/home\/wp-json\/wp\/v2\/media\/308"}],"wp:attachment":[{"href":"https:\/\/www.lianeallen.com\/home\/wp-json\/wp\/v2\/media?parent=307"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.lianeallen.com\/home\/wp-json\/wp\/v2\/categories?post=307"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.lianeallen.com\/home\/wp-json\/wp\/v2\/tags?post=307"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}