Goodbye developer blog, hello data-blog!












GBIF has a new blog!











What is it?


A place for GBIF staff and guest bloggers to contribute:

  • Statistics 

  • Graphs 

  • Tutorials 

  • Ideas 

  • Opinions 



Who can contribute?


If you would like to contribute you can contact jwaller@gbif.org. Guest blogs are very welcome.


How can I write a post?


There is a short turtorial on the blog github.


What about the developer blog?


The developer blog will remain up as an archive, but there are no plans to actively post new content here.






Comments

Post a Comment

Popular posts from this blog

Optimizing Writes in HBase

Image
I've written a few times about our work to improve the scanning performance of our cluster (parts  1 , 2 , and  3 ) since our highest priority for HBase is being able to serve requests for downloads of occurrence records (which require a full table scan). But now that the scanning is working nicely we need to start writing new records into our occurrence table as well as  cleaning raw data and  interpreting it into something more useful for the users of our data portal . That processing is built as Hive queries that read from and write back to the same HBase table. And while it was working fine on small test datasets, it all blew up once I moved the process to the full dataset. Here's what happened and how we fixed it. Note that we're using CDH3u3, with the addition of Hive 0.9.0, which we patched to support HBase 0.90.4. The problem Our processing is Hive queries which run as Hadoop MapReduce jobs. When the mappers were running they would eventually fail (repe...
Read more

Lots of columns with Hive and HBase

We're in the process of rolling out a long awaited feature here at GBIF, namely the indexing of more fields from Darwin Core . Until the launch of our now HBase-backed occurrence store (in the fall of 2013) we couldn't index more than about 30 or so terms from Darwin Core because we were limited by our MySQL schema. Now that we have HBase we can add as many columns as we like! Or so we thought. Our occurrence download service gets a lot of use and naturally we want downloaders to have access to all of the newly indexed fields. The way our downloads work is as an Oozie workflow that executes a Hive query of an HDFS table (more details in this Cloudera blog ). We use an HDFS table to significantly speed up the scan speed of the query - using an HBase backed Hive table takes something like 4-5x as long. But to generated that HDFS table we need to start from a Hive table that _is_ backed by HBase. Here's an example of how to write a Hive table definition for an HBase-backed tab...
Read more

Darwin Core Archives for Species Checklists

Image
GBIF has long had an ambition for supporting the sharing of annotated species checklists through the network. Realising this ambition has been frustrated by the lack of a data exchange standard of sufficient scope and simplicity as to promote publication of this type of resource. In 2009, the Darwin Core standard data set was formerly ratified by the TDWG, Biodiversity Information Standards. The addition of new terms, and a means of expressing these terms in a simplified and extensible text-based format, paved the way for the development of a data exchange profile for exchanging species checklists known as the Global Names Architecture (GNA) Profile. Species checklists, published in this format, can be zipped into single, portable, 'archive' files. Here I introduce two example archives that illustrate the flexible scope of the format. The first represents a very simple species checklist while the second is a more richly documented taxonomic catalogue. The contents ...
Read more