faulker045

By targeting data locality, full table scans of HBase using MapReduce across 373 million records are reduced from 19 minutes to 2.5 minutes.  We've been posting some blogs about HBase Performance which are all based on the PerformanceEvaluation tools supplied with HBase.  This has helped us understand many characteristics of our system, but in some ways has sidetracked our tuning - namely investigating channel bonding  to help increase inter machine bandwidth believing it was our primary limitation.  While that will help for many things (e.g. the copy between mappers and reducers), a key usage pattern involves full table scans of HBase (spawned by Hive ) and in a well setup environment network traffic should be minimal for this.  Here I describe how we approached this problem, and the results. The environment We run Ganglia for cluster monitoring (and ours is public ) and Puppet to provision machines.  As an aside, without these tools or an equivalent ...

Hive 0.9.0 was  released  at the beginning of this month and it contains a lot of very nice improvements. Thanks to all involved! Unfortunately it drops compatibility with HBase 0.90.x due to two issues which introduced a dependency on HBase 0.92: https://issues.apache.org/jira/browse/HIVE-2748 https://issues.apache.org/jira/browse/HIVE-2764 Fortunately these were relatively easy to revert so that's what we did because we wanted to all the 0.9.0 goodness on our HBase 0.90.4 cluster (CDH3u3). I've forked Hive on Github and reverted the parts of those two issues ( HIVE-2748 , HIVE-2764 ) that were causing problems. For all those "stuck" with HBase 0.90 (e.g. CDH3 users) we've also deployed this custom Hive HBase Handler to our own Maven repository and will maintain that for the foreseeable future. You can just download the jar file and use it in your projects or use our Maven repository: gbif-thirdparty http://repository.gbif.org/content/repositories/thirdpart...

Update: See also part 1 and part 3 . In my  last post  I described my initial foray into testing our HBase cluster performance using the PerformanceEvaluation class.  I wasn't happy with our conclusions, which could largely be summed up as "we're not sure what's wrong, but it seems slow".  So in the grand tradition of people with itches that wouldn't go away, I kept scratching.  Everything that follows is based on testing with PerformanceEvaluation (the jar patched as in the  last post ) using a 300M row table built with PerformanceEvaluation sequentialWrite 300 , and tested with PerformanceEvaluation scan 300 .  I ran the scan test 3 times, so you should see 3 distinct bursts of activity in the charts.  And to recap our hardware setup - we have 3 regionservers and a separate master. The first unsettling  ganglia metric  that kept me digging was of ethernet bytes_in and bytes_out.  I'll recreate those here: Figure 1 - bytes_in (MB/...

Update: See also followup posts: part 2 and part 3 . In the last post Lars talked about  setting up Ganglia  for monitoring our Hadoop and HBase installations.  That was in preparation for giving HBase a solid testing run to assess its suitability for hosting our index of occurrence records.  One of the important features in our new Data Portal will be the "Download" function that lets people download occurrences matching some search criteria and currently that process is a very manual and labour intensive one, so automating it will be a big help to us.  Using HBase it would be implemented as a full table scan, and that's why I've spent some time testing our scan performance. Anyone who has been down this road will probably have encountered the myriad opinions on what will improve performance (some of them conflicting) along with the seemingly endless parameters that can be tuned in a given cluster.  The overall result of that kind of research is: "Yo...

We're getting serious in our Hadoop adoption. The first process (our so called "rollover") is now in production and it uses Hadoop, Hive, Oozie and various other parts of the Hadoop ecosystem. Our next step is evaluating HBase and its performance on our (small and aging) cluster. To do that properly and to fix a rather embarrassing situation we first had to get proper monitoring up and running for our cluster. So far we've only had Cacti stats for OS level things (CPU, I/O, etc.) but we were missing actual Hadoop statistics. So we've now set up Ganglia at GBIF and the best news is it's public  and using the very latest Ganglia 3.3 which was released only a few days ago in February 2012. The setup was relatively painless. Ganglia was just nice to work with. To get monitoring of HBase working we had to apply HBASE-4854  because it's not included in our Hadoop distribution (CDH3u2). Thanks to Lars George for the hint. So we can happily report that Ganglia 3...

Guest post from Jörg Holetschek, Botanic Garden and Botanical Museum Berlin-Dahlem. The traditional way of sharing occurrence data with GBIF has been web-service-based for years. Data publishers have used one of the existing provider software packages ( DiGIR , BioCASe or TAPIR Link ) to expose their data as a DiGIR-, BioCASe- or TAPIR-compliant web service. Biodiversity networks such as GBIF used harvesters to crawl and index the records published by these services, an approach that works fine for small and medium-sized datasets, but runs into difficulties when record numbers hit the millions: Harvesting can take days and puts a heavy load on both the publisher and the crawler. To overcome this, GBIF recently introduced DarwinCore Archives for storing all information of a dataset to be published in a single file. GBIF directly ingesting this file eliminates the time-consuming back-and-forth communication between data provider and harvester, speeding up the process and reducing load f...

This post originally appeared on the Canadensys blog and is a follow-up of the post Customizing the IPT . As mentioned at the very end of my post about customizing the IPT , I face a problem when I want to install a new version of the GBIF Integrated Publishing Toolkit : installing it will overwrite all my customized files! Luckily Tim Robertson gave me a hint on how to solve this: a shell script to reapply my customization. Here's how it works (for Mac and Linux systems only): Comparing the customized files with the default files First of all, I need to compare my customized files with the files from the new IPT. They might have changed to include new functionalities or fix bugs. So, I installed the newest version of IPT on my localhost , opened the default files and compared them with my files. Although there are tools to compare files, I mostly did this manually. The biggest change in version 2.0.3 was the addition of localization, for which I'm using a different UI, so I h...