Sebastián Pardo bio photo

Sebastián Pardo

Email Twitter LinkedIn Github   ORCID   ResearchGate   Google Scholar

Every now and them I spend some of my time playing with bird-related data. Lately I’ve been tinkering with code to interface with eBird, a global citizen science initiative that records bird sightings throughout the world, and even helps you maintain your own lists (for those of you who are listers, that is…).

The eBird API interface allows for downloading their most up-to-date latest checklist, and can be accessed in R using the ebirdtaxonomy function in the rebird package. Unfortunately, this checklist doesn’t include any high-level taxonomic information, such as order and family, for each species.

I was looking for a way to compare my eBird sightings across taxa, so after a while of searching for different taxonomic checklists and ways of merging them, I’ve come up with a way that’s only slightly convoluted to include higher taxonomic info into any eBird checklist.

library(dplyr)
library(tidyr)
library(lazyeval)
library(rebird)
library(tidyr)
library(xml2)

eBird’s taxonomic checklist

First, we load eBird’s taxonomic checklist and separate the scientific name into two columns, genus and species:

ebirdtax <- ebirdtaxonomy(cat = "species") %>% 
  separate(sciName, c("genus", "species"), sep = " ", remove = FALSE)
ebirdtax %>% select(genus, species, everything())
## Source: local data frame [10,473 x 11]
## 
##          genus       species speciesCode category                comName
##          (chr)         (chr)       (chr)    (chr)                  (chr)
## 1     Struthio       camelus     ostric2  species         Common Ostrich
## 2     Struthio molybdophanes     ostric3  species         Somali Ostrich
## 3         Rhea     americana     grerhe1  species           Greater Rhea
## 4         Rhea       pennata     lesrhe2  species            Lesser Rhea
## 5  Nothocercus        julius     tabtin1  species Tawny-breasted Tinamou
## 6  Nothocercus    bonapartei     higtin1  species       Highland Tinamou
## 7  Nothocercus nigrocapillus     hootin1  species         Hooded Tinamou
## 8      Tinamus           tao     grytin1  species           Gray Tinamou
## 9      Tinamus    solitarius     soltin1  species       Solitary Tinamou
## 10     Tinamus       osgoodi     blatin1  species          Black Tinamou
## ..         ...           ...         ...      ...                    ...
## Variables not shown: sciNameCodes (chr), sciName (chr), taxonID (chr),
##   taxonOrder (dbl), comNameCodes (chr), bandingCodes (chr)

This checklist has 10473 species and, as already mentioned, doesn’t contain any columns with order or family information. We need to obtain that information somewhere else and match it to this checklist. For ease of comparison throughout this post, we’ll just focus on distinct genera.

ebirdgen <- select(ebirdtax, genus) %>%
  distinct(genus)
ebirdgen
## Source: local data frame [2,226 x 1]
## 
##           genus
##           (chr)
## 1      Struthio
## 2          Rhea
## 3   Nothocercus
## 4       Tinamus
## 5  Crypturellus
## 6    Rhynchotus
## 7   Nothoprocta
## 8       Nothura
## 9     Taoniscus
## 10     Eudromia
## ..          ...

This leaves us with 2226 unique genera.

BirdLife’s taxonomic checklist

There are other organizations that also maintain their own complete bird species checklists, but do include information on order and families for each species. One of these is BirdLife International. We need to download the latest version of this checklist; for simplicity I’ve already turned this list into a .csv file which you can download here (you can also download the original .zip file with the complete information here).

A little bit of wrangling is required to make this checklist comparable with eBird’s taxonomy:

birdlifetax <- read.csv("~/Projects/stats-ebird/BirdLife-v8.csv", 
                        stringsAsFactors = FALSE) %>% tbl_df %>%
  select(Order, Family_name, Scientific.name, everything())
birdlifetax
## Source: local data frame [11,862 x 14]
## 
##               Order   Family_name          Scientific.name    Family
##               (chr)         (chr)                    (chr)     (chr)
## 1  STRUTHIONIFORMES Struthionidae         Struthio camelus Ostriches
## 2  STRUTHIONIFORMES Struthionidae   Struthio molybdophanes Ostriches
## 3  STRUTHIONIFORMES Struthionidae         Struthio camelus Ostriches
## 4  STRUTHIONIFORMES       Rheidae Pterocnemia tarapacensis     Rheas
## 5  STRUTHIONIFORMES       Rheidae           Rhea americana     Rheas
## 6  STRUTHIONIFORMES       Rheidae             Rhea pennata     Rheas
## 7  STRUTHIONIFORMES       Rheidae        Rhea tarapacensis     Rheas
## 8  STRUTHIONIFORMES       Rheidae                Rhea nana     Rheas
## 9  STRUTHIONIFORMES       Rheidae             Rhea pennata     Rheas
## 10 STRUTHIONIFORMES     Tinamidae       Nothocercus julius  Tinamous
## ..              ...           ...                      ...       ...
## Variables not shown: Common.name (chr), Authority (chr),
##   BirdLife.taxonomic.treatment (chr), IUCN_RL_2015 (chr), Synonyms
##   (chr), Alternative.common.names (chr), Taxonomic.notes (chr),
##   Taxonomic.sources. (chr), SISRecID (int), SpcRecID (int)

As you can see, this second checklist contains more species than eBird (11862). We’ll split the scientific name into genus and species, capitalize only the first letter of each order (using a small function) , and select all unique genera with their respective order and family columns:

capFirst <- function(x) {
  paste0(toupper(substring(x, 1, 1)), tolower(substring(x, 2)))
}

birdlifegen <- birdlifetax %>%
  separate(Scientific.name, c("genus", "species"), sep = " ") %>%
  mutate(order = capFirst(Order)) %>%
  select(order, family = Family_name, genus) %>%
  distinct(genus)
birdlifegen
## Source: local data frame [2,219 x 3]
## 
##               order        family        genus
##               (chr)         (chr)        (chr)
## 1  Struthioniformes Struthionidae     Struthio
## 2  Struthioniformes       Rheidae  Pterocnemia
## 3  Struthioniformes       Rheidae         Rhea
## 4  Struthioniformes     Tinamidae  Nothocercus
## 5  Struthioniformes     Tinamidae      Tinamus
## 6  Struthioniformes     Tinamidae Crypturellus
## 7  Struthioniformes     Tinamidae   Rhynchotus
## 8  Struthioniformes     Tinamidae  Nothoprocta
## 9  Struthioniformes     Tinamidae      Nothura
## 10 Struthioniformes     Tinamidae    Taoniscus
## ..              ...           ...          ...

The BirdLife checklist also contains more unique genera than eBird’s checklist. What is interesting to see is which ones (and how many) genera are different between these checklists.

anti_join(ebirdgen, birdlifegen, by = "genus") %>%
  select(genus) %>% distinct(genus)
## Source: local data frame [173 x 1]
## 
##            genus
##            (chr)
## 1        Euodice
## 2     Spermestes
## 3    Odontospiza
## 4   Paludipasser
## 5  Sporaeginthus
## 6      Granatina
## 7     Coccopygia
## 8   Pachyphantes
## 9     Carpospiza
## 10        Alario
## ..           ...

There are 173 genera in the eBird checklist that are not in the BirdLife checklist! It’s important to know that the difference from anti_join (and also setdiff) is asymmetrical, therefore the order in which we write the arguments matters. Just out of curiosity, we can have a look at the symmetric difference between these sets:

sym_diff <- function(a,b) unique(c(setdiff(a,b), setdiff(b,a)))

ebirdbl.diff <- sym_diff(ebirdgen$genus, birdlifegen$genus)
data.frame(genus = ebirdbl.diff) %>% tbl_df
## Source: local data frame [339 x 1]
## 
##              genus
##             (fctr)
## 1             Chen
## 2       Oressochen
## 3      Oceanodroma
## 4     Ichthyophaga
## 5        Crecopsis
## 6      Anurolimnas
## 7  Aenigmatolimnas
## 8      Porphyriops
## 9    Mustelirallus
## 10 Chroicocephalus
## ..             ...

So between the eBird and BirdLife checklists, there are 339 genera that are are in one but not the other!

IOC’s taxonomic checklist

There are too many genera missing to enter by hand, so we need to find another data source to fill in the gaps. The IOC World Bird List is another well-maintained bird species checklist. Their spreadsheet file is way too messy to work with, however we can parse the XML file they provide. I have very little experience with XML code but I am very grateful that Scott Chamberlain, who works at ROpenSci, conjured some black magic and came up with code to turn the XML into a data frame (you can see his gist here):

# sourcing the gist creates a data frame named df
devtools::source_gist("https://gist.github.com/sckott/c0437a71a889793e30d5")
iocgen <- mutate(df, order = capFirst(order))
iocgen
## Source: local data frame [2,284 x 3]
## 
##               order        family        genus
##               (chr)         (chr)        (chr)
## 1      Tinamiformes     Tinamidae      Tinamus
## 2      Tinamiformes     Tinamidae  Nothocercus
## 3      Tinamiformes     Tinamidae Crypturellus
## 4      Tinamiformes     Tinamidae   Rhynchotus
## 5      Tinamiformes     Tinamidae  Nothoprocta
## 6      Tinamiformes     Tinamidae      Nothura
## 7      Tinamiformes     Tinamidae    Taoniscus
## 8      Tinamiformes     Tinamidae     Eudromia
## 9      Tinamiformes     Tinamidae    Tinamotis
## 10 Struthioniformes Struthionidae     Struthio
## ..              ...           ...          ...

The IOC checklist has more genera than either of the checklists examined previously. Now we combine the IOC and BirdLife genera and compare it with the eBird checklist:

allgenera <- full_join(iocgen, birdlifegen, 
                       by = c("order", "family", "genus"))
allgenera %>% arrange(genus)
## Source: local data frame [2,833 x 3]
## 
##               order          family        genus
##               (chr)           (chr)        (chr)
## 1       Apodiformes     Trochilidae     Abeillia
## 2  Caprimulgiformes     Trochilidae     Abeillia
## 3     Passeriformes       Cettiidae   Abroscopus
## 4     Passeriformes       Sylviidae   Abroscopus
## 5       Galliformes        Cracidae      Aburria
## 6     Passeriformes    Meliphagidae Acanthagenys
## 7     Passeriformes      Thraupidae  Acanthidops
## 8     Passeriformes     Emberizidae  Acanthidops
## 9     Passeriformes    Fringillidae     Acanthis
## 10    Passeriformes Acanthisittidae Acanthisitta
## ..              ...             ...          ...

By sorting the data frame by genus, we can see that in certain cases (like for the genus Abeillia) the order and family information does not match between these two checklists. Here I made a judgement call and after a quick scan through Wikipedia I decided that the IOC checklist had the most updated taxonomic placement. For example, the family Trochilidae are currently placed in the order Apodiformes rather than Caprimulgiformes, therefore, the first instance of this genus is the correct one. As full_join places rows from the data frame specified first (iocgen in this case) before those from the other data frame, we can use distinct to keep the first instance of each genus:

allgenera <- allgenera %>% distinct(genus) 
allgenera %>% arrange(genus)
## Source: local data frame [2,406 x 3]
## 
##            order          family           genus
##            (chr)           (chr)           (chr)
## 1    Apodiformes     Trochilidae        Abeillia
## 2  Passeriformes       Cettiidae      Abroscopus
## 3    Galliformes        Cracidae         Aburria
## 4  Passeriformes    Meliphagidae    Acanthagenys
## 5  Passeriformes      Thraupidae     Acanthidops
## 6  Passeriformes    Fringillidae        Acanthis
## 7  Passeriformes Acanthisittidae    Acanthisitta
## 8  Passeriformes    Acanthizidae       Acanthiza
## 9  Passeriformes    Meliphagidae Acanthorhynchus
## 10 Passeriformes    Acanthizidae     Acanthornis
## ..           ...             ...             ...

So, we kept the first instance of Abeillia, and so on. Now we a have a comprehensive list of genera, with their respective family and order, from two separate checklists. We can now match it to the eBird checklist:

ebirdorders <- left_join(ebirdgen, allgenera, by = "genus") %>%
  select(order, family, genus)
ebirdorders
## Source: local data frame [2,226 x 3]
## 
##               order        family        genus
##               (chr)         (chr)        (chr)
## 1  Struthioniformes Struthionidae     Struthio
## 2        Rheiformes       Rheidae         Rhea
## 3      Tinamiformes     Tinamidae  Nothocercus
## 4      Tinamiformes     Tinamidae      Tinamus
## 5      Tinamiformes     Tinamidae Crypturellus
## 6      Tinamiformes     Tinamidae   Rhynchotus
## 7      Tinamiformes     Tinamidae  Nothoprocta
## 8      Tinamiformes     Tinamidae      Nothura
## 9      Tinamiformes     Tinamidae    Taoniscus
## 10     Tinamiformes     Tinamidae     Eudromia
## ..              ...           ...          ...

We have merged all genera, with their order and family data, from the BirdLife and IOC checklists and now successfully merged it with the eBird checklist. Even though the BirdLife and IOC checklists are up-to-date, it is likely that some genera included in eBird aren’t included in either of the previous lists.

ebirdmissing <- ebirdorders %>% filter(is.na(order))
ebirdmissing
## Source: local data frame [39 x 3]
## 
##    order family          genus
##    (chr)  (chr)          (chr)
## 1     NA     NA     Oressochen
## 2     NA     NA   Ichthyophaga
## 3     NA     NA      Crecopsis
## 4     NA     NA  Mustelirallus
## 5     NA     NA  Rhamphomantis
## 6     NA     NA      Damophila
## 7     NA     NA   Calorhamphus
## 8     NA     NA       Guarouba
## 9     NA     NA    Euchrepomis
## 10    NA     NA Cercomacroides
## ..   ...    ...            ...

Filling in the gaps

There are still 39 genera for which we do not have order or family info. Given that the eBird checklist is ordered taxonomically, one way around this is by checking the order and family values around each NA value: if they are the same, then we can use this value to fill in the blanks:

NAs <- which(ebirdorders$genus %in% ebirdmissing$genus)
NAs
##  [1]   30  304  342  351  526  721  806  946  966  997 1234 1241 1248
## [14] 1303 1317 1403 1405 1406 1407 1408 1427 1428 1540 1733 1737 1741
## [27] 1745 1754 1758 1853 1878 1909 1976 2162 2163 2186 2202 2209 2221
print(ebirdorders[(NAs[1]-1):(NAs[1]+1),])
## Source: local data frame [3 x 3]
## 
##          order   family      genus
##          (chr)    (chr)      (chr)
## 1 Anseriformes Anatidae Pteronetta
## 2           NA       NA Oressochen
## 3 Anseriformes Anatidae Chloephaga

There are some cases where there are more than one NA in a row, and even sometimes these gaps are in between different families:

print(ebirdorders[2161:2164,])
## Source: local data frame [4 x 3]
## 
##           order       family             genus
##           (chr)        (chr)             (chr)
## 1 Passeriformes Fringillidae          Linurgus
## 2            NA           NA Pseudochloroptila
## 3            NA           NA            Alario
## 4 Passeriformes Fringillidae           Serinus
print(ebirdorders[1402:1409,])
## Source: local data frame [8 x 3]
## 
##           order        family          genus
##           (chr)         (chr)          (chr)
## 1 Passeriformes Campephagidae    Campochaera
## 2            NA            NA    Malindangia
## 3 Passeriformes Campephagidae         Lalage
## 4            NA            NA      Celebesia
## 5            NA            NA Cyanograucalus
## 6            NA            NA      Analisoma
## 7            NA            NA      Edolisoma
## 8 Passeriformes   Neosittidae  Daphoenositta

We can make an ugly for loop to deal with this situation, in which it checks the adjacent non-NA values, and if they are identical, uses them to replace the NAs in between:

for (i in 1:nrow(ebirdorders)) {
  if (is.na(ebirdorders$family[i])) {
    nextval <- which(!is.na(ebirdorders$family[i+1:(i+9)]))[1]
    nextval2 <-which(!is.na(ebirdorders$order[i+1:(i+9)]))[1]
    if (identical(ebirdorders$family[i-1], ebirdorders$family[i+nextval])) {
      ebirdorders$family[i] <- ebirdorders$family[i-1]
    }
    if (identical(ebirdorders$order[i-1], ebirdorders$order[i+nextval2])) {
      ebirdorders$order[i] <- ebirdorders$order[i-1]
    }
  }
}

ebirdorders %>% filter(is.na(order))
## Source: local data frame [0 x 3]
## 
## Variables not shown: order (chr), family (chr), genus (chr)
ebirdorders %>% filter(is.na(family))
## Source: local data frame [8 x 3]
## 
##           order family          genus
##           (chr)  (chr)          (chr)
## 1    Piciformes     NA   Calorhamphus
## 2 Passeriformes     NA    Euchrepomis
## 3 Passeriformes     NA    Ceratopipra
## 4 Passeriformes     NA      Celebesia
## 5 Passeriformes     NA Cyanograucalus
## 6 Passeriformes     NA      Analisoma
## 7 Passeriformes     NA      Edolisoma
## 8 Passeriformes     NA  Grammatoptila

There are now only 8 genera with missing family values, but none that are missing their respective order as we managed to fill in all of those. The remaining gaps are from NAs between different families, hence we can’t infer the family based on their position in the checklist. We’ll have to add these by hand:

for (i in 1:nrow(ebirdorders)) {
  if (is.na(ebirdorders$family[i])) {
    if (ebirdorders$genus[i] %in% c("Calorhamphus")) {
      ebirdorders$family[i] <- "Megalamidae"
      }
    if (ebirdorders$genus[i] %in% c("Ceratopipra")) {
      ebirdorders$family[i] <- "Pipridae"
      }
    if (ebirdorders$genus[i] %in% c("Celebesia","Cyanograucalus",
                                      "Analisoma", "Edolisoma")) {
      ebirdorders$family[i] <- "Campephagidae"
      }
    if (ebirdorders$genus[i] %in% c("Grammatoptila")) {
      ebirdorders$family[i] <- "Leiothrichidae"
      }
    if (ebirdorders$genus[i] %in% c("Euchrepomis")) {
      ebirdorders$family[i] <- "Thamnophilidae"
      }
  }
}

anyNA(ebirdorders$family)
## [1] FALSE

Voilà! No more missing values! We can join this list of genera back to the full taxonomic checklist with all species, or to your own checklist of sightings downloaded from eBird’s website:

ebirdfull <- left_join(ebirdtax, ebirdorders, by = "genus")
anyNA(ebirdfull$family)
## [1] FALSE

So, there it is! A slightly complicated way to add order and family information to the eBird taxonomic checklist. If there’s and eBird developers reading this, please feel free to add it to the eBird-1.1-SpeciesReference results fields! ;)