---
category: literaturenote
citekey: matterrwebdatahighlevelinterface2016
title: "RWebData: A High-Level Interface to the Programmable Web"
authors: "Matter, Ulrich"
year: 2016
date: 2016-03-01 2016-03-01
publication: "arXiv:1603.00293 [cs, stat]"
url: "http://arxiv.org/abs/1603.00293"
zotero_key: U7MT8H2B
zotero_storage: 5U7FCUBV
collections: R
folder: Liiklussageduse kaudne hindamine/04_Tehniline
firstAuthor: "Matter, Ulrich"
status: converted
---
# <span id="page-0-0"></span>RWebData: A High-Level Interface to the Programmable Web

# Ulrich Matter

University of Basel

#### Abstract

The rise of the programmable web offers new opportunities for the empirically driven social sciences. The access, compilation and preparation of data from the programmable web for statistical analysis can, however, involve substantial up-front costs for the practical researcher. The R-package RWebData provides a high-level framework that allows data to be easily collected from the programmable web in a format that can directly be used for statistical analysis in R [\(R Core Team](#page-19-0) [2013\)](#page-19-0) without bothering about the data's initial format and nesting structure. It was developed specifically for users who have no experience with web technologies and merely use R as a statistical software. The core idea and methodological contribution of the package are the disentangling of parsing web data and mapping them with a generic algorithm (independent of the initial data structure) to a flat table-like representation. This paper provides an overview of the high-level functions for R-users, explains the basic architecture of the package, and illustrates the implemented data mapping algorithm.

Keywords: R, programmable web, big public data, web api, rest.

# 1. Introduction

Digital data from the Internet has in many ways become part of our daily lives. Broadband facilities, the separation of data and design, as well as a broader adaption of certain web technology standards increasingly facilitate the integration of data across different software applications and hardware devices over the web. The Internet is increasingly becoming a programmable web<sup>1</sup> , where data is published not only in HTML-based websites for human readers, but also in standardized machine-readable formats to be "shared and reused across application, enterprise, and community boundaries" [\(W3C](#page-20-0) [2013\)](#page-20-0). The technical ecology of this programmable web consists essentially of web servers providing data in formats such as Extensible Markup Language (XML) via Application Programming Interfaces (APIs)<sup>2</sup>

<sup>1</sup> I use the term programmable web synonymously for "Semantic Web" or "Web of Data" and as conceptually motivated by [Swartz](#page-20-1) [\(2013\)](#page-20-1).

<sup>2</sup>Web APIs are a collection of predefined HTTP requests and response messages to facilitate the programmatic exchange of data between a web server and clients.

to other server- or client-side applications (see, e.g., [Nolan and Temple Lang](#page-19-1) [2014](#page-19-1) for a detailed introduction to web technologies for R-programmers). In the conceptual framework of this paper, APIs serve a dual function: they are the central nodes between different web applications and, at the same time, the central access points for researchers when they want to systematically collect and analyze data from the programmable web. More and more of these access points are becoming available every day. Moreover, this trend is likely to continue with the increasing number of people who have devices to access the Internet and the increasing amount of data recorded by embedded systems (i.e., sensors and applications in devices such as portable music players or cars that automatically feed data to web services).[3](#page-0-0)

The rise of the programmable web offers various new opportunities for data-driven sciences. First, for the social sciences, the programmable web provides big data covering every-day human activities and interrelationships as people leave digital traces by using mobile devices and applications based on APIs. The systematic analysis of such data is likely to offer new insights in fields as diverse as economics, political science, and sociology (see [Matter and](#page-19-2) [Stutzer](#page-19-2) [2015a](#page-19-2) for a review of the arguments in the case of political economics and political science, as well as, e.g., [Dodds](#page-18-0) et al. [2011;](#page-18-0) [DiGrazia](#page-18-1) et al. [2013;](#page-18-1) Preis [et al.](#page-19-3) [2013;](#page-19-3) [Barber´a](#page-18-2) [2015;](#page-18-2) [Matter and Stutzer](#page-19-4) [2015b](#page-19-4) for primary research based on data collected via web APIs). In fact, web APIs might become an important domain of data sources for the evolving field of computational social science (see, e.g., [Lazer](#page-18-3) et al. [2009;](#page-18-3) [Cioffi-Revilla](#page-18-4) [2010;](#page-18-4) [Conte](#page-18-5) et al. [2012;](#page-18-5) [Giles](#page-18-6) [2012\)](#page-18-6). Second, researchers from all fields of science can provide their own prepared data via APIs as a resource to facilitate research, and the consequent aggregation of data sets from diverse disciplines offers new opportunities to create new 'scientific data mash-ups' [\(Bell](#page-18-7) [2009\)](#page-18-7) and drive innovation. The provision and hosting of APIs by research institutes and scientists for other scientists is already common practice in the life sciences (see, e.g., the API to the NCBI Gene Expression Omnibus data repository, [NCBI](#page-19-5) [2014\)](#page-19-5). Third, the use of standardized protocols and data formats to exchange data over the web via APIs offers, in general, new approaches to facilitate reproducibility and replicability of research.

While the advantages of accessing the programmable web and integrating it into research projects are very promising, the practical utilization of this technology comes at a cost and demands that researchers possess a specific skill set and a certain knowledge of web technologies. The new R-package RWebData substantially reduces these costs by providing several high-level functions that facilitate the exploration and systematic collection of data from APIs based on a Representational State Transfer (REST) architecture[4](#page-0-0) . Moreover, the package contains a unified framework to summarize, visualize, and convert nested/tree-structured web

<sup>3</sup>The number of publicly accessible web APIs has grown from around 1,000 at the end of 2008 to over 10,000 at the end of 2013 [ProgrammableWeb](#page-19-6) [\(2014\)](#page-19-6). [Turner](#page-20-2) et al. [\(2014\)](#page-20-2) estimate that the share of available digital data stemming from embedded systems will rise to 10% by 2020 and will include up to 32 billion devices connected to the Internet. See also [Helbing and Pournaras](#page-18-8) [\(2015\)](#page-18-8) for a discussion of how this 'Internet of things' could be fostered on a crowd-sourced basis and employed for big-data analytics.

<sup>4</sup>See [Richardson and Amundsen](#page-19-7) [\(2013\)](#page-19-7) for an introduction to REST APIs.

data that works independently of the data's initial format (XML/RSS, JSON, YAML) as well as a framework to easily create R packages as client libraries for any REST API. The package is aimed at empirical researchers using R as their daily data analysis- and statistics tool, but who do not have a background in computer science or data science. In addition, several lower level functions of RWebData might also be useful for more advanced R-programmers who whish to develop client applications to interact with REST APIs. The package thus bridges the gap between the fundamental and very valuable R-packages that integrate several web technologies into the R-environment (see, e.g., [Temple Lang](#page-20-3) [2013b](#page-20-3)[,a;](#page-20-4) [Couture-Beil](#page-18-9) [2013;](#page-18-9) [Ooms](#page-19-8) et al. [2014\)](#page-19-8) and the statistical analysis of the programmable web universe.

The development and use of RWebData is motivated in Section [2,](#page-2-0) which explains the challenges of extracting data from the programmable web for statistical analysis and introduces the packages that enable R-users to work with web technologies. Section [3](#page-5-0) presents the central data mapping strategy and algorithm developed in RWebData as well as an overview of the package's basic architecture. Section [4](#page-10-0) illustrates the high-level functionality of RWebData for a typical user. In Section [5,](#page-15-0) more advanced examples show how RWebData can be used to write Open Source Interfaces/API client libraries. The concluding discussion in Section [6](#page-16-0) reviews the potential of RWebData for the empirically driven social sciences in terms of big public data access as well as the reproducibility and replicability of research based on data from web APIs.

RWebData is free, open source (under GLP-2 and GLP-3 license), and published on Bitbucket (<https://bitbucket.org/ulrich-matter/rwebdata>). It can directly be installed from the R-console (using the devtools-package; [Wickham and Chang](#page-20-5) [2015\)](#page-20-5) with the command install\_bitbucket('ulrich-matter/RWebData'). This paper describes version 0.1 of RWebData, which requires R version 3.0.2 or a subsequent version.

# 2. Motivation

<span id="page-2-0"></span>How to extract data from the programmable web in a format suitable for statistical analysis? Particularly social scientists are often confronted with APIs whose initial purpose is not to provide data for scientific research in order to make the researchers' lives easier, but to facilitate the integration of the data in dynamic websites and smartphone applications.[5](#page-0-0) In such a context, API server and API client share the problem domain, but do not share the same goal. This makes it per se harder to write an API client in order to compile data from the web API (see [Richardson and Amundsen](#page-19-7) [2013](#page-19-7) for a discussion of this issue). The API query methods might, for example, not provide the length and breadth of data that the researcher is looking for using a single query, but only with a combination of several query

<sup>5</sup>The vast majority of APIs are explicitly made for web developers and are hosted by companies or NGOs that have nothing to do with academic research (according to [Santos](#page-20-6) [2012,](#page-20-6) only about 195 of over 7,000 APIs listed in the ProgrammableWeb.com API-directory were related to 'science' in 2012.).

methods. Importantly, the returned data are – due to the initial purpose of most APIs – usually not in a format that can be directly integrated in a statistical analysis. Researchers trying to compile data from such an API are thus confronted with two main challenges:

- 1. The need to query and combine data from different API methods which is likely to involve many requests to the API and demands a basic knowledge of how to interact with an API from within the R-environment.
- 2. The extraction and conversion of the data records from a nested web data format such as XML, JSON, or YAML to a table-like representation that can be directly used for statistical analyses.

An additional challenge is to overcome the two issues above in a way that supports the reproducibility and replicability of research based on API data (including the pre-processing of raw data). As pointed out by [Matter and Stutzer](#page-19-2) [\(2015a\)](#page-19-2), this is an increasingly important issue in the context of social science research which needs to be resolved, as the new digital data sources hinder the replicability of research owing to the present 'unique purpose' of the new data sets.

# 2.1. Interacting with REST web APIs

In order to visit a certain website, we normally type a website's address in a web browser's address bar. The browser (a type of web client) then sends a message to the web server behind that address, requesting a copy of the website and then parses and renders the returned website (usually a HTML document) in the browser window. In REST terminology, the website's address is an URL and the website which the URL locates is a resource. The document that the server sends in response to the client's request is a representation of that resource. Importantly, every URL points only to one resource and every resource should have only one URL (this is one of the REST principles and is referred to as addressability). The message that the browser sends to the web server is a HTTP GET request, a HTTP method that essentially requests a representation of a given resource.

REST web APIs work basically the same way as the website outlined above. The crucial difference is, that their design is intended for programmable clients (rather than humans using a web browser) and that they therefore consist of URLs pointing to resources that are not optimized for graphical rendering, but which instead contain the raw data (often in a format such as XML or JSON that facilitates the integration of the data in a HTML document).

Interacting with a REST API from within the R-environment, therefore, means, sending HTTP requests to a web server and handling the server's response with R. In addition to the basic functionality necessary for this, which are delivered in the base-package [\(R Core](#page-19-0)

[Team](#page-19-0) [2013\)](#page-19-0), more detailed functions for HTTP requests are provided in packages such as RCurl [\(Lang](#page-18-10) [2013a\)](#page-18-10) and httr [\(Wickham](#page-20-7) [2014a\)](#page-20-7). RWebData builds on the libcurl-based RCurl package which is designed to interact with web servers hosting a REST API. The implementation is focused on a robust download of resources that checks the received HTTP response for potential problems (e.g., unexpected binary content of the HTTP responses' body-entity) to make sure that the user does not have to specify anything other than the URL.

#### 2.2. Web data conversion

Most functions related to the fitting and testing of statistical models as well as to exploratory data analysis and data visualization in the R computing environment work on data represented in data-frames (a table-like flat representation in which each row represents a datarecord/observation and each column a variable describing these records).[6](#page-0-0) Data-frames are probably the most common R-objects used by researchers when conducting an empirical analysis with R. Many input/output-functionalities in R serve to import (export) data into (from) R as data-frames.[7](#page-0-0) However, when it comes to reading data from web APIs into R, this is not necessarily the case. The reason lies in the nature of web data formats which allow for more flexibility than solely a table-like data representation and come (in the context of web APIs) typically in a nested structure. While a conversion from one table in, e.g., a XML-document to a data-frame is straightforward (see, i.e., xmlToDataFrame() in [Temple Lang](#page-20-3) [2013b\)](#page-20-3), a conversion of a more complex XML-document with a nested data structure to a flat table-like data representation in R or any other computing environment is ex ante less clear and depends on the nature of the data and the purpose the data is converted for.[8](#page-0-0) This is particularly the case for data provided by a non-scientific API that is explicitly made for web developers in order to integrate the data in dynamic websites or mobile applications and not for researchers to integrate the data in their analysis. Not surprisingly, there are different solutions offered in the R-environment to map web data formats such as JSON and XML to R-objects.[9](#page-0-0) Most

<sup>6</sup>Technically, a data-frame can contain other r-objects than just scalars (i.e., another data-frame). However, in the vast majority of applications in the context of statistical analysis and data visualization, data-frames are used for a table-like flat data representation.

<sup>7</sup>See, i.e., data from CSV- and similar text files: read.table() in [R Core Team](#page-19-0) [\(2013\)](#page-19-0) or similar functions in [Wickham and Francois](#page-20-8) [\(2015\)](#page-20-8), Microsoft Excel files: read.xls() in [Warnes](#page-20-9) et al. [\(2014\)](#page-20-9), data from other statistical computing environments such as Stata: read.dta() in [R Core Team](#page-19-9) [\(2014\)](#page-19-9), or data from ODBCdata bases sqlQuery() in [Ripley and Lapsley](#page-20-10) [\(2013\)](#page-20-10).

<sup>8</sup>See, e.g., the classical problem of mapping a set of XML-documents to a relational data base scheme (RDBS; i.e., a set of linked tables) with or without knowing the scheme behind the XML-documents [\(Moh](#page-19-10) [et al.](#page-19-10) [2000;](#page-19-10) [Men-hin and Fu](#page-19-11) [2001\)](#page-19-11). See also, in the R context, the different implementations of mapping JSON to R-objects in [Couture-Beil](#page-18-9) [\(2013\)](#page-18-9), [Lang](#page-18-11) [\(2013b\)](#page-18-11), or [Ooms](#page-19-8) et al. [\(2014\)](#page-19-8) as well as a detailed discussion of this matter in [Ooms](#page-19-12) [\(2014\)](#page-19-12).

<sup>9</sup>See, e.g., the CRAN Task View on web technologies and services ([http://cran.r-project.org/web/](http://cran.r-project.org/web/views/WebTechnologies.html) [views/WebTechnologies.html](http://cran.r-project.org/web/views/WebTechnologies.html)) for an overview of popular R-packages that parse data from the web and map it to R-objects. Several contributions in this area also rather focus on the automated information extraction from traditional websites made for human interaction. Such methods are commonly summarized under the

packages represent web data as nested lists containing objects of different classes such as atomic vectors or data-frames. After converting the original web data to R-objects, the user, therefore, often has to extract and rearrange the data records of interest in order to obtain a data representation for statistical analysis. This process becomes even more complex, as queries to the same API method might provide slightly different results depending on the amount of detail embedded in the data records. In these cases, data extraction can become particularly costly if the data set that needs to be compiled depends on many API requests involving various API methods.

# 3. Data mapping strategy and basic architecture

<span id="page-5-0"></span>Web data provided via REST APIs are typically in a format such as XML, JSON, or YAML and are not structured in tables containing data values in rows and columns, but are rather organized in a nested (tree-like) structure. However, independent of the data format and data structure, documents provided by a particular REST API have basic aspects in common. Based on these commonalities, this section presents a conceptual and terminological framework as well as a description of how this framework is used to develop the data mapping strategy applied in RWebData and serves as the foundation of the data mapping algorithm which will be outlined later on.

## 3.1. Original data structure and data semantics

From the researcher's or statistician's point of view, the data sets provided by web APIs obey, independent of the raw data format and nesting structure, some very basic data semantics. For these data semantics, I use mainly the same terminology that is nicely outlined by [Wickham](#page-20-11) [\(2014b,](#page-20-11) p. 3):

- 1. "A dataset is a collection of values, usually either numbers (if quantitative) or strings (if qualitative)."
- 2. "Every value belongs to a variable and an observation."
- 3. "A variable contains all values that measure the same underlying attribute (like height, temperature, duration) across units."
- 4. "An observation contains all values measured on the same unit (like a person, or a day, or a race) across attributes."

terms "web scraping" or "screen scraping". See, e.g., the R-package rvest [\(Wickham](#page-20-12) [2015\)](#page-20-12) for functions that facilitate web scraping as well as [Munzert](#page-19-13) et al. [\(2014\)](#page-19-13) for a practical introduction to web scraping with R.

In addition, each observation and each variable belongs to a specific observation type. Thus, in a multi-dimensional data set describing, for example, a city, observation types could be buildings or citizens. The following fictional XML example further illustrates this point.

```
<?xml version="1.0" encoding="UTF-8"?>
<firm>
  <employees>
    <employee>
      <firstName>John</firstName>
      <secondName>Smith</secondName>
    </employee>
    <employee>
      <firstName>Peter</firstName>
      <secondName>Pan</secondName>
    </employee>
  </employees>
  <shareholders>
    <shareholder>
      <ID>S1</ID>
      <Name>Karl Marx</Name>
    </shareholder>
    <shareholder>
      <ID>S2</ID>
      <Name>Bill Gates</Name>
    </shareholder>
  </shareholders>
  <firmName>MicroCapital Ltd</firmName>
  <firmID>123</firmID>
</firm>
```

The XML document contains a data set describing a firm. The variables "firstName" and "secondName" describe observations of type "employee", while observations of type "shareholder" are described by the variables "ID" and "Name". Finally, the variables "firmName" and "firmID" describe the firm itself, which builds another type of observation. The following subsection illustrates how the data would be mapped according to the procedure implemented in RWebData.

#### 3.2. Mapping nested web data to data-frames

The core idea behind the data mapping procedure in RWebData is built around the basic se-

mantics outlined above. According to this system, documents returned from APIs can contain data describing observational units of one type or several different types. RWebData returns one data-frame for each observation type. Observations and variables belonging to different types are, thus, collected in different data-frames, each describing one type of observational unit. Some observations might describe the document (or main type of observational unit) itself (i.e., metadata). These are collected in a separate data-frame. Tables [1](#page-7-0) (a-c) present the XML-document from the example above after mapping to data-frames according to the outlined mapping procedure. The next subsection explains how this process is implemented in RWebData.

<span id="page-7-0"></span>

|   | firmName         |                      | firmID          |            |   | firstName         | secondName |
|---|------------------|----------------------|-----------------|------------|---|-------------------|------------|
|   |                  | 123                  |                 |            | 1 | John              | Smith      |
| 1 | MicroCapital Ltd |                      |                 |            | 2 | Peter             | Pan        |
|   | (a) Firm type    |                      |                 |            |   | (b) Employee type |            |
|   |                  |                      | ID              | Name       |   |                   |            |
|   |                  | 1                    | S1<br>Karl Marx |            |   |                   |            |
|   |                  | 2                    | S2              | Bill Gates |   |                   |            |
|   |                  | (c) Shareholder type |                 |            |   |                   |            |

Table 1: The same data as in the XML code example above, but mapped to data-frames according to the generic RWebData mapping procedure. While (a) contains data describing the firm (the main observational unit) itself, (b) and (c) contain the observations and variables describing the observation types employee and shareholder, respectively.

#### 3.3. Parsing and generic mapping of different web data formats

In order to map nested web data to data-frames, RWebData applies, in a first step, existing parsers for different formats (mime-types) of the data in the body of the HTTP response from the API to read the data into R. [10](#page-0-0) Independent of the initial format, the data is parsed and coerced to a (nested) list representing the tree-structure of the raw data. In a next step, the data is mapped to one or several data-frames depending on the nesting structure and the recurrence of observation types and variables. Appendix [A.I](#page-21-0) presents a detailed description of the mapping algorithm. The data-frames are returned to the user either directly (in a list) or as part of an apiresponse-object.

The core idea behind the generic approach to web data conversion in the RWebData package is thus to disentangle the parsing of web data from the mapping of the data to data-frames. This allows the parsers for different web data formats to be relatively simple, while the

<sup>10</sup>In the case of JSON, either the parser provided in [Ooms](#page-19-8) et al. [\(2014\)](#page-19-8) or (in order to increase robustness in special cases) the one contributed by [Lang](#page-18-11) [\(2013b\)](#page-18-11) is applied. XML-documents or RSS-documents are parsed with the parser provided in [Temple Lang](#page-20-3) [\(2013b\)](#page-20-3) and YAML-documents with the parser provided in [Stephens](#page-20-13) [\(2014\)](#page-20-13).

same mapping algorithm is focused on the data semantics and can be applied independent of the initial raw data format. In addition, it allows different parsers' advantages to be combined in order to make the parsing process more robust. The suggested procedure goes hand in hand with the object-oriented approach applied in RWebData and provides summary and plot methods that work independent of the initial data format. The modular design of mapping web data to data-frames facilitates the extension of RWebData's compatibility with additional web data formats or alternative parsers that are, for example, optimized for vast web documents. Parsers simply need to read the web data as (nested) lists or in a format that conserves the tree-structure of the raw web data and can be easily coerced to a nested list.

# 3.4. Basic data mapping algorithm

Once the web data-document is downloaded from an API and coerced to a nested list, the algorithm consists essentially of two procedures:

- 1. The extraction of different observation types (and their respective observations and variables) as sub-trees. While reversely traversing the whole data-tree, the algorithm checks at each level (node) of the tree whether either the whole current sub-tree or one of its siblings can be considered an observation type. If so, the respective sub-tree is extracted and saved in a deposit variable. If not, the algorithm traverses further down in the tree-structure. Checks for observation types are defined as a set of control statements that incorporate the above outlined data semantics. Essentially, the recurrence of the same variables as siblings (containing the actual data values as leaf elements) is decisive in order to recognize observation types. The result of this step is a set of sub-trees, each sub-tree containing one observation type and its respective observations and variables.
- 2. For each of the resulting observation types (in the form of sub-trees), the respective observations are extracted as character vectors and bound as rows in a data-frame, while the variable names are conserved and added as column names. This procedure is built to be robust to observations described by a differing number of variables. The result of this step is a set of data-frames, each describing one observation type and containing individual observations as rows and variables as columns.

Singly occurring leaf nodes that are not nested within one of the detected observation types are collected along the way and then returned in one data-frame. According to the logic of the data semantics and the algorithm outlined above, these remaining data can be seen as metadata describing the data set as a whole. Appendix [A.I](#page-21-0) presents a detailed, more technical outline of the data mapping algorithm.

# 3.5. Basic architecture

RWebData is specifically written to give practical researchers a high-level interface to compile data from the programmable web. The common user will thus normally not be confronted with the range of internal processes outlined above. The work-flow with RWebData for an average user thus mainly involves the specification of what data should be requested from what API (in the form of either a list, a data-frame or simply a string with a URL) to one of RWebData's high-level functions in order to obtain the desired data mapped to data-frames. Figure [1](#page-9-0) illustrates the basic architecture of RWebData, summarizing its internal functionality and the processing procedure when the package is used by an average user.

<span id="page-9-0"></span>![](_page_9_Figure_4.jpeg)

Figure 1: Basic architecture of the RWebData package.

RWebData's high level functions take either lists, data frames, or a URL (as character string) as input values and return a list of data-frames or an apidata-object. First, RWebData generates an apirequest-object based on which the HTTP GET and responses are handled. Upon successful interaction with the API, an apiresponse-object, containing, inter alia, the raw web data, is generated. Subject to consideration of the mime-type of the raw data, the data is then preprocessed, parsed, and coerced to a nested list. Finally, the data mapping algorithm is applied in order to extract the data in the form of data frames as outlined in the previous section. The latter builds the main part of the package. The procedure is straight-forward and user-friendly in practice.

# 4. Basic functionality

<span id="page-10-0"></span>In order to describe the basic usage of RWebData, this section offers a step-by-step introduction to the high-level functionality of the package. Some of the functions described here offer more options which are not all discussed in this section. Details on all these options are provided by the respective R help files. Users are generally encouraged to read the detailed documentation of RWebData on GitHub (<https://github.com/GivenTheData/RWebData>).

RWebData's implementation is motivated by the convention over configuration paradigm and thus provides a way for the user to interact with APIs using as few specifications as necessary to access the data. RWebData then converts data in the most common formats provided by APIs to one data-frame or a list of several data-frames. Hence, the user does not need to specify any HTTP options or understand what XML or JSON is and is able to obtain the data directly in the expected format for statistical analysis. There are primarily two high-level functions in RWebData that provide this functionality in different ways: getTabularData() and apiData().

# 4.1. Fetching data from REST APIs

The function getTabularData() provides a straightforward way to get data from web APIs as data-frames. In the simplest case, the function takes a string containing the URL to the respective API resource as an input.[11](#page-0-0) The function then handles the HTTP request and response, parses the body of the response (depending on the mime-type) and automatically extracts the data records from the nested data structure as data-frames. This enables us to fetch data from different APIs providing data in different formats with essentially the same command.

Consider, for example, the World Bank Indicators API which provides time series data on financial indicators of different countries.[12](#page-0-0) We want to use data from that API to investigate how the United States' public dept was affected by the financial crisis in 2008. All we need in order to download and extract the data is the URL to the respective resource on the API.[13](#page-0-0)

```
R> u <- paste0("http://api.worldbank.org/countries/USA/indicators", # address
+ "/DP.DOD.DECN.CR.GG.CD?", # query method
+ "&date=2005Q1:2013Q4") # parameters
```

<sup>11</sup>The URL to the resource which a user wants to query is very easy to find in any REST API documentation. How to call the respective resource methods of an API with URLs is the essential part of such documentations. <sup>12</sup>Note that this is an example of an API which is – unlike most APIs – explicitly made for researchers retrieving data from the web. Nevertheless, retrieving data from this API with functions from, e.g., the XML package [\(Temple Lang](#page-20-3) [2013b\)](#page-20-3) is not necessarily straightforward for users without a background in web technologies.

<sup>13</sup>A URL to an API resource typically includes the base address of the API, a part specifying the specific API query method or resource, and some query parameters. How to build query-URLs in the specific case of the World Bank Indicators API is well documented on <http://data.worldbank.org/node/203>.

```
R> usdept <- getTabularData(u)
```

Without bothering about the initial format[14](#page-0-0), the returned data is already in the form of a data-frame and is ready to be analyzed (e.g., by plotting the time series as presented in Figure [2\)](#page-11-0):

```
R> require(zoo)
R> plot(as.ts(zoo(usdept$value, as.yearqtr(usdept$date))),
     ylab="U.S. public dept (in USD)")
```

![](_page_11_Figure_5.jpeg)

Figure 2: Plot of the time series on United States public dept extracted from the World Bank Indicators API

The same approach of fetching data with getTabularData() can be applied to any other REST API that provides data in either XML/RSS, JSON, or YAML. Instead of a URL,

<sup>14</sup>The World Bank Indicators API provides time series data by default as XML in a compressed text file. Handling solely one API query of this type might already involve many steps to fetch and extract the data with the existing lower level functions. And, thus, might be tedious for a user who only has limited experience with web technologies.

one can also use a list of request parameters plus the base URL that directs to the API's server as function arguments. We use this approach in order to download data from the HarvardEvents API (<https://manual.cs50.net/api/events/>), which provides data about current or past events at Harvard University. Data on events can be fetched in various formats. In the following example, we query the API for upcoming lectures at Harvard and request a different data format each time (note the different format in the output parameter).

```
R> hae_api <- "http://events.cs50.net/api/1.0/events?"
R> haexml <- getTabularData(list(output="xml", q="lecture"), base.url=hae_api)
R> haejson <- getTabularData(list(output="json", q="lecture"), base.url=hae_api)
R> haerss <- getTabularData(list(output="rss", q="lecture"), base.url=hae_api)
```

In each case, getTabularData() automatically converts the raw web data into data-frames.[15](#page-0-0) The output to this example is presented in Appendix [A.II.](#page-23-0)

While the data format in the examples above varies, the underlying data structure is relatively simple in all these examples. The World Bank Indicators API is specifically devised to provide data for statistical analysis.The HarvardEvents API, although primarily designed for developers, provides data that can be naturally represented in one table (with the exception of responses in RSS format). The next section considers examples where the provided data cannot easily be thought of as one single table.

# 4.2. Nested data structures

The high-level function getTabularData() automatically handles nested data and converts them to a list of various data-frames. It does not, however, provide any information on the initial nesting structure and can only handle individual requests. Alternatively, we can use apiData() in order to exploit RWebData's internal classes and methods to handle requests to APIs. The simplest way to call apiData() is again by using a URL (as a string) pointing to an API resource. apiData() returns an object of class apiresponse. Such apiresponse objects contain additional information about the executed request to the API and support generic plot and summary functions illustrating the structure and content of the retrieved web document. The following example demonstrates the summary methods for apiresponse objects with data from the Ergast API (<http://ergast.com/mrd/>) on Formula 1 race results.

R> f1 <- apiData('http://ergast.com/api/f1/2013/1/results.json', + shortnames=TRUE)

<sup>15</sup>Although an API offers its data in different formats, it does not necessarily send exactly the same content for the same request in different formats. Different formats might serve different purposes for web developers. It thus makes sense that some contain additional information. Therefore, data requests with getTabularData() to the same API but in different formats do not automatically lead to identical results. While haexml and haejson are almost identical in the example above, haerss differs from the other results.

```
R> summary(f1)
```

```
API data summary:
=================
```

The API data has been split into the following 2 data frames:

```
Length Class Mode
metadata 20 data.frame list
Results 27 data.frame list
```

The respective data frame(s) contain the following variables:

## 1. metadata:

```
xmlns, series, url, limit, offset, total, season, round, 1,
raceName, circuitId, 2, circuitName, lat, long, locality,
country, date, time, path,
```

#### 2. Results:

```
number, position, positionText, points, driverId,
permanentNumber, code, url, givenName, familyName, dateOfBirth,
nationality, constructorId, url, name, nationality, grid,
laps, status, millis, time, rank, lap, time, units, speed,
path,
```

The summary method called by the generic summary(), provides an overview of the variables included in the data and shows how RWebData has split the data into several data-frames. This is particularly helpful in an early phase of a research project when exploring an API for the first time.

The next example demonstrates the visualization of nested data returned from the Open States API[16](#page-0-0). We query data on a legislator in the Council of the District of Columbia with apiData() and call the generic plot() on the returned apiresponse object. Figure [3](#page-14-0) shows the result of the plot command below. Note that the Open States API is free to use for registered users. Upon registration a api-key is issued that is then added as a parametervalue to each API request (indicated with "[YOUR-API-KEY]" in the example below).

```
R> url <- "http://openstates.org/api/v1/legislators/DCL000004/
+ ?apikey=[YOUR-API-KEY]"
```

<sup>16</sup>See <https://sunlightlabs.github.io/openstates-api/> for details.

```
R> p <- apiData(url)
R> plot(p, type="jitter")
```

The apiresponse plot method illustrates the nesting structure of the original data by drawing entities (variables or types) as nodes and nesting relationships as edges in a Reingold-Tilford tree-graph [\(Reingold and Tilford](#page-19-14) [1981;](#page-19-14) see also [Csardi and Nepusz](#page-18-12) [2006](#page-18-12) for the implementation in R on which RWebData relies.). The option type="jitter" shifts the nodes slightly vertically to make long node names better readable in the graph.

The transformed data (in the form of a list of data-frames) saved in an apiresponse-object can be accessed with getdata(). In the current example, the data has been split into two data-frames: one with metadata containing general variables describing the legislator, and one containing data on the legislator's roles in the 2011-2012 session.

```
R> pdata <- getdata(p)
R> summary(pdata)
```

Length Class Mode metadata 23 data.frame list 2011-2012 11 data.frame list

<span id="page-14-0"></span>![](_page_14_Figure_6.jpeg)

Figure 3: Reingold-Tilford tree-graph illustrating the nested structure of the raw JSON data from the Open States API.

## 4.3. Interactive sessions

So far, we have only considered individual requests to APIs. In practice, users might want to query an API in many ways in an interactive session to explore the data and assemble the data set that meets their needs. This can easily be done with RWebData. We continue with the Open States API example. First, we fetch general data on all legislators in the DC Council.

```
R> url <- "http://openstates.org/api/v1/legislators/?state=dc&chamber=upper
+ &apikey=[YOUR-API-KEY]"
R> dc_council <- getTabularData(url)
```

The goal is to combine these general data with data on the legislators' roles. In particular, we want to download the detailed role data on all legislators and combine these within one data set. As there is no API method provided to obtain these data with one call to the API, we use apiDownload() to handle all individual requests at once. By inspecting the dc\_council data-frame from above, we see that the fifth column of that data-frame contains all the IDs (id) pointing to the respective resources on individual council members. We simply paste these ids with the legislators-method and use the resulting URLs as the function argument to apiDownload() in order to obtain all the data compiled in one data-frame.

```
R> head(names(dc_council))
[1] "fax" "last_name" "updated_at" "full_name" "id"
[6] "first_name"
R> api_requests <- paste0("http://openstates.org/api/v1/legislators/",
+ dc_council$id,
+ "/?apikey=",
+ "YOUR-API-KEY")
```

R> dc\_leg\_roles <- apiDownload(api\_requests)

During the download, apiDownload() indicates the number of queries processed on a progress bar printed to the R console. Moreover, apiDownload() periodically saves the processed data locally in a temporary file. This relieves the working memory assigned to the R-session and makes large downloads robust to network interruptions. A summary of the resulting dc\_leg\_roles is presented in Appendix [A.III.](#page-24-0)

# <span id="page-15-0"></span>5. Writing interfaces to REST APIs

In the context of an ongoing research project, it might be more comfortable to have a specific function for specific queries to the same API. We could manually program such a function based on RWebData's internal functions or using the functions of other packages. However, RWebData provides a way to write such functions automatically. Given the parameters and the base URL for the respective API method/resource, generateQueryFunction() writes a function that includes all the functionality of the package to handle queries to that specific API method/resource. The same works for APIs that accept requests via form URLs, such as the BisTip.com Search API. Here, only the base URL and the respective parameters have to be specified.

```
R> bi_trip <- "http://www.bistip.com/api/v1/trips.json?"
R> bi_seek <- "http://www.bistip.com/api/v1/seeks.json?"
R> bi_params <- list(from=NA, to=NA) # parameters without default value
R> bitstipTrip <- generateQueryFunction(x=bi_params, base.url=bi_trip)
R> bitstipSeek <- generateQueryFunction(x=bi_params, base.url=bi_seek)
```

The new functions bitstipTrip() and bitstipSeek() can now be used to query the BitsTip.com API.

```
R> all_t <- bitstipSeek(from="tokyo")
R> j_s <- bitstipTrip(from="jakarta", to="singapore")
R> names(j_s)[1:4]
[1] "departure_date_medium_format" "origin_location"
[3] "arrival_date_medium_format" "period"
```

With only five lines of code, we have generated a complete BitsTip.com API R-client library (or Open Source Interface; [Matter and Stutzer](#page-19-2) [2015a\)](#page-19-2) that automatically provides the web data as data-frames.

# 6. Discussion

<span id="page-16-0"></span>Empirically driven social sciences can substantially profit from the rapidly growing programmable web as a data source covering countless dimensions of social activity. However, the purpose and goals that drove the initial development of web APIs focused on providing general data for public use via dynamic websites and other applications, and not on the provision of data formatted for scientific research and statistical analyses. This leads to technical hurdles a researcher has to overcome in order to compile such data in a format that is suitable for statistical analysis. The presented R package RWebData offers a simple high-level interface that helps to overcome such technical hurdles (and the respective costs) associated with the statistical analysis of data from the programmable web. The package facilitates a frictionless and well documented raw data compilation and data preparation process that substantially increases the replicability and reproducibility of original research based on data from the programmable web. As pointed out by [Matter and Stutzer](#page-19-2) [\(2015a\)](#page-19-2), empirical research based on newly available big public data from web data sources is a challenge demanding scientific rigor, as the repository of 'unique' data sets rapaciously grows. The need for data precision is even greater in the age of big data: Concision must be an essential characteristic of data recording and retrieval with regard to parsing and compilation. Data identification and presentation need to be restricted to minimum informational cues. Rigor applies not only to the final prepared data used in the analysis, but also to the documentation of the data preparation and data selection process. With RWebData, an R-script documenting how the package's high-level functions have been applied to compile data as well as any further step in the data preparation and analysis are enough to ensure the replicability of a study based on big public data from the programmable web.

# Acknowledgements

I am grateful to Dietmar Maringer, Armando Meier, Reto Odermatt, Michaela Slotwinski, Alois Stutzer, as well as seminar participants at the University of Basel and the University of Oxford for helpful remarks. Special thanks go to Ingmar Schlecht for many productive discussions on software development and the methodological aspects of this paper. I also thank Joerg Kalbfuss for excellent research assistance. The author acknowledges financial support from the University of Basel Research Fund.

# References

- <span id="page-18-2"></span>Barber´a P (2015). "Birds of the Same Feather Tweet Together: Bayesian Ideal Point Estimation Using Twitter Data." Political Analysis, 23(1), 76–91.
- <span id="page-18-7"></span>Bell G (2009). "Foreword/Digital Libraries for Data and Documents: Just Like Modern Document Libraries." In T Hey, S Transley, K Tolle (eds.), The Fourth Paradigm: Data-Intensive Scientific Discovery. Microsoft Research, Redmond, Washington.
- <span id="page-18-4"></span>Cioffi-Revilla C (2010). "Computational Social Science." Wiley Interdisciplinary Reviews: Computational Statistics, 2(3), 259–271.
- <span id="page-18-5"></span>Conte R, Gilbert N, Bonelli G, Cioffi-Revilla C, Deffuant G, Kertesz J, Loreto V, Moat S, Nadal JP, Sanchez A, Nowak A, Flache A, San Miguel M, Helbing D (2012). "Manifesto of Computational Social Science." The European Physical Journal Special Topics, 214(1), 325–346.
- <span id="page-18-9"></span>Couture-Beil A (2013). rjson: JSON for R. R package version 0.2.13, URL [http://CRAN.](http://CRAN.R-project.org/package=rjson) [R-project.org/package=rjson](http://CRAN.R-project.org/package=rjson).
- <span id="page-18-12"></span>Csardi G, Nepusz T (2006). "The igraph Software Package for Complex Network Research." InterJournal, Complex Systems, 1695. URL <http://igraph.org>.
- <span id="page-18-1"></span>DiGrazia J, McKelvey K, Bollen J, Rojas F (2013). "More Tweets, More Votes: Social Media as a Quantitative Indicator of Political Behavior." PLOS ONE, 8(11), e79449.
- <span id="page-18-0"></span>Dodds PS, Harris KD, Kloumann IM, Bliss CA, Danforth CM (2011). "Temporal Patterns of Happiness and Information in a Global Social Network: Hedonometrics and Twitter." PLOS ONE, 6(12), e26752.
- <span id="page-18-6"></span>Giles J (2012). "Making the Links." Nature, 448(7412), 448–450.
- <span id="page-18-8"></span>Helbing D, Pournaras E (2015). "Build Digital Democracy." Nature, 527(7576), 33–34.
- <span id="page-18-10"></span>Lang DT (2013a). "RCurl: General network (HTTP/FTP/...) client interface for R. R package version 1.95-4.1." http://CRAN.R-project.org/package=RCurl.
- <span id="page-18-11"></span>Lang DT (2013b). RJSONIO: Serialize R objects to JSON, JavaScript Object Notation. R package version 1.0-3, URL <http://CRAN.R-project.org/package=RJSONIO>.
- <span id="page-18-3"></span>Lazer D, Pentland A, Adamic L, Aral S, Barab´asi AL, Brewer D, Christakis N, Contractor N, Fowler J, Gutmann M, Jebara T, King G, Macy M, Roy D, Van Alstyne M (2009). "Computational Social Science." Science, 323(5915), 721–723.

- <span id="page-19-2"></span>Matter U, Stutzer A (2015a). "pvsR: An Open Source Interface to Big Data on the American Political Sphere." PLoS ONE, 10(7), e0130501. [doi:10.1371/journal.pone.0130501](http://dx.doi.org/10.1371/journal.pone.0130501).
- <span id="page-19-4"></span>Matter U, Stutzer A (2015b). "The Role of Lawyer-Legislators in Shaping the Law: Evidence from Voting Behavior on Tort Reforms." Journal of Law and Economics, 58(2), 357–384.
- <span id="page-19-11"></span>Men-hin Y, Fu AWc (2001). "From XML to Relational Databases." In Proceedings of the 81h International Workshop on Knowledge Representation Meets Databases (KRDB 2001), Rome, Italy.
- <span id="page-19-10"></span>Moh CH, Lim EP, Ng WK (2000). "Re-engineering Structures from Web Documents." In Proceedings of the fifth ACM Conference on Digital libraries, pp. 67–76. ACM.
- <span id="page-19-13"></span>Munzert S, Rubba C, Meissner P, Nyhuis D (2014). Automated Data Collection with R: A Practical Guide to Web Scraping and Text Mining. John Wiley & Sons, Chichester, UK.
- <span id="page-19-5"></span>NCBI (2014). "Programmatic Access to GEO." URL [http://www.ncbi.nlm.nih.gov/geo/](http://www.ncbi.nlm.nih.gov/geo/info/geo_paccess.html) [info/geo\\_paccess.html](http://www.ncbi.nlm.nih.gov/geo/info/geo_paccess.html).
- <span id="page-19-1"></span>Nolan D, Temple Lang D (2014). XML and Web Technologies for Data Sciences with R. UseR! Springer, New York.
- <span id="page-19-12"></span>Ooms J (2014). "The jsonlite Package: A Practical and Consistent Mapping Between JSON Data and R Objects." ArXiv e-prints. <1403.2805>.
- <span id="page-19-8"></span>Ooms J, Temple Lang D, Wallace J (2014). jsonlite: A smarter JSON encoder/decoder for R. R package version 0.9.8, URL <http://CRAN.R-project.org/package=jsonlite>.
- <span id="page-19-3"></span>Preis T, Moat HS, Bishop SR, Treleaven P, Stanley HE (2013). "Quantifying the Digital Traces of Hurricane Sandy on Flickr." Scientific Reports, 3.
- <span id="page-19-6"></span>ProgrammableWeb (2014). "Programmableweb Research Center: Growth in Web APIs From 2005 to 2013." URL <www.programmableweb.com/api-research>.
- <span id="page-19-0"></span>R Core Team (2013). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. URL <http://www.R-project.org/>.
- <span id="page-19-9"></span>R Core Team (2014). foreign: Read Data Stored by Minitab, S, SAS, SPSS, Stata, Systat, Weka, dBase, ... R package version 0.8-59, URL [http://CRAN.R-project.org/package=](http://CRAN.R-project.org/package=foreign) [foreign](http://CRAN.R-project.org/package=foreign).
- <span id="page-19-14"></span>Reingold EM, Tilford JS (1981). "Tidier Drawings of Trees." IEEE Transactions on Software Engineering, 7(2), 223–228.
- <span id="page-19-7"></span>Richardson L, Amundsen M (2013). RESTful Web APIs. O'Reilly Media, Cambridge.

- <span id="page-20-10"></span>Ripley B, Lapsley M (2013). RODBC: ODBC Database Access. R package version 1.3-10, URL <http://CRAN.R-project.org/package=RODBC>.
- <span id="page-20-6"></span>Santos W (2012). "195 Science APIs: Springer, EPA and NCBI." URL [http://www.](http://www.programmableweb.com/news/195-science-apis-springer-epa-and-ncbi/2012/03/28) [programmableweb.com/news/195-science-apis-springer-epa-and-ncbi/2012/03/28](http://www.programmableweb.com/news/195-science-apis-springer-epa-and-ncbi/2012/03/28).
- <span id="page-20-13"></span>Stephens J (2014). yaml: Methods to Convert R Data to YAML and Back. R package version 2.1.11, URL <http://CRAN.R-project.org/package=yaml>.
- <span id="page-20-1"></span>Swartz A (2013). "Aaron Swartz's A Programmable Web: An Unfinished Work." In J Hendler, Y Ding (eds.), Synthesis Lectures on The Semantic Web: Theory and Technology. Morgan & Claypool Publishers.
- <span id="page-20-4"></span>Temple Lang D (2013a). RCurl: General network (HTTP/FTP/...) client interface for R. R package version 1.95-4.1, URL <http://CRAN.R-project.org/package=RCurl>.
- <span id="page-20-3"></span>Temple Lang D (2013b). XML: Tools for Parsing and Generating XML Within R and S-Plus. R package version 3.95-0.2, URL <http://CRAN.R-project.org/package=XML>.
- <span id="page-20-2"></span>Turner V, Gantz JF, Reinsel D, Minton S (2014). "The Digital Universe of Opportunities: Rich Data and the Increasing Value of the Internet of Things." White paper, IDC, Framingham, MA.
- <span id="page-20-0"></span>W3C (2013). "W3C Semantic Web Activity: What Is the Semantic Web?" URL [www.w3.](www.w3.org/2001/sw/) [org/2001/sw/](www.w3.org/2001/sw/).
- <span id="page-20-9"></span>Warnes GR, Bolker B, Gorjanc G, Grothendieck G, Korosec A, Lumley T, MacQueen D, Magnusson A, Rogers J, others (2014). gdata: Various R Programming Tools for Data Manipulation. R package version 2.13.3, URL [http://CRAN.R-project.org/package=](http://CRAN.R-project.org/package=gdata) [gdata](http://CRAN.R-project.org/package=gdata).
- <span id="page-20-7"></span>Wickham H (2014a). httr: Tools for Working with URLs and HTTP. R package version 0.4, URL <http://CRAN.R-project.org/package=httr>.
- <span id="page-20-11"></span>Wickham H (2014b). "Tidy Data." Journal of Statistical Software, 59(10), 1–23.
- <span id="page-20-12"></span>Wickham H (2015). rvest: Easily Harvest (Scrape) Web Pages. R package version 0.2.0, URL <http://CRAN.R-project.org/package=rvest>.
- <span id="page-20-5"></span>Wickham H, Chang W (2015). devtools: Tools to Make Developing R Packages Easier. R package version 1.7.0, URL <http://CRAN.R-project.org/package=devtools>.
- <span id="page-20-8"></span>Wickham H, Francois R (2015). readr: Read Tabular Data. R package version 0.1.0, URL <http://CRAN.R-project.org/package=readr>.

# Appendix

#### <span id="page-21-0"></span>A.I. Data mapping algorithm

As pointed out in the main text, RWebData parses and coerces the raw web data to a nested list representing the tree-structure of the data. Call this list ~x.

The data mapping algorithm consists of two main parts. First, ~x is split into n lists representing sub-trees. One for each observation type in the data. The key problem that the algorithm has to solve at this step is the identification of cutting points (i.e., what part of the tree belongs to what observation type). Second, each resulting sub-tree is then split into individual character vectors. One for each observation. The individual observations are then stacked together in one data-frame with each vector (observation) as a row. Algorithm [A1](#page-0-0) presents a formal description of these procedures. In the resulting data-frames, each row i represents one observation, and each column j represents a variable/characteristic of the n observations. The data-frames are then returned in a list.

In order to make the formal descriptions of the procedures in the data mapping algorithm conveniently readable, the pseudo-code describes simple versions of these procedures (that are not necessarily most efficient). The algorithms' R-implementations in RWebData are more efficient and contain more control statements to ensure robustness. The actual R-implementation relies partly on existing R functions and favors vectorization over for-loops in some cases.

#### Algorithm A1 Data mapping algorithm

```
1: procedure Types(~x)
2: T ypes ← empty list
3: deposit ← empty list
4: if ~x is an observation type then
5: add ~x to T ypes
6: return T ypes
7: end if
8: if ~x contains a part i at the highest nesting level that is an observation type then
9: add i to T ypes
10: remove i from ~x
11: end if
12: for all elements i in ~x do
13: if i is a non-empty list then
14: if i is an observation type then
15: add i to T ypes
16: else
17: apply this very procedure to i . recursive call
18: add the resulting observation types to T ypes
19: add the remaining leaves (metadata) to deposit
20: end if
21: else
22: add i to deposit . it's a leaf node (i.e., just a value)
23: end if
24: end for
25: end procedure
26: procedure Observations(T ypes)
27: rows ← empty list
28: for all ~ obstype in T ypes do
29: for all observation in ~ obstype do
30: unlist and transpose observation . extract leaves as vector
31: add resulting vector to rows
32: end for
33: bind rows to one data-frame (preserving variable names)
34: end for
35: end procedure
```

#### <span id="page-23-0"></span>A.II. HarvardEvents API example

```
R> hae_api <- "http://events.cs50.net/api/1.0/events?"
R> haexml <- getTabularData(list(output="xml", q="lecture"), base.url=hae_api)
R> haejson <- getTabularData(list(output="json", q="lecture"), base.url=hae_api)
R> haerss <- getTabularData(list(output="rss", q="lecture"), base.url=hae_api)
R> # comparison of results based on xml/json data:
R> dim(haexml)
[1] 9 10
R> names(haexml)
[1] "summary" "dtstart" "dtend"
[4] "location" "description" "calname"
[7] "id" "calname" "INPUT_:_output"
[10] "INPUT_:_q"
R> haexml[1,4]
[1] "Waterhouse Room, Gordon Hall, HMS"
R> dim(haejson)
[1] 9 10
R> names(haejson)
[1] "summary" "dtstart" "dtend"
[4] "location" "description" "calname"
[7] "id" "calname" "INPUT_:_output"
[10] "INPUT_:_q"
R> haejson[1,4]
[1] "Waterhouse Room, Gordon Hall, HMS"
R> # rss response is differently structured:
R> summary(haerss)
```

```
Length Class Mode
metadata 8 data.frame list
item 8 data.frame list
R> haerss$item[1,]
                        guid title
1 http://events.cs50.net/18370722 JCSW Sponsorship Panel
                        link description category
1 http://events.cs50.net/18370722 <NA> events
                      pubDate INPUT_:_output INPUT_:_q
1 Mon, 04 May 2015 12:00:00 -0400 rss lecture
R>
R>
A.III. Open States API
R> url <- "http://openstates.org/api/v1/legislators/?state=dc&chamber=upper
+ &apikey=[YOUR-API-KEY]"
R> c_council <- getTabularData(url)
R> api_requests <- paste0("http://openstates.org/api/v1/legislators/",
                   dc_council$id,
                   "&apikey=",
                   "YOUR-API-KEY")
 |
 | | 0%
 |
 |============================== | 50%
 |
 |============================================================| 100%
R> # explore the data
R> summary(dc_leg_roles)
        Length Class Mode
metadata 48 data.frame list
2013-2014 13 data.frame list
```

2011-2012 13 data.frame list roles 13 data.frame list

R> dim(dc\_leg\_roles\$roles)

[1] 39 13

#### Affiliation:

Ulrich Matter Faculty of Business and Economics University of Basel Peter Merian-Weg 6 4002 Basel, Switzerland

E-mail: [ulrich.matter@unibas.ch](mailto:ulrich.matter@unibas.ch)