ennet package provides a set of functions that extracts information from the en-net online forum. This set of functions was built on top of the rvest package which provides robust and performant web scraping functions and the dplyr package which provides a full suite of data manipulation functions. The ennet package was designed to be able to interact with how the en-net online forum has been structured.

en-net website structure

The en-net online forum website has a very clear and clean structure. The opening page is a list of thematic areas which are linked to each of their respective webpages. In each of these thematic area webpages is another list, this time a list of topics raised within the thematic area. These topics are the text that an online user provides as the title for the question she/he is going to ask. Each of the topics are then again linked to their respective webpages that show the actual full question raised and the ensuing responses and discussion stemming from that question.

The en-net online forum structure can be summarised graphically as follows:



Getting list of thematic areas

To get a list of thematic areas along with the link to the webpage of each thematic area, we use the get_themes() function as follows:

## Load ennet package
library(ennet)

## Get all thematic areas from en-net
get_themes()

which results in

#> # A tibble: 18 x 2
#>    themes                                           links                       
#>    <chr>                                            <chr>                       
#>  1 Announcements & Nutritionists needed             https://www.en-net.org/foru…
#>  2 Assessment and Surveillance                      https://www.en-net.org/foru…
#>  3 COVID-19 and nutrition programming               https://www.en-net.org/foru…
#>  4 Coverage assessment                              https://www.en-net.org/foru…
#>  5 Cross-cutting issues                             https://www.en-net.org/foru…
#>  6 Food assistance                                  https://www.en-net.org/foru…
#>  7 Infant and young child feeding interventions     https://www.en-net.org/foru…
#>  8 Management of At Risk Mothers and Infants        https://www.en-net.org/foru…
#>  9 Micronutrients                                   https://www.en-net.org/foru…
#> 10 Partnerships for research                        https://www.en-net.org/foru…
#> 11 Management of wasting/acute malnutrition         https://www.en-net.org/foru…
#> 12 Prevention and management of stunting            https://www.en-net.org/foru…
#> 13 Scaling Up Nutrition (SUN)                       https://www.en-net.org/foru…
#> 14 Upcoming trainings                               https://www.en-net.org/foru…
#> 15 Other thematic area                              https://www.en-net.org/foru…
#> 16 Multi-sector nutrition programming               https://www.en-net.org/foru…
#> 17 Adolescent nutrition                             https://www.en-net.org/foru…
#> 18 Simplified Approaches for the Management of Acu… https://www.en-net.org/foru…

The resulting table has two columns - the first is named themes which contains the various thematic areas on the en-net online forum, and the second is named links which contains the corresponding URL for the webpages for each of the thematic areas.

This will be useful when choosing which themes to focus on when extracting information. This function outputs an object of the appropriate class and structure as the required input for the get_themes_topics() function which would lend to piped operations between the two functions (see below).

Getting list of topics from thematic area/s

To get a list of topics for a particular theme, we use the get_theme_topics() function as follows:

## Load dplyr to facilitate data manipulation
library(dplyr)

## Extract data from "Coverage assessment" theme
get_themes() %>%
  filter(themes == "Coverage assessment") %>%
  select(links) %>%
  as.character() %>%
  get_theme_topics()

which results in

#> # A tibble: 94 x 7
#>    Theme    Topic                 Views Replies Author   Posted     Link        
#>    <chr>    <chr>                 <int>   <int> <chr>    <date>     <chr>       
#>  1 Coverag… Resources for covera… 15752      11 Tamsin … 2011-12-06 https://www…
#>  2 Coverag… SLEAC                  1597       2 Anonymo… 2021-01-12 https://www…
#>  3 Coverag… Use of single covera…  5362       7 Hugh Lo… 2019-11-10 https://www…
#>  4 Coverag… Real and theoretical…  4349       3 Abdul    2019-10-10 https://www…
#>  5 Coverag… Single Coverage for …  3172       4 Ben All… 2019-07-05 https://www…
#>  6 Coverag… Cox's Bazar Refugee …  3410       4 Hugh Lo… 2019-03-28 https://www…
#>  7 Coverag… Wide Area Survey (St…  3066       2 Anonymo… 2019-03-09 https://www…
#>  8 Coverag… Disconnection betwee…  3040       2 Tammam … 2019-03-03 https://www…
#>  9 Coverag… Can we classify the …  3218       2 Anonymo… 2018-11-18 https://www…
#> 10 Coverag… Routine monitoring d…  3104       0 Anonymo… 2018-11-14 https://www…
#> # … with 84 more rows

The resulting table contains information on all the topics within the Coverage assessment thematic area including URL links to the corresponding webpages for each topic

To get a list of topics for multiple themes, we use the get_themes_topics() function as follows:

## Extract data from "Coverage assessment" theme and "Food assistance" theme
get_themes() %>%
  filter(themes %in% c("Coverage assessment", "Food assistance")) %>%
  get_themes_topics()

which results in

#> # A tibble: 126 x 7
#>    Theme    Topic                 Views Replies Author   Posted     Link        
#>    <chr>    <chr>                 <int>   <int> <chr>    <date>     <chr>       
#>  1 Coverag… Resources for covera… 15752      11 Tamsin … 2011-12-06 https://www…
#>  2 Coverag… SLEAC                  1597       2 Anonymo… 2021-01-12 https://www…
#>  3 Coverag… Use of single covera…  5362       7 Hugh Lo… 2019-11-10 https://www…
#>  4 Coverag… Real and theoretical…  4349       3 Abdul    2019-10-10 https://www…
#>  5 Coverag… Single Coverage for …  3172       4 Ben All… 2019-07-05 https://www…
#>  6 Coverag… Cox's Bazar Refugee …  3410       4 Hugh Lo… 2019-03-28 https://www…
#>  7 Coverag… Wide Area Survey (St…  3066       2 Anonymo… 2019-03-09 https://www…
#>  8 Coverag… Disconnection betwee…  3040       2 Tammam … 2019-03-03 https://www…
#>  9 Coverag… Can we classify the …  3218       2 Anonymo… 2018-11-18 https://www…
#> 10 Coverag… Routine monitoring d…  3104       0 Anonymo… 2018-11-14 https://www…
#> # … with 116 more rows

The resulting table contains information on all the topics within the Coverage assessment and Food assistance thematic area including URL links to the corresponding webpages for each topic.

Getting discussions from topic/s

To get a list of discussions for a particular topic, we use the get_topic_discussions() function as follows:

get_themes() %>%
  filter(themes == "Coverage assessment") %>%
  get_themes_topics() %>%
  filter(Topic == "Resources for coverage assessment") %>%
  select(Link) %>%
  as.character() %>%
  get_topic_discussions()

which results in

#> # A tibble: 12 x 10
#>    theme  topic user  userCode job   role  date_time           type  code  post 
#>    <chr>  <chr> <chr> <chr>    <chr> <chr> <dttm>              <chr> <chr> <chr>
#>  1 Cover… Reso… Tams… user24   en-n… Foru… 2011-12-06 11:59:00 ques… 574   "Dea…
#>  2 Cover… Reso… Mark… user31   Cons… Freq… 2011-12-06 13:24:00 answ… 1536  "And…
#>  3 Cover… Reso… Erne… user999  Kati… Tech… 2011-12-22 18:02:00 answ… 1603  "Wor…
#>  4 Cover… Reso… Saul… user1000 Dire… Tech… 2012-01-11 08:59:00 answ… 1664  "And…
#>  5 Cover… Reso… Mark… user31   Cons… Freq… 2012-05-01 12:29:00 answ… 2076  "Jus…
#>  6 Cover… Reso… Erne… user999  Kati… Tech… 2012-05-26 09:48:00 answ… 2180  "We …
#>  7 Cover… Reso… Erne… user999  Kati… Tech… 2012-07-05 11:19:00 answ… 2278  "Dea…
#>  8 Cover… Reso… Mark… user31   Cons… Freq… 2012-09-17 16:40:00 answ… 2442  "Upd…
#>  9 Cover… Reso… Mark… user31   Cons… Freq… 2012-11-01 16:19:00 answ… 2615  "And…
#> 10 Cover… Reso… Mark… user31   Cons… Freq… 2013-04-30 17:19:00 answ… 3288  "The…
#> 11 Cover… Reso… Mark… user31   Cons… Freq… 2013-09-05 09:56:00 answ… 3445  "Her…
#> 12 Cover… Reso… Alex… user107… Acti… Norm… 2020-06-11 17:27:00 answ… 7723  "The…

The resulting table contains information on all the discussions within the topic on Resources for coverage assessment within the thematic area of Coverage assessment including the text data on the question and the ensuing reply/replies to the question.

To get a list of discussions for a set of topics, we use the get_topics_discussions() function as follows:

get_themes() %>%
  filter(themes %in% c("Coverage assessment", "Food assistance")) %>%
  get_themes_topics() %>%
  get_topics_discussions()

which results in

#> # A tibble: 509 x 10
#>    theme  topic user  userCode job   role  date_time           type  code  post 
#>    <chr>  <chr> <chr> <chr>    <chr> <chr> <dttm>              <chr> <chr> <chr>
#>  1 Cover… Reso… Tams… user24   en-n… Foru… 2011-12-06 11:59:00 ques… 574   "Dea…
#>  2 Cover… Reso… Mark… user31   Cons… Freq… 2011-12-06 13:24:00 answ… 1536  "And…
#>  3 Cover… Reso… Erne… user999  Kati… Tech… 2011-12-22 18:02:00 answ… 1603  "Wor…
#>  4 Cover… Reso… Saul… user1000 Dire… Tech… 2012-01-11 08:59:00 answ… 1664  "And…
#>  5 Cover… Reso… Mark… user31   Cons… Freq… 2012-05-01 12:29:00 answ… 2076  "Jus…
#>  6 Cover… Reso… Erne… user999  Kati… Tech… 2012-05-26 09:48:00 answ… 2180  "We …
#>  7 Cover… Reso… Erne… user999  Kati… Tech… 2012-07-05 11:19:00 answ… 2278  "Dea…
#>  8 Cover… Reso… Mark… user31   Cons… Freq… 2012-09-17 16:40:00 answ… 2442  "Upd…
#>  9 Cover… Reso… Mark… user31   Cons… Freq… 2012-11-01 16:19:00 answ… 2615  "And…
#> 10 Cover… Reso… Mark… user31   Cons… Freq… 2013-04-30 17:19:00 answ… 3288  "The…
#> # … with 499 more rows

The resulting table contains information on all the discussions within all the topics within the thematic areas of Coverage assessment and Food assistance including the text data on the question and the ensuing reply/replies to the question.