Methodology
Data Pipeline The majority of this research relies on The GDELT Project, an online news database. For accessing data from GDELT, we relied on BigQuery as opposed to other common methods. There are various ways in which GDELT data can be accessed, but we found BigQuery to be the quickest and most comprehensive dataset that didn’t require the parsing and merging of a collection of json and/or ngram files. We also found that a lot of the preprocessed data, including translation, sentiment, and image URLs lived primarily in databases rather than json feeds.
For this project, we relied on two tables that were merged prior to save. The first of these tables was ‘gdelt-bq.gdeltv2.gkg_partitioned’. From this table, we extracted the date, source name, sharing image, translation information, document identifier (URL), themes, tone, and locations. Because we knew the subset of sources being queried, here we could supply base URLs to filter off for our data.
The second table we queried prior to merge was ‘gdelt-bq.gdeltv2.gal’. From this table, we accessed the site URL (for merge), the article title, the article language, and the outlet name. The primary reason for pulling data from two tables was because we needed the article headlines and these only came in this gdeltv2.gal table. Otherwise, we would have been able to rely on the results from the first table referenced. Additionally, we selected these tables because they included non-English headlines and news sources.
Resources
The preparation pipeline for our data can be found in this iPython Notebook: Preparation Pipeline (Colab)
Processed datasets used for analysis, below:
Complete Articles Dataset
8,218,648 Articles (7,905,339 Unique Articles)
Download: Complete Articles - 16.33GB
Population/Fertility Articles Dataset
416,565 Articles (405,801 Unique Articles)
Download: Population/Fertility Articles - 1.31GB
Population/Fertility Articles (Excluding Covid Coverage) Dataset
16,941 Articles (16,197 Unique Articles)
Download: Population/Fertility Articles (Non-Covid) - 56.3MB
Non-population Articles Dataset
7,802,083 Articles (7,499,538 Unique Articles)
Download: Non-population Articles Dataset - 15.03GB
Coverage, tone and theme, and country analysis can be found in the iPython notebooks below:
Coverage Counts Notebook: Colab
Tone & Theme Analysis Notebook: Colab
Country Analysis Notebook: Colab
Image Analysis Pipeline
Each image is passed into an image processing pipeline that involves various stages of preprocessing, object detection, and metadata generation.
Initially, each image is downscaled to a lower resolution and undergoes a color transformation from the BGR to the RGB scheme, preparing it for subsequent stages of processing.
The core of this pipeline is object detection, executed via the pre-configured YoloV3 neural network. Leveraging the model’s predefined labels, the function is designed to identify people within each image, outputting associated bounding boxes and confidence levels. This detection process employs predefined network confidence and threshold parameters, in conjunction with the dimensions of the resized image, to optimize detection accuracy.
In the post-detection phase, OpenAI’s CLIP model is used to generate metadata for each detected person. Specifically, it produces predictions for age and perceived gender from the cropped images. To obtain a more comprehensive description, the original, uncropped image is also introduced to the CLIP model.
The output of this pipeline, a rich set of image metadata, serves as the primary dataset for further downstream analysis.
Image Analysis Appendix
Images for Articles/Country (Full dataset)
| Country | Article Count | Images | Coverage |
| Brazil | 168093 |
80735 |
0.48 |
| Egypt | 40571 |
16926 |
0.41 |
| France | 50908 |
27939 |
0.54 |
| Hungary | 15381 |
11531 |
0.75 |
| India | 103993 |
82932 |
0.79 |
| Japan | 17522 |
15548 |
0.88 |
| Nigeria | 22372 |
15826 |
0.70 |
| United States | 72384 |
58199 |
0.80 |
Images for Articles/Country (non-Covid-19 dataset)
| Country | Article Count | Images | Coverage |
| United States | 2685 | 2459 | 0.92 |
| Nigeria | 2575 | 2281 | 0.89 |
| France | 1251 | 1104 | 0.88 |
| India | 4024 | 3153 | 0.78 |
| Hungary | 328 | 256 | 0.78 |
| Brazil | 3050 | 2303 | 0.76 |
| Japan | 463 | 332 | 0.72 |
| Egypt | 2565 | 1576 | 0.61 |
Topic Modeling: France
| Topic | Count | Example |
| People holding signs | 175 | A group of people holding up signs in front of a crowd |
| Group of people, crowd | 124 | A group of people in orange suits next to an inflatable boat |
| Close-up of woman | 96 | An older woman with grey hair wearing a white jacket |
| Wearing face mask | 67 | A woman wearing a face mask talking to a doctor |
| Hospitals and medical staff | 55 | A group of doctors in a room with a patient |
| Buildings | 54 | An aerial view of an apartment complex in a city |
| Pregnancy, babies, mothers | 54 | A woman holding a baby in her arms |
| Suit and tie | 52 | A man wearing a suit and tie |
| Close up of hands | 42 | A person holding a book in their hand |
| Contraceptive pills, contraceptives | 34 | A close up view of many different types of contraceptive pills |
| Microphone, podium | 31 | A man standing at a podium in front of a microphone |
| Unsorted Images | 175 | A close up picture of a string of white yarn |
| Total | 1104 |
Topic Modeling: United States
| Topic | Count | Example |
| People holding signs | 360 | A group of people marching down a street holding signs |
| Suit and tie, microphone | 302 | A man in a suit and tie sitting in front of a microphone |
| Group of people, standing | 239 | A group of people standing in the rain with water splashing all over them |
| Close-up of woman | 220 | A woman with long blonde hair wearing a black dress |
| Buildings, signage | 178 | A view of a health center with a sign on the wall |
| Split picture, two people side-by-side | 168 | A picture of Britney Spears and a man |
| Baby, bed | 109 | A woman holding a baby in a hospital bed |
| Books, posters | 106 | A New York tax book sitting on top of a table |
| Headphones, radio tower | 87 | NPR logo |
| Person holding object | 67 | A person holding a phone in their hands |
| Hospital equipment, medical staff | 66 | A hospital bed in a room with a monitor on it |
| Pills, contraceptives | 64 | A close up view of a box of contraceptive pills |
| People wearing face masks | 46 | An older man wearing a black face mask |
| Animals, trees | 37 | A bald eagle sitting on a tree branch |
| Body of water | 32 | Three boats are in the water near the shore |
| Forest fire, smoke | 26 | A fire burning in the middle of a forest |
| Unsorted Images | 345 | A milk carton sitting on top of a table |
| Total | 2459 |
| Topic | Count | Example |
| Group of people, standing | 265 | A crowd of people walking down a street |
| Suit and tie | 210 | A man wearing a suit and tie |
| Misc. People | 142 | A person wearing a hat and glasses |
| Head scarf | 138 | A woman in a pink dress and a pink head scarf |
| Flags and people holding signs | 137 | A group of people holding up signs in front of a building |
| Pregnant women, babies and children | 123 | A woman is using a stethoscope on a pregnant belly |
| Person sitting at desk, papers | 93 | A man in a white hat is sitting at a desk |
| Maps and charts | 82 | A map of Nigeria with the words "2023 elections"written underneath |
| Doctors, hospitals | 82 | A person wearing a white lab coat and blue gloves |
| Person speaking at podium, microphone | 80 | A man in a blue jacket is speaking at a press conference |
| Buildings | 76 | An aerial view of a large city with buildings |
| People wearing face masks | 56 | A woman wearing a face mask while looking at her cell phone |
| Hands holding objects | 55 | A pair of dirty hands holding a pair of scissors |
| Group of men | 52 | A group of people in white suits working in a room |
| US Supreme Court | 27 | A group of people standing in front of the supreme court |
| President Buhari Muhammadu | 26 | President Muhammadu Buhari has been elected president of Nigeria |
| Family, field | 20 | A family is holding hands in front of a green field |
| Contraceptives | 17 | Contraceptive pills and contraceptives are on a table |
| Unsorted Images | 282 | A couple of stuffed animals that are next to each other |
| Images in small clusters, not listed above | 394 | A faucet with water pouring out |
| Total | 2281 |
| Topic | Count | Example |
| Group of people standing | 407 | A group of people sitting around a table in a room |
| Man wearing suit and tie | 299 | A man wearing a suit and tie |
| Flag, arabic writing | 141 | A man sitting at a desk in front of two flags |
| Woman wearing head scarf | 140 | A woman wearing a head scarf sitting at a desk |
| Building | 91 | An aerial view of an urban area with buildings |
| Person wearing glasses | 70 | Person wearing glasses |
| Two people | 68 | Two men sitting on rocks near a body of water |
| Busy street with traffic | 59 | A busy street filled with lots of people |
| Group of people wearing masks | 47 | A group of people sitting in chairs wearing masks |
| Pregnant woman, babies | 28 | A woman holding a baby in her arms |
| Family | 28 | A person holding paper cut-outs of a family |
| Body of water | 22 | An aerial view of a large body of water |
| Doctors, medicine | 22 | An older man is being examined by a doctor |
| Variety of food | 11 | There are many different kinds of fruit on the table |
| Unsorted Images | 93 | A breast cancer awareness pink ribbon is shown |
| Images in small clusters, not listed above | 30 | An image of an orange line going down a graph |
| Total | 1576 |
| Topic | Count | Example |
| Animals | 36 | A group of swans floating on top of a body of water |
| Man and woman | 34 | A woman in a blue dress standing next to a man in a black jacket |
| Woman posing for picture | 40 | A woman in a see-through outfit on the red carpet |
| Buildings, urban | 20 | An aerial view of a residential area with many trees |
| Group of people, standing | 18 | A group of people standing around a conference room table |
| Babies, children | 17 | A woman holding a baby in her arms |
| Suit and tie | 13 | A person wearing a suit and tie |
| Pregnant women | 12 | A pregnant woman is holding her belly with two hands |
| Pills, contraception | 10 | A close up view of a variety of contraceptive pills |
| Unsorted Images | 33 | A man drinking a cup of coffee while looking at a laptop |
| Images in small clusters, not listed above | 38 | An image of a video game screen |
| Total | 256 |
| Topic | Count | Example |
| Babies, children | 54 | A man and a woman holding a little girl |
| Person standing | 32 | A group of young people in graduation attire |
| People wearing face masks | 30 | A man and a woman wearing face masks |
| Buildings, signage | 29 | A building that has a huawei logo on it |
| Portrait of woman | 24 | A woman in front of a world map |
| Two people | 18 | A composite image of a woman and a man |
| Misc. objects | 16 | Two cans of beer are side by side |
| Computer, laptop | 12 | A woman sitting at a table using a laptop computer |
| Group of people sitting | 11 | A group of people sitting around a table |
| People, street | 11 | A crowd of people walking across a street |
| Suit and tie, microphone | 19 | A person wearing a suit and tie |
| Graphs, charts | 7 | A bar chart showing the number of women in managerial positions |
| Unsorted Images | 37 | A deer standing next to a tree in the woods |
| Images in small clusters, not listed above | 69 | A man in a vest and glasses standing in front of a tree |
| Total | 332 |
| Topic | Count | Example |
| Close-up of woman | 313 | A woman smiling with her hands near her face |
| Man and woman | 162 | A man and a woman are posing for a picture |
| Man in suit and tie, microphone | 141 | A man in a suit and tie holding a microphone |
| Pregnant women, babies | 139 | A young woman holding a pregnancy test in her hand |
| Group standing on street | 111 | A group of people standing under a street light |
| Surgical masks, lab coats | 109 | A woman wearing a surgical mask looking at a clipboard |
| Close-up of man | 100 | A split photo of a man and a man with a beard |
| Person laying on bed, sitting on couch | 81 | A woman laying in a bed next to a man |
| Buildings, urban | 78 | A police car parked in front of a white building |
| Cell phones, calendars | 75 | A person holding a cell phone in their hand |
| Pills, pill bottles | 62 | There is an image of a menstrual cup next to a contraceptive pill |
| Group holding hands | 50 | A group of people standing around a banner |
| People sitting on ground | 48 | A woman sitting on the ground with her hands on her face |
| People sitting at table | 47 | A group of people sitting at a table talking |
| Patients, Medical staff | 46 | A group of surgeons performing surgery on a patient |
| Statues | 40 | An old statue in the middle of a field |
| Computer screens | 35 | A computer screen with a black background |
| Older women | 35 | An older woman in a purple floral shirt holding a basket |
| Group of women | 35 | A collage of three women holding a megaphone |
| Body of water, city | 29 | An aerial view of a city with a body of water in the background |
| Unsorted Images | 269 | There are many balloons that are on the table |
| Images in small clusters, not listed above | 268 | A cat sitting on the ground wearing a neck tie |
| Total | 2303 |
| Topic | Count | Example |
| Group of people standing | 361 | A large group of people standing in the street |
| Man in tie and suit, microphone | 332 | A man in a suit and tie sitting in front of a microphone |
| Woman with long hair | 221 | A woman with long black hair wearing a white dress |
| Children and babies | 211 | A man and a woman holding a baby |
| Exotic animals | 158 | An illustration of a group of elephants and birds |
| Signs, Flags | 156 | A person holding up a sign in front of a building |
| Man and woman | 149 | A man and a woman are posing for a picture |
| Lab coats, stethoscopes | 138 | A person in a white lab coat and blue gloves |
| People wearing face masks | 119 | A man wearing a face mask sitting at a table |
| Buildings, urban | 90 | An aerial view of a city with many buildings |
| Cell phones | 88 | A hand is holding a smart phone with a logo on the screen |
| Henna, hands | 84 | A person 's hands with henna tattoos |
| Pregnant women | 65 | A woman holding her pregnant belly |
| Close-up of person | 51 | A person wearing a white shirt |
| Named political figures | 46 | Narendra Modi in front of a microphone |
| Floods, bodies of water | 44 | A group of people in life jackets and life jackets in the water |
| Group of men | 43 | Two men standing next to each other in front of flower |
| Domestic Animals | 43 | A group of dogs laying on top of a sandy beach |
| Contraceptives, pills | 39 | A person holding a contraceptive pill |
| Unsorted Images | 278 | There is a person that is holding a model of a brain |
| Images in small clusters, not listed above | 28 | A woman is brushing a boy's teeth with an electric toothbrush |
| Total | 3153 |