🌏 MultiClimate: Multimodal Stance Detection on Climate Change Videos 🌎

Abstract

Climate change (CC) has attracted increasing attention in NLP in recent years. However, detecting the stance on CC in multimodal data is understudied and remains challenging due to a lack of reliable datasets. To improve the understanding of public opinions and communication strategies, this paper presents MultiClimate, the first open-source manually-annotated stance detection dataset with 100 CC-related YouTube videos and 4,209 frame-transcript pairs. We deploy state-of-the-art vision and language models, as well as multimodal models for MultiClimate stance detection. Results show that text-only BERT significantly outperforms image-only ResNet50 and ViT. Combining both modalities achieves state-of-the-art, 0.747/0.749 in accuracy/F1. Our 100M-sized fusion models also beat CLIP and BLIP, as well as the much larger 9B-sized multimodal IDEFICS and text-only Llama3 and Gemma2, indicating that multimodal stance detection remains challenging for large language models.

MultiClimate Dataset Examples

Image-transcript (Label)	Example 1	Example 2	Example 3
Image (Support)
Transcript (Support)	Together we can address the causes and impacts of climate change to ensure that our spectacular ocean and great lakes continue to thrive while supporting our economies and values.	Together we can address the causes and impacts of climate change to ensure that our spectacular ocean and great lakes continue to thrive while supporting our economies and values.	And revealing their wonders for generations to come.
Image (Neutral)
Transcript (Neutral)	[Music]	[Music]	[Music]
Image (Oppose)
Transcript (Oppose)	This double threat of climate change and war pushes people out of their homes, breaks food and supply chains amplifies diseases and weakens health care services.	By 2050 the need for humanitarian aid is expected to double if urgent steps are not taken.	We cannot let this happen.

Overview Workflow

1. Filter videos by CC (Creative Commons) LICENSE

2. Make sure there is the transcript and available in English
   E.g., The UK's Climate Change Act is a unique law does not have the transcript because the voice channel is only pure background music.
   E.g., Climate Change: The Water Paradigm only provides Vietnamese
   E.g., The IAEA’s Multifaceted Approach to Climate Change only has French
   E.g., Which countries are historically responsible for climate change? has no sound
   E.g., Sea Otters that Fight Climate Change only background music
   E.g., Listening to Women: adapting to climate change mainly in English, but interviewees with other languages so no transcripts in these parts
   E.g., Forest conservation as a critical buffer subtitles disabled
   E.g., Climate Change subtitles disabled
   E.g., The consequences of climate change subtitles disabled
   E.g., This is Earth in the next 100 years if we don't act on climate change | Ways to Change the World only background music
   E.g., Untold stories of climate change loss and damage in the LDCs: Rwanda subtitles disabled
   E.g., Untold stories of climate change loss and damage in the LDCs: Nepal (Nepali) only in Nepali
   E.g., Impact of Climate Change on Snow Leopard Habitat ony background music
   E.g., A new course for the Congo: Forests, climate change and education subtitles in French
   E.g., Combating the effects of climate change in the Caribbean subtitles not in English
   E.g., Forests Migrate But Not Fast Enough For Climate Change | World Economic Forum only background music
   E.g., Protagonists, Together against climate change. subtitles in Spain
   E.g., Fighting climate change with underwater meadows | UpLink only background music

3. Download the video and scripts using youtube.ipynb
   Now also frame extraction in youtube.ipynb, first image is also removed, just need to change the URL, id and abbreviation of the video (step 3 to step 4.2 included)

4. Extract a series of images from a given video HCCAW.mp4 (03:56)?

   1. Extract frame every 5 seconds using ffmpeg -i CCUIM.mp4 -r 0.2 -start_number 0 CCUIM-%03d.jpg
   2. Remove the first image -000.jpg
   3. Add the title of the video as the textual part for the image -001.jpg (0 seconds)

5. Replace the time to label ( 0️⃣:neutral, 1️⃣:support, 2️⃣:oppose): human/machine labeling the ground-truth

6. Evaluate via accuracy, weighted F1 score

Alternative: Extract scene-changing frames

ffmpeg -i HCCAW.mp4 -vf "select='gt(scene,0.4)'" -vsync vfr frame-%2d.jpg

Dataset Statistics

Set	#	Videos	0️⃣ (16)	1️⃣ (58)	2️⃣ (26)
Train🔵	3,372	Others	1,036	1,449	887
Dev🟣	417	CCGFS, CCIAP, CICC, EFCC, FIJI, HCCAB, HRDCC, HUSNS, MACC, SAPFS	83	204	130
Test🔴	420	ACCFP, CCAH, CCSAD, CCUIM, EIB, EWCC, GGCC, SCCC, TICC, WICC	73	194	153
Total	4,209		1,192	1,847	1,170

Dataset Overview

• 👍: Videos (animations) selected and annotated by both annotators
• ✅: Check sign that full sentences and relabeling are done

#	Youtube-ID	Title	Link	Annotators (IAA)	Label
👍1	brN2_YNAEK8	About the Committee on Climate Change	ACCC	✅Jiawen, Longfei (0.9152)🔵	1️⃣
👍2	NTP5a1M2Z6I	Addressing Climate Change through Fiscal Policies in Indonesia	ACCFP	✅Jiawen, Longfei (0.6978)🔴	1️⃣
3	t-MiN_0kYhM	Analysing impacts of air quality policies on health and climate change | Denise Mauzerall	AIAQ	✅Jiawen	1️⃣
4	BQr593iCEn8	AI and digital twins: Tools to tackle climate change	AIDT	✅Jiawen	1️⃣
5	T4CTgXYI2kA	1-Arctic Methane and Climate Change	AMCC	✅Jiawen	0️⃣
6	xajNochi7tM	Big data and climate change	BDCC	✅Jiawen	0️⃣
7	eTqKLJ_o9yQ	Bitcoin Energy Consumption & Climate Change: Does Bitcoin Use Too Much Energy? [2021]	BECCC	✅Jiawen	2️⃣
8	iKzdIzN4q2Q	Belize: Women in Fisheries Forum 5: Gender and Climate Change - Understanding the Link	BWFF	✅Jiawen	1️⃣
9	s4ly6o-VT90	Connections between air quality and climate - English - Sept. 2021	CBAQC	✅Jiawen	2️⃣
👍10	FGs2QQWFqyQ	The Crucial Connection: Climate Change and Health | Kaiser Permanente	CCAH	✅Jiawen, Longfei (0.7778)🔴	2️⃣
11	1tGWJ-NkcGU	Climate change, biodiversity and nutrition - Helping local heroes tell their stories	CCBN	✅Jiawen	1️⃣
12	lAop3wreUek	Climate change, biodiversity and nutrition nexus	CCBNN	✅Jiawen	1️⃣
13	4VXSrQospVY	Can climate change and biodiversity loss be tackled together?	CCCBL	✅Jiawen	1️⃣
14	DRXQ9ixPbD8	Combating climate change in the Pacific	CCCP	✅Jiawen	1️⃣
15	TdxNG8L4JCM	Climate Change and Conflict in Somalia	CCCS	✅Longfei	1️⃣
16	8fbrnAAg7VM	Climate change and development	CCD	✅Jiawen	2️⃣
17	nXOB8YPyc04	Climate Change and Food Supply	CCFS	✅Jiawen	2️⃣
18	MNdF-eVRWX4	Climate Change Fuelling Wilder Weather	CCFWW	✅Jiawen	2️⃣
19	v24wT16OU2w	Climate Change, Global Food Security, and the U.S. Food System	CCGFS	✅Jiawen🟣	1️⃣
20	CA8iTY7iMCk	Climate Change and our Health (ADB Insight Full Episode)	CCH	✅Jiawen	1️⃣
21	mPE7D0wRYoU	Climate Change: Health Equity Stories from The Colorado Trust (English subtitles)	CCHES	✅Jiawen	2️⃣
22	KxBAiad3Xto	Climate change in the Australian Alps	CCIAA	✅Jiawen	1️⃣
23	yeih2v4P25A	Climate Change: It’s About Health | Kaiser Permanente	CCIAH	✅Jiawen	1️⃣
👍24	m95K7LClIC4	350.org - Climate Change Is About Power	CCIAP	✅Jiawen, Longfei (0.6716)🟣	1️⃣
25	b919Fb-P3N8	Climate Change Impacts for Canadian Directors	CCICD	✅Jiawen	2️⃣
26	A7ktYbVwr90	Climate change is simple: David Roberts at TEDxTheEvergreenStateCollege	CCIS	✅Jiawen	2️⃣
27	yVvVk2zNSbo	Untold stories of climate change loss and damage in the LDCs: Sierra Leone	CCISL	✅Jiawen	2️⃣
28	lNBP7aRskVE	Climate Change: Mitigate or Adapt	CCMA	✅Jiawen	1️⃣
👍29	5DVa8xBgToc	Climate Change for South African Directors	CCSAD	✅Jiawen, Longfei (0.4080)🔴	1️⃣
👍30	GeksVaAnMzc	Climate Change or Social Change: The Role of Blockchain	CCSC	✅Jiawen, Longfei (0.3244)	1️⃣
31	zI9h-HTBHO8	Climate Change: Take Action	CCTA	✅Jiawen	1️⃣
32	fN-ZnY61_C8	Climate Change: The Philippines	CCTP	✅Jiawen	1️⃣
👍33	Vve6zge_RsA	Climate change unlikely to increase malaria burden in West Africa	CCUIM	✅Jiawen, Longfei (0.6328)🔴	0️⃣
34	RZ-N5KwBaVc	Climate Change is Water Change	CCWC	✅Jiawen	1️⃣
35	MnTm89dSHhA	Climate Change and Water Quality	CCWQ	✅Jiawen	1️⃣
36	OL8a1YEhk_o	Honest Government Ad | Climate Emergency & School Strikes	CESS	✅Jiawen	2️⃣
👍37	b7LiW66cSM4	How should Coronavirus influence the fight against Climate Change?	CICC	✅Jiawen, Longfei (0.5681)🟣	1️⃣
38	vD0lx_b8jNM	COP28: Nuclear Science and Technology for Climate Change Adaptation	COP	✅Jiawen	1️⃣
39	MeFbo0z0xi4	Crop production - Climate change affects biosphere | Earth Hazards | meriSTEM	CPCC	✅Jiawen	1️⃣
40	FhyUbeDVM3k	Capturing and transforming CO2 to mitigate climate change	CTCM	✅Jiawen	1️⃣
41	Gu5NKLxqTak	Diet and Climate Change: Cooking Up a Storm	DACC	✅Jiawen	1️⃣
42	OfYGx-N_gB0	Deforestation and Climate Change	DFCC	✅Jiawen	1️⃣
43	CG3pN7qQqZI	This Is How Denmark Protects Its Cities Against Climate Change	DPIC	✅Jiawen	1️⃣
44	Ry-ei9Bu8UI	Developing tools for equality in climate change planning in Tanzania	DTECC	✅Jiawen	1️⃣
45	M17pm2iPT_c	Effects Of Climate Change In MN Discussed At Seminar	ECCDS	✅Jiawen	1️⃣
46	HzL9hUOh_K4	Ecosystems at risk from Climate Change	EFCC	✅Jiawen🟣	1️⃣
👍47	kjTAWBPPez0	Earth Is Blue: Climate Change in your Sanctuaries	EIB	✅Jiawen, Longfei (0.6474)🔴	1️⃣
👍48	Qmxg97Ae9Wg	Extreme Weather and Climate Change, EarthNow	EWCC	✅Jiawen, Longfei (0.6903)🔴	2️⃣
👍49	Wrb4b28dgcU	Forests and Climate Change	FCC	✅Jiawen, Longfei (0.6061)	2️⃣
50	9DaUn0geq4U	Fiji: Standing tall against climate change threats	FIJI	✅Jiawen🟣	2️⃣
👍51	ison6lQozDU	Food loss and waste are among the main causes of climate change	FLW	✅Jiawen, Longfei (0.8182)🔵	2️⃣
52	M9wSP16P9xM	+Forest, together against climate change	FTACC	✅Longfei	1️⃣
👍53	epZ9Rw-i8Mo	Greenland's glaciers and Climate Change, Danish Broadcasting Corporation - Denmark	GGCC	Jiawen, ✅Longfei (0.7362)🔴	2️⃣
54	S9Z_h1_LQ0o	How Climate Change Affects Biodiversity	HCCAB	✅Longfei🟣	1️⃣
55	me14ikumMZE	How Climate Change Affects the Ecosystem	HCCAE	✅Longfei	2️⃣
56	CGoNpwN0mrs	How Climate Change /Actually/ Works...in 4 Minutes	HCCAW	✅Longfei	0️⃣
57	KTA5onaECFE	How climate change influences geopolitics – Interview with Francesco Femia	HCCIG	✅Longfei	2️⃣
58	RFsxDqQWjhk	How COVID-19 is impacting air pollution and climate change	HCI	✅Longfei	1️⃣
59	ivN1QIvdBUI	How do we change our attitude towards climate change? Christiana Figueres	HDWC	✅Longfei	1️⃣
60	jDueuwB3Tcs	Human Health, Vector-Borne Diseases, and Climate Change	HHVBD	✅Longfei	0️⃣
61	e_8upuAySOI	Human Rights Day 2012: Climate Change and Human Rights	HRDCC	✅Longfei🟣	2️⃣
62	vsbcasoudtM	How are scientists helping whale-watchers adapt to climate change?	HSHWA	✅Longfei	0️⃣
👍63	mc1qAnGGGCE	How Solar Panels Work To Reduce Climate Change	HSPW	✅Jiawen, Longfei (0.9255)🔵	1️⃣
64	RYsZ0NdHKyc	How the US National Security Policy Incorporates Climate Change – Interview with Alice Hill	HUSNS	✅Longfei🟣	1️⃣
65	OSA944ShtmE	IMRF: Statement from IOM’s Migration, Environment, Climate Change, and Risk Reduction Division	IMRF	✅Longfei	1️⃣
66	OKPzj-l7gp0	INCAS: Monitoring for Climate Change	INCAS	✅Longfei	0️⃣
67	ks7rCR7-mF0	Migration and Climate Change in the Global Compact for Migration	MACC	✅Longfei🟣	1️⃣
👍68	ko4cUnzoPic	Science Action: What's the unique role of methane in climate change?	MICC	✅Jiawen, Longfei (0.3641)	2️⃣
69	vB3_49ULzf0	NASA's Climate Advisor Discusses Climate Change	NASA	✅Longfei	1️⃣
70	EwHtHsBeRIA	Overview of the C-ROADS Climate Change Policy Simulator	OCCC	✅Longfei	0️⃣
71	CUdBaExvHy4	President Clinton On the Cost of Climate Change	PCOCC	✅Longfei	1️⃣
72	A6uRlax7AuE	Preserve the Wonder - Climate change action	PWCCA	✅Longfei	1️⃣
73	yTo3zmn3u84	Removing atmospheric greenhouse gases to prevent dangerous climate change	RAGG	✅Longfei	1️⃣
👍74	_P31w8E_5Zc	Regenerative Agriculture: A Solution to Climate Change	RASCC	✅Jiawen, Longfei (0.6393)	1️⃣
75	Btqy4-H03dE	Ron Cohen on "BEACO2N" Climate Change Sensors	RCCCS	✅Longfei	1️⃣
76	fCo77PW2G6Y	Reframing climate change science to include indigenous and local knowledge - Dr Tero Mustonen	RCCS	✅Longfei	0️⃣
77	s3ViLeAG6_U	Richard H Thaler on climate change	RHTCC	✅Longfei	0️⃣
78	G6JfQwonm78	Rare plant - Snakeshead Fritillary - defies climate change	RPDCC	✅Longfei	1️⃣
79	K4dpmfzEASo	Strengthening Africa’s pastoral food systems transformation in the face of climate change	SAPFS	✅Longfei🟣	0️⃣
👍80	N2l_AjZGVQo	System change NOT Climate change: Can we leverage the digital age to get there? - D^2S Agenda	SCCC	✅Jiawen, Longfei (0.7243)🔴	1️⃣
81	se-BRvZuu7k	Scientists drill deep in Antarctic ice for clues to climate change	SDDA	✅Longfei	0️⃣
82	ZsbSI8UrPYA	Saint Lucia and Climate Change Adaptation (English)	SLCCA	✅Longfei	1️⃣
83	id4DZ0NiKk4	Stanford Students Tackle Climate Change	SSTCC	✅Longfei	0️⃣
84	pCraV8ahpYo	The connections between climate change and mental health	TCBCC	✅Longfei	0️⃣
85	tqavP5lotNo	Transforming our Economy to Combat Climate Change	TECCC	✅Longfei	1️⃣
👍86	L555lOp_0pQ	Trade, Investment, and Climate Change in Asia and the Pacific	TICC	Jiawen, ✅Longfei (0.8719)🔴	1️⃣
87	xWYwSgvZh38	The inequalities of climate change - ICRC	TIOCC	✅Longfei	2️⃣
88	jsZ2_WFtlDU	Things in Your Home that are Linked to Climate Change	TIYH	✅Longfei	1️⃣
89	763lGy43spk	The technology fighting climate change	TTFCC	✅Longfei	1️⃣
90	5KtGg-Lvxso	To understand climate change, understand these three numbers.	TUCC	✅Longfei	0️⃣
91	SDxmlvGiV9k	UK Climate Change Risk Assessment 2017 - Urgent priorities for the UK	UKCC	✅Longfei	2️⃣
92	eIcWgCjTHWM	Voices from Vanuatu: Climate Change Impacts and Human Mobility	VFVCC	✅Longfei	2️⃣
93	ii9mx391VVk	View from the Pacific: ‘Climate change is real’	VPCC	✅Longfei	1️⃣
👍94	_IVDYaQDNCg	Wildfires and Climate Change Attribution: It's Complicated!	WCCA	Jiawen, ✅Longfei (0.3478)	2️⃣
95	TM_6C9szLOI	Why focus on human security when working on climate change adaptation?	WFHSW	✅Longfei	1️⃣
👍96	MPiFBW0NnWY	What is Climate Change?	WICC	Jiawen, ✅Longfei (0.8381)🔴	2️⃣
97	mgBYo6eG80U	What is climate change? | Earth Hazards | meriSTEM	WICCE	✅Longfei	2️⃣
98	iXvyExAzQ58	What is the Impact of Solar Energy and Solar Panels on Climate Change?	WISE	✅Longfei	1️⃣
99	iFmoMhVb6iw	Cuomo: Walk the Talk on Climate Change	WTCC	✅Longfei	1️⃣
100	6ObqydfPGLI	Yale Professor Tony Leiserowitz Discusses American Perceptions of Climate Change	YPTL	✅Longfei	0️⃣

Note that we use a maximum of 5 characters when naming the link for simplicity. If frame-transcript labels in one video are same, we rank 2️⃣ > 1️⃣ > 0️⃣ due to inbalanced data.

Citation

Please kindly cite if you find this repository helpful.

@article{wang2024multiclimate,
  title={MultiClimate: Multimodal Stance Detection on Climate Change Videos},
  author={Wang, Jiawen and Zuo, Longfei and Peng, Siyao and Plank, Barbara},
  journal={arXiv preprint arXiv:2409.18346},
  year={2024}
}