Mitigating Harms of Digitally Manipulated Media

Why Digitally Manipulated Media?

In our last-in person sprint in November 2023, we tried to arrive at issue areas surrounding digitally manipulated images and videos (henceforth, DMM). National elections are just around the corner (some state-wide ones have just finished), and the prevalence of manipulated media has already increased. Besides it affecting the dynamics of online and offline life, there has also been interest in the issue from international organisations, and an increasing number of media agencies are also investigating and fact-checking DMM. With the unbounded growth of generative AI applications and the susceptibility of users to fall victim to such content, manipulated media is a growing and perilous issue.

Why use the term Digitally Manipulated Media

While popular and journalistic writing often uses the word deepfake, we find it analytically confusing. Manipulated videos and images often use a combination of techniques: some involving AI, some not. We use digitally manipulated media as a term that covers the range of manipulations from cheap or shallow fakes to deepfakes. The goal for this sprint was to understand the entire spectrum, and the range of possible approaches for response. Some kinds of manipulations might be easier to detect technically while some might be too sophisticated for existing automated detection techniques. Hence we use the term digitally manipulated media as a catch-all term for all manipulations to videos and images.

Areas of Intervention

Through the week long sprint, we identified different points/areas of intervention to counter DMM. Following a broad mapping of all possible interventions, we arrived at specific areas where Tattle is best placed to intervene. You can read more about the

Taxonomy

• Characterization of properties/features of DMM in terms of its domain and the manipulation technique used to forge them; evaluating what kind of secondary analysis can be done from this information (like potential impact)
• Following the Donovan continuum of shallow/cheapfake → deepfake
• The importance of images in an increasingly non-text world and how this impacts the flow of information
• Quantify deep fakes vs cheap fakes
• Different kinds of images - memes vs non-humour

Metadata

DMM detection, documentation and identification can benefit from metadata labels. During the sprint, what metadata labels could apply were discussed. What metadata would be relevant when dealing with DMMs? They could either be related to subject matter; they could be related to source or otherwise involve network analysis; or they could relate to technical properties of the image. Some initial labels the group agreed on are identifying the DMM technique used to alter the image, availability of both original and altered image, and broad area of interest/subject matter (ie news, politics, gender etc).

We tried to come up with a schema that could account for changes in the future. For instance, one could have a metadata field to mark if there is a forgery in an image. One could imagine a scenario where this forgery was detected by a certain algorithm and not a human. In this case, we felt that our metadata schema needed to consider fields that could help capture this. A possible schema for this could have been:

identified by
	human
		ID
	machine
		technique_name
		url
		sha
		confidence_score

We reviewed ML, image forensics and Social Science papers on relevant topics and made an incomplete list of useful metadata. Some of them are:

authoritative event description(s)
source
forgery_technique
domain
	political, health etc

We did a preliminary literature review on datasets containing forged images. We found some technical metadata fields from the literature - forgery category (real/fake), bounding box, segmentation mask, forgery boundary, facial landmarks. Some other fields that could be are gender, age, skin colour, face occlusion, background scenery in the image.

Repository

Having a repository of some kind to assist with aspects of countering DMMs came up as being useful, with a few possible functions for this repository. It could act as a single source of truth for images on certain widely used platforms/protocols, where a digitally manipulated media could be identified against the original. The use of watermarks and other proofs of authenticity could be assisted by this repository. It could serve to counter cases where there are either repeat offenses or regular targets of DMM. There are precedents for this in projects like StopNCII which maintain a hash database of non-consensual images and help platforms take them down.

Detection

• Investigating the type of AI tools and other digital technology required to detect forgery and the level of human assistance required in executing/running such technology
• Checking to see who are the most frequent victims of DMM and building a case base around them

Blocker

• Platforms can flag/label content that is DMM and warn users
• Once such a post gets flagged as such, it stops travelling through users' feeds
• Functionalities can be built into Uli that allow it to redact DMM content

Social Network Analysis

Social Media websites like X (Twitter) can be thought of in the context of a social network (graph). Where there are relationships between users, these relationships are interactions between the users. Interactions are in the form of retweeting and following each other, where each user is a node and each interaction is an edge in the network. Previous work has looked at the spread of text-based hateful content on Twitter from a network level. It would be an interesting study to look at the spread of deepfake images from a similar network perspective. Some questions that can be answered through such a study are what happens to the information over time and what are some qualitative and quantitative properties of users posting deepfake content.

Helpline

• A LLM can be built into Uli so that the tool can respond to queries, related to DMM, filed through it
• A trusted channel/helpline can be constructed to provide solutions/remediation
• We can facilitate the creation of support groups for victims

Social Awareness

• Nudity can be normalised to prevent the sharing of sex-based videos and any coercive/discriminatory behaviours and practices that arise from it
• Gender sensitising all platforms through advocacy so that they are aware of the gendered nature and impact of DMM
• Public service announcements and how-to videos of citizens/users can safeguard themselves against DMM through the detection of forged media

Survivor Storytelling

• Elucidating the impact of DMM through storytelling led by the survivors/victims of such attacks can then lead to the creation of support groups
• Elaborating on how DMM affects the perception of reality vs textual information

Law

• Facilitating law enforcement to act upon digitally manipulated media

Platform

• Having ownership rights over images and operationalising them through platforms
• Building out specific reporting features/tools on platforms that enable users to report DMM
• Social media platforms such as Twitter creating a feature through which users can add annotations about a specific DMM

Things We Have Learned from the Sprint

We opted to construct a taxonomy of cases of DMM we could find via Google News and through news reportage. For this activity, we looked at the domain that the DMM touched upon (from politics, sports, entertainment, economy and elections) and also noted down the manipulation technique used to create the DMM. Alongside this, we are constructing a dataset of these cases with the originally morphed image and also some metadata labels, that will help us automate the detection of morphed images when looking at larger datasets. You can see the WIP dataset here.

We also noted the following observations from our activity:

• Joan Donovan uses the Deepfake to Cheapfake spectrum but it isn't clear that the linearity is a technical fact or a subjective assessment of what appears more real/ more doctored
• Where DMM is harder to detect through technical means, there might be more space for a central actor - regulator/law enforcement agency or platform
• Images are a common modality for cross-posting on platforms. We make a distinction in the screenshots of impostor accounts, vs. doctoring of images of real events/accounts. So, where the manipulation originates is important. In DMMs, the manipulation happens in images:
• This is not a DMM:
  • No, Ricky Ponting Did Not Call BCCI A 'Cricket Mafia'; Fake Quote Viral | BOOM
  • News Outlets Including TOI, Zee News, PTI Fall for Amartya...
  • How False Claims Of India's GDP Touching $4 Trillion Went Viral | BOOM
• This is a DMM:
  • Fact Check: Morphed image of AAP banner says now Gujarat will read namaaz
  • Video Of Mukesh Ambani Asking People To Invest In 'Laila...
• Absence of an overview focused on and in Global South/Majority World
• Image forensics is not a prevalent public discourse
• How public-ness is weaponized
  • Two strategies for using AI: face-swap and lip-syncing. Both rely on the availability of videos
  • A public figure is used to generate legitimacy

You can follow our future developments on this page.