Signs, Symbols and Other Visual Clues

By Marek Tuszynski

In Short: How to investigate or aid your investigation by exploring visual signs or symbols, especially in cases where very little direct information is visible, available or accessible.

This is an example of a rather bad use of signs to help visitors navigate an unknown space. These red footprints are supposed to help visitors find the emergency department registration desk in one of the largest hospitals in Berlin, Germany. Why red? - Foto by Marek Tuszynski

Imagine someone having a moment of anxiety in the midst of a medical emergency and seeing the red footprints in the picture above. What meaning is being conveyed?

Here we focus on various types of signs, symbols and images and explore the meanings of these visuals, which may help you move forward with your investigation.

Exploring or deciphering the meanings of symbols and signs is not a new field. It is, however, more familiar to those who study iconography (the meaning of symbols and signs) in areas such as painting, sculpture and architecture. It is also integral to the study of archaeology, ethnography, sociology and other fields. The study or investigation of organised criminal actors such as gangs and crime syndicates (just think of Yakuza in Japan) often relies on understanding their visual languages, for example: tattoos, patterns and symbols on fabrics, tags and other ways of visually marking bodies or objects.

We rely on symbols in almost all aspects of our everyday lives. We buy products marked with symbols and labels to help us make choices about what to consume; we use stickers, badges and other forms of markings to define our broader identities. This phenomenon is as common for sport fans as it is for music enthusiasts. It is also extremely common in different occupations, from forestry to engineering to the military.

We use visual codes to sort things, to register them or to disguise their provenance, to mark territories, distinguish functions or convey specific messages – which may often be messages of resistance or commentary. Think of labels, tags and painted marks as well as emojis, ASCII codes, memes or even icons. Symbols and signs constitute a much wider form of communication beyond the reach of just words.

We see symbols and signs everywhere: on buildings, pavement, clothes, devices, trees and vehicles. With the proliferation of digital communication, our use of symbols and signs has expanded as much as our spaces of communication, beyond the limitations of our physical expressions and actions. Observing and analysing these new signs and symbols can help us understand what new forms of expression and communication actually mean, how they redefine the way we express ourselves, communicate, organise, mobilise and learn. And they might constitute critical clues, references, identities or other traces that can enrich the way we investigate.

A single sign might mean many things; but it might also be misleading or meaningless in a specific context. On the other hand, when accumulated, signs can change our understanding of what they represent or signify, especially if they add up to certain visual patterns or narratives that can tell us stories we wouldn’t be able to see otherwise. They can help us define actors, their roles and even their intentions. However, while it is fairly easy to find correlations between various types of visual materials, correlation is not a proof or guarantee of causation. You need to find multiple ways of verifying assumptions based on patterns, connections or hidden meanings behind symbols and signs, especially in their original context.

Signs and symbols as evidence

First, we should define the difference between a sign and a symbol - not that it matters, but it might be useful when thinking about them.

One distinction is that a sign extends language and communicates with people who have incorporated it to their vocabulary. A sign can also be made by gesture or lack of gesture; a sign often has some direct communication behind it (such as an instruction). A symbol, on the other hand, is more ephemeral, as it is not an instruction but rather a representation of something (e.g. process, function and other things like objects).

This section is not about how to distinguish between these two and other forms of visual or non-visual clues, but rather about how to expand one’s field of vision to not miss or neglect them while investigating.

When studying signs and symbols in our investigations, it’s important to consider that there are several different types of them, roughly grouped into 3 categories, of which the first two are the most common: formal and informal signs or symbols.

Formal signs and symbols

Some examples of formal signs and symbols are things like car registration plates, or tail numbers of planes, or symbols used on products and devices. These are determined by specific governmental or corporate standards or rules, and usually need to be linked to some documentation and processes. This often means that we can find their meanings somewhere and we can derive other information from them, such as potential ownership, historical records and past transactions, etc.

Informal sign and symbols

These can be used for different purposes, for example to make work more streamlined and efficient. Graffiti signs on pavement or graphical signs on trees might be placed there to signal certain information to particular workers, for example. They are intentional but often temporary, often arbitrary and might mean different things in different places. Looking at them and understanding the context in which they appear is crucial. By observing these informal signs we can learn something about past, present or future activities or presences in given spaces. This category also covers the use of symbols to manifest one’s affiliations. These might follow certain codes and behaviours, such as patches used by formal or informal groups to add an extra layer of belonging and recognition.

And there are also those Incidental or unintended signs and symbols.

These include a sort of visual metadata that might come from unintended behaviours such as using a specific tool like a printer that leaves a unique code on paper (see this case for extended reading on what printer codes can reveal) or device with its own characteristics like a specific font, tag, visual mark – or a unique way of making and creating things.

Note:

One might think that patterns and repetition would be one way to decode or interpret signs and symbols. Sometimes, however, a single occurrence of a sign or symbol might be just as relevant. Categorisations are useful when we try to understand things, but life is full of surprises. It is more important to stay open-minded and observant and think of expanding your field of vision around you, because sometimes one small clue might give you a big opening to a story you would not have otherwise been able to access.

On the other hand, we are saturated with signs and symbols. Some of them might also be there to confuse, mislead and intimidate us; some might be genuinely incidental and give you a false sense of connections that do not exist, relationships that never happened or might make you believe in total nonsense. Let’s approach signs and symbols with these precautions in mind.

From looking at various investigations from different fields, it is clear that it’s important to write about how we can use this method as a valid method – or sometimes the only available one – in establishing some initial parameters of an investigation. This is particularly the case when access to direct information is complicated or, frankly, not possible because information is either missing or is purposely obfuscated.

To illustrate how exploring and understanding signs and symbols can sometimes be an effective investigative approach, we will look at a few cases of investigators who use this method when digging into issues that are otherwise inaccessible to outsiders. We will look at the work of Tala F. Saleh, who investigated the sectarian distribution of actors in civil war-torn Beirut by analysing signs and symbols in political graffiti on the streets; at the works of a renowned artist collecting army badges and revealing entire networks of covert or secret US army divisions and their areas of operations and interest; at the findings of an artist reading telecommunication signs on pavements, streets and buildings to gain a better understanding of communication networks and their ownership in the city; at a researcher investigating gestures and artefacts captured in photographs documenting the politics of housing control in East Jerusalem; at a security expert mapping bots based on their visual representation; and at how reading hidden information from travel codes can reveal an abundance of details about people’s habits.

These are just a few powerful examples of how you can go about spotting, reading, understanding and exploring the meanings of signs and symbols in your investigation. While we do not offer a step-by-step guide here, we explore ways of figuring out what makes some people better at working with this method and when and how it can be used. It’s about encouraging a specific mindset of digging into the peripheral spaces where information can be found.

As much as these cases are fascinating and inspiring, as well as revealing and sometimes tragic, we would like to invite you to look beyond the actual stories and focus on the methods used. We want you to see them as a set of clearly defined tactics that could be applied to different contexts and situations.

Politics of graffiti, graffiti of politics

“Marking Beirut: A City Revealed through its Graffiti,” Tala F. Saleh, 2009. Photo of one of the book’s spreads documenting the visual research conducted by Tala F. Saleh. Book owned and photographed by the author of this text as of 2019.

While living and studying in Beirut, Tala F. Saleh became interested in the amount and diversity of graffiti expression happening in the city at the time. As an outsider, she decided to use her design and visualisation skills to look at the symbols embedded specifically in political graffiti. She did so – as she later said in her 2009 book, “Marking Beirut: A City Revealed through its Graffiti” – to gain an outsider’s view of an internal struggle.

Tala was in Beirut immediately after the war with Israel in the summer of 2006, and documented its dense graffiti symbolism until 2007. The questions she was asking then, as she points out in the book, were:

• “What does this stencil offer?

• Does it say anything in particular about the city and its communities?

• Who does it speak to?” Almost immediately, she realised that Beirut’s graffiti was very specific, in that it was mainly about politics, war and social struggle.

In the intense period of four summer months in 2007, she decided to photograph and document every single sign and stencil, street by street, wall after wall throughout the entire city centre. In doing so, she also realised that most of the visuals were comprised of political logos, slogans and related social commentary. By meticulously accumulating this visual material, she realised that graffiti in Beirut actually could be used to identify its neighbourhoods and communities, and therefore divided the city into politically affiliated areas.

“Marking Beirut: A City Revealed through its Graffiti,” Tala F. Saleh, 2009. Photo of two pages showing the documentation process of the visual research. Book owned and photographed by the author, ibid.

Tala conducted her work very systematically by walking and taking pictures, documenting the symbols on the map, accumulating images in a simple image database, deciphering logos and symbols and learning the meaning and context to build a consistent narrative that would constitute an outsider’s view of an internal struggle. It is worth noting that her work prior to this investigation already focused on graffiti in the city in its pre-war period (1975-2009) as well as on what she called “Western forms of political graffiti,” which enabled her to use her existing visual and critical skills in this process.

Tala’s analysis of Beirut’s graffitied slogans, logos and symbols focused on their style, medium, message, location, space and frequency (or repetition). She extended this study by looking at how this visual communication had been appropriated and changed over time. Just like many other forms of communication, it was dynamic and ever-changing. That led her to conclude that this visual culture is a culture of unofficial forms of propaganda and mass (local) communication, intended to keep the inhabitants of the city (the audience) aware of political and spacial divisions, opinions, conflicts and influences.

“Marking Beirut: A City Revealed through its Graffiti,” Tala F. Saleh, 2009. Photo of two pages documenting the geographical mapping of accumulations and appearance patterns of specific groups of symbols. Book owned and photographed by the author, ibid.

This investigation captured an ephemeral status quo of the city and its inhabitants, a status quo that practically disappeared before the book was published in 2009. Saleh’s work is not entirely unique: there were earlier attempts to work with graffiti as the political message and the walls of the city as the medium. An academic, Maria Chakhtoura, documented graffiti in Beirut in the period from 1975 to 1978 and published her work under the title “La Guerre Des Graffiti,” while there are other studies in reading and interpreting the fabric of the city through its graffiti culture.

The point of this example here goes beyond the visual language of graffiti. Nowadays, the same forms of representation, communication and affiliation are proliferating not only in urban and rural physical spaces but also in virtual spaces of social networks and online communities. Instead of graffiti, their members or frequenters are using memes, animated GIFs, emoticons, ASCII art, and hashtags to represent their views, affiliations, alliances and identities. These symbols constitute a significant visual material that carries a thick layer of information.

The makers of these new forms of visual propaganda and influence have their own style. What is more interesting from an investigative point of view is that they also carry other elements, which, in the case of graffiti, were seemingly untraceable, such as metadata. But even graffiti can contain a kind of metadata, and finding a way to collect and interpret that metadata is as important in offline formats as it is in online formats of visual information. Graffiti has its own style, unique tags, specific materials (types of paint), and appears in specific places and at specific times. These elements, when collected, might constitute significant metadata. Another issue is that these forms of spreading visual meanings are prone to different forms of erasure, such as overprinting or appropriation. The same visuals might also mean different things in different times and contexts.

Note:

Exploring symbolic visual languages does not have to focus on the political aspect of images, it can also explore the functional meaning and role that symbols play in various professions.

Let’s move from the inhabitants of the city and their symbolic expression to the workers of the city and the sign language they use to make their jobs more efficient. When reverse-engineered and understood by a bystander, these signs give us a unique understanding of urban infrastructure and its power dynamics.

We leave Beirut for New York and follow artist and writer Ingrid Burrington.

Professional efficiency

Cable markings from left to right: “fiber optic cable,” “duct width markings,” “point of entry,” company marking “xo communications.” All images from “Networks Of New York: An Illustrated Field Guide to Urban Internet Infrastructure,” Malville House, 2016, author Ingrid Burrington, images by Mariana Druckman.

Ingrid is also an explorer and author – though she doesn’t describe herself as such. She does not like unanswered questions; or more precisely, she dislikes not trying to answer even the weirdest questions one can ask. She is also a wanderer, a modern day flaneur exploring urban spaces, taking notes of the weirdest and made-to-be-invisible fragments of urban infrastructures, in particular those related to communications and the transfer of information. She takes pictures, reads manuals, talks to people, writes tonnes of notes and spends time putting it all together.

Manhole covers and handholes: “Time Warner Cable of New York,” manhole “East Jordan Iron Works - Communications,” “Verizon” handhole. All images from “Networks of New York: An Illustrated Field Guide to Urban Internet Infrastructure,” Malville House 2016, author Ingrid Burrington, images by Mariana Druckman.

From Ingrid’s work we can discover that learning the visual vocabulary of a group of experts can reveal more about the substance of their work than other sources. It can also lead to a better understanding of invisible structures and infrastructures – in her case, the underground networks of cables and services they deliver as well as their politics in terms of ownership, access and proliferation. On a more meta level, it also shows how old structures get reused and appropriated by new actors.

Ingrid’s work emphasises the importance of observing and learning informal visual languages of actors who might not represent the politics of what’s behind their work, but who are focused on the practical and logistical levels of their activities. However, because we often are all about streamlining and simplifying our work flows, we often use visual languages to help ourselves and others involved to do our work more efficiently. An important part of Ingrid’s practice is not only relying on intelligent guessing but also talking to people who make these signs – they are the best sources to explain their language.

The same method Ingrid uses can also be applied in non-urban contexts, where forestry workers often use visual symbols when planning specific types of work they will be undertaking. These signs and symbols – what we might also call “visual footprints” – might be left unintentionally by tools, devices or equipment that have been used in various processes of making or destroying things. For further examples illustrating these kind of clues, it is worth visiting the Center for Land Use Interpretation, which is a research and education organisation interested in understanding the nature and extent of human interaction with the surface of the earth, and in finding new meanings in the intentional and incidental forms that we individually and collectively create.

Now, from workers of the city let’s move to the specialists that operate under the privilege of secrecy, in particular those who don’t want to let outsiders know what they are actually up to – sometimes because they are working on confidential matters requiring higher security of national or global magnitude.

Strictly secret

“I Could Tell You But Then You Would Have to Be Destroyed By Me: Emblems from the Pentagon’s Black World,” Trevor Paglen, 2008, Second Melville House Printing, scan of the fragment of the book cover by the author of the text, 2019.

The entire work portfolio of American artist Trevor Paglen is a must-see and read for anyone who would like to gain expertise and learn the tactics of exposing the invisible, in this case very literally. Here, we look at his work that culminated in a book in 2008 called _**I Could Tell You But Then You Would Have To be Destroyed By Me: Emblems from The Pentagon’s Black World.

Paglen was trying to figure out how the budget for US government operations related to fighting terrorism campaigns since 9/11 was steadily growing, forming a so called “black budget” that officially appeared under the category of “Research, Development, Test and Evaluation” with names of projects and activities and undisclosed amounts of money. Looking at the difference between the total budget known to the public and the sum of disclosed numbers suggested that the amount going to the “black budget” was about $US 30 billion. Yet, how exactly it was spent, what sort of activities, structures or infrastructures it supported had been kept in the dark, mostly because of its relationship to national security. There was no formal, legal or direct way of exploring these questions.

Instead of pursuing traditional forms of finding such information, the artist turned to the collectors of military memorabilia: uniforms, buttons, wings, cords, ribbon racks, badges, etc. One of the collectors in particular, whom he calls Merlin, introduced him to what he called “patch intel” (intel stands for intelligence/information). He explained that however elusive these patches might look, with their somewhat naive symbols of stars, lighting bolts and other rather strange creatures or devices, they constitute a symbolic visual language enabling those who wear them to identify themselves and their affiliation. However, these symbols and patches illustrate a rather interesting paradox: why would classified activities often undertaken under state secrecy acquire visual representation that risks undermining all the efforts to keep their work behind closed doors?

What Paglen learned by amassing his collection of patches was that something as ephemeral as pride makes soldiers rely on insignia, and wearing insignia shows others that one is part of something bigger than themselves. As with other cases in this text, this is a very important and common characteristic, regardless of levels of secrecy or danger or a need for elusiveness. Groups of people undertaking such activities might purposefully or incidentally (driven by pride or need of belonging or simple recognition of their craft) create a visual language of symbols that when learned and deciphered might reveal a lot of nuanced information about who they are, what they are working on, where this work is happening, etc.

As above, this is a scan from the book presenting the patch from page 16, under the title “Special Projects Flight Test Squadron.” The author deciphers symbols visible on the patch as follows: “The collection of stars a group of 5 plus one reveals the number 51 aka secret base known as Area 51; the wizard is the mascot of the squadron; the symbol on the right hand side of the mascot is an engineering symbol representing an unknown value appearing on the radar – which stands for the stealth character of the aircraft; on the right side, the symbol of a falling object represents hollow aluminium spheres dropped from the sky to calibrate radars; the lighting bolt usually depicts electronic warfare; the sword refers to the ‘Bird of Prey,’ a declassified Boeing stealth demonstrator; while the handle of the sword depicts the shape of the aircraft operated.”

The book of patches showcases almost 60 patches with descriptions. This collection and its deciphering does not completely uncover all the programs hidden behind military secrecy, nor does it expose specific details about these programs. However, it creates a visual map giving us, the ones kept in the unknown, a rough understanding of what some of them are working on, where they are based and how much taxpayer money goes into their work.

Paglen’s work is one of the best illustrations of how to exploit a very common fact, namely that even if you are dealing with groups or activities that are protected from public scrutiny, the members of these groups, the specialists working on these secretive issues, are often proud of their activities and want to be recognised, if not publicly then among themselves at least for their skills, expertise and affiliations.

Photography, mapping, power

A few years ago we at Exposing the Invisible featured the work of Hagit Keisar, an artist, researcher and activist from East Jerusalem. At the time of her research into house demolition in this part of the city, Hagit was struggling to get access to the archives documenting those procedures. All her direct attempts to get access were refused, because her intention was seen as contesting or possibly criticising their work. She changed her approach when one of the people she interviewed, a former retired house demolition inspector, revealed to her that one of the key tools they used in their work was photography.

Photograph copied by Hagit Keisar from the municipality archive, depicting an image of houses marked for demolition.

This is what all the officers and inspectors do – they take pictures as evidence. So Hagit asked the archive to allow her to study the way they used photography in their practices and, to her surprise, she got immediate access to the documentation, which until then had been inaccessible to her. This not only led to unprecedented access but also allowed her to discover the internal visual language that house inspectors were using. That gave her a series of important clues when analysing their work and, in particular, the politics, abuse of power and inside rationale of their work.

Note:

You can read all about Hagit’s experience, her creative investigative research and artistic work on the politics of house demolitions in East Jerusalem in this article: “Photography, mapping and power,” as well as watch the documentary and transcripts of an entire series featuring citizen investigators using innovative tactics to expose hidden layers of problems in their communities: “From My Point of View.”

What’s in a code name

Yet another example, which illustrates the characteristic or vulnerability of people needing symbols to communicate about their craft, is the work of M.C. McGrath, a security researcher and activist who exploited publicly available CVs on LinkedIn. He scraped the data of almost 30,000 names of people who in their employment, skills and experience descriptions were using code names of US surveillance and intelligence programmes. Code names have no meaning to outsiders, since they are often acronyms or random names.

Note:

M.C. McGrath created the Transparency Toolkit, which provides a set of tools to collect data from various open data sources and was the basis for his ICWATCH launched in 2015. ICWATCH is a database of an estimated 27,000 LinkedIn resumes of people who appeared to work in the intelligence sector. The database included information about the intelligence community, surveillance programmes and other information that was very much private but that had been posted publicly via the professional networking platform, LinkedIn. At the time of his project launch in 2015, M.C. described his work and findings in an interview with the Exposing the Invisible team, speaking about researching with different types of open data and why he believes holding the individual to account is important.

M.C. made a very basic observation that people who work on covert (undercover) projects are still doing it for work and in this, like in any other business, people move between jobs and companies on a regular basis. A CV is the kind of document that by default requires one to make some specific references to skills, tools or activities that a potential new employer should be able to recognise.

In the case of the Transparency Toolkit, the most crucial set of variables were the code names of programmes or references to specific tools, often also hidden behind specialised codes. The lesson learned here is not to get deterred by jargon or code languages that might appear initially meaningless and boring or seem to have no apparent meaning to people outside of a specific trade.

Hashtags of evidence

A hashtag is yet another example of something between an acronym or a code name that can be used to investigate who is behind specific information distributed on social media, in particular on Twitter but also on Instagram or Facebook. We have addressed the hashstag in our previous work on Exposing the Invisible, with “Automated Sectarianism and Pro-Saudi Propaganda on Twitter” by Mark Owen Jones and “Disclosures of a #Hashtag” by Hadi Al Khatib.

In the first case, Marc Owen Jones, at the time a Research Fellow at the Institute for Arab and Islamic Studies at Exeter University and a researcher and director at Bahrain Watch, exposed an industrial scale automated propaganda campaign (aka fake Twitter accounts) in the Persian Gulf. Marc initially went after what seemed most suspicious: the repetition of identical tweets posted by different accounts. From this observation he also discovered that these same accounts used the same set of hashtags. He first accumulated all the tweets and information about the accounts intending to see if there were any patterns. That correlation between use of specific hashtags at the same time led him to an accumulation of accounts that he then analysed, looking at the time of creation of these accounts, number of followers, location and biography. In short, the initial observations, the gathered data and its analysis allowed Marc to discover an automated propaganda machine of bots used to promote certain ideas while suppressing others.

Visualisation made on Gephi by Marc Owen Jones of the #Bahrain hashtag on 22 June 2016.

In the second case, Hadi Al Khatib of the Syrian Archive wrote a piece for Exposing the Invisible back in 2016 explaining and showcasing the possibilities of using hashtags for investigations and raising awareness around possible threats so users can make informed choices on how to use Twitter. Hadi focused on two cases of how hashtags have been used during two conferences: Black Hat and DefCon in 2016. These examples enabled him to explain how, by analysing hashtags, an investigator can learn about the background of participants and networks of attendees of these gatherings. The methods and tools to analyse hastags and other related information are explained in his article “Disclosures of a #Hashtag.”

This exercise makes a very different case about the ways hashtags can be use to identify people rather than bots – including where they were or are and what networks they belong to. His investigation helped establish a unique social graph that otherwise would be extremely hard to create. Such an analysis might constitute an excellent starting point in our quest to understand issues, networks, or to help identify individuals with specific roles, positions or functions in their respective networks.

Using Carto (previously known as CartoDB), a free web-based tool to create interactive maps, Hadi and his team geolocated the tweets they had extracted.

Our Data in a PNR

Another case that explores symbols and their meanings, or their rich content in this case, shows how Passenger Name Records (PNR) can be read and decoded, and what that means for us.

The PNR is a six-digit code assigned to anyone when they book a flight. Nowadays they are often represented as bar codes on our travel documents, especially boarding cards. A PNR barcode might look very innocent and rather meaningless, and in fact it’s commonly understood as a one-off code enabling quick identification of travel details.

In fact, a PNR is an identifier that contains a rich data set of detailed information about the traveller. Each time we get a new ticket, we get a different PNR, unless we are serviced by the same travel agency. The PNR data is maintained and operated mainly by two large databases: the Computer Reservation System (CRS) or a Global Distribution System (GDS). The code includes such information as:

• the traveller’s name

• address

• ID or passport details

• other contact details

• frequent flyer program information

• all ticketing information

• full travel itinerary

• form of payment information

• check-in and seat information

• meal preferences

• other preferences and much more.

In addition, it can also include detailed information outlining your reasons for travel. That’s a lot of data points.

In Tactical Tech’s article “Booking Flights: Our Data Flies with Us” written by Paz Pena, Leil-Zahra Mortada and Rose Regina Lawrence, we explored ways of decoding bar codes, learning PNR numbers and using this knowledge to access the personal details of the people behind them. This is rather self-explanatory: such simple information hidden in a bar code could potentially be used by anyone to gain access to much more detailed and personal information.

The reason for mentioning it here is not only to show how an investigator can use such information but also to raise awareness about the fact that all of us, including the most sophisticated investigators, are travelling, participating, staying and entering places through layers of access codes often presented to us as unintelligible numbers, codes or symbols. However, when we look at them through the lens of a machine (such as a bar-code reader) they reveal more human-readable details that might lead to unexpected findings and conclusions.

From the article “Booking Flights: Our Data Flies with Us,” by Paz Pena, Leil-Zahra Mortada and Rose Regina Lawrence, Tactical Tech

Decoding the invisible

There are a few last things worth mentioning here – one is about spaces that contain no information, and the other about obfuscation.

The first case sounds fairly abstract – if there is no information then how can there be something?

That is how some scientists go about their research. Astronomy is a good example here – it is often possible to determine the existence of a planet or a star by deducing it first from the triangulation of various secondary facts and observations. This method is useful in narrowing down where to look for information in areas where the space you need to cover is fairly vast. This specific method might be useless in the toolbox of an investigator on Earth; however, the thinking behind it should not be. The lesson to learn here: looking for negative, empty spaces or gaps in data sets might lead you to better focus your resources, for one; and secondly, you might be able to learn more from the absence of something.

Let’s look at the work of Sam Raphael, Crofton Black and Ruth Blakeley and their Investigation into the CIA Torture Programme.

Snapshot by the author of an Extract from the Committee Study, with dates, CIA officers and cable data redacted, and with PSEUDONYMS for black sites and contractors; page 24; “CIA Torture Unredacted: An Investigation Into the CIA Torture Programme;” Project by The Rendition Project, The Bureau of Investigative Journalism, University of Westminster, The University of Sheffield, 2019; written by Sam Raphael, Crofton Black and Ruth Blakeley.

The challenge they faced was obvious and unresolvable – they finally got documents that could shine some light on the grievous issue they were investigating: that is, the CIA’s post-9/11 torture programme, its victims, methods and those involved. Unfortunately, almost all the important information has been redacted, as on the above image. Their meticulous work over four years produced perhaps the most detailed public account of the entire torture programme led by the CIA in the time period of the so-called “war on terror” between 2001 and 2009. It is very important to read the entire report to understand the scale, depth and extent of this programme, including detailed profiles of prisoners, locations of secret sites, complex networks of private companies involved in the programme and a detailed overview of apparent complicity by a number of key countries.

Here we focus on one of the methods the team of investigators and researchers used; that is, how were they able to un-redact these heavily redacted documents to such an impressive extent? They were dealing with enormous amounts of redactions – some pages were almost entirely blacked out, as on the above picture; some information was replaced with codes, such as “country a” or “destination site b.” The most important un-redacting process involved a technical analysis of the text itself – learning that specific fonts were used (Times New Roman, at 12p, happens to be a proportional font where each letter has different width), that the text was aligned to the left (spacing was always the same between characters), that the document used a standardised way of showing dates such as “Month, DD, YEAR” and additionally, that only individual words and figures had been redacted as opposed to entire paragraphs or sentences.

This method of deciphering what was hidden behind black marks, combined with the use of very comprehensive data sets of names, locations and times, which enabled correlating and triangulating known information with obfuscated information, yielded an unprecedented result. Bringing to light what was purposefully hidden was not only important and significant, but also created an unexpected consequence of teaching those engaging with redaction how not to do it the next time around.

Note:

Read the report “CIA Torture Unredacted - An Investigation Into the CIA Torture Programme” – project by The Rendition Project, The Bureau of Investigative Journalism, University of Westminster, The University of Sheffield, 2019.

The second case here is about obfuscation – hiding under a different image and identity.

So far we have looked at examples of where large quantities of images, symbols and signs in aggregation create a larger story. However, sometimes it might be enough to gain basic and important clues from a single image rather than from a series of them.

The most interesting example comes from an investigation by a renowned security expert. The expert was trying to figure out the person or people behind a massive botnet called Mirai, which was causing havoc on the internet in 2016, at a scale never before seen. Security expert Brian Krebs spent a long time tracking any and every clue that could help him reveal the real authors of this botnet.

This is an image created by one of the suspected authors of Mirai Botnet that Krebs came across - Dreadiscool’s account on Spigot Minecraft forum since 2013 includes some interesting characters photoshopped into this image. https://krebsonsecurity.com/2017/01/who-is-anna-senpai-the-mirai-worm-author/

The investigator was able to list a number of various nicknames linked to real people as potential suspects. In one case, while trying to make a link with the nickname Dreadiscool, whose identity he was able to disclose, he came across a meme created by the user hiding behind this pseudonym. The meme appropriated a well known image from the movie Pulp Fiction, where the heads of the actors John Travolta and Samuel L. Jackson are replaced with the heads of two hackers known as Vyp0r and Tucker Preston. The anime figure also visible in the image is of Yamada – a character from the manga movie series B Gata H Hei.

While exploring this path, Krebs found that the user Dreadiscool had listed other films that they had watched on the same forum. Among nine titles, he discovered the title “Mirai Nikki” – another manga movie series. This, however, was a weak link because it might have been purely coincidental – anime is popular among certain types of young hackers. Nevertheless, this clue helped him explore this path of the investigation further, finally leading to the discovery that the user Dreadiscool had also been using the pseudonym Anna Senpai (a user who leaked the source code of the Mirai Botnet – the much feared malware back in 2016). Through a series of further correlations, he uncovered the real name behind these nicknames was Paras Jha. After gathering enough clues, Krebs exposed the work of Paras in a series of articles. The case was picked up by law enforcement and led to his arrest and conviction in 2018.

This investigation yet again brings together the self-assurance of the main actor with the very scrupulous work of an investigator triangulating and correlating various clues and following them to their ends, eliminating those that lead nowhere and following those that open unexpected avenues for further investigation.

There is also something ephemeral about these kind of visual clues. Once correctly attributed, they might be misleading when used in different contexts, because their creators or users might try to use them differently to mislead or “play” those who try to decode them. In addition, confusion may arise when they get taken up by different actors or they just change their meaning over time. An illustrative example here is the history of the use of the infamous Pepe the frog.

Extract from a comic strip (Boy’s Club #1) featured an anthropomorphic frog “Pepe.” This image started the Pepe the frog internet meme. From Wikipedia https://en.wikipedia.org/wiki/Pepe_the_Frog#/media/File:Feels_good_man.jpg

Although mainly known in the US and Europe as the most iconic meme/image recently (and wrongfully) associated with the alt-right and white nationalist movement primarily in the US, Pepe the frog then became an image of so-called kek – a parody of religion that led to Kekistan, a fictional country representing so called “shitposters” – trolls contesting any notion of political correctness. The most interesting shift happened when the same image of Pepe the frog started to appear during the protests in Hong Kong, where protesters used it as a symbol of liberty and resistance against the contested extradition bill and police brutality. It turned out that those who used it were totally unaware of the right-wing connotation of the image and of its origin and history – initially completely unrelated to the far-right or any other political statement – which in fact started in 2005 with the work of cartoonist Matt Furie in an obscure zine called Boy’s Club. Some people started wrongly associating the ideas behind the Hong Kong protests with the alt-right – based on what the image has been mostly know for in its previous recent life.

There is more

This selection of cases far from fully explores and describes all the possible ways of reading symbols, signs, acronyms, codes and other alphanumerical information as well as more ephemeral images in the process of an investigation.

But there are some basic principles that you can take away from the investigators and investigations we looked at. It is hard to present them as a step-by-step formula but they have a few things in common worth noting:

• They are based on expanding the periphery of what one is looking at, or exploring any and every visual clue possible, even those that might seem irrelevant or unrelated to the core investigation.

• They often rely on unique expertise related to this sort of visual material. Many of the investigators engage and talk to those who make these images or use them, and they often find people who collect and analyse them for purposes different then those of the investigators (e.g. plane-spotters are most often interested in planes – not in corruption, covert flights or other uses of airplanes).

• Investigators take such clues with lots of reservations and scepticism as they know that interpretation, understanding, correlation and triangulation of information might be at any point corrupted for unknown reasons – and they use this method mostly as a supporting framework, together with other, more elaborate means of evidence collection and verification.

---

Published March 2020

Resources

Articles and Guides

• Automated Sectarianism and Pro-Saudi Propaganda on Twitter by Mark Owen Jones on Exposing the Invisible (archived copy from Wayback Machine available here).

• Booking Flights: Our Data Flies with Us by Paz Pena, Leil-Zahra Mortada and Rose Regina Lawrence on Tactical Tech (archived copy from Wayback Machine available here).

• CIA Torture Unredacted: An Investigation Into the CIA Torture Programme. Project by The Rendition Project, The Bureau of Investigative Journalism, University of Westminster, The University of Sheffield, 2019; written by Sam Raphael, Crofton Black and Ruth Blakeley (archived copy from Wayback Machine available here).

• Disclosures of a #Hashtag by Hadi Al Khatib on Exposing the Invisible (archived copy from Wayback Machine available here).

• From My Point of View - Hagit: photography, mapping and power, by Exposing the Invisible. (archived copy from Wayback Machine available here).

• Ingrid Burrington, Tactical Cartography -video sketch and text transcript, by Exposing the Invisible (archived copy from Wayback Machine available here).

• I Could Tell You But Then You Would Have to Be Destroyed By Me: Emblems from the Pentagon’s Black World, by Trevor Paglen, 2008. A video recording and text transcript of a Trevor Paglen talk about his book is available from c-span.org here.

• Marking Beirut: A City Revealed through its Graffiti, by Tala F. Saleh, 2009. Book available for online purchase. See related 2010 interview with the author (archived copy from Wayback Machine available here).

• Networks Of New York: An Illustrated Field Guide to Urban Internet Infrastructure, by Ingrid Burrington, 2016. Book available for online purchase. A book review is available here (archived copy from Wayback Machine available here).

Tools and Databases

• Center for Land Use Interpretation. A research and education organisation interested in understanding the nature and extent of human interaction with the surface of the earth. (archived snapshot of the website on Wayback Machine available here).

• ICWatch, by M.C. McGrath / Transparency Toolkit. A database of an estimated 27,000 LinkedIn resumes of people who appeared to work in the intelligence sector. (archived snapshot of the website on Wayback Machine available here).

• Syrian Archive. A Syrian-led and initiated collective of human rights activists dedicated to curating visual documentation relating to human rights violations and other crimes committed by all sides during the conflict in Syria with the goal of creating an evidence-based tool for reporting, advocacy and accountability purposes (archived snapshot of the website on Wayback Machine available here).

• Transparency Toolkit by M.C. McGrath. A set of tools to collect data from various open data sources. (archived snapshot of the website on Wayback Machine available here)

Glossary

term-ascii

ASCII - stands for the American Standard Code for Information Interchange. It is a code for representing 128 English characters as numbers, with each letter assigned a number from 0 to 127. For example, the ASCII code for uppercase M is 77. Computers use ASCII to represent text for easier transfer of information to other computers. (source Webopedia).

term-bot

Bot – also called web robot or internet bot, is a software application that runs automated tasks over the internet. For example, a Twitter bot that posts automated messages and news feeds.

term-graffiti

Graffiti - writings or word-based drawings made on walls or other surfaces, usually in public spaces and without permission, as a form of artistic expression, political expression, advocacy etc.

term-hashtag

Hashtag - symbol introduced by the number sign, or hash symbol, #, is a type of metadata tag used on social networks such as Twitter and other microblogging services. It lets users apply dynamic, user-generated tagging that helps other users easily find messages with a specific theme or content. (source Wikipedia)

term-iconography

Iconography - the study of the meaning of symbols and signs.

term-meme

Meme - an idea, image, text, etc., usually with humorous features, that is created, copied and spread rapidly by Internet users, often with slight mutations and variations.

term-metadata

Metadata - information that describes properties of a file, be it image, document, sound recording, map etc. For example the contents of an image are the visible elements in it, while the date the image was taken, the location and device it was taken on, are called metadata.

term-sign

Sign - a sign extends language and communicates with people who have incorporated it to their vocabulary. A sign can also be made by gesture or lack of gesture it often has some direct communication behind it (such as an instruction).

term-symbol

Symbol - as opposed to a sign, a symbol is more ephemeral as it is not an instruction but rather a representation of something, e.g. process, function and other things like objects.