We live in an age where the majority of developed nations are dominated by the Internet. It has been infused into most facets of the human experience, such as communication and socialization, as well as almost every process and industry. With proper consideration, collection and analysis of Internet data for security and safety reasons can yield a wealth of intelligence and positive outcomes. However, it can also lead to negative outcomes and public perception without proper consideration.

The early days of social, deep, and dark web data analysis may have been a bit of a Wild West, but that hasn’t been the case for quite some time now. Below we’ll discuss the Good, the Bad, and the Ugly when it comes to using Web Intelligence in your security strategy.

Humans are natural storytellers. Since civilizations were based on the transfer of knowledge throughout time, it’s only logical that the Information Age has resulted in communication and socialization on social media, forums, and entire networks such as the Dark Web. And boy do we communicate a lot. Taking only Twitter into account – there is an average of more than 500 million Tweets sent each day. Now consider all the other platforms that exist across the Internet and you’ll start to understand just how much digital communication is occurring at any specific moment in time.

In the context of security – there is something to be said about the age-old quote, “Straight from the horse’s mouth”. Primary-source intelligence will always be the most timely, relevant, accurate, specific, and actionable. While social media platforms and forums can all seem innocuous enough, these very same platforms and others like them are means for criminal organizations, terrorist groups, nation-states, and any other threat actors that loom on the horizon to communicate. This gives security teams a treasure trove of information to work with. The conceptualization, planning, discussion, and execution of attacks are just waiting to be discovered and cataloged.

Web Intelligence isn’t a magic easy button. It’ll always require a human’s brain and experience in order to analyze and interpret the information they’ve found. Without a proper plan and clearly defined algorithm in place to gather and prioritize data, intelligence collection will drain a significant amount of time and resources. 

Web Intelligence can quickly become a bottomless pit of invested effort. This is largely in part of a few things at play:

The most obvious barrier to having a successful Web Intelligence program is the sheer amount of data to vet and validate in order to take action on anything of substantial value. There are just too many posts, articles, discussions, and pages that emerge, change, and disappear. Keeping track of disparate sources takes away time from investigating information that is actually useful.

Media Sonar Web Intelligence & Investigation platform pulls in a vast amount of data across disparate sources. Access intelligence from the surface web, social media platforms, deep and dark web networks, forums, etc. Craft specific search queries and rules in order to find the unique pieces of intelligence you need to act on.

Another barrier to Web Intelligence is having the cursory knowledge to understand where to focus your efforts and where to look. Where do your threats talk the most? What are they sharing and on what platforms? What are they saying and how do they talk?  Without at least a little bit of knowledge of these things, all your efforts can be dumped into a data source that will only end up fruitless.

Media Sonar adds and collects from data sources that are emergent in the threat landscape – like Telegram and Raidforums – and provides easy filters for sites, so analysts can focus on finding relevant content rather than all their time finding where the content may be.

Not all the threats out there on the web are obvious; there are many that are extremely professional and dedicated to their trade and the digital universe allows for a lot of tradecraft to be employed.

While there are no surefire solutions to countering ALL digital tradecraft, analysts need a tool that allows for some degree of investigation and the ability to learn the most about their adversaries as is possible in the face of obfuscation tactics. Media Sonar Pathfinder, in conjunction with our OSINT Toolkit, allows users to explore found content, extract data entities such as usernames or email addresses, and automatically resolves common data points between posts/content.

Expertise can play a big part in mitigating these barriers, but so can tools designed for Web Intelligence gathering and analysis. With the right features, security teams are not only able to find indicators of potential threats but also investigate them and make intelligence about them actionable.

Even when lawful, some facets of Web Intelligence are highly governed by ethical, moral, and societal frameworks that can turn on you if not properly framed, planned, and executed. Chris Bousquet, in an article for the Ash Center of Harvard, highlights an incredible point, “There’s a term in the software engineering world – ‘garbage in, garbage out’”. Without a very well-defined set of algorithms or mission parameters and goals for the end-data, Web Intelligence collection and analysis endeavors can easily become biased, and therefore – skewed. There’s a major difference between wanting to monitor a social movement/hashtag and wanting to monitor for violent outbursts or crime that branches from a peaceful protest. While monitoring open-source data in this way is technically legal – the revelation of a poorly planned monitoring program can create a public outcry.

Once a Web Intelligence program has been clearly defined and scoped – again, with an effort to avoid any biases – that’s when properly defined and tested algorithms and queries come in. While it can be attractive to start collecting intelligence from low-hanging fruit such as a protest hashtag – it starts to cross the line of things like Free Speech. Not everyone at a peaceful protest is or will be complicit in any criminal activity that spawns from the event. And while the passive scanning and collection of that data may seem innocuous to you, it can present more of a problem if the general public were to discover it without being aware of your end goal (ex.: stop violent actions that spawn from a peaceful protest).

Analysts need tools that allow them to fine-tune their queries and algorithms. While most modern tools will try to offer as much automation and efficiency as possible, it’s a careful balance in terms of the workload required on the practitioner’s end. Quick searches and topic selection are all nice features in terms of efficiency but they can easily be too broad or unfocused for your end goal. You can begin to collect data that isn’t needed and therefore, could run the risk of public scrutiny.

Media Sonar not only includes quick-filtering features to make setting up a monitoring and analysis program quick and easy – but it also allows for very detailed query logic to be created. Contain your mission parameters and data collection as tightly as possible to avoid some of these societal and ethical pitfalls of Web Intelligence.

Organizations are moving away from the traditional “defend-and-respond” approach towards a more mature and proactive security strategy. Web Intelligence plays a major role and it’s critical that analysts carefully consider Web Intelligence tools that can help them better navigate the good, bad, and ugly of intelligence. Understanding the context, ideas and identifying information about your threats, in the face of tradecraft, all feeds into turning data into intelligence.

Web Intelligence tools like Media Sonar helps security teams implement and automate collection, analysis and investigation. Stay consistent with your mission plan, reduce the amount of time and expertise required, and work to avoid ethical and public-perception issues that can arise.

Whether you’re just entering your endeavors to incorporate Web Intelligence into your security program, or you’re an extremely mature security organization that already ingests different forms of Web Intelligence – Media Sonar can assist you.