Using the Right Tool to Gather Publicly Available Information

Open-Source Intelligence (OSINT) is usually defined as intelligence produced by collecting, evaluating, and analyzing publicly available information to answer a specific intelligence question. 

Here are some common ways in which OSINT is used today:

• Security and Intelligence: 
• Business and Market Research
• Investigative Journalism
• Academic Research
• Legal Proceedings

It is an exceptional tool for gathering information on a wide range of topics such as security threats, market research, and competitive intelligence, and can also be used by a variety of organizations and individuals to inform decision-making and strategy.  

Challenge

OSINT is transparent and can be easily verified. This means that organizations can be confident in the accuracy and reliability of the information they gather. It also offers many advantages over other forms of intelligence collection, making it a valuable tool for a wide range of organizations and individuals.

The client approached us with the challenge: develop an OSINT (open source intelligence) tool, which aims to collect, consolidate, and make a detailed accurate analysis of personal data and user-friendly presentations.

Solution

To build and develop an OSINT tool, we proposed and did the following:

• Data mining from third-party providers (Social media, trade registers, search engines).
• Data collection, processing, and consolidation.
• Data indexed and searchable.
• Machine learning analysis.
• Event-driven architecture on AWS.

What technologies did we use to make this a reality?

Our team consisting of a Tech Lead, Project Manager, Data Engineers,  ML Developers,  FE Engineers, and a Quality Assurance Engineer used Machine learning, Python (Starlette Framework), TypeScript (ReactJS), AuroraDB, ElasticSearch, DynamoDB, Docker and worked with AWS Services for Containerization, Messaging, Storage.

Results

The results of our ready-to-use solution are looking encouraging. We successfully managed to deliver the following:

• An OSINT tool that is production ready.
• The client has already onboarded the first customer.
• The solution also gives our client the advantage to fast delivery reports.
• The machine learning prediction success rate amounts to +91%.