How we partnered with an existing internal team to boost their productivity
We collaborated with a leading software development firm to support them on a project that required additional expertise and technical know-how. The project involved centralizing data from different sources into an easily accessible location for a cybersecurity company based in the USA. The company is known for leveraging large datasets to identify and counteract malicious software, and needed a platform to improve their internal data management.
The client wanted to build a data platform capable of centralizing and managing various data sources, making them accessible for further processing and analysis. This required handling a significant amount of data with good scalability and monitoring capabilities to ensure efficient data processing.
Our data engineers were brought in as external resources to enhance our client's existing team and contribute to the project's success. They were quickly integrated into the team and began working on coordinated tasks. We made sure to maintain active communication among team members, including daily meetings, constant feedback, and messaging.
Our data engineers were brought in as external resources to enhance our client's existing team and contribute to the project's success. They were quickly integrated into the team and began working on coordinated tasks. We made sure to maintain active communication among team members, including daily meetings, constant feedback, and messaging.
We leveraged different technologies to build an efficient ETL. Our team developed code, created functions, and implemented different pipeline components to receive data from various sources, process it, and then upload it to a centralized database. The technologies we used included Snowflake for the database, AWS for cloud infrastructure, Terraform for infrastructure management, and Prefect for workflow automation.
Cyber-security practices requires immediate processing for early identification of threats and fast responses to potential security incidents. We needed to process data in real-time and avoid delays caused by waiting for larger files to become available. To achieve this, we opted to handle smaller data blocks rather than a single large file containing millions of records per day. We leveraged Apache Kafka and Confluent. With the integration of these cutting-edge technologies, we were able to achieve a processing time of approximately 1 minute, ensuring our client had access to the most up-to-date information available.
After almost two years of development, the new data platform centralizes diverse data sources and empowers our client's teams with valuable insights. Staff members have successfully migrated to this new platform, enabling them to access and analyze data more efficiently.
"They seamlessly integrated into our team, functioning just like one of us when we engage with a client. The feedback was abundant and positive, and they consistently fulfilled their commitments. We had a great experience working with eidos."
