Fundada en 2012 por consultores experimentados The Whiteam nace como consultora tecnológica de calidad con una misión clara; ayudar a las compañías de todo el mundo a optimizar su rentabilidad empresarial a través de un uso eficiente de las tecnologías de la información.

Job description

The WhITeam is looking for a motivated Data Engineers to support the realisation of a Customs Data Hub prototype. This technical annex outlines the tasks and responsibilities of a Data Engineer service which will play a crucial role in the design and implementation of data-related aspects within the prototype project. The Data Engineer will focus on managing data pipelines, metadata, data processing (both batch and real-time), and validating the system's ability to handle the intended data models. The service provider for this role should possess a strong background in data engineering, with expertise in managing data pipelines, validating data models, and implementing data processing solutions. The Data Engineer should have hands-on experience with data integration, ETL (Extract, Transform, Load) processes, data quality assurance, and data governance.

Job requirements

I. Data Pipeline Design and Management:

a) Collaborate with stakeholders to understand data requirements, sources, and potential transformation processes.

b) Design and implement scalable and efficient data pipelines to extract, transform, and load data from various sources into the prototype system.

c) Ensure data pipeline robustness, reliability, and scalability by addressing potential bottlenecks and implementing appropriate monitoring and error-handling mechanisms.

d) Collaborate with IT engineers to integrate data pipelines with the overall system architecture and components.

II. Metadata Management:

e) Define and implement metadata management strategies, including data dictionaries, data lineage, and data cataloguing, to enable effective data governance and discoverability.

f) Establish and document data naming conventions, standards, and documentation to maintain consistency and facilitate data understanding for stakeholders.

g) Collaborate with IT engineers to incorporate metadata management processes within the prototype system, enabling metadata-driven functionalities.

III. Personal Data Management:

h) Define and implement personal data management strategies, to enable effective governance, discoverability and compliance with GDPR regulations.

i) Collaborate with IT engineers to incorporate compliant personal data management requirements in the implementation.

IV. Data Model Validation:

j) Collaborate with stakeholders to understand the intended (Customs) data models to be applied after the prototype.

k) Validate the system's ability to handle and process the intended data models, ensuring compatibility, performance, and scalability.

l) Perform data model validation exercises, using simulated data or representative samples, to verify the system's capability to support the intended data models.

m) Provide recommendations and adjustments to the system's design to align with the intended data models and ensure their successful implementation.

V. Data Processing (Batch and Real-time):

n) Design and implement batch data processing workflows, ensuring efficient data extraction, transformation, and loading processes.

o) Develop real-time data processing solutions, including streaming data ingestion, data enrichment, and event-driven architectures, where applicable.

p) Optimize data processing pipelines for performance, scalability, and reliability.

q) Collaborate with IT engineers to integrate batch and real-time data processing components into the overall system architecture.

VI. Data Quality Assurance:

r) Establish data quality standards, processes, and metrics to ensure data accuracy, consistency, and completeness within the target system.

s) Implement data validation and cleansing techniques to identify and address data quality issues.

t) Collaborate with IT engineers to incorporate data quality assurance processes into the overall system architecture and workflows.

VII. Coordination and Advisory Role:

u) Coordinate with IT engineers to ensure seamless integration of data-related components within the prototype system.

v) Provide guidance and advisory support to IT engineers on data engineering best practices, data processing techniques, and data management strategies.

w) Collaborate with stakeholders to address data-related challenges, provide technical expertise, and offer recommendations for optimizing data workflows and processes.

The company

Being part of THEWHITEAM means collaborate with a company constituted by professionals with extensive experience in technology consulting.

We certainly believe that companies and clients are the key to moving forward in the sector, but is the people who build the opportunities. We consider it essential that our organisation is based on our greatest asset and value-added brand, which is our team.