EMMANUEL OFFISONG
Cloud Data Engineer | DevOps
Lagos
Summary
As a highly skilled Cloud Data Engineer and DevOps Engineer, I specialize in designing, implementing, and managing robust data solutions while optimizing and automating deployment pipelines for seamless software delivery. My expertise encompasses cloud platforms such as AWS and Azure, as well as data warehousing, infrastructure management, site reliability engineering, ETL processes, and containerization. I am dedicated to delivering scalable and secure solutions, providing value for any organization I find myself in.
Overview
5
5
years of professional experience
1
1
Certification
Work History
Cloud Data Engineer | DevOps
Data2bots
03.2023 - Current
- Consulted for TMobile Netherlands, , and Emesa to help them build their data infrastructure from the Raw Layer, Qualified Layer, and Information Factory layer
- Developed Airflow Dags for orchestrating data pipelines that delivered about 50M+ rows to
- Redshift clusters
- Created a DevOps lifecycle that builds and deploys infra changes automatically on a
- Kubernetes cluster
- Developed a custom solution to orchestrate dbt models on AWS using AWS CICD
- TOOLS: AWS Lambda, AWS Codebuild, AWS Codecommit, Custom Python scripts
- Eventbridge
- Create a cloud infrastructure in GCP where web developers can easily deploy their web applications
- TOOLS: Cloud run, Cloud build, Artifact Registry, Terraform, GCS, Google Pub-Sub
- Built an end-to-end monitoring and alerting system for different resources like Glue, DMS, and Lambda using AWS
- TOOLS: Lambda, Terraform, Glue, DMS, SQS, SNS, Cloudwatch events, Cloud subscription filters, Eventbus
- Designed and built a dynamic AWS cloud networking infrastructure that allows specific access to AWS resources in a VPC and private subnet using a VPN client, secures AWS resources such as Glue, RDS instances, and EC2, amongst others in the subnets, and improved security by 60% by allowing only streamlined access using AWS Client VPN
- Created and scheduled data migration tasks and optimized the query time of Redshift by
Data Engineer
Estate Intel
Lagos
05.2022 - 02.2023
- Designed and developed scalable data pipelines for Nigeria's largest real estate data bank
- Built the machine learning valuation model and a twitter bot for vesper.africa
- Created an automated scraping pipeline using Apache Airflow and Python to get new information from multiple sources
- I designed the cloud data infrastructure using Terraform and Cloud Formation Templates
- Optimized Apache Spark and Amazon Redshift clusters that reduced costs by about 30%
- Created a central source of truth of analytics for data from multiple sources
- Worked with tools like Powerbi and Qgis to create dashboards and heatmaps that make end users understand the insights better.
Data Engineer | AI Researcher
NitHub
01.2022 - 05.2022
- I worked on a language translation model that translates English into the Yoruba language
- Create an AI ecosystem where people from diverse cultural backgrounds can come together and exchange knowledge and resources for the development of AI in Nigeria
- Built batch data ingestion pipelines using Spark, Pandas, Airflow, and AWS
- Built streaming solutions to effectively process IOT data using AWS Kinesis.
Data Engineer | DevOps Engineer
CapturedByDuk
Lagos
01.2019 - 01.2022
- Built a streaming pipeline using Spark and Confluent Kafka
- Built ETL pipelines using AWS Lambda, AWS Glue, and Redshift
- Used Apache Nifi for data ingestion from different sources
- Built monitoring and alerting systems that create incidents and Jira tickets per workflow failures
- Built whole applications infrastructure by using Ansible, Terraform, and AWS
- Monitored multiple Kubernetes clusters using tools like Grafana and Prometheus
- Used Powerbi to create interactive dashboards.
Education
B.Sc - Systems Engineering
University of Lagos
Skills
Cloud: AWS, Azure
DevOps Tools: Bash Scripting, Terraform, GitHub Actions, Jenkins, Azure DevOps, AWS CICD(Codecommit, Codebuild, Code pipeline), Ansible, Prometheus, Grafana, Makefile, Docker, Kubernetes, Helm, AWS CloudFormation
Data Tools: Apache Spark, AWS Glue, Azure DataFactory, dbt, Apache Kafka, Google PubSub, Apache Airflow, Alembic, AWS Lambda, Azure Databricks, Azure Stream Analytics, AWS Kinesis
Databases: Postgres, MySql, SQL ServerData warehouses: AWS Redshift, Snowflake, Azure Synapse Analytics, BigqueryVisualization: Looker, PowerBi, QGisSoft Skills: Agile Methodology, Communication, Technical Writing, Presentation, Collaboration
Accomplishments
- Over the years, I have written technical articles for companies like Airbyte, Redpanda, Paperspace, Estuary Flow, Decodable, amongst others. You can check out my articles using this link https://github.com/Techtacles/Technical-Articles
Additional Information
- Built and managed the end-to-end infrastructure of a book recommendation machine learning project using Terraform, Github Actions, Docker, AWS Lambda, AWS ECR, AWS ECS.
- Deployed a containerized deep learning framework for a music playlist generator machine learning model using Terraform, Github Actions, AWS ECR, AWS EKS. https://github.com/Techtacles/Playlist_Generator/tree/aws-cloud
- Build and deployed an end to end data pipeline using Terraform, Github Actions, Airflow, Docker, Kubernetes and AWS resources. https://github.com/Techtacles/kubernetes-de-project
- Built a data pipeline consisting of a batch layer and a streaming layer that processes information and sends notification using Apache Kafka, Apache Spark, AWS S3, Lamda, Glue, SNS. https://github.com/Techtacles/batch-streaming-data-pipeline
- Built an end-to-end deep learning framework to detect early signs of breast cancer using MRI images with about 80% accuracy using Jenkins, Vector database, Python and AWS.
Certification
AWS Solutions Architect Associate
Timeline
Cloud Data Engineer | DevOps
Data2bots
03.2023 - Current
Data Engineer
Estate Intel
05.2022 - 02.2023
Data Engineer | AI Researcher
NitHub
01.2022 - 05.2022
Data Engineer | DevOps Engineer
CapturedByDuk
01.2019 - 01.2022
B.Sc - Systems Engineering
University of Lagos