Data Engineer

Functie Data Engineer
Aanvraagnummer 93099
Expertise Java, Big Data Technologies
Regio Hilversum
Startdatum ASAP
Duur 3 maanden +
Werkzaamheden

You will be working on the collecting, storing, processing, and analyzing of huge sets of data. The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them. You will also be responsible for integrating them with the architecture used across the company. Aside from this, you will be challenged everyday day!

You will be responsible for :

The Design and implement distributed data processing pipelines using Spark, Hive, Sqoop, Python, and other tools and languages prevalent in the Hadoop ecosystem.

You’re able to design and implement end to end solution You can build and utilities, user defined functions, and frameworks to better enable data flow patterns.

  • Research, evaluate and utilize new technologies/tools/frameworks centered around Hadoop and other elements in the Big Data space.
  • You iDefine and build data acquisitions and consumption strategies Build and incorporate automated unit tests, participate in integration testing efforts.
  • Work with teams to resolving operational & performance issues
  • Work with architecture/engineering leads and other teams to ensure quality solutions are implements, and engineering best practices are defined and adhered to.
  • Proven ability to work cross functional teams to deliver appropriate resolution

You’re Qualification:

  • MS/BS degree in a computer science field or related discipline
  • 6+ years’ experience in large-scale software development
  • 1+ year experience in Hadoop or big data technologies.
  • Strong Java programming, Python, shell scripting, and SQL
  • Strong development skills around Hadoop, Spark, Hive, and Pig
  • Good understanding of file formats including JSON, Parquet, Avro, and others
  • Experience with performance/scalability tuning, algorithms and computational complexity
  • Ability to understand relational database schemas
  • Experience with AWS components and services, particularly, EMR, S3, and Lambda
  • Automated testing, Continuous Integration / Continuous Delivery

solliciteer direct