DATA ENGINEER / KAFKA / SPARK / HADOOP / SQL / SCALA / DATA INGESTION
My client is looking for a Data Engineer to help provision valuable business data to the Data Platform. You will work closely with Data Platform Engineers, Data Scientists, and InfraDev Engineers across multiple platforms and technologies to create a highly scalable, secure data platform.
As a Data Engineer, you will:
- Build data ingestion / integration pipelines for large scale real-time (streaming) and batch applications
- Build capabilities to efficiently integrate new data sources into common platform
- Develop necessary transformation logic / event logic for data processing across pipeline
- Design and develop all necessary transformation logic, business logic, and preparation to support needs of business
- Care about your craft, be aware of new technologies and be keen to use them when appropriate (e.g. Continuous Integration, Continuous Deployment)
- Feed your insights to the product team to continuously improve the platform
- Be an active contributing part of an agile development team
- Be diligent and positive about tracking (JIRA), system monitoring, security, auditing and other ways of ensuring the quality of the platform and the processes around it
You should have experience with:
- Manipulating data with a variety of tools and programming languages, e.g. SQL, Java, Python
- Building streaming & batch applications with technologies including: JMS, Kafka, Kinesis, Spark Streaming
- Working with data formats including JSON, XML, Flatfiles, Sequence, Avro, Parquet, etc.
- Cloud computing and related challenges around security, bandwidth, etc.
- Understanding statistics and data science to work closely with business analyst
- Kafka, Spark, Hadoop, Hive
- Scala, Java, R, Python
- PostgreSQL, ElasticSearch
- Git, Jira