Sr. Big Data Engineer at Levelset in Remote
A Data Engineer at Levelset is responsible for sourcing, assembling, transforming, and loading large data sets about US contractors, sub-contractors, suppliers, projects, payment documents, etc across various internal and external sources (i.e. Saas applications and data systems, US County Clerk data). You will be building production grade data pipelines across our stack which currently consists of technologies like MySQL, MongoDB, RedShift, Neo4J, Redis, Kafka, RabbitMQ, Python, PHP, Airflow and other modern tools in a highly scalable microservices architecture. Responsibilities Provide technical guidance and contribute to the definition, architecture, development, integrations, QA, documentation, and support of our construction payment data platform Perform analysis of large data sets using tools and techniques from big data ecosystems like Hadoop and MapReduce Design, implement, and develop distributed data infrastructure and pipelines for high volume / high velocity multi-tiered data storage with high availability and elegant fault tolerance Design and implement reliable data pipeline features used for multi-sourced data collection, cleansing, transformations, validation rules, clustering, and reliable data distribution streams (messaging queues or API services) Create reliable data development processes / systems for consistent auditing, logging, monitoring, and debugging of data ingestion pipelines Help create reliable and automated QA for big data pipelines to maintain high data quality / integrity What You Bring BS degree in computer science, computer engineering or equivalent 3-5 years SQL experience on a variety of RDBMS, and enjoy optimizing queries; have also developed scalable solutions using NoSQL databases working with data sets comprised of millions or billions of records is comfortable 3-5 years experience in Java, Python, Spark, Scala, Hadoop and Cloud technologies (AWS is preferred) Ideally you'll have exposure across multiple big data technologies such as Hadoop, MapReduce, HDFS, Cassandra, HBase, Hive, Flume, Sqoop, Kafka, Scala Familiarity with AWS scripting and automation Experienced in developing microservices is a plus Must be able to quickly understand technical and business requirements and can translate them into technical implementations
Salary Range:
$80K -- $100K
Minimum Qualification
Database Administration, Software DevelopmentEstimated Salary: $20 to $28 per hour based on qualifications.
Salary Range:
$80K -- $100K
Minimum Qualification
Database Administration, Software DevelopmentEstimated Salary: $20 to $28 per hour based on qualifications.
|
|
 | |
