A Hadoop developer is responsible for development, programming, coding of information Technology solutions and operations of systems that store and manage large amounts of data. The Hadoop developers is responsible for documenting detailed system specifications, unit testing, oversee work results, code reviews providing training and serve as a technical SME for other Developers. We are looking for a Big Data Engineer that will work on the collecting, storing, processing, and analyzing of huge sets of data. Provide production support, maintaining and implementing large complex projects. Most Hadoop developers have a computer software background and have a degree in information systems, software engineering, computer science, or mathematics.
Responsibilities
• Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities
• Implementing ETL process using APACHE NIFI
• Monitoring performance and advising any necessary infrastructure changes
• Experience with Spark and SparkR.
• Provide mentor and training to resources within Spark and SparkR ETL framework.
• Troubleshoot and debug any Hadoop ecosystem run time issues
• Understanding the requirements of input to output transformations
• Cleaning data as per business requirements using streaming API’s or user defined functions.
• Build distributed, reliable and scalable data pipelines to ingest and process data in real-time, defining Hadoop Job Flows
• Assess the quality of datasets for a hadoop data lake


Skills and Qualifications:
• Proficient understanding of distributed computing principles.
• Proficiency with ETL infrastructure and integration of data from multiple data sources, Implementing ETL process using APACHE NIFI
• Proficient experience with Spark and SparkR
• Management of Hadoop cluster, with all included services such as Hive, HBase, mapReduce, Spark, SparkR, Python, Scala and Sqoop
• Ability to solve any ongoing issues with operating the cluster and identifying performance bottlenecks.
• Proficiency with Hadoop v2, MapReduce, HDFS
• Experience with building stream-processing systems, using solutions such as Storm or Spark-Streaming.
• Good knowledge of Big Data querying tools, such as Pig, Hive, and Impala
• Experience with integration of data from multiple data sources
• Experience with NoSQL databases, such as HBase, Cassandra, MongoDB
• Knowledge of various ETL techniques and frameworks, such as Flume
• Experience with various messaging systems, such as Kafka, Scoop
• Experience with Big Data ML toolkits, such as Mahout, SparkML, or H2O
• Good understanding of Lambda Architecture, along with its advantages and drawbacks
• Experience with Cloudera/MapR/Hortonworks
• Health Care Experience is s plus