Data Engineer

Job Summary:   Software engineering skills combined with the demands of a high volume, highly-visible analytics platform make this an exciting challenge for the right candidate. Are you passionate about digital media, entertainment, and software services? Do you like big challenges and working within a highly motivated team environment? As a software engineer in the Data Experience (DX) team, you will research, develop, support, and deploy solutions in real-time distributing computing architectures. The DX big data team is a fast-moving team of world-class experts who are innovating in providing user-driven, self-service tools for making sense and making decisions with high volumes of data. We are a team that thrives on big challenges, results, quality, and agility.   Who does the data engineer work with? Big Data software engineering is a diverse collection of professionals who work with a variety of teams ranging from other software engineering teams whose software integrates with analytics services, service delivery engineers who provide support for our product, testers, operational stakeholders with all manner of information needs, and executives who rely on big data for data backed decisioning.   What are some interesting problems you'll be working on? Develop systems capable of processing millions of events per second and multi-billions of events per day, providing both a real-time and historical view into the operation of our wide-array of systems. Design collection and enrichment system components for quality, timeliness, scale and reliability. Work on high-performance real-time data stores and a massive historical data store using best-of-breed and industry-leading technology.   Where can you make an impact? Comcast DX is building the core components needed to drive the next generation of data platforms and data processing capability. Running this infrastructure, identifying trouble spots, and optimizing the overall user experience is a challenge that can only be met with a robust big data architecture capable of providing insights that would otherwise be drowned in an ocean of data. Success in this role is best enabled by a broad mix of skills and interests ranging from traditional distributed systems software engineering prowess to the multidisciplinary field of data science.   Responsibilities: Develop solutions to big data problems utilizing common tools found in the ecosystem. Develop solutions to real-time and offline event collecting from various systems. Develop, maintain, and perform analysis within a real-time architecture supporting large amounts of data from various sources. Analyze massive amounts of data and help drive prototype ideas for new tools and products. Design, build and support APIs and services that are exposed to other internal teams Employ rigorous continuous delivery practices managed under an agile software development approach Ensure a quality transition to production and solid production operation of the software Skills & Requirements: 5+ years programming experience Bachelors or Masters in Computer Science, Statistics or related discipline Experience in software development of large-scale distributed systems - including proven track record of delivering backend systems that participate in a complex ecosystem. Experience working on big data platforms in the cloud or on traditional Hadoop platforms AWS Core Kinesis IAM S3/Glacier Glue DynamoDB SQS Step Functions Lambda API Gateway Cognito EMR RDS/Auora CloudFormation CloudWatch Languages Python Scala/Java Spark Batch, Streaming, ML Performance tuning at scale Hadoop Hive HiveQL YARN Pig Scoop Ranger Real-time Streaming Kafka Kinesis Data File Formats: Avro, Parquet, JSON, ORC, CSV, XML NoSQL / SQL Microservice development RESTful API development CI/CD pipelines Jenkins / GoCD AWS CodeCommit CodeBuild CodeDeploy CodePipeline Containers Docker / Kubernetes AWS Lambda Fargate EKS Analytics Presto / Athena QuickSight Tableau Test-driven development/test automation, continuous integration, and deployment automation Enjoy working with data - data analysis, data quality, reporting, and visualization Good communicator, able to analyze and clearly articulate complex issues and technologies understandably and engagingly. Great design and problem solving skills, with a strong bias for architecting at scale. Adaptable, proactive and willing to take ownership. Keen attention to detail and high level of commitment. Good understanding in any: advanced mathematics, statistics, and probability. Experience working in agile/iterative development and delivery environments. Comfort in working in such an environment.  Requirements change quickly and our team needs to constantly adapt to moving targets. 189630
Salary Range: NA
Minimum Qualification
Not Specified years

Don't Be Fooled

The fraudster will send a check to the victim who has accepted a job. The check can be for multiple reasons such as signing bonus, supplies, etc. The victim will be instructed to deposit the check and use the money for any of these reasons and then instructed to send the remaining funds to the fraudster. The check will bounce and the victim is left responsible.