Experienced Big Data Engineer - Software Development and Team Leadership Opportunity with American Express

Introduction to American Express and Our Technology Team American Express is a global services company that provides customers with access to products, insights, and experiences that enrich lives and build business success. Our technology team is at the forefront of innovation, leveraging cutting-edge technologies to deliver exceptional customer experiences and drive business growth. As a member of our diverse tech group, you will have the opportunity to engineer, code, and develop software that makes us an essential part of our customers' digital lives. Job Overview We are seeking a highly skilled and experienced Big Data Engineer to join our team. As a Big Data Engineer, you will be responsible for designing, developing, and testing software applications that enable us to process and analyze large datasets. You will work closely with cross-functional teams to identify business requirements and develop solutions that meet those needs. This is a full-time, remote opportunity that offers a competitive salary of $28 per hour and the chance to work with a talented team of professionals who are passionate about technology and innovation. Key Responsibilities Develop and test software applications, including ongoing refactoring of code and driving continuous improvement in code structure and quality Participate in code reviews and automated testing Troubleshoot basic software components and identify code defects for remediation Enable the deployment, support, and monitoring of software across test, integration, and production environments Automate solutions in test or production environments Scale applications based on demand projections Lead a small team of engineers and inspire them to achieve their full potential Influence colleagues with innovative changes and improvements by challenging the status quo and demonstrating risk-taking Essential Qualifications Bachelor's degree in Engineering, Computer Science, or a related field 3+ years of software development experience and leading teams of engineers and scrum teams 3+ years of hands-on experience working with Hadoop, Hive, Spark (core, SQL, and PySpark) Strong data warehousing concepts Knowledge of financial reporting environment is a plus Expertise in distributed environment Hands-on experience with programming languages such as Core Java or Python/Scala Expertise in Hadoop and Spark architecture and its working principle Hands-on experience with writing and optimizing complex SQL (Hive/PySpark-dataframes), improving joins while handling large amounts of data Experience with UNIX shell scripting Ability to design and develop advanced data pipelines for batch and real-time data processing Should have experience in analysis, design, development, testing, and implementation of system applications Demonstrated ability to create and document technical and functional specifications and analyze software and system processing flows Preferred Qualifications Knowledge of cloud platforms like GCP/AWS, building microservices, and scalable solutions is a plus 3+ years of experience with designing and building solutions using Kafka streams or queues Experience with GitHub/Bitbucket and using CI/CD pipelines Experience with NoSQL databases such as HBase, Couchbase, MongoDB is good to have Excellent technical and analytical skills Good communication skills Exceptional project management skills Career Growth Opportunities and Learning Benefits At American Express, we are committed to helping our employees grow and develop their skills. As a Big Data Engineer, you will have access to a range of training and development opportunities, including online courses, workshops, and conferences. You will also have the chance to work with a talented team of professionals who are passionate about technology and innovation, and who can provide guidance and support as you grow in your career. Work Environment and Company Culture American Express is a dynamic and inclusive workplace that values diversity and promotes a culture of innovation and collaboration. Our technology team is a close-knit community of professionals who are passionate about technology and innovation, and who are committed to delivering exceptional customer experiences. As a Big Data Engineer, you will be part of a team that is dedicated to excellence and that is always looking for new and better ways to solve problems and deliver results. Compensation, Perks, and Benefits American Express offers a competitive salary and benefits package that includes health insurance, retirement savings, and paid time off. As a Big Data Engineer, you will also have access to a range of perks and benefits, including flexible working hours, remote work options, and a comprehensive wellness program. Conclusion If you are a highly skilled and experienced Big Data Engineer who is looking for a new challenge and opportunity to grow your career, we encourage you to apply for this role. At American Express, we are committed to innovation, diversity, and excellence, and we are looking for talented professionals who share our values and our passion for technology and innovation. Don't miss out on this opportunity to join our team and help us shape the future of technology and customer experience.