AIML - Senior Manager, ML Product Evaluation Engineering
Cupertino, California
Summary
Would you like to play a part in shipping groundbreaking technology for large-scale systems, natural language, and artificial intelligence? Join the Siri ML Systems Evaluation Engineering (MLSEE) team at Apple! We interact with various engineering teams to identify issues and test requirements, drive the quality of Siri features, and create frameworks that support evaluation and qualification. You will work with the people who craft the intelligent assistant that helps millions of people get things done - just by asking!
As a senior manager, you will play a critical role in driving the evaluation methods of various platforms and product areas for Siri. You will drive coverage and fidelity of testing and evaluation for pre-ship hill climbing and readiness for a wide range of AIML products, including Apple Intelligence features. You will be crafting new and leveraging existing methodologies and frameworks to validate the end-to-end quality of new features. Working alongside SW and ML engineering product teams as a critical partner, you will utilize a suite of available libraries and frameworks - test automation, LLM-based evaluators, and failure analysis tools - and drive requirements for improving those. You will be responsible for benchmarking performance for new features/platforms and driving improvement recommendations.
Description
We are seeking an expert senior engineering manager to lead Siri's intelligent services and platforms (iPhone, HomePod, CarPlay, Apple Watch, iPad, MacBook) evaluation and qualification. You will be responsible for partnerships with product engineering that will drive the test and evaluation strategy, frameworks, performance testing framework, and solutions that can scale. You will work alongside SW and ML engineering product teams as a critical partner. You will need to drive efficiency through the adoption of LLMs for testing and evaluation, including evaluators, auto-triage of issues, and auto-error-attribution.
This role will collaborate with Siri Engineering teams, create a quality strategy in partnership with the product roadmap, work on client integration and performance automation frameworks, analyze results, and reach consensus as we assess the impact of changes, new features, and overall readiness for each release. This is a fast-paced role with high visibility, impact, and influence in building one of the most advanced AI systems in the industry.
Would you like to play a part in shipping groundbreaking technology for large-scale systems, natural language, and artificial intelligence? Join the Siri ML Systems Evaluation Engineering (MLSEE) team at Apple! We interact with various engineering teams to identify issues and test requirements, drive the quality of Siri features, and create frameworks that support evaluation and qualification. You will work with the people who craft the intelligent assistant that helps millions of people get things done - just by asking!
As a senior manager, you will play a critical role in driving the evaluation methods of various platforms and product areas for Siri. You will drive coverage and fidelity of testing and evaluation for pre-ship hill climbing and readiness for a wide range of AIML products, including Apple Intelligence features. You will be crafting new and leveraging existing methodologies and frameworks to validate the end-to-end quality of new features. Working alongside SW and ML engineering product teams as a critical partner, you will utilize a suite of available libraries and frameworks - test automation, LLM-based evaluators, and failure analysis tools - and drive requirements for improving those. You will be responsible for benchmarking performance for new features/platforms and driving improvement recommendations.
Description
We are seeking an expert senior engineering manager to lead Siri's intelligent services and platforms (iPhone, HomePod, CarPlay, Apple Watch, iPad, MacBook) evaluation and qualification. You will be responsible for partnerships with product engineering that will drive the test and evaluation strategy, frameworks, performance testing framework, and solutions that can scale. You will work alongside SW and ML engineering product teams as a critical partner. You will need to drive efficiency through the adoption of LLMs for testing and evaluation, including evaluators, auto-triage of issues, and auto-error-attribution.
This role will collaborate with Siri Engineering teams, create a quality strategy in partnership with the product roadmap, work on client integration and performance automation frameworks, analyze results, and reach consensus as we assess the impact of changes, new features, and overall readiness for each release. This is a fast-paced role with high visibility, impact, and influence in building one of the most advanced AI systems in the industry.
Created: 2024-06-29
Reference: 200556905
Country: United States
State: California
City: Cupertino
About Apple
Founded in: 1976
Number of Employees: 154000
Website: https://www.apple.com/
Career site: https://www.apple.com/careers/us/
Wikipedia: https://en.wikipedia.org/wiki/Apple_Inc.
Instagram: https://www.instagram.com/apple/
LinkedIn: https://www.linkedin.com/company/apple
Similar jobs:
-
AIML - Director of Engineering, Siri Device Ecosystem
Apple in Cupertino, California -
AIML - Sr Manager, Program Office, Data Operations
Apple in Cupertino, California -
Specialist Sales Rep, GenAI/ML, AGS-NAMER-US-COV-SPEC-DATA\u0026AI-AIML/GenAI
Amazon in San Francisco, California💸 $73900 per year -
AIML - Software Engineer, Machine Learning Platform and Technologies
Apple in Cupertino, California -
AIML - Machine Learning Engineer/Scientist, Siri and Information Intelligence
Apple in Cupertino, California -
AIML - Senior Software Engineer, Privacy - Machine Learning Platform and Technology
Apple in Santa Clara Valley (Cupertino), California -
AIML - Software Engineer, Siri Speech
Apple in Cupertino, California -
AIML - Senior Machine Learning Engineer, Computer Vision, Siri and Information Intelligence
Apple in Cupertino, California -
AIML - Full Stack Engineer, Siri and Information Intelligence
Apple in Cupertino, California -
AIML - Director of Business Operations
Apple in Cupertino, California -
AIML - Sr Engineering Program Manager, Advanced Question Answering Systems
Apple in Cupertino, California -
AIML - Senior Data Infrastructure Software Engineer, Machine Learning Platform and Technology
Apple in Santa Clara Valley (Cupertino), California -
AIML - Principal Staff Software Engineer, Privacy - Machine Learning Platform and Technology
Apple in Cupertino, California -
AIML- Machine Learning Engineer, Machine Learning Platform & Infrastructure
Apple in Cupertino, California -
AIML - Software Development Engineer, MLPT
Apple in Cupertino, California -
AIML - Sr ML Engineer, Data and ML Innovation
Apple in Cupertino, California -
AIML - Sr Engineering Program Manager, Siri Metrics & Evaluation Methods
Apple in San Francisco, California -
AIML - ML Engineer, Siri Search Relevance
Apple in Cupertino, California -
AIML - Machine Learning Engineer, Foundation Models
Apple in Cupertino, California -
AIML - Director of Engineering
Apple in Cupertino, California