About Careertakes

👉 Important disclosure: Careertakes is a third-party recruiting platform supporting this hiring process. If selected, you will be employed directly by our client, Software Development.

Applicants for this role may also receive access to additional matched opportunities through the Careertakes platform.

---

What You’ll Do

Confidential Client is building production AI agents for real users. As an entry-level Data Analyst focused on AI agent analytics, you will measure agent quality, analyze user-agent interactions, and translate behavioral data into actionable insights that improve models, prompts, tooling, and product workflows.

• Define and maintain metrics for task completion, response quality, latency, reliability, and cost
• Analyze agent conversation logs, tool usage, errors, and user feedback to identify failure modes
• Build, maintain, and optimize SQL queries, dashboards, and recurring reports
• Support A/B experiments for prompts, models, tools, and workflows and analyze experiment results
• Communicate findings and prioritized recommendations to AI engineering, evaluation, and product teams
• Help operationalize monitoring and alerting for agent performance

---

Qualifications

Required

• Recent graduate in Data Science, Statistics, Computer Science, Mathematics, Economics, or related field

• Strong SQL skills and working familiarity with Python

• Good understanding of statistics, experimentation, and data visualization best practices

• Clear written and verbal communication skills; able to present technical findings to non-technical partners

• Comfortable working independently and in remote or distributed teams

• Strong analytical mindset and attention to data quality and detail

---

Preferred (Nice-to-Have)

• Prior experience analyzing LLM/chatbot/agent interactions or conversational data
• Familiarity with product analytics or event-tracking systems (event schemas, instrumentation)
• Experience with pandas, Tableau, Looker, or similar analysis & visualization tools
• Interest in AI evaluation, prompt engineering, human-AI interaction, or model robustness

---

Why Join

• High ownership and measurable impact on production AI agents used by real people
• Work at the intersection of AI, agents, and product design
• Gain hands-on experience in AI evaluation, experimentation, and production analytics
• Learn from cross-functional teams (AI engineers, product managers, researchers)

---

Workplace & Legal

Confidential Client is committed to fair hiring practices and will comply with applicable U.S. employment laws for roles posted in each state. This role is open to recent graduates and is full-time. The position is remote-friendly; candidates should be able to work effectively in distributed teams and within U.S. employment requirements.

---

Equal Opportunity & Hiring Transparency

Careertakes and our client are Equal Opportunity Employers committed to building a diverse and inclusive workforce. We prohibit discrimination or harassment of any kind. To support a fair and efficient hiring process, AI tools may be used to assist with application review or resume screening. These tools do not replace human decision-making. Final hiring decisions are made by people.

If you have questions about how your data is used, please contact us directly.