About Careertakes

👉 Important disclosure: Careertakes is a third-party recruiting platform supporting this hiring process. If selected, you will be employed directly by our client, Technology, Information and Internet.

Applicants for this role may also receive access to additional matched opportunities through the Careertakes platform.

---

Position overview

Confidential Client is hiring a Software Engineer - Python/Typescript (Remote, independent contractor) to support AI model development through high-quality data evaluation, rubric creation, and engineering of evaluation workflows. This contract role is expected to run ~3 months with an immediate start. Preferred commitment is ~40 hours/week with at least six hours overlap with Pacific Time.

---

What You’ll Do

• Design, refine, and improve rubrics and evaluation criteria for preference and annotation data used in LLM/AI research.
• Review, label, and audit dataset examples against high-quality standards; identify subtle issues and inconsistencies.
• Build and maintain reproducible pipelines and lightweight infrastructure that support data generation, collection, and evaluation.
• Translate qualitative judgments into measurable, consistent evaluation processes for scale.
• Analyze results and communicate findings through concise reports and actionable recommendations.
• Collaborate with researchers and engineers to iterate on evaluation methods and data quality improvements.
• Apply strong technical judgment to explain why examples are high or low quality and to preserve evaluation consistency.
• Deliver practical solutions in a fast-moving environment and adapt to changing technical requirements.

---

What We’re Looking For (Required)

• Proficiency in Python or TypeScript and experience applying those languages to engineering workflows or code maintenance.
• Hands-on experience with preference data collection, rubric creation, or related annotation/evaluation workflows.
• Strong written and verbal communication; able to summarize technical findings clearly and concisely.
• Demonstrated quality judgment—able to explain why an example is effective (not just whether it meets minimal requirements).
• Ability to work independently as a contractor and collaborate effectively with cross-functional teams (researchers, engineers).
• Comfortable working in ambiguous, fast-paced projects while maintaining high standards for delivery.

---

Preferred Qualifications

• Experience navigating and contributing to large codebases and developer tools (e.g., VS Code) — particularly relevant for TypeScript work.
• Prior work building or maintaining data pipelines and evaluation infrastructure.
• Familiarity with AI model training, preference-data evaluation, or quality-driven LLM workflows.
• Experience converting subjective assessments into clear criteria and checks that can be applied consistently at scale.

---

Logistics & Compensation

• Work arrangement: Fully remote (independent contractor).
• Contract type: Independent contractor (expected duration ~3 months).
• Time commitment: 40 hours/week preferred. If you cannot do 40 hours, state your weekly availability and preferred overlap with Pacific Time (6+ hours overlap preferred).
• Start: Immediate/ASAP.
• Compensation: Market-based; candidates should provide a specific expected hourly rate to help determine mutual fit.
• Benefits: This contractor engagement does not include medical insurance or paid leave.

---

Interview Process

• One interview (~30 minutes) focused on technical capability and fit.

---

Keywords (for applicants)

• Python, TypeScript, preference data, rubric creation, data evaluation, data pipelines, LLM, remote contractor

---

Equal Opportunity & Hiring Transparency

Careertakes and our client are Equal Opportunity Employers committed to building a diverse and inclusive workforce. We prohibit discrimination or harassment of any kind. To support a fair and efficient hiring process, AI tools may be used to assist with application review or resume screening. These tools do not replace human decision-making. Final hiring decisions are made by people.

If you have questions about how your data is used, please contact us directly.