AI Developer Tools
Observability, evaluation, data and fine-tuning tools for AI applications. 170 tools listed. See also the guide to the best AI Developer Tools tools.
Your tool could be here
Featured spots are labeled "Sponsored" and sit at the top of the directory and your category.
Get featured →-
Agenta
Open-source platform for prompt management, evaluation and observability of LLM apps.
-
Agnost AI
Monitors AI agents in production, finds silent failures and turns trace data into tested fixes.
-
Aim
Open-source experiment tracker and AI observability tool.
-
Amazon SageMaker
AWS platform for building, training and deploying machine learning models.
-
Amazon Textract
AWS service that extracts text, tables and forms from documents.
-
Anchorhead
Evals for frontier research agents.
-
Appen
Provider of training data, data annotation and model evaluation for AI.
-
Argilla
Open-source collaboration tool for building and labeling high-quality AI datasets.
-
Arize Phoenix
Open-source observability and evaluation tool for LLM apps.
-
Arthur AI
AI evaluation, monitoring and governance platform for models and agents.
-
Ashr
Enterprise post-training, monitoring and continual learning.
-
assistant-ui
Open-source React components for AI chat interfaces.
-
Aucert
AI agents that explore and test your mobile app on every build and return evidence-backed verdicts.
-
Axolotl
Open-source tool for fine-tuning language models.
-
BAML
Open-source language and toolchain from Boundary for type-safe structured LLM prompts and outputs.
-
beacon.sh
Open-source self-improving memory layer for AI coding agents.
-
BentoML
Open-source framework and cloud for packaging and serving AI models.
-
BioStack Platforms
Healthcare and preclinical datasets, annotation and infrastructure for AI model training and evaluation.
-
Braintrust
Evaluation and observability platform for AI products.
-
Carrot Labs
AI cost management across providers.
-
Chainlit
Open-source Python framework for building conversational AI frontends.
-
Channel3
Product-search and enrichment API that gives shopping agents structured catalog data, merchant offers, and sales-commission integrations.
1,000 free credits/mo, then $0.007/credit -
Chunkr
Document parsing and chunking API for RAG applications.
-
Cleanlab
Tools that detect and fix issues in AI training data and flag untrustworthy LLM responses.
-
ClearML
Open-source platform for ML experiment tracking, orchestration and serving.
-
Clocknext
Billing and monetization infrastructure for AI-native apps and agents.
-
Comet ML
ML experiment tracking, evaluation and model management.
-
Confident AI
LLM evaluation platform built on the open-source DeepEval framework.
-
Cudo Compute
Distributed cloud compute marketplace.
-
CVAT
Annotation platform for computer vision and audio data, available open source or as a cloud service.
-
Datadog LLM Observability
Datadog monitoring, tracing and evaluation for LLM applications and AI agents.
-
Datalab
Document conversion and OCR models and API from the makers of Marker and Surya.
-
Datasaur
Enterprise AI platform for private LLMs, with data labeling tools for NLP and LLM projects.
-
Dedica
Checks whether AI agents can successfully use your website.
-
DeepEval
Open-source LLM evaluation framework.
-
DeepSpeed
Microsoft deep learning optimization library for training and inference at scale.
-
Distilabel
Framework for synthetic data generation and AI feedback.
-
Docling
Open-source toolkit for converting documents into structured formats for AI.
-
dstack
Open-source orchestration for AI training and inference across clouds.
-
DVC
Open-source data and model versioning for ML projects.
-
E2B
Secure cloud sandboxes for running code produced by AI agents.
-
Encord
Data platform for annotating and managing multimodal AI training data.
-
Eraser
AI diagramming and technical design docs.
-
Evidently AI
Open-source and cloud tools for evaluating, testing and monitoring ML and LLM systems.
-
Extend
Document processing platform for parsing and extracting data with AI.
-
Feast
Open-source feature store for machine learning.
-
Fiddler AI
Monitoring, evaluation and guardrails platform for enterprise ML models, LLM apps and AI agents.
-
Firecrawl
API that crawls websites and returns clean, LLM-ready data.
-
Floot Payments
Payments for apps built inside ChatGPT and Claude.
-
Flyte
Open-source orchestration platform for data, ML and AI workflows.
-
Future AGI
Evaluation and observability platform for AI agents and LLM apps.
-
Galileo
AI evaluation and observability platform for measuring, monitoring and protecting LLM apps and agents.
-
Giskard
Testing and red-teaming platform that finds vulnerabilities and quality issues in LLM agents.
-
Glasshand
Rents real iPhones and Macs in a data center for AI agents to operate.
-
Google Document AI
Google Cloud document processing and extraction service.
-
Gradio
Python library for building and sharing web demos of machine learning models.
-
Guardrails AI
Framework for validating and guarding LLM outputs.
-
Guidance
Microsoft-originated library for controlling LLM outputs.
-
Hamming
Evaluation and monitoring platform for voice and conversational AI agents.
-
hardware intelligence
AI tools that help engineers design computer chips.
-
Helicone
Open-source LLM observability and gateway platform.
-
HoneyHive
Observability and evaluation platform for testing and monitoring AI agents.
-
Hopsworks
AI lakehouse and feature store platform for ML and LLM systems.
-
Hue
Simulated apps, data and stateful environments for testing AI agents.
-
HyperProbe
Debug production code without redeploying, with AI help to find and fix incidents.
-
iMerit
Data annotation and AI data solutions with expert annotators.
-
Inngest
Durable workflow engine for background jobs and AI agents.
-
Instructor
Library for getting structured outputs from language models.
-
io.net
Decentralized GPU network for AI.
-
Jam
Bug reporting browser extension that captures screen, console and network logs, with AI debugging.
-
Jina AI
Embeddings, reranker and web reader models and APIs for search.
-
Jupyter AI
Open-source generative AI extension for JupyterLab with a chat interface and notebook magics.
-
Kedro
Open-source Python framework for reproducible data science pipelines.
-
Kestrel AI
AI agents for platform engineering that automate incident response, cloud provisioning, CI/CD and developer requests.
-
Kili Technology
Data labeling and evaluation platform for training and testing AI models.
-
Kiln
Free desktop app for building AI tasks, synthetic datasets, evals and fine-tunes.
-
KServe
Open-source model serving on Kubernetes.
-
Kubeflow
Open-source toolkit for running machine learning workflows on Kubernetes.
-
Label Studio
Open-source data labeling tool for many data types.
-
Labelbox
Data labeling platform and expert workforce for training and evaluating AI models.
-
Lakera
Security platform for protecting generative AI applications.
-
Laminar
Open-source observability and evaluation platform for AI agents.
-
Langfuse
Open-source LLM engineering platform for tracing, evaluation and prompt management.
-
LangSmith
Observability, evaluation and deployment platform for LLM applications.
-
LangWatch
LLM observability, evaluation and agent testing platform.
-
llama.cpp
Open-source C/C++ library for running language models efficiently on local hardware.
-
LlamaFactory
Open-source toolkit with a web UI for fine-tuning 100+ LLMs and vision-language models.
-
LlamaParse
LlamaIndex document parser that turns complex PDFs into LLM-ready text.
-
Ludwig
Open-source declarative framework for building custom AI models and fine-tuning LLMs.
-
Lumiko
Browser demo recording tool aimed at product teams.
-
Lunary
Open-source platform for LLM observability, prompt management and evaluations.
-
Maritime
Cloud computers customized for any AI agent.
-
Maxim AI
Platform for simulating, evaluating and observing AI agents, and maker of the open-source Bifrost AI gateway.
-
Mermaid Chart
Diagramming platform from the creators of Mermaid, with AI diagram generation.
-
Metaflow
Open-source framework for building and managing ML and AI systems.
-
Mindee
Document parsing API for invoices, receipts and IDs.
-
MinerU
Open-source tool that converts PDFs into machine-readable Markdown and JSON.
-
Mirascope
Open-source Python toolkit for building LLM applications with a unified interface across providers.
-
Mirrors
Turns an AI agent's recurring mistakes into targeted, real-time corrections.
-
MLflow
Open-source platform for managing the machine learning and GenAI lifecycle.
-
MLX
Apple open-source array framework for machine learning on Apple silicon.
-
NeMo Guardrails
NVIDIA open-source toolkit for adding programmable guardrails to LLM apps.
-
NVIDIA Dynamo-Triton
NVIDIA open-source inference serving software (formerly Triton Inference Server) for deploying AI models at scale.
-
Okareo
Platform for testing, evaluating and monitoring AI agents and LLM apps.
-
OpenCV
Open-source computer vision library.
-
Openlayer
AI evaluation, testing and governance platform for ML and LLM systems.
-
OpenLIT
Open-source, OpenTelemetry-native observability for LLM apps, agents and GPUs.
-
Opik
Open-source platform from Comet for evaluating and monitoring LLM applications.
-
Oumi
Open platform for building, training and improving specialized AI models and agents.
-
Outlines
Library for constraining language model output to defined structures.
-
Papaya
Analyzes AI agent traces and code to improve quality, cut latency and reduce cost.
-
Paperspace
Cloud GPUs and notebooks for machine learning workloads.
-
Patronus AI
AI research lab that studies where models fail and builds training data and simulated environments to improve them.
-
PEFT
Hugging Face library for parameter-efficient fine-tuning.
-
Postman
API platform for designing, testing and documenting APIs, with AI tools for building agents.
-
Prime Intellect
Decentralized AI compute and research platform for training and collaborating on models.
-
Prized
Persistent cloud environments for coding agents.
-
Prodigy
Developer annotation tool for NLP and computer vision from Explosion.
-
Promptfoo
Open-source tool for testing and red-teaming prompts, models and agents.
-
PromptLayer
Prompt management, versioning and evaluation platform for LLM teams.
-
Pushary
Sends AI agent permission prompts to the right person's phone or Mac.
-
Puter
Open-source internet OS with AI and developer APIs.
-
Pydantic Logfire
Observability platform from the Pydantic team for tracing apps and LLM calls, built on OpenTelemetry.
-
Ragas
Open-source toolkit for evaluating retrieval-augmented generation applications.
-
Ray
Open-source framework for scaling Python and AI workloads.
-
Reducto
Document parsing API that converts PDFs, spreadsheets and scans into structured data for AI applications.
-
Respan
Open-source AI gateway with built-in observability and evals for LLM apps and agents (formerly Keywords AI).
-
Roark
Simulates and scores voice AI agents before launch and monitors production calls.
-
Roboflow
Computer vision platform for labeling, training and deploying models.
-
Runpod
GPU cloud with on-demand pods and serverless compute for AI training, inference and batch jobs.
-
Sama
Data annotation and model validation for computer vision and generative AI.
-
Scale AI
Data labeling, evaluation and AI application services for enterprises and governments.
-
sentence-transformers
Python library for text and image embeddings.
-
Sentry
Error and performance monitoring platform with Seer, an AI agent for debugging issues.
-
SGLang
Open-source fast serving framework for large language and vision-language models.
-
Simantic
Runs and tests real firmware on simulated hardware, from a single microcontroller to connected devices.
-
Simulithic
AI agents that map user journeys in your app, test them on every pull request and watch production.
-
SkyPilot
Open-source framework for running AI workloads on any cloud or Kubernetes.
-
Snorkel AI
Data development platform for labeling and curating AI training and evaluation data.
-
Sonarly
Autonomous AI on-call engineer that triages alerts and opens fix PRs.
-
spaCy
Open-source library for industrial-strength natural language processing in Python.
-
Specsight
Product observability for teams shipping with AI.
-
Streamlit
Python framework for building data and AI web apps.
-
SuperAnnotate
Data labeling platform for vision and LLM datasets.
-
Superlog
AI monitoring that finds missing logs and traces, catches errors and automates investigations.
-
Superserve
Cloud virtual machines and sandboxes for long-running AI agents, with persistent storage, pause and resume, and programmable networking.
Free trial; paid from $20/mo with usage billing -
Surge AI
Human data and RLHF platform for training and evaluating AI models.
-
telemetry.dev
LLM observability focused on AI agents.
-
Temporal
Durable execution platform for reliable workflows, used to run AI agents and long-running processes.
-
TensorBoard
Open-source visualization toolkit for machine learning experiments.
-
Testifly
Autonomous AI QA that writes and maintains tests with little prompting.
-
Thunder Compute
Affordable cloud GPUs for AI.
-
Toloka
Human data platform for AI training, evaluation and red teaming.
-
torchtune
PyTorch library for fine-tuning LLMs.
-
Traceloop
LLM observability platform built on OpenLLMetry, an open-source OpenTelemetry extension.
-
Transformers
Hugging Face library of pretrained models for text, vision, audio and multimodal tasks.
-
TRL
Hugging Face library for post-training LLMs with SFT, DPO and RL.
-
TruLens
Open-source library for evaluating and tracing LLM apps and agents.
-
Ultralytics YOLO
Computer vision models and platform for object detection and segmentation.
-
Unsloth
Open-source library for faster, lower-memory fine-tuning of language models.
-
Unstructured
Turns documents into structured data for LLM pipelines.
-
V7
AI data labeling and document-processing agents for enterprises.
-
Vellum
Platform for building, testing and deploying AI agents and workflows.
-
vLLM
Open-source high-throughput inference engine for large language models.
-
Voltage Park
GPU cloud for large-scale AI training and inference.
-
Voyage AI
Embedding and reranking models for retrieval.
-
Weave
Weights & Biases toolkit for tracking and evaluating LLM applications.
-
Weights & Biases
Platform for tracking machine learning experiments, managing models and evaluating LLM applications.
-
WhyLabs
AI observability platform for data and model monitoring.
-
ZenML
Open-source MLOps framework for building portable ML and LLM pipelines.