Home › Best AI tools › AI Developer Tools
Best AI Tools for Ai development and operations (170 options)
170 tools for AI development and operations in the directory. Observability, evaluation, data and fine-tuning tools for AI applications.
How to read this page. We do not rank tools or publish scores. This page lists every AI Developer Tools tool in the directory with its official link and pricing model so you can shortlist, then try the ones that fit. Pricing labels come from each vendor's pricing page when we could confirm them; "See site" means we could not.
At a glance
- 170 tools listed; 39 free to use, 38 with a free tier, 8 paid, 85 where pricing is not confirmed here.
- With a free plan or tier: Agenta, Aim, assistant-ui, Axolotl, beacon.sh, Braintrust, Carrot Labs, Chainlit, Channel3, Cleanlab, Clocknext, Comet ML, Confident AI, CVAT, Datalab, DeepSpeed, Distilabel, Docling, DVC, Eraser, Feast, Fiddler AI, Firecrawl, Future AGI, Gradio, Guidance, Helicone, HoneyHive, Hopsworks, Inngest, Instructor, Jam, Jupyter AI, Kedro, Kiln, KServe, Laminar, Langfuse, LangWatch, llama.cpp, LlamaFactory, Ludwig, Lunary, Maritime, Metaflow, Mindee, MinerU, MLflow, MLX, NeMo Guardrails, OpenCV, Oumi, Outlines, Paperspace, PEFT, Postman, Prized, Pydantic Logfire, Ragas, Ray, Respan, Roboflow, sentence-transformers, Sentry, SGLang, spaCy, Streamlit, telemetry.dev, TensorBoard, Thunder Compute, torchtune, Traceloop, Transformers, TRL, TruLens, Ultralytics YOLO, vLLM.
What to look for
Check language and framework support, self-hosting options and how the tool fits your existing deployment and monitoring stack.
All 170 tools
| Tool | What it does | Pricing | Link |
|---|---|---|---|
| Agenta | Open-source platform for prompt management, evaluation and observability of LLM apps. | Freemium | Official site |
| Agnost AI | Monitors AI agents in production, finds silent failures and turns trace data into tested fixes. | See site | Official site |
| Aim | Open-source experiment tracker and AI observability tool. | Free | Official site |
| Amazon SageMaker | AWS platform for building, training and deploying machine learning models. | See site | Official site |
| Amazon Textract | AWS service that extracts text, tables and forms from documents. | See site | Official site |
| Anchorhead | Evals for frontier research agents. | See site | Official site |
| Appen | Provider of training data, data annotation and model evaluation for AI. | See site | Official site |
| Argilla | Open-source collaboration tool for building and labeling high-quality AI datasets. | See site | Official site |
| Arize Phoenix | Open-source observability and evaluation tool for LLM apps. | See site | Official site |
| Arthur AI | AI evaluation, monitoring and governance platform for models and agents. | See site | Official site |
| Ashr | Enterprise post-training, monitoring and continual learning. | See site | Official site |
| assistant-ui | Open-source React components for AI chat interfaces. | Freemium | Official site |
| Aucert | AI agents that explore and test your mobile app on every build and return evidence-backed verdicts. | See site | Official site |
| Axolotl | Open-source tool for fine-tuning language models. | Free | Official site |
| BAML | Open-source language and toolchain from Boundary for type-safe structured LLM prompts and outputs. | See site | Official site |
| beacon.sh | Open-source self-improving memory layer for AI coding agents. | Freemium | Official site |
| BentoML | Open-source framework and cloud for packaging and serving AI models. | See site | Official site |
| BioStack Platforms | Healthcare and preclinical datasets, annotation and infrastructure for AI model training and evaluation. | See site | Official site |
| Braintrust | Evaluation and observability platform for AI products. | Freemium | Official site |
| Carrot Labs | AI cost management across providers. | Freemium | Official site |
| Chainlit | Open-source Python framework for building conversational AI frontends. | Free | Official site |
| Channel3 | Product-search and enrichment API that gives shopping agents structured catalog data, merchant offers, and sales-commission integrations. | Freemium | Official site |
| Chunkr | Document parsing and chunking API for RAG applications. | See site | Official site |
| Cleanlab | Tools that detect and fix issues in AI training data and flag untrustworthy LLM responses. | Freemium | Official site |
| ClearML | Open-source platform for ML experiment tracking, orchestration and serving. | See site | Official site |
| Clocknext | Billing and monetization infrastructure for AI-native apps and agents. | Freemium | Official site |
| Comet ML | ML experiment tracking, evaluation and model management. | Freemium | Official site |
| Confident AI | LLM evaluation platform built on the open-source DeepEval framework. | Freemium | Official site |
| Cudo Compute | Distributed cloud compute marketplace. | See site | Official site |
| CVAT | Annotation platform for computer vision and audio data, available open source or as a cloud service. | Freemium | Official site |
| Datadog LLM Observability | Datadog monitoring, tracing and evaluation for LLM applications and AI agents. | See site | Official site |
| Datalab | Document conversion and OCR models and API from the makers of Marker and Surya. | Freemium | Official site |
| Datasaur | Enterprise AI platform for private LLMs, with data labeling tools for NLP and LLM projects. | See site | Official site |
| Dedica | Checks whether AI agents can successfully use your website. | See site | Official site |
| DeepEval | Open-source LLM evaluation framework. | See site | Official site |
| DeepSpeed | Microsoft deep learning optimization library for training and inference at scale. | Free | Official site |
| Distilabel | Framework for synthetic data generation and AI feedback. | Free | Official site |
| Docling | Open-source toolkit for converting documents into structured formats for AI. | Free | Official site |
| dstack | Open-source orchestration for AI training and inference across clouds. | See site | Official site |
| DVC | Open-source data and model versioning for ML projects. | Free | Official site |
| E2B | Secure cloud sandboxes for running code produced by AI agents. | See site | Official site |
| Encord | Data platform for annotating and managing multimodal AI training data. | See site | Official site |
| Eraser | AI diagramming and technical design docs. | Freemium | Official site |
| Evidently AI | Open-source and cloud tools for evaluating, testing and monitoring ML and LLM systems. | See site | Official site |
| Extend | Document processing platform for parsing and extracting data with AI. | See site | Official site |
| Feast | Open-source feature store for machine learning. | Free | Official site |
| Fiddler AI | Monitoring, evaluation and guardrails platform for enterprise ML models, LLM apps and AI agents. | Freemium | Official site |
| Firecrawl | API that crawls websites and returns clean, LLM-ready data. | Freemium | Official site |
| Floot Payments | Payments for apps built inside ChatGPT and Claude. | See site | Official site |
| Flyte | Open-source orchestration platform for data, ML and AI workflows. | See site | Official site |
| Future AGI | Evaluation and observability platform for AI agents and LLM apps. | Freemium | Official site |
| Galileo | AI evaluation and observability platform for measuring, monitoring and protecting LLM apps and agents. | See site | Official site |
| Giskard | Testing and red-teaming platform that finds vulnerabilities and quality issues in LLM agents. | See site | Official site |
| Glasshand | Rents real iPhones and Macs in a data center for AI agents to operate. | Paid | Official site |
| Google Document AI | Google Cloud document processing and extraction service. | See site | Official site |
| Gradio | Python library for building and sharing web demos of machine learning models. | Free | Official site |
| Guardrails AI | Framework for validating and guarding LLM outputs. | See site | Official site |
| Guidance | Microsoft-originated library for controlling LLM outputs. | Free | Official site |
| Hamming | Evaluation and monitoring platform for voice and conversational AI agents. | See site | Official site |
| hardware intelligence | AI tools that help engineers design computer chips. | See site | Official site |
| Helicone | Open-source LLM observability and gateway platform. | Freemium | Official site |
| HoneyHive | Observability and evaluation platform for testing and monitoring AI agents. | Freemium | Official site |
| Hopsworks | AI lakehouse and feature store platform for ML and LLM systems. | Freemium | Official site |
| Hue | Simulated apps, data and stateful environments for testing AI agents. | See site | Official site |
| HyperProbe | Debug production code without redeploying, with AI help to find and fix incidents. | See site | Official site |
| iMerit | Data annotation and AI data solutions with expert annotators. | See site | Official site |
| Inngest | Durable workflow engine for background jobs and AI agents. | Freemium | Official site |
| Instructor | Library for getting structured outputs from language models. | Free | Official site |
| io.net | Decentralized GPU network for AI. | See site | Official site |
| Jam | Bug reporting browser extension that captures screen, console and network logs, with AI debugging. | Freemium | Official site |
| Jina AI | Embeddings, reranker and web reader models and APIs for search. | See site | Official site |
| Jupyter AI | Open-source generative AI extension for JupyterLab with a chat interface and notebook magics. | Free | Official site |
| Kedro | Open-source Python framework for reproducible data science pipelines. | Free | Official site |
| Kestrel AI | AI agents for platform engineering that automate incident response, cloud provisioning, CI/CD and developer requests. | See site | Official site |
| Kili Technology | Data labeling and evaluation platform for training and testing AI models. | See site | Official site |
| Kiln | Free desktop app for building AI tasks, synthetic datasets, evals and fine-tunes. | Free | Official site |
| KServe | Open-source model serving on Kubernetes. | Free | Official site |
| Kubeflow | Open-source toolkit for running machine learning workflows on Kubernetes. | See site | Official site |
| Label Studio | Open-source data labeling tool for many data types. | See site | Official site |
| Labelbox | Data labeling platform and expert workforce for training and evaluating AI models. | See site | Official site |
| Lakera | Security platform for protecting generative AI applications. | See site | Official site |
| Laminar | Open-source observability and evaluation platform for AI agents. | Freemium | Official site |
| Langfuse | Open-source LLM engineering platform for tracing, evaluation and prompt management. | Freemium | Official site |
| LangSmith | Observability, evaluation and deployment platform for LLM applications. | See site | Official site |
| LangWatch | LLM observability, evaluation and agent testing platform. | Freemium | Official site |
| llama.cpp | Open-source C/C++ library for running language models efficiently on local hardware. | Free | Official site |
| LlamaFactory | Open-source toolkit with a web UI for fine-tuning 100+ LLMs and vision-language models. | Free | Official site |
| LlamaParse | LlamaIndex document parser that turns complex PDFs into LLM-ready text. | See site | Official site |
| Ludwig | Open-source declarative framework for building custom AI models and fine-tuning LLMs. | Free | Official site |
| Lumiko | Browser demo recording tool aimed at product teams. | See site | Official site |
| Lunary | Open-source platform for LLM observability, prompt management and evaluations. | Freemium | Official site |
| Maritime | Cloud computers customized for any AI agent. | Freemium | Official site |
| Maxim AI | Platform for simulating, evaluating and observing AI agents, and maker of the open-source Bifrost AI gateway. | See site | Official site |
| Mermaid Chart | Diagramming platform from the creators of Mermaid, with AI diagram generation. | Free trial | Official site |
| Metaflow | Open-source framework for building and managing ML and AI systems. | Free | Official site |
| Mindee | Document parsing API for invoices, receipts and IDs. | Freemium | Official site |
| MinerU | Open-source tool that converts PDFs into machine-readable Markdown and JSON. | Free | Official site |
| Mirascope | Open-source Python toolkit for building LLM applications with a unified interface across providers. | See site | Official site |
| Mirrors | Turns an AI agent's recurring mistakes into targeted, real-time corrections. | See site | Official site |
| MLflow | Open-source platform for managing the machine learning and GenAI lifecycle. | Free | Official site |
| MLX | Apple open-source array framework for machine learning on Apple silicon. | Free | Official site |
| NeMo Guardrails | NVIDIA open-source toolkit for adding programmable guardrails to LLM apps. | Free | Official site |
| NVIDIA Dynamo-Triton | NVIDIA open-source inference serving software (formerly Triton Inference Server) for deploying AI models at scale. | See site | Official site |
| Okareo | Platform for testing, evaluating and monitoring AI agents and LLM apps. | Free trial | Official site |
| OpenCV | Open-source computer vision library. | Free | Official site |
| Openlayer | AI evaluation, testing and governance platform for ML and LLM systems. | See site | Official site |
| OpenLIT | Open-source, OpenTelemetry-native observability for LLM apps, agents and GPUs. | See site | Official site |
| Opik | Open-source platform from Comet for evaluating and monitoring LLM applications. | See site | Official site |
| Oumi | Open platform for building, training and improving specialized AI models and agents. | Free | Official site |
| Outlines | Library for constraining language model output to defined structures. | Free | Official site |
| Papaya | Analyzes AI agent traces and code to improve quality, cut latency and reduce cost. | See site | Official site |
| Paperspace | Cloud GPUs and notebooks for machine learning workloads. | Freemium | Official site |
| Patronus AI | AI research lab that studies where models fail and builds training data and simulated environments to improve them. | See site | Official site |
| PEFT | Hugging Face library for parameter-efficient fine-tuning. | Free | Official site |
| Postman | API platform for designing, testing and documenting APIs, with AI tools for building agents. | Freemium | Official site |
| Prime Intellect | Decentralized AI compute and research platform for training and collaborating on models. | See site | Official site |
| Prized | Persistent cloud environments for coding agents. | Freemium | Official site |
| Prodigy | Developer annotation tool for NLP and computer vision from Explosion. | Paid | Official site |
| Promptfoo | Open-source tool for testing and red-teaming prompts, models and agents. | See site | Official site |
| PromptLayer | Prompt management, versioning and evaluation platform for LLM teams. | See site | Official site |
| Pushary | Sends AI agent permission prompts to the right person's phone or Mac. | Free trial | Official site |
| Puter | Open-source internet OS with AI and developer APIs. | See site | Official site |
| Pydantic Logfire | Observability platform from the Pydantic team for tracing apps and LLM calls, built on OpenTelemetry. | Freemium | Official site |
| Ragas | Open-source toolkit for evaluating retrieval-augmented generation applications. | Free | Official site |
| Ray | Open-source framework for scaling Python and AI workloads. | Free | Official site |
| Reducto | Document parsing API that converts PDFs, spreadsheets and scans into structured data for AI applications. | See site | Official site |
| Respan | Open-source AI gateway with built-in observability and evals for LLM apps and agents (formerly Keywords AI). | Freemium | Official site |
| Roark | Simulates and scores voice AI agents before launch and monitors production calls. | See site | Official site |
| Roboflow | Computer vision platform for labeling, training and deploying models. | Freemium | Official site |
| Runpod | GPU cloud with on-demand pods and serverless compute for AI training, inference and batch jobs. | See site | Official site |
| Sama | Data annotation and model validation for computer vision and generative AI. | See site | Official site |
| Scale AI | Data labeling, evaluation and AI application services for enterprises and governments. | See site | Official site |
| sentence-transformers | Python library for text and image embeddings. | Free | Official site |
| Sentry | Error and performance monitoring platform with Seer, an AI agent for debugging issues. | Freemium | Official site |
| SGLang | Open-source fast serving framework for large language and vision-language models. | Free | Official site |
| Simantic | Runs and tests real firmware on simulated hardware, from a single microcontroller to connected devices. | See site | Official site |
| Simulithic | AI agents that map user journeys in your app, test them on every pull request and watch production. | See site | Official site |
| SkyPilot | Open-source framework for running AI workloads on any cloud or Kubernetes. | See site | Official site |
| Snorkel AI | Data development platform for labeling and curating AI training and evaluation data. | See site | Official site |
| Sonarly | Autonomous AI on-call engineer that triages alerts and opens fix PRs. | See site | Official site |
| spaCy | Open-source library for industrial-strength natural language processing in Python. | Free | Official site |
| Specsight | Product observability for teams shipping with AI. | See site | Official site |
| Streamlit | Python framework for building data and AI web apps. | Free | Official site |
| SuperAnnotate | Data labeling platform for vision and LLM datasets. | See site | Official site |
| Superlog | AI monitoring that finds missing logs and traces, catches errors and automates investigations. | See site | Official site |
| Superserve | Cloud virtual machines and sandboxes for long-running AI agents, with persistent storage, pause and resume, and programmable networking. | Free trial | Official site |
| Surge AI | Human data and RLHF platform for training and evaluating AI models. | See site | Official site |
| telemetry.dev | LLM observability focused on AI agents. | Freemium | Official site |
| Temporal | Durable execution platform for reliable workflows, used to run AI agents and long-running processes. | See site | Official site |
| TensorBoard | Open-source visualization toolkit for machine learning experiments. | Free | Official site |
| Testifly | Autonomous AI QA that writes and maintains tests with little prompting. | See site | Official site |
| Thunder Compute | Affordable cloud GPUs for AI. | Freemium | Official site |
| Toloka | Human data platform for AI training, evaluation and red teaming. | See site | Official site |
| torchtune | PyTorch library for fine-tuning LLMs. | Free | Official site |
| Traceloop | LLM observability platform built on OpenLLMetry, an open-source OpenTelemetry extension. | Freemium | Official site |
| Transformers | Hugging Face library of pretrained models for text, vision, audio and multimodal tasks. | Free | Official site |
| TRL | Hugging Face library for post-training LLMs with SFT, DPO and RL. | Free | Official site |
| TruLens | Open-source library for evaluating and tracing LLM apps and agents. | Free | Official site |
| Ultralytics YOLO | Computer vision models and platform for object detection and segmentation. | Freemium | Official site |
| Unsloth | Open-source library for faster, lower-memory fine-tuning of language models. | See site | Official site |
| Unstructured | Turns documents into structured data for LLM pipelines. | See site | Official site |
| V7 | AI data labeling and document-processing agents for enterprises. | See site | Official site |
| Vellum | Platform for building, testing and deploying AI agents and workflows. | Free trial | Official site |
| vLLM | Open-source high-throughput inference engine for large language models. | Free | Official site |
| Voltage Park | GPU cloud for large-scale AI training and inference. | Paid | Official site |
| Voyage AI | Embedding and reranking models for retrieval. | See site | Official site |
| Weave | Weights & Biases toolkit for tracking and evaluating LLM applications. | See site | Official site |
| Weights & Biases | Platform for tracking machine learning experiments, managing models and evaluating LLM applications. | See site | Official site |
| WhyLabs | AI observability platform for data and model monitoring. | See site | Official site |
| ZenML | Open-source MLOps framework for building portable ML and LLM pipelines. | See site | Official site |
Related guides
- Alternatives to beacon.sh
- Alternatives to Firecrawl
- Alternatives to Glasshand
- Alternatives to Langfuse
- Alternatives to Maritime
- Alternatives to Paperspace
- Alternatives to Pydantic Logfire
- Alternatives to Scale AI
- Alternatives to Testifly
- Langfuse vs LangSmith
- Firecrawl vs Crawl4AI
- Scale AI vs Labelbox
- Pydantic Logfire vs Langfuse
- Spider vs Firecrawl
- Lasso Security vs Lakera
- vLLM vs SGLang
- Glasshand vs Maritime
- beacon.sh vs Mem0
- Testifly vs Momentic
- Paperspace vs Lambda
- Prized vs Maritime