IntermediatePython

CuratorNVIDIA NeMo's open-source data preprocessing toolkit

Curator is an open-source data preprocessing toolkit developed by NVIDIA's NeMo team, specifically engineered for large language model training. It offers a scalable, modular pipeline for essential tasks such as text cleaning, quality filtering, and deduplication, helping developers extract high-quality data from raw corpora efficiently. With core components rewritten in Rust, it delivers exceptional performance and integrates smoothly into existing data pipelines. The project primarily uses Python and is licensed under Apache-2.0.

1.6K Stars

290 Forks

230 Issues

108 Views

Python

Apache-2.0

IndexedJune 26, 2026

GitHub repository

Project Overview

Overview

Curator is an open-source data preprocessing toolkit from NVIDIA's NeMo team, designed for large language model (LLM) training. It enables developers to efficiently extract high-quality data from raw corpora, improving training outcomes.

Core Features

Text cleaning: removes noise and irrelevant content.
Quality filtering: selects high-quality samples based on rules or models.
Deduplication: eliminates duplicate data to reduce redundancy.

Technology Stack

The project is primarily written in Python, but its core components are rewritten in Rust for superior performance. This hybrid design combines Python's ease of use with Rust's high efficiency.

License

Curator is licensed under Apache-2.0, allowing free use and modification.

Getting Started

According to the README summary, Curator is designed as a modular pipeline that integrates seamlessly into existing data processing flows. For detailed installation and usage instructions, please refer to the official documentation.

LLM data preprocessingNVIDIA NeMoopen-source data toolkittext cleaningdata deduplicationRust performanceAI data pipelinesMinHashdata qualityPython tools

Project Rating

0.0 (0 Reviews)

Frequently Asked Questions

What is Curator: NVIDIA NeMo's open-source data preprocessing toolkit?

What language is Curator: NVIDIA NeMo's open-source data preprocessing toolkit written in?

Curator: NVIDIA NeMo's open-source data preprocessing toolkit is primarily written in Python.

What license is Curator: NVIDIA NeMo's open-source data preprocessing toolkit under?

Curator: NVIDIA NeMo's open-source data preprocessing toolkit is released under the Apache-2.0 license.

Related Projects

No results yet

Explore More

Similar Tools

PakBot

PakBot is billed as the first AI assistant made for Pakistan, with free chat in Urdu, English, Punjabi, Sindhi and Pashto, plus image generation, voice chat and web search.

Filently

Filently is a Swiss-built AI tool that automatically renames and files Google Drive documents, learning your existing folders and reading dates, vendors and amounts by OCR.

Nika

Nika is a suite of AI companion products, including the Nika AI girlfriend, the Sebastian AI boyfriend and the Aurora City dating simulator. It offers text, voice and multimedia roleplay through Telegram and a web app, with a level-based bond system.

WeiClaw

WeiClaw is a hardware device that pairs with Agent-running computers, watching Agent status and toggling the host between sleep and active on demand.

Tomo

Tomo is an AI assistant that lives inside WhatsApp and Telegram, requiring no downloads or onboarding mazes. Users can text natural language like 'dentist Thursday 3pm' and it syncs to Google Calendar, remembers context, sends proactive morning briefings, and learns routines.