Automatically select calibration datasets for code models by changwangss · Pull Request #2107 · intel/auto-round

changwangss · 2026-07-30T15:09:18Z

Description

Automatically select code-oriented calibration datasets when quantizing a pure-text code model without an explicitly configured calibration dataset.

Add is_code_model() following the existing model-detection style.
- Detect code-specialized models from the model name and existing config metadata without additional Hub requests.
- Avoid false positives such as encoder, decoder, codec, and unrelated parent directories named code.
Select an exact-size code calibration dataset:
- datasets <= 3.6.0: OpenCodeInstruct 50% and GitHub Code Clean 50%.
- datasets > 3.6.0: OpenCodeInstruct only, because script-based GitHub Code Clean is no longer supported.
- Enable cross-sample concatenation for OpenCodeInstruct so short examples can form fixed-length calibration sequences.
Apply automatic selection only to pure-text LLMs that require calibration.
Preserve explicit user dataset choices, including both NeelNanda/pile-10k and its pile-10k alias.
- If dataset / --dataset is omitted, normal models continue to use Pile-10k, while detected code models use the code-oriented calibration dataset.
- If the user explicitly specifies any dataset, that choice is left unchanged.
Keep the existing diffusion default behavior (coco2014) unchanged.

Type of Change

New feature

Related Issues

Fixes or relates to ##1986

Checklist Before Submitting

My code has been tested locally.
Documentation has been updated as needed.
New or updated tests are included where applicable.
The CUDA CI has passed. You can trigger it by commenting /azp run Unit-Test-CUDA-AutoRound.

AutoRoundBot · 2026-07-31T01:33:38Z

/azp run Unit-Test-CUDA-AutoRound

azure-pipelines · 2026-07-31T01:33:49Z

Azure Pipelines successfully started running 1 pipeline(s).

wenhuach21 · 2026-07-31T07:13:19Z

+    "codestral",
+    "deepseekcoder",
+    "granitecode",
+    "magicoder",


Is it still doable if the user wants to use Pile-10k for a code model?

fixed the issue.
added a dataset_was_explicitly_set flag, changed the dataset default to None across all entry points, and then selected the appropriate default dataset only after determining whether the user had explicitly specified one.

for more information, see https://pre-commit.ci

AutoRoundBot · 2026-07-31T08:44:38Z

/azp run Unit-Test-CUDA-AutoRound

azure-pipelines · 2026-07-31T08:44:47Z

Azure Pipelines successfully started running 1 pipeline(s).

wenhuach21 · 2026-08-03T02:29:21Z

        quant_lm_head: bool = False,
        to_quant_block_names: Optional[Union[str, list[str]]] = None,
-        dataset: Union[str, list, tuple, torch.utils.data.DataLoader] = "NeelNanda/pile-10k",
+        dataset: Optional[Union[str, list, tuple, torch.utils.data.DataLoader]] = None,


I think the API entry of AutoRound is not this one, so you need to change auto_round/autoround.py /AutoRound as well
@n1ck-guo is it possible to merge your pr ASAP, the different entries confuse developers

Already addressed in commit 9961d8e. The public AutoRound API in auto_round/autoround.py now defaults dataset to None and forwards it unchanged, allowing the downstream orchestrator to distinguish an omitted dataset from an explicit user choice.

There is no conflict， this is the correct modification.

wenhuach21 · 2026-08-03T02:30:35Z

-        self.dataset = dataset
-        if self.dataset is None:
-            self.dataset = "NeelNanda/pile-10k"
+        self.dataset_was_explicitly_set = dataset is not None


based on the code, there is no need to save dataset_was_explicitly_set to self

Fixed. dataset_was_explicitly_set is now a local variable because it is only needed during initialization and does not need to be stored on self.

wenhuach21 · 2026-08-03T02:36:52Z

+
+        datasets_version = datasets.__version__
+    parsed_version = Version(str(datasets_version))
+    sources = [("opencode-instruct:concat=true", 50), ("github-code-clean", 40), ("mbpp:split=train:concat=true", 10)]


is there any data to support this?

Updated based on our discussion. MBPP has been removed from the automatically selected mixture. For datasets <= 3.6.0, OpenCodeInstruct and GitHub Code Clean are now selected with a 50/50 ratio. For newer datasets versions, the code falls back to OpenCodeInstruct only because script-based GitHub Code Clean is no longer supported.

wenhuach21 · 2026-08-03T02:39:09Z

+    sources = [("opencode-instruct:concat=true", 50), ("github-code-clean", 40), ("mbpp:split=train:concat=true", 10)]
+    if parsed_version > _GITHUB_CODE_CLEAN_MAX_DATASETS_VERSION:
+        sources = [source for source in sources if source[0] != "github-code-clean"]
+        logger.warning(


warning_once

Fixed. Changed logger.warning() to logger.warning_once() to avoid repeatedly reporting the same datasets-version compatibility warning when multiple AutoRound instances are created.

AutoRoundBot · 2026-08-03T06:43:37Z

/azp run Unit-Test-CUDA-AutoRound

azure-pipelines · 2026-08-03T06:43:48Z

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: changwangss <chang1.wang@intel.com>

n1ck-guo

LGTM

n1ck-guo · 2026-08-03T08:12:10Z

        quant_lm_head: bool = False,
        to_quant_block_names: Optional[Union[str, list[str]]] = None,
-        dataset: Union[str, list, tuple, torch.utils.data.DataLoader] = "NeelNanda/pile-10k",
+        dataset: Optional[Union[str, list, tuple, torch.utils.data.DataLoader]] = None,


There is no conflict， this is the correct modification.

AutoRoundBot · 2026-08-03T08:38:18Z

/azp run Unit-Test-CUDA-AutoRound

azure-pipelines · 2026-08-03T08:38:28Z

Azure Pipelines successfully started running 1 pipeline(s).

changwangss added 2 commits July 30, 2026 22:58

feat: auto-select code calibration datasets

497e6f8

refactor: simplify code model detection

546cd73

changwangss requested a review from wenhuach21 July 31, 2026 06:28

wenhuach21 reviewed Jul 31, 2026

View reviewed changes

changwangss and others added 2 commits July 31, 2026 15:51

fix: preserve explicit calibration datasets

9961d8e

[pre-commit.ci] auto fixes from pre-commit.com hooks

829fe95

for more information, see https://pre-commit.ci

wenhuach21 reviewed Aug 3, 2026

View reviewed changes

changwangss requested a review from n1ck-guo August 3, 2026 02:36

wenhuach21 reviewed Aug 3, 2026

View reviewed changes

Merge branch 'main' into feat/auto-code-calibration-dataset-v3

ba7add0

fix: address code calibration dataset review comments

9efca15

Signed-off-by: changwangss <chang1.wang@intel.com>

n1ck-guo approved these changes Aug 3, 2026

View reviewed changes

Uh oh!

Conversation

changwangss commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of Change

Related Issues

Checklist Before Submitting

Uh oh!

AutoRoundBot commented Jul 31, 2026

Uh oh!

azure-pipelines Bot commented Jul 31, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

changwangss Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AutoRoundBot commented Jul 31, 2026

Uh oh!

azure-pipelines Bot commented Jul 31, 2026

Uh oh!

wenhuach21 Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AutoRoundBot commented Aug 3, 2026

Uh oh!

azure-pipelines Bot commented Aug 3, 2026

Uh oh!

n1ck-guo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AutoRoundBot commented Aug 3, 2026

Uh oh!

azure-pipelines Bot commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

changwangss commented Jul 30, 2026 •

edited

Loading

changwangss Jul 31, 2026 •

edited

Loading

wenhuach21 Aug 3, 2026 •

edited

Loading