Skip to content

FEAT Add Turkish Prompt-Injection dataset loader - #2365

Draft
Enes Deniz (3nesdeniz) wants to merge 1 commit into
microsoft:mainfrom
3nesdeniz:feat/turkish-prompt-injection-dataset
Draft

FEAT Add Turkish Prompt-Injection dataset loader#2365
Enes Deniz (3nesdeniz) wants to merge 1 commit into
microsoft:mainfrom
3nesdeniz:feat/turkish-prompt-injection-dataset

Conversation

@3nesdeniz

Copy link
Copy Markdown

Adds a remote SeedDatasetProvider loader for the open, CC BY 4.0 Turkish Prompt-Injection dataset (3nesdeniz/turkish-prompt-injection-1k), pinned to an immutable HF revision.

Turkish-native prompt-injection coverage is scarce in the existing seed datasets. This loader adds paired benign/attack Turkish examples (instruction override & system-prompt extraction, jailbreak personas, obfuscation & data exfiltration, and agentic tool abuse) with Turkish morphological variation. The benign hard negatives share surface vocabulary with the attacks while being fully legitimate, so false-positive behavior can be measured directly.

Changes:

  • _TurkishPromptInjectionDataset with typed label (attack/benign/all), split, and attack_families filtering, pinned to an immutable HF revision, with schema and label-dependent validation. Follows the established _RemoteDatasetLoader pattern.
  • Registers the loader and its enums for discovery.
  • Adds 11 focused unit tests (mocked HF fetch + validation/error cases, including the type(label) is int bool-rejection guard).
  • Adds a bibliography/reference entry.

Testing:

  • pytest of the new test file: 11 passed.
  • Live-loaded the published test split end-to-end (79 attacks / 34 benign) and verified split and family filters against real data.
  • black, isort, and flake8 clean.

Labels describe prompt-injection decisions rather than content harms, so they are retained in seed metadata and harm_categories is intentionally empty (matching the agentic boundary-pairs loader convention).

Add a remote SeedDatasetProvider for the CC BY 4.0 Turkish Prompt-Injection
dataset (3nesdeniz/turkish-prompt-injection-1k), pinned to an immutable HF
revision. Supports label (attack/benign/all), split, and Turkish attack-family
filtering, with schema and label-dependent validation. Registers the loader and
enums for discovery, adds focused unit tests, and a bibliography entry.
@3nesdeniz

Copy link
Copy Markdown
Author

Converting this to a draft and gating it on the design discussion in #2379, per the catalog-direction process Adrian Gavrila (@adrian-gavrila) outlined on #2175 — I'd rather get maintainer direction before asking for review, instead of repeating the build-before-discuss sequencing. The implementation here is complete and tested (unit suite + a live load of the published test split) and will be ready to finalize if/when the direction on #2379 lands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant