Instrument Data To Allotrope

What This Skill Does

• This skill converts instrument files into the standardized Allotrope Simple Model (ASM) format, facilitating LIMS upload, data lake integration, and data engineering handoff.
• It automates schema transformations, parses instrument outputs, and generates production-ready code, streamlining data engineering tasks and ensuring data standardization.

When to Use

• Convert instrument data to ASM format.
• Prepare data for LIMS upload.
• Integrate instrument data into data lakes.
• Automate schema transformations.
• Parse proprietary instrument formats.
• Generate Python parser code for data engineers.

Key Features

• Detects instrument type from file contents.
• Parses files using allotropy library.
• Generates ASM JSON output.
• Creates flattened CSV output.
• Provides Python parser code for handoff.
• Offers guidance on field classification.

Manual Installation

Workflow Overview

Detect instrument type from file contents (auto-detect or user-specified) Parse file using allotropy library (native) or flexible fallback parser Generate outputs:

• ASM JSON (full semantic structure)
• Flattened CSV (2D tabular format)
• Python parser code (for data engineer handoff) Deliver files with summary and usage instructions

When Uncertain: If you're unsure how to map a field to ASM (e.g., is this raw data or calculated? device setting or environmental condition?), ask the user for clarification. Refer to references/field_classification_guide.md for guidance, but when ambiguity remains, confirm with the user rather than guessing.

Quick Start

Install requirements first

pip install allotropy pandas openpyxl pdfplumber --break-system-packages

Core conversion from allotropy.parser_factory import Vendor from allotropy.to_allotrope import allotrope_from_file

Convert with allotropy asm = allotrope_from_file("instrument_data.csv", Vendor. BECKMAN_VI_CELL_BLU)

Output Format Selection

ASM JSON (default) - Full semantic structure with ontology URIs Best for: LIMS systems expecting ASM, data lakes, long-term archival Validates against Allotrope schemas

Flattened CSV - 2D tabular representation Best for: Quick analysis, Excel users, systems without JSON support Each measurement becomes one row with metadata repeated

Both - Generate both formats for maximum flexibility

Calculated Data Handling

IMPORTANT: Separate raw measurements from calculated/derived values.

Raw data → measurement-document (direct instrument readings) Calculated data → calculated-data-aggregate-document (derived values)

Calculated values MUST include traceability via data-source-aggregate-document:

"calculated-data-aggregate-document": { "calculated-data-document": [{ "calculated-data-identifier": "SAMPLE_B1_DIN_001", "calculated-data-name": "DNA integrity number", "calculated-result": {"value": 9.5, "unit": "(unitless)"}, "data-source-aggregate-document": { "data-source-document": [{ "data-source-identifier": "SAMPLE_B1_MEASUREMENT", "data-source-feature": "electrophoresis trace" }] } }] }

Common calculated fields by instrument type:

See references/field_classification_guide.md for detailed guidance on raw vs. calculated classification.

Validation

Always validate ASM output before delivering to the user:

python scripts/validate_asm.py output.json python scripts/validate_asm.py output.json --reference known_good.json # Compare to reference python scripts/validate_asm.py output.json --strict # Treat warnings as errors

Validation Rules: Based on Allotrope ASM specification (December 2024)

Source: https://gitlab.com/allotrope-public/asm

Soft Validation Approach:

• Unknown techniques, units, or sample roles generate warnings (not errors) to allow for forward compatibility.
• If Allotrope adds new values after December 2024, the validator won't block them—it will flag them for manual verification.
• Use --strict mode to treat warnings as errors if you need stricter validation.

What it checks: Correct technique selection (e.g., multi-analyte profiling vs plate reader) Field naming conventions (space-separated, not hyphenated) Calculated data has traceability (data-source-aggregate-document) Unique identifiers exist for measurements and calculated values Required metadata present Valid units and sample roles (with soft validation for unknown values)

Supported Instruments

See references/supported_instruments.md for complete list. Key instruments:

Detection & Parsing Strategy

Tier 1: Native allotropy parsing (PREFERRED)

Always try allotropy first. Check available vendors directly:

from allotropy.parser_factory import Vendor

List all supported vendors for v in Vendor: print(f"{v.name}")

Common vendors: AGILENT_TAPESTATION_ANALYSIS (for TapeStation XML) BECKMAN_VI_CELL_BLU THERMO_FISHER_NANODROP_EIGHT MOLDEV_SOFTMAX_PRO APPBIO_QUANTSTUDIO ... many more

When the user provides a file, check if allotropy supports it before falling back to manual parsing. The scripts/convert_to_asm.py auto-detection only covers a subset of allotropy vendors.

Tier 2: Flexible fallback parsing

Only use if allotropy doesn't support the instrument. This fallback: Does NOT generate calculated-data-aggregate-document Does NOT include full traceability Produces simplified ASM structure

Use flexible parser with:

Tier 3: PDF extraction

For PDF-only files, extract tables using pdfplumber, then apply Tier 2 parsing.

Pre-Parsing Checklist

Before writing a custom parser, ALWAYS:

Check if allotropy supports it - Use native parser if available Find a reference ASM file - Check references/examples/ or ask user Review instrument-specific guide - Check references/instrument_guides/ Validate against reference - Run validate_asm.py --reference

Common Mistakes to Avoid