Metadata Builder
================

The Metadata Builder converts a dataset's structure, filenames, existing
sidecars, and readable image headers into a transparent set of metadata
inventories, BIDS-oriented templates, validation summaries, and review tables.

It can be used after creating a new dataset in NIM Studio or independently to
inspect and curate an existing raw, derivative, or study-level dataset.

The Metadata Builder does not assume that automatically inferred values are
complete or scientifically correct. All generated modality templates are marked
for manual review, and missing required or recommended fields remain visible
before export.

Overview
--------

The Metadata Builder scans a selected dataset recursively and summarizes its
participants, sessions, files, modalities, tasks, acquisitions, runs, suffixes,
existing JSON files, NIfTI files, and readable DICOM metadata.

During the scan, NIM Studio combines information from:

* Folder structure
* BIDS-style filenames
* Existing sidecar JSON files
* NIfTI headers, when ``nibabel`` is installed
* DICOM headers, when ``pydicom`` is installed

The resulting metadata are associated with their detected source so that users
can distinguish values inferred from filenames, folder organization, existing
JSON files, NIfTI headers, and DICOM headers. :contentReference[oaicite:0]{index=0}

Supported Dataset Contexts
--------------------------

The Metadata Builder can scan:

* Raw BIDS datasets
* Derivative datasets
* Study-level dataset roots
* Partially organized BIDS-like datasets
* Existing datasets created outside NIM Studio

The selected root is classified as ``raw``, ``derivative``, or ``study`` using
its directory context and detected contents. For derivative datasets, the
application also pre-fills the pipeline name using the selected folder name
when no value has been entered. :contentReference[oaicite:1]{index=1}

Dataset Description
-------------------

The module provides an editable interface for constructing
``dataset_description.json``.

Supported fields include:

Required fields
^^^^^^^^^^^^^^^

* ``Name``
* ``BIDSVersion``

Dataset type
^^^^^^^^^^^^

* ``raw``
* ``derivative``
* ``study``

Recommended and optional fields
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* ``License``
* ``Authors``
* ``Keywords``
* ``Acknowledgements``
* ``HowToAcknowledge``
* ``Funding``
* ``EthicsApprovals``
* ``ReferencesAndLinks``
* ``DatasetDOI``

Derivative provenance fields
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

For derivative datasets, users can document:

* ``GeneratedBy.Name``
* Pipeline version
* Pipeline or processing description
* Source dataset URL
* Source dataset DOI
* Source dataset version

A derivative dataset cannot save a selected ``dataset_description.json`` unless
``GeneratedBy.Name`` has been provided. :contentReference[oaicite:2]{index=2}
:contentReference[oaicite:3]{index=3}

When a dataset has been scanned, NIM Studio may also add a structural summary
containing the number of detected participants, participant-session rows,
files, and datatypes. It records that the metadata templates were generated by
NIM Studio and still require user verification. :contentReference[oaicite:4]{index=4}

Participant Management
----------------------

The Metadata Builder creates participant-level summaries from detected
``sub-<label>`` folders and filenames.

The generated ``participants.tsv`` can include:

* Participant identifier
* Detected sessions
* Detected datatypes or modalities
* Detected tasks
* Detected acquisition labels
* Detected runs
* Number of files associated with each participant

Researchers may also define custom participant columns, such as:

* ``age``
* ``sex``
* ``diagnosis``
* ``group``
* ``site``

Custom columns are created as empty fields for later completion. NIM Studio does
not infer clinical or demographic values from the dataset. :contentReference[oaicite:5]{index=5}

The accompanying ``participants.json`` documents the meaning of each generated
column, including researcher-defined variables.

Session Management
------------------

When sessions are detected, the module can generate ``sessions.tsv`` and
``sessions.json``.

Each participant-session row may contain:

* Participant identifier
* Session identifier
* Datatypes detected in that session
* Tasks detected in that session
* Number of files detected

The current implementation generates a combined session table at the selected
output root. It does not automatically create separate BIDS
``sub-<label>_sessions.tsv`` files within every participant directory.

Scan Inventory
--------------

The generated ``scans.tsv`` acts as a dataset-wide file inventory.

For each scanned file it may include:

* Relative filename
* Participant identifier
* Session identifier
* Datatype
* Suffix
* Task
* Acquisition
* Run
* Description
* Space
* Extension
* Whether an existing JSON sidecar was detected
* Missing required metadata fields
* Manual-review status

The corresponding ``scans.json`` describes these columns.

This inventory is broader than the standard participant-level BIDS
``scans.tsv`` convention and should therefore be treated as a NIM Studio
dataset audit and metadata-curation table unless it is manually adapted to a
specific BIDS scope. :contentReference[oaicite:6]{index=6}

Header Extraction
-----------------

NIfTI Metadata
^^^^^^^^^^^^^^

When ``nibabel`` is available, NIM Studio attempts to read:

* Image dimensions
* Voxel size
* Temporal zoom as a proposed ``RepetitionTime``
* NIfTI data type

Header extraction is best-effort. Files that cannot be read are skipped without
preventing the remainder of the scan. :contentReference[oaicite:7]{index=7}

DICOM Metadata
^^^^^^^^^^^^^^

When ``pydicom`` is available, NIM Studio reads DICOM headers without loading
pixel data and maps available fields into BIDS-oriented names.

Examples include:

* Manufacturer
* Scanner model
* Device serial number
* Software version
* Magnetic-field strength
* Coil name
* Institution information
* Sequence and protocol information
* Repetition time
* Echo time
* Inversion time
* Flip angle
* Pixel bandwidth
* Slice thickness
* Image rows and columns

Limited PET-oriented fields may also be extracted when they are available,
including tracer name, injected radioactivity, radionuclide half-life, and
start time. Header availability varies across vendors and export formats.
:contentReference[oaicite:8]{index=8}

Existing Sidecars
-----------------

When a matching JSON sidecar exists, NIM Studio reads it and incorporates its
values into the metadata record.

The current merge order is:

#. Filename-derived metadata
#. NIfTI-header metadata
#. DICOM-header metadata
#. Existing JSON metadata

Later sources replace earlier values when the same field is present.
Consequently, existing sidecar values take precedence over automatically
inferred header or filename values. :contentReference[oaicite:9]{index=9}
:contentReference[oaicite:10]{index=10}

Modality Sidecar Templates
--------------------------

The Metadata Builder groups detected files by likely sidecar scope and creates
JSON templates such as:

.. code-block:: text

   T1w.json
   task-rest_bold.json
   task-nback_events.json
   dwi.json
   eeg.json

Templates are populated from representative metadata values detected across
the dataset.

Each generated template contains a ``NIMStudioMetadataStatus`` section
recording:

* That manual review is required
* Detected datatype and suffix
* Number of files represented
* Required fields
* Recommended fields
* Missing required fields
* Missing recommended fields
* Sources of populated values

These templates are review aids. They are not automatically certified as
complete BIDS sidecars. :contentReference[oaicite:11]{index=11}

Validation Summary
------------------

NIM Studio compares detected metadata against an internal registry of selected
required and recommended fields.

The current registry includes configurations for several common modalities,
including:

* Anatomical MRI
* Functional MRI and SBRef
* Diffusion MRI
* Field maps
* ASL
* PET
* EEG
* MEG
* iEEG
* Physiological recordings
* MR spectroscopy

The validation view reports how often required and recommended fields are
missing across scanned files. Missing recommended fields do not prevent export.
Missing required modality fields trigger a warning, but users may still export
the files explicitly as review templates. :contentReference[oaicite:12]{index=12}
:contentReference[oaicite:13]{index=13}

This is a targeted metadata completeness assessment based on the NIM Studio
registry. It is not a complete implementation of the official BIDS validation
rules.

Metadata Master Table
---------------------

The module can generate a merged metadata table in CSV or XLSX format.

Each row represents a scanned file and may contain:

* Dataset root and dataset type
* Participant and session identifiers
* BIDS entities
* Filename and relative path
* Existing-sidecar status
* Scanner and acquisition information
* Timing parameters
* Image dimensions and voxel size
* PET metadata
* Metadata source information
* Required and recommended fields
* Missing fields
* Manual-review status

The table provides a centralized metadata inventory suitable for review,
harmonization, quality control, and preparation for later curation.
:contentReference[oaicite:14]{index=14}

Available Outputs
-----------------

Users can preview and selectively save:

* ``dataset_description.json``
* ``participants.tsv``
* ``participants.json``
* ``sessions.tsv``
* ``sessions.json``
* ``scans.tsv``
* ``scans.json``
* Modality sidecar JSON templates
* ``metadata_master.csv``
* ``metadata_master.xlsx``

XLSX export requires ``openpyxl``. NIfTI and DICOM header extraction require
``nibabel`` and ``pydicom`` respectively. If these optional packages are not
installed, the remaining metadata functions can still be used.

Before saving, users choose the destination directory. Existing files with the
same names are written to that location, so users should inspect the selected
folder and retain backups before exporting into an existing dataset.
:contentReference[oaicite:15]{index=15}

Recommended Workflow for a New Project
--------------------------------------

For a project being created from the beginning, the recommended sequence is:

#. **Create the research project**

   Use the Project Builder to establish the complete project structure,
   including governance, design, code, data, analysis, outputs, and manuscript
   branches.

#. **Create the dataset structure**

   Use the Dataset Builder to define cohorts, participants, sessions,
   modalities, BIDS components, and custom extensions.

#. **Add or import source data**

   Preserve an untouched source copy. Store vendor exports and non-curated
   material under an appropriate ``sourcedata`` or protected source location.

#. **Transform files when required**

   Use the BIDS Transformer when existing files require reviewed routing,
   organization, or BIDS-oriented renaming.

#. **Scan the resulting dataset**

   Open the Metadata Builder and select the raw BIDS root, derivative root, or
   relevant study dataset.

#. **Complete dataset-level information**

   Enter the dataset name, BIDS version, dataset type, authors, licence,
   funding, ethics information, references, DOI, and acknowledgement
   instructions.

#. **Review participant and session summaries**

   Verify detected participant identifiers, sessions, modalities, tasks,
   acquisitions, and file counts. Add custom participant columns where needed.

#. **Review header-derived metadata**

   Inspect NIfTI and DICOM values and confirm that timing, scanner, sequence,
   PET, and acquisition parameters are correct.

#. **Review validation findings**

   Resolve required metadata gaps and assess missing recommended fields.

#. **Preview every selected output**

   Inspect the dataset description, participant tables, session tables, scan
   inventory, modality templates, and metadata master table before saving.

#. **Save metadata to the appropriate scope**

   Ensure that root-level files are saved at the dataset root and that modality
   templates are placed at the correct inheritance level for the intended
   files.

#. **Run an official BIDS validation**

   Use an official BIDS validator after the generated files have been reviewed
   and placed correctly.

#. **Assess the completed dataset**

   Use the Data Management Dashboard to review metadata coverage, dataset
   health, BIDS-like organization, and FAIR readiness.

Recommended Workflow for an Existing Project
--------------------------------------------

For an existing dataset or research project, the Metadata Builder can be used
independently.

#. **Create a verified backup**

   Work from a protected copy during beta testing, particularly when exporting
   files into the dataset itself.

#. **Identify the correct scan root**

   Select the raw dataset root, a derivative dataset, or the specific study
   directory that should be inventoried. Avoid selecting an overly broad
   institutional storage root.

#. **Scan before changing the dataset**

   Run the metadata scan and review the detected dataset type, participants,
   sessions, datatypes, existing JSON files, and readable headers.

#. **Inspect the metadata master table**

   Use the merged inventory to identify inconsistent naming, missing sidecars,
   incomplete fields, repeated scanner values, and modality-specific gaps.

#. **Review existing JSON precedence**

   Confirm that existing sidecars contain correct values, because they take
   precedence over NIfTI-, DICOM-, and filename-derived candidates.

#. **Complete dataset-level fields**

   Populate or correct ``dataset_description.json`` information. For derivative
   datasets, document the generating pipeline and source dataset.

#. **Treat generated sidecars as templates**

   Compare proposed modality templates against acquisition protocols, scanner
   exports, task documentation, and existing sidecars before using them.

#. **Export to a review directory first**

   For important existing datasets, save generated metadata into a separate
   review folder rather than immediately writing over dataset files.

#. **Merge changes deliberately**

   Move only approved files into the dataset and preserve previous versions
   through version control, checksums, or a documented archive.

#. **Validate and audit**

   Run an official BIDS validator, followed by the NIM Studio Data Management
   Dashboard and Duplicate Audit where appropriate.

What the Metadata Builder Does Not Do
-------------------------------------

The current beta does not:

* Guarantee that a dataset is fully BIDS-compliant.
* Replace the official BIDS Validator.
* Convert DICOM images into NIfTI.
* Determine scientifically correct acquisition parameters when headers are
  absent or ambiguous.
* Resolve every BIDS inheritance relationship automatically.
* Determine the correct scope for every generated sidecar.
* Generate complete ``events.tsv`` content from task logs.
* Infer demographic, clinical, or protected participant information.
* Validate consent, ethics approval, anonymization, or GDPR compliance.
* Remove identifying DICOM fields.
* Guarantee that a NIfTI temporal zoom represents the intended BIDS
  ``RepetitionTime`` in every case.
* Harmonize conflicting metadata across scanners, sites, sessions, or cohorts.
* Preserve existing files automatically when exporting to the same location.
* Make missing required fields scientifically valid by inserting placeholders.

Beta Safety Notice
------------------

Generated metadata should initially be treated as draft documentation and
curation material.

During the beta phase, users should:

* Test the module on non-critical or backed-up data.
* Review every inferred field.
* Compare header-derived values with acquisition protocols.
* Keep identifying and confidential metadata within approved environments.
* Export into a separate review directory when working with established
  datasets.
* Validate the final dataset independently before analysis, publication, or
  sharing.