Refactor streamline docs - #30
Conversation
… of human specific)
There was a problem hiding this comment.
These images look great! Is there an SVG version that can be stored for easier editing (in case we want to do any minor adjustments in the future)?
There was a problem hiding this comment.
Or if you used slides, or some other software to create otherwise - the 'source' files for the images
|
|
||
| EMBER is planning development of one analytical platform: | ||
|
|
||
| - **EMBER-Hearth**: enables sandboxed analyses in the cloud on both of these platforms. |
There was a problem hiding this comment.
| - **EMBER-Hearth**: enables sandboxed analyses in the cloud on both of these platforms. | |
| - **EMBER-Hearth**: enables analysis pipelines such as spike sorting and pose estimation inference in the cloud through both of these platforms. |
Suggest removing the term 'sandbox' since it can mean different things depending on context (e.g., dandi.sandbox.emberarchive.org would not carry the meaning nor be the place to keep such things)
Suggesting these two example analyses since they are the currently active planning ones that we believe we can actually offer (note no current public guarantee of end-to-end, pure web aspect such as manual annotations - that may continue to take longer than forseeable future depending on when external packages are made public)
| 1. Standardize your data. | ||
| 2. Create an account for each appropriate storage platform using instructions in the [User Guide](../user-guide/index.md). |
There was a problem hiding this comment.
| 1. Standardize your data. | |
| 2. Create an account for each appropriate storage platform using instructions in the [User Guide](../user-guide/index.md). | |
| 1. Create an account for each appropriate storage platform using instructions in the [User Guide](../user-guide/index.md). | |
| 2. Standardize your data. |
Recommend starting with this since registration is low-hanging fruit and takes an admin 1-2 days to get around to vetting and approving
| ### Standardizing Data | ||
| Data standardization is required for uploading to either of the EMBER storage platforms to enable easy and accessible reuse of data by secondary users who were not involved in the original study that created the data. | ||
|
|
||
| The general approach is to organize your data using the BIDS schema and convert relevant data into NWB. |
There was a problem hiding this comment.
| The general approach is to organize your data using the BIDS schema and convert relevant data into NWB. | |
| The general approach is to organize your dataset using the BIDS schema and convert relevant file contents per session into NWB. |
Some verbal clarity on the usage of BIDS vs. NWB (consistent with the explicit descriptions just beneath)
|
|
||
| The general approach is to organize your data using the BIDS schema and convert relevant data into NWB. | ||
|
|
||
| BIDS broadly describes how to rename files and folders and organize them into a standardized directory. It als requires associated metadata files to help explain the experiment and data. |
There was a problem hiding this comment.
| BIDS broadly describes how to rename files and folders and organize them into a standardized directory. It als requires associated metadata files to help explain the experiment and data. | |
| BIDS broadly describes how to rename files and folders and organize them into a standardized directory. It also requires associated metadata files to help explain the experiment and data. |
There was a problem hiding this comment.
Hmmm seems we might need codespell!
There was a problem hiding this comment.
🎺 @yarikoptic can you set them up with a basic codespell action?
|
|
||
| More specific instructions for how to standardize data can be found in the [User Guide](../user-guide/index.md). | ||
|
|
||
| Feel free to use AI tools like ChatGPT and Codex to help you convert your data according to the standards. |
There was a problem hiding this comment.
| Feel free to use AI tools like ChatGPT and Codex to help you convert your data according to the standards. | |
| Feel free to use AI tools like ChatGPT, Codex, or Claude to help you convert your data according to the standards. |
Or are you not even allowed to use the dreaded word within the APL Brain organization XD
|
|
||
| [https://bids-specification.readthedocs.io/en/stable/modality-specific-files/magnetoencephalography.html#meg-recording-data]() | ||
|
|
||
| #### BEP032 (iEEG / microelectrode extensions) |
There was a problem hiding this comment.
| #### BEP032 (iEEG / microelectrode extensions) | |
| #### BEP032 (microelectrode extensions) |
iEEG is thankfully already completely covered! (with .nwb as well!)
|
|
||
| ### BIDS Extensions Supported | ||
|
|
||
| #### Intracranial / Microelectrode Data |
There was a problem hiding this comment.
| #### Intracranial / Microelectrode Data | |
| #### iEEG / EEG |
They wanted microelectrode to mean things like tetrodes, neuropixels, utah arrays, etc
| ## 1. Relevant Data Standards | ||
| Before formatting your dataset, familiarize yourself with the standards supported by the BBQS Program and the EMBER archive (defined by the BBQS DCAIC Data Standards Working Group): | ||
|
|
||
| [https://docs.google.com/document/d/1vIJ01La9G76FfGywS3IbG4o1GR4qquS31MoJ2B4_4os/edit?usp=sharing]() |
There was a problem hiding this comment.
| [https://docs.google.com/document/d/1vIJ01La9G76FfGywS3IbG4o1GR4qquS31MoJ2B4_4os/edit?usp=sharing]() | |
| [Guidelines for BBQS data standards](https://docs.google.com/document/d/1vIJ01La9G76FfGywS3IbG4o1GR4qquS31MoJ2B4_4os/edit?usp=sharing) |
There was a problem hiding this comment.
The other URLs are nice looking, but this one stands out
| !!! warning "`ndx-multisubjects`" | ||
| This extension is undergoing development, and currently will not pass the validator. You can still use and upload this file to EMBER-DANDI using `--validation skip` argument until the validators are updated or this extension is incorporated into core NWB. |
There was a problem hiding this comment.
| !!! warning "`ndx-multisubjects`" | |
| This extension is undergoing development, and currently will not pass the validator. You can still use and upload this file to EMBER-DANDI using `--validation skip` argument until the validators are updated or this extension is incorporated into core NWB. |
Thinking of these as docs more for public consumption (as BBQS would be handled by data ingest task force), this is more of a detail I would leave to an issue thread that can be tracked, described, and closed over time
Also runs the risk of forgetting its here in this doc after its been fixed
| 1. Standardize your data. | ||
| 2. Create an account for each appropriate storage platform using instructions in the [User Guide](../user-guide/index.md). | ||
| 3. Upload the standardized data to the appropriate storage platform(s) using instructions in the [User Guide](../user-guide/index.md). | ||
|
|
There was a problem hiding this comment.
| !!! info "Important" | |
| If you are a member of the **BBQS consortium**, please reach out to the Data Ingest Task Force by starting a discussion on the [BBQS Helpdesk](https://github.com/orgs/brain-bbqs/discussions) to request a streamlined pathway for uploading and data standardization. |
There was a problem hiding this comment.
While I would not go into detail about the white glove service here (since this is more for public-facing docs), still useful to leave a note here in case a BBQS member does start looking into these instructions
| sub-02 34 F | ||
| ``` | ||
|
|
||
| #### 3.3 iEEG Metadata (BEP032) |
There was a problem hiding this comment.
| #### 3.3 iEEG Metadata (BEP032) | |
| #### 3.3 iEEG Metadata |
BEP032 doesn't touch iEEG at all, though the two can share many commonalities; there are various metadata entries that differ between the two (as various things such as coordinate systems and probe files make more sense for deeply implanted devices as opposed to iEEG
| ```json | ||
| { | ||
| "SamplingFrequency": 48000, | ||
| "Microphone": "Shure SM7B" |
There was a problem hiding this comment.
If you could do me a favor and check thoroughly against BEP047 to confirm - I had thought this might be called 'Device' and might have an extra separate field for the 'DeviceManufacturerer' or something along those lines
|
|
||
| [https://bids.neuroimaging.io/tools/validator.html]() | ||
|
|
||
| We do not recommend using the online data validator for datasets containing PHI/PII. |
There was a problem hiding this comment.
May I ask why? It is a static client side web app after all, just like our other HEARTH apps - granted there can be 'extra' details for true HIPAA compliance such as only going to the website on a secure computer and network (usually at the hospital in question)
| deno run -ERWN jsr:@bids/validator <dataset> | ||
| ``` | ||
|
|
||
| When using an extension, you must use the flag `--schema` with the link to the pull request specific to the extension. |
There was a problem hiding this comment.
| When using an extension, you must use the flag `--schema` with the link to the pull request specific to the extension. | |
| When using a BEP standard, you must use the flag `--schema` with the link to the pull request specific to the schema. |
Strictly speaking BIDS does not have 'extensions' like NWB does - though they have indicated they may want to enable that in the future, doesn't exist today
| [https://bids-specification.readthedocs.io/en/stable/modality-specific-files/magnetoencephalography.html#meg-recording-data]() | ||
| [https://bids-specification.readthedocs.io/en/stable/modality-specific-files/magnetoencephalography.html#meg-recording-data](https://bids-specification.readthedocs.io/en/stable/modality-specific-files/magnetoencephalography.html#meg-recording-data) | ||
|
|
||
| #### BEP032 (iEEG / microelectrode extensions) |
There was a problem hiding this comment.
| #### BEP032 (iEEG / microelectrode extensions) | |
| #### BEP032 (microelectrode extension) |
|
|
||
| ### BIDS Extensions Supported | ||
|
|
||
| #### Intracranial / Microelectrode Data |
There was a problem hiding this comment.
| #### Intracranial / Microelectrode Data | |
| #### iEEG / EEG |
| [https://bids-specification.readthedocs.io/en/stable/modality-specific-files/magnetoencephalography.html#meg-recording-data]() | ||
| [https://bids-specification.readthedocs.io/en/stable/modality-specific-files/magnetoencephalography.html#meg-recording-data](https://bids-specification.readthedocs.io/en/stable/modality-specific-files/magnetoencephalography.html#meg-recording-data) | ||
|
|
||
| #### BEP032 (iEEG / microelectrode extensions) |
There was a problem hiding this comment.
| #### BEP032 (iEEG / microelectrode extensions) | |
| #### BEP032 (microelectrode extension) |
| !!! warning "`ndx-multisubjects`" | ||
| This extension is undergoing development, and currently will not pass the validator. You can still use and upload this file to EMBER-DANDI using `--validation skip` argument until the validators are updated or this extension is incorporated into core NWB. |
There was a problem hiding this comment.
| !!! warning "`ndx-multisubjects`" | |
| This extension is undergoing development, and currently will not pass the validator. You can still use and upload this file to EMBER-DANDI using `--validation skip` argument until the validators are updated or this extension is incorporated into core NWB. |
see https://github.com/aplbrain/BBQS-EMBER-docs/pull/30/changes#r3927280571
| To use EMBERvault, you must validate your dataset locally. The EMBERvault system is intended to store standardized data for reuse but does not currently format and validate data in-platform. | ||
|
|
||
| Data uploaded to either EMBERVault or EMBER-DANDI must be properly standardized. | ||
| ***Can this section be updated to jsut use the dandi validator?*** |
| Once your user account and project have been set up, you are ready to upload your data to EMBER-Vault! | ||
| # Data Upload | ||
| Once your user account and project have been set up, you are ready to upload your [***validated***](../user-guide/data-standardization.md#5-validation-required-before-upload) data to EMBER-Vault! | ||
| To use EMBERvault, you must validate your dataset locally. The EMBERvault system is intended to store standardized data for reuse but does not currently format and validate data in-platform. |
There was a problem hiding this comment.
| To use EMBERvault, you must validate your dataset locally. The EMBERvault system is intended to store standardized data for reuse but does not currently format and validate data in-platform. | |
| To use EMBER-Vault, you must validate your dataset locally. The EMBER-Vault system is intended to store standardized data for reuse but does not currently format and validate data in-platform. |
Should it not be EMBER-Vault to match EMBER-DANDI, EMBER-HEARTH, etc.?
There was a problem hiding this comment.
Yeah I guess it looks that way everywhere else
|
|
||
| The use of AI tools to write conversion scripts is encouraged. Once you have decided on your conversion strategy, prompt your favorite AI tool with this strategy, pointing it at your unstandardized dataset. | ||
|
|
||
| To help you get started, we've published some of the prompts and conversion scripts we've put together for certain datasets (link) |
| !!! success "EMBER-verified" | ||
| EMBER-DANDI datasets that meet this second quality check will be denoted as "EMBER Verified". |
There was a problem hiding this comment.
| !!! success "EMBER-verified" | |
| EMBER-DANDI datasets that meet this second quality check will be denoted as "EMBER Verified". | |
| !!! success "EMBER-Verified" | |
| EMBER-DANDI datasets that meet this second quality check will be denoted as "EMBER-Verified". |
Ooo this sounds nice - has anyone worked on the branding designs yet? Badges, stickers, trophies?
yarikoptic
left a comment
yarikoptic
left a comment
There was a problem hiding this comment.
No non-descript AI use for "conversion scripts" encouragement please!
|
|
||
| ### AI tools | ||
|
|
||
| The use of AI tools to write conversion scripts is encouraged. Once you have decided on your conversion strategy, prompt your favorite AI tool with this strategy, pointing it at your unstandardized dataset. |
There was a problem hiding this comment.
oh, please do not !!! FWIW: I promise to adamantly breath into the backs of those who will be pushing such encouragement -- it is wrong! It can potentially result in tons of hallucinated metadata or data, forgotten or simply wrong conversions, etc
For the best use of AI, we must promote their proper use! (in skills, docs, harnesses) and overall such instructions should be not much different between "for humans" or "for AI": they all
- must know about existing conversion tools and how to use them, instead of creating their own!
- if an ad-hoc conversion script is to be developed for some particular niche case, we can recommend strategies (in my order of preference)
- if particular aspect or data modality support is missing from otherwise overall well fitting existing tool (like e.g. neuroconv) -- consider developing (with AI or "by hand") missing component and submitting as a PR following that project conventions for code organization etc. That is the best way to guarantee correctness of your code and also most efficient work as your code would become then usable by others as part of that bigger package
- if there is really no existing or fitting conversion tool, creating a dedicated ad-hoc script could be approached BUT its development should follow some well tested and established AI development routine or harness to guarantee robust and correct operation!
There was a problem hiding this comment.
I read this as proper use implied. Indeed we should have an entire separate set of guidelines for what that means, which could be cross-linked from here
| 2. reuse data for new analyses or models. | ||
|
|
||
| # What standards are accepted? | ||
| EMBER accepts two standards: BIDS and NWB. |
There was a problem hiding this comment.
that statement is wrong in its core as BIDS standard relies on number of standards internally (nifti, HED, ...) as well as NWB really and they cover different levels of standardization -- as standards group document highlights! So, many more standards "accepted". And "DANDI layout" (which is accepted by DANDI for e.g. a collection of NWB files) not even a "standard" per se.
I bet you can get a reasonable answers if you feed your AI "friend" with the document from standards WG -- the "Guidelines for BBQS data standards" (not sure I am allowed to share a URL here) and asking to get really concise, no AI slop, answers to them, and then curate/adjust. @oruebel -- can that doc migrate into some repo on github in its markdown form to simplify reuse / contributions etc.
Alternatively, would it be ok if I create such an instance using my AI brewed https://github.com/con/docflow -- and share publicly? it should even then have the comments in .json (BTW -- there is lots non addressed)
There was a problem hiding this comment.
I bet you can get a reasonable answers if you feed your AI "friend" with the document from standards WG -- the "Guidelines for BBQS data standards" (not sure I am allowed to share a URL here) and asking to get really concise, no AI slop, answers to them, and then curate/adjust. @oruebel -- can that doc migrate into some repo on github in its markdown form to simplify reuse / contributions etc.
That doc is referenced and linked already in this PR
| @@ -0,0 +1,99 @@ | |||
| # Why do I need to standardize my data? | |||
There was a problem hiding this comment.
I think all the questions should get under ## Frequently Asked Questions section and there should be some overarching # BBQS Standards section with ## Introduction pointing to the "Guidelines for BBQS data standards" developed by @oruebel et al.
There was a problem hiding this comment.
oh @CodyCBakerPhD already referenced this doc in the other file, so likely my above comment is not applicable here on restructuring but likely the file to be renamed to reflect that it is some form of FAQs?
There was a problem hiding this comment.
yeap, found it later. recommendation/instructions stand
4 participants
Have reorganized documentation according to:
There's a lot more that can be done in terms of refinement, but figured this would be a good place to start getting feedback.