Analysis Documentation Practices for Researchers

Documenting an analysis means keeping a record complete enough that someone else, or you in eight months, can see what data went in, what was done to it, why, and how to rerun it and get the same numbers. That record has four parts: the data and its provenance, the methods and assumptions, the code and software environment, and the decisions you made along the way. This guide covers what to write down, how to organize it, how to handle clinical and other sensitive data, and where agentic analytics fits into the process.

TL;DR

• Document at two levels: the project (tools, methods, collection) and the file (variable meanings, coding schemes, transformations).
• Record decisions and their rationale, not only code: exclusions, model choices, and whether each analysis was exploratory or confirmatory.
• Keep a replication folder: original data, scripts, analysis data, and a README with software requirements and step-by-step instructions.
• For patient or special category data, document the lawful basis, anonymization or pseudonymization steps and retention, and keep identifiable data out of cloud tools where rules require it.
• PlotStudio AI adds an agentic layer to your existing tools that plans, executes and records analyses so the documentation is produced as the work happens.

Table of Contents

• What “analysis documentation” actually means
• What to record: the minimum a reviewer will ask for
• The five W’s, which are really six
• A step-by-step structure a supervisor can follow
• Version control and audit trails
• Documenting clinical and sensitive-data analyses
• What to check in analysis software
• Our perspective: documentation as a by-product, not a chore
• How PlotStudio AI supports each documentation habit
• Try it on a real dataset
• A short routine for the last three months of a thesis
• FAQ
• Sources

What “analysis documentation” actually means

The phrase covers three different things, and search results mix them up.

Document analysis is a qualitative method: you study policies, reports or records as your data. It is a research technique, not a record-keeping practice.

Good Documentation Practices (GDocP) come from regulated industries. They follow the ALCOA-plus principle: data should be Attributable, Legible, Contemporaneous, Original, Accurate, Complete, Consistent, Enduring, and Available. The aim is records that stay truthful, retrievable and traceable across their lifecycle. If you work on a regulated trial, these principles apply to your analysis files too.

Documenting a research analysis is what this article is about. It borrows the spirit of ALCOA (who did what, when, on which original data) and adds what scientific reviewers need: the reasoning behind each analytical choice and a way to reproduce the output.

What to record: the minimum a reviewer will ask for

A University at Buffalo guide to documentation and metadata lists what researchers should capture: data provenance and protocols; collection, processing and analysis methods; decisions and their rationale; software, scripts and versions; version history; and variable names, labels, units, coding schemes, missing-data codes and types.

In practice, turn that into a checklist.

Data

• Where each dataset came from, who collected it, when, and under which protocol.
• A data dictionary: variable names, labels, units, types, coding schemes and the codes you use for missing values.
• The original, untouched files, stored separately from any cleaned version.
• Every modification, including anonymization. The UK Data Service asks for study-level documentation covering the purpose of data creation, content, collection methods, processing protocols, modifications and quality assurance.

Methods and assumptions

• The statistical model or test, and why it fits the design (for example, mixed-effects models for repeated measures, Cox proportional hazards for time-to-event data).
• The assumptions you checked and how: residual diagnostics, proportional hazards tests, normality checks, and what you did when they failed.
• How you handled missingness, outliers and multiple comparisons.
• Inclusion and exclusion rules for observations. A 2019 National Academies report says reports should describe methods, instruments, measurement procedures, data exclusion and inclusion decisions, statistical inference details and uncertainty.

Code and environment

The National Academies’ reproducibility report says computational reproducibility requires documenting input data, intermediate results, output data, study methods in executable form, and environment details such as operating system, hardware and library dependencies. Writing “analyzed in R” is not enough. Record the version of R or Python and of every package.

Decisions

This is the part most people skip. Note every fork: you dropped site C because of a protocol deviation, you switched from a t-test to a Mann-Whitney test after seeing skewed residuals, you added a covariate. The same National Academies report advises clarifying whether a study is exploratory or confirmatory. Label each analysis accordingly, and say what was planned in advance. The Stanford data science handbook likewise recommends planning statistical analyses beforehand and separating exploratory from confirmatory work.

A rule of thumb from Stanford’s data management guide: document more than you think is necessary.

The five W’s, which are really six

The Buffalo guide frames completeness with the five W’s, which it lists as Who, What, When, Where, Why, and How. Applied to an analysis log:

If an entry can’t answer all six, someone will eventually have to guess.

A step-by-step structure a supervisor can follow

The TIER Documentation Protocol gives a clear model: assemble a replication folder with the original data and metadata, processing and analysis scripts, the analysis data, and a README that states software requirements and gives step-by-step replication instructions.

Here is a workable version.

1. Create the folder structure on day one. Separate raw/, processed/, scripts/, output/ and docs/. Never edit files in raw/.
2. Write the analysis plan before you look at outcomes. State hypotheses, primary and secondary outcomes, models and the multiplicity strategy.
3. Script every transformation. Cleaning done by hand in a spreadsheet leaves no trace. Cleaning done in code is the record.
4. Keep a decision log. One dated line per judgment call, with the reason.
5. Capture the environment. Record software and package versions. In R, renv records package versions and dependencies.
6. Save logs and outputs together. Statistical output should live beside the code that produced it.
7. Write the README last, then test it. Ask a colleague to follow it cold. Where they stumble is where your documentation has a gap.

A 2021 PMC article on data analysis workflows describes a reproducible workflow in three phases: Explore, Refine and Produce. Your documentation should change with the phase. Exploration notes can be rough; the produce phase needs a clean, rerunnable pipeline. For more on this arc, see our guide to PhD data analysis workflow and one-command reproducible workflows.

Version control and audit trails

For researchers who code, Git handles version history and collaboration. A primer on reproducible research in R recommends Git for version control, R Markdown for dynamic documents that combine text, code and data, and renv for package versions. It is written for researchers with minimal or no coding knowledge and uses RStudio.

Regulated work raises the bar. A University Hospitals Bristol and Weston study data SOP says clinical research data changes need an authorized process and an audit trail. Sophisticated systems should log changed fields and user identities; simpler systems should keep dated, version-numbered copies that identify who made the change. The same SOP expects a live data-management plan describing data flows, with every version retained in the trial master file.

If you want to see how audit trails are framed more broadly, standards-first document audit trails give a useful view of how change records are kept.

The University of Edinburgh’s trials unit sets a clear rule in its statistical analysis SOP: analyses must be repeatable and documented with the software and version, analysis datasets, programs, log files and output files in a clearly labeled folder. It also says amending data inside statistical programs (hard coding) is generally not acceptable, and specifically not acceptable in CTIMP trials. Errors found during final analysis should be fixed through a documented database unlock or flagged in a report footnote.

That last point matters for everyone, regulated or not. A silent fix inside a script is the kind of change a reviewer can’t see and you won’t remember.

Documenting clinical and sensitive-data analyses

When the data are patient records, the documentation itself becomes part of your compliance. The rules differ by country, so check the law where your data are collected and stored. The WHO’s guidance on sharing health-related data makes the same point.

Distinguish anonymization from pseudonymization, and record which you did. WHO describes anonymization as irreversibly removing identifying variables. De-identification or pseudonymization masks identifiers or replaces them with a code whose key can enable re-identification. Your methods should say which applies, and where any key is held.

Document the lawful basis. The UK Information Commissioner’s Office says documentation for special category processing should record the data categories, the processing condition and lawful basis, and whether retention and deletion policies were followed, or why they were not. The ICO also says high-risk processing, such as large-scale processing or genetic or biometric data combined with other criteria, requires a data protection impact assessment. This is UK GDPR guidance; US researchers should check HIPAA, IRB conditions and institutional policy, and EU researchers the GDPR itself.

Describe retention. IARC’s 2024 data protection policy says teams should give clear information about the controller, purposes, transfers, data-subject rights and storage period or its criteria, and should periodically review whether retained data can become anonymous, more strongly pseudonymized or aggregated.

Plan for data you can’t share. When participant-level data can’t be public for ethical, legal or confidentiality reasons, WHO says to publish a metadata record with a persistent identifier and a data-availability statement describing the restrictions, the access-application process and reuse conditions. Write that statement early, not at submission.

If you are an NHS or hospital researcher, ask your data protection officer or information governance lead before choosing any tool that touches identifiable data.

What to check in analysis software

When evaluating tools for a privacy-sensitive, reproducible workflow, regulatory guidance on computerized systems offers a good checklist. India’s Good Documentation Practices guidance for laboratory systems asks for validation before introduction and after changes, controls for data integrity, confidentiality, access and retrieval, audit trails, and retention of both original and amended records. Translated for an analysis tool:

• Does it run on your machine, or does data leave it?
• Is the code it ran visible and exportable?
• Can you review the approach before execution?
• Are intermediate results kept, not just final charts?
• Can a collaborator open the record and see how each number was produced?

Our perspective: documentation as a by-product, not a chore

Most documentation fails for one reason: it is written after the fact, from memory, by someone who is tired. The deadline-driven reality of a PhD means the decision log gets abandoned in month two.

We built PlotStudio AI around a different idea. If an analysis is planned, executed and inspected in structured steps, the documentation can be a by-product of the work rather than a second job. In PlotStudio AI, specialized agents plan the investigation, write and run real code, inspect intermediate results and synthesize findings. Plan Mode lets you review and change the methods, assumptions and steps before anything runs, so the rationale exists before the result does. Skills encode how your lab or discipline approaches analysis, including required procedures, statistical conventions and reporting expectations, so the same conventions apply on every project.

We don’t see this as a replacement for R, Python, Stata, SPSS, SAS, Jupyter or RStudio. Those tools provide the statistical computing. PlotStudio AI is an agentic analytics layer on top that can plan, execute, inspect, validate and document complete analyses. We think it is becoming a must-have layer in the modern research toolkit, particularly for researchers who were never trained to code and are learning R under a deadline.

A caution we hold ourselves to: no tool removes your responsibility to understand the method. The point of reviewable plans and exported code is that you can check the work, and a supervisor can too.

— Aymen

How PlotStudio AI supports each documentation habit

For methods write-ups drawn from this record, see our guide on writing a methods section, and for the broader picture, research reproducibility for analysts. More reading on reproducible research is collected on our site.

Try it on a real dataset

PlotStudio AI has a free trial, and you can see plans on the pricing page. Managed Credits cost $69.99 per month, and Bring Your Own Key costs $39.99 per month. Academic users can take Bring Your Own Key at $399.90 per year on the academic page. Start with a dataset you have already analyzed, run it through Plan Mode, and compare the exported notebook with your own scripts. That test tells you more than any feature list.

A short routine for the last three months of a thesis

If the deadline is close, you don’t need a perfect system. You need a minimum viable record.

• This week: freeze the raw data, write the data dictionary, and list every analysis already run.
• Each analysis: one paragraph in the decision log with the question, method, assumptions checked and result.
• Each month: rerun everything from raw data in a clean session. If a number changes, find out why now, not at the viva.
• Before submission: write the README, record software versions, and give a colleague the folder.

For researchers who need more structure on the analysis side, our exploratory data analysis guide and PhD data analysis help cover the earlier steps.

FAQ

What are good documentation practices in clinical research?

They are record-keeping standards, usually built on ALCOA-plus, that make data attributable, legible, contemporaneous, original, accurate, complete, consistent, enduring and available. In practice that means audit trails for data changes, retained versions of plans and datasets, and documented software and versions for analyses. Your sponsor, institution and local regulator set the exact requirements.

What should I include in a README for my analysis?

State the software and versions needed, describe each folder and file, and give step-by-step instructions to run the analysis from the original data to the final outputs. The TIER protocol treats this README as the entry point to the whole replication folder. Test it by having someone else follow it.

Do I need to document exploratory analyses?

Yes, briefly. Label them as exploratory, note what you tried and why, and keep them separate from confirmatory tests. The National Academies advises clarifying whether a study is exploratory or confirmatory, because that changes how readers interpret p-values.

Can I document an analysis if I don’t code?

Yes. Start with a decision log and data dictionary, which need no code. Then use tools that produce code and records for you, such as R Markdown in RStudio or an agentic analytics tool that exports notebooks, and make sure you can read and justify what they produce.

Sources

• Good Documentation Practices guidance document
• Best Practices: Documentation and Metadata, University at Buffalo
• Document Your Process, Stanford Data Management and Sharing
• Reproducibility and Replicability in Science, National Academies
• Improving Reproducibility and Replicability in Research, National Academies brief
• Study-level documentation, UK Data Service
• Open, rigorous and reproducible research handbook
• TIER Documentation Protocol v2.0
• Study data SOP, University Hospitals Bristol and Weston
• Statistical Analysis and Reporting SOP, ECTU
• Principles for data analysis workflows, PMC
• Primer on Reproducible Research in R, PMC
• Special category data rules, ICO
• Sharing and reuse of health-related data for research, WHO
• IARC Data Protection Policy

Recommended

• Research Reproducibility: A Practical Guide for Analysts
• Analysis Readiness: Six Items Academic Teams Need for IRB/GDPR
• Data Anonymization Techniques: A Practitioner’s Guide
• PhD Data Analysis Help: A Rigorous Workflow Guide