You're comparing three ad creatives, auditing treatment responses, or checking whether feature adoption differs by user tier. Chi square test usage fits when your data are counts in categorical groups and the question is whether observed frequencies differ from what chance or a reference distribution would produce. It doesn't estimate a causal effect or predict an outcome. It tests whether the pattern is compatible with a null hypothesis.
PlotStudio connects this statistical workflow with agentic analytics. Instead of returning one isolated answer, it can plan a multi-step analysis, write and run Python locally, inspect expected counts, and save the result as a reproducible Analysis Page with narrative, charts, code, and statistics.
Table of Contents
- What Chi Square Tests Are Used For
- Goodness-of-Fit and Independence Explained
- Assumptions You Must Verify Before Running the Test
- Running a Chi Square Test Step by Step in Python and R
- Alternatives When Assumptions Fail
- Two Real Examples With Effect Size and Reporting
- Common Mistakes and Misconceptions to Avoid
- Reproducible Chi Square Workflows With Agentic Analytics
What Chi Square Tests Are Used For
A marketing analyst might have click or conversion counts for three subject lines. A clinician might compare treatment response categories across groups. A product manager might ask whether account tier and feature adoption are associated. In each case, the raw material is categorical frequency data, not a set of continuous measurements such as revenue, height, or response time.
Two forms handle most introductory workflows:
- Goodness-of-fit: one categorical variable is compared with a hypothesized distribution.
- Test of independence: two categorical variables are compared in a contingency table to assess whether they're associated.
A third standard form, the test of homogeneity, compares the distribution of one categorical outcome across several populations or groups. These forms belong to a family introduced by Karl Pearson in 1900, a milestone in the development of categorical-data inference described in this historical review.
The question the test can answer
The underlying question is simple:
Do these counts differ more than we'd reasonably expect from chance under the null model?
For goodness-of-fit, the null model specifies category probabilities. For independence, it assumes the row and column variables have no relationship. Expected counts come from that model, then the test compares them with the observed counts.
Chi-square usage is broad because the method only needs category counts. It appears in clinical research, genetics, sociology, education, manufacturing quality control, and market research, as summarized by JMP's overview of the chi-square test of independence. Analysts can also explore market research use cases where categorical comparisons support decisions about audiences, responses, and preferences.
Don't ask chi-square to tell you which creative caused a lift, how large the business impact is, or what a new user will do next. Use an effect size, confidence interval, regression model, or predictive method for those questions.
Goodness-of-Fit and Independence Explained
Start with the data shape, not the formula. Suppose a die is rolled and the analyst records how often each face appears. A goodness-of-fit test compares those observed frequencies with a distribution specified in advance, such as equal probabilities for each face. The result tells you whether the observed pattern is inconsistent with that reference distribution.
The independence test uses two categorical variables. An email experiment might cross subject line with opened versus not opened. The null hypothesis says the variables are independent, so the proportion opened should not depend on subject line. A small p-value indicates evidence against that independence assumption, but it doesn't identify the mechanism or establish that the subject line caused the behavior.
The calculation underneath
For either form, the statistic is:
χ² = Σ((O − E)² / E)
Here, O is an observed count and E is the count expected under the null model. Each cell contributes a squared discrepancy scaled by its expected frequency. Large discrepancies in cells with small expected counts can contribute strongly, which is why expected-frequency diagnostics matter.
For an independence table, the expected count for a cell is calculated from its row total, column total, and grand total:
Eᵢⱼ = row total × column total / grand total
The degrees of freedom depend on the design. For a goodness-of-fit test with k categories, the approximate reference distribution has k − 1 degrees of freedom, because the category counts are constrained by the estimated total Yale's goodness-of-fit explanation. For an independence table, the usual calculation is based on the number of rows and columns.
| Dimension | Goodness-of-Fit | Test of Independence |
|---|---|---|
| Variables | One categorical variable | Two categorical variables |
| Input shape | Category counts | Contingency table |
| Null hypothesis | Observed distribution matches the specified distribution | Variables are independent |
| Expected counts | Based on hypothesized proportions | Based on row and column margins |
| Typical use | Check whether responses follow a target mix | Check whether group and outcome are associated |
| Main limitation | The reference distribution must be defensible | Observations and expected counts must support the approximation |
A significant result means the observed pattern is difficult to reconcile with the null model at the chosen decision threshold. It doesn't mean every category differs, and it doesn't supply a practical effect size by itself.
Assumptions You Must Verify Before Running the Test
Treat assumptions as gates, not suggestions. A chi-square result can look precise while resting on a table that doesn't support the approximation.
Check the data and study design
The input must be counts, and categories must be mutually exclusive. A person, transaction, patient, or account should contribute to the relevant frequency structure according to the design. Percentages alone aren't enough unless you can recover the underlying counts.
Observations also need to be independent. Repeated measurements from the same person, matched subjects, or clustered survey responses don't behave like unrelated observations. A clustered sample can produce artificially small p-values because the nominal row count overstates the amount of independent information.
The expected-frequency rule is specific. For independence tests, the approximation is generally considered reliable when expected counts are 5 or more in at least 80% of cells, with no expected count below 1 the methodological discussion of chi-square validity. The check applies to expected counts, not observed counts.
Run the diagnostics before interpreting p
After fitting the null model, inspect the expected matrix cell by cell. A table can have a substantial total sample and still contain a rare category with a problematic expected frequency. Also review how the sample was collected, whether categories overlap, and whether the same unit appears more than once.
A useful pre-publication checklist is:
- Confirm frequency data: Use counts rather than proportions alone.
- Review independence: Identify repeated measures, matching, clustering, and household or site effects.
- Inspect expected counts: Check the full expected matrix, not just the raw table.
- Assess sampling scope: Decide whether the sampling design supports the population claim.
- Choose an alternative if needed: Don't force an asymptotic p-value through a sparse table.
For planning and data-collection questions, this guide to sample-size determination can help you think through the design before analysis.
The distinction between observed and expected counts is especially important. An observed cell of zero isn't automatically disqualifying. What matters is whether the null model expects enough observations in that cell and whether the broader table satisfies the approximation conditions.
Running a Chi Square Test Step by Step in Python and R
A defensible workflow starts with raw categorical records or a clearly documented count table. First, define the unit of observation and category labels. Then create the contingency table, calculate margins, run the test, inspect expected frequencies, and save the complete output.
In Python, SciPy returns the statistic, p-value, degrees of freedom, and expected frequencies:
import numpy as np
from scipy.stats import chi2_contingency, chisquare
table = np.array([
[42, 18],
[35, 25]
])
chi2, p_value, df, expected = chi2_contingency(table)
print("chi2:", chi2)
print("p:", p_value)
print("df:", df)
print(expected)
For goodness-of-fit, supply observed counts and the expected counts or probabilities:
observed = np.array([18, 12, 10, 11, 9])
expected = np.array([12, 12, 12, 12, 12])
chi2, p_value = chisquare(
f_obs=observed,
f_exp=expected
)
print(chi2, p_value)
In R, chisq.test() handles both forms:
table <- matrix(
c(42, 18,
35, 25),
nrow = 2,
byrow = TRUE
)
result <- chisq.test(table)
result$statistic
result$p.value
result$parameter
result$expected
For goodness-of-fit:
observed <- c(18, 12, 10, 11, 9)
expected <- c(12, 12, 12, 12, 12)
result <- chisq.test(
x = observed,
p = expected / sum(expected)
)
result$statistic
result$p.value
result$parameter
| Step | Python, scipy | R, stats |
|---|---|---|
| Independence test | chi2_contingency(table) |
chisq.test(table) |
| Goodness-of-fit | chisquare(f_obs, f_exp) |
chisq.test(x, p = ...) |
| Statistic | First returned value | result$statistic |
| p-value | Second returned value | result$p.value |
| Degrees of freedom | Third returned value | result$parameter |
| Expected matrix | Fourth returned value | result$expected |
Use a tidy source table when your raw data have one row per observation. Group by the categorical variables, count rows, and preserve the code that creates the matrix. If data refreshes are possible, keep the test in a script or notebook with documented package versions and any random seed used for simulation.
Code generation can reduce boilerplate, but it doesn't remove the need to inspect the table. This discussion of Python code generation is relevant when you want generated code that remains reviewable.
Practical rule: Never trust the asymptotic p-value until you've printed the expected-frequency matrix.
Alternatives When Assumptions Fail
A failed assumption is a decision point, not an invitation to tweak the table until chi-square becomes convenient.
For a 2×2 table with sparse expected counts, Fisher's exact test is the usual alternative. It calculates an exact probability under the table margins rather than relying on the chi-square approximation. It's especially appropriate when the sample is small or a rare outcome creates thin cells.
For a larger table, the choice is less automatic. A G-test, also called a log-likelihood-ratio test, compares observed and expected counts using a likelihood-based discrepancy. It can be useful when expected counts are borderline, but it still requires thoughtful handling of sparse categories.
Monte Carlo methods provide another route. You can simulate tables under the null structure, calculate the chi-square statistic for each simulated table, and estimate the p-value from the resulting reference distribution. This preserves the original table dimensions while avoiding blind reliance on the asymptotic approximation.
The requested decision thresholds are useful as operating rules:
- 2×2, small sample or an expected count below 5: Prefer Fisher's exact test.
- Larger sparse table with expected counts between 1 and 5: Consider the G-test or an exact-style method suited to the table.
- Adequate overall sample with a few low cells: Consider a simulated chi-square p-value.
- Ordinal categories: Use a trend-sensitive method when the ordering is part of the research question.
- Paired categorical observations: Use a paired-data method such as McNemar's test, not ordinary independence chi-square.
Yates' correction is mainly used for 2×2 tables. It replaces each ordinary Pearson contribution with (|O − E| − 0.5)² / E, which decreases the statistic and can increase the p-value the correction formula and its effect. It's a modeling choice, not a universal repair for poor data.
Effect-size measures remain important after switching tests. For 2×2 tables, report phi where appropriate. For larger tables, report Cramér's V, alongside the p-value and an interval when your reporting framework supports one. This overview of nonparametric tests provides broader context for choosing methods when standard assumptions fail.
Two Real Examples With Effect Size and Reporting
The p-value is only one part of a categorical analysis. A useful report gives the table, test choice, degrees of freedom, p-value, effect size, and an interpretation tied to the decision.
Marketing conversion comparison
A marketing team compares subject-line variants A and B across 4,200 recipients. The resulting 2×2 conversion table is:
| Variant | Converted | Not converted | Total |
|---|---|---|---|
| A | 126 | 1,974 | 2,100 |
| B | 151 | 1,949 | 2,100 |
| Total | 277 | 3,923 | 4,200 |
The Pearson test produces χ² = 6.14, df = 1, and p = .013. The reported effect size is Cramér's V = 0.038.
| Example | Test | Statistic | df | p | Effect size |
|---|---|---|---|---|---|
| Marketing comparison | Pearson chi-square independence | 6.14 | 1 | .013 | Cramér's V = 0.038 |
| Clinical comparison | Omnibus chi-square independence | Report from fitted table | Table-dependent | Report exact value | Cramér's V, if applicable |
An APA-style sentence could read: A chi-square test of independence indicated an association between subject-line variant and conversion, χ²(1, N = 4,200) = 6.14, p = .013, Cramér's V = .038. The result is statistically detectable, but the effect is practically tiny. A team deciding whether to ship the variant should consider the business value, uncertainty, implementation cost, and confidence interval, not the p-value alone.
Clinical treatment and response categories
A clinical analysis can use treatment group as the row variable and response category as the column variable. For example, a three-category outcome might contain improved, unchanged, and worsened, with treatment and comparison groups forming the rows.
The omnibus chi-square asks whether the response distribution differs across treatment groups. If it's significant, don't claim that every response category drives the result. Inspect standardized or adjusted residuals cell by cell, apply an appropriate multiplicity adjustment for post-hoc comparisons, and report which treatment-response combinations contribute most strongly.
A transparent reporting template is:
A chi-square test of independence examined the relationship between treatment group and response category, χ²(df, N = [sample size]) = [statistic], p = [exact p], Cramér's V = [effect size]. Adjusted residuals indicated that [specific cells] contributed most to the association. The confidence interval for [effect measure] was [interval], where applicable.
The clinical interpretation should distinguish an omnibus association from a clinically meaningful benefit. If the table is sparse, return to the assumption gate and choose an exact or simulated method before interpreting residuals.
Common Mistakes and Misconceptions to Avoid
Most chi-square errors happen before the software prints the p-value. Analysts pass the wrong table, misread the degrees of freedom, or treat significance as a measure of importance.
| Mistake | Why It Happens | Correct Fix |
|---|---|---|
| Swapping observed and expected counts | Both inputs look like frequency matrices | Label objects explicitly and inspect the expected matrix |
| Running the test on percentages | Percentages are easier to copy into a report | Use raw counts, preserving denominators |
| Miscalculating degrees of freedom | Analysts apply a one-dimensional rule to a contingency table | Match the formula to the test design |
| Treating p as effect size | A small p-value feels like a large result | Report phi or Cramér's V and interpret practical impact |
| Calling non-significance proof of independence | “Not detected” gets rewritten as “absent” | Say the analysis didn't provide sufficient evidence of association |
| Ignoring direction | The omnibus result doesn't show which cells differ | Inspect proportions, residuals, and planned contrasts |
| Assuming normality is required | The word “square” sounds like a continuous-data method | Check categorical-data and frequency assumptions instead |
A large table can produce a very small p-value for a modest association. For example, a result with p = 0.001 and Cramér's V = 0.04 should be described as statistically detectable but weak in magnitude, not as a powerful relationship. The numeric example is an illustration of interpretation, not a universal threshold.
Another common mistake is reporting only “significant” or “not significant.” State the test, statistic, degrees of freedom, exact p-value, sample size where relevant, and effect size. If the test was corrected, exact, or simulated, say so.
Before publishing, ask:
- Did I use counts rather than percentages?
- Are categories mutually exclusive?
- Are observations independent?
- Did I inspect expected counts?
- Did I choose the correct test for paired or clustered data?
- Did I report magnitude and direction?
- Does the conclusion stay within what the test can establish?
Reproducible Chi Square Workflows With Agentic Analytics
A defensible chi-square result is reproducible. Keep the raw count source, table-construction logic, package environment, test call, expected matrix, degrees of freedom, p-value, effect-size calculation, and reporting text connected in one auditable workflow. That chain lets another analyst trace the result back to the source rather than trusting a copied value in a slide deck.
A practical setup includes:
- Version control: Commit the analysis script or notebook.
- Pinned environment: Record Python or R package versions.
- Single entry point: Rebuild the table and results from the source data.
- Assumption output: Save expected counts and warnings for sparse cells.
- Persistent reporting: Export the results table and narrative together.
Agentic analytics suits this workflow because it can investigate across multiple steps rather than translate one question into one query. PlotStudio is built for the individual analyst and researcher: you upload data, review or edit the proposed method in Plan Mode, and let an AI data analyst write and run real Python in an embedded local engine. It can inspect the data, construct tables, check assumptions, generate charts, and save the work as an Analysis Page.
The distinction matters. An answer is a data point. An analysis is actionable, reproducible intelligence with methodology, diagnostics, and an audit trail. PlotStudio keeps the generated code inspectable and supports export to Jupyter notebooks and PDF, while saved pages can be revisited and connected through workspace insights. Data stays on the user's machine during local execution, which is relevant for sensitive research datasets.
An independent review by The Effortless Academic describes PlotStudio as a purpose-built tool for academic data work and discusses its data-quality evaluation and publication-oriented figures. That kind of workflow supports the principle that automation should expose the reasoning path, not hide it.
For a deeper treatment of defensible research practice, see this guide to research reproducibility.
Use PlotStudio AI to upload a categorical dataset, review a chi-square analysis plan, inspect the generated local Python and expected counts, and save the result as an auditable Analysis Page. It's a practical way to turn chi square test usage from a one-off calculation into a reproducible workflow you can revisit, export, and defend.