Where LLMs Actually Help in Marketing, With a Check for Each
LLMs help where the output is a draft you can check without asking the model again. What measured experiments show, and a check for each of five marketing tasks.
Three platforms, three answers, one budget to defend. Your GA4 export already holds the answer, and the first finding is free and never leaves your browser. Get your first finding free
By Joris van Huët, Founder & CEOUpdated 3 min read
They help where the output is a draft you can check without asking the model again. The paper notes that some tasks, "though seemingly similar in difficulty level", fall outside the model's reach, so every job below comes with a check.
What do measured results say about where LLMs help?
Dell'Acqua and colleagues (working paper, 2023, abstract via SciSpace/SSRN) randomly assigned 758 consultants at Boston Consulting Group to no AI, GPT-4, or GPT-4 with a prompting overview. On 18 realistic tasks inside the frontier, consultants using AI "completed 12.2% more tasks on average, and completed task 25.1% more quickly" (Dell'Acqua et al., 2023). On a task selected to be outside it, they "were 19 percentage points less likely to produce correct solutions compared to those without AI".
Brynjolfsson, Li and Raymond (Quarterly Journal of Economics, 2025, peer-reviewed) studied 5,172 customer-support agents: "Access to AI assistance increases worker productivity, as measured by issues resolved per hour, by 15% on average".
None of these measures marketing analysis, and no public measurement is cited here for, say, the time saved when an LLM drafts a ShopifyQL query.
Which marketing tasks are worth giving to an LLM?
Five tasks, each with its check:
- Draft a query or a formula. Shopify's help page says "You can use Sidekick to help generate and modify ShopifyQL queries for your reports." Check: run it on ten rows you've counted by hand, then compare the total with Shopify's Sales report.
- Clean campaign and source names. GA4's URL builder page says "Parameter values are case sensitive", so utm_source=google and utm_source=Google are different values. An LLM can propose a mapping. Check: list the distinct values before and after, and read every one that changed.
- Explain a report, a setting or an error. Check: open the platform's help page and read its last-updated date before you act on the explanation.
- Summarize a finished table for someone else. Liu and colleagues (TACL 2023, peer-reviewed) found that performance "significantly degrades when models must access relevant information in the middle of long contexts", so paste the table, not the raw export. Check: every number in the summary must appear in the table.
- Draft ad variants and test ideas. Kıcıman and colleagues (arXiv, 2023) write that LLMs "may be used by human domain experts to save effort in setting up a causal analysis". Check: run the variants as an A/B test in the ad platform and judge them on your own conversions, and turn each test idea into something you can look up or measure.
Which jobs stay with a method you can check?
The estimate of what a channel caused, and the budget call that follows.
Rule of thumb: if you can't check it in minutes without the model, it isn't ready for the deck.
Sources, 30 September 2026: Navigating the Jagged Technological Frontier (Dell'Acqua et al., 2023); Generative AI at Work (Quarterly Journal of Economics, 2025); Improving Generative Ad Text on Facebook using Reinforcement Learning (arXiv, 2025); Overview of ShopifyQL syntax (Shopify Help Center); URL builders (Google Analytics Help); Lost in the Middle (TACL 2023); Causal Reasoning and Large Language Models (arXiv, 2023); How is ChatGPT's behavior changing over time? (arXiv, 2023).
Related answers
Frequently asked questions
What can an LLM do well in marketing analytics?
Draft queries and formulas, clean campaign names, explain reports, summarize finished tables and suggest tests. Each has a check you can run without the model: run the query on rows you've counted, list the distinct values, read the help page date, find every number, and run an A/B test.Should an LLM decide my budget split?
Use it to lay out the options and the test.How do I check an LLM's summary of a report?
Find every number in the summary in the source table, and check each claim against it. Paste the table rather than the raw export: a peer-reviewed study found that performance degrades when relevant information sits in the middle of a long input.
Go deeper: Incrementality testing, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.
Keep reading
Terms in this article
- AnalyticsAnalytics is the systematic computational analysis of data. It reveals customer behavior and measures campaign performance.
- AttributionAttribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
- Causal AnalysisCausal Analysis identifies true cause-and-effect relationships in data, moving beyond correlation to show how marketing actions directly impact outcomes.
- Causal InferenceCausal inference is the set of methods for estimating how much an outcome changes because of a cause, rather than alongside it. It works by comparing what happened with an estimate of what would have happened without the cause.
- ConversionConversion is a specific, desired action a user takes in response to a marketing message, such as a purchase or a sign-up.
- Google AnalyticsGoogle Analytics is a web analytics service that tracks and reports website traffic.
- Holdout TestA holdout test is an experiment that keeps a random part of an audience from seeing a campaign and compares its results with those of the part that could see it. The held-out part is the control group, so the gap in outcomes estimates what the campaign added.
- ShopifyShopify is an ecommerce platform for creating online stores and selling products. Attribution modeling shows which marketing channels drive traffic and conversions within Shopify.