Popper Probe

    1

    Convert fuzzy research claims into sharp, falsifiable hypothesis files through an adversarial Popperian dialogue.

    Free

    1 useSecurity scanned

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    popper-probe

    Example session with this skill installed

    I think LLMs perform better at reasoning tasks when the prompt is written in the style of a Socratic dialogue. I want to turn this into a testable hypothesis.

    • Read your context and instructions
    • Compiled the popper-probe

    I'll help you sharpen that claim into a falsifiable hypothesis.

    Process Summary

    We've completed the Popperian intake dialogue. I have operationalized the "Socratic style" as a specific prompt template, defined the "reasoning tasks" as the Big-Bench Hard (BBH) suite, and identified a 5% accuracy delta as the falsifier.

    hypothesis.md

    ---
    slug: llm-socratic-reasoning-delta
    created: 2024-05-22
    status: active
    falsifiability_gate: passed
    literature_pass: none
    ---
    
    # GPT-4o accuracy on BBH tasks increases by >=5% when using Socratic-style prompting compared to zero-shot chain-of-thought.
    
    > ## Original framing
    > "I think LLMs perform better at reasoning tasks when the prompt is written in the style of a Socratic dialogue."
    
    ## Operational restatement
    GPT-4o (version 2024-05-13) will achieve at least a 5 percentage point higher aggregate score on the Big-Bench Hard (BBH) reasoning benchmark when using the 'Socratic-v1' template compared to a standard 'Zero-shot CoT' baseline.
    
    ## Falsifier(s)
    - The Socratic-v1 template results in an aggregate score within ±1% of the Zero-shot CoT baseline.
    - The Socratic-v1 template produces a statistically significant decrease in accuracy across >50% of individual sub-tasks.
    
    ## Test design
    - **Methods:** Automated evaluation using the BBH evaluation harness.
    - **Design:** Randomized trial of 250 prompts per sub-task. Temperature set to 0.0 for reproducibility.
    - **Instruments:** OpenAI API (GPT-4o), Python evaluation script.
    
    ## Auxiliary assumptions
    - The Socratic-v1 template does not exceed the model's context window limits.
    - The BBH tasks are not already contaminated within the model's training data for these specific prompt styles.
    
    ## Distinctiveness
    - **Prediction:** This hypothesis predicts that the structural framing of dialogue improves latent reasoning, whereas competing 'token-limit' theories suggest performance is purely a function of response length.
    
    ## Intake log
    - 2024-05-22: Initial intake completed. User refined 'better' to '>=5% accuracy'.
    

    Next steps

    1. Run python3 scripts/validate_hypothesis.py popper-corpus/llm-socratic-reasoning-delta/hypothesis.md to confirm schema.
    2. Place any related research papers in popper-corpus/llm-socratic-reasoning-delta/refs/.
    3. Begin the experimental run using the identified BBH harness.
    hypothesis.md

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Research often stalls because hypotheses are too fuzzy to test. This skill acts as a technical sparring partner, forcing you to convert empirical conjectures into sharp, falsifiable markdown files within your local corpus. It uses a structured Popperian dialogue to strip away ambiguity and define exactly what observation would prove a claim wrong.

    What it does

    • Operational restatement translates vague ideas into measurable, third-party verifiable claims with specific units and magnitudes.
    • Falsifier identification requires you to name the exact experimental result that would force you to abandon the hypothesis.
    • Test design sketching documents methods, instruments, controls, and sample size intuition for the proposed experiment.
    • Auxiliary assumption logging captures the hidden requirements that must be true for your test to remain valid.
    • Automated corpus management writes approved hypotheses to a local markdown directory with YAML frontmatter for future tracking.

    How it works

    1. Preamble offers an optional state-of-the-art orientation based on your supplied papers or its own internal knowledge.
    2. Sharpening runs through four specific probes: restatement, falsifier, test design, and distinctiveness.
    3. Drafting generates a hypothesis file based on your dialogue and presents it for your review.
    4. Validation writes the final .md file to your popper-corpus directory once you approve the draft.

    Frameworks & tools

    This skill produces standard Markdown with YAML frontmatter. It includes a schema validation step using a provided Python script (validate_hypothesis.py) and handles local file I/O for repository management.

    Why this beats prompting it yourself

    Generic LLMs tend to be overly agreeable, validating weak claims instead of challenging them. This skill is hardcoded to be adversarial, blocking the "termination path" until a concrete falsifier is named, which prevents researchers from committing to unfalsifiable work.

    Use cases

    • Converting a whitepaper intuition into a rigorous experimental design.
    • Documenting the assumptions behind a new machine learning architecture.
    • Auditing a fuzzy business claim before spending resources on A/B testing.
    • Building a structured research corpus for a doctoral thesis or lab project.

    Known limitations

    Does not perform automated web searches unless explicitly requested in Probe 0. It cannot audit your existing corpus for contradictions or log new observations against old files.

    Requires local file system access. Only analyzes user-supplied papers for literature passes. Does not support automated web searches for papers outside the optional Probe 0 fallback.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    1 use

    Times used to date

    Free forever

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 1 month ago

    • Free with an Agensi account

    Permissions

    Network Access
    Listed1 month ago

    What's inside

    Frequently Asked Questions

    This skill prevents researchers from wasting resources on 'unfalsifiable' work by acting as an adversarial partner that refuses to finalize a project until you have defined a specific, measurable result that would prove your theory wrong.

    The skill generates standardized Markdown files with YAML frontmatter, specifically designed for local repository management and easy integration with tools like Obsidian, Notion, or GitHub.

    No, this skill is designed to be adversarial; it is hardcoded to challenge your assumptions and block the creation of the final hypothesis file until you have met rigorous Popperian criteria.

    You will receive the 'intake' skill configuration, a specialized Python validation script (validate_hypothesis.py) for schema checking, and documentation for setting up your local hypothesis corpus directory.

    While the core logic is framework-agnostic, the skill requires an environment capable of local file I/O and Python execution to run the included validation and file management scripts.

    New in Research & Analysis