0 cumulative citations
View corpus contextStudent-drawn decomposition diagrams reliably prompt GPT-4.1 to produce working Python code, suggesting multimodal inputs—not just text—can reshape programming instruction and assessment.
Drawing Your Programs: Exploring the Applications of Visual-Prompting with GenAI for Teaching and Assessment
Student-authored problem-decomposition diagrams can serve as effective multimodal prompts for GPT-4.1, yielding high-quality Python code in an introductory programming course.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
When designing a program, both novice programmers and seasoned developers alike often sketch out -- or, perhaps more famously, whiteboard -- their ideas. Yet despite the introduction of natively multimodal Generative AI models, work on Human-GenAI collaborative coding has remained overwhelmingly focused on textual prompts -- largely ignoring the visual and spatial representations that programmers naturally use to reason about and communicate their designs. In this proposal and position paper, we argue and provide tentative evidence that this text-centric focus overlooks other forms of prompting GenAI models, such as problem decomposition diagrams functioning as prompts for code generation in their own right enabling new types of programming activities and assessments. To support this position, we present findings from a large introductory Python programming course, where students constructed decomposition diagrams that were used to prompt GPT-4.1 for code generation. We demonstrate that current models are very successful in their ability to generate code from student-constructed diagrams. We conclude by exploring the implications of embracing multimodal prompting for computing education, particularly in the context of assessment.
Summary
Main Finding
Multimodal GenAI (GPT-4.1) can reliably interpret student-drawn problem-decomposition diagrams and generate corresponding Python code. In a classroom case study, the model fully extracted diagram-defined function structure in 81.2% of cases and showed a strong correlation (r = 0.776, p < 0.001) between the number of human-identified functions and functions generated. Most mismatches arose from ambiguous or idiosyncratic diagram annotations and legibility issues rather than a fundamental inability to read diagrams.
Key Points
- Motivation
- Developers and novices naturally use diagrams and sketches to reason about program structure. Modern GenAI models that accept images make diagram-based (visual) prompting feasible and authentic.
- Visual prompting can reduce representational transformations (diagram → text → code), lowering extraneous cognitive load for learners.
- Pedagogical affordances
- Diagrams can serve as prompts for code generation, enabling new formative activities (sketch → generate → test → refine) and scalable summative assessments based on behavioral tests of generated programs.
- Types of visual-specification assessments suggested: ERDs → database schemas, decomposition diagrams → behavioral program tests, UML class diagrams → object-oriented code verification.
- Empirical results (case study)
- Participants: 133 students in a large introductory Python course.
- Task: Students sketched decomposition diagrams for implementing “Evil Hangman” on single sheets of paper.
- Prompting: GPT-4.1 was given only the diagram image plus minimal instruction to implement “a program” from the diagram (no task context supplied).
- Evaluation: Two researchers coded ground-truth functions (inter-rater reliability r = 0.97). Model outputs were classified as Fully / Partially / Failed based on alignment of function names and docstrings with diagram-ground-truth.
- Outcomes: 81.2% full extraction success; models tended to generate more functions than human raters.
- Sources of divergence
- Ambiguous annotations: notes or speech-bubble-like annotations were sometimes interpreted as separate functions.
- Inputs vs functions: labeled input boxes were sometimes treated as functions by the model.
- Helpful extras: models sometimes added sensible “helper” functions (display, load wordlist, error handling) not present in diagrams.
- Legibility and incomplete diagrams caused failed extractions.
- Practical considerations
- Visual prompting is promising, but diagrams need conventions/guidelines (to reduce ambiguity) and behavioral testing as the principal assessment method (since students define their own interfaces).
- Risks include overreliance on GenAI, potential academic-integrity concerns, and the need to scaffold critical evaluation of model outputs.
Data & Methods
- Study setting: In-person assignment in a large intro CS course at a U.S. research university; IRB-approved.
- Sample: 133 students produced hand-drawn program decomposition diagrams on letter-sized paper.
- Prompting pipeline: Diagrams were supplied as image inputs to GPT-4.1 with minimal instructions to implement a program based on the depicted decomposition. No explicit domain/context was given to the model beyond the image.
- Coding & evaluation:
- Two researchers independently coded the number and content of functions present in each diagram (inter-rater reliability r = 0.97), reconciled disagreements for ground truth.
- Model-generated functions compared to ground truth and categorized as Fully, Partially, or Failed (based on function name/docstring alignment and quantity).
- Analysis of error sources for mismatches (annotation confusion, input/function ambiguity, helper functions, legibility).
- Quantitative highlights: correlation r = 0.776 between human-identified and model-generated function counts; 81.2% full extraction success.
Implications for AI Economics
- Labor demand and task reallocation
- Short-term automation potential: Routine translation of design into boilerplate or multi-function program structure may be increasingly automated, reducing demand for entry-level coding labor focused on implementing straightforward decompositions.
- Skill premium shift: Demand will rise for higher-level skills—problem decomposition, prompt engineering (especially visual prompts), human-AI coordination, verification, and model auditing. These skills may command wage premia.
- New roles: Growth in roles such as “prompt designer/architect,” multimodal interface designers, AI-assisted curriculum developers, and evaluators who validate AI-generated code.
- Productivity and firm-level impacts
- Increased developer productivity: Faster code prototyping from sketches can compress development cycles and reduce costs for routine components, shifting developer effort toward complex design, integration, and correctness.
- Firm strategy: Companies may invest in internal multimodal GenAI tooling (diagram-to-code pipelines) for rapid prototyping and onboarding, changing capital allocation toward AI-enabled platforms.
- Markets and products
- Education technology market: Opportunity for diagram-to-code authoring tools, automated diagram-based assessment platforms, and training materials focused on decomposition and visual prompting.
- Certification and credentialing: As assessments shift toward specification and behavioral testing of AI-generated outputs, new credentialing models and test designs will be needed; this could create market demand for standardized diagram-spec formats and evaluation services.
- Measurement, hiring, and signaling
- Hiring signals may evolve: Portfolios could emphasize decomposition artifacts and the ability to specify correct behavior (visual designs + validation), not only raw code submissions. Employers will need new evaluative criteria for AI-collaboration skills.
- Credential inflation risk: If AI lowers the barrier for producing working code from specifications, credentials based on code production may lose discriminative value, increasing reliance on validated decomposition or problem-design assessments.
- Equity and distributional effects
- Access inequality: Benefits accrue to those and institutions that can afford multimodal GenAI tools and training; unequal access could widen skill and wage gaps.
- Geographic effects: Lower-cost regions currently supplying implementation labor may see contractions in demand for routine tasks, while demand for higher-level coordination and validation roles could be more concentrated in skill hubs.
- Assessment of automation risks and policy
- Imperfect automation: Models make errors (ambiguous annotations, legibility problems). Economic gains depend on integrating testing/QA; thus, initially the effect is augmentation rather than full replacement.
- Policy implications: Education and labor policies should support upskilling in decomposition, model-assessment skills, and equitable access to multimodal GenAI tools. Accreditation standards may need revision to incorporate AI-assisted work and verify core competencies.
- Investment and research signals
- Strong commercial opportunity for multimodal GenAI applications that reliably convert diagrams to verified code; investors may prioritize startups building robust diagram-to-code toolchains and assessment platforms.
- Further R&D needed on standardized diagram representations, error mitigation, and automated behavioral testing to make market solutions dependable for high-stakes settings.
Summary takeaway for economists and policymakers: Visual prompting via multimodal GenAI can materially change how programming tasks are expressed and executed, shifting value from routine implementation toward decomposition, specification, and verification. The transition creates productivity upside and new market niches, but also raises distributional, credentialing, and regulatory questions that merit proactive policy and educational responses.
Assessment
Paper Typedescriptive
Evidence Strengthlow — The paper provides feasibility/tentative evidence that GPT-4.1 can generate code from student-created decomposition diagrams, but it does not establish causal effects (e.g., on learning, productivity, or assessment validity), lacks control or counterfactual comparisons, and appears to rely on outcome measures and evaluation procedures that are not described as rigorously validated.
Methods Rigormedium — The study leverages real classroom data from a large introductory Python course and applies a concrete multimodal prompting procedure (student diagrams → GPT-4.1 → generated code), which strengthens ecological validity; however, it lacks randomized or quasi-experimental design, clear pre-registered evaluation protocols, and detailed reporting of sample sizes, annotation procedures, and inter-rater reliability, limiting internal validity and reproducibility.
SampleData come from a large introductory undergraduate Python programming course in which students produced problem-decomposition diagrams; those student-constructed diagrams were used as prompts for GPT-4.1 and the resulting code was evaluated (metrics not fully specified in the abstract). Exact sample size, task complexity, and evaluation rubric are not provided.
Themeshuman_ai_collab skills_training productivity
GeneralizabilitySingle-course, introductory Python context (novice programmers); may not generalize to experienced developers or non-educational settings, Student-created diagrams (novice-generated visual representations) may differ from professional design artefacts, Findings are specific to GPT-4.1 and could change with other or future models, Likely limited to low-to-moderate code complexity typical of introductory programming assignments, Unclear evaluation procedures reduce external validity for broader claims about learning gains or workplace productivity
Claims (6)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Work on Human-GenAI collaborative coding has remained overwhelmingly focused on textual prompts, largely ignoring the visual and spatial representations that programmers naturally use to reason about and communicate their designs. Research Productivity | negative | research focus on prompt modalities (text vs. visual) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| There exist natively multimodal Generative AI models (e.g., GPT-4.1) that can accept non-textual inputs. Other | positive | availability/capability of multimodal generative models |
Reading fidelity
high
Study strength
high
|
not reported
|
| Problem decomposition diagrams can function as prompts for code generation in their own right, enabling new types of programming activities and assessments. Output Quality | positive | effectiveness of decomposition diagrams as prompts for code generation |
Reading fidelity
high
Study strength
medium
|
not reported
|
| We present findings from a large introductory Python programming course where students constructed decomposition diagrams that were used to prompt GPT-4.1 for code generation. Other | null_result | implementation of a course-based study using student-constructed diagrams to prompt a multimodal model |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Current models (GPT-4.1) are very successful in their ability to generate code from student-constructed diagrams. Output Quality | positive | success/accuracy of code generation from diagram prompts |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Embracing multimodal prompting has implications for computing education, particularly for assessment design. Training Effectiveness | positive | implications for assessment and pedagogy in computing education |
Reading fidelity
high
Study strength
speculative
|
not reported
|