Taking Control: Midjourney x ControlNet – Studio Carlos Banon

AI Creative Challenge 4.0_ Winner01

Become A Digital Member

Subscribe only for €3.99 per month.
Cancel anytime!

Advance your design skills

join PAACADEMY’s online workshops to learn more about parametric and computational design

Midjourney x ControlNet

Introduction to the Workshop:

Workflows in design have been completely redefined since AI platforms such as Midjourney and Stable Diffusion impacted Architecture and Design. This workshop offers technical knowledge and fosters creative thinking through the integration of Midjourney and Stable Diffusion with Rhinoceros to produce open-ended and controlled designs with industry-ready standards. On one hand, we will learn to expand architectural concepts using Midjourney’s commands, such as /image, /blend, and /remix, including image prompts as guidance. On the other hand, we will develop solid skills to further control geometry, materiality, and atmosphere creation using Stable Diffusion + Controlnet, including inpainting techniques. The workshop will combine both tools to maximize potential and optimize them for various design objectives.

Scope of the Workshop:

Introduction and Overview:

  • A brief introduction to Midjourney and Stable Diffusion
  • Overview of their impact on architectural design workflows
  • Understanding the significance of integrating these tools to produce 3D Models

Midjourney for Architectural Concepts:

  • Introduction to Midjourney’s commands: /image, /blend, and /remix
  • Understanding the use of image prompts for concept generation
  • Hands-on exercises to create and expand architectural concepts using Midjourney

Stable Diffusion + Controlnet for Conceptualization, Materiality and Atmosphere Control:

  • Introduction to Design Possibilities with Stable Diffusion and ControlNet
  • Techniques for controlling materiality and atmosphere in architectural design.
  • Hands-on exercises to develop skills in materiality and atmosphere manipulation

Combining Midjourney, Stable Diffusion, and Rhinoceros for Optimized Design:

  • Strategies for integrating Midjourney and Stable Diffusion within Rhinoceros
  • Methods for optimizing design workflows for various objectives. Impainting.
  • Collaborative exercises to practice combining tools for design exploration

Wrap-up and Next Steps:

  • Recap of the key learnings from the workshop
  • Opportunities for further exploration and skill development
  • Q&A
Midjourney x ControlNet
Midjourney x ControlNet

Learning Objectives:

  • Develop a comprehensive understanding of Midjourney and Stable Diffusion, and their articulation with Rhinoceros to redefine architectural design workflows.
  • Enhance participants’ skills in combining Midjourney’s commands (/image, /blend, /remix) with image prompts for generating and expanding innovative architectural concepts.
  • Build proficiency in controlling materiality and atmosphere using Stable Diffusion + ControlNet, enabling iterations for the creation of immersive and adaptive spaces.
  • Master the integration of Midjourney and Stable Diffusion within Rhinoceros to optimize design workflows, maximize potential, and adapt to various design objectives.


Day 1: Setting Foundations. (5h)

  • Introduction: The Spectrum of AI-Driven Design Workflows using Midjourney and Stable Diffusion – Maximizing the Strengths of Each Platform 
  • Lecture: When Midjourney Comes to Town: a Design Paradigm Change in Architecture.
  • Tutorial 1: Industry-Ready Architectural Image Generation with Midjourney: prompting with /imagine
  • Tutorial 2: Working with Self and External References: workflows with /blend and /remix
  • Lecture: Taking geometric control: Introduction to Stable Diffusion and ControlN
  • Tutorial 3: Setting up Stable Diffusion – Interface, Introduction, Models, and Initial Experiments
  • Tutorial 4: Strategic Prompting: Inpainting Techniques
  • Project Theme and Assignment for Tomorrow  (Remote Follow Up and Troubleshooting in Miro)

Day 2: Control and Production. (4h)

  • Project Interim Review and General Discussion
  • Tutorial 5: Fine-Tuning Constraints: Implementing Multiple ControlNet Layers
  • Tutorial 6: Moving across Midjourney, Stable Diffusion, and Rhinoceros 3D Models. Strategies to create 3D models
  • Project Development (Miro)
  • Presentation of Results: Final Jury Review with Distinguished Guests
  • Gigapixel Enhancement. Selection of the Best Three Projects for Awards


  • Rhinoceros3D (basic notion of 3D modeling is required) + vRAY (trial)
  • Stable Diffusion and ControlNet (step by step installation guide will be provided. A cloud server will be available for students to process images remotely)
  • Photoshop (elemental notions)
  • Midjourney: Standard membership recommended. (No prior experience needed. Subscription required)

Important Notes:


Carlos Banon

Carlos Bañón is an award-winning architect working in the intersection of AI, Digital Design and Advanced Manufacturing methods in Architecture. Co-Founder of his practice Subarquitectura Architects in Spain, and tenured Associate Professor at the prestigious Singapore University of Technology and Design (SUTD), he has also held visiting professorships at MIT and EPFL.

Bañón has been recognized with the Design of the Year Prize by the London Design Museum, Singapore’s President Design Award, the Architizer A+ Award, and featured in prestigious publications such as AD, Dezeen, and ArchDaily. As a trailblazer in AI-driven design, Bañón actively shapes the future of architecture through innovative built projects and shares his expertise in lectures, workshops, and publications. Bañón’s creations have received international acclaim, with multiple selections for the Venice Biennale of Architecture, Triennale di Milano, Seul Biennale of Architecture, and a Mies van der Rohe Award nomination.

Share with a friend:

Learn about parametric and computational from the online courses at the PAACADEMY:

Leave a Comment

Your email address will not be published. Required fields are marked *

Become A Digital Member

Subscribe only for €3.99 per month. Cancel anytime!

Weekly Newsletter in Your Inbox

Explore More

Sponsored Content

Subscribe to our weekly newsletter