Rails Packwerk Refactor
I have been interested in building Rails with much less accidental coupling. Until recently, Engines have been the best way to do that, but the setup effort is rather heavy (to integrate the migrations, tests, namespaces, …). In fact, heavy enough that most people do without modules.
Packwerk however, makes it easy to use modules and critcially migrate toward modules overtime. Packwerk allows you to organize into modules, without enforcing boundaries - until you are ready to fully refactor and disentagle your code.
To demonstrate Packwerk’s usage lets start with a Standard Rails Monolith and iteratively transform it into discrete packages with boundaries. The file structure of a fully packaged project is easy to visually understand.
The routes file often will give us a clue to or application boundaries (espescially if name spaces or scopes are in use), alternatively a well structured engine architechture may clarify the application boundaries.
For example I have packaged a standard rails application - here are the before and after for the file system (compared to the scoped routes file).
Here is the packaged rails dependency graph:
NOW IDEALLY, by looking within app/packages
and the packwerk.png
the next person will have a decent idea about what the code does and where.
An out line of how to create this basic app is in the appendix - for those who want to follow along.
Setup Packwerk
Packwerk its self is a simple gem install. Crucially, Packwerk has a companion tool graphwerk
to visualize your modules and their dependencies. We will install and demostrate both.
Prequisits
In oder to use graphwerk you must have graphviz installed:
brew install graphviz
does the trick on MacOS
You will also need to use at least Ruby 2.6+
and Rails 6.0+
with Zeitwerk
as the loader (which is the default for new Projects).
Install Packwerk
This will be a very simple projects (too simple to need modules), but that also makes the examples easier to grock.
We will start with a fresh rails projects to keep the complexity low and make it straight-forward to follow along before using in your own established projects.
Now add the packwerk and graphwerk gems to ./Gemfile
# Gemfile
...
# Packwerk (modular Rails) - dependency tooling and mapping
gem "packwerk", "~> 2.2"
gem 'graphwerk', group: %i[development test]
of course normally, I would install rspec and a slew of other tools, but the focus here is simply the usage of Packwerk and its associated tools.
Now of course we need to finalize the install and config:
bundle install
# make it easy to
bundle binstub packwerk
# create intial packwerk config files
bin/packwerk init
now we should see a file ./packwerk.yml
with all the configs commented out. We will learn to configure as we go.
Now lets be sure dependency visualization tool works:
bin/rails graphwerk:update
Now we should see an intial dependency map named ./packwerk.png
looking like:
Configure Packages
First, we need a location to place your packages - let’s make a folder:
mkdir app/packages
Now Tell rails how to find & load the code in your packages (remember this is dependent on Zeitwerk) config.paths.add 'app/packages', glob: '*/{*,*/concerns}', eager_load: true
to config/application.rb
. Now it will look something like:
# config/application.rb
require_relative "boot"
require "rails/all"
# Require the gems listed in Gemfile, including any gems
# you've limited to :test, :development, or :production.
Bundler.require(*Rails.groups)
module Packaged
class Application < Rails::Application
# Initialize configuration defaults for originally generated Rails version.
config.load_defaults 7.0
# config packages fur packwerk
config.paths.add 'app/packages', glob: '*/{*,*/concerns}', eager_load: true
end
end
Finally, let the controllers know how to find the views within packages app/controllers/application_controller.rb
to:
# app/controllers/application_controller.rb
class ApplicationController < ActionController::Base
append_view_path(Dir.glob(Rails.root.join('app/packages/*/views')))
end
Using Packwerk
From doing this a few times - I’ve learned its easiest to work from the outside in. In other words extract actions first and organize them first. Then move the models into the appropriate packages. Then package (most of rails - assets is trickier - so I jut leave it alone
Marketing Package
We can add our proposed packages with:
mkdir app/packages/marketing
mkdir app/packages/marketing/public
mkdir -p app/packages/marketing/views
mkdir app/packages/marketing/controllers
cp package.yml app/packages/marketing/package.yml
if you have helpers, etc make those folders too.
now we can copy the code (controllers & views):
mv app/controllers/landing_controller.rb app/packages/marketing/controllers/.
mv app/views/landings app/packages/marketing/views/landings
now start rails and test that the base config is good (then we will start with enforcing package cofig)
if you get the error:
LandingController#index is missing a template for request formats: text/html
check that app/controllers/application_controller.rb
looks like
# app/controllers/application_controller.rb
class ApplicationController < ActionController::Base
append_view_path(Dir.glob(Rails.root.join('app/packages/*/views')))
end
if you get the error uninitialized constant LandingController
check that config/application.rb
has the following code:
...
module Packwerk
class Application < Rails::Application
# Initialize configuration defaults for originally generated Rails version.
config.load_defaults 7.0
...
# load the packages
config.paths.add 'app/packages', glob: '*/{*,*/concerns}', eager_load: true
end
end
If you forgot this config - you will need to restart rails!
Now that we have confirmed our config - we need to config boundary enforcement in the package’s packages.yml
file config. So lets configure the package in the file app/packages/marketing/package.yml
- we will use the simplest possible config.
# app/packages/marketing/package.yml
# Turn on dependency checks for this package
enforce_dependencies: true
# Turn on privacy checks for this package
enforce_privacy: true
# this allows you to modify what your package's public path is within the package
# code that this package publicly shares with other packages
public_path: public/
# A list of this package's dependencies
# Note that packages in this list require their own `package.yml` file
# '.' - we are dependent on the root application
dependencies:
- '.'
Now that the code works, lets generate a new diagram of our app using:
bin/rails graphwerk:update
if your graph looks the same as before, check that app/packages/marketing/packages.yml
is there and configured.
If all is well the new graph in packwerk.png
will look like:
Now that the package is recognized, lets if packwerk finds any problems
bin/packwerk check
Ideally, packwerk finds no problems and we get:
No offenses detected
No stale violations detected
In a full production system this command would then be integrated into the CI to only allow clean packages to commit.
Package Managers
basically the same as for marketing
lets now make our new manage
package structure:
mkdir app/packages/managers
mkdir app/packages/managers/public
mkdir app/packages/managers/views
mkdir app/packages/managers/controller
cp app/packages/marketing/package.yml app/packages/managers/package.yml
move the appropriate files:
mv app/controllers/users_controller.rb app/packages/manage/controller/users_controller.rb
mv app/controllers/blogs_controller.rb app/packages/manage/controller/blogs_controller.rb
mv app/packages/manage/views/users app/packages/manage/views/users
mv app/packages/manage/views/blogs app/packages/manage/views/blogs
Let’s be sure things still work by going to: localhost:3000/manage/users
Let’s check that the dependencies look like we expect:
bin/rails graphwerk:update
and hopefully we see:
now let’s check for violations using: bin/packwerk check
and hopefully we see:
No offenses detected
No stale violations detected
Writers Package
Guests Package
Core Package
mkdir -p app/packages/core
mkdir -p app/packages/core/public
mkdir -p app/packages/core/models
cp app/packages/managers/package.yml app/packages/core/package.yml
lets move the models
mv app/models/blog.rb app/packages/core/models/blog.rb
mv app/models/user.rb app/packages/core/models/user.rb
Let’s see if the code still works like before.
Now finally, let’s check our enforcement with:
bin/packwerk check
oops - now we have multiple dependency
and privacy
violations:
app/packages/writers/controllers/posts_controller.rb:21:12
Dependency violation: ::Blog belongs to 'app/packages/core', but 'app/packages/writers' does not specify a dependency on 'app/packages/core'.
Are we missing an abstraction?
Is the code making the reference, and the referenced constant, in the right packages?
Inference details: this is a reference to ::Blog which seems to be defined in app/packages/core/models/blog.rb.
To receive help interpreting or resolving this error message, see: https://github.com/Shopify/packwerk/blob/main/TROUBLESHOOT.md#Troubleshooting-violations
problem is we haven’t told our packages they depend on core - to fix we will update package.yml files in managers, writers, guests to depend on core. Marketing and rails application don’t need to change. we will make them look like:
# app/packages/marketing/package.yml
# Turn on dependency checks for this package
enforce_dependencies: true
# Turn on privacy checks for this package
enforce_privacy: true
# this allows you to modify what your package's public path is within the package
# code that this package publicly shares with other packages
public_path: public/
# A list of this package's dependencies
# Note that packages in this list require their own `package.yml` file
# '.' - we are dependent on the root application
dependencies:
- 'app/packages/core'
- '.'
If you are still getting the above error check your tests. Unfortunately, packwerk doesn’t tell zou the offending file, but you can search for the model reference with excluding app/packages
check again:
app/packages/writers/controllers/profiles_controller.rb:5:16
Privacy violation: '::User' is private to 'app/packages/core' but referenced from 'app/packages/writers'.
Is there a public entrypoint in 'app/packages/core/public/' that you can use instead?
Inference details: this is a reference to ::User which seems to be defined in app/packages/core/models/user.rb.
To receive help interpreting or resolving this error message, see: https://github.com/Shopify/packwerk/blob/main/TROUBLESHOOT.md#Troubleshooting-violations
now we have a privacy error - we need to move our models to public - as a way to declare we intend to share them
lets put our models in public (if zou want them in public/models - zou will need to make more tweaks to rails paths)
Let’s be sure our dependency map is now updated
bin/rails graphwerk:update
and we should see:
Rails Package
pretty good but folders still messy - lets make rails in a packagr too
mkdir app/packages/rails
Move everything remaining (our rails files)
- app/assets
- app/packages
into app/packages/rails
Overview
Hanami 2.x will use a structure like Packwerk and is built on dry-rb, which also has many intresting features and useful patterns. When Hanami 2.x is released, a similar dependency checker like packwerk would be valuable.
Currently with Rails one can achieve most of what Hanami 2.x framework offers with Packwerk and dry-rails.
Benefits
Drawbacks
Resources
For on organizing Ruby code via packages / modules
Hanami 2.x
- Hanami 2.x Code
- Hanami 2.x Blog - currently Hanami 2.0 is in Beta and is targeted for API usage (if you are willing to assemble all the parts Hanami 2.0 will also offers front-end services), Hanami 2.1 will be focused on delivering a fully integrated full stack.
- dry-rb code
- dry-rb Website
- rom-rb code
- rom-rb website - speed focus alternative to Rail’s ActiveRecord. While many ORMs focus on objects and state tracking, ROM focuses on data and transformations (a bit like Elixir’s Ecto). This project is heavily influenced by
- Hanami Mastery - tutorials about dry-rb gems and Hanami 2.x concepts in order to help people coming from Rails to understand and use Hanami 2.x effectively.
Modular Rails using Packwerk
- Shopify - Packwerk Code
- Shopify - Packwerk Docs
- Shopify - Packwerk Debug Help
- Shopify - Video introducing Packwerk
- Shopify - on Monoliths
- Shopify - enforcing-modularity-rails-apps-packwerk
- Package-Based-Rails-Applications Book, by Stephan Hagemann
- modularization-with-packwerk
- packwerk-to-delimit-bounded-contexts
Modular Rails using Engines
- Rails Engines Docs
- Component-Based-Rails-Applications Website, Stephan Hagemann - many links and articles on using Engines and enforcing boundaries
- Component-Based Rails Applications Book, 2018, by Stephan Hagemann
- Modular-Rails Book / Website, by Thibault Denizet
Appendix - Create the App
Quick Start
Lets make a this simple Blog App as our starting point.
NOTE: Packwerk require Ruby 2.6 or newer and Rails must be configured with Zeitwerk.
rails new packaged --javascript=esbuild --css=tailwind --skip-active-storage
cd packaged
bin/rails db:create
# marketing
bin/rails g controller landing index --no-helper
# manager
bin/rails g scaffold user full_name email --no-helper
bin/rails g scaffold blog content user:references --no-helper
# bin/rails g scaffold comment note blog:references --no-helper
bin/rails db:migrate
# writers
bin/rails g controller profiles index show edit update --no-helper
bin/rails g controller posts index show new create edit update delete --no-helper
# bin/rails g controller discussion index new create edit update delete --no-helper
# patrons
bin/rails g controller authors index show --no-helper
bin/rails g controller articles index show --no-helper
Routes
now lets update the routes with:
# config/routes.rb
scope 'guests' do
resources :authors, only: %i[index show] do
resources :articles, only: %i[index show]
end
end
scope 'writers' do
resources :profiles, only: %i[index show edit update] do
resources :posts
end
end
scope 'managers' do
resources :users do
resources :blogs
end
end
get '/landing', to: 'landing#index'
root 'landing#index' # root path route ("/")
end