Skip to content

ORM Module

ci go report codecov Deps PkgGoDev

Overview

Yokai provides a fxorm module, allowing your application to interact with databases.

It wraps the orm module, based on GORM.

Installation

First install the module:

go get github.com/ankorstore/yokai/fxorm

Then activate it in your application bootstrapper:

internal/bootstrap.go
package internal

import (
    "github.com/ankorstore/yokai/fxcore"
    "github.com/ankorstore/yokai/fxorm"
)

var Bootstrapper = fxcore.NewBootstrapper().WithOptions(
    // load fxorm module
    fxorm.FxOrmModule,
    // ...
)

Configuration

This module provides the possibility to configure the database driver:

  • sqlite for SQLite databases
  • mysql for MySQL databases
  • postgres for PostgreSQL databases
  • sqlserver for SQL Server databases

You can also provide to the ORM the databasedsn, some config, and configure SQL queries automatic logging and tracing.

configs/config.yaml
modules:
  orm:
    driver: mysql                                               # driver to use
    dsn: "user:password@tcp(localhost:3306)/db?parseTime=True"  # database DSN to use
    config:
      dry_run: false                                            # disabled by default
      skip_default_transaction: false                           # disabled by default
      full_save_associations: false                             # disabled by default
      prepare_stmt: false                                       # disabled by default
      disable_automatic_ping: false                             # disabled by default
      disable_foreign_key_constraint_when_migrating: false      # disabled by default
      ignore_relationships_when_migrating: false                # disabled by default
      disable_nested_transaction: false                         # disabled by default
      allow_global_update: false                                # disabled by default
      query_fields: false                                       # disabled by default
      translate_error: false                                    # disabled by default
    log:
      enabled: true  # to log SQL queries, disabled by default
      level: info    # with a minimal level
      values: true   # by adding or not clear SQL queries parameters values in logs, disabled by default
    trace:
      enabled: true  # to trace SQL queries, disabled by default
      values: true   # by adding or not clear SQL queries parameters values in trace spans, disabled by default

See GORM Config for more details about the modules.orm.config configuration keys.

For security reasons, you should avoid to hardcode DSN sensible parts (like the password) in your config files, you can use the env vars placeholders instead:

# ./configs/config.yaml
modules:
  orm:
    driver: mysql
    dsn: "${MYSQL_USER}:${MYSQL_PASSWORD}@tcp(${MYSQL_HOST}:${MYSQL_PORT})/${MYSQL_DATABASE}?parseTime=True"

Usage

You can declare your models, for example:

internal/model/example.go
package model

import (
    "gorm.io/gorm"
)

type ExampleModel struct {
    gorm.Model
    Name string
}

This module makes available the DB in Yokai dependency injection system.

To access it, you just need to inject it where needed, for example in a repository to manage your ExampleModel:

internal/repository/example.go
package repository

import (
    "context"
    "sync"

    "github.com/foo/bar/internal/model"
    "gorm.io/gorm"
)

type ExampleRepository struct {
    mutex sync.Mutex
    db    *gorm.DB
}

func NewExampleRepository(db *gorm.DB) *ExampleRepository {
    return &ExampleRepository{
        db: db,
    }
}

func (r *ExampleRepository) Find(ctx context.Context, id int) (*model.ExampleModel, error) {
    r.mutex.Lock()
    defer r.mutex.Unlock()

    var exampleModel model.ExampleModel

    res := r.db.WithContext(ctx).Take(&exampleModel, id)
    if res.Error != nil {
        return nil, res.Error
    }

    return &exampleModel, nil
}

func (r *ExampleRepository) Create(ctx context.Context, exampleModel *model.ExampleModel) error {
    r.mutex.Lock()
    defer r.mutex.Unlock()

    res := r.db.WithContext(ctx).Create(exampleModel)

    return res.Error
}

Like any other services, the ExampleRepository needs to be registered to have its dependencies autowired:

internal/register.go
package internal

import (
    "github.com/foo/bar/internal/repository"
    "go.uber.org/fx"
)

func Register() fx.Option {
    return fx.Options(
        // register the ExampleRepository
        fx.Provide(repository.NewExampleRepository),
        // ...
    )
}

Migrations

This module provides the possibility to run your schemas migrations.

At bootstrap

To run the migrations automatically at bootstrap, you just need to pass the list of models you want to auto migrate to RunFxOrmAutoMigrate():

internal/bootstrap.go
package internal

import (
    "github.com/ankorstore/yokai/fxcore"
    "github.com/ankorstore/yokai/fxorm"
    "github.com/foo/bar/internal/model"
)

// ...

func Run(ctx context.Context) {
    Bootstrapper.WithContext(ctx).RunApp(
        // run ORM migrations for the ExampleModel model
        fxorm.RunFxOrmAutoMigrate(&model.ExampleModel{}),
        // ...
    )
}

func RunTest(tb testing.TB, options ...fx.Option) {
    // ...

    Bootstrapper.RunTestApp(
        tb,
        // test options
        fx.Options(options...),
        // run ORM migrations for the ExampleModel model for tests
        fxorm.RunFxOrmAutoMigrate(&model.ExampleModel{}),
        // ...
    )
}

Dedicated command

A preferable way to run migrations is via a dedicated command.

You can create it in the cmd/ directory of your application:

cmd/migrate.go
package cmd

import (
    "github.com/ankorstore/yokai/fxcore"
    "github.com/ankorstore/yokai/fxorm"
    "github.com/ankorstore/yokai/log"
    "github.com/foo/bar/internal/model"
    "github.com/spf13/cobra"
    "go.uber.org/fx"
    "gorm.io/gorm"
)

func init() {
    rootCmd.AddCommand(migrateCmd)
}

var migrateCmd = &cobra.Command{
    Use:   "migrate",
    Short: "Run application ORM migrations",
    Run: func(cmd *cobra.Command, args []string) {
        // bootstrap, apply migrations then shutdown
        fxcore.NewBootstrapper().
            WithOptions(fxorm.FxOrmModule).
            WithContext(cmd.Context()).
            RunApp(
                fx.Invoke(func(logger *log.Logger, db *gorm.DB, sd fx.Shutdowner) error {
                    logger.Info().Msg("starting ORM auto migration")

                    // run ORM migrations for the ExampleModel model
                    err := db.AutoMigrate(&model.ExampleModel)
                    if err != nil {
                        logger.Error().Err(err).Msg("error during ORM auto migration")
                    } else {
                        logger.Info().Msg("ORM auto migration success")
                    }

                    // shutdown
                    return sd.Shutdown()
                }),
            )
    },
}

You can then execute this command when needed by running app migrate from a dedicated step in your deployment pipeline.

Performance

See general GORM performance recommendations.

Disable Default Transaction

Gorm performs write (create/update/delete) operations by default inside a transaction to ensure data consistency, which is not optimized for performance.

You can disable it in the configuration:

configs/config.yaml
modules:
  orm:
    config:
      skip_default_transaction: true # disable default transaction

Cache Prepared Statement

To create a prepared statement when executing any SQL (and cache them to speed up future calls):

configs/config.yaml
modules:
  orm:
    config:
      prepare_stmt: true # enable prepared statements

Health Check

This module provides a ready to use OrmProbe, to be used by the health check module.

It will perform a ping to the configured database connection to ensure it is healthy.

You just need to register it:

internal/register.go
package internal

import (
    "github.com/ankorstore/yokai/fxhealthcheck"
    "github.com/ankorstore/yokai/orm/healthcheck"
    "go.uber.org/fx"
)

func Register() fx.Option {
    return fx.Options(
        // register the OrmProbe probe for startup, liveness and readiness checks
        fxhealthcheck.AsCheckerProbe(healthcheck.NewOrmProbe),
        // ...
    )
}

Logging

You can enable the SQL queries automatic logging with modules.orm.log.enabled=true:

configs/config.yaml
modules:
  orm:
    log:
      enabled: true  # to log SQL queries, disabled by default
      level: debug   # with a minimal level
      values: true   # by adding or not clear SQL queries parameters values in logs, disabled by default

To get logs correlation, your need to propagate the context with WithContext():

res := r.db.WithContext(ctx).Take(&exampleModel, id)

As a result, in your application logs:

DBG latency="54.32µs" sqlQuery="SELECT * FROM `examples` WHERE `examples`.`id` = 1 AND `examples`.`deleted_at` IS NULL LIMIT 1" sqlRows=1

If needed, you can obfuscate the SQL values from your SQL queries with modules.orm.log.values=false, this will replace the values in your logs with ?:

DBG latency="54.32µs" sqlQuery="SELECT * FROM `examples` WHERE `examples`.`id` = ? AND `examples`.`deleted_at` IS NULL LIMIT 1" sqlRows=1

Tracing

You can enable the SQL queries automatic tracing with modules.orm.trace.enabled=true:

configs/config.yaml
modules:
  orm:
    trace:
      enabled: true  # to trace SQL queries, disabled by default
      values: true   # by adding or not clear SQL queries parameters values in trace spans, disabled by default

To get traces correlation, your need to propagate the context with WithContext():

res := r.db.WithContext(ctx).Take(&exampleModel, id)

As a result, in your application trace spans attributes:

db.system: "mysql"
db.statement: "SELECT * FROM `examples` WHERE `examples`.`id` = 1 AND `examples`.`deleted_at` IS NULL LIMIT 1"
...

If needed, you can obfuscate the SQL values from your SQL queries with modules.orm.trace.values=false, this will replace the values in your trace spans with ?:

db.system: "mysql"
db.statement: "SELECT * FROM `examples` WHERE `examples`.`id` = ? AND `examples`.`deleted_at` IS NULL LIMIT 1"
...

Testing

This module provide support for the sqlite databases, making your tests portable (in memory, no database required):

configs/config.test.yaml
modules:
  orm:
    driver: sqlite   # use sqlite driver
    dsn: ":memory:"  # in memory

You can then retrieve your components using the DB, and make actual database operations:

internal/repository/example_test.go
package repository_test

import (
    "testing"

    "github.com/foo/bar/internal/model"
    "github.com/foo/bar/internal/repository"
    "github.com/stretchr/testify/assert"
    "go.uber.org/fx"
    "gorm.io/gorm"
)

func TestExampleRepository(t *testing.T) {
    var gormDB *gorm.DB
    var exampleRepository repository.ExampleRepository

    internal.RunTest(t, fx.Populate(&gormDB, &exampleRepository))

    // prepare your test data in the sqlite database
    exampleRepository.Create(
        context.Background(),
        &model.ExampleModel{
            Name: "test",
        },
    )

    // some tests ...

    // close DB
    db, err := gormDB.DB()
    assert.NoError(t, err)

    err = db.Close()
    assert.NoError(t, err)
}

In test mode, the module won't automatically close the database connection on shutdown, to allow database manipulation after the RunTest() execution.