Files
gf/.github/workflows/ci-main.yml
John Guo 852c3dda62 feat(contrib/drivers/dm&pgsql&mssql&oracle): add Replace/LastInsertId features support for dm/pgsql/mssql/oracle (#4547)
This pull request introduces significant improvements to the handling of
the `Replace` and `Save` operations for multiple database drivers,
especially for MSSQL and PostgreSQL. The changes ensure that these
operations now auto-detect primary keys when conflict columns are not
explicitly provided, improving usability and aligning behavior across
drivers. Additionally, the pull request updates related tests to reflect
these enhancements and includes some minor documentation and code
cleanup.

**Key changes:**

### Enhanced Replace/Save Logic for Database Drivers

* **MSSQL Driver:**
- `Replace` and `Save` operations now auto-detect primary keys if
`OnConflict` is not specified, using the `MERGE` statement for upsert
functionality. If no primary key is found in the data, a detailed error
is returned.
[[1]](diffhunk://#diff-87815aa559a927e2de09bd05148f9841dfc06a1b5f3ecc5e3d5fcb80323a87f8L23-R61)
[[2]](diffhunk://#diff-87815aa559a927e2de09bd05148f9841dfc06a1b5f3ecc5e3d5fcb80323a87f8L43-L59)
- Updated tests to verify that `Replace` correctly updates or inserts
records, and that missing conflict columns are properly handled.
[[1]](diffhunk://#diff-bdbde9d7d6ee14c795343767b414740c4396f4dd3e97788b1f9d4e615405a42dL141-R151)
[[2]](diffhunk://#diff-26338e93e473300b1313936eb0f6826546473793442f24715fa294b595f7a805L2661-R2707)

* **PostgreSQL Driver:**
- Similar to MSSQL, `Replace` and `Save` now auto-detect primary keys
for conflict resolution if `OnConflict` is not set, and treat `Replace`
as a `Save` operation.
- Adjusted tests to ensure `Save` and `Replace` work as expected,
including verifying data replacement and insertion.
[[1]](diffhunk://#diff-c22703c37ebb6836c332f7cd2ada570577ba4564fe39886db02f7c2d0e7a2048L93-R93)
[[2]](diffhunk://#diff-c22703c37ebb6836c332f7cd2ada570577ba4564fe39886db02f7c2d0e7a2048R102)
[[3]](diffhunk://#diff-c22703c37ebb6836c332f7cd2ada570577ba4564fe39886db02f7c2d0e7a2048L110-R130)

* **DM Driver:**
- Improved conflict detection: now checks that at least one primary key
exists in the provided data when `OnConflict` is not specified, and
provides clearer error messages.
- Refactored to use the core method for primary key detection and
removed redundant code.

### Minor Improvements and Documentation

* Added clarifying comments to `DoInsert` methods for ClickHouse, DM,
MSSQL, Oracle, and PostgreSQL drivers, specifying that the input list
must have at least one validated record.
[[1]](diffhunk://#diff-f2e003895041ed3c52b91bb8c270696adc3528d77c39d2f7137af3396267444cR19)
[[2]](diffhunk://#diff-f51b30e3f0b0f1284b905385a89992efd0de2fe9ff8c5a4062344dfab17d428eR23)
[[3]](diffhunk://#diff-87815aa559a927e2de09bd05148f9841dfc06a1b5f3ecc5e3d5fcb80323a87f8L23-R61)
[[4]](diffhunk://#diff-f61dac3fcfd5df4a3936cd8743499c8c0fc45f4f5d0f5398ed84a0cb1603202cR24)
[[5]](diffhunk://#diff-c1dfed79aaa3a432057d2bd74d270e4b4094ebcf72984f1161d4972bea009410R16-R72)
* Minor code and comment cleanups, including improved formatting and
error handling.
[[1]](diffhunk://#diff-f61dac3fcfd5df4a3936cd8743499c8c0fc45f4f5d0f5398ed84a0cb1603202cR37)
[[2]](diffhunk://#diff-f61dac3fcfd5df4a3936cd8743499c8c0fc45f4f5d0f5398ed84a0cb1603202cL96-R98)
[[3]](diffhunk://#diff-f61dac3fcfd5df4a3936cd8743499c8c0fc45f4f5d0f5398ed84a0cb1603202cL106-L116)
[[4]](diffhunk://#diff-a17b44c76aaac53d1f164a2bb9440a5531659f4355e7ccfabdadff8dc8633c09L170-R171)
[[5]](diffhunk://#diff-56189fa9ae1df51716b50d34d7fe56bfe67a330e8ac2c6b0de7b958db6817ed5R83-R98)

### Workflow and Documentation Updates

* Updated example Docker commands in the CI workflow for consistency and
clarity.
[[1]](diffhunk://#diff-a1a3cb9bdeb5541d148091d973cf266aa3b317e6415a86630e816cbe27cf8b9cL57-R57)
[[2]](diffhunk://#diff-a1a3cb9bdeb5541d148091d973cf266aa3b317e6415a86630e816cbe27cf8b9cL78-R78)
[[3]](diffhunk://#diff-a1a3cb9bdeb5541d148091d973cf266aa3b317e6415a86630e816cbe27cf8b9cL92-R92)
[[4]](diffhunk://#diff-a1a3cb9bdeb5541d148091d973cf266aa3b317e6415a86630e816cbe27cf8b9cL106-R106)
[[5]](diffhunk://#diff-a1a3cb9bdeb5541d148091d973cf266aa3b317e6415a86630e816cbe27cf8b9cL153-R153)
[[6]](diffhunk://#diff-a1a3cb9bdeb5541d148091d973cf266aa3b317e6415a86630e816cbe27cf8b9cL164-R164)
* Removed outdated note about `Replace` support from the SQLite driver
documentation.

These changes improve the consistency, reliability, and developer
experience when performing upsert operations across different database
backends.

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Lance Add <1196661499@qq.com>
2025-12-09 15:46:41 +08:00

290 lines
8.6 KiB
YAML

# The main codes build and unit testing running workflow.
name: GoFrame Main CI
on:
push:
branches:
- master
- develop
- personal/**
- feature/**
- enhance/**
- fix/**
pull_request:
branches:
- master
- develop
- personal/**
- feature/**
- enhance/**
- fix/**
workflow_dispatch:
inputs:
debug:
type: boolean
description: 'Enable tmate Debug'
required: false
default: false
# This allows a subsequently queued workflow run to interrupt previous runs
concurrency:
group: '${{ github.workflow }} @ ${{ github.event.pull_request.head.label || github.head_ref || github.ref }}'
cancel-in-progress: true
env:
TZ: "Asia/Shanghai"
# for unit testing cases of some components that only execute on the latest go version.
LATEST_GO_VERSION: "1.25"
jobs:
code-test:
strategy:
matrix:
# 🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥
# When adding new go version to the list, make sure:
# 1. Update the `LATEST_GO_VERSION` env variable.
# 🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥🔥
go-version: [ "1.23", "1.24", "1.25" ]
goarch: [ "386", "amd64" ]
runs-on: ubuntu-latest
# Service containers to run with `code-test`
services:
# Etcd service.
# docker run -p 2379:2379 -e ALLOW_NONE_AUTHENTICATION=yes bitnamilegacy/etcd:3.4.24
etcd:
image: bitnamilegacy/etcd:3.4.24
env:
ALLOW_NONE_AUTHENTICATION: yes
ports:
- 2379:2379
# Redis backend server.
redis:
image : redis:7.0
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
# Maps tcp port 6379 on service container to the host
- 6379:6379
# MySQL backend server.
# docker run \
# -p 3306:3306 \
# -e MYSQL_DATABASE=test \
# -e MYSQL_ROOT_PASSWORD=12345678 \
# mysql:5.7
mysql:
image: mysql:5.7
env:
MYSQL_DATABASE : test
MYSQL_ROOT_PASSWORD: 12345678
ports:
- 3306:3306
# MariaDb backend server.
# docker run \
# -p 3307:3306 \
# -e MYSQL_DATABASE=test \
# -e MYSQL_ROOT_PASSWORD=12345678 \
# mariadb:11.4
mariadb:
image: mariadb:11.4
env:
MARIADB_DATABASE: test
MARIADB_ROOT_PASSWORD: 12345678
ports:
- 3307:3306
# PostgreSQL backend server.
# docker run \
# -p 5432:5432 \
# -e POSTGRES_PASSWORD=12345678 \
# -e POSTGRES_USER=postgres \
# -e POSTGRES_DB=test \
# -v postgres:/Users/john/Temp/postgresql/data \
# postgres:17-alpine
postgres:
image: postgres:17-alpine
env:
POSTGRES_PASSWORD: 12345678
POSTGRES_USER: postgres
POSTGRES_DB: test
TZ: Asia/Shanghai
ports:
- 5432:5432
# Set health checks to wait until postgres has started
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
# MSSQL backend server.
# docker run \
# -p 1433:1433 \
# -e ACCEPT_EULA=Y \
# -e SA_PASSWORD=LoremIpsum86 \
# -e MSSQL_USER=root \
# -e MSSQL_PASSWORD=LoremIpsum86 \
# mcr.microsoft.com/mssql/server:2022-latest
mssql:
image: mcr.microsoft.com/mssql/server:2022-latest
env:
TZ: Asia/Shanghai
ACCEPT_EULA: Y
MSSQL_SA_PASSWORD: LoremIpsum86
ports:
- 1433:1433
options: >-
--health-cmd="/opt/mssql-tools18/bin/sqlcmd -S localhost -U sa -P ${MSSQL_SA_PASSWORD} -N -C -l 30 -Q \"SELECT 1\" || exit 1"
--health-start-period 10s
--health-interval 10s
--health-timeout 5s
--health-retries 10
# ClickHouse backend server.
# docker run \
# -p 9000:9000 -p 8123:8123 -p 9001:9001 \
# clickhouse/clickhouse-server:24.11.1.2557-alpine
clickhouse-server:
image: clickhouse/clickhouse-server:24.11.1.2557-alpine
ports:
- 9000:9000
- 8123:8123
- 9001:9001
# Polaris backend server.
# docker run \
# -p 8090:8090 -p 8091:8091 -p 8093:8093 -p 9090:9090 -p 9091:9091 \
# polarismesh/polaris-standalone:v1.17.2
polaris:
image: polarismesh/polaris-standalone:v1.17.2
ports:
- 8090:8090
- 8091:8091
- 8093:8093
- 9090:9090
- 9091:9091
# Oracle 11g server.
# docker run \
# -e ORACLE_ALLOW_REMOTE=true \
# -e ORACLE_SID=XE \
# -e ORACLE_DB_USER_NAME=system \
# -e ORACLE_DB_PASSWORD=oracle \
# -p 1521:1521 \
# loads/oracle-xe-11g-r2:11.2.0
oracle-server:
image: loads/oracle-xe-11g-r2:11.2.0
env:
ORACLE_ALLOW_REMOTE: true
ORACLE_SID: XE
ORACLE_DB_USER_NAME: system
ORACLE_DB_PASSWORD: oracle
ports:
- 1521:1521
# dm8 server
# docker run -p 5236:5236 loads/dm:v8.1.2.128_ent_x86_64_ctm_pack4
dm-server:
image: loads/dm:v8.1.2.128_ent_x86_64_ctm_pack4
ports:
- 5236:5236
zookeeper:
image: zookeeper:3.8
ports:
- 2181:2181
steps:
# TODO: szenius/set-timezone update to node16
# sudo ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
- name: Setup Timezone
uses: szenius/set-timezone@v2.0
with:
timezoneLinux: "Asia/Shanghai"
- name: Checkout Repository
uses: actions/checkout@v5
- name: Setup tmate Session
uses: mxschmitt/action-tmate@v3
if: ${{ github.event_name == 'workflow_dispatch' && inputs.debug }}
with:
detached: true
limit-access-to-actor: false
- name: Free Disk Space
run: |
df -h /
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/hostedtoolcache/CodeQL /opt/hostedtoolcache/Python || true
df -h /
- name: Start Apollo Containers
run: docker compose -f ".github/workflows/apollo/docker-compose.yml" up -d --build
- name: Start Nacos Containers
run: docker compose -f ".github/workflows/nacos/docker-compose.yml" up -d --build
- name: Start Redis Cluster Containers
run: docker compose -f ".github/workflows/redis/docker-compose.yml" up -d --build
- name: Start Consul Containers
run: docker compose -f ".github/workflows/consul/docker-compose.yml" up -d --build
- name: Setup Golang ${{ matrix.go-version }}
uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
cache-dependency-path: '**/go.sum'
- name: Install Protoc
uses: arduino/setup-protoc@v3
with:
version: "31.x"
repo-token: ${{ secrets.GITHUB_TOKEN }}
- name: Install the protocol compiler plugins for Go
run: |
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
export PATH="$PATH:$(go env GOPATH)/bin"
- name: Before Script
run: bash .github/workflows/scripts/before_script.sh
- name: Build & Test
if: ${{ (github.event_name == 'push' && github.ref != 'refs/heads/master') || github.event_name == 'pull_request' }}
run: bash .github/workflows/scripts/ci-main.sh
- name: Build & Test & Coverage
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }}
run: bash .github/workflows/scripts/ci-main.sh coverage
- name: Stop Redis Cluster Containers
run: docker compose -f ".github/workflows/redis/docker-compose.yml" down
- name: Stop Apollo Containers
run: docker compose -f ".github/workflows/apollo/docker-compose.yml" down
- name: Stop Nacos Containers
run: docker compose -f ".github/workflows/nacos/docker-compose.yml" down
- name: Stop Consul Containers
run: docker compose -f ".github/workflows/consul/docker-compose.yml" down
- name: Report Coverage
uses: codecov/codecov-action@v4
# Only report coverage on the latest go version
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' && matrix.go-version == env.LATEST_GO_VERSION }}
with:
flags: go-${{ matrix.go-version }}-${{ matrix.goarch }}
token: ${{ secrets.CODECOV_TOKEN }}