Github Action 自托管服务器踩坑记录

我们的一个项目中使用 Github Action 来进行 CI/CD, 我们的痛点在于 Github Action 提供的缓存存储限制在10G, 而我们某个 job 的缓存比较大, 大约是 3.6G, 再加上别的缓存, 导致存储空间不够, 最终导致CI运行失败. 所以我们希望构建自托管服务器来解决这个问题. 其实一开始考虑的是能不能升级配置, 但是没找到能升级缓存存储大小的办法.

问题如图:
avatar
avatar

配置过程

  1. 使用 Terraform 配一台服务器, 使用的是阿里云, 具体配置如下, 可以参考

main.tf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
locals {
  vpc_id      = alicloud_vpc.vpc.id
  vswitch_ids = [for vswitch in alicloud_vswitch.vswitch : vswitch.id]
  image_id    = data.alicloud_images.images.images.0.id
}

# ------------------------ VPC 内网 ------------------------

resource "alicloud_vpc" "vpc" {
  vpc_name    = var.name
  cidr_block  = var.vpc_cidr
  enable_ipv6 = true // 开启 IPV6
}

resource "alicloud_vswitch" "vswitch" {
  count        = length(var.zones)
  vswitch_name = "vswitch-${count.index}"
  cidr_block   = var.vswitch_cidrs[count.index]
  vpc_id       = local.vpc_id
  zone_id      = var.zones[count.index]
}

# ------------------------ 安全组 -------------------------

resource "alicloud_security_group" "sg" {
  security_group_name = var.name
  description         = "Security group for ECS"
  vpc_id              = local.vpc_id
}

resource "alicloud_security_group_rule" "allow_ssh" {
  type              = "ingress"
  ip_protocol       = "tcp"
  port_range        = "22/22"
  security_group_id = alicloud_security_group.sg.id
  cidr_ip           = "0.0.0.0/0"
}

# ------------------------ 服务器镜像 --------------------

data "alicloud_images" "images" {
  owners     = "system"
  os_type    = "linux"
  name_regex = "^ubuntu_22+_[0-9]+_x64*"
}



# ------------------------ Runner 服务器 ----------------

# 创建弹性 IP
resource "alicloud_eip" "eip" {
  bandwidth            = "20"
  internet_charge_type = "PayByBandwidth"
}

# 创建 SSH 密钥对
resource "alicloud_ecs_key_pair" "key_pair" {
  key_name_prefix = var.name
  public_key      = var.public_key_pair
}

# 创建 ECS 实例
resource "alicloud_instance" "runner" {
  instance_name        = var.name
  host_name            = ""     # 删了, 需要自己填
  image_id             = var.runner_image != "" ? var.runner_image : local.image_id
  instance_type        = var.runner_instance_type
  spot_strategy        = "SpotAsPriceGo"
  security_groups      = [alicloud_security_group.sg.id]
  key_name             = alicloud_ecs_key_pair.key_pair.key_pair_name
  vswitch_id           = local.vswitch_ids[0]
  system_disk_category = "cloud_essd"
  system_disk_size     = 200
}

# 为 ECS 实例绑定公网 IP
resource "alicloud_eip_association" "runner_eip" {
  allocation_id = alicloud_eip.eip.id
  instance_id   = alicloud_instance.runner.id
}

provider.tf

1
2
3
4
5
6
7
8
9
10
11
12
terraform {
  required_providers {
    alicloud = {
      source  = "aliyun/alicloud"
      version = "1.258.0"
    }
  }
}

provider "alicloud" {
  region = var.region
}

variables.tf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
variable "name" {
  type        = string
  description = "名称"
  default     = "swan-lake-runner"
}

variable "region" {
  type        = string
  description = "地域"
  default     = "cn-hongkong"
}

variable "vpc_cidr" {
  type        = string
  description = "VPC 网段"
  default     = "192.168.0.0/16"
}

variable "vswitch_cidrs" {
  type        = list(string)
  description = "子网网段"
  default     = ["192.168.0.0/20", "192.168.16.0/20", "192.168.32.0/20"]
}

variable "zones" {
  type        = list(string)
  description = "可用区"
  default     = ["cn-hongkong-b", "cn-hongkong-c"]
}

variable "public_key_pair" {
  type        = string
  description = "服务器公钥"
  default     = ""   # 需要自己填
}

variable "runner_image" {
  type        = string
  description = "runner 服务器对应的镜像"
  default     = ""
}

variable "runner_instance_type" {
  type        = string
  description = "runner 服务器的实例规格"
  default     = "ecs.u1-c1m2.2xlarge" # 默认 8 核 16 G
}
  1. 按照脚本配置服务器

github 上选好服务器信息就会有脚本, 服务器上新建一个用户 levit (config.sh不能再sudo权限下运行), 照着配置即可

如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Download

# Create a folder
$ mkdir actions-runner && cd actions-runner# Download the latest runner package
$ curl -o actions-runner-linux-x64-2.328.0.tar.gz -L https://github.com/actions/runner/releases/download/v2.328.0/actions-runner-linux-x64-2.328.0.tar.gz
Copied!
# Optional: Validate the hash
$ echo "01066fad3a2893e63e6ca880ae3a1fad5bf9329d60e77ee15f2b97c148c3cd4e  actions-runner-linux-x64-2.328.0.tar.gz" | shasum -a 256 -c# Extract the installer
$ tar xzf ./actions-runner-linux-x64-2.328.0.tar.gz
Configure

# Create the runner and start the configuration experience
$ ./config.sh --url {项目地址} --token {token, 一小时过期}# Last step, run it!
$ ./run.sh

Using your self-hosted runner
# Use this YAML in your workflow file for each job
runs-on: self-hosted

接下来我们要将其注册为服务, 使用svc.sh这个脚本

1
2
3
cd /home/levit/actions-runner
sudo ./svc.sh install levit
sudo ./svc.sh start
  1. 测试运行

直接用原先的配置文件, 只是将服务器配置改了

  • 出现问题, 没有docker, 原因是我们的CI拉了一个 postgresql 的镜像, 这里就体现出自托管的第一个问题, 不能无缝迁移

avatar

  • 于是先安装Docker, 安装完之后, 还是失败:
1
2
3
4
5
6
7
-  Stop containers

Value cannot be null. (Parameter 'network')

- Initialize containers

Exit code 1 returned from process: file name '/usr/bin/docker', arguments 'version --format '{{.Server.APIVersion}}''.

其实有点奇怪, 我已经将用户放到了Docker组里, 但没关系, 我直接修改 docker.sock 权限, 将其设置为所有用户都可读可写

1
2
sudo crontab -e
@reboot chmod 666 /var/run/docker.sock
  • 接着运行, 没有权限错误了, 但还是报错:

avatar

需要为某个job配置cargo环境, 这里也体现出了不能无缝迁移的问题:

1
2
- name: Install Rust toolchain
  uses: dtolnay/rust-toolchain@stable
  • 接着继续rerun, 还是报错:
1
2
3
4
5
6
7
8
error: linking with cc failed: exit status: 1

  |
  = note:  "cc" "-m64" "/tmp/rustcJw9CYQ/symbols.o" "<17 object files omitted>" "-Wl,--as-needed" "-Wl,-Bstatic" "/home/levit/actions-runner/_work/swan-lake/swan-lake/target/debug/deps/{libjsonschema-290691234c978af9,libbytecount-2256add9d3494a3e,libuuid_simd-bb3f0d49847d5f65,liboutref-c83ec375ecf62c12,libvsimd-5cbd82f94125c6a6,libfancy_regex-d11fb54c91ac00d7,libbit_set-0ee784b9472a9e26,libbit_vec-45738dfdd44d120a,libemail_address-6f5ee3998a9edbf7,libfraction-a1a8f9b7d5c27b05,liblazy_static-0aeb88bcc2e3d662,libnum_cmp-c8fad974b14e04e1,libreferencing-c9c6db8602fb3ed6,libfluent_uri-e970ea4acfa64631,libborrow_or_share-a5a2f252111b68c2}.rlib" "<sysroot>/lib/rustlib/x86_64-unknown-linux-gnu/lib/{libtest-*,libgetopts-*,libunicode_width-*,librustc_std_workspace_std-*}.rlib" "/home/levit/actions-runner/_work/swan-lake/swan-lake/target/debug/deps/{libblake3-7dd5ff8fef01e5db,libconstant_time_eq-4fa9b96eb7cef840,libarrayref-83025b7747475057,libopendal-5fb99e01659bf0da,libquick_xml-2a726846ed0b77c8,libreqsign-02438016364b1fd4,libhex-fbbed2810870e39e,libhome-b9d87c5140a51805,libsha2-a6835826c9bc8fce,libsha1-8f2e0fe7b8043ddd,libcpufeatures-69aa18d40df9d02d,libhmac-b419e6aa0f87987b,libreqwest-8dd681c75f772da9,libhyper_rustls-2ac018dc46dd3942,libwebpki_roots-e984893442cdc9cb,libtokio_rustls-8fd037e35f978f21,librustls-9de24b5a794bb3e6,libwebpki-471800d671557465,libring-2bccc933dbdc7831,libuntrusted-50ea818375ed8b6b,librustls_pki_types-63588602b5356c3c,libzeroize-84b9e390cb3b2ce7,libserde_urlencoded-8fb4b2858d139ab2,libtower_http-ba8c76ed943aff1f,libiri_string-be77eb48b0dd18d7,libtower-03506a7144375102,libtower_layer-58bbf4e5492fefb4,libhyper_util-c8f54a2132bc4c90,libipnet-856ee1c52243b1ef,libhyper-ec2ba3f3ffc392b6,libwant-ccae98356ec19395,libtry_lock-9ed3f89773057675,libatomic_waker-93d4097655d001c5,libhttp_body_util-b841ad22673700f3,liburl-ffd43874b94bc1b1,libidna-16a375d8ec8b433b,libutf8_iter-d47e0c5887c2fee5,libidna_adapter-0bdb7cf28e38733d,libicu_properties-dcf335287fb53c38,libicu_properties_data-ef8f86fdfbcc0e0e,libicu_normalizer-50ee150a59eead80,libicu_normalizer_data-39e9a9fa20f7672d,libicu_collections-20fc61c1dfeefcd0,libpotential_utf-47ef1a08876992b2,libicu_provider-aed8a73499001d21,libicu_locale_core-49e8c067c2d83c01,libtinystr-2cf2777381234737,liblitemap-bc100b32c3f97fac,libwriteable-a52f6d98e9661a1a,libzerovec-597d45153ac865a7,libzerotrie-218f59a7cfea8e78,libyoke-fcff048bcc363888,libstable_deref_trait-017ced0fa0571f24,libzerofrom-ac19be1ca0dfc650,libform_urlencoded-bd0d0fbbafd906ce,libsync_wrapper-55a890af0295b61d,libuuid-0b66264e995123c1,libmd5-6eccc4d1e631f5a6,libdigest-10395b78375d1f22,libsubtle-23bbe7a6e0b68b65,libconst_oid-370fbf126c15b32d,libblock_buffer-e56da7547172d3f4,libcrypto_common-e0e228f8488d58de,libgeneric_array-7e23144c5c814a34,libtypenum-152f75a5de6f7595,libhttp_body-053cb9e3f8ed3998,libbackon-233256edcef81425,libduckdb-95603ea9f150e109,libcast-23c2d92181e4be65,librust_decimal-3a376e9eda03cbc8,libarrayvec-8dd224ed0246c278,libfallible_streaming_iterator-9cdaf58f18b7d270,libfallible_iterator-17e4eb4eb537d2df,libstrum-f004b271180ce62e,libarrow-e26ae7adc15735eb,libarrow_row-760ac79c4d87d72a,libarrow_string-aaecd9e5edee204c,libregex-f77a3b2038129dbb,libregex_automata-8c19a88e8ad2ac66,libaho_corasick-0ed81590f2ae18b2,libregex_syntax-6318a9bbcc70ced8,libarrow_cast-e0ce1c0c5a5d9dc0,libatoi-667e8fb5c0f7cc28,libbase64-5aabd82bcf109ae7,libcomfy_table-69681f41ade57e39,libunicode_width-535dbec589893981,libstrum-8b8379ce8ef4445c,liblexical_core-1264b85422e3b2f0,liblexical_write_float-3ee841bd625feb66,liblexical_write_integer-468ac8f136f41133,liblexical_parse_float-9af0b2e51353789e,liblexical_parse_integer-689c92091f66a68f,liblexical_util-3dc2d2ef1665e293,libarrow_arith-0a30238d3f424d4d,libarrow_ord-31adcb1135aea171,libarrow_select-0b5465830701d4f2,libarrow_array-b499d37295fbb251,libahash-91420521ef482c9a,libarrow_data-c7f07ee3675e2ee1,libarrow_schema-0f5c678e7d8ab040,libarrow_buffer-4d94fdf5a9c84fd8,libhalf-2cec35d15a3ba3a6,libnum-e31c77d560b6ed7e,libnum_iter-3bf292bd3a0b1ea4,libnum_rational-2f326e8f05386278,libnum_complex-3893fceabc412a37,libnum_bigint-7bfd5f2b4b0eae53,libnum_integer-2bc708a80e89d54d,liblibduckdb_sys-d2f9a4a2654aa606,libcrossbeam_channel-4a74ef3461b2451a,libcrossbeam_utils-b7dcd0e43924ba99,libdotenvy-fa31cbed99044194,libnanoid-369636c001a90ae4,librand-f7b13ee099f478c1,librand_chacha-ee16fdd86986addd,libppv_lite86-40603323d9bc0c77,libzerocopy-03d72ed3ef7ec978,librand_core-3f4bc136a9f8484b,libgetrandom-2244fa2595ff2ae5,libmockall-57000000ec03b54a,libpredicates_tree-fe4e274869d64127,libtermtree-b2cbb17b0c17fe59,libpredicates-5ffee91d19bdc757,libanstyle-65570d73b831b970,libpredicates_core-cdd3de0a0cc9c035,libfragile-6960d9b3dde335f1,libdowncast-ce412692fd2fad81,libdiesel-2e77d666e64ef451,libbyteorder-488d2f8c43033f2b,libr2d2-7047a2af7716e7e4,libscheduled_thread_pool-b7560a0f27b10c8b,libdowncast_rs-d956fda7291209d2,libpq_sys-613af9466fab2839,libchrono-2851518c3a894a06,libiana_time_zone-bed1638e18244077,libnum_traits-1b559d869beb4a95,libanyhow-b3ab1b3ee9153273,libconfig-b3f3a9edc1412b20,libpathdiff-2077368ba25a7e35,libjson5-bb97d519c5a16250,libpest-b5e01d7e0f8b9acc,libucd_trie-f7c5d35f723550c4,libserde_untagged-4a6c6bcb80d21995,liberased_serde-addc3b85e7772f16,libtypeid-159a1644fa1de970,libron-eda3421753d935b1,libbase64-0aaa390cbd040b05,libtoml-11d123cbace28d1f,libtoml_parser-84870e95f74b4dab,libserde_spanned-aa5879e78c196976,libtoml_datetime-70432bb51aa8c02f,libini-41fe15a0b123963d,libordered_multimap-d7df216402c6ef24,libhashbrown-636f0ae3866e6f51,libdlv_list-ab975e9b23150c72,libyaml_rust2-35f01721fb7d1a41,libarraydeque-9874e418356542b1,libhashlink-34f8ccc830689adc,libhashbrown-7cc99169ca7078db,libfoldhash-113770b6792d23fa,libconvert_case-98659eae1c820f18,libunicode_segmentation-b4ff539827683099,librocket-1e5fd66656b5db40,libnum_cpus-6387fee21d1a0ad8,libtempfile-8074f510846ee675,libgetrandom-2ce7e6772d0c2425,libfastrand-63c674c69accf2ad,librustix-c9b2c583ad8755a3,libbitflags-afc9ce5c1d0c623b,liblinux_raw_sys-9a541f4ad566f31f,libserde_json-3d64847e6513f4f1,libryu-21bbdb07b7fa6bff,librocket_http-8a3a24add199c040,libcookie-6f1bfbfc26c971ee,libstable_pattern-bece7a67d83edd81,libref_cast-91d5091170254cf5,libpercent_encoding-ad967e46737740dd,libhyper-bc3ee1bb594b9cef,libsocket2-7ef03947986246ee,libh2-5244c0590e467da7,libtower_service-68697946deac773c,libtracing-d1be752364854f9a,libtracing_core-0a0894d85926a626,libonce_cell-918b1d32890f5f9e,libhttp_body-fecb75fabf17fd50,libhttpdate-9c38e1c21d90a8e0,libhttp-255fd2d49b9c9b98,libmulter-dc063061b16b8c25,libmime-3c061a1728c96449,libtokio_util-e05c9aed2d20433d,libhttparse-f765f69e51c8e6db,libspin-5ee876b9442aaf34,libencoding_rs-524161b9015a3f42,libhttp-904eaef600962caf,libfnv-4bf9ac57e0281eeb,libeither-89c65796d86a9c5f,libtokio_stream-8da1b17b994df557,libatomic-66cbc3942dd4d353,libstate-4f6b7fdcfbc81fcb,libparking_lot-dac38b95d2132e90,libparking_lot_core-cdb53501ce5dd3d3,libcfg_if-60f5305215eca12c,libsmallvec-d53022e7c549079b,liblock_api-56202e6727c2ce82,libscopeguard-eba723b2b8e43bcd,libubyte-8238217ac05a6acf,liblog-aad30686f737d430,libtime-cb5617d4825c11e4,libitoa-40fdb2a09307e577,libtime_core-be06e82fde38b763,libnum_conv-a1775535d8d4be76,libderanged-7bd7c8530ef0dc97,libpowerfmt-89450eb64bb1f678,libfigment-7d55d05e23e020cd,libtoml-9ce9d12b8941fe4b,libtoml_edit-8361924b848be875,libserde_spanned-3ab11f909bb27821,libindexmap-f412c1254fe67957,libequivalent-efa3f710fdf26075,libhashbrown-6a661c2b0deea163,libwinnow-5a7f45c1a3478eaf,libtoml_write-82dc00e761d89a9f,libtoml_datetime-fd870c4ed5df893a,libuncased-6be05f8539468c86,libpear-6a8df34eb094490e,libinlinable_string-56d105fc07a37f0d,libserde-fff0f7d1cc17c00b,libserde_core-41a3274965227519,libtokio-6d23371785a8c671,libsignal_hook_registry-19c82e32b9ed1172,libsocket2-f423ee368e086efe,libbytes-b337d0e4a1d885b9,libmio-ca722db78bb7e719,libfutures-59583a8e3cbe66be,libfutures_executor-0cad4b417028c71c,libfutures_util-5e047ee12fcb02b3,libmemchr-ccc31091145e243c,libfutures_io-4d3aa44137d734c5,libslab-95d912b52b35c49c,libfutures_channel-33102af0aaf9dea2,libfutures_sink-7c593bcd15c8e07a,libfutures_task-abd7a6a3eeb97385,libpin_utils-a287e5d51f28dc7e,libasync_stream-0fbb92eebe43219d,libpin_project_lite-c14b565038496e22,libfutures_core-a0306a7584cf855d,libyansi-584d76a9158e288b,libis_terminal-7542f11ce0673d82,liblibc-575f2dcbb5441d05}.rlib" "<sysroot>/lib/rustlib/x86_64-unknown-linux-gnu/lib/{libstd-*,libpanic_unwind-*,libobject-*,libmemchr-*,libaddr2line-*,libgimli-*,librustc_demangle-*,libstd_detect-*,libhashbrown-*,librustc_std_workspace_alloc-*,libminiz_oxide-*,libadler2-*,libunwind-*,libcfg_if-*,liblibc-*,librustc_std_workspace_core-*,liballoc-*,libcore-*,libcompiler_builtins-*}.rlib" "-Wl,-Bdynamic" "-lstdc++" "-lpq" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-L" "/tmp/rustcJw9CYQ/raw-dylibs" "-B<sysroot>/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld" "-fuse-ld=lld" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/home/levit/actions-runner/_work/swan-lake/swan-lake/target/debug/build/blake3-c020cd84003e629c/out" "-L" "/home/levit/actions-runner/_work/swan-lake/swan-lake/target/debug/build/blake3-c020cd84003e629c/out" "-L" "/home/levit/actions-runner/_work/swan-lake/swan-lake/target/debug/build/libduckdb-sys-b01f71e00958eae1/out" "-L" "/home/levit/actions-runner/_work/swan-lake/swan-lake/target/debug/build/ring-3e41e38a32e34795/out" "-L" "<sysroot>/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/home/levit/actions-runner/_work/swan-lake/swan-lake/target/debug/deps/swan_lake-ed37ca88dbfde654" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs"
  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: rust-lld: error: unable to find library -lpq
          collect2: error: ld returned 1 exit status

需要安装一个 libpq-dev 库:

1
sudo apt-get update && sudo apt-get install -y libpq-dev

现在终于可以运行成功了!🥰

总结

缓存还是上传的github云端, 还是受到容量限制, 并没有解决我们的痛点, 下次行动前应该多调研一下😇

存在问题:

  1. 环境不一定匹配, 不一定能够无缝迁移, 需要自己进行一些额外配置

  2. 缓存不能存在本地, 需要上传云端(这里就很难受, 还是受到10G容量的限制, 并且上传受到带宽限制, 十分耗时)

自托管服务器适合的场景:

  1. 项目体量大, 每天在CI上的耗时很长, 需要自托管服务器来降本. 因为Github提供的免费运行时是2000分钟(个人免费用户), 超过之后按量计费, 价格还是比较高的.

  2. 项目对于性能或者网络等配置有特定要求

所以说对于个人开发者和小项目基本没必要上自托管服务器, 直接用Github提供的就行了, 省时省力.