19900472
CustomerApi.Jobs.UpdateUsage
Queue
update_usage
Attempt
5 of 5
Priority
0
Tags
...
Node
customer_api@10.10.0.159
Queue Time
00:00.377
Run Time
00:05.694
Inserted
13h ago
Scheduled
13h ago
Completed
—
Cancelled
—
Discarded
13h ago
Args
%{"account_id" => "38184", "salesforce_id" => "a248c000002VUuTAAW"}
Meta
%{"orig_scheduled_at" => 1772798405689023}
Errors
Attempt 5—13h ago
** (MatchError) no match of right hand side value:
{:error,
[
%{
"errorCode" => "UNABLE_TO_LOCK_ROW",
"fields" => [],
"message" => "unable to obtain exclusive access to this record or 1 records: 0018c00002IqwU5AAJ"
}
]}
(customer_api 1.21.0+0fcb3e63) lib/customer_api/jobs/update_usage.ex:10: CustomerApi.Jobs.UpdateUsage.process/1
(oban_pro 1.6.12) lib/oban/pro/worker.ex:1156: Oban.Pro.Worker.process/3
(oban 2.20.3) lib/oban/queue/executor.ex:145: Oban.Queue.Executor.perform/1
(oban 2.20.3) lib/oban/queue/executor.ex:77: Oban.Queue.Executor.call/1
(elixir 1.19.5) lib/task/supervised.ex:105: Task.Supervised.invoke_mfa/2
(elixir 1.19.5) lib/task/supervised.ex:40: Task.Supervised.reply/4
Attempt 4—13h ago
** (Ch.Error) Code: 74. DB::Exception: Cannot read from file /mnt/clickhouse-cache/sharedS3DiskCache/41d/41d44fb28d710916cab2a82cedfc9dde/7004487680: , errno: 5, strerror: Input/output error: Cache info: Buffer path: ch-s3-376/aa8129e2-f47e-4b75-bbcb-8441591d3836/mergetree/gsx/tsibrrwjkpxgrncvixrteckpwmhmt, hash key: 41d44fb28d710916cab2a82cedfc9dde, file_offset_of_buffer_end: 7004751286, read_until_position: 7005697694, internal buffer end: 263606, read_type: CACHED, last caller: 4f654dcd-46f5-44cc-a7e7-028e2c44c4e9:977, file segment info: File segment: [7004487680, 7008681983], key: 41d44fb28d710916cab2a82cedfc9dde, state: DOWNLOADED, downloaded size: 4194304, reserved size: 4194304, downloader id: None, current write offset: 7008681984, caller id: 4f654dcd-46f5-44cc-a7e7-028e2c44c4e9:977, kind: Regular, unbound: 0: (while reading column timestamp): (attempt to read data part all_25410_31001_5_100676293 (state Active) failed with error. It can mean that some retryable error happened, or data is corrupted. Please retry the query: While executing MergeTreeSelect(pool: PrefetchedReadPool, algorithm: Thread). (CANNOT_READ_FROM_FILE_DESCRIPTOR) (version 25.10.1.7375 (official build))
(ecto_sql 3.13.4) lib/ecto/adapters/sql.ex:1113: Ecto.Adapters.SQL.raise_sql_call_error/1
(ecto_ch 0.8.6) lib/ecto/adapters/clickhouse.ex:323: Ecto.Adapters.ClickHouse.execute/5
(ecto 3.13.5) lib/ecto/repo/queryable.ex:241: Ecto.Repo.Queryable.execute/4
(ecto 3.13.5) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
(ecto 3.13.5) lib/ecto/repo/queryable.ex:163: Ecto.Repo.Queryable.one/3
(customer_api 1.21.0+0fcb3e63) lib/customer_api/services/account_usage.ex:339: CustomerApi.Services.AccountUsage.ch_usage_with_account_metrics/1
(customer_api 1.21.0+0fcb3e63) lib/customer_api/services/account_usage.ex:42: CustomerApi.Services.AccountUsage.feature_usage/1
(customer_api 1.21.0+0fcb3e63) lib/customer_api/jobs/update_usage.ex:8: CustomerApi.Jobs.UpdateUsage.process/1
Attempt 3—13h ago
** (Ch.Error) Code: 74. DB::Exception: Cannot read from file /mnt/clickhouse-cache/sharedS3DiskCache/41d/41d44fb28d710916cab2a82cedfc9dde/7004487680: , errno: 5, strerror: Input/output error: Cache info: Buffer path: ch-s3-376/aa8129e2-f47e-4b75-bbcb-8441591d3836/mergetree/gsx/tsibrrwjkpxgrncvixrteckpwmhmt, hash key: 41d44fb28d710916cab2a82cedfc9dde, file_offset_of_buffer_end: 7004751286, read_until_position: 7005697694, internal buffer end: 263606, read_type: CACHED, last caller: d3798af2-7245-4627-95f1-77bc729de08d:2065, file segment info: File segment: [7004487680, 7008681983], key: 41d44fb28d710916cab2a82cedfc9dde, state: DOWNLOADED, downloaded size: 4194304, reserved size: 4194304, downloader id: None, current write offset: 7008681984, caller id: d3798af2-7245-4627-95f1-77bc729de08d:2065, kind: Regular, unbound: 0: (while reading column timestamp): (attempt to read data part all_25410_31001_5_100676293 (state Active) failed with error. It can mean that some retryable error happened, or data is corrupted. Please retry the query: While executing MergeTreeSelect(pool: PrefetchedReadPool, algorithm: Thread). (CANNOT_READ_FROM_FILE_DESCRIPTOR) (version 25.10.1.7375 (official build))
(ecto_sql 3.13.4) lib/ecto/adapters/sql.ex:1113: Ecto.Adapters.SQL.raise_sql_call_error/1
(ecto_ch 0.8.6) lib/ecto/adapters/clickhouse.ex:323: Ecto.Adapters.ClickHouse.execute/5
(ecto 3.13.5) lib/ecto/repo/queryable.ex:241: Ecto.Repo.Queryable.execute/4
(ecto 3.13.5) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
(ecto 3.13.5) lib/ecto/repo/queryable.ex:163: Ecto.Repo.Queryable.one/3
(customer_api 1.21.0+0fcb3e63) lib/customer_api/services/account_usage.ex:339: CustomerApi.Services.AccountUsage.ch_usage_with_account_metrics/1
(customer_api 1.21.0+0fcb3e63) lib/customer_api/services/account_usage.ex:42: CustomerApi.Services.AccountUsage.feature_usage/1
(customer_api 1.21.0+0fcb3e63) lib/customer_api/jobs/update_usage.ex:8: CustomerApi.Jobs.UpdateUsage.process/1
Attempt 2—13h ago
** (Ch.Error) Code: 74. DB::Exception: Cannot read from file /mnt/clickhouse-cache/sharedS3DiskCache/41d/41d44fb28d710916cab2a82cedfc9dde/7004487680: , errno: 5, strerror: Input/output error: Cache info: Buffer path: ch-s3-376/aa8129e2-f47e-4b75-bbcb-8441591d3836/mergetree/gsx/tsibrrwjkpxgrncvixrteckpwmhmt, hash key: 41d44fb28d710916cab2a82cedfc9dde, file_offset_of_buffer_end: 7006584832, read_until_position: 7007099300, internal buffer end: 2097152, read_type: CACHED, last caller: 1dc3994a-9881-4620-a2e1-250763289696:43166, file segment info: File segment: [7004487680, 7008681983], key: 41d44fb28d710916cab2a82cedfc9dde, state: DOWNLOADED, downloaded size: 4194304, reserved size: 4194304, downloader id: None, current write offset: 7008681984, caller id: 1dc3994a-9881-4620-a2e1-250763289696:43166, kind: Regular, unbound: 0: While reading or decompressing ch-s3-376/aa8129e2-f47e-4b75-bbcb-8441591d3836/mergetree/gsx/tsibrrwjkpxgrncvixrteckpwmhmt (position: 7006584832, typename: DB::AsynchronousBoundedReadBuffer, compressed data header: <uninitialized>): (while reading column timestamp): (attempt to read data part all_25410_31001_5_100676293 (state Active) failed with error. It can mean that some retryable error happened, or data is corrupted. Please retry the query: While executing MergeTreeSelect(pool: PrefetchedReadPool, algorithm: Thread). (CANNOT_READ_FROM_FILE_DESCRIPTOR) (version 25.10.1.7375 (official build))
(ecto_sql 3.13.4) lib/ecto/adapters/sql.ex:1113: Ecto.Adapters.SQL.raise_sql_call_error/1
(ecto_ch 0.8.6) lib/ecto/adapters/clickhouse.ex:323: Ecto.Adapters.ClickHouse.execute/5
(ecto 3.13.5) lib/ecto/repo/queryable.ex:241: Ecto.Repo.Queryable.execute/4
(ecto 3.13.5) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
(ecto 3.13.5) lib/ecto/repo/queryable.ex:163: Ecto.Repo.Queryable.one/3
(customer_api 1.21.0+0fcb3e63) lib/customer_api/services/account_usage.ex:339: CustomerApi.Services.AccountUsage.ch_usage_with_account_metrics/1
(customer_api 1.21.0+0fcb3e63) lib/customer_api/services/account_usage.ex:42: CustomerApi.Services.AccountUsage.feature_usage/1
(customer_api 1.21.0+0fcb3e63) lib/customer_api/jobs/update_usage.ex:8: CustomerApi.Jobs.UpdateUsage.process/1
Attempt 1—13h ago
** (Ch.Error) Code: 74. DB::Exception: Cannot read from file /mnt/clickhouse-cache/sharedS3DiskCache/6c5/6c543e16b07e23c540acb15065003505/6488588288: , errno: 5, strerror: Input/output error: Cache info: Buffer path: ch-s3-8c1/aa8129e2-f47e-4b75-bbcb-8441591d3836/mergetree/wmm/otjakdkhehvnzblfbazpsdxzhrdvx, hash key: 6c543e16b07e23c540acb15065003505, file_offset_of_buffer_end: 6490938227, read_until_position: 6491411216, internal buffer end: 2349939, read_type: CACHED, last caller: 35266f85-4350-45b3-a102-382321c274d8:2107, file segment info: File segment: [6488588288, 6492782591], key: 6c543e16b07e23c540acb15065003505, state: DOWNLOADED, downloaded size: 4194304, reserved size: 4194304, downloader id: None, current write offset: 6492782592, caller id: 35266f85-4350-45b3-a102-382321c274d8:2107, kind: Regular, unbound: 0: (while reading column timestamp): (attempt to read data part all_35815_41324_5_100676293 (state Active) failed with error. It can mean that some retryable error happened, or data is corrupted. Please retry the query: While executing MergeTreeSelect(pool: PrefetchedReadPool, algorithm: Thread). (CANNOT_READ_FROM_FILE_DESCRIPTOR) (version 25.10.1.7375 (official build))
(ecto_sql 3.13.4) lib/ecto/adapters/sql.ex:1113: Ecto.Adapters.SQL.raise_sql_call_error/1
(ecto_ch 0.8.6) lib/ecto/adapters/clickhouse.ex:323: Ecto.Adapters.ClickHouse.execute/5
(ecto 3.13.5) lib/ecto/repo/queryable.ex:241: Ecto.Repo.Queryable.execute/4
(ecto 3.13.5) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
(ecto 3.13.5) lib/ecto/repo/queryable.ex:163: Ecto.Repo.Queryable.one/3
(customer_api 1.21.0+0fcb3e63) lib/customer_api/services/account_usage.ex:339: CustomerApi.Services.AccountUsage.ch_usage_with_account_metrics/1
(customer_api 1.21.0+0fcb3e63) lib/customer_api/services/account_usage.ex:42: CustomerApi.Services.AccountUsage.feature_usage/1
(customer_api 1.21.0+0fcb3e63) lib/customer_api/jobs/update_usage.ex:8: CustomerApi.Jobs.UpdateUsage.process/1