Greenplum通过gp_dist_random('gp_id') 在所有节点调用某个函数

简介:

使用greenplum时,如果需要调用一个函数,这个函数很可能就在master执行,而不会跑到segment上去执行。
例如 random()函数。
通过select random()来调用的话,不需要将这条SQL发送到segment节点,所以执行计划如下,没有gather motion的过程。

postgres=# explain analyze select random();  
                                       QUERY PLAN                                         
----------------------------------------------------------------------------------------  
 Result  (cost=0.01..0.02 rows=1 width=0)  
   Rows out:  1 rows with 0.017 ms to end, start offset by 0.056 ms.  
   InitPlan  
     ->  Result  (cost=0.00..0.01 rows=1 width=0)  
           Rows out:  1 rows with 0.004 ms to end of 2 scans, start offset by 0.059 ms.  
 Slice statistics:  
   (slice0)    Executor memory: 29K bytes.  
   (slice1)    Executor memory: 29K bytes.  
 Statement statistics:  
   Memory used: 128000K bytes  
 Total runtime: 0.074 ms  
(11 rows)  

如果要让这个函数在segment执行,怎么办呢?
通过gp_dist_random('gp_id')来调用,gp_dist_random的参数是一个可查询的视图,或表。

postgres=# explain analyze select random() from gp_dist_random('gp_id');  
                                                               QUERY PLAN                                                                  
-----------------------------------------------------------------------------------------------------------------------------------------  
 Gather Motion 240:1  (slice1; segments: 240)  (cost=0.00..4.00 rows=240 width=0)  
   Rows out:  240 rows at destination with 6.336 ms to first row, 59 ms to end, start offset by 4195 ms.  
   ->  Seq Scan on gp_id  (cost=0.00..4.00 rows=1 width=0)  
         Rows out:  Avg 1.0 rows x 240 workers.  Max 1 rows (seg0) with 0.073 ms to first row, 0.075 ms to end, start offset by 4207 ms.  
 Slice statistics:  
   (slice0)    Executor memory: 471K bytes.  
   (slice1)    Executor memory: 163K bytes avg x 240 workers, 163K bytes max (seg0).  
 Statement statistics:  
   Memory used: 128000K bytes  
 Total runtime: 4279.445 ms  
(10 rows)  

gp_id在每个segment中都有一条记录,所以以上SQL会在每个SEGMENT中调用一次random()并返回所有结果,例如我的测试环境中有240个segment, 那么以上SQL将返回240条记录。

在gp_id的定义中,介绍了gp_dist_random用它可以做一些管理的工作:
譬如查询数据库的大小,查询表的大小,其实都是这样统计的。
src/backend/catalog/postgres_bki_srcs

/*-------------------------------------------------------------------------  
 *  
 * gp_id.h  
 *        definition of the system "database identifier" relation (gp_dbid)  
 *        along with the relation's initial contents.  
 *  
 * Copyright (c) 2009-2010, Greenplum inc  
 *  
 * NOTES  
 *    Historically this table was used to supply every segment with its  
 * identification information.  However in the 4.0 release when the file  
 * replication feature was added it could no longer serve this purpose  
 * because it became a requirement for all tables to have the same physical  
 * contents on both the primary and mirror segments.  To resolve this the  
 * information is now passed to each segment on startup based on the  
 * gp_segment_configuration (stored on the master only), and each segment  
 * has a file in its datadirectory (gp_dbid) that uniquely identifies the  
 * segment.  
 *  
 *   The contents of the table are now irrelevant, with the exception that  
 * several tools began relying on this table for use as a method of remote  
 * function invocation via gp_dist_random('gp_id') due to the fact that this  
 * table was guaranteed of having exactly one row on every segment.  The  
 * contents of the row have no defined meaning, but this property is still  
 * relied upon.  
 */  
#ifndef _GP_ID_H_  
#define _GP_ID_H_  
  
  
#include "catalog/genbki.h"  
/*  
 * Defines for gp_id table  
 */  
#define GpIdRelationName                        "gp_id"  
  
/* TIDYCAT_BEGINFAKEDEF  
  
   CREATE TABLE gp_id  
   with (shared=true, oid=false, relid=5001, content=SEGMENT_LOCAL)  
   (  
   gpname       name     ,  
   numsegments  smallint ,  
   dbid         smallint ,  
   content      smallint   
   );  
  
   TIDYCAT_ENDFAKEDEF  
*/  

查询数据库大小的GP函数

postgres=# \df+ pg_database_size  
                                                                                                     List of functions  
   Schema   |       Name       | Result data type | Argument data types |  Type  |  Data access   | Volatility |  Owner   | Language |      Source code      |                         Description                           
------------+------------------+------------------+---------------------+--------+----------------+------------+----------+----------+-----------------------+-------------------------------------------------------------  
 pg_catalog | pg_database_size | bigint           | name                | normal | reads sql data | volatile   | dege.zzz | internal | pg_database_size_name | Calculate total disk space usage for the specified database  
 pg_catalog | pg_database_size | bigint           | oid                 | normal | reads sql data | volatile   | dege.zzz | internal | pg_database_size_oid  | Calculate total disk space usage for the specified database  
(2 rows)  

其中pg_database_size_name 的源码如下:
很明显,在统计数据库大小时也用到了select sum(pg_database_size('%s'))::int8 from gp_dist_random('gp_id');

Datum  
pg_database_size_name(PG_FUNCTION_ARGS)  
{  
        int64           size = 0;  
        Name            dbName = PG_GETARG_NAME(0);  
        Oid                     dbOid = get_database_oid(NameStr(*dbName));  
  
        if (!OidIsValid(dbOid))  
                ereport(ERROR,  
                                (errcode(ERRCODE_UNDEFINED_DATABASE),  
                                 errmsg("database \"%s\" does not exist",  
                                                NameStr(*dbName))));  
                                                  
        size = calculate_database_size(dbOid);  
          
        if (Gp_role == GP_ROLE_DISPATCH)  
        {  
                StringInfoData buffer;  
                  
                initStringInfo(&buffer);  
  
                appendStringInfo(&buffer, "select sum(pg_database_size('%s'))::int8 from gp_dist_random('gp_id');", NameStr(*dbName));  
  
                size += get_size_from_segDBs(buffer.data);  
        }  
  
        PG_RETURN_INT64(size);  
}  

不信我们可以直接查询这个SQL,和使用pg_database_size函数得到的结果几乎是一样的,只差了calculate_database_size的部分。

postgres=# select sum(pg_database_size('postgres'))::int8 from gp_dist_random('gp_id');  
      sum         
----------------  
 16006753522624  
(1 row)  
  
postgres=# select pg_database_size('postgres');  
 pg_database_size   
------------------  
   16006763924106  
(1 row)  

gp_dist_random('gp_id')本质上就是在所有节点查询gp_id,
gp_dist_random('pg_authid')就是在所有节点查询pg_authid,
例如:

postgres=# select * from gp_dist_random('gp_id');  
  gpname   | numsegments | dbid | content   
-----------+-------------+------+---------  
 Greenplum |          -1 |   -1 |      -1  
 Greenplum |          -1 |   -1 |      -1  
 Greenplum |          -1 |   -1 |      -1  
 Greenplum |          -1 |   -1 |      -1  
 Greenplum |          -1 |   -1 |      -1  
 Greenplum |          -1 |   -1 |      -1  
 Greenplum |          -1 |   -1 |      -1  
 Greenplum |          -1 |   -1 |      -1  
 Greenplum |          -1 |   -1 |      -1  
 Greenplum |          -1 |   -1 |      -1  
。。。。。。  

如果不想返回太多记录,可以使用limit 来过滤,但是执行还是会在所有的segment都执行,如下:

postgres=# explain analyze select random() from gp_dist_random('gp_id') limit 1;  
                                                                  QUERY PLAN                                                                     
-----------------------------------------------------------------------------------------------------------------------------------------------  
 Limit  (cost=0.00..0.04 rows=1 width=0)  
   Rows out:  1 rows with 5.865 ms to first row, 5.884 ms to end, start offset by 4212 ms.  
   ->  Gather Motion 240:1  (slice1; segments: 240)  (cost=0.00..0.04 rows=1 width=0)  
         Rows out:  1 rows at destination with 5.857 ms to end, start offset by 4212 ms.  
         ->  Limit  (cost=0.00..0.02 rows=1 width=0)  
               Rows out:  Avg 1.0 rows x 240 workers.  Max 1 rows (seg0) with 0.062 ms to first row, 0.063 ms to end, start offset by 4228 ms.  
               ->  Seq Scan on gp_id  (cost=0.00..4.00 rows=1 width=0)  
                     Rows out:  Avg 1.0 rows x 240 workers.  Max 1 rows (seg0) with 0.060 ms to end, start offset by 4228 ms.  
 Slice statistics:  
   (slice0)    Executor memory: 463K bytes.  
   (slice1)    Executor memory: 163K bytes avg x 240 workers, 163K bytes max (seg0).  
 Statement statistics:  
   Memory used: 128000K bytes  
 Total runtime: 4288.007 ms  
(14 rows)  
目录
相关文章
|
SQL 关系型数据库 MySQL
将MySQL 数据迁移到 PostgreSQL
将MySQL 数据迁移到 PostgreSQL 可以采用以下步骤: 安装 PostgreSQL 数据库:首先,需要安装 PostgreSQL 数据库。可以从官方网站(https://www.postgresql.org/)下载最新版本的 PostgreSQL,并根据官方指南进行安装。 创建 PostgreSQL 数据库:在 PostgreSQL 中创建与 MySQL 数据库相对应的数据库。可以使用 pgAdmin 或命令行工具(如 psql)来创建数据库。例如,如果在 MySQL 中有一个名为 "mydb" 的数据库,那么可以在 PostgreSQL 中创建一个具有相同名称的数据库。 导
6688 0
|
Linux
解决CentOS yum安装Mysql8提示“公钥尚未安装”或“密钥已安装,但是不适用于此软件包”的问题
解决CentOS yum安装Mysql8提示“公钥尚未安装”或“密钥已安装,但是不适用于此软件包”的问题
6300 0
|
关系型数据库 PostgreSQL
PostgreSQL listagg within group (order by) 聚合兼容用法 string_agg ( order by) - 行列变换,CSV构造...
标签 PostgreSQL , order-set agg , listagg , string_agg , order 背景 listagg — Rows to Delimited Strings The listagg function transforms values from a g...
7469 0
|
存储 关系型数据库 MySQL
【MySQL】在 Centos7 环境安装 MySQL -- 详细完整教程
【MySQL】在 Centos7 环境安装 MySQL -- 详细完整教程
|
运维 Linux 虚拟化
VMware虚拟机安装教程,Windows下安装VMware虚拟机,附VMware下载,Windows各版本系统镜像下载
虚拟机技术允许一台物理机运行多个操作系统,提升资源利用率,节省成本。通过快照、克隆等功能,实现系统快速恢复与复制,提高运维效率。本文详细介绍VMware虚拟机的安装步骤、Windows镜像下载及系统安装激活流程,适合初学者快速入门。
13371 0
|
Docker Windows 容器
clash代理docker实现镜像拉取
clash代理docker实现镜像拉取
7496 1
|
关系型数据库 Linux 数据库
|
存储 运维 分布式计算
HDFS 如何容错?
【8月更文挑战第31天】
946 0
|
存储 Linux Shell
【Linux】VScode远程连接Linux
文指导读者如何在VScode中下载安装、配置插件以实现与Linux的远程连接,包括使用XShell、安装必要的插件如C/C++ExtensionPack,并介绍了基本操作和VScode内置命令行功能。
763 0
|
Linux
如何检查CentOS版本:5种方法
这个文件包含了CentOS的详细版本信息,包括版本号、架构等。
4632 0