OSS

Flink

If you have already configured oss access through Flink (Via Flink FileSystem), here you can skip the following configuration.

Put paimon-oss-0.9.0.jar into lib directory of your Flink home, and create catalog:


1. CREATE CATALOG my_catalog WITH (
2. 'type' = 'paimon',
3. 'warehouse' = 'oss://<bucket>/<path>',
4. 'fs.oss.endpoint' = 'oss-cn-hangzhou.aliyuncs.com',
5. 'fs.oss.accessKeyId' = 'xxx',
6. 'fs.oss.accessKeySecret' = 'yyy'
7. );

Spark

If you have already configured oss access through Spark (Via Hadoop FileSystem), here you can skip the following configuration.

Place paimon-oss-0.9.0.jar together with paimon-spark-0.9.0.jar under Spark’s jars directory, and start like


1. spark-sql \
2. --conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
3. --conf spark.sql.catalog.paimon.warehouse=oss://<bucket>/<path> \
4. --conf spark.sql.catalog.paimon.fs.oss.endpoint=oss-cn-hangzhou.aliyuncs.com \
5. --conf spark.sql.catalog.paimon.fs.oss.accessKeyId=xxx \
6. --conf spark.sql.catalog.paimon.fs.oss.accessKeySecret=yyy

Hive

If you have already configured oss access through Hive (Via Hadoop FileSystem), here you can skip the following configuration.

NOTE: You need to ensure that Hive metastore can access oss.

Place paimon-oss-0.9.0.jar together with paimon-hive-connector-0.9.0.jar under Hive’s auxlib directory, and start like


1. SET paimon.fs.oss.endpoint=oss-cn-hangzhou.aliyuncs.com;
2. SET paimon.fs.oss.accessKeyId=xxx;
3. SET paimon.fs.oss.accessKeySecret=yyy;

And read table from hive metastore, table can be created by Flink or Spark, see Catalog with Hive Metastore


1. SELECT * FROM test_table;
2. SELECT COUNT(1) FROM test_table;

Trino

From version 0.8, paimon-trino uses trino filesystem as basic file read and write system. We strongly recommend you to use jindo-sdk in trino.

You can find How to config jindo sdk on trino here. Please note that:

Use paimon to replace hive-hadoop2 when you decompress the plugin jar and find location to put in.
You can specify the core-site.xml in paimon.properties on configuration hive.config.resources.
Presto and Jindo use the same configuration method.