Skip to content

[Improvement][Seatunnel-web] Bump seatunnel version from 2.3.8 to 2.3.10 #291

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 16 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,12 @@ Click it if your want to know more about our design. 👉🏻[Design](https://gi

## Compatibility

| SeaTunnel Web Version | SeaTunnel Version | Doc |
|-----------------------|------------------------|------------------------------------------------------------------------------|
| 1.0.3-SNAPSHOT | 2.3.8 / 2.3.9-SNAPSHOT | [Docs](https://github.com/apache/seatunnel-web/blob/main/README.md) |
| 1.0.2 | 2.3.8 | [Docs](https://github.com/apache/seatunnel-web/blob/1.0.2-release/README.md) |
| 1.0.1 | 2.3.3 | [Docs](https://github.com/apache/seatunnel-web/blob/1.0.1-release/README.md) |
| 1.0.0 | 2.3.3 | [Docs](https://github.com/apache/seatunnel-web/blob/1.0.0-release/README.md) |
| SeaTunnel Web Version | SeaTunnel Version | Doc |
|-----------------------|--------------------------|------------------------------------------------------------------------------|
| 1.0.3-SNAPSHOT | 2.3.10 / 2.3.11-SNAPSHOT | [Docs](https://github.com/apache/seatunnel-web/blob/main/README.md) |
| 1.0.2 | 2.3.8 | [Docs](https://github.com/apache/seatunnel-web/blob/1.0.2-release/README.md) |
| 1.0.1 | 2.3.3 | [Docs](https://github.com/apache/seatunnel-web/blob/1.0.1-release/README.md) |
| 1.0.0 | 2.3.3 | [Docs](https://github.com/apache/seatunnel-web/blob/1.0.0-release/README.md) |

## How to start

Expand All @@ -52,16 +52,16 @@ You have two ways to get the SeaTunnel installer package. Build from source code
* Get the source package from https://seatunnel.apache.org/download or https://github.com/apache/seatunnel.git
* Please follow [Build SeaTunnel From Source Code](https://seatunnel.apache.org/docs/start-v2/locally/deployment#build-seatunnel-from-source-code) to build the SeaTunnel.
* After building, it is necessary to set an environment variable `ST_WEB_BASEDIR_PATH` to represent the location of the data source shade package. A custom class loader will be used to load the data source shade package based on this. For example: `ST_WEB_BASEDIR_PATH=/seatunnel-web-dist/target/apache-seatunnel-web-1.0.3-SNAPSHOT/`
* Then you can get the installer package in `${Your_code_dir}/seatunnel-dist/target`, For example:`apache-seatunnel-2.3.8-bin.tar.gz`
* Run `tar -zxvf apache-seatunnel-2.3.8-bin.tar.gz` to unzip the installer package.
* Run `cd apache-seatunnel-2.3.8 & sh bin/seatunnel-cluster.sh -d` to run the SeaTunnel Zeta Engine Server.
* Then you can get the installer package in `${Your_code_dir}/seatunnel-dist/target`, For example:`apache-seatunnel-2.3.10-bin.tar.gz`
* Run `tar -zxvf apache-seatunnel-2.3.10-bin.tar.gz` to unzip the installer package.
* Run `cd apache-seatunnel-2.3.10 & sh bin/seatunnel-cluster.sh -d` to run the SeaTunnel Zeta Engine Server.
* Please confirm that port 5801 is being monitored by the SeaTunnelServer process.

##### 2.1.2 Download installer package and deploy
The other way to install SeaTunnel Zeta Engine Server is download the installer package from https://seatunnel.apache.org/download and deploy.

* Download and install connector plugin(Some third-party dependency packages will also be automatically downloaded and installed during this process, such as hadoop jar). You can get the step from https://seatunnel.apache.org/docs/start-v2/locally/deployment.
* Run `cd apache-seatunnel-2.3.8 & sh bin/seatunnel-cluster.sh -d` to run the SeaTunnel Zeta Engine Server.
* Run `cd apache-seatunnel-2.3.10 & sh bin/seatunnel-cluster.sh -d` to run the SeaTunnel Zeta Engine Server.

#### 2.2 Init database

Expand All @@ -86,7 +86,7 @@ sh build.sh code
1. Edit `seatunnel-server/seatunnel-app/src/main/resources/application.yml` Fill in the database connection information
![img.png](docs/images/application_config.png)
2. Edit `seatunnel-server/seatunnel-app/src/main/resources/application.yml`, add `jwt.secretKey` value. Eg: https://github.com/apache/seatunnel(Notice that cannot be too short).
3. Copy `apache-seatunnel-2.3.8/connectors/plugin-mapping.properties` file to `seatunnel-web/seatunnel-server/seatunnel-app/src/main/resources` dir.
3. Copy `apache-seatunnel-2.3.10/connectors/plugin-mapping.properties` file to `seatunnel-web/seatunnel-server/seatunnel-app/src/main/resources` dir.
4. Run `seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/SeatunnelApplication.java` If there are no errors reported, the seatunnel web backend service is successfully started. Notice that, you must set `-DSEATUNNEL_HOME=${your_seatunnel_install_path}` like this:

![img.png](docs/images/idea_st_home.png)
Expand Down Expand Up @@ -127,21 +127,21 @@ You have two ways to get the SeaTunnel installer package. Build from source code
##### 3.1.1 Build from source code
* Get the source package from https://seatunnel.apache.org/download or https://github.com/apache/seatunnel.git
* Build installer package use maven command `./mvnw -U -T 1C clean install -DskipTests -D"maven.test.skip"=true -D"maven.javadoc.skip"=true -D"checkstyle.skip"=true -D"license.skipAddThirdParty" `
* Then you can get the installer package in `${Your_code_dir}/seatunnel-dist/target`, For example:`apache-seatunnel-2.3.8-bin.tar.gz`
* Then you can get the installer package in `${Your_code_dir}/seatunnel-dist/target`, For example:`apache-seatunnel-2.3.10-bin.tar.gz`

##### 3.1.2 Download installer package
The other way to get SeaTunnel Zeta Engine Server installer package is download the installer package from https://seatunnel.apache.org/download and install plugins online.

* Download and install connector plugin(Some third-party dependency packages will also be automatically downloaded and installed during this process, such as hadoop jar). You can get the step from https://seatunnel.apache.org/docs/start-v2/locally/deployment.
* After completing the previous step, you will receive an installation package that can be used to install SeaTunnel Zeta Engine Server on the server. Run `tar -zcvf apache-seatunnel-2.3.8-bin.tar.gz apache-seatunnel-2.3.8`
* After completing the previous step, you will receive an installation package that can be used to install SeaTunnel Zeta Engine Server on the server. Run `tar -zcvf apache-seatunnel-2.3.10-bin.tar.gz apache-seatunnel-2.3.10`

##### 3.1.3 Deploy SeaTunnel Zeta Server
After 3.1.1 or 3.1.2 you can get an installer package `apache-seatunnel-2.3.8-bin.tar.gz`, Then you can copy it to you server node and deploy reference https://seatunnel.apache.org/docs/seatunnel-engine/deployment.
After 3.1.1 or 3.1.2 you can get an installer package `apache-seatunnel-2.3.10-bin.tar.gz`, Then you can copy it to you server node and deploy reference https://seatunnel.apache.org/docs/seatunnel-engine/deployment.

##### 3.1.4 Deploy SeaTunnel Zeta Client In SeaTunnel Web Run Node
If you use SeaTunnel Web, you need deploy a SeaTunnel Zeta Client in the SeaTunnel Web run Node. **If you run SeaTunnel Zeta Server and SeaTunnel Web in same node, you can skip this step**.

* Copy `apache-seatunnel-2.3.8-bin.tar.gz` to the SeaTunnel Web node and unzip it **in the same path of SeaTunnel Zeta Server node**.
* Copy `apache-seatunnel-2.3.10-bin.tar.gz` to the SeaTunnel Web node and unzip it **in the same path of SeaTunnel Zeta Server node**.
* Set `SEATUNNEL_HOME` to environment variable like SeaTunnel Zeta Server node.
* Config `hazelcast-client.yaml` reference https://seatunnel.apache.org/docs/seatunnel-engine/deployment#6-config-seatunnel-engine-client
* Run `$SEATUNNEL_HOME/bin/seatunnel.sh --config $SEATUNNEL_HOME/config/v2.batch.config.template`, If this job run finished, it indicates successful client deployment.
Expand Down Expand Up @@ -183,7 +183,7 @@ tar -zxvf apache-seatunnel-web-${project.version}.tar.gz
![image](docs/images/application_config.png)

* Copy `$SEATUNNEL_HOME/config/hazelcast-client.yaml` to `apache-seatunnel-web-${project.version}/conf/`
* Copy `apache-seatunnel-2.3.8/connectors/plugin-mapping.properties` file to `apache-seatunnel-web-${project.version}/conf/` dir.
* Copy `apache-seatunnel-2.3.10/connectors/plugin-mapping.properties` file to `apache-seatunnel-web-${project.version}/conf/` dir.

#### 3.6 Start SeaTunnel Web

Expand Down
24 changes: 12 additions & 12 deletions README_CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,16 +42,16 @@ SeaTunnel是下一代超高性能、分布式、海量数据集成工具。它
* 从 https://seatunnel.apache.org/download 或 https://github.com/apache/seatunnel.git 获取源包
* 请按照 [从源码构建 SeaTunnel 来构建 SeaTunnel](https://seatunnel.apache.org/zh-CN/docs/start-v2/locally/deployment/#从源码构建seatunnel)。
* 在构建之后,需要设置一个环境变量`ST_WEB_BASEDIR_PATH`来表示数据源shade包的位置。将使用自定义类加载器来基于此加载数据源shade包。例如:`ST_WEB_BASEDIR_PATH=/seatunnel-web-dist/target/apache-seatunnel-web-1.0.3-SNAPSHOT/`
然后你可以在`${Your_code_dir}/seatunnel-dist/target`下获取安装包,例如:`apache-seatunnel-2.3.8-bin.tar.gz`。
* 执行`tar -zxvf apache-seatunnel-2.3.8-bin.tar.gz`解压安装包。
* 运行`cd apache-seatunnel-2.3.8 & sh bin/seatunnel-cluster.sh -d`运行SeaTunnel Zeta Engine Server。
然后你可以在`${Your_code_dir}/seatunnel-dist/target`下获取安装包,例如:`apache-seatunnel-2.3.10-bin.tar.gz`。
* 执行`tar -zxvf apache-seatunnel-2.3.10-bin.tar.gz`解压安装包。
* 运行`cd apache-seatunnel-2.3.10 & sh bin/seatunnel-cluster.sh -d`运行SeaTunnel Zeta Engine Server。
* SeaTunnel Zeta Engine Server默认占用5801端口,请确认端口5801正在被SeaTunnelServer进程占用。

##### 2.1.2 下载安装程序包并进行部署
下载安装包并部署SeaTunnel Zeta Engine Server的另一种安装方式是从 https://seatunnel.apache.org/download 下载安装包并部署。

* 下载并安装连接器插件(一些第三方依赖包也会在此过程中自动下载并安装,如Hadoop jar)。您可以从 https://seatunnel.apache.org/docs/2.3.8/start-v2/locally/deployment 获得该步骤。
* 运行`cd apache-seatunnel-2.3.8 & sh bin/seatunnel-cluster.sh -d`运行SeaTunnel Zeta Engine Server。
* 下载并安装连接器插件(一些第三方依赖包也会在此过程中自动下载并安装,如Hadoop jar)。您可以从 https://seatunnel.apache.org/docs/2.3.10/start-v2/locally/deployment 获得该步骤。
* 运行`cd apache-seatunnel-2.3.10 & sh bin/seatunnel-cluster.sh -d`运行SeaTunnel Zeta Engine Server。
#### 2.2 初始化数据库

1. 编辑 `seatunnel-server/seatunnel-app/src/main/resources/script/seatunnel_server_env.sh` 文件, 填写已安装的数据库 address, port, username, and password. 下面是一个例子:
Expand All @@ -77,7 +77,7 @@ sh build.sh code
![img.png](docs/images/application_config.png)

2. 编辑 `seatunnel-server/seatunnel-app/src/main/resources/application.yml` 添加 `jwt.secretKey` 值. 例如: `https://github.com/apache/seatunnel` (注意: 不能太短).
3. 复制 `apache-seatunnel-2.3.8/connectors/plugin-mapping.properties` 文件 到 `seatunnel-web/seatunnel-server/seatunnel-app/src/main/resources` 目录.
3. 复制 `apache-seatunnel-2.3.10/connectors/plugin-mapping.properties` 文件 到 `seatunnel-web/seatunnel-server/seatunnel-app/src/main/resources` 目录.
4. 运行 `seatunnel-server/seatunnel-app/src/main/java/org/apache/seatunnel/app/SeatunnelApplication.java` 如果没有报错,说明seatunnel web后端服务启动成功。注意,你必须设置 `-DSEATUNNEL_HOME=${your_seatunnel_install_path}` 像这样:

![img.png](docs/images/idea_st_home.png)
Expand Down Expand Up @@ -118,23 +118,23 @@ npm run dev
##### 3.1.1 从源码编译
* 从 https://seatunnel.apache.org/download 或 https://github.com/apache/seatunnel.git 获取源码包
* 使用maven命令编译安装包 `./mvnw -U -T 1C clean install -DskipTests -D"maven.test.skip"=true -D"maven.javadoc.skip"=true -D"checkstyle.skip"=true -D"license.skipAddThirdParty" `
* 然后您可以在`${您的代码目录}/seatunnel-dist/target`中获得安装包,例如:`apache-seatunnel-2.3.8-bin.tar.gz`
* 然后您可以在`${您的代码目录}/seatunnel-dist/target`中获得安装包,例如:`apache-seatunnel-2.3.10-bin.tar.gz`

##### 3.1.2 下载安装包
获取 SeaTunnel Zeta 引擎服务安装包的另一种方式是从 https://seatunnel.apache.org/download 下载安装包并在线安装插件。

* 下载并安装连接器插件(这个过程中也会自动下载和安装一些第三方依赖包,比如 hadoop jar)。您可以参考 https://seatunnel.apache.org/docs/2.3.8/start-v2/locally/deployment 获取详细步骤。
* 完成上述步骤后,您将获得一个可以用于在服务器上安装 SeaTunnel Zeta 引擎服务的安装包。运行 `tar -zcvf apache-seatunnel-2.3.8-bin.tar.gz apache-seatunnel-2.3.8`
* 下载并安装连接器插件(这个过程中也会自动下载和安装一些第三方依赖包,比如 hadoop jar)。您可以参考 https://seatunnel.apache.org/docs/2.3.10/start-v2/locally/deployment 获取详细步骤。
* 完成上述步骤后,您将获得一个可以用于在服务器上安装 SeaTunnel Zeta 引擎服务的安装包。运行 `tar -zcvf apache-seatunnel-2.3.10-bin.tar.gz apache-seatunnel-2.3.10`

##### 3.1.3 部署 SeaTunnel Zeta 服务端

在完成 3.1.1 或 3.1.2 后您可以获得一个安装包 `apache-seatunnel-2.3.8-bin.tar.gz`,然后您可以将其复制到服务器节点并参考 https://seatunnel.apache.org/docs/seatunnel-engine/deployment 进行部署。
在完成 3.1.1 或 3.1.2 后您可以获得一个安装包 `apache-seatunnel-2.3.10-bin.tar.gz`,然后您可以将其复制到服务器节点并参考 https://seatunnel.apache.org/docs/seatunnel-engine/deployment 进行部署。

##### 3.1.4 在 SeaTunnel Web 运行节点部署 SeaTunnel Zeta 客户端

如果您使用 SeaTunnel Web,您需要在 SeaTunnel Web 运行节点部署一个 SeaTunnel Zeta 客户端。**如果 SeaTunnel Zeta 服务端和 SeaTunnel Web 在同一节点,您可以跳过此步骤**。

* 将 `apache-seatunnel-2.3.8-bin.tar.gz` 复制到 SeaTunnel Web 节点并解压,解压**到与 SeaTunnel Zeta 服务端节点相同的路径下**。
* 将 `apache-seatunnel-2.3.10-bin.tar.gz` 复制到 SeaTunnel Web 节点并解压,解压**到与 SeaTunnel Zeta 服务端节点相同的路径下**。
* 和 SeaTunnel Zeta 服务端节点相同,设置 `SEATUNNEL_HOME` 环境变量。
* 参考 https://seatunnel.apache.org/docs/seatunnel-engine/deployment#6-config-seatunnel-engine-client 配置 `hazelcast-client.yaml`。
* 运行 `$SEATUNNEL_HOME/bin/seatunnel.sh --config $SEATUNNEL_HOME/config/v2.batch.config.template`,如果该作业运行结束,表示客户端部署成功。
Expand Down Expand Up @@ -176,7 +176,7 @@ tar -zxvf apache-seatunnel-web-${project.version}.tar.gz
![image](docs/images/application_config.png)

* 复制 `$SEATUNNEL_HOME/config/hazelcast-client.yaml` 到 `apache-seatunnel-web-${project.version}/conf/`
* 复制 `apache-seatunnel-2.3.8/connectors/plugin-mapping.properties` 文件到 `apache-seatunnel-web-${project.version}/conf/` 目录.
* 复制 `apache-seatunnel-2.3.10/connectors/plugin-mapping.properties` 文件到 `apache-seatunnel-web-${project.version}/conf/` 目录.

#### 3.6 运行 SeaTunnel Web

Expand Down
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@
<guava.version>33.2.1-jre</guava.version>
<checker.qual.version>3.10.0</checker.qual.version>
<awaitility.version>4.2.0</awaitility.version>
<seatunnel-framework.version>2.3.8</seatunnel-framework.version>
<seatunnel-framework.version>2.3.10</seatunnel-framework.version>
<oracle-jdbc.version>21.5.0.0</oracle-jdbc.version>
<postgresql.version>42.4.3</postgresql.version>
<sqlserver.version>9.2.1.jre8</sqlserver.version>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@

package org.apache.seatunnel.app.bean.connector;

import org.apache.seatunnel.api.common.PluginIdentifier;
import org.apache.seatunnel.api.configuration.util.OptionRule;
import org.apache.seatunnel.app.domain.response.connector.ConnectorFeature;
import org.apache.seatunnel.app.domain.response.connector.ConnectorInfo;
Expand All @@ -25,7 +26,6 @@
import org.apache.seatunnel.common.config.Common;
import org.apache.seatunnel.common.config.DeployMode;
import org.apache.seatunnel.common.constants.PluginType;
import org.apache.seatunnel.plugin.discovery.PluginIdentifier;
import org.apache.seatunnel.server.common.SeatunnelErrorEnum;
import org.apache.seatunnel.server.common.SeatunnelException;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
package org.apache.seatunnel.app.bean.env;

import org.apache.seatunnel.api.configuration.util.OptionRule;
import org.apache.seatunnel.api.env.EnvOptionRule;
import org.apache.seatunnel.api.options.EnvOptionRule;
import org.apache.seatunnel.app.dynamicforms.AbstractFormOption;
import org.apache.seatunnel.app.dynamicforms.FormStructure;
import org.apache.seatunnel.app.thirdparty.framework.SeaTunnelOptionRuleWrapper;
Expand All @@ -37,7 +37,7 @@ public class JobEnvCache {
@Getter private final FormStructure envFormStructure;

public JobEnvCache() {
OptionRule envOptionRules = EnvOptionRule.getEnvOptionRules();
OptionRule envOptionRules = new EnvOptionRule().optionRule();
envFormStructure =
SeaTunnelOptionRuleWrapper.wrapper(
envOptionRules.getOptionalOptions(),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

package org.apache.seatunnel.app.domain.response.connector;

import org.apache.seatunnel.plugin.discovery.PluginIdentifier;
import org.apache.seatunnel.api.common.PluginIdentifier;

import lombok.AllArgsConstructor;
import lombok.Data;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ public class EngineServiceImpl extends SeatunnelBaseServiceImpl implements IEngi
Lists.newArrayList(
// new Engine("Spark", "2.4.0"),
// new Engine("Flink", "1.13.6"),
new Engine(EngineType.SeaTunnel, "2.3.8")));
new Engine(EngineType.SeaTunnel, "2.3.10")));

@Override
public List<Engine> listSupportEngines() {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ public long createJob(JobReq jobReq) throws CodeGenerateUtils.CodeGenerateExcept
.name(DEFAULT_VERSION)
.id(uuid)
.engineName(EngineType.SeaTunnel)
.engineVersion("2.3.8");
.engineVersion("2.3.10");
if (BusinessMode.DATA_INTEGRATION.equals(jobReq.getJobType())) {
builder.jobMode(JobMode.BATCH);
} else if (BusinessMode.DATA_REPLICA.equals(jobReq.getJobType())) {
Expand Down
Loading
Loading