Deploy Tez Client Plugin & Configurations
April 11, 2025 ยท View on GitHub
Deploy Tez Client Plugin
- Append client jar to package which is set by 'tez.lib.uris'.
In production mode, you can append client jar (rss-client-tez-XXXXX-shaded.jar) to package which is set by 'tez.lib.uris'.
In development mode, you can append client jar (rss-client-tez-XXXXX-shaded.jar) to HADOOP_CLASSPATH.
- Update tez-site.xml to enable Uniffle.
| Property Name | Default | Description |
|---|---|---|
| tez.am.launch.cmd-opts | -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC org.apache.tez.dag.app.RssDAGAppMaster | enable remote shuffle service |
| tez.rss.coordinator.quorum | coordinatorIp1:19999,coordinatorIp2:19999 | coordinator address |
Note that the RssDAGAppMaster will automatically disable slow start (i.e., tez.shuffle-vertex-manager.min-src-fraction=1, tez.shuffle-vertex-manager.max-src-fraction=1).
Tez Specific Configurations
| Property Name | Default | Description |
|---|---|---|
| tez.rss.avoid.recompute.succeeded.task | false | Whether to avoid recompute succeeded task when node is unhealthy or black-listed |
| tez.rss.client.max.buffer.size | 3k | The max buffer size in map side. Control the size of each segment(WrappedBuffer) in the buffer. |
| tez.rss.client.batch.trigger.num | 50 | The max batch of buffers to send data in map side. Affect the number of blocks sent to the server in each batch, and may affect rss_worker_used_buffer_size |
| tez.rss.client.send.thread.num | 5 | The thread pool size for the client to send data to the server. |
| tez.shuffle.mode | remote | Use Remote Shuffle if the value is set to 'remote' or use default config value, or set 'local' to use local shuffle when needs to fall back. |
Remote Spill (Experimental)
In cloud environment, VM may have very limited disk space and performance. This experimental feature allows to reduce tasks to spill data to remote storage (e.g., hdfs)
| Property Name | Default | Description |
|---|---|---|
| tez.rss.reduce.remote.spill.enable | false | Whether to use remote spill |
| tez.rss.remote.spill.storage.path | The remote spill path | |
| tez.rss.reduce.remote.spill.replication | 1 | The replication number to spill data to Hadoop FS |
| tez.rss.reduce.remote.spill.retries | 5 | The retry number to spill data to Hadoop FS |
Notice: this feature requires the MEMORY_LOCALFILE_HDFS storage type mode.