TensorFlow Determinism

February 27, 2023 ยท View on GitHub

Announcements

Enabling Determinism in Latest TensorFlow Version

From TensorFlow version 2.8 onwards, op-determinism is enabled using tf.config.experimental.enable_op_determinism. For more information, see the official documentation. You may also want to refer to the detailed status of GPU-determinism in TensorFlow in this current repository.

Patch Changes

In version 0.4.0 of this package, the distribution name changed from tensorflow-determinism to framework-reproducibility and the package name changed from tfdeterminism to fwr13y. These changes reflect an intention going forward for this repo to increasingly support reproducibility in multiple deep learning frameworks.

Users of tfdeterminism.patch will need to use the to-be-deprecated fwr13y.d9m.tensorflow.patch and will be encouraged to migrate to using fwr13y.d9m.tensorflow.enable_determinism instead, which is intended to provide compatibility, indefinitely, with future versions of TensorFlow.

Best-Effort Determinism

Version 0.4.0 (and later versions) of this package include a function (fwr13y.d9m.tensorflow.enable_determinism) that can be applied to any version of TensorFlow to obtain the latest and best solutions, including any new patches (including for earlier versions of TensorFlow).

It may be instructive to view the code of this relatively simple function to see specifically how op-determinism is enabled in various versions of TensorFlow.

Installation

Note that, currently, you only need to install, import, and call into this package if you're using a version of TensorFlow for which there is a determinism patch available (and you're using a nondeterministic op that would be patched). There is currently no patch available for TensorFlow versions 2.7 or greater because the effect of the patches that were developed for earlier versions have been upstreamed into these newer versions of TensorFlow.

Use pip to install:

pip install framework-reproducibility

This will install a package that can be imported as fwr13y. The installation of framework-reproducibility will not automatically install TensorFlow. The intention of this is to allow you to install your chosen version of TensorFlow. You can import your chosen version of TensorFlow either before or after you import fwr13y. However, you must import your chosen version of TensorFlow before invoking any functionality in fwr13y.

Determinism Solutions in Various Versions of TensorFlow

For the latest status of GPU-determinism for different versions of TensorFlow (viewed from an op-by-op perspective), refer to the detailed status in this current repository. Although in the short-term, solutions may be deployed as patches for stock TensorFlow or via the NGC TensorFlow container images, the long-term intention and plan is to continue upstreaming all solutions into stock TensorFlow, or, more likely, implementing solutions directly in stock TensorFlow initially and only.

What follows in this section is a high-level view of the status of GPU-determinism in different versions of TensorFlow.

Stock TensorFlow Version 2.11+

The latest version of stock TensorFlow (version 2.11 at the time of writing) implements most of the currently-available GPU-deterministic op solutions. It is missing deterministic tf.sparse.sparse_dense_matmul, which is provided by NGC TF Docker image version 21.04+. Note that this specific solution is not robust and will never be upstreamed to stock TensorFlow.

The following Python code is running on a machine in which pip package tensorflow=2.11.0 has been installed correctly.

import tensorflow as tf
tf.keras.utils.set_random_seed(1)
tf.config.experimental.enable_op_determinism
# Now build your graph and train it

For more informatiom about tf.config.experimental.enable_op_determinism, see the official documentation.

Stock TensorFlow version 2.11 with GPU support can be installed as follows:

pip install tensorflow=2.11.0

The TensorFlow project includes detailed instructions for installing TensorFlow with GPU support.

If you would like to enable determinism while automatically taking advantage of any patches that might be released in this current repository (before they reach stock TensorFlow), you could, alternatively, enable determinism as follows:

import tensorflow as tf
import fwr13y.d9m.tensorflow as tf_determinism
tf.keras.utils.set_random_seed(1)
tf_determinism.enable_determinism()
# Now build your graph and train it

Note that, at the time of writing (2023-02-06), it seems unlikely that we'll release any more patches in this current repository for TensorFlow, preferring instead to immediately upstream enahancements to stock TensorFlow.

NVIDIA GPU Cloud (NGC) TensorFlow Docker Images

NGC TensorFlow Docker images, starting with version 19.06, implement GPU-deterministic op functionality. Version 19.12 (and beyond) also implements multi-algorithm deterministic cuDNN convolutions, which solves the problem of some layer configurations causing an exception to be thrown with the message "No algorithm worked!". Version 20.03 (and beyond) also implements deterministic backprop for bilinear resizing. Version 21.04 (and beyond) implements deterministic tf.sparse.sparse_dense_matmul.

In Python code running inside the container, deterministic ops can be enabled as follows:

import tensorflow as tf
import fwr13y.d9m.tensorflow as tf_determinism
# Set seeds
tf_determinism.enable_determinism()
# Now build your graph and train it

The following table shows which version of TensorFlow each NGC Docker image version is based on:

NGC TF Image VersionTensorFlow Version
19.061.13
19.07 - 19.101.14
19.11 - 20.011.15 / 2.0
20.02 - 20.031.15 / 2.1
20.06 - 20.081.15 / 2.2
20.09 - 20.121.15 / 2.3
21.02 - 21.051.15 / 2.4
21.06 - 21.081.15 / 2.5
21.09 - 21.121.15 / 2.6
22.01 - 22.021.15 / 2.7
22.03 - 22.051.15 / 2.8
22.06 - 22.091.15 / 2.9
22.10 - 22.121.15 / 2.10
23.01 -1.15 / 2.11

This table may be updated periodically, but will most likely not be completely up-to-date when you're reading this.

Note that, for now, the NGC TensorFlow container images continue to support a GPU-performance-optimized TensorFlow API version 1 variant (using a -tf1 docker image repository tag), for those who have not yet migrated to TensorFlow API version 2. The source code for this can be found at GitHub/NVIDIA/TensorFlow.

For information about pulling and running the NVIDIA NGC Docker images, see these instructions.

Stock TensorFlow Version < 2.1

Versions 1.14, 1.15, and 2.0 of stock TensorFlow implement a reduced form of GPU-deterministic op functionality, which must be supplemented with a patch provided in this repo. The following Python code is running on a machine in which pip package tensorflow-gpu=2.0.0 has been installed correctly and on which framework-reproducibility has also been installed (as shown in the installation section above).

import tensorflow as tf
import fwr13y.d9m.tensorflow as tf_determinism
# Set seeds
tf_determinism.enable_determinism()
# Now build your graph and train it

Stock TensorFlow with GPU support can be installed as follows:

pip install tensorflow-gpu=2.0.4

The TensorFlow project includes detailed instructions for installing TensorFlow with GPU support.

Additional Ingredients in the Determinism Recipe

Deterministic op functionality, such as that enabled by TF_DETERMINISTIC_OPS=1 (in stock TensorFlow versions up to and including 2.7) or by tf.config.experimental.enable_op_determinism (in stock TensorFlow version 2.8 and onwards), can only contribute to fully-deterministic operation of a model or training regime in the context of a deterministic system. The following are notes on various other items that may need to be addressed in order to ensure that your model trains or infers with perfect reproducibility.

Guidance on obtaining reproducible operation from TensorFlow, as well as the status as of 2021-08-06, can be found in these slides.

Various aspects of this section may be out-of-date or non-optimally maintained. Nevertheless, it has been retained in case you might find it useful. I recommend following the simpler instructions above first, and then exploring more deeply if determinism has not been achieved.

Seeds

You'll also need to set any and all appropriate random seeds:

SEED = 123
random.seed(SEED)
np.random.seed(SEED)
tf.random.set_seed(SEED)

At some point, possibly in TensorFlow version 2.8, it may have become sufficient to simply call tf.keras.utils.set_random_seed(SEED) instead.

You also may need to set the environment variable PYTHONHASHSEED to a reproducible value before you start the python process.

In the TensorFlow version 1 API, tf.random.set_seed was tf.set_random_seed.

In most models, the effect of setting tf.random.set_seed is to ensure that the trainable variables (the weights and biases) in your model are pseudorandomly initialized the same way each time. Every time tf.random.set_seed is called, with a particular seed value, the pseudorandom number generator that TensorFlow uses to initialize the trainable variables is reset ("seeded") deterministically according to that seed.

If your model is not training deterministically, a good starting point is to confirm that the trainable variables prior to training are set the same way on every run, this can be done by calling the following function after calling model.compile and before calling model.fit:

def summarize_keras_trainable_variables(model, message):
  s = sum(map(lambda x: x.sum(), model.get_weights()))
  print("summary of trainable variables %s: %.13f" % (message, s))
  return s

Assuming the the trainable variable are being reproducibly reset, the function can also be used after training has completed to confirm that the training was deterministic:

model.compile(..)
summarize_keras_trainable_variables(model, "before training")
model.fit(...)
summarize_keras_trainable_variables(model, "after training")

An equivalent function can be used for non-keras models. It might also be preferable to use a hash function rather than sum.

If you're using dropout, which introduces a pseudorandom dropout sequence during training, then to achieve deterministic results you will need to reset the pseudorandom number generator that is used to produce those dropout sequences. The pseudorandom dropout sequence introduced by tf.keras.Dropout layers will be reset by tf.random.set_seed.

If you would like to control the seed used for dropout independently of the seed used for trainable variable initialization, then you can call tf.random.set_seed just before training (e.g. just before calling model.fit but after constructing the model and running model.compile). However, if you would like to explicitly control the seed used for the dropout sequence, then you can specify it using the seed argument of the tf.keras.layers.Dropout constructor.

Note that the shuffle argument of the Keras model fit method defaults to True (enabled). This pseudorandom shuffling is controlled by a pseudorandom number generator that is also reset reproducibly by tf.random.set_seed, probably the same pseudorandom number generator that TensorFlow uses throughout.

For more information, including about the relationship between the global seed (set via tf.random.set_seed) and the seed parameter that some ops have, see the documentation for tf.random.set_seed. This documentation suggests that, for determinism, if tf.random.set_seed has been called then the seed parameter of other ops does not need to be set. This did not used to be true and there may still be instances where the seed parameter of a given op must be set, to obtain determinism, even after tf.random.set_seed has been called.

Dataset Sharding

If you're using tf.data.Dataset, you should not shard the dataset. This is achieved by either not calling the shard() method, or by setting its num_shards parameter to 1.

From at least TensorFlow 2.2 onward, if not earlier, tf.data.Dataset::shard appears to operate deterministically.

Data-Loader Parallelism

When the data-loader pipeline is stateful and is replicated into multiple asynchronous threads, the threads can interact with each other, resulting in non-deterministic operation of the overall data-loader functionality. The most common example of this is when pseudorandom number generation is used in the data-loader pipeline, such as for data augmentation. When the same underlying pseudorandom number generator state is used in all the threads, it will result in nondeterministic functionality. This happens when the default pseudorandom number generator (e.g. from numpy) is used in the data-loader. There are three solutions to this:

  1. Make sure that the instances of the objects operating in the parallel threads have their own pseudorandom number generator state. For example, see numpy.random.Generator.
  2. Generate the pseudorandom augmentation parameters, each set associated with each example, in a single thread and then pass them into a stateless, parallelized augmentation process.
  3. Run all the data-loader code in only one thread.

How you run all the data-loader code in only one thread depends on how you're running the data-loader. If you're using tf.keras.Model::fit() or tf.keras.Model::fit_generator() then workers should be set to not more than 1.

Since the validation process runs in the main thread, if the validation process uses the same stateful data pipeline as the training data-loader then these two processes will also run in separate threads and you'll wind up with the same problem. In this case, you need to set workers to 0 (zero), which will cause the data-loader to be run in the main thread.

If you're using tf.data.Dataset, it may not be possible to instantiate different pseudorandom number generator state for each of the parallel calls in methods such as map (controlled by num_parallel_calls). A deterministic solution is to serialize the pseudorandom generation of the data augmentation parameters (i.e. num_parallel_calls=1) and then feed these into a subsequent, parallelized augmentation stage (or stages). For more information, and example code, see github/NVIDIA/framework-determinism issue 36.

When deterministic ops are expected, version 2.7 of TensorFlow will force num_parallel_calls to 1 for map and interleave stages where a map_func contains stateful ops. This may reduce the performance of data-loaders built using tf.data.Dataset that have not been restructured as described above. A future version of TensorFlow may perform the required restructuring automatically.

The methods of tf.data.Dataset (such as both map and interleave) that have a num_parallel_calls parameter also have a deterministic parameter. When set to True, deterministic forces the elements to be produced in a repeatable order when num_parallel_calls is greater than 1; when set to False, the constraint is relaxed; if not set, then tf.data.Options.experimental_deterministic (which defaults to True) controls the behavior. Therefore, for determinism, don't set the deterministic parameter in any of the methods and also don't change tf.data.Options.experimental_deterministic from the default.

tf.image ops that use pseudorandom number generators (and therefore produce a different result every time they are executed), such as tf.image.sample_distorted_bounding_box, have stateless versions (e.g. tf.image.stateless_sample_distorted_bounding_box) which will always produce the same result every time they are executed with the same seed parameter. These ops may be useful in the creation of a deterministic data-loader. A random number per example could be created and added in a tf.data.Dataset stage with no parallelism (num_parallel_calls=1) and then this random number could be passed as the seed parameter of multiple, different stateless random image ops in later, parallelized (num_parallel_calls > 1) stages.

tf.data.experimental.enable_debug_mode can be called to quickly disable all potential sources of nondeterminism in subsequently defined tf.data input pipelines when running in eager mode, which makes ruling-out nondeterminism from a tf.data input pipeline easy and fast.

Update for stock TF versions 2.7 and 2.8: functionality has been, or is being, added to a new make_deterministic grappler pass that attempts to modify instances of tf.data.Dataloaders to guarantee deterministic operation when op-determinism is expected/enabled. Please refer to the notes for the MakeDeterministic class declaration for more information. The link provided is for a specific commit point, so reference the notes at the commit point that is relevant to you.

While Loop Parallelism

In TF1, the use of tf.while_loop when parallel_iterations is greater than 1 (note that 10 is the default) may introduce nondeterminism into model functionality. Additionally, the AutoGraph Transformations, that operate while compiling code into a graph when (TF2 API) tf.function (or the use of the @tf.function decorator) is used, may lead to loops being implemented using tf.while_loop and, therefore, parallelized.

The current work-around, to prevent this nondeterminism, is to use tf.autograph.experimental.set_loop_options inside the for loop, with parallel_iterations=1.

It has not yet been confirmed whether this nondeterminism is specific to operation on a GPU or if it is a general issue in TensorFlow.

In TF2, the configuration of parallel_iterations in a tf.while_loop does not affect the order of stateful operations, and therefore tf.while_loop can be used without setting parallel_iterations to 1.

Gradient Gating

In some TensorFlow API interfaces, it is possible to limit the amount of paralellism that is allowed during back-propagation calculations.

If used, tf.gradients (not supported in eager execution) should have its gate_gradients parameter set to True (the default is False).

The non-Keras, TF1 API optimizers, based on the tf.compat.v1.train.Optimizer, such as tf.compat.v1.train.AdamOptimizer, accept a gate_gradients parameter in their minimize and compute_gradient methods. If this is set to tf.compat.v1.train.Optimizer.GATE_NONE then there is an increased probability of introducing nondeterminism in the backprop. If not specified,gate_gradients defaults to tf.compat.v1.train.Optimizer.GATE_OP, which theoretically could lead to the introduction of nondeterminism, although I have not yet seen this happen in a real application. The setting of this parameter that minimizes parallelism in the backprop calculation (and leads to the lowest performance) is tf.compat.v1.train.Optimizer.GATE_GRAPH. If you've removed all other sources of nondeterminism and nondeterminism is still being introducted somewhere, then you could try this setting, if it's available to you. I don't recommend changing gate_gradients to GATE_GRAPH as a standard practice.

The Keras optimizers, such as tf.keras.optimizers.SGD, which are now the standard optimizers in the TF2 API, do not offer a gate_gradients parameter in the minimize or get_gradients methods. There is also no ability to control gradient gating on tf.GradientTape for calculation of gradients in eager execution.

The reduced availablilty of control of gradient gating in TF2, with eager execution and an increased reliance on the (often high-level) Keras interface, doesn't seem to be a real problem with respect to GPU-determinism.

Protobufs

When serializing a graph into a protobuf, you should pass True to the SetSerializationDeterministic method of the CodedOutputStream class.

Multi-GPU using Horovod

If you're using Horovod for multi-GPU training, you may need to disable Tensor Fusion (assuming that the nondeterminism associated with Tensor Fusion has not yet been resolved, see Horovod PR 1130):

os.environ['HOROVOD_FUSION_THRESHOLD']='0'

Multi-GPU using tf.distribute Strategies

Prior to TensorFlow version 2.3, when using tf.data.Dataset::shuffle with tf.distribute.MirroredStrategy (or perhaps any tf.distribute strategy), setting reshuffle_each_iteration=True introduces nondeterminism. This appears to have been fixed in TensorFlow version 2.3. See TF issue 38197 for more information.

The last time I checked, the seed parameter of the shuffle method needed to be set in order to obtain determinism, although the latest documentation for tf.random.set_seed suggests that this should not be necessary as long as tf.random.set_seed has been called.

Creating a tf.random.Generator under a tf.distribute.Strategy::scope() will result in different replicas getting different random number streams.

CPU

If you want to obtain determinism when your ops are running on the CPU, you may need to limit the number of CPU threads used. In the TF1 API, this can be acheived as follows:

config = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1,
                                  inter_op_parallelism_threads=1)
with tf.compat.v1.Session(config=config):
  ...

In the TF2 API, it can be achieved like this:

tf.config.threading.set_intra_op_parallelism_threads(1)
tf.config.threading.set_inter_op_parallelism_threads(1)

It should not be necessary to limit the number of CPU threads used when your ops are not running on the CPU (e.g. when they're running on a GPU).

Detailed Status of Determinism in TensorFlow and Beyond

Confirmed and likely sources of non-determinism, along with any existing solutions, are being tracked here.

GPU-Specific Sources of Non-Determinism

Historic GPU-Specific Sources of Non-Determinism

In the past, tf.math.reduce_sum and tf.math.reduce_mean operated nondeterministically when running on a GPU. This was resolved before TensorFlow version 1.12. These ops now function deterministically by default when running on a GPU.

Confirmed Current GPU-Specific Sources of Non-Determinism (With Solutions)

The information in this section has been moved to a separate Status of GPU-Determinism in TensorFlow page, and expanded.

Other Possible GPU-Specific Sources of Non-Determinism

Going beyond the above-mentioned sources, in the TensorFlow master branch on 2021-02-26, afer release 2.4, the following files call CUDA atomicAdd either directly or indirectly. This makes them candidates for the injection of nondeterminism.

  • dilation_ops_gpu.cu.cc
  • maxpooling_op_gpu.cu.cc
  • multinomial_op_gpu.cu.cc
  • scatter_functor_gpu.cu.h
  • stateful_random_ops_gpu.cu.cc
  • svd_op_gpu.cu.cc

Unless you are using TensorFlow ops that depend on these files (i.e. ops with similar names), then your model will not be affected by these potential sources of non-determinism.

Update: as of stock TensorFlow version 2.8, these ops have been enhanced to either operate deterministically on GPU or to throw an exception when used with the expectation of determinism. I have a task, which may never be completed, to update this documentation to reflect this accurately.

Beyond atomicAdd, there are ten other CUDA atomic functions whose use could lead to the injection of non-determinism, such as atomicCAS (the most generic, atomic compare and swap). Note also that the word 'atomic' was present in 167 files in the TensorFlow repo and some of these may be related to the use of CUDA atomic operations. It's important to remember that it's possible to use CUDA atomic operations without injecting non-determinism, and that, therefore, when CUDA atomic operations are present in op code, it doesn't guarantee that the op can inject non-determinism into the computation.

Sources of Non-Determinism in TensorFlow Unrelated to GPU

  • Issue 29101: Random seed not set in graph context of Dataset#map. This may have been resolved in version 1.14 of TensorFlow.
  • tf.while_loop with parallel_iterations > 1 (default is 10). See While Loop Parallelism

Sources of Non-Determinism Beyond TensorFlow

  • TensorRT timing-based kernel schedule. Each time an inference engine is generated, it could be slightly different, particularly if there is varying load on the machine used to run TensorRT. There is a solution planned for this.
  • Horovod Tensor Fusion. Work-around: disable Tensor Fusion by setting the environment variable HOROVOD_FUSION_THRESHOLD to '0'. See Horovod PR 1130.
  • Before this project started (in 2018), PyTorch was widely considered to have a more complete and coherent GPU-determinism story than TensorFlow. At the time of writing (2020-02-25), it is no longer clear that one framework is superior to the other in this regard. For more information about determinism in PyTorch, see the reproducibility documentation in the PyTorch repo.

This section catalogs relevant links.

TensorFlow Issues

GitHub issues in the TensorFlow project (probably an incomplete list):

NumberTitleDate OpenedStatus
2652Backward pass of broadcasting on GPU is non-deterministic2016-06-03Closed
2732Mention that GPU reductions are nondeterministic in docs2016-06-08Closed
13932Non-determinism from tf.data.Dataset.map with random ops2017-10-23Closed
16889Problems Getting TensorFlow to behave Deterministically2018-02-09Open
18037tf.sparse_tensor_dense_matmul makes small errors with
tf.float32 matrices on GPU		2018-03-27Closed
18096Feature Request: Support for configuring deterministic
options of cuDNN conv ...		2018-03-29Open
22398CUDA implementation of BiasAddGrad op is non-determinstic2018-09-19Closed
29101Random seed not set in graph context of Dataset#map2019-05-28Open
38151Test deterministic cuDNN CTC loss2020-04-01Open
38185Add GPU-deterministic back-prop for fused
softmax/cross-entropy ops		2020-04-02Open
38197Model not deterministic, even though
os.environ['TF_DETERMINISTIC_OPS'] = '1' set		2020-04-03Closed
39751Non-deterministic behaviour: tf.math.unsorted_segment_sum
uses CUDA Atomic Operations		2020-05-21Open
40514TFBertForSequenceClassification: Non-deterministic when
training on GPU ...		2020-06-16Closed
42033Add deterministic tf.image.crop_and_resize backprop2020-08-04Open
47174EfficientNet models from TensorFlow.Keras not being
reproducible on GPU		2021-02-15Open
51978D9m unimplemented exception for AUC metric and
TF_DETERMINISTIC_OPS		2021-09-13Closed
53771Deterministic selection of deterministic cuDNN
convolution algos removed in TF 2.5		2022-01-14Open
53792Use enable_op_determinism + Fixed seed + same
hardware still get diff results in 2.8.		2022-01-17Open
53846tf.data.experimental.sample_from_datasets
non-deterministic in multi-GPU		2022-01-20Open
54259Possible issue with tf.data.Dataset in 2.7
(but probably Petastorm-related)		2022-02-03Closed
54276Deterministic GPU impl of unsorted segment
reduction op not available on Windows		2022-02-04Open
54442Reproducible init of trainable variables (TVs)
even when the number of TVs changes		2022-02-17Open

GitHub issues in dependent or related projects (probably an incomplete list):

ProjectNumberTitleDate OpenedStatus
Keras12800Unable to get reproducible results using Keras / TF on GPU2019-05-07Closed
Tensorpack902How to run Tensorpack training with deterministic behavior2018-09-20Closed
transformers5603Non-deterministic training issue on GPU: TF-BERT2020-06-16Open
ONNX Runtime4611Inference on GPU is not deterministic2020-07-24Closed

TensorFlow Pull Requests

The following pull requests (and some inidividual commits) are those in the TensorFlow GitHub repo (github.com/tensorflow/tensorflow) that are directly related to this project. As we have discovered, 1.8% of all commits seem to reference, or have some relationship with, "determinism" or "deterministic". As of 2020-01-30, that was 1,391 commits.

This list is incomplete. For example, there are PRs and commits that I have recorded in a documentation update queue, for possible future inclusion here.

IDTitleStatusDate MergedVersion
24747Add cuDNN deterministic env variable (only
for convolution).		merged2019-01-151.14
25269Add deterministic cuDNN max-poolingmerged2019-01-301.14
25796Added tests for TF_CUDNN_DETERMINISTICmerged2019-02-221.14
c27901Add a decorator to disable autotuning during
test executions.		merged2019-03-131.14
29667Add release note about TF_CUDNN_DETERMINISTICmerged2019-08-061.14
31389Enhance release notes related to
TF_CUDNN_DETERMINISTIC		merged2019-08-071.14
31465Add GPU-deterministic tf.nn.bias_addmerged2019-10-172.1
32979Fix typo in release noteclosed
33483Fix small typo in v2.0.0 release notemerged2019-10-252.1
33803Enable tf.nn.bias_add python op tests
to work in eager mode		merged2020-02-122.2
33900Address problems with use_deterministic_cudnn
test decorator		merged2020-01-092.2
34887Add info about TF_DETERMINISTIC_OPS to v2.1
release notes		merged2019-12-092.1
34951Add multi-algorithm deterministic cuDNN
convolutions		merged2020-01-272.2
35006Fix version 2.1 release note regarding
TF_DETERMINISTIC_OPS		merged2019-12-202.1
e31951[XLA/GPU] Convert reduction into tree reduction
using padding		merged2020-01-072.2
8b7a31[XLA] Respect TF_DETERMINISTIC_OPS env variable
for reductions		merged2020-02-192.2
37377[XLA] follow-up on GPU-deterministic reductionsmerged2020-03-092.3
9e0961Use the CUDNN_CTC_LOSS_ALGO_DETERMINISTIC
algorithm ...		merged2020-03-102.3
38089Add reminder to test deterministic cuDNN CTC lossclosed
38509List deterministic op func bug fixes in v2.2
release notes		merged2020-04-152.2
39243GPU-deterministic tf.image.resize (bilinear)merged2020-09-222.4
44717Add to rel notes: deterministic
tf.image.resize (bilinear)		merged2020-11-132.4
47419Support all fp types in GPU SparseTensorDenseMatMulmerged2021-03-082.5
47749Add GPU determinisim for fp types in GPU
SparseTensorDenseMatMul		closed
47772Add segment reduction op exceptions for
GPU determinism		merged2021-03-182.5
47925Add softmax/cross-entropy op exceptions for
GPU determinism		merged2021-04-052.6
47974Add GPU implem of sparse segment reduction
ops		merged2021-05-052.6
48581Update release notes in branch r2.5closed
48688Add CPU-focused tests for fused
softmax/cross-entropy ops		merged2021-04-262.6
48905Add GPU excepts, CPU d9m, and tests to
crop_and_resize		merged2021-05-132.6
49178Add non-sparse softmax/xent GPU-determinismmerged2021-06-042.6
50070Add sparse softmax/xent GPU-determinismmerged2021-09-102.7
50135Factor core/kernels RequireDeterminism() into
library		merged2021-06-092.6
50355Add d9m-unimplemented exceptions to sparse/sparse
matmul		merged2021-06-232.6
50505Add d9m-unimplemented exception-throwing to fused
batch-norm		merged2021-07-082.7
50640Enhance r2.6 release notesmerged2021-07-082.6
0f7b11Add internal function to enable/disable op determinismmerged2021-07-262.7
51023Add unimplemented exception to nearest-neighbor
resizing		merged2021-08-022.7
a4b531Raise error if random ops used with determinism
without seed		merged2021-08-102.7
51140Add unimplemented exception to
tf.image.adjust_contrast		merged2021-08-192.7
51392Add GPU-deterministic segment reductionsclosed
5a51f1Add determinism checks & tests for
DebugNumericSummaryV2		merged2021-08-312.7
fc91e1Add make_deterministic grappler passmerged2021-09-032.7
51861Replacement for 51392 (w/ deterministic kernels
optionally enabled)		merged2021-09-072.7
eb95f1Add d9m checks+tests for ScatterNd,
ScatterNdUpdate, & TensorScatter		merged2021-09-092.7
51920Add d9m-unimplemented exception for
tf.nn.depthwise_conv2d		merged2021-09-152.7
03ba31Make GPU scatter ND ops deterministic by running them on CPUmerged2021-09-172.7
c0e2e1Handle MapAndBatch in make_deterministic
grappler pass		merged2021-09-102.7
f0e6c1Add determinism exception to DenseBincountmerged2021-09-222.7
52227Add determinism tests for tf.nn.ctc_lossmerged2021-10-052.7
52971Add op-determinism info to version 2.7
release notes		merged2021-11-102.7
4dc6d1Have GpuScatterExpander always rewrite
Scatter when op-d9m enabled		merged2021-12-162.8
53465Add v2.8 release notesmerged2021-12-222.8
ced761Fix issue where convolutions were not
deterministic		merged2022-01-192.8
53826r2.8 cherry-pick req: Fix issue where
convs were not deterministic		merged2022-01-202.8
53847Add release note about deterministic
selection of conv algos		merged2022-01-212.8
54119Add disable for depthwise-conv d9m-unimplemented
exception		merged2022-02-012.9
55657Add GPU-determinism to tf.nn.depthwise_conv2dmerged2022-04-262.10

Notes:

  1. These are individual commits.

Other TensorFlow Organization Pull Requests

These are relevant pull requests against repositories in github.com/tensorflow other than github.com/tensorflow/tensorflow

The following list is probably incomplete.

RepositoryNumberTitleDate OpenedStatus
community346RFC: Enhancing determinism in TF2021-01-19merged
community370RFC: [determinism] Improve list of ops in2021-03-19merged
community386RFC: [determinism] Add tf.nn.depthwise_conv2d to op list in2021-04-28open

PyTorch Pull Requests

The following list is definitely incomplete and should be moved to a PyTorch-specific section of this current repository.

IDTitleStatusDate MergedVersion
33795Enhance reproducibility documentationmerged2020-03-061.5

Horovod Pull Requests

The following list is definitely incomplete and should be moved to a section of this current repository which is focused on determinism beyond TensorFlow.

IDTitleStatusDate MergedVersion
1130Add grouped allreduce featureOpen

Miscellaneous

The following list is almost certainly incomplete.