SOK DLRM Benchmark

This is the SOK DLRM benchmark on Criteo Terabyte dataset.

Prepare Dataset

git clone https://gitlab-master.nvidia.com/dl/hugectr/hugectr.git
cd hugectr/
cd sparse_operation_kit/documents/tutorials/DLRM_Benchmark/
# train_data.bin and test_data.bin is the binary dataset generated by hugectr
# $DATA is the target directory to save the splited dataset
python3 split_bin.py train_data.bin $DATA/train
python3 split_bin.py test_data.bin $DATA/test

Environment

docker run --privileged=true --gpus=all -it --rm nvcr.io/nvidia/merlin/merlin-tensorflow-training:21.11
git clone https://gitlab-master.nvidia.com/dl/hugectr/hugectr.git
cd hugectr/
cd sparse_operation_kit/
mkdir build
cd build
# use "-DSM=70" in V100
cmake -DSM=80 ..
make -j
make install
cd ..
cp -r sparse_operation_kit /usr/local/lib/

Run Benchmark

cd hugectr/sparse_operation_kit/documents/tutorials/DLRM_Benchmark/
# FP32 Result
horovodrun -np 8 ./hvd_wrapper.sh python3 main.py --data_dir=($DATA) --global_batch_size=65536 --xla --compress --custom_interact --eval_in_last

# AMP result
horovodrun -np 8 ./hvd_wrapper.sh python3 main.py --data_dir=($DATA) --global_batch_size=65536 --xla --amp --custom_interact --eval_in_last

Performance

8 x A100 (82GB embedding table):

batch size	exit criteria	frequent of evaluation	xla	custom interact	amp	compress	training time (minutes)	evaluating time (minutes)	total time (minutes)	average time of iteration (ms)	throughput(samples/second)
65536	1 epoch	at end	yes	yes	no	yes	5.93	0.09	6.02	5.55	12.08M
65536	1 epoch	at end	yes	yes	yes	no	5.06	0.07	5.13	4.74	14.51M
55296	AUC > 0.8025	every 3793 steps	yes	yes	no	yes	5.23	1.44	6.67	4.87	11.66M
55296	AUC > 0.8025	every 3793 steps	yes	yes	yes	no	4.99	1.26	6.25	4.64	12.50M

8 x V100 (82GB embedding table):

batch size	exit criteria	frequent of evaluation	xla	custom interact	amp	compress	training time (minutes)	evaluating time (minutes)	total time (minutes)	average time of iteration (ms)	throughput(samples/second)
65536	1 epoch	at end	yes	yes	no	yes	17.52	0.19	17.71	16.42	4.02M
65536	1 epoch	at end	yes	yes	yes	no	10.20	0.15	10.35	9.56	6.99M
55296	AUC > 0.8025	every 3793 steps	yes	yes	no	yes	16.45	3.59	20.04	14.45	3.85M
55296	AUC > 0.8025	every 3793 steps	yes	yes	yes	no	9.69	2.54	12.23	8.52	6.62M

Note: The custom interact op can be seen in here, be careful to clone the repo from https://github.com/NVIDIA/DeepLearningExamples/, instead of cloning it from https://gitlab-master.nvidia.com/wraveane/tensorflow-dot-based-interact as written in the README.

Profile

cd hugectr/sparse_operation_kit/documents/tutorials/DLRM_Benchmark/
nsys profile --sample=none --backtrace=none --cudabacktrace=none --cpuctxsw=none --trace-fork-before-exec=true horovodrun -np 8 ./hvd_wrapper.sh python3 main.py --data_dir=($DATA) --global_batch_size=65536 --xla --compress --custom_interact --early_stop=30