Start with this issue:

https://github.com/tensorflow/tensorflow/issues/9400

This can explain why I use official resnet model and cannot do memory growth and set memory limit.