Write a function to implement chunked dataset streaming for out-of-memory training on massive corpora

Question

May i know how to write a function to implement chunked dataset streaming for out-of-memory training on massive corpora.