NLP Tasks

  • Sentiment analysis: is a text positive or negative?
  • Text generation (in English): provide a prompt and the model will generate what follows.
  • Name entity recognition (NER): in an input sentence, label each word with the entity it represents (person, place, etc.)
  • Question answering: provide the model with some context and a question, extract the answer from the context.
  • Filling masked text: given a text with masked words (e.g., replaced by [MASK]), fill the blanks.
  • Summarization: generate a summary of a long text.
  • Translation: translate a text in another language.
  • Feature extraction: return a tensor representation of the text.

PyTorch Data Parallel Modes

  • Data Parallel (accelerator=’dp’) (multiple-gpus, 1 machine)
  • DistributedDataParallel (accelerator=’ddp’) (multiple-gpus across many machines (python script based)).
  • DistributedDataParallel (accelerator=’ddp_spawn’) (multiple-gpus across many machines (spawn based)).
  • DistributedDataParallel 2 (accelerator=’ddp2’) (DP in a machine, DDP across machines).
  • Horovod (accelerator=’horovod’) (multi-machine, multi-gpu, configured at runtime)
  • TPUs (tpu_cores=8|x) (tpu or TPU pod)