PyTorch 1.10 is now ready for production and offers a rich ecosystem of tools and libraries for deep learning, computer vision, natural language processing and more. Here’s how you can get started with PyTorch. […]
PyTorch (further articles in English) is an open source machine learning framework that is used for both research prototypes and production use. According to the source code repository, PyTorch provides two important features:
- Tensor calculations (like NumPy) with strong GPU acceleration.
- Deep neural networks built on a tape-based autograd system.
Originally developed at Idiap Research Institute, NYU, NEC Laboratories America, Facebook and Deepmind Technologies with the support of the Torch and Caffe2 projects, PyTorch now has a thriving open source community. PyTorch 1.10, released in October 2021, has commits from 426 contributors, and the repository currently has 54,000 stars.
This article gives an overview of PyTorch, including new features in PyTorch 1.10 and a short guide to getting started with PyTorch.
The evolution of PyTorch
Early on, academics and researchers were attracted to PyTorch because it was easier to use for model development with graphics processing units (GPUs) than TensorFlow. PyTorch is set to Eager Execution Mode by default, which means that its API calls are executed when they are called, instead of being added to a graph to be executed later.
TensorFlow has since improved its support for eager execution mode, but PyTorch is still very popular in the academic and research community.
PyTorch is now ready for production and allows you to easily switch between eager and graph mode with TorchScript and speed up the way to production with TorchServe. The torch.distributed backend enables scalable distributed training and performance optimization in research and production, and a rich ecosystem of tools and libraries extends PyTorch and supports development in the areas of computer vision, natural language processing and more.
Furthermore, PyTorch is supported by the most important cloud platforms, including Alibaba, Amazon Web Services (AWS), Google Cloud Platform (GCP) and Microsoft Azure. Cloud support enables smooth development and easy scaling.
What’s new in PyTorch 1.10
According to the PyTorch blog, the updates to PyTorch 1.10 focused on improving training and performance, as well as ease of use for developers. For details, see the PyTorch 1.10 release Notes. Here are just a few highlights of this version:
- CUDA Graphs APIs have been integrated to reduce CPU overhead for CUDA workloads.
- Multiple front-end APIs like FX, torch.special and nn.Module parameterization has been switched from beta to Stable. FX is a Pythonic platform for converting PyTorch programs; torch.special implements special functions such as gamma and Bessel functions.
- A new LLVM-based JIT compiler supports automatic fusion on both CPUs and GPUs. The LLVM-based JIT compiler can merge sequences of torch library calls to improve performance.
- Android NNAPI support is now available as a beta version. NNAPI (Android’s Neural Networks API) allows Android applications to run compute-intensive neural networks on the most powerful and efficient parts of the chips powering mobile phones, including GPUs and specialized neural processing units (NPUs).
The PyTorch 1.10 version contained over 3,400 commits, which indicates a very active project that focuses on improving performance through a variety of methods.
Getting started with PyTorch
Reading the version updates will not bring you much if you do not understand the basics of the project or do not know how to use it.
The PyTorch tutorial page offers two ways: one for those who are familiar with other deep learning frameworks and one for newbies. If you need the beginner track, which introduces tensors, records, autograd and other important concepts, I suggest that you follow it and use the Run option in Microsoft Learn, as shown in Figure 1.
If you are already familiar with deep learning concepts, I recommend that you run the Quickstart notebook shown in Figure 2. You can also click on “Run in Microsoft Learn” or “Run in Google Colab” or use the notebook locally.
Interesting PyTorch projects
As can be seen on the left side of the screenshot in Figure 2, PyTorch has numerous recipes and tutorials. In addition, there are numerous models and examples of their use, usually in the form of notebooks. Three projects from the PyTorch ecosystem seem particularly interesting to me: Captum, PyTorch Geometric (PyG) and skorch.
Captum
As noted on the GitHub repository of this project, the word captum in Latin means “understand”. As described on the repository page and elsewhere, Captum is “a model interpretation library for PyTorch”. It contains a variety of gradient- and perturbation-based attribution algorithms that can be used to interpret and understand PyTorch models. It also provides quick integration for models created with domain-specific libraries such as torchvision, torchtext, and others.
PyTorch Geometric (PyG)
PyTorch Geometric (PyG) is a library that data scientists and others can use to write and train neural graph networks for applications related to structured data. As explained on his GitHub repository page:
PyG offers methods for deep learning on graphs and other irregular structures, also known as geometric deep learning. In addition, it consists of easy-to-use mini-batch loaders for processing many small and single huge graphs, multi-GPU support, distributed graph learning via Quiver, a large number of popular benchmark datasets (based on simple interfaces for creating your own data), the GraphGym experiment manager and helpful transformations, both for learning on any graphs and on 3D networks or point clouds.
skorch
skorch is a Scikit-learn-compatible neural network library that wraps PyTorch. The goal of skorch is to enable the use of PyTorch with sklearn. If you are familiar with sklearn and PyTorch, there is no need to learn new concepts, and the syntax should be well known to you. In addition, Skorch abstracts the training loop, which eliminates the need for a lot of standard code. A simple net.fit(X, y) is sufficient, as shown in Figure 5.
Conclusion
Overall, PyTorch is one of the few top-notch deep neural network frameworks with GPU support. You can use it for model development and production, you can run it on-premises or in the cloud, and you can find a lot of ready-made PyTorch models that you can use as a starting point for your own models.
*Martin Heller is a contributing editor and reviewer at InfoWorld. As a former consultant for web and Windows programming, he developed databases, software and websites from 1986 to 2010. Most recently, he served as Vice President of Technology and Education at Alpha Software and as Chairman and CEO at Tubifi.
Outsourced Software Development Services | Dedicated Software Development Team
