Python package upgrade & troubleshooting
Description
- Upgrading Python or its packages is a complex process requiring significant attention and time
- This document aims to provide a step-by-step guide for upgrading Python or its packages
- Several steps are outlined to be followed to avoid any potential issues
- The primary objective is to streamline the upgrade process, ensuring it unfolds seamlessly while mitigating any potential issues that might arise along the way
Building the local image
- Upgrade the packages versions in the /devops/docker_build/pyproject.toml
- After the upgrade, the first step is to build the local image
- The build command is described in the /docs/work_tools/all.docker.how_to_guide.md#multi-architecture-build
Run the tests
- Create a new task-specific gdoc, e.g., CmTask7256_Upgrade_Pandas.docx
- Run all the tests from the orange repo against the new dev image
- Fast
- Slow
- Superslow
- QA
- Example:
i run_fast_tests --stage local --version {new version}
Document Errors
- Collect all the failures in the GDOC:
- Test name
- Traceback
- Group errors by type, e.g.,
# Error type 1 ## Test name 1 traceback ## Test name 2 traceback # Error type 2 ## Test name 1 traceback ## Test name 2 traceback # Error type N ...
Identify forward-compatible fixes
Forward-compatible fix is a fix that works on both versions of a Python package, i.e. on the current one and the target one.
- Identify forward-compatible changes
- Move them to a separate GDOC section to isolate from the other changes
- For each forward-compatible fix file a separate GH issue and a separate PR to master
Handle non-forward compatible fixes
- File an issue for the error
- Describe the solution in the GDOC
- Apply changes to the main PR
- Run the tests for this error type and make sure they do not fail
- Repeat this process for all the types of errors
- Once resolved, run all the regressions locally on the new local image from the orange repo and make sure the regressions are green
Image Release
- Run release command described in the /docs/work_tools/all.docker.how_to_guide.md#command-to-run-the-release-flow
- Follow the post-release check-list in the /docs/work_tools/all.docker.how_to_guide.md#post-release-check-list
Merge Base PR
- Run the regressions from GitHub on the main PR
- Check that the GitHub Actions is picking up new image and all regressions
(fast, slow, superslow) are green on
cmamp,orangeandkaizenflow - Merge the main PR on all the repos (
cmamp,orange,kaizenflow)